Date: Tue, 5 Aug 2014 12:51:37 -0400 From: Rich Felker <dalias@...c.org> To: musl@...ts.openwall.com Subject: Re: openmp/pthreads and fork... On Mon, Aug 04, 2014 at 10:56:51PM -0700, Isaac Dunham wrote: > Hello, > I've been packaging OpenBLAS for Alpine Linux, and an issue came to my > attention: > OpenBLAS (optionally) uses pthreads or OpenMP, and OpenMP uses pthreads. > OpenBLAS implements the BLAS (Basic Linear Algebra Subprograms) API/ABI, > like ATLAS, MKL, and so on. > Some programs that use BLAS will fork(); python is one of these. > With OpenBLAS, this had caused "random" segfaults due to use of threads > by the library both before and after the fork. > The OpenBLAS issue is here: > https://github.com/xianyi/OpenBLAS/issues/294 > > They worked around this using pthread_atfork() to cleanup before the fork(). This is unlikely to work. pthread_atfork is basically unable to do what it was designed for, since, after fork, the new thread in the child is not the owner of any mutexes that were obtained by the parent before forking, and thus cannot unlock them. It also cannot destroy and reinitialize them since they're locked. It may be possible to work around this by using semaphores instead of mutexes but I have not been able to get (and haven't really pushed for) a definitive answer on whether this is valid (there's a question whether using the forked copy of the semaphore is even valid; it's certainly not, formally, if the original process was multi-threaded since sem_wait is not AS-safe). This issue is somewhat documented in the rationale for pthread_atfork: http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_atfork.html But the way it's written, it's not at all clear that the result of the analysis there is that using pthread_atfork just doesn't work. I believe this is going to be changed in the next version of POSIX to make it clear. > I see some claims that calling any pthread functions in the child would be > UB, so I'm wondering about a few things: > -Is the last-mentioned claim correct? Yes. See: http://pubs.opengroup.org/onlinepubs/9699919799/functions/fork.html "A process shall be created with a single thread. If a multi-threaded process calls fork(), the new process shall contain a replica of the calling thread and its entire address space, possibly including the states of mutexes and other resources. Consequently, to avoid errors, the child process may only execute async-signal-safe operations until such time as one of the exec functions is called. Fork handlers may be established by means of the pthread_atfork() function in order to maintain application invariants across fork() calls." This is not stated clearly (what does "to avoid errors" mean? what does "may only" mean?), but the important part is "the child process may only execute async-signal-safe operations until such time as one of the exec functions is called". This is generally taken as implying that calls to any AS-unsafe function invoke UB. > -What is musl's behavior? musl makes no attept to synchronize malloc with fork, and doing so is difficult if you want to maintain the current requirement that fork be AS-safe (which is going to be removed in the next version of POSIX) and somewhat bloated even if not difficult. A similar issue applies to other code that uses locks internally -- stdio, time zone loading, dlopen, etc. Grepping for __lock or LOCK would find most if not all such uses. Also, many synchronization primitives (e.g. all mutexes except default/normal type) store their owner as a thread id, which is wrong after fork. So attempting to use them after fork, if they were locked before, is not going to work. That's about all I'm aware of that breaks after fork, but there may be more, and at present there's no intentional plan to avoid breaking more. > -How correct, and how likely to work with musl, is the fix for libgomp > mentioned here: > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=60035 Very unlikely since it's calling free which could run with malloc locks permanently deadlocked. > (and for that matter, the equivalent workaround referred to in OpenBLAS > issue 294)? > -Is there a safe (non-crashing) way to use/write libraries that might or > might not be built with threading--whether POSIX-specified or just > "working with musl" ? As nsz said, using fork in this kind of way is just unsafe. It was unsafe even without threads, but got a lot worse with threads. The fundamental issues are that forking results in all resources that were conceptually "owned" or "locked" by the parent having two owners after the fork, and certain state (e.g. process id, but also lots of other more subtle things) changes out from under the program in the child. Rich
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.