Date: Wed, 12 Jan 2022 14:52:39 +0000 From: Harmen Stoppels <me@...menstoppels.nl> To: Rich Felker <dalias@...c.org> Cc: musl@...ts.openwall.com Subject: Re: Cannot dlopen() an already loaded shared library by its SONAME name > > > > 1. Julia  splits binary dependencies into separate packages, so when > > liba.so depends on libb.so, they live in a different dir, where > > the absolute and relative paths are only known when the julia > > interpreter has started, so neither rpaths or LD_LIBRARY_PATH can > > be used. > > So they dlopen libb.so, and then dlopen liba.so in that > > order, and then assume liba.so does not have to locate libb.so > > again, because its soname is already seens before. > > The proposed workaround was: don't list libb.so in the > > DT_NEEDED of liba.so (that is, if you're already doing the work of > > the linker, you might as well not use the linker at all for locating > > libs). However, being able to run executables shipped with julia > > packages would still be nice (e.g. a subprocess with LD_LIBRARY_PATH > > set properly) > > > > 2. The Nix / Guix / Spack people are trying to reduce startup time of > > executables with many shared libraries (as well as fixing library > > paths once and for all to keep executables run deterministically). > > In Guix there's a blog post where they call this the "stat storm" , > > and they solve it in a glibc patch: using context dependent ld.so.cache, > > that is, a reverse mapping soname => library path. > > In Nix the proposal to fix the "stat storm" is to replace DT_NEEDED > > in executables with absolute paths of all required libs (also > > transient ones). This works fine, except on musl, where a dlopen by > > soname will still do a search. > > This could be solved much better by making an application-specific > directory full of symlinks to the libraries it uses and putting that > directory as the first thing in the program binary's rpath. So the proposal is basically to replicate an application-specific ld.so.cache in the filesystem? Create a dir per ELF file, put its path in the RPATH, fill the dir with symlinks from soname => library. One problem with this is that $ORIGIN starts behaving differently compared to ld.so.cache. $ORIGIN is now relative to the directory of the symlink, not to the realpath of library. So if a library of a dependent package dlopen's a library in its own prefix by soname, relying on an rpath of say $ORIGIN/plugins, this will fail if the symlink dir is just a flat file list. So it means you'd effectively need to merge the prefixes, and this won't fly in Nix / Spack. Also it won't be a manageable solution for Julia, cause they require immutability of each prefix (and if they knew the relative path ahead of time, they wouldn't need this at all). > but having this happen on libraries without any SONAME is really an anti-feature. What do you mean? I think we're talking only about libraries that have a SONAME, do you mean dlopen-by-soname? So, do I understand correctly that loading a lib by path & putting its soname in a dict, so that future libs opened by soname can early exit is fine? But loading a lib by soname first, and then by path, and upon opening it happens to have a soname seen before, but is a different file (st_dev/st_ino), then it should continue with this lib, not early exit? Download attachment "publickey - me@...menstoppels.nl - 0xFD537C88.asc" of type "application/pgp-keys" (1815 bytes) Download attachment "signature.asc" of type "application/pgp-signature" (510 bytes)
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.