Date: Thu, 27 Aug 2015 20:38:16 -0700 From: Khem Raj <raj.khem@...il.com> To: musl@...ts.openwall.com Subject: Re: Adjustments to roadmap On Thu, Aug 27, 2015 at 7:43 PM, Rich Felker <dalias@...c.org> wrote: > Hey everyone, > > Between travel for a CII meeting last month (from which there's > exciting stuff I still need to follow up on, especially tlsify), work, > and making personal time to enjoy the summer, I've gotten rather > behind on musl releases. I think I'm caught up with everything > important for 1.1.11 though, and plan to release it ASAP. > > During this period of release delay, I haven't been updating the > roadmap on the wiki, but lots of new ideas have come up for enhancing > musl that may be appropriate to prioritize somewhere in the next few > release cycles. I'd like to share some of them here and get feedback > on what the community/users would like to see: > > ---------------------------------------------------------------------- > > 1. Unifying static/shared libc .o files. > > Right now, all the files in musl are built twice, once as .o for > inclusion in libc.a, and once as ".lo" (my naming convention) for .lo is conflicting with libtool generated objects btw. > inclusion in libc.so. This of course requires twice the time and space > to build, and also has semi-gratuitously separate code paths that need > to be maintained. I would like to eliminate that. > > One of the obvious big reasons for building libc.so's files separately > is that they need to be PIC. But for static-PIE support, which musl > now has and which I'm trying to get upstream in GCC/binutils, libc.a > needs to be built as PIC too. I did some tests, and the size increase > of libc.a by using PIC is about 1-3% on "important" archs and still > only 4-6% on some of the worst ones. I don't have figures on > performance cost, though, and it's harder to measure. > > If we do get rid of the dual-build, it will still be possible to build > a non-PIC libc.a via something like --disable-pic, which would imply > --disable-shared. It would just be a separate run of the build, which > is somewhat inconvenient now but much less so if we add out-of-tree > build support. And of course getting rid of the double-build in the > makefile would simplify it and make developing out-of-tree build > support easier. This seems a good proposal. We need some data on arches like mips > > I have more detailed notes on this topic I'll post later. > > 2. Rewriting malloc > > It came to my attention during this last release cycle that musl's > strategies for avoiding contention in malloc were leading to excessive > heap growth/fragmentation, contrary to my explicit intent. This was > the issue Timo Teräs reported for which I made a mitigation as commit > c3761622e8168b0c6453637ac82e70b09af3e8e9. > > I don't yet have a proposed long-term direction for malloc. Some brief > testing determined that the fine-grained locking we have now, despite > being theoretically of dubious benefit, actually does help; switching > to a single lock resulted in significant performance regressions. > > In addition, I've lost a good deal of faith in the whole "dlmalloc" > style heap-management approach. Worst-case fragmentation is expressed > by the fact that allocations as small as M/N can fail, where M is the > "total free memory" (total minus used) and N is the number of live > allocations. It should be possible to achieve much better bounds by > avoiding excessive mixing of different-sized chunks, and there are > known highly-inefficient ways to do this, but I haven't found any good > ones. > > What I'd like to do at the same time is eliminate most or all of the > need for merging of free chunks -- this is the main bottleneck to > concurrency. > > 3. Symbol versioning > > Right now musl's dynamic linker ignores symbol versioning information > except to select the "current" version of a symbol the same way ld > does. This is sufficient for most things, if you don't want to support > old library ABIs and mixed (i.e. broken) ABIs, but it breaks some > hacks GCC is doing in their target libs (libgcc_s.so, ...), and some > setups that arguably "should" work, especially for C++ apps. What's > probably worse is that you get silent breakage when an app is trying > to use symbol versioning, expecting it to work, and it doesn't. > > I don't think the status quo is a reasonable option. We should either > teach GCC that musl targets don't support symbol versioning, and make > sure apps/libs' build systems detect this, or we should make them > work. My leaning is towards the latter. we can detect using triplets so apps should be able to conflgure for musl. I think symbol versioning could be a good thing for musl for backward compatibilty with itself who knows what future holds for musl, we might have to deal with own past. but it should be a build time option, for glibc if we disable versioned symbols code size reduces significantly. > > This is not an endorsement of symbol versioning. It's a poor > approximation of the best possible solution to a problem that's not > fully solvable, and it DOES introduce the possibility of silent > breakage in most places where it's used, but there are ways to use it > safely for some limited purposes, and supporting these seems to be the > path of least headache. > > 4. Dynamic linker library chain changes > > There is some ambiguity of global symbol definition priority when a > library is first loaded as RTLD_LOCAL then later moved to the global > namespace with RTLD_GLOBAL, but basically I think what musl is doing > right now is "wrong" short of an official interpretation of the > standard to the contrary. > > If this is resolved, I'd like to rework how musl does its linked list > of shared libraries. Right now there's just one list with a flag for > each library indicating whether it's in the global namespace or not. > This forces searches for symbols to step through libraries that are > not relevant, and also forces TLS initialization for new threads to > step through all the libraries even if only a few actually have TLS. > > What I'd like to do is have 3 or more separate lists: one for all the > libraries (for gdb and dl_iterate_phdr to use), one representing the > global namespace, and one for just the libraries with TLS. This trades > a very small amount of memory in 'struct dso' for significant > performance improvements in programs with large numbers of libraries. > > ---------------------------------------------------------------------- > > In addition to the above, there are all of the existing roadmap items > on the wiki which are open for discussion of how they should be > prioritized. The big projects are roughly: > > - Atomics refactorization/deduplication > - Bits refactorization/deduplication > - Out-of-tree builds > - LC_COLLATE > - IDN > - Advanced glibc ABI-compat features in dynamic linker > - Documentation how about nommu ? > > Apologies for the slow progress lately. Don't worry though, there's > still lots more good stuff to come for musl. Some goals around code size might be interesting too. > > Rich
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.