[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date: Thu, 27 Aug 2015 22:43:48 -0400
From: Rich Felker <dalias@...c.org>
To: musl@...ts.openwall.com
Subject: Adjustments to roadmap
Hey everyone,
Between travel for a CII meeting last month (from which there's
exciting stuff I still need to follow up on, especially tlsify), work,
and making personal time to enjoy the summer, I've gotten rather
behind on musl releases. I think I'm caught up with everything
important for 1.1.11 though, and plan to release it ASAP.
During this period of release delay, I haven't been updating the
roadmap on the wiki, but lots of new ideas have come up for enhancing
musl that may be appropriate to prioritize somewhere in the next few
release cycles. I'd like to share some of them here and get feedback
on what the community/users would like to see:
----------------------------------------------------------------------
1. Unifying static/shared libc .o files.
Right now, all the files in musl are built twice, once as .o for
inclusion in libc.a, and once as ".lo" (my naming convention) for
inclusion in libc.so. This of course requires twice the time and space
to build, and also has semi-gratuitously separate code paths that need
to be maintained. I would like to eliminate that.
One of the obvious big reasons for building libc.so's files separately
is that they need to be PIC. But for static-PIE support, which musl
now has and which I'm trying to get upstream in GCC/binutils, libc.a
needs to be built as PIC too. I did some tests, and the size increase
of libc.a by using PIC is about 1-3% on "important" archs and still
only 4-6% on some of the worst ones. I don't have figures on
performance cost, though, and it's harder to measure.
If we do get rid of the dual-build, it will still be possible to build
a non-PIC libc.a via something like --disable-pic, which would imply
--disable-shared. It would just be a separate run of the build, which
is somewhat inconvenient now but much less so if we add out-of-tree
build support. And of course getting rid of the double-build in the
makefile would simplify it and make developing out-of-tree build
support easier.
I have more detailed notes on this topic I'll post later.
2. Rewriting malloc
It came to my attention during this last release cycle that musl's
strategies for avoiding contention in malloc were leading to excessive
heap growth/fragmentation, contrary to my explicit intent. This was
the issue Timo Teräs reported for which I made a mitigation as commit
c3761622e8168b0c6453637ac82e70b09af3e8e9.
I don't yet have a proposed long-term direction for malloc. Some brief
testing determined that the fine-grained locking we have now, despite
being theoretically of dubious benefit, actually does help; switching
to a single lock resulted in significant performance regressions.
In addition, I've lost a good deal of faith in the whole "dlmalloc"
style heap-management approach. Worst-case fragmentation is expressed
by the fact that allocations as small as M/N can fail, where M is the
"total free memory" (total minus used) and N is the number of live
allocations. It should be possible to achieve much better bounds by
avoiding excessive mixing of different-sized chunks, and there are
known highly-inefficient ways to do this, but I haven't found any good
ones.
What I'd like to do at the same time is eliminate most or all of the
need for merging of free chunks -- this is the main bottleneck to
concurrency.
3. Symbol versioning
Right now musl's dynamic linker ignores symbol versioning information
except to select the "current" version of a symbol the same way ld
does. This is sufficient for most things, if you don't want to support
old library ABIs and mixed (i.e. broken) ABIs, but it breaks some
hacks GCC is doing in their target libs (libgcc_s.so, ...), and some
setups that arguably "should" work, especially for C++ apps. What's
probably worse is that you get silent breakage when an app is trying
to use symbol versioning, expecting it to work, and it doesn't.
I don't think the status quo is a reasonable option. We should either
teach GCC that musl targets don't support symbol versioning, and make
sure apps/libs' build systems detect this, or we should make them
work. My leaning is towards the latter.
This is not an endorsement of symbol versioning. It's a poor
approximation of the best possible solution to a problem that's not
fully solvable, and it DOES introduce the possibility of silent
breakage in most places where it's used, but there are ways to use it
safely for some limited purposes, and supporting these seems to be the
path of least headache.
4. Dynamic linker library chain changes
There is some ambiguity of global symbol definition priority when a
library is first loaded as RTLD_LOCAL then later moved to the global
namespace with RTLD_GLOBAL, but basically I think what musl is doing
right now is "wrong" short of an official interpretation of the
standard to the contrary.
If this is resolved, I'd like to rework how musl does its linked list
of shared libraries. Right now there's just one list with a flag for
each library indicating whether it's in the global namespace or not.
This forces searches for symbols to step through libraries that are
not relevant, and also forces TLS initialization for new threads to
step through all the libraries even if only a few actually have TLS.
What I'd like to do is have 3 or more separate lists: one for all the
libraries (for gdb and dl_iterate_phdr to use), one representing the
global namespace, and one for just the libraries with TLS. This trades
a very small amount of memory in 'struct dso' for significant
performance improvements in programs with large numbers of libraries.
----------------------------------------------------------------------
In addition to the above, there are all of the existing roadmap items
on the wiki which are open for discussion of how they should be
prioritized. The big projects are roughly:
- Atomics refactorization/deduplication
- Bits refactorization/deduplication
- Out-of-tree builds
- LC_COLLATE
- IDN
- Advanced glibc ABI-compat features in dynamic linker
- Documentation
Apologies for the slow progress lately. Don't worry though, there's
still lots more good stuff to come for musl.
Rich
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
