Date: Sun, 10 May 2020 18:09:15 -0400 From: Rich Felker <dalias@...c.org> To: musl@...ts.openwall.com Subject: upstreaming mallocng Finishing initial development of and upstreaming mallocng is the main item for this release cycle, and I think we're close to there. The main question is whether there will be any serious performance or memory usage regressions relative to oldmalloc, which may be problematic for some users. With that in mind, I'm considering supporting a build-time choice of oldmalloc for at least the first couple releases with mallocng, so that there's an easy path back for users who are hit by any such regression that can't be solved immediately. If offering oldmalloc as an option, I think I'd like to apply the patch for the runaway heap expansion issue first: https://www.openwall.com/lists/musl/2019/04/12/4 This does cost some level of performance, but the loss should be well-understood. Arguably this should have been applied at the time I wrote it, but I wasn't really excited about the performance hit without having something better to offer in place of it. With that said, here are some of the things users should expect to see from mallocng: - More predictable behavior. Availability of memory should be influenced a lot less by history of allocator activity within the process's lifetime, especially on 64-bit archs where virtual address space fragmentation is not an issue. Non-essential patterns of ballooning fragmentation, like in glibc issue https://sourceware.org/bugzilla/show_bug.cgi?id=14581 which musl oldmalloc also exhibited, should also not happen anymore. - Memory getting returned to the system, aggressively unless it bounces back and forth enough that we back off to avoid pathologically bad performance. Software that builds then frees large data structures should expect to mostly return to usage levels from before the allocation after it's all freed, rather than getting stuck at near-peak usage like what happens with dlmalloc type allocator. - No more runaway heap expansion (see above thread). - Differences in performance characteristics. Expected to perform better at high usage, anywhere between noticably better and noticably worse at low usage levels. (Where worse is severe, efforts will be made to mitigate it as much as possible, but only if it's reported.) - Lower memory overhead at high usage. Ideally similar, but possibly somewhat higher, memory overhead at low usage. Tuning out all of the causes of gratuitous excess usage and over-eagar preallocation when the nominal usage is low, without clobbering performance, has been the big focus of recent development. - Byte-accurate malloc_usable_size. This is necessary to make accesses out to malloc_usable_size offset not produce UB in the eyes of the compiler/fortify, which may generate traps (e.g. for memset(p, 42, malloc_usable_size(p))). Now would be a great time for feedback on large-scale use of mallocng, if anyone's up for it. You can put it in LD_PRELOAD via .profile or even in init for system-wide use. Rich
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.