Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Mon, 19 Sep 2022 15:21:27 -0400
From: Rich Felker <>
To: Gabriel Ravier <>
Cc: baiyang <>, James Y Knight <>,
	musl <>, Florian Weimer <>
Subject: Re: The heap memory performance (malloc/free/realloc) is
 significantly degraded in musl 1.2 (compared to 1.1)

On Mon, Sep 19, 2022 at 09:07:57PM +0200, Gabriel Ravier wrote:
> On 9/19/22 20:14, Szabolcs Nagy wrote:
> >* baiyang <> [2022-09-20 01:40:48 +0800]:
> >>I looked at the code of tcmalloc, but I didn't find any of the problems you mentioned in the implementation of malloc_usable_size (see: ).
> >>
> >>On the contrary, similar to musl, tcmalloc also directly uses the return value of malloc_usable_size in its realloc implementation to determine whether memory needs to be reallocated:
> >>
> >>I think this is enough to show that the return value of malloc_usable_size in tcmalloc is accurate and reliable, otherwise its own realloc will cause a segment fault.
> >obviously internally the implementation can use the internal chunk size...
> >
> >GetSize(p) is not the exact size (that the user allocated) but an internal
> >size (which may be larger) and that must not be exposed *outside* of the
> >malloc implementation (other than for diagnostic purposes).
> >
> >you can have 2 views:
> >
> >(1) tcmalloc and jemalloc are buggy because they expose an internal
> >     that must not be exposed (becaues it can break user code).
> >
> >(2) user code is buggy if it uses malloc_usable_size for any purpose
> >     other than diagnostic/statistics (because other uses are broken
> >     on many implementations).
> >
> >either way the brokenness you want to support is a security hazard
> >and you are lucky that musl saves the day: it works hard not to
> >expose internal sizes so the code you seem to care about can operate
> >safely (which is not true on tcmalloc and jemalloc: the compiler
> >may break that code).
> While I would agree that using malloc_usable_size is generally not a
> great idea (it's at most acceptable as a small micro-optimization,
> but I would only ever expect it to be seen in very well-tested code
> in very hot loops, as it is indeed quite easily misused), it seems
> like a bit of a stretch to say that all of:
> - sqlite3 (
> - systemd
> (
> , along with all files using MALLOC_SIZEOF_SAFE, i.e.
> src/basic/alloc-util.c, src/basic/compress.c, src/basic/fileio.c,
> src/basic/memory-util.h, src/basic/recurse-dir.c,
> src/basic/string-util.c, src/libsystemd/sd-netlink/netlink-socket.c,
> src/shared/journal-importer.c, src/shared/varlink.c,
> src/test/test-alloc-util.c and src/test/test-compress.c)
> - rocksdb (
> , along with at least 20 other uses)
> - folly (
> - lzham_codec (
> - quickjs
> (
> - redis
> (,
> along with a few other uses elsewhere)
> along with so many more well-known projects that I've given up on
> listing them, are all buggy because of their usage of
> malloc_usable_size...

Depending on how you interpret the contract of malloc_usable_size
(which was historically ambigious), either (1) or (2) above is
*necessarily* true. It's not a matter of opinion just logical
consequences of the choice you make.

Moreover, it's not at all a stretch to say 7+ popular projects have
gigantic UB they don't care to fix. The whole story of musl has been
finding *hundreds* of such projects, and eventually getting lots of
them fixed.


Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.