Date: Mon, 19 Sep 2022 21:07:57 +0200 From: Gabriel Ravier <gabravier@...il.com> To: baiyang <baiyang@...il.com>, James Y Knight <jyknight@...gle.com>, musl <musl@...ts.openwall.com>, Florian Weimer <fweimer@...hat.com>, dalias@...c.org Subject: Re: The heap memory performance (malloc/free/realloc) is significantly degraded in musl 1.2 (compared to 1.1) On 9/19/22 20:14, Szabolcs Nagy wrote: > * baiyang <baiyang@...il.com> [2022-09-20 01:40:48 +0800]: >> I looked at the code of tcmalloc, but I didn't find any of the problems you mentioned in the implementation of malloc_usable_size (see: https://github.com/google/tcmalloc/blob/9179bb884848c30616667ba129bcf9afee114c32/tcmalloc/tcmalloc.cc#L1099 ). >> >> On the contrary, similar to musl, tcmalloc also directly uses the return value of malloc_usable_size in its realloc implementation to determine whether memory needs to be reallocated: https://github.com/google/tcmalloc/blob/9179bb884848c30616667ba129bcf9afee114c32/tcmalloc/tcmalloc.cc#L1499 >> >> I think this is enough to show that the return value of malloc_usable_size in tcmalloc is accurate and reliable, otherwise its own realloc will cause a segment fault. > obviously internally the implementation can use the internal chunk size... > > GetSize(p) is not the exact size (that the user allocated) but an internal > size (which may be larger) and that must not be exposed *outside* of the > malloc implementation (other than for diagnostic purposes). > > you can have 2 views: > > (1) tcmalloc and jemalloc are buggy because they expose an internal > that must not be exposed (becaues it can break user code). > > (2) user code is buggy if it uses malloc_usable_size for any purpose > other than diagnostic/statistics (because other uses are broken > on many implementations). > > either way the brokenness you want to support is a security hazard > and you are lucky that musl saves the day: it works hard not to > expose internal sizes so the code you seem to care about can operate > safely (which is not true on tcmalloc and jemalloc: the compiler > may break that code). While I would agree that using malloc_usable_size is generally not a great idea (it's at most acceptable as a small micro-optimization, but I would only ever expect it to be seen in very well-tested code in very hot loops, as it is indeed quite easily misused), it seems like a bit of a stretch to say that all of: - sqlite3 (https://github.com/sqlite/sqlite/blob/master/src/mem1.c) - systemd (https://github.com/systemd/systemd/blob/main/src/basic/alloc-util.h , along with all files using MALLOC_SIZEOF_SAFE, i.e. src/basic/alloc-util.c, src/basic/compress.c, src/basic/fileio.c, src/basic/memory-util.h, src/basic/recurse-dir.c, src/basic/string-util.c, src/libsystemd/sd-netlink/netlink-socket.c, src/shared/journal-importer.c, src/shared/varlink.c, src/test/test-alloc-util.c and src/test/test-compress.c) - rocksdb (https://github.com/facebook/rocksdb/blob/main/table/block_based/filter_policy.cc , along with at least 20 other uses) - folly (https://github.com/facebook/folly/blob/main/folly/small_vector.h) - lzham_codec (https://github.com/richgel999/lzham_codec/blob/master/lzhamdecomp/lzham_mem.cpp) - quickjs (https://raw.githubusercontent.com/bellard/quickjs/master/quickjs.c) - redis (https://github.com/redis/redis/blob/unstable/src/networking.c, along with a few other uses elsewhere) along with so many more well-known projects that I've given up on listing them, are all buggy because of their usage of malloc_usable_size...
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.