Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date: Fri, 8 Aug 2014 13:19:52 -0400
From: Rich Felker <dalias@...c.org>
To: musl@...ts.openwall.com
Subject: Re: Replacing malloc

On Fri, Aug 08, 2014 at 04:15:29PM +0400, Alexander Monakov wrote:
> [changing topic, subject adjusted]
> 
> On Fri, 8 Aug 2014, Rich Felker wrote:
> > The fourth issue is much bigger: replacing malloc is UB and does not
> > work, especially not on musl. :-)
> 
> Whoa.  Let me ask for further clarifications.
> 
> You probably don't need me to tell you that most people expect that replacing
> malloc would in fact work, with at least two use cases in mind: a tracking
> wrapper around libc malloc (obtained via dlsym), or an entire custom allocator
> that obtains fresh memory via mmap.  So you can LD_PRELOAD a "malloc
> debugging library" or preload or even link against an alternative allocator.

A wrapper "works" as long as it does not expect to see all calls (it
won't see ones internal to libc). So if it's just for keeping stats on
application memory usage, it's probably fine. But if it's trying to
actually encapsulate the allocations (e.g. increase N and add its own
headers around them), you'll run into serious problems when you pass a
pointer from a wrapped malloc to an unwrapped free, or vice versa.

> Of course it's not without inherent issues.  If the alternative allocator
> provides malloc/realloc/calloc/free, it's going to see an unexpected but
> legitimate free when the application passes pointer obtained via
> posix_memalign.  Or when the application obtains a free()-able pointer via
> other libc functionality such as asprintf and the libc is linked in such a way
> that internal malloc calls are not interposable.  Or on glibc a malloc wrapper
> needs to handle malloc->dlsym->malloc recursion.

Right. I've raised similar issues with glibc, which attempts to
support malloc replacements but really doesn't (because of similar
issues). The new C11 aligned_alloc is again a problem since many/most
malloc replacements will fail to provide it. 

My view is that if they want to support this, they need to provide
detailed documentation for what's required to make it work, including
a list of symbols that need to be provided. This is complicated by
namespace clash issues. For instance providing mallinfo wrongly
imposes on the standard namespace, but not providing it might lead to
applications which want mallinfo calling the "wrong one".

Of course future-proofing this against new malloc-subsystem functions
yet to be added is also very difficult, and I don't know the right way
to do it. I just know that the current way is a time-bomb...

> I hope above you didn't mean to say that anybody wishing to use malloc
> wrappers or custom mmap-based malloc replacements on musl should abandon all
> hope, period; but merely that it is not for production use, and attempting to

It's definitely not for production use, at least not with arbitrary
versions of musl. It may be possible, with some testing and analysis,
when doing static linking with a known version of musl or in an
environment you have full control over (like an embedded system).

> do so should be with care, for instance if gnash library uses custom malloc,
> it may not return pointers to that memory to be free()'d by the main
> executable (calling libc's free).  But it would be like that on any libc.  So
> I have to wonder what "especially not on musl" stands for.

The situation used to be much worse, since many custom mallocs attempt
to use sbrk and thereby corrupt the state of the internal malloc's
heap. A while back we had a big discussion about this which ended in
disabling sbrk (making it always-fail with nonzero arguments). That
caught a lot of issues where applications were previously corrupting
the heap (now they immediately fail with ENOMEM messages, or they
fallback to mmap and work).

The remaining aspects to "especially not on musl" are related to the
fact that, unlike glibc, musl does not go out of its way to attempt to
support malloc replacements:

- For dynamic linking, musl always calls malloc/free directly, not
  symbolically, so it will not use your replacements. This matters
  especially for the dynamic linker that uses malloc before anything
  else is loaded (glibc's dynamic linker instead uses a temporary
  malloc at load time and switches later) and which uses malloc
  internals for being able to "donate back" unused writable memory
  from dynamic linking for use by malloc.

- For static linking, musl will use whatever malloc symbol appears
  first in the link order (there's no way around this). So the
  inconsistency in behavior may be surprising. Also, this means that
  if you call any of the memalign-type functions (including: if they
  get called internally! but I don't think they do) and your malloc
  replacement does not provide them, the internal memalign's
  assumptions about the heap structure will probably corrupt your
  malloc's state.

- Since sbrk intentionally does not work, some malloc implementations
  might not work at all.

I can't think of any other reasons it's _especially_ bad to try this
on musl right now, but there may be more.

> But so far every time I speak about a problem with musl the problem is
> deeper than I initially think -- so please clarify :)

:)

Basically, each implementation has its own manifestation of the UB of
replacing malloc. I think musl's current manifestation might be less
similar to application expectations than other implementations'. It's
also not a behavior that we document, support, or try to preserve
across versions; it's the way it is because the way it is currently
makes implementation internals the simplest.

Rich

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.