[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20251104103138.GG3520958@port70.net>
Date: Tue, 4 Nov 2025 11:31:38 +0100
From: Szabolcs Nagy <nsz@...t70.net>
To: The 8472 <the8472.rs@...inite-source.de>
Cc: Rich Felker <dalias@...c.org>, Alejandro Colomar <alx@...nel.org>,
Thiago Macieira <thiago@...ieira.org>,
Florian Weimer <fw@...eb.enyo.de>, libc-alpha@...rceware.org,
musl@...ts.openwall.com, Arthur O'Dwyer <arthur.j.odwyer@...il.com>,
Jonathan Wakely <jwakely@...hat.com>
Subject: Re: Re: realloci(): A realloc() variant that works in-place
* The 8472 <the8472.rs@...inite-source.de> [2025-11-04 00:51:16 +0100]:
> On 03/11/2025 22:28, Rich Felker wrote:
> >
> > OK, this whole conversation is mixing up unrelated things:
> >
> > 1. In-place realloc to avoid relatively-expensive memcpy
> > 2. In-place realloc to avoid updating pointers
> >
> > The case where mremap would be used is utterly irrelevant to (1). And
> > further, the cost of the mremap operation is so high (syscall
> > overhead, page table/TLB synchronization) that any cost of updating
> > pointers because the object moved is dwarfed and thereby irrelevant
> > too.
> >
> > So I don't see why anyone should care about this case.
> >
> > Moreover, I see (2) as entirely misguided. The whole provenance model
> > makes it broken to try to rely on pointer values not changing, and no
> > code should be trying to do that. A new allocator interface should not
> > be pandering to this very fragile, very likely to be broken by
> > compiler transformations, utterly backwards practice. Just treat the
> > old pointer as invalid and always update like you're supposed to,
> > regardless of whether the value is different.
> >
> > Rich
> >
>
> On the Rust side we have uses for both these scenarios, and more.
>
> A) A strictly in-place realloc is useful for collections and
> arenas that have outstanding borrows (thus cannot move)
> but want to try growing in-place before they have to allocate
> another chunk.
>
> For those a metadata-update or mremap without MAYMOVE is fine.
how useful?
> B) Collections that want to resize and can change their pointer
> but need custom data movement, i.e. not a plain memcpy from the old
> to the new location. A VecDeque that needs to copy its front and
> tail to different locations after a resize. A Vec wants to copy
> fewer bytes than its allocated size.
>
> In these cases mremap(MREMAP_MAYMOVE) is fine but memcpy should
> be avoided and we would fallback to malloc + custom copy operations
> + free.
does it actually help or always falls back?
> C) Alignment-changing reallocations, for example to go from a Box<[u8]>
> to Box<[f32]>. In those cases mremap and memcpy are both fine but for
> large allocations the former would be preferred.
>
>
> Combinations are also possible, for example converting
> a Box<str> to an Arc<str> requires alignment changes and custom
> data rearrangment (B + C).
are these common operations?
> To cover those different cases jemalloc's API[0] seems
> like good starting point:
>
> void *rallocx(void *ptr, size_t size, int flags);
> size_t xallocx(void *ptr, size_t size, size_t extra, int flags);
>
>
> xallocx covers the strict in-place reallocation, including making
> best-effort extension requests.
>
> reallocx generally allows moving and already allows alignment-changes
> through MALLOCX_ALIGN flags. The flags could be further extended
> with MAY_MOVE (mremap) and MAY_COPY (memcpy) flags.
is there an analysis with actual numbers so one
can see the tradeoffs instead of speculating?
we know that users think a new api would be
useful, but some evidence is needed.
>
>
> The 8472
>
>
> [0] https://jemalloc.net/jemalloc.3.html
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
