[<prev] [<thread-prev] [day] [month] [year] [list]
Message-ID: <6bbe5d8b-4f6a-4330-9ce9-d210ec508f34@infinite-source.de>
Date: Tue, 4 Nov 2025 00:51:16 +0100
From: The 8472 <the8472.rs@...inite-source.de>
To: Rich Felker <dalias@...c.org>, Alejandro Colomar <alx@...nel.org>
Cc: Thiago Macieira <thiago@...ieira.org>, Florian Weimer <fw@...eb.enyo.de>,
libc-alpha@...rceware.org, musl@...ts.openwall.com,
Arthur O'Dwyer <arthur.j.odwyer@...il.com>,
Jonathan Wakely <jwakely@...hat.com>
Subject: Re: Re: realloci(): A realloc() variant that works in-place
Hello,
On 03/11/2025 22:28, Rich Felker wrote:
> On Mon, Nov 03, 2025 at 10:36:07AM +0100, Alejandro Colomar wrote:
>> Hi Rich,
>>
>> On Sun, Nov 02, 2025 at 07:28:57PM -0500, Rich Felker wrote:
>>> On Mon, Nov 03, 2025 at 12:58:39AM +0100, Alejandro Colomar wrote:
>>>>> All this will need fine-tuning once implementations exist.
>>>>>
>>>>>> So, why not require the caller to not ask too much? We could go back to
>>>>>> reporting an error if there's not enough memory.
>>>>>>
>>>>>> Of course, it would still guarantee no errors when shrinking, but
>>>>>> I think we could error out when growing.
>>>>>
>>>>> I'd prefer no errors either way. If there isn't memory to grow the underlying
>>>>> space (a brk() system call returns ENOMEM), then realloci() returns as much as
>>>>> it could get but not more.
>>>>
>>>> The problem is that this is asking the implementation to speculate.
>>>>
>>>> Consider the case that a realloci() implementation knows that the
>>>> requested size fails. Let's put some arbitrary numbers:
>>>>
>>>> old_size = 10000;
>>>> requested_size = 30000;
>>>>
>>>> It knows the block can grow to somewhere between 10000 (which it
>>>> currently has) and 30000 (the system reported ENOMEM), but now it has
>>>> the task of allocating as much as it can get. Should it do a binary
>>>> search of the size? Try 20000, then if it fails try 15000, etc.?
>>>> That's speculation, and it would make this function too slow.
>>>
>>> I don't see any plausible implementation in which this involved a
>>> binary search. Either you have fixed-size slots in which case you just
>>> look at the size of the slot to see what the max obtainable is, or you
>>> have a dlmalloc-like situation where you check the size of the
>>> adjacent free block (if any) to determine the max obtainable. These
>>> are O(1) operations.
>>
>> I was thinking of mremap(2) without MREMAP_MAYMOVE.
>
> OK, this whole conversation is mixing up unrelated things:
>
> 1. In-place realloc to avoid relatively-expensive memcpy
> 2. In-place realloc to avoid updating pointers
>
> The case where mremap would be used is utterly irrelevant to (1). And
> further, the cost of the mremap operation is so high (syscall
> overhead, page table/TLB synchronization) that any cost of updating
> pointers because the object moved is dwarfed and thereby irrelevant
> too.
>
> So I don't see why anyone should care about this case.
>
> Moreover, I see (2) as entirely misguided. The whole provenance model
> makes it broken to try to rely on pointer values not changing, and no
> code should be trying to do that. A new allocator interface should not
> be pandering to this very fragile, very likely to be broken by
> compiler transformations, utterly backwards practice. Just treat the
> old pointer as invalid and always update like you're supposed to,
> regardless of whether the value is different.
>
> Rich
>
On the Rust side we have uses for both these scenarios, and more.
A) A strictly in-place realloc is useful for collections and
arenas that have outstanding borrows (thus cannot move)
but want to try growing in-place before they have to allocate
another chunk.
For those a metadata-update or mremap without MAYMOVE is fine.
B) Collections that want to resize and can change their pointer
but need custom data movement, i.e. not a plain memcpy from the old
to the new location. A VecDeque that needs to copy its front and
tail to different locations after a resize. A Vec wants to copy
fewer bytes than its allocated size.
In these cases mremap(MREMAP_MAYMOVE) is fine but memcpy should
be avoided and we would fallback to malloc + custom copy operations
+ free.
C) Alignment-changing reallocations, for example to go from a Box<[u8]>
to Box<[f32]>. In those cases mremap and memcpy are both fine but for
large allocations the former would be preferred.
Combinations are also possible, for example converting
a Box<str> to an Arc<str> requires alignment changes and custom
data rearrangment (B + C).
To cover those different cases jemalloc's API[0] seems
like good starting point:
void *rallocx(void *ptr, size_t size, int flags);
size_t xallocx(void *ptr, size_t size, size_t extra, int flags);
xallocx covers the strict in-place reallocation, including making
best-effort extension requests.
reallocx generally allows moving and already allows alignment-changes
through MALLOCX_ALIGN flags. The flags could be further extended
with MAY_MOVE (mremap) and MAY_COPY (memcpy) flags.
The 8472
[0] https://jemalloc.net/jemalloc.3.html
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
