Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Sun, 10 Jan 2016 11:33:22 +0100
From: Szabolcs Nagy <>
Subject: Re: Possible infinite loop in qsort()

* Rich Felker <> [2016-01-09 23:05:16 -0500]:
> On Sat, Jan 09, 2016 at 10:07:19AM +0100, Felix Janda wrote:
> > Markus Wichmann wrote:
> > > This is the Leonardo number precompute loop in qsort():
> > > 
> > >    for(lp[0]=lp[1]=width, i=2; (lp[i]=lp[i-2]+lp[i-1]+width) < size; i++);
> > > 
> > 
> > musl enforces that object sizes should not be greater than PTRDIFF_MAX.
> > See for example the discussion at
> > 
> >
> > 
> > So there will not be objects of size 3GB with musl on x32. Since the
> > Leonardo numbers grow slower than 2^n in general no overflow should
> > happen if "size" is valid. Otherwise, UB was invoked.
> Note also that if you do want to use this code on an implementation
> without such a guarantee, only the case where the member size is 1 can
> possibly have >SIZE_MAX/2 members. In that case, you can massively
> optimize out the whole sort by just counting the number of times each
> byte appears (in size_t[UCHAR_MAX+1] space which is tiny), sorting the
> pairs (value,count) using the comparison function, then writing out
> each value the appropriate number of times.

one element array would have to be special cased too:

qsort(p, 1, SIZE_MAX, cmp)

would oob access lp

in musl

qsort(p, 1, SIZE_MAX/2, cmp)

would loop for more than necessary before lp[i] reaches >=size
but it's harmless because lp is large enough.

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.