Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Wed, 14 Sep 2016 12:32:53 +0200
Subject: Re: incompatibility between libtheora/mmx and musl ?

On Tue, Sep 13, 2016 at 04:43:04PM -0400, Rich Felker wrote:
> > Libtheora contains an assembler part for ia32 and x86_64 which has been
> > in use for many years, with at least both glibc and uclibc.
> > 
> > Now, musl-based builds of libtheora for ia32 with this code enabled
> > lead to the encoder segfaulting. No problem when using the C version.

> >
> > 
> > There is also a corresponding
> > 
> >
> The most likely explanation is that they're overflowing a heap buffer.
> Perhaps it would be possible to pad all their malloc/realloc calls
> with +64 or so to see if that makes the problem go away. If so that
> gives a good starting point for tracking down the bug.

Thanks for the suggestion, indeed libtheora seems to consistently use
ogg_malloc() which is redefinable.

Building with

 #define _ogg_malloc(x)  malloc((x)+256)
 #define _ogg_calloc(x,y)  calloc((x)+256,(y))
 #define _ogg_realloc(y,x) realloc((y),(x)+256)
 #define _ogg_free    free

instead of the default

 #define _ogg_malloc  malloc
 #define _ogg_calloc  calloc
 #define _ogg_realloc realloc
 #define _ogg_free    free

did not make any difference. The crash on a test file occurs in the same
way and the resulting partial output file is as long as otherwise.

This may mean that this is not a simple overflowing but rather
overwriting or reading distant "random" places (?) (register corruption?)


Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.