Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Tue, 4 Sep 2012 08:57:21 +0200
From: magnum <>
Subject: Re: memory usage (was: [john-users] JTR against 135 millions MD5 hashes)

On 4 Sep, 2012, at 5:47 , Solar Designer <> wrote:

> Jim -
> On Mon, Sep 03, 2012 at 11:12:18PM -0400, wrote:
>> On Mon, Sep 3, 2012 at 8:49 PM, Solar Designer wrote:
>>> I did not realize that there were wasteful allocations in prepare() 
>>> and valid().  Weren't they temporary?
>> They were done using alloc_memory_tiny.  Now, I simply use static 
>> buffers.  When raw hashes were being used, and a few other cases, there 
>> could be one or more of these type allocations, which are very similar 
>> to memory leaks, when done on temp transient data.
> Did you include fixes for this in the fixes branch?  If not yet, can you
> do that?  I guess those changes are non-invasive.

We were not using mem_alloc_tiny but alloc/free so the problem was not so much wasted memory but fragmentation. Anyway, it is improved in all branches including -fixes.

>>> Another potential source of memory usage reduction are the alignments.
>>> For raw MD5 hashes, a 4-byte alignment should suffice (they're 4x4
>>> bytes), yet we were using 8-byte alignment on 64-bit builds.
>> Very good point. I had not even thought of things like these new 
>> alignment requirements.  I really think for most formats, 4 is the 
>> correct alignment, due to dereference of an arch_word_32 in some of the 
>> lookup functions.
>> I do not have time to fully address all of the issues brought up in your 
>> reply, but will try to look more in depth at this tomorrow afternoon.
> I think that for now our priority should be specifying proper alignments
> for each format's binary and salt (in bleeding).

I do not know of *any* format using larger alignment than ARCH_WORD_32 except for salts (where we sometimes pass a pointer).

BTW nearly all formats introduced since we implemented this got the DEFAULT_ALIGN so the problem has got worse instead of better. Maybe we should set DEFAULT_ALIGN to -1 and have the self-test bitch about any use of it (but this should be muted in releases).


Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.