[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Tue, 4 Sep 2012 08:57:21 +0200
From: magnum <john.magnum@...hmail.com>
To: john-dev@...ts.openwall.com
Subject: Re: memory usage (was: [john-users] JTR against 135 millions MD5 hashes)
On 4 Sep, 2012, at 5:47 , Solar Designer <solar@...nwall.com> wrote:
> Jim -
>
> On Mon, Sep 03, 2012 at 11:12:18PM -0400, jfoug@....net wrote:
>> On Mon, Sep 3, 2012 at 8:49 PM, Solar Designer wrote:
>>> I did not realize that there were wasteful allocations in prepare()
>>> and valid(). Weren't they temporary?
>>
>> They were done using alloc_memory_tiny. Now, I simply use static
>> buffers. When raw hashes were being used, and a few other cases, there
>> could be one or more of these type allocations, which are very similar
>> to memory leaks, when done on temp transient data.
>
> Did you include fixes for this in the fixes branch? If not yet, can you
> do that? I guess those changes are non-invasive.
We were not using mem_alloc_tiny but alloc/free so the problem was not so much wasted memory but fragmentation. Anyway, it is improved in all branches including -fixes.
>>> Another potential source of memory usage reduction are the alignments.
>>> For raw MD5 hashes, a 4-byte alignment should suffice (they're 4x4
>>> bytes), yet we were using 8-byte alignment on 64-bit builds.
>>
>> Very good point. I had not even thought of things like these new
>> alignment requirements. I really think for most formats, 4 is the
>> correct alignment, due to dereference of an arch_word_32 in some of the
>> lookup functions.
>>
>> I do not have time to fully address all of the issues brought up in your
>> reply, but will try to look more in depth at this tomorrow afternoon.
>
> I think that for now our priority should be specifying proper alignments
> for each format's binary and salt (in bleeding).
I do not know of *any* format using larger alignment than ARCH_WORD_32 except for salts (where we sometimes pass a pointer).
BTW nearly all formats introduced since we implemented this got the DEFAULT_ALIGN so the problem has got worse instead of better. Maybe we should set DEFAULT_ALIGN to -1 and have the self-test bitch about any use of it (but this should be muted in releases).
magnum
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
