Openwall GNU/*/Linux - a small security-enhanced Linux distro for servers
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date: Mon, 3 Sep 2012 23:12:18 -0400 (EDT)
From: jfoug@....net
To: john-dev@...ts.openwall.com
Subject: RE: memory usage (was: [john-users] JTR against 135
 millions MD5 hashes)

Thank you for the long and insightful write up.


On Mon, Sep 3, 2012 at 8:49 PM, Solar Designer wrote:
> Jim -
>
> Thank you for sharing your thoughts on this.
>
> On Mon, Sep 03, 2012 at 11:16:44AM -0400, jfoug@....net wrote:
>> The memory reduction due to the source() function, and also due to 
>> removing some wasteful allocations within the prepare() and valid() 
>> within dynamic.  I believe the wasteful allocations in dynamic have 
>> been removed, all the way back to the J7-RC.   The memory 
>> requirements were greatly reduced early on, but have slowly started 
>> to be lost some (the bitmaps).  However, total mem usage is still 
>> much less than it was, when we were keeping the full source buffer 
>> for each hash.
>
> I did not realize that there were wasteful allocations in prepare() 
> and
> valid().  Weren't they temporary?

They were done using alloc_memory_tiny.  Now, I simply use static 
buffers.  When raw hashes were being used, and a few other cases, there 
could be one or more of these type allocations, which are very similar 
to memory leaks, when done on temp transient data.

> Another potential source of memory usage reduction are the alignments.
> For raw MD5 hashes, a 4-byte alignment should suffice (they're 4x4
> bytes), yet we were using 8-byte alignment on 64-bit builds.

Very good point. I had not even thought of things like these new 
alignment requirements.  I really think for most formats, 4 is the 
correct alignment, due to dereference of an arch_word_32 in some of the 
lookup functions.

I do not have time to fully address all of the issues brought up in your 
reply, but will try to look more in depth at this tomorrow afternoon.

Jim.

Powered by blists - more mailing lists

Your e-mail address:

Powered by Openwall GNU/*/Linux - Powered by OpenVZ