Date: Mon, 3 Sep 2012 23:12:18 -0400 (EDT) From: jfoug@....net To: john-dev@...ts.openwall.com Subject: RE: memory usage (was: [john-users] JTR against 135 millions MD5 hashes) Thank you for the long and insightful write up. On Mon, Sep 3, 2012 at 8:49 PM, Solar Designer wrote: > Jim - > > Thank you for sharing your thoughts on this. > > On Mon, Sep 03, 2012 at 11:16:44AM -0400, jfoug@....net wrote: >> The memory reduction due to the source() function, and also due to >> removing some wasteful allocations within the prepare() and valid() >> within dynamic. I believe the wasteful allocations in dynamic have >> been removed, all the way back to the J7-RC. The memory >> requirements were greatly reduced early on, but have slowly started >> to be lost some (the bitmaps). However, total mem usage is still >> much less than it was, when we were keeping the full source buffer >> for each hash. > > I did not realize that there were wasteful allocations in prepare() > and > valid(). Weren't they temporary? They were done using alloc_memory_tiny. Now, I simply use static buffers. When raw hashes were being used, and a few other cases, there could be one or more of these type allocations, which are very similar to memory leaks, when done on temp transient data. > Another potential source of memory usage reduction are the alignments. > For raw MD5 hashes, a 4-byte alignment should suffice (they're 4x4 > bytes), yet we were using 8-byte alignment on 64-bit builds. Very good point. I had not even thought of things like these new alignment requirements. I really think for most formats, 4 is the correct alignment, due to dereference of an arch_word_32 in some of the lookup functions. I do not have time to fully address all of the issues brought up in your reply, but will try to look more in depth at this tomorrow afternoon. Jim.
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.