Date: Tue, 4 Sep 2012 08:57:21 +0200 From: magnum <john.magnum@...hmail.com> To: john-dev@...ts.openwall.com Subject: Re: memory usage (was: [john-users] JTR against 135 millions MD5 hashes) On 4 Sep, 2012, at 5:47 , Solar Designer <solar@...nwall.com> wrote: > Jim - > > On Mon, Sep 03, 2012 at 11:12:18PM -0400, jfoug@....net wrote: >> On Mon, Sep 3, 2012 at 8:49 PM, Solar Designer wrote: >>> I did not realize that there were wasteful allocations in prepare() >>> and valid(). Weren't they temporary? >> >> They were done using alloc_memory_tiny. Now, I simply use static >> buffers. When raw hashes were being used, and a few other cases, there >> could be one or more of these type allocations, which are very similar >> to memory leaks, when done on temp transient data. > > Did you include fixes for this in the fixes branch? If not yet, can you > do that? I guess those changes are non-invasive. We were not using mem_alloc_tiny but alloc/free so the problem was not so much wasted memory but fragmentation. Anyway, it is improved in all branches including -fixes. >>> Another potential source of memory usage reduction are the alignments. >>> For raw MD5 hashes, a 4-byte alignment should suffice (they're 4x4 >>> bytes), yet we were using 8-byte alignment on 64-bit builds. >> >> Very good point. I had not even thought of things like these new >> alignment requirements. I really think for most formats, 4 is the >> correct alignment, due to dereference of an arch_word_32 in some of the >> lookup functions. >> >> I do not have time to fully address all of the issues brought up in your >> reply, but will try to look more in depth at this tomorrow afternoon. > > I think that for now our priority should be specifying proper alignments > for each format's binary and salt (in bleeding). I do not know of *any* format using larger alignment than ARCH_WORD_32 except for salts (where we sometimes pass a pointer). BTW nearly all formats introduced since we implemented this got the DEFAULT_ALIGN so the problem has got worse instead of better. Maybe we should set DEFAULT_ALIGN to -1 and have the self-test bitch about any use of it (but this should be muted in releases). magnum
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.