Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date: Mon, 16 May 2011 15:49:47 +0200
From: magnum <>
Subject: Re: binary hashes and BINARY_SIZE

On 2011-03-29 21:11, Alain Espinosa wrote:
> On 3/29/11, Solar Designer<>  wrote:
>> ... moving some code from cmp_one() to cmp_exact().  Alain?  Many
>> other "formats" are similarly not optimized in this respect.  Not that I
>> care much (with modern RAM sizes), but Alain started to compare memory
>> usage of different tools here:
>> ;-)
>> JtR's memory usage at NTLM will grow by 4 MB or 8 MB with the large hash
>> tables patch, but we can reduce it by 8 MB or 12 MB with the BINARY_SIZE
>> optimization.
> I make a rapid read and do not fully understand how john works. When i
> have some free time i read again and change the NTLM behavior. I think
> this 'john internal documentation' is very useful for john developers
> when starting a new format. Add it to the wiki or a 'How to support a
> new format'?
> I think memory usage/managment make an appreciable influence in
> performance which a large number of hashes.

As the current code only use 4 bytes for cmp_all() and cmp_one() I'm 
pretty sure a BINARY_SIZE of just 4 is plenty enough. cmp_exact() would 
have to be re-written so it creates a full NT hash from scratch and 
compares with that. This can use a copy of the (full) generic code as it 
doesn't need to be fast.

I had a go at re-arranging the current cmp_one() and cmp_exact in a way 
I believe is "more proper" (for the current binary size). I thought this 
wouldn't affect performance in any direction (as cmp_all() was not 
affected) but for some reason my benchmark dropped 5%. It must be one of 
those random how-stuff-ends-up cases.

Anyway, I enclose the code in case it helps someone think. I started to 
look at x86-64.S but that's way over my head. I could do it for 
crypt_all_generic as an example. This would be dead simple, just store b 
(instead of a, b, c, d) in the output array, making it a quarter of its 
current size. The SSE and MMX versions would need more work I guess, 
re-designing the buffers.

BTW I was experimenting the other day with defaulting to the old 
"non-fully-Unicode" crypt_all assembly code, and only use the new code 
when needed, but that too just made a performance drop, coincidental I 


View attachment "NT_fmt_cmp_fns.c" of type "text/x-csrc" (2741 bytes)

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.