Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Sun, 18 Dec 2011 22:56:34 +0100
From: magnum <>
Subject: Re: 1.7.9-jumbo

On 12/18/2011 08:22 PM, Solar Designer wrote:
> On Sun, Dec 18, 2011 at 07:41:07PM +0100, magnum wrote:
>> Memory comes back to me now... for the non-utf8 versions of set_key(),
>> the 1.7.8 code was buggy. It truncated at 28 instead of 27 (but this was
>> only a problem when running external mode). I have a patch somewhere
>> that I can post soon, that re-introduces the tests as if external.c was
>> not truncating, but that does it in a more effective way - and properly
>> at max length. And if 1.7.8 would still be a tad faster after that, I
>> will just say that's because it was buggy.
> Are you saying that 1.7.8-jumbo's benchmark results for NT were
> inflated because of a bug (I don't see how this can be the case...),
> and that this has been corrected?

I'm sorry for the confusion. I did not recall the details so I thought 
the code in J5 was completely without length checks. I have checked 
current code, some git history and my sore brains and I believe this is 
what happened:

1. I found out NT had a truncation bug that only showed up in external 
mode (it truncated at 28).

2. I fixed the truncation bug in NT and asked how to do with external.c 
as I thought that would be a smarter fix.

At this point I could not do much about the performance drop, without a 
truncating external mode.

3. I implemented truncation in external.c (before your answer, since 
dynamic and other formats really depend on it)

4. I dropped the new length check (so there was NO length check 
anywhere) and expected a boost but to my surprise that was even slower! 
So I left it with the correct (but no longer needed) truncation and 
forgot all about it (at this point I should have reverted the non-utf8 
set_key() changes before posting the patches).

5. Jumbo-5 was released with the code as in (2) above.

Now, we can choose to revert set_key() and set_key_encoding() to the 
"buggy" versions in 1.7.8 - the bug is now harmless as long as external 
truncates. But I would much rather figure out why the speed went from 
26500K (original code) to 24000K (no length check at all) by completely 
removing the extra condition in the for loop. [These figures are for my 
intel laptop, the J5 code (correct length check) is 25500K]. How can a 
dropped conditional make a significant performance drop?

>> For set_key_utf8(), the code present in jumbo-5 has got to be faster
>> than 1.7.8, and now properly truncates at 27. In UTF-8 mode we must
>> always take care of truncation since the "reported" max length is 3x27.
>> Try running "-test -enc:utf8" comparing 178J8 and 179J5, that's a quick
>> test only including Unicode formats.
> 1.7.8-jumbo-8:
> $ ./john -te -fo=nt -enc=utf8
> Benchmarking: NT MD4 [128/128 X2 SSE2-16] in UTF-8 mode... DONE
> Raw:    21421K c/s real, 21421K c/s virtual
> 1.7.9-jumbo-5:
> $ ./john -te -fo=nt -enc=utf8
> Benchmarking: NT MD4 [128/128 X2 SSE2-16] in UTF-8 mode... DONE
> Raw:    20427K c/s real, 20427K c/s virtual
> (I repeated both runs twice, as usual - got similar results.)
> So 1.7.8-jumbo-8 reports slightly faster speed even for utf8.

IIRC it went from like 21000K to 25000K on the AMD when I produced that 
patch, I was pretty happy with it. My intel laptop show a smaller boost, 
from 20500K to 21500K. I too repeated my tests. This is what I meant 
with "compiler randomness". BTW I use gcc-4.6 on the laptop and it may 
have been gcc-4.5 on the AMD.

Besides, the UTF-8 change is still needed, regardless of external.c


Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.