Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Mon, 05 May 2008 09:14:53 +0200
From: Simon Marechal <>
Subject: Re: working with large files

Solar Designer a écrit :
> Well, what JtR does right now is to completely rebuild the per-salt hash
> table each time there's a guess for that salt (and a hash gets removed).
> The obvious optimization here is to update the existing per-salt hash
> table instead.  If there's demand for a speedup in this area, even
> though it is negligible for long-living sessions, then perhaps this
> should be done.
> Alternatively, you could choose to not remove hashes for cracked
> passwords at all.  This means that there may be duplicate guesses if
> duplicate candidate passwords are tried (but it's not much of a problem
> since "john --show" will only report each guess just once).  It also
> means that the reported effective c/s rate will be artificially higher -
> this will not correspond to any meaningful speedup, and overall things
> may get slower for long-living sessions because keeping more hashes in
> the tables results in a higher number of collisions.


here is a patch to my tree with the changes. It should however be 
possible to patch vanilla john versions, except with the part about the 
rawMD5_fmt.c changes.

Here is what it (supposedly) does:
* increases the size of the largest hash table from 4096 to 65536 entries
* increases accordingly the size of the hashes returned by the raw-md5 
* add a "found" marker to the db_password structure
* when a password is found, the corresponding db_password structure is 
marked as such, and then the hash table is searched for this entry and 
this entry is removed
* if the password was the first in the related db_salt "list", it is 
removed and then this list is parsed until a password that is not found 
is encountered. All consecutive found passwords that are parsed are then 

The two last items dramatically alter the performance when working with 
large files. The last item ensures that salts where all passwords have 
been found are cleaned.

It seems to work for me (doesn't crash, seems to clean the salts 
properly, dramatically increase speed in the rare case you have 100000+ 
passwords per salt). As always with stuff I submit, it is a workaround 
to an immediate problem i encountered, so there are cleaner ways to do 
this (suggested by Solar, or changing the linked-list structures to 
something faster), so don't use it on something important. Feedback 
would however be tremendously useful.


Download attachment "largepasswordset.patch.gz" of type "application/x-gzip" (1607 bytes)

To unsubscribe, e-mail and reply
to the automated confirmation request that will be sent to you.

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.