john-users - Re: NON-MPI parallelizing patch against 1.7.2.

Openwall wordlists collection for password cracking (20+ languages)
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date: Mon, 5 Jun 2006 22:20:28 +0400
From: Solar Designer <solar@...nwall.com>
To: john-users@...ts.openwall.com
Subject: Re: NON-MPI parallelizing patch against 1.7.2.

On Mon, Jun 05, 2006 at 05:00:54PM +0200, Otheus (aka Timothy J. Shelling) wrote:
> This patch
> 
> (a) fixes problems when multiple john processes on different machines are
> trying to access the same file (ie, john.pot). It replaces the flock() calls
> with lockf() calls. You even see a message on stdout when two processes
> collide on the same file.

As I've explained in a private e-mail, I've been meaning to change John
to use fcntl() locks instead of flock() - and to use those locks on more
platforms (currently, OS_FLOCK is never enabled with "make generic").
fcntl() has a higher chance of being supported and reliable over NFS.

The john.pot corruption that you've provided examples of did not look
like it is locking-related, although it could have been.

> Solar, I hope you will consider incorporating these into your baseline code.
> Since there's no MPI code, and it performs reasonably well, and has minimal
> code changes, I think it will be to your liking.

I don't intend to incorporate these specific changes, however I accept
them as a good illustration of what functionality you would like to see
in John.

Speaking of your checksumming function - it is a good idea, however the
particular function results in a highly non-uniform distribution with a
high number of nodes.  For example, with MPIbykeysum from your original
patch and the number of nodes set to 100, the first 10 nodes (numbers 0
to 9) get less than 5,000 of candidate passwords per node out of the
first 1 million of candidate passwords generated with "-i" with the
supplied all.chr, whereas nodes 60 to 69 get 12,000 to 13,000 of
candidate passwords per node out of the same 1 million.  If you
implement a checksumming function internal to John, you may want to use
the CRC-32 implementation instead.

MPIbyKeySumThenCount would have been a good idea except that I don't
think it works correctly with recovered sessions.

If you would like me to possibly place any of your patches in contrib/,
please generate them according to the instructions I've provided a few
days ago.  I'd expect a newer revision of the MPI patch from you,
including the relevant additions to john.conf.

Thanks,

-- 
Alexander Peslyak <solar at openwall.com>
GPG key ID: B35D3598  fp: 6429 0D7E F130 C13E C929  6447 73C3 A290 B35D 3598
http://www.openwall.com - bringing security into open computing environments