john-users - Re: to MPI or not MPI john?

Openwall Project		/home Owl JtR Pro crypt pam_passwdqc tcb phpass scanlogd popa3d msulogin / Linux BIND / advisories presentations / services donations / wordlists passwords / community lists wiki CVSweb mirrors signatures
bringing security into open environments

Order Openwall Wordlists CD (20+ languages) with delivery worldwide or download

[<prev] [next>] [<thread-prev] [thread-next>] [month] [year] [list]

Date: Tue, 06 Jun 2006 22:11:44 +1000
From: David Luyer <david@...er.net>
To: <john-users@...ts.openwall.com>
Subject: Re: to MPI or not MPI john?


On 6/6/06 9:20 PM, "Randy B" <aoz.syn@...il.com> wrote:

> I know a lot of parallelization people want to see an approach that
> allows loose clusters to work on a given problem set, but I for one am
> interested simply to use my SMP machines more efficiently.  I think it
> would move us along much more quickly (and scratch my itch sooner...
> ;-) )  if focus was initially placed on solving the parallelization
> problem, then on distribution to multiple nodes.  If I *really* want
> to run a non cluster-aware program on multiple nodes of a cluster,
> I'll wrap it in something like Mosix.

I think I've mentioned my quick and dirty approach to distributing
john before, both on SMP and on multiple machines; ignore the early
passes because they all happen within a relatively short period, and
focus on incremental, then just distribute individual incremental
passes to individual machines, think about how large-scale distributed
efforts (SETI@...e etc) work, and map that to john;

    - hand out work block to client:

            - client requests work block

            - server answers: you have incremental pass #5 to work on

            - client works on incremental pass #5

    - wait for completion of work block

    - hand out new work block

    - repeat

The advantage is that you don't have to worry about splitting the
work evenly, servers falling behind (if they have unequal capability
or unequal hash distribution), wasted CPU in generating keys which
won't be checked, etc.
 
I've only done this in dodgy ways with manual editing on the status
files and merging of john.pot files etc.  But it could easily be
automated as part of john itself.  The issues to think about are:

    - marking a work block as "not done" if a server dies while
      working on it, losing its state file

    - automatically distributing updates to passwords cracked
      (can't assume NFS distribution of john.pot, some people
      may wish to utilize CPU resources across multiple security
      domains)

    - having the one server hand out tasks on hash types/workloads?
      (it is not uncommon to have a mix of 3DES and MD5 and want
      to find plaintexts for both)

David.