john-dev - current large hash file speeds and bottlenecks (was: ldr_split

Follow @Openwall on Twitter for new release announcements and other news

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20150918165705.GA25301@openwall.com>
Date: Fri, 18 Sep 2015 19:57:05 +0300
From: Solar Designer <solar@...nwall.com>
To: john-dev@...ts.openwall.com
Subject: current large hash file speeds and bottlenecks (was: ldr_split_line() performance regression)

On Tue, Sep 15, 2015 at 09:29:35PM -0700, Fred Wang wrote:
> That is a *whole* lot better than when you started.  I'm still interested in seeing what happens with --fork=32 on the larger machines, but - wow!  Well done!

Thanks.  For the currently committed code (which isn't my latest, and is
a bit slower than what I run on 2x E5420 now) and the currently
committed john.conf defaults, I get on 2x E5-2670 v1:

--fork=32:

real    0m17.399s
user    2m38.080s
sys     0m19.943s

--fork=16:

real    0m19.137s
user    1m59.104s
sys     0m9.040s

--fork=8:

real    0m23.691s
user    1m46.217s
sys     0m4.615s

Loading takes a little over 10 seconds on one core, the rest is cracking.

This machine is supposed to be slower than the 2x E5-2680 v2 that you
reported similar speeds for.

This is a non-OpenMP build ("./configure --disable-openmp") made with:

gcc version 4.9.1 20140922 (Red Hat 4.9.1-10) (GCC)

(from devtoolset-3 for Scientific Linux 6), and the command line is:

time ./john -form=raw-md5 -w=10m.pass -ru=fred-best64 -nolog -v=1 -save-mem=1 -fork=32 29m.txt

1709703 passwords are cracked in all cases, like before.

The next hurdle is "--show", which is still unreasonably slow: takes
several minutes to display those same passwords we've cracked in ~20
seconds.  I haven't looked into it yet.  I think this might be a
regression with some changes in jumbo as I seem to recall "--show"
working no slower than normal loading of large files before.  I hope
it's something trivial.

After that, speeding up loading of large salted hash databases, which is
hopefully mostly about tuning of SALT_HASH_LOG.

And maybe we need to have john print one-line tips each time it's run on
a terminal, like e.g. radare2 does.  Suggest options like "-nolog -v=1
-save-mem=1", randomly or when it feels a given run would likely benefit
from them.

Alexander

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.