[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Wed, 13 Jul 2016 09:11:24 +0200
From: Albert Veli <albert.veli@...il.com>
To: john-users@...ts.openwall.com
Subject: Re: Loading a large password hash file
For the record, on my computer it's faster to split the hashfile and loop
than waiting for the whole file to load. About 10 million lines per
hashfile seems to be a good value for my computer:
split -l 10000000 huge_hashlist.txt
that creates split files with filenames xaa, xab etc. Then loop:
for hl in x*; do ./john --fork=4 --format=Raw-SHA1 <more arguments> $hl;
done
On Sun, Jul 10, 2016 at 3:38 PM, Solar Designer <solar@...nwall.com> wrote:
> On Sun, Jul 10, 2016 at 12:51:41AM +0300, Solar Designer wrote:
> > On Sun, Jul 10, 2016 at 12:46:14AM +0300, Solar Designer wrote:
> > > perl -e 'use Digest::SHA1 qw(sha1_hex); for ($i = 0; $i < 200000000;
> $i++) { print sha1_hex($i), "\n"; }'
> > >
> > > which is 8200000000 bytes. On a machine with enough RAM, JtR loaded it
> > > in 6 minutes, and the running "john" process uses 13 GB.
> > [...]
> > > There's also the --save-memory option, which may actually speed things
> > > up when you don't have enough RAM. But that's sub-optimal, and high
> > > memory saving levels may hurt cracking speed a lot. They also hurt
> > > loading time when there would have been enough RAM to load the hashes
> > > without memory saving. I've just tried --save-memory=2 on the 200M
> > > SHA-1's file, and it looks like it'll load in about 1 hour (instead of
> > > 6 minutes), consuming something like 11 GB. So probably not worth it
> in
> > > this case.
> >
> > My --save-memory=2 test completed loading in about 40 minutes, and the
> > "john" process uses a little over 9 GB. So in this case it's a 7x
> > increase in loading time to save 30% of memory. Cracking is about 5x
> > slower. Usually not worth it, but if you had 12 GB RAM and didn't want
> > to split the input file in two, it could help.
>
> Setting "NoLoaderDupeCheck = Y" in john.conf reduced the loading times
> to 3 minutes, with or without --save-memory=2. Memory usage stayed the
> same (13 GB or 9 GB, depending on --save-memory).
>
> All of this is on E5-2670 v1, using only one core during the loading, so
> at 3.3 GHz. A modern desktop CPU will work a bit faster.
>
> Commenting out "#define REVERSE_STEPS" in rawSHA1_fmt_plug.c made almost
> no difference for loading time, at least with "NoLoaderDupeCheck = Y".
>
> Alexander
>
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
