Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Wed, 13 May 2015 21:24:18 +0300
From: Aleksey Cherepanov <lyosha@...nwall.com>
To: john-dev@...ts.openwall.com
Subject: Re: get_binary_*() and get_hash_*() methods

Solar,

On Wed, May 13, 2015 at 07:53:30PM +0300, Solar Designer wrote:
> Aleksey,
> 
> On Wed, May 13, 2015 at 07:37:21PM +0300, Aleksey Cherepanov wrote:
> > On Thu, May 07, 2015 at 05:39:45PM +0300, Solar Designer wrote:
> > > You can see a dirty hack like this here:
> > > 
> > > git show 9a6f4f6f69903763e664f03d2adee97486eca9de DES_bs_b.c
> > 
> > I tried to perform lookup in crypt_all() to exit earlier not finishing
> > the computations.
> 
> The same(?) can be achieved by exiting crypt_all() early all the time
> and completing the computation in cmp_one() when needed.  IIRC, myrice's
> raw SHA-512 on GPU formats did that, completing the computation in
> cmp_one() on CPU only when needed.

Not exactly the same: I build the check into the algo so I don't need
global vars to complete computation. Though cmp_one should be called
so rarely that computation of the whole hash from key should be ok.

> > I tried it against raw-sha512, scalar implementation
> > with 2 instructions reversed (endianity altering and addition of the
> > state), I compared h variable (it should be the closest result). Also
> > I replaced salt->index(index) with var & 0xfff (salt->hash_size == 2).
> 
> OK, but raw SHA-512 on one core isn't slow enough for this to matter.

"slow enough"? Did you mean "fast enough" as I can get from the text
below?

> These tricks are for faster hashes or/and for use with parallelization,
> to keep the checks parallelized as well and performed in the same
> threads that computed the corresponding hashes (thus, to avoid data
> transfers between different cores).
> 
> > salt argument is 0 in self tests, so I tested it with
> > --skip-self-tests:
> > 5g 0:00:01:27  3/3 0.05721g/s 1351Kp/s 1351Kc/s 20274KC/s gok3sb
> > 5g 0:00:01:27  3/3 0.05747g/s 1295Kp/s 1295Kc/s 19426KC/s 159518*
> > #1 is with early exit, #2 without that additional check.
> > 
> > The difference (3%) may be a fluctuation.
> 
> Yes, may be just a fluctuation, or may be for real, but either way it's
> quite small.
> 
> > Is there any solution to not skip self tests?
> 
> Yes.  crypt_all() tells the caller whether there _might_ be matches.
> You simply return that, yes, there _might_ be matches every time you're
> called with parameters that you have no optimized code for.  For
> example, if you only support salt->hash_size == 2, you can return
> *count + 1 whenever you're called with salt->hash_size != 2.  Of course,
> you'll do this after computing the hashes sufficiently for your
> get_hash*() and cmp_*() to work properly.

So self tests can't really check the correctness of gpu part of
crypt_all() of fast formats because gpus should check hashes on gpu
and do not return them back for checks in cmp_one().

Or at least benchmark is not accurate if the check is in crypt_all().

> > I had such problem too. Why is there 0 in binary field? Should not
> > such entries be removed? Is it due to multithreading?
> 
> I'm not familiar with Sayantan's code, but "binary" may be NULL in
> "single crack" mode.  See single.c:

So it is normal behaviour. Thanks!


BTW having high MAX_KEYS_PER_CRYPT, it may be interesting to construct
a small bitmap (or a similar structure) in the end of crypt_all() to
be used in cmp_all()/cmp_one() to speed up the check.

-- 
Regards,
Aleksey Cherepanov

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.