john-users - Re: Dupes recognition based on internal representation of ciphertext?

Openwall GNU/*/Linux 3.0 - a small security-enhanced Linux distro for servers
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Sun, 18 Dec 2005 00:44:35 +0300
From: Solar Designer <solar@...nwall.com>
To: john-users@...ts.openwall.com
Subject: Re: Dupes recognition based on internal representation of ciphertext?

Frank (and others),

It's been some months, but I've finally included a (somewhat limited)
solution for this problem into 1.6.40.  I might do it better after 1.7.

The new rules for JtR patches implementing hashes with case-insensitive
encodings are as follows:

1. valid() should allow for lower/upper/mixed case to be used in the
case-insensitive portions of the encodings.

2. split() must unify the case of hash encodings, typically by
converting them to all-lowercase (except for case-sensitive portions,
if any).

3. The new flag FMT_SPLIT_UNIFIES_CASE must be set.

Those implementing this kind of patches to JtR may wish to review the
changes I've applied to LM_fmt.c between JtR 1.6.39 and 1.6.40 for an
example:

	http://cvsweb.openwall.com/cgi/cvsweb.cgi/Owl/packages/john/john/src/LM_fmt.c.diff?r1=1.1;r2=1.3

Thanks,

-- 
Alexander

On Tue, Jun 14, 2005 at 12:03:57AM +0400, Solar Designer wrote:
> Frank,
> 
> It's a very interesting issue you've raised.  Sorry I didn't get
> around to responding to you earlier.
> 
> On Sun, Jun 05, 2005 at 09:37:58PM +0200, Frank Dittrich wrote:
> > It looks like the dupes recognition in cracker.c (crk_process_guess)
> > is based on the internal representation of the ciphertext.
> 
> Correct.
> 
> Arguably, the loader should be enhanced to also use internal
> representations when it avoids loading dupes(*) for cracking and when
> it displays cracked passwords.  I haven't tried doing it yet, but I
> expect the latter change to resolve the problem you point out below.
> 
> (*) The current logic in JtR is to only load duplicate hashes when in
> "single crack" or batch modes, and only when login names differ.
> 
> Alternatively, the split() method for affected hash types should be
> enhanced to canonicalize the text representations.
> 
> > $ ./john --format=raw-md5 --wordlist=p h
> > Loaded 3 password hashes with no different salts (Raw MD5 [raw-md5])
> > abc              (3)
> > abc              (2)
> > abc              (1)
> > guesses: 3  time: 0:00:00:00 100%  c/s: 5.55  trying: abc
> > 
> > it looks like all passwords have been guessed.
> > 
> > However, only one hash gets saved in john.pot:
> > 
> > $ cat john.pot
> > 900150983cD24fB0d6963F7d28E17f72:abc
> > 
> > $ ./john --show h
> > 3:abc
> > 
> > 1 password cracked, 2 left
> [...]
> > Of course, for raw MD5 the problem can be avoided by just
> > translating all hashes to lower case.
> 
> BTW, this is best done in split().
> 
> > But there might exist hash algorithms which use different external
> > representations for the same hash.
> 
> Yes.  In fact, LM hashes, as implemented in the official JtR, are
> affected by this (but non-all-uppercase LM hash strings are rare, so
> this was never reported to me).
> 
> Thank you for the bug report!
> 
> > In this case, it's unfortunate that not all external representations
> > get saved in john.pot.
> 
> Well, yes, but it won't be optimal to have multiple representations
> stored in the file, yet not have yet another representation of the
> same hash recognized by "john --show" without having to crack the hash
> again.
> 
> Meanwhile, you can use the trivial fix to cracker.c if you like: in
> the log_guess() function call, remove the "dupe ? NULL : ".
> 
> Thanks again,
> 
> -- 
> Alexander Peslyak <solar at openwall.com>
> GPG key ID: B35D3598  fp: 6429 0D7E F130 C13E C929  6447 73C3 A290 B35D 3598
> http://www.openwall.com - bringing security into open computing environments
> 
> Was I helpful?  Please give your feedback here: http://rate.affero.net/solar