john-users - Re: Incremental mode progress and ETA

Openwall wordlists collection for password cracking (20+ languages)
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Wed, 23 Feb 2011 03:11:49 +0300
From: Solar Designer <solar@...nwall.com>
To: john-users@...ts.openwall.com
Subject: Re: Incremental mode progress and ETA

On Wed, Feb 23, 2011 at 12:50:09AM +0100, magnum wrote:
> As a spinoff of recent experiments with better MPI splitting, I ended up 
> writing a progress (percentage) and ETA patch for Incremental mode.
> 
> It is dead simple and works just fine but I'd like to improve it if 
> possible. It calculates the total number of possible candidates from 
> real_count and this seem to end up with a valid figure in all 
> situations. So how many have we tried? The current solution is just a 
> 64-bit counter that increments for every candidate produced.

The problem here is that you have to store that counter in .rec, which
is a change of the file format.  I am going to make such a change anyway
(need that counter for other reporting to be added anyway), but along
with reworking the incremental mode, not on its own.

> Now to the actual question: Is there by any chance some kind of 
> reasonably light formula that could be used to calculate this number 
> instead, from the various parameters and variables already used in the 
> process?

Maybe, but like I said I am going to add the counter anyway, for all
modes at once - but along with the incremental mode rework to avoid
changing the .rec file format twice.

> Or is that question the reason we don't already have this 
> feature?

No, that's not the reason.  The real reason/excuse is in the FAQ.

> Unless there is such a formula, with all cool patches pouring in now I 
> think I'll release it as-is, not just for MPI but as a separate patch. 

I am concerned that this might result in people running into issues with
restoring sessions.  In particular, what if one wants to go "back" to a
version of John without your patch (maybe a newer version of John even,
just without the patch)?  Will they not be able to restore the session
because of your .rec file format change?

> We could have a Makefile feature for disabling it in case it ends up 
> hitting performance in some environments. On my gear it doesn't seem to 
> have any measurable impact on performance even with the very fastest 
> hashes, in fact it sometimes ends up faster.

That's curious.  Perhaps you did the smart thing of not incrementing the
counter for every candidate password, but updating it once per crypt_all()?

> However there is another advantage with calculating it from already used 
> data: we could finally get to know how far that low-prio job we've been 
> running for ages have progressed (my current patch disables progress 
> output unless the counter was saved in an extra line in the .rec file). 
> Come to think of it, if there is an easy-but-pretty-slow way of 
> calculating it, I could keep the counter and only use the formula when 
> restoring an old job that lacks the counter.

This makes sense.  Yes, you could implement it.  Sorry, I am not in the
mood of producing pseudo-code (or C code) for you now.  I feel that I
have other priorities.

Thanks,

Alexander