john-users - Couple of questions regarding rules (preprocessor) and truncation

Openwall GNU/*/Linux 3.0 - a small security-enhanced Linux distro for servers
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date: Fri, 18 Mar 2011 20:46:55 +0100
From: magnum <rawsmooth@...dband.net>
To: john-users@...ts.openwall.com
Subject: Couple of questions regarding rules (preprocessor) and truncation

Solar,

Here's a cut down version of a rule line I experiment with:

[List.Rules:T1]
 >[2-9A-Z] '\p[2-9A-Z] A[0z]"1234"

I do want this rule to go to Z for formats that supports that length. 
But what I would prefer, and actually did try first, is this:

 >[2-*] '\p[2-*] A[0z]"1234"

This would produce a lot less *rules* when running against 
length-impaired formats like DES or LM and many other formats too. Could 
this be made allowed? Or is it already, using some escape I did not try?
I understand the preprocessor can't know about variables but *, + and - 
are constants and should be known at pp time, right?

The full ruleset I tried expands to over 750,000 rules from four lines. 
If I could use those constants in the preprocessor it would 
automatically cut down to more like 155,000 rules when running against 
DES or 133,000 rules when running LM, *without* making separate versions 
for different length formats. That would make a difference.


Anyway, accepting the situation and just trying to mitigate some of the 
duplicates, I split the example line into two:

[List.Rules:T2]
 >[2-9A-Z] '\p[2-9A-Z] A0"1234"
 >[2-9A-Z] '\p[2-9A-Z] Az"1234"

Now, trying this with one long word, and truncation at 8:

user@...:~$ cat wordlist
longcandidate

user@...:~$ ./john -wo:wordlist -stdout=8 -ru:T2
1234lo
1234lon
1234long
lo1234
lon1234
long1234
longc123
longca12
longcan1
longcand
words: 10  time: 0:00:00:00 100.00% (ETA: Fri Mar 18 19:00:27 2011) 
w/s: 500  current: longcand

That worked fine, 12 duplicates was muted. Lowering it to 7 works too, 
or raising it. But why are dupes not muted if I go below 7?

user@...:~$ ./john -wo:wordlist -stdout=6 -ru:T2
1234lo
1234lo
1234lo
1234lo
1234lo
1234lo
1234lo
1234lo
1234lo
1234lo
lo1234
lon123
long12
longc1
longca
longca
longca
longca
longca
longca
longca
words: 21  time: 0:00:00:00 100.00% (ETA: Fri Mar 18 19:01:26 2011) 
w/s: 1050  current: longca

Actually *one* duplicate is muted, compared to using -ru=T1. I do not 
understand which, or why.

Last, a related question: can I expect DES to behave just like 
--stdout=8 (and LM just like --stdout=7) when it comes to muting 
duplicates? I mean, is it the very same parts of John that does it, 
stdout or not?

cheers
magnum