Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date: Sun, 3 Apr 2016 11:17:57 -0500
From: jfoug <jfoug@...nwall.net>
To: john-users@...ts.openwall.com
Subject: Re: New hybrid external elite mode


On 4/3/2016 9:52 AM, Rich Rumble wrote:
> A lot of this can be done with some better rules, see and old thread I 
> started in 2010: http://www.openwall.com/lists/john-users/2010/08/02/8 
> I wanted to leet replace each possible cobmination: remember -> 
> r3member-> r3m3mber -> r3m3mb3r I haven't tried this external mode 
> yet, but does it do the same thing? -rich 

Yes, I listed that a  lot of this 'can' be done with rules, but it is 
very inefficient to do this using rules. You have to run your wordlist 
through the rules sets 1000's of times, so produce the proper words.  On 
each run through, the rule will throw away almost every word, with the 
exception of the 'few' that have the specific letters in the exact 
locations you want.

Take a look at this new external mode. It is pretty fast.  I also wrote 
a case mangling external script.  For a baseline speed check, I did a 
comparison with it against rexgen code doing case conversion (which now 
has to be done using replacement alphabets, since case folding was 
removed from the library).   When used in an actual production run 
(raw-md5 against a single hex line), the external case was getting about 
9.3m/s while the rexgen was getting about 4.6m/s, so this external mode 
is running pretty fast.

The external leet code will likely run MUCH faster than multiple leet 
rules, since it is not wasting a lot of time disqualifying words.  It 
simply takes each word as it comes over, and builds the transform set, 
and then properly permutes the next word at each step.

The external hybrid mode is a very powerful word mutation engine. It can 
also be pretty fast, if scripts are written properly.  For transforms, 
where 1 word will be converted into 10's or 100's or 1000's of final 
words, if the next() function is written to be very thin and fast, then 
overall the script will run very fast.

For another qwik-n-dirty speed test, I wanted to test against rules, 
where I KNOW the rules are efficient. So I tested with a simple append 
000 to 999 (1000 items), and ran it using raw-md5 against a 20m 
wordlist. I then re-ran using mask (which should be faster than rules), 
again just to get a baseline

$ ../run/john -rules=:'$[0-9]$[0-9]$[0-9]' in -form=raw-md5 
-w=/c/phpbb/dictionaries/mc-allpass.txt
0g 0:00:03:32 DONE (2016-04-03 10:23) 0g/s 9215Kp/s 9215Kc/s 9215KC/s 
▒▒▒▒▒▒▒▒▒999..▒▒▒▒▒▒▒▒▒▒▒999

$ ../run/john -extern=app_3_nums in -form=raw-md5 
-w=/c/phpbb/dictionaries/mc-allpass.txt
0g 0:00:04:11 DONE (2016-04-03 11:05) 0g/s 7811Kp/s 7811Kc/s 7811KC/s 
▒▒▒▒▒▒▒▒▒▒▒920..▒▒▒▒▒▒▒▒▒▒▒999


$ ../run/john -mask=?w?d?d?d in -form=raw-md5 
-w=/c/phpbb/dictionaries/mc-allpass.txt
0g 0:00:01:53 DONE (2016-04-03 10:53) 0g/s 17234Kp/s 17234Kc/s 17234KC/s 
▒▒▒▒▒▒▒▒▒▒▒127..▒▒▒▒▒▒▒▒▒▒▒777

One BIG difference, is that rules produces 1000 rules. Each word is 
given to ONE rule until all words are used, and then all words are given 
to the next rule, etc.  In the external mode, a word is sent to the 
engine, and then ALL candidates for that word are produced one after the 
other, then the next word is done.  -mask mode works in the same way 
that the external-hybrid engine does (depth first).

Btw, here is the external script. It could have been written to be a 
little faster, but I wrote it quickly, and to produce candidates in the 
same order as the rules mode produces.  Well, it produces the numbers  
in the same order.


[List.External:app_3_nums]
int len;
int val;
void init() {/*do nothing*/}
void new() {
     val=len=0; while (word[len++]) ; ++len;
     /* note, len is pointing 3 chars past the word */
     word[len+1] = 0;
}
void next() {
     word[len] = '0'+val%10;
     if (val%10==0) {
         word[len-1] = '0'+(val/10)%10;
         if (val%100==0) word[len-2] = '0'+val/100;
     }
     if (++val > 1000) word[0] = 0;
}

PS, I was able to rewrite the script, and get it to be almost as fast as 
the rule.

$ ../run/john -extern=app_3_nums in -form=raw-md5 
-w=/c/phpbb/dictionaries/mc-allpass.txt
0g 0:00:03:47 DONE (2016-04-03 11:16) 0g/s 8617Kp/s 8617Kc/s 8617KC/s 
▒▒▒▒▒▒▒▒▒▒▒920..▒▒▒▒▒▒▒▒▒▒▒999

[List.External:app_3_nums]
int len;
int val;
void init() {/*do nothing*/}
void new() {
     val=len=0; while (word[len++]) ; ++len;
     word[len-2] = '0'-1;
     word[len-1] = '0';
     word[len]   = '0';
     word[len+1] = 0;
}
void next() {
     ++word[len];
     if (val%10==0) {
         word[len] = '0';
         ++word[len-1];
         if (val%100==0) {
             word[len-1] = '0';
             ++word[len-2];
         }
     }
     if (++val > 1000) { word[0] = 0; return; }
}

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.