Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Mon, 3 May 2021 19:53:42 +0200
From: Solar Designer <solar@...nwall.com>
To: john-users@...ts.openwall.com
Subject: Re: source of information for John's charset files

On Mon, May 03, 2021 at 04:18:15PM +0200, Solar Designer wrote:
> On Sun, May 02, 2021 at 11:00:34PM -0400, Matt Weir wrote:
> > Side note, I just saw your most recent results of training/running against
> > RockYou. I'm willing to admit I'm wrong if you are getting better results
> > training without dupes. That's just contrary to what I've seen in the past.
> > I might need to run some tests of my own to look into this.
> 
> Note: better results when the test set is also without dupes.  However,
> I think that's what matters after most dupes are eliminated using a
> wordlist anyway in real-world usage of our tools.

I found an easily publicly available example where training without
dupes produces worse results even when the test set is also without
dupes.  Running through RockYou as a wordlist first corrects that.

I took phpbb-withmd5.txt.bz2 from here:

https://wiki.skullsecurity.org/Passwords

phpbb 	phpbb.txt.bz2 (868,606 bytes) 	n/a 	2009-01 	Ordered by commonness
								Cracked from md5 by Brandon Enright
								(97%+ coverage)
phpbb with count 	phpbb-withcount.txt.bz2 (872,867 bytes) 	n/a
phpbb with md5		phpbb-withmd5.txt.bz2 (4,117,887 bytes) 	n/a

My 3 training sets are: RockYou with dupes, RockYou unique (full 14.3M
this time), HIBP v7 cracked excluding fbobh_* (458M).

Cracked phpbb uniques with incremental mode at 1 billion candidates are:

RockYou with dupes - 34.7%
RockYou unique     - 33.6%
HIBP v7 cracked    - 30.9%

RockYou wordlist   - 39.3%

RockYou wordlist together with:

RockYou with dupes - 51.3%
RockYou unique     - 51.5%
HIBP v7 cracked    - 50.6%

RockYou wordlist with best64 rules (only, no incremental):

RockYou + best64   - 56.3%

RockYou wordlist with best64 rules (above) together with the incremental
runs (first set of results above):

RockYou with dupes - 60.4%
RockYou unique     - 60.5%
HIBP v7 cracked    - 60.1%

All of these combined - 62.3%.

Of course, real attacks on fast hashes like this would test many more
candidates and get almost all hashes cracked.  These simulations I run
are to see how effective the approaches are in case similar passwords
were used with slow (non-)hashes and/or with many different salts.

Alexander

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.