Date: Wed, 5 May 2021 14:14:51 +0200 From: Solar Designer <solar@...nwall.com> To: john-users@...ts.openwall.com Subject: Re: source of information for John's charset files On Mon, May 03, 2021 at 04:18:15PM +0200, Solar Designer wrote: > My expectation is that training on unique passwords only will in fact > reduce the number of cracked accounts when only incremental mode is > used. However, after having run through RockYou as a wordlist, it's not > obvious whether it's beneficial for incremental mode to also favor the > repeated passwords like 123456. > > What you suggest about excluding e.g. just the top 10k makes sense. > Another approach I thought of, but I don't recall trying, is to apply a > logarithmic scale to the counts. For example, for passwords appearing > 1000+ times include them 4 times, for 100+ include them 3 times, etc. I experimented with this now, and it may very well be a way to have the best of both worlds, or at least a reasonable balance. Compared to RockYou unique, I am getting the most improvement (overall across different test sets) by adding to it a list of RockYou passwords that appeared 3+ times on the original with-duplicates list. In other words, a password that appeared 3+ times is listed twice, otherwise just once. Of course, this doesn't distinguish frequencies of passwords within top ~1 million, so e.g. "password" ranks way lower than it does in our current default .chr files generated from the original with-duplicates RockYou. However, "123456" still ranks first, because not only it but also its substrings rank high. I also tried adding top 100k and top 10k lists (giving 4 repeats for passwords in top 10k), which hurt my tests on all-unique test sets a tiny bit (possibly just noise), but it also brought "password" only a bit higher. > I decided to test these not only at 1 billion candidates, but also at > other points. I use three training sets: RockYou with dupes (same as > was used to generate our currently bundled .chr files - in fact, I just > reuse ascii.chr from there), RockYou unique shuffled and 1M test set > removed from it (so 13.3M training set), and HIBP v7 458M cracked (after > removal of the fbobh_* pattern). The test set is always the mentioned > 1M from RockYou unique shuffled. Here are the percentages cracked at > 10M, 100M, 1G, 10G, 100G candidates: > > RockYou with dupes - 4.6%, 10.2%, 20.2%, 33.3%, 48.0% > RockYou -1M unique - 4.7%, 11.2%, 21.5%, 35.0%, 48.3% > HIBP v7 cracked - 3.2%, 8.7%, 17.8%, 30.0%, 44.5% > > So despite of "RockYou -1M unique" being the only one 100% out-of-sample > test (no password appears in both the training and the test set) and > also having the smallest training set (at 13.3M), it outperforms the two > other tests across this whole range. > > Of course, HIBP performing worse doesn't necessarily mean it's a worse > choice in general - just that it's a worse fit for RockYou. We've also > seen that when using a portion of HIBP as the test set, things are the > other way around - training on the rest of HIBP produces better results > than training on RockYou does. Here's the mix of my new potion: HIBP v7 cracked - x1 (also includes RockYou) RockYou unique - x30 RockYou top 1M 3+ - x31 This gives a list of roughly double the size of HIBP v7 cracked: ~458M to ~924M. Generating a .chr file with --external=filter_ascii uses ~896M from there (excludes the nested hashes, among other things). This gives the highest weight to passwords appearing on RockYou 3+ times, then to the rest of RockYou, and hopefully only uses HIBP to resolve ties and as a fallback where RockYou lacks a definitive pattern. The results directly comparable to those above, also repeated below for comparison, are: RockYou with dupes - 4.6%, 10.2%, 20.2%, 33.3%, 48.0% RockYou -1M unique - 4.7%, 11.2%, 21.5%, 35.0%, 48.3% HIBP v7 cracked - 3.2%, 8.7%, 17.8%, 30.0%, 44.5% New mix - 4.9%, 10.8%, 20.8%, 34.0%, 47.6% Also for comparison, the best I am able to achieve by training on (full) RockYou only (processed in various ways) at 1 billion candidates (middle column above) is 22.0%. To remind, the above uses 1M of RockYou unique shuffled as the test set, so except for the "RockYou -1M unique" line this is in-sample testing. So these good results don't mean a lot, but they do mean that the usage of HIBP hurting this test is mostly gone with the new mix. This matters more together with another result: As I mentioned earlier in this thread, training on HIBP v7 cracked not surprisingly provided the best result when testing on HIBP v7 as well, including with non-overlapping training and test sets. The results at 1 billion candidates were 5.4% for training on RockYou with duplicates, 6.0% for RockYou unique, and 6.6% for HIBP cracked unique. Well, here's a new result: the new mix above achieves almost 6.6% as well (to be specific, 6.64% vs. 6.59%). So maybe it's a good balance for targeting yet unknown password sets. > Also curious is how many different passwords the different training sets > crack. At the 100G mark, the three runs above cracked a total of 52.5%. Adding the new mix increases the total for the four 100G pots to 52.7% - not much increase from the previous 52.5%. Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.