Date: Mon, 2 Aug 2010 07:43:32 +0400 From: Solar Designer <solar@...nwall.com> To: john-users@...ts.openwall.com Subject: Re: KoreLogic Contest.. first set of stats/rules First of all, Minga (and others at KoreLogic) - thank you for the contest! On Sun, Aug 01, 2010 at 07:32:07PM -0500, Minga Minga wrote: > We have released the first set of stats/information from the contest: > > How the passwords were created: > http://contest.korelogic.com/how_passes_created.html > > Here are the "answers": > http://contest.korelogic.com/plaintexts.html Thank you for publishing this. I was indeed curious about some of it. > Most relevant - how they were created. > http://contest.korelogic.com/rules.html > Those are the john the ripper rules used to CREATE the contest. Great. This is a bit disappointing to me, though, in two ways: 1. Didn't you promise to release a ruleset for actual use when attacking real-world passwords? So far, you've only released these snippets that you used to generate the passwords. Only a subset of them are candidates for inclusion in an actual-use ruleset, and this needs more work (as you correctly propose). 2. Your use of the preprocessor is minimal. The ruleset can be made a lot shorter by more extensive use of the preprocessor. OK, maybe this tells me that this stuff is hard to grasp, although I don't see how I could have made the syntax significantly simpler or the documentation significantly better (more examples? but there are plenty in the default john.conf's ruleset). Any suggestions are welcome. For example, the 90-line KoreLogicRulesReplaceNumbers ruleset: [List.Rules:KoreLogicRulesReplaceNumbers] /0s01 /0s02 /0s03 ... /9s96 /9s97 /9s98 can be shortened to 10 lines without any efficiency loss (and no readability loss either): /0 s0[1-9] /1 s1[02-9] /2 s2[013-9] /3 s3[0-24-9] /4 s4[0-35-9] /5 s5[0-46-9] /6 s6[0-57-9] /7 s7[0-68-9] /8 s8[0-79] /9 s9[0-8] In fact, it can be shortened even further to one line: /[0-9] s\0[0-9] Q but then there's slight efficiency loss (some candidate passwords are almost generated, then rejected with the "Q" command). (Maybe I need to enhance JtR with detection and dropping of effective-no-op and always-reject rules, then it'd be OK to "inadvertently" produce them with the preprocessor.) I've verified that both of my proposed replacements produce the exact same results as yours (same MD5 hash of output on a test wordlist). Another example of preprocessor non-use (note: these lines start with the greater-than character, which I think many mail readers may display improperly since this syntax is typically used for quoting in e-mail): [List.Rules:KoreLogicRulesAdd1234_Everywhere] Az"1234" >0A"1234" >1A"1234" >2A"1234" >3A"1234" >4A"1234" >5A"1234" >6A"1234" >7A"1234" >8A"1234" >9A"1234" This is literally the same as: Az"1234" >[0-9]A\0"1234" Also, both produce some duplicates. This is solved with: <1Az"1234" >[0-9]A\0"1234" which generates all the same candidate passwords but no duplicates. This highlights another problem: the first one of these two lines is only useful when applied to an empty string, thereby producing the candidate password "1234". Probably not what you meant (better to have "1234" in your wordlist). Thus, we further reduce this to one line: >[0-9]A\0"1234" Your KoreLogicRulesReplaceLettersCaps, which is 26 lines: # This is a lamer/faster version of --rules:nt [List.Rules:KoreLogicRulesReplaceLettersCaps] /asaA /bsbB ... /ysyY /zszZ can be replaced with one line (verified, exact same output): /[a-z] s\0\p[A-Z] BTW, I like the idea behind this one (I mean your idea of toggling the case of just one type of character at a time). :-) Then, a block like: /asa@[:c] /asa4[:c] /AsA4[:c] /AsA@[:c] /bsb8[:c] /BsB8[:c] /ese3[:c] /EsE3[:c] may be rewritten as one line: /\r[aaAAbBeE] s\0\r\p[@44@...3] [:c] These two are exactly the same. And the single line can be extended to include the rest of your rules in that block (46 lines), or this can be split over, say, 4 lines for readability. As to your comment about "[:c]" producing duplicates, yes, but it did not produce as many duplicates in JtR's supplied ruleset because the lines use "l" (lowercase) in there. They do it primarily to avoid having to generate separate rules for lower- and uppercase letters to replace. Then "c" is far more likely to produce a different string. To avoid generating duplicates, you can use: /\r[aaAAbBeE] s\0\r\p[@44@...3] [:M] \p[:c] \p[:Q] OK, that's more than enough for now. Basically, this needs more work. BTW, spaces and colons are OK to have/generate - they have no performance cost in recent versions of JtR (are squeezed out early on). > In the next few weeks, Id us, as a community, to try to improve these rules > and hopefully incorporate them into everyone's rules. The represent years > of password analysis of over 3 million passwords from environments > that enforce password complexity. Sounds great, but frankly as it relates to many of the "rule blocks", I seriously doubt that they're effective "enough" on real-world passwords. Even in this contest, even when we (our team) figured out many patterns and could encode them into rules, we then only ran the "slower" ones of the rules (those producing more combinations) against the fastest hashes only - we did not run them against slow hashes, it was just not worth the time. That's even when we _knew_ we'd get a few hits. In the real world, we would not know. That said, I agree that we should analyze these rules, identify those that are generally useful, optimize them, and put into some ruleset(s) recommended for use. Maybe produce two such rulesets: one for fast hashes and/or short wordlists, the other for slow/salted hashes and/or for large wordlists. Thanks again, Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.