Date: Thu, 1 Jun 2017 15:54:39 +0200 From: Solar Designer <solar@...nwall.com> To: john-users@...ts.openwall.com Subject: Re: Need help for understanding rule preprocessor Hi Alex, On Wed, May 31, 2017 at 11:40:11PM -0500, Alex Liu wrote: > I am a computer science student at The University of Chicago. I am using > john (version: 1.7.9-jumbo-7_omp [macosx-x86-64]) and got confused by the > preprocessor command. I wonder what the following two commands mean: > > -[:c] <* !?A \p1[lc] p > -[:c] l /a /e /l /o /s sa4 se3 sl1 so0 ss$ (?\p1[za] \p1[:c] > > Specifically, what does [:c] and -[:c] mean? What does \p1[za] and \p1[lc] > mean here? Since you correctly refer to these pieces as preprocessor commands, it appears that you already found they're documented in doc/RULES - but I guess the documentation is hard to understand. What the preprocessor does is expand individual original ruleset lines into a possibly larger number of rules. In this case, 2 lines are expanded into 4, as you can test and find out in this way: [user@...t run]$ cat john-local.conf [List.Rules:test] -[:c] <* !?A \p1[lc] p -[:c] l /a /e /l /o /s sa4 se3 sl1 so0 ss$ (?\p1[za] \p1[:c] [user@...t run]$ echo -n test | md5sum | cut -d' ' -f1 > pw [user@...t run]$ echo test | ./john --pipe --rules=test --session=test --verbosity=4 --format=raw-md5 pw Using default input encoding: UTF-8 Loaded 1 password hash (Raw-MD5 [MD5 128/128 AVX 4x3]) Warning: no OpenMP support for this hash type, consider --fork=32 Press Ctrl-C to abort, or send SIGUSR1 to john process for status 0g 0:00:00:00 0g/s 13.33p/s 13.33c/s 13.33C/s tests..Tests Session completed [user@...t run]$ fgrep -w Rule test.log 0:00:00:00 - Rule #1: '-: <* !?A l p' accepted as '<*!?Alp' 0:00:00:00 - Rule #2: '-c <* !?A c p' accepted as '<*!?Acp' 0:00:00:00 - Rule #3: '-: l /a /e /l /o /s sa4 se3 sl1 so0 ss$ (?z :' accepted as 'l/a/e/l/o/ssa4se3sl1so0ss$(?z' 0:00:00:00 - Rule #4: '-c l /a /e /l /o /s sa4 se3 sl1 so0 ss$ (?a c' accepted as 'l/a/e/l/o/ssa4se3sl1so0ss$(?ac' This is with recent bleeding-jumbo. The much older 1.7.9* might not have needed the "--verbosity=4" option for this, but otherwise contains this logging functionality too. Here you can see the resulting 4 rules after preprocessor expansion (left) and also after processing of the rule reject flags and nop squeezing (right). Now I'll explain what these specific pieces do: The dashes at the start of lines indicate rule reject flags, quoting doc/RULES: Rule reject flags. -: no-op: don't reject -c reject this rule unless current hash type is case-sensitive We want the "<* !?A l p" rule to be used unconditionally, but the "<* !?A c p" rule used on case-sensitive hash types only. That's because using the latter on a case-insensitive hash type would be redundant, since the difference between the "l" (lowercase) and "c" (capitalize) commands would be lost. For this optimization, we need the "-c" rule reject flag on the latter rule. We also don't want to spend two separate lines on these similar rules, so we compact them into one line using the preprocessor. To do this, we introduce the "-:" no-op rule reject flag into the first rule and then combine the two rule reject flags as "-[:c]". Then we also need to combine the "l" and "c" commands, but we want them to be applied along with the corresponding rule reject flags. We do this with "\p1[lc]", where the "\p1" requests processing "in parallel" with list or range "number one", which in this case is our two rule reject flags that are the first preprocessor expansion found on this line. In the second original line, we also use a no-op rule command (on top of a no-op rule reject flag), for a similar reason. As you can see, this no-op command gets squeezed out and does not affect runtime performance. Yes, this is way too much complexity for saving two lines, but when you get used to these concepts and use them a lot they end up making many rulesets many times shorter (and easier to maintain too, as there's less duplication so changing something common to multiple ultimate rules requires fewer edits). The below isn't about the preprocessor anymore, but now let's see how the rule reject flags actually reject rules: [user@...t run]$ rm test.log [user@...t run]$ echo test | ./john --pipe --rules=test --session=test --verbosity=4 --format=LM pw Using default input encoding: UTF-8 Using default target encoding: CP850 Loaded 2 password hashes with no different salts (LM [DES 128/128 AVX-16]) Warning: poor OpenMP scalability for this hash type, consider --fork=32 Will run 32 OpenMP threads Press Ctrl-C to abort, or send SIGUSR1 to john process for status 0g 0:00:00:00 0g/s 6.666p/s 6.666c/s 13.33C/s TESTS Session completed [user@...t run]$ fgrep -w Rule test.log 0:00:00:00 - Rule #1: '-: <* !?A l p' accepted as '<*!?Alp' 0:00:00:00 - Rule #2: '-c <* !?A c p' rejected 0:00:00:00 - Rule #3: '-: l /a /e /l /o /s sa4 se3 sl1 so0 ss$ (?z :' accepted as 'l/a/e/l/o/ssa4se3sl1so0ss$(?z' 0:00:00:00 - Rule #4: '-c l /a /e /l /o /s sa4 se3 sl1 so0 ss$ (?a c' rejected We requested treating our hash(es) as LM, a case-insensitive hash type. As a result, the "-c" rule reject flags actually rejected the two rules (out of four). Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.