Date: Tue, 25 Jan 2011 10:56:16 +0100 From: Frank Dittrich <frank_dittrich@...mail.com> To: john-users@...ts.openwall.com Subject: Re: Password file hash type analysis in JtR Robert Harris wrote: > I know that Unix/Linux and Windows password files can have multiple hash > types in them, I think the following output/utility would be useful in JtR. What is your use case? > Potential new output: > > The following password hash format(s) was/were discovered in the input file > <file>: > > <format1> > > <format2> > > ... > > <formatx> Is that just one line describing the format? As you would specify it on the command line (with --format=) or as john would report it (DES vs. Traditional DES, BF vs. OpenBSD Blowfish, ...)? > Line(s) 1-30 contained <format1> type passwords > > Line(s) 31-60 contained <format2> type passwords > > ... > > Line(s) 61-90 contained <formatx> type passwords Without listing the corresponding hashes/passwords? Depending on your needs, something like this might work: sample pot file: $LM$44efce164ab921ca:123456 10xCP/g3c1sQc:password 20ewVEw7R2GAw:password 10vrzSg43HfWI:123456 10xCP/g3c1sQc:password 10.6.TrD/0sas:trustno1 trEJ1uz6ul/WU:trustno1 $1$12$1YZMQPfPxS3.kStGKUBCO.:1111 e10adc3949ba59abbe56e057f20f883e:123456 $LM$b757bf5c0d87772f:1234 cut -d: -f 1 -s john.pot|grep -n "^" > cracked_hashes cracked_hashes would then look like this: 1:$LM$44efce164ab921ca 2:10xCP/g3c1sQc 3:20ewVEw7R2GAw 4:10vrzSg43HfWI 5:10xCP/g3c1sQc 6:10.6.TrD/0sas 7:trEJ1uz6ul/WU 8:$1$12$1YZMQPfPxS3.kStGKUBCO. 9:e10adc3949ba59abbe56e057f20f883e 10:$LM$b757bf5c0d87772f Rename your john.pot file before continuing. (Actually, the jumbo-patched john has an option --pot=NAME, but I never used it... That's why I just renamed john.pot) The next step probably requires the jumbo patch, due to --show=LEFT. (And, for our use case, it requires an empty pot file.) for f in DES BSDI MD5 BF AFS LM; do ./john --format=$f --show=LEFT cracked_hashes > cracked_hashes_$f; done You may also write it across multiple lines, the shell will prompt you for the continuation: for f in DES BSDI MD5 BF AFS LM > do > ./john --format=$f --show=LEFT cracked_hashes > cracked_hashes_$f > done Check how the hashes are distributed according to hash type: wc -l cracked_hashes_* $ wc -l cracked_hashes_* 0 cracked_hashes_AFS 0 cracked_hashes_BF 0 cracked_hashes_BSDI 5 cracked_hashes_DES 4 cracked_hashes_LM 1 cracked_hashes_MD5 10 total On first glance, this looks good. However, I know I included 6 DES hashes (one duplicate). 2:10xCP/g3c1sQc 6:10xCP/g3c1sQc Just 5 of these appear in cracked_hashes_DES, because the duplicate has been removed. (I'm not sure if this is unintended or not. If a user wants to remove cracked hashes, but nevertheless continue to use --single mode, keeping the duplicates might be better.) So, where does the additional hash come from? I included 2 LM hashes: 1:$LM$44efce164ab921ca 10:$LM$b757bf5c0d87772f cracked_hashes_LM, however, contains (in addition to the expected content): 9:2:$LM$be56e057f20f883e 9:1:$LM$e10adc3949ba59ab These 2 additional lines are the first and second half of the hash e10adc3949ba59abbe56e057f20f883e in line 9 of my original cracked_hashes file. (While john stores LM hashes with a preceding $LM$ marker, the algorithm internally accepts output generated by other tools. That's why there's an ambiguity.) e10adc3949ba59abbe56e057f20f883e is a raw-MD5 hash which didn't appear in any cracked_hashes_* file, because I just used the format list of the unpatched john. Using the complete list: $ for f in DES BSDI MD5 BF AFS LM NT XSHA PO raw-MD5 MD5-gen \ > IPB2 raw-sha1 md5a hmac-md5 phpass-md5 KRB5 bfegg \ > nsldap ssha openssha oracle oracle11 MYSQL \ > mysql-sha1 mscash lotus5 DOMINOSEC \ > NETLM NETNTLM NETLMv2 NETNTLMv2 NETHALFLM \ > mssql mssql05 epi phps mysql-fast pix-md5 sapG \ > sapB md5ns HDAA DMD5 crypt > do > ./john --format=$f --show=LEFT cracked_hashes > cracked_hashes_$f > done produces this output (to stderr): Generic crypt(3) module: hash encoding string length 20, type id $L appears to be unsupported on this system; will not load such hashes. Generic crypt(3) module: hash encoding string length 32, type id #0 appears to be unsupported on this system; will not load such hashes. And this list of files: $ wc -l cracked_hashes_* 0 cracked_hashes_AFS 0 cracked_hashes_BF 0 cracked_hashes_bfegg 0 cracked_hashes_BSDI 6 cracked_hashes_crypt 5 cracked_hashes_DES 0 cracked_hashes_DMD5 0 cracked_hashes_DOMINOSEC 0 cracked_hashes_epi 0 cracked_hashes_HDAA 0 cracked_hashes_hmac-md5 0 cracked_hashes_IPB2 0 cracked_hashes_KRB5 4 cracked_hashes_LM 1 cracked_hashes_lotus5 1 cracked_hashes_MD5 0 cracked_hashes_md5a 0 cracked_hashes_MD5-gen 0 cracked_hashes_md5ns 1 cracked_hashes_mscash 0 cracked_hashes_mssql 0 cracked_hashes_mssql05 0 cracked_hashes_MYSQL 0 cracked_hashes_mysql-fast 0 cracked_hashes_mysql-sha1 0 cracked_hashes_NETHALFLM 0 cracked_hashes_NETLM 0 cracked_hashes_NETLMv2 0 cracked_hashes_NETNTLM 0 cracked_hashes_NETNTLMv2 0 cracked_hashes_nsldap 0 cracked_hashes_NT 0 cracked_hashes_openssha 0 cracked_hashes_oracle 0 cracked_hashes_oracle11 0 cracked_hashes_phpass-md5 0 cracked_hashes_phps 0 cracked_hashes_pix-md5 0 cracked_hashes_PO 1 cracked_hashes_raw-MD5 0 cracked_hashes_raw-sha1 0 cracked_hashes_sapB 0 cracked_hashes_sapG 0 cracked_hashes_ssha 0 cracked_hashes_XSHA 19 total A mysteriously increased number of hashes? $ sort -u cracked_hashes_*|wc -l In total I have 12 different hashes. Several hashes appear to be valid for multiple hash types. $ sort cracked_hashes_*|uniq -c |grep -v "^ *1 " 2 2:10xCP/g3c1sQc 2 3:20ewVEw7R2GAw 2 4:10vrzSg43HfWI 2 6:10.6.TrD/0sas 2 7:trEJ1uz6ul/WU 2 8:$1$12$1YZMQPfPxS3.kStGKUBCO. 2 9:e10adc3949ba59abbe56e057f20f883e These are the duplicates. e10adc3949ba59abbe56e057f20f883e appears twice, because it is valid for lotus5 and raw-MD5. (Additionally, it has been stored in cracked_hashes_mscash as 9:M$9#e10adc3949ba59abbe56e057f20f883e.) And crypt recognizes all hash types supported by your system, see man 3 crypt That means you could exclude crypt from the list of "hash formats", unless you used john --format=crypt to crack hashes not currently supported by an individual implementation (SHA-256 or SHA-512). I didn't include any SHA-256 or SHA-512 hash. After removing cracked_hashes_crypt, these hashes remain: $ $ sort -n cracked_hashes_* 1:$LM$44efce164ab921ca 2:10xCP/g3c1sQc 3:20ewVEw7R2GAw 4:10vrzSg43HfWI 6:10.6.TrD/0sas 7:trEJ1uz6ul/WU 8:$1$12$1YZMQPfPxS3.kStGKUBCO. 9:1:$LM$e10adc3949ba59ab 9:2:$LM$be56e057f20f883e 9:e10adc3949ba59abbe56e057f20f883e 9:e10adc3949ba59abbe56e057f20f883e 9:M$9#e10adc3949ba59abbe56e057f20f883e 10:$LM$b757bf5c0d87772f Line 5 of my original cracked_hashes cracked hashes file disappeared (same hash as in line 2). Line 9 produced 5 lines of "output", 4 of them unwanted: The 2 lines in cracked_hashes_LM, one in cracked_hashes_mscash, and either the one in cracked_hashes_lotus5 or the one in cracked_hashes_raw-MD5. Because the hash e10adc3949ba59abbe56e057f20f883e appears unchanged in cracked_hashes_lotus5 and in cracked_hashes_raw-MD5, the only way to find which is the right format is: Try to crack the password again, using the contents of your original john.pot file. Before you continue working with john, don't forget to rename the .pot file back to john.pot. Unless I misunderstood your requirement, I think there's no need to develop a new tool. (On the other side, a tool could remove some of the ambiguity I mentioned. But this could also be done with some more thought put into some scripting.) Regards, Frank
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.