Date: Wed, 2 Jan 2013 22:52:42 +0100 From: Frank Dittrich <frank_dittrich@...mail.com> To: john-users@...ts.openwall.com Subject: Multiple formats accepting the same raw hashes Hi all, with the ever-increasing number of hash formats supported by John the Ripper, it is inevitable, at least for raw hashes, that there are multiple hash formats which "understand" the same raw input. Yesterday, we had a short discussion of possible issues resulting from this fact on john-dev, and we felt we might need input from john users for the decision how to handle such cases. (After a somewhat lengthy explanation, there will be a few questions for you at the end of my mail.) A recent example is the BLAKE2 format that has been added to the github repository. Prior to the introduction of this format, the following hashes would have been interpreted as raw-sha512 hashes: 4245af08b46fbb290222ab8a68613621d92ce78577152d712467742417ebc1153668f1c9e1ec1e152a32a9c242dc686d175e087906377f0c483c5be2cb68953e b109f3bbbc244eb82441917ed06d618b9008dd09b3befd1b5e07394c706a8bb980b1d7785e5976ec049b46df5f1326af5a2ea6d103fd07c95385ffab0cacbc86 $ ./john testpw Loaded 2 password hashes with no different salts (Raw SHA-512 [32/32 OpenSSL]) guesses: 0 time: 0:00:00:02 0.00% (3) c/s: 169771 trying: sposy - serrun guesses: 0 time: 0:00:00:03 0.00% (3) c/s: 195360 trying: hEd - borrix guesses: 0 time: 0:00:00:04 0.00% (3) c/s: 202503 trying: jazm21 - jethon Session aborted Now that BLAKE2 has been added to github, this is what happens if you use the latest git snapshot: $ ./john testpw Warning: detected hash type "blake2-512", but the string is also recognized as "raw-sha512" Use the "--format=raw-sha512" option to force loading these as that type instead Loaded 2 password hashes with no different salts (BLAKE2 512 [32/32]) guesses: 0 time: 0:00:00:02 0.00% (3) c/s: 139432 trying: critas - monniel guesses: 0 time: 0:00:00:03 0.00% (3) c/s: 151740 trying: 49382609 - 49184819 Session aborted This means, the above hashes are now ambiguous. Because there was no --format= option on the command line, john happens to use the first format which identifies one of the hashes in the input file as valid, and uses this format (here: blake2-512). However, john also mentions which other hash formats would support at least one of the hashes in the input file (here: raw-sha512). If you want to force a specific format to be used, you'll have to use the --format= option, either if you just want to suppress the checks which other formats support the hashes given in the input file, or if you want to make sure a certain format gets used: $ ./john testpw --format=blake2-512 Loaded 2 password hashes with no different salts (BLAKE2 512 [32/32]) guesses: 0 time: 0:00:00:02 0.00% (3) c/s: 154519 trying: meneste - saa Session aborted or $ ./john testpw --format=raw-sha512 Loaded 2 password hashes with no different salts (Raw SHA-512 [32/32 OpenSSL]) guesses: 0 time: 0:00:00:03 0.00% (3) c/s: 152979 trying: crestert - 0700013 Session aborted In most cases, formats which accept raw hashes allow to use a prefix, to avoid ambiguity. E.g., if --format= is not used, this hash will be interpreted as a blake2-512 hash: $BLAKE2$4245af08b46fbb290222ab8a68613621d92ce78577152d712467742417ebc1153668f1c9e1ec1e152a32a9c242dc686d175e087906377f0c483c5be2cb68953e And this one as a raw-sha512 hash: $SHA512$b109f3bbbc244eb82441917ed06d618b9008dd09b3befd1b5e07394c706a8bb980b1d7785e5976ec049b46df5f1326af5a2ea6d103fd07c95385ffab0cacbc86 When john finds the password, it always stores the format specific hash representation in john.pot, not the raw hash, even if the input file just contains raw hashes. Also, when you start a john session without specifying --format, the format which is used gets written into the .rec file. That means, if you interrupt a session and later restore it, john will continue to use the same format (even if you upgraded john, and the new version knows more hash formats than the old version). To summarize: You don't risk to loose any work if a newer john version supports more hash formats, but you can't rely on the assumption that future john versions will per default use the same hash format as earlier john versions used on the same input file. To ensure that raw SHA512 hashes will only be interpreted as raw-sha512 format, we would have to implement solutions or find workarounds which also have disadvantages. We could not allow new formats to be added which accept raw hashes, if the same raw hashes are already accepted by another format. Disadvantage: If you only have raw hashes, you might not know what hash algorithm has been used to create these hashes. In this case, it is a feature that john lists all the possible hash formats which you can try to use. We could allow new formats to support raw hashes, but make sure that the format which in older john versions is the only format supporting these hashes will be the one which is used as the default format (i.e., will be used unless --format= is specified). Disadvantages: Additional effort for developers when implementing new formats and for future maintenance is required. While currently the only format supporting a particular raw hash is probably also the most frequently used format for such a raw hash, this might change in future. (IMO, this doesn't really apply in this case: neither raw SHA512 nor raw BLAKE2-512 are good password hash algorithms, so I hope we won't see many raw BLAKE2-512 hashes in the future. But we would aim to find a general solution, nut just a crude workaround which only applies in this particular case,) So, at the end of a long mail, finally a few questions: How important is it for you that john mentions which supported hash algorithms might be used to crack a given set of hashes, instead of silently using the first hash format which supported raw hashes of a particular fixed length? How important is it for you that the default hash format when starting a new password crack session on the same input file remains the same even across different john versions? (The same question asked in another way: Would you like developers to spend time on this "problem" instead of adding new hash formats, improving performance of existing formats, fixing real bugs?) Frank
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.