Date: Sat, 5 Jan 2013 02:34:19 +0100 From: Frank Dittrich <frank_dittrich@...mail.com> To: john-dev@...ts.openwall.com Subject: Re: New plugin load order magic On 01/05/2013 01:10 AM, magnum wrote: > On 1 Jan, 2013, at 23:30 , Frank Dittrich <frank_dittrich@...mail.com> wrote: >> On 01/01/2013 10:48 PM, magnum wrote: >>> What we could do if you insist, is force raw-sha512 to load before blake. >> I do not insist. I am just unsure what the best solution is. >> Knowing what formats *could* possibly crack a given hash also is an >> important feature. >> May be you are right, and it is more important to let the user know that >> there are possibly more formats which could crack a given hash than >> allowing him not to specify --format=, and still expect a raw hash to be >> treated as raw-sha512. >> >>> Either by renaming one of the source files so the plugin globbing will put them in a different order, or by making raw-sha512 a non-plugin. >> >> No, spending/causing effort for such crude workarounds is not what I >> wanted, although, in this particular case it might be a bit unfortunate >> that BLAKE2 would grab raw-sha512 hastes, even if BLAKE2 probably isn't >> really used as a password hash function. >> There might be users who will complain on john-users when they are not >> longer able to crack raw-sha512 hashes if they forget the --format switch. >> >> May be this really should have been discussed on john-users as well, so >> tomorrow I'll try to get this discussed on john-users. > > After Lukas' struct name fixes, I opted to try the struct-name sort order, committed now. I took the chance to rename fmt_rawSHA512 to fmt_raw0_SHA512 which ensures it loads before the Blake2 format. BTW I also renamed (talking format *labels* now) blake2-512 to raw-blake2-512, I hope anyone doesn't disagree with that? > > Some review of fmt_registers.h did not reveal any other cases I should fix. The MSSQL formats are in correct order, as is the NETLM ones, raw-sha1, NT and . > > Here is the old load order: > (des, bsdi, md5, bf, afs, lm, dynamic_n) > bfegg, dmd5, dominosec, epi, fortigate, hdaa, ipb2, krb4, krb5, mschapv2, > netlm, netlmv2, netntlm, netntlmv2, nethalflm, md5ns, nt, phps, po, > sybasease, xsha512, xsha, chap, clipperz, crc32, sha256crypt, sha512crypt, > dmg, dragonfly3-32, dragonfly3-64, dragonfly4-32, dragonfly4-64, drupal7, > encfs, episerver, formspring, gost, gpg, hmac-sha224, hmac-sha256, > hmac-sha384, hmac-sha512, hmailserver, ike, keepass, keychain, keyring, > krb5pa-md5, krb5pa-sha1, lp, lastpass, lotus5, md4-gen, mediawiki, mongodb, > mscash, mscash2, mssql, mssql05, mssql12, mysql-sha1, mysql, mysqlna, npdf, > nsldap, nt2, nk, o5logon, odf, office, oldoffice, oracle11, oracle, osc, > pbkdf2-hmac-sha512, phpass, pix-md5, pkzip, postgre, pst, putty, pwsafe, > racf, radmin, blake2-512, raw-md4, raw-md5, raw-sha1, raw-sha1-linkedin, > raw-sha224, raw-sha256, raw-sha384, raw-sha512, raw-md5u, salted-sha1, sapb, > sapg, sha1-gen, sip, ssh-ng, strip, sunmd5, sxc, openvms, vnc, wbb3, wowsrp > (hmac-md5, hmac-sha1, raw-sha, django, tc_ripemd160, tc_sha512, tc_whirlpool, > raw-sha1-ng, crypt, trip, ssh, pfx, pdf, wpapsk, rar, zip, dummy) > > Here is the new one: > (des, bsdi, md5, bf, afs, lm, dynamic_n, bfegg) > dmd5, dominosec, epi, fortigate, formspring, hdaa, ipb2, krb4, krb5, > keepass, mschapv2, mysql, nethalflm, netlm, netlmv2, netntlm, netntlmv2, > md5ns, nt, nt2, osc, phps, po, sybasease, openvms, xsha, xsha512, wowsrp, > chap, clipperz, crc32, sha256crypt, sha512crypt, dmg, dragonfly3-32, > dragonfly3-64, dragonfly4-32, dragonfly4-64, drupal7, encfs, episerver, > gost, gpg, hmac-sha224, hmac-sha256, hmac-sha384, hmac-sha512, hmailserver, > ike, keychain, keyring, krb5pa-sha1, lp, lotus5, md4-gen, mediawiki, > mongodb, mscash, mscash2, krb5pa-md5, mssql, mssql05, mssql12, mysql-sha1, > mysqlna, nk, npdf, nsldap, o5logon, odf, office, oldoffice, oracle, > oracle11, pbkdf2-hmac-sha512, phpass, pix-md5, pkzip, postgre, pst, putty, > pwsafe, racf, radmin, raw-sha512, raw-blake2-512, raw-md4, raw-md5, > raw-sha1, raw-sha1-linkedin, raw-sha224, raw-sha256, raw-sha384, raw-md5u, > salted-sha1, sapb, sapg, sha1-gen, sip, lastpass, ssh-ng, strip, sunmd5, > sxc, vnc, wbb3, hmac-md5 > (hmac-sha1, raw-sha, django, tc_ripemd160, tc_sha512, tc_whirlpool, > raw-sha1-ng, crypt, trip, ssh, pfx, pdf, wpapsk, rar, zip, dummy) > > Please test/review/comment. I began testing this on my 32bit Linux system even without your commit (i.e., without the renaming of fmt_raw_SHA512 to fmt_raw0_SHA512, and with manually changing Makefile for the sort order). First, I built an OMP version with plugin formats sorted by LC_ALL=C sort. Then, another one using LC_ALL=C sort -r, for the reverse sort order. Afterwards, I started test runs for each of these two versions, and checked the output with relbench. (unstable-jumbo)run $ ./relbench test-john-sort.txt test-john-sort-r.txt More than one benchmark for NT MD4:Raw in file 1 More than one benchmark for NT MD4:Raw in file 2 Number of benchmarks: 293 Minimum: 0.76404 real, 0.76404 virtual Maximum: 1.73783 real, 1.35359 virtual Median: 0.99895 real, 0.99926 virtual Median absolute deviation: 0.00562 real, 0.00576 virtual Geometric mean: 0.99826 real, 0.99783 virtual Geometric standard deviation: 1.04313 real, 1.03040 virtual $ ./relbench -v test-john-sort.txt test-john-sort-r.txt |grep "^Ratio:"|cut -f 2- 0.76404 real, 0.76404 virtual MS SQL 2005 SHA-1:Only one salt 0.83397 real, 0.98742 virtual OpenPGP / GnuPG Secret Key:Only one salt 0.84306 real, 0.84306 virtual Oracle 11g SHA-1:Many salts 0.90196 real, 0.90196 virtual BLAKE2 512:Raw 0.92598 real, 0.95994 virtual CRC-32:Many salts 0.92773 real, 0.93744 virtual pfx:Raw 0.94533 real, 0.95016 virtual generic crypt(3) DES:Many salts 0.94980 real, 0.94980 virtual Raw SHA-1 LinkedIn:Raw 0.95147 real, 0.94198 virtual generic crypt(3) DES:Only one salt 0.96503 real, 0.97477 virtual dynamic_82: sha512($p.$s):Only one salt ... 1.03409 real, 1.01804 virtual EPiServer salted SHA-1/SHA-256:Only one salt 1.04082 real, 1.00388 virtual WPA-PSK PBKDF2-HMAC-SHA-1:Raw 1.04139 real, 1.04121 virtual EPiServer salted SHA-1/SHA-256:Many salts 1.04252 real, 1.02131 virtual PKZIP:Many salts 1.04392 real, 1.02818 virtual SIP MD5:Raw 1.05087 real, 1.02437 virtual PKZIP:Only one salt 1.05487 real, 1.05487 virtual MS SQL SHA-1:Only one salt 1.05829 real, 1.05829 virtual Raw SHA-1:Raw 1.11241 real, 1.11241 virtual MySQL 4.1 double-SHA-1:Raw 1.73783 real, 1.35359 virtual PostgreSQL MD5 challenge-response:Raw When I repeated the experiment, I noticed that even with the same john binary, I had such variations: $ ./relbench -v test-john-sort.txt test-john-sort2.txt |grep "^Ratio:"|cut -f 2-|sort -n 0.74209 real, 1.00234 virtual SXC SHA-1 Blowfish:Raw 0.95602 real, 0.95602 virtual dynamic_25: sha1($s.$p):Many salts 0.95703 real, 0.95573 virtual pfx:Raw 0.96532 real, 0.98522 virtual HalfLM C/R DES:Only one salt 0.97442 real, 1.00415 virtual dynamic_82: sha512($p.$s):Many salts 0.97468 real, 0.98952 virtual PDF MD5 SHA-2 RC4 / AES:Many salts 0.97595 real, 0.98092 virtual MySQL Network Authentication SHA1:Raw 0.97690 real, 0.99166 virtual Mac OS X 10.7+ salted SHA-512:Many salts 0.97710 real, 0.99660 virtual HalfLM C/R DES:Many salts 0.97721 real, 0.98221 virtual Fortigate FortiOS:Many salts ... 1.01613 real, 1.00592 virtual DragonFly BSD $3$ SHA-256 w/ bug, 64-bit:Many salts 1.01698 real, 1.01187 virtual DragonFly BSD $3$ SHA-256 w/ bug, 32-bit:Only one salt 1.01837 real, 1.01320 virtual SIP MD5:Raw 1.02181 real, 1.00658 virtual NTLMv1 C/R MD4 DES (ESS MD5):Only one salt 1.03106 real, 1.01046 virtual PDF MD5 RC4:Only one salt 1.03986 real, 1.03986 virtual generic crypt(3) DES:Only one salt 1.04056 real, 1.04586 virtual generic crypt(3) DES:Many salts 1.04082 real, 1.00971 virtual WPA-PSK PBKDF2-HMAC-SHA-1:Raw 1.04978 real, 1.01811 virtual PKZIP:Only one salt 1.69725 real, 1.34221 virtual PostgreSQL MD5 challenge-response:Raw I tried to keep the system idle besides the john --test, but who knows. May be I shouldn't have used OMP. (BTW: I think we really need another script to support relbench: a script which checks several benchmark files created with the same binary, just picks the best c/s rates for each hash format, and writes a new "best-of" benchmark result to stdout.) So, I started another test on my 64bit Linux system after fetching the latest git (latest commit: 018ddb99). Then, I uncommented HAVE_NSS, HAVE_GMP and HAVE_KRB5, used $ make clean; make linux-x86-64-native renamed ../run/john to ../run/john-sort Then, I changed "$(SORT)" to "$(SORT) -r" (both occurrences in Makefile) Then, to save some time, I just removed john.o, fmt_externs.h and mt_registers (this shouldn't be necessary because I changed Makefile). Then, I just used $ make linux-x86-64-native (without a make clean. Then, I renamed ../run/john to ../run/john-sort-r, and repeated the experiment I did on my 32bit Linux system. This time, the differences weren't that large (Ratio from 0.922 to 1.08, with one exception: The dummy format was 1.24 times faster with the john-sort-r binary! This was easily reproducable, even if I tested --format=dummy separately. 58949K vs. 73213K. For some reason I don't understand, the 2 binaries differed in size! john-sort: 2146752 bytes. john-sort-r: 2146816 bytes. When I do a $ make clean before I build the john binary with the reverted sort order, both binaries have the same size, but performance for --format=dummy drops to the performance of the john-sort version. When I change the Makefile again (use $(SORT) without -r), remove john.o, and build a new john version without a make clean, I get a binary of size of 2146816 bytes. The dummy format test gives between 62825K and 64214K now, which is somewhere between the c/s rates of the other binaries. Admittedly, dummy isn't the most important format. But I don't understand these fairly reproducible c/s rate differences. And I really don't know why the binaries differ in size. May be I should get some sleep and hope someone else can come up with an explanation. Frank
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.