Date: Sat, 7 May 2011 21:35:10 -0500 From: "jfoug" <jfoug@....net> To: <john-dev@...ts.openwall.com> Subject: RE: John core change patch (and md5-gen, etc) (was john-users) Sorry, sent to wrong list :( >-----Original Message----- >From: jfoug Sent: Saturday, May 07, 2011 9:34 PM > >v-2 of this change just posted. This release has had a lot of back and >forth testing (build testing and runtime detection testing). I feel >very good about this release now. It should be ready to go. I placed the >post-jumbo-1-md5-2 patch there. I have also built the diff against >vanilla 1.7.7 and put it on the wiki also, calling it 1.7.7.jumbo-2-RC1- >diff. > >I know Solar hates bottom quotes, but I have left most of the original >post here, so people can see what was originally listed as included. >The changes for the -v2 are these: > >bf_fmt.c init() >phpassMD5_fmt_orig init() valid() added fmt_default_prepare to >structure. >PHPS_fmt_orig.c init() valid() added fmt_default_prepare to >structure. >SKEY_fmt.c valid() added fmt_default_prepare to structure. >md5_gen_fmt.c Fixed numerous signed vs unsigned warnings >md5_mmx.S Fixed data align bug (causing crashes) >format.c $dummy% hash string was too short, and giving a >'prepare' failure. Same 'shorter' check was already in loader.c > >there were no bugs found in the md5_gen code (which I am glad about). >There were many warnings about signed vs unsigned char pointers, etc. >Those have been cleaned up. There were only a few spots, since they were >mostly in #defines, but dozens and dozens of places they were used. 3 >of the missing older style format are not included in a 'normal' build, >but I updated them to the new format any way. The bf_fmt.c has it's >init inside of an openmp which I do not build with, so it was missed. >The crash bug in md5_mmx.S was never seen by me, because there was >special #defines for MINGW32, so my size was already right. However, >the other non MINGW builds cored for SSE2 builds under a x64 compiler. >NOTE, the same data alignment is there for SHA1_mmx.S, but crashing was >not observed there. The item missed in the format.c was due to reduced >hash size of the dummy format. It is new with 1.7.7 and something I had >not accounted for, since most of my build/testing was done on 1.7.6 and >is quite old, done over the last year or so, even though only released >in full today. The reduction from 13 bytes to 7 was already done in >loader.c However, I needed the same check in format.c after the call to >the new prepare() function. > >I am actually happy that so few issues turned up, and that they were >very trivial over all, since this was a pretty intrusive change, and >since the md5_gen code was about a 20-30% rewrite/new code. I think the >core changes being release today should work well. > >There will soon be some patches release which provide 'thin' support for >many formats. The patch will move the existing 'fat' format to a new >name, and then drop in a thin format to the original file name. There >are already a couple examples today, where the fat format is still >hanging around, but the thin format has taken over (phpass, PHPS, etc). > >Also, there will be a patch sooner or later to get any changes needed to >get the intrinsic code working properly and fully with md5_gen. I know >the new (27) and (28) (OpenBSD md5) will not work properly on MD%_PARA >builds. I am hopeful that is most of the changes, and that the other >code 'still' works. But whatever is needed, will be added. There may >also be some intrinsic stuff needed to get the SHA1 to use it in the >md5_gen code (SSE2 is already supported, but it is a little slower than >'native' SHA1). > >Jim. > >>-----Original Message----- >>From: JimF Sent: Friday, May 06, 2011 7:12 PM >> >>This has been put on the wiki. It is actually 'based' as jumbo-2 (in >>the >>params.h, but that can be changed as needed). Here are the changes: >> >>If one of these 'not included', is the recent changes to the 'generic' >>(md5_gen), and the format changes (along with bench/format/loader), >then >>hold up a little. I have been working hard on it. I have got these >>things. >>Many of these items were in the 1.7.6 patch. Many have been added >since >>that patch. >> >>New formats: >> >>md5_gen(18) | PostOffice MD5 (PO) >>md5_gen(19) | Cisco PIX >>md5_gen(20) | Cisco PIX (salted) >>md5_gen(21) | HTTP Digest Access Auth >>md5_gen(22) | md5(sha1($p)) >>md5_gen(23) | sha1(md5($p)) >>md5_gen(24) | sha1($p.$s) >>md5_gen(25) | sha1($s.$p) >>md5_gen(26) | sha1-raw >>md5_gen(27) | OpenBSD MD5 (SSE working but SSE-para has not been added >>to >>these 2 formats yet) >>md5_gen(28) | Apache MD5 >> >>Significant changes to code in the md5_gen. I now try to use the >>'md5_body' >>code, both the MD5_X2 and 'single', depending upon is listed in arch.h >>NOTE to do this, I had to expose some functions within md5_std.c (they >>were >>static). Also, I had to call the init function, for MD5_IMM builds >>(that >>issue mad me pull my hair out for a while). This was a big change, but >>makes a big difference. >> >>md5_gen is now working/tested with big-endian systems (only a X2 >>actually >>built and tested). However, all changes should work on an X1 build. >>There >>were numerous changes needed to get many of the formats working. Also, >>many >>of the better optimizations had to be done differently, so as to not >>have to >>call swap functions all over the place. >> >>The format structure has been changed. All format files have been >>updated. >>New method is 'prepare'. Also, the init and valid take a pointer to >the >>format structure being used. >> >>The md5 gen (from the interface john sees, the format structure), is >now >>a >>'normal' format, and not a format that morphs itself. Thus when john >>loads >>the md5_gen, it will >> >>I have (again) gotten loader code updated (with the core changes). >NOTE >>, >>there is one section I am not sure of. That section is the code that >>fixes >>a 'bugus' salt, or something like that. >> >>All 'special' code for md5_gen format has been removed from bench.c and >>format.c >> >>All 'special' loader code for formats has been removed from loader.c. >>It is >>now contained within the prepare function of the *_fmt.c files. >> >>Changes made to md5-mmx.S and sha1-mmx.S (and the header files), to get >>SSE2/MMX code working for VC builds. To do this, you still have to >>build >>with MinGW (to build the .S files, into .o files). The main changes >are >>additions of marshal functions usable by VC to properly setup the stack >>and >>registers, prior to jumping to the 'real' functions. All of this code >is >>properly #defined, so it will not impact any other build at all. >> >>New 'filter' in john.conf >> >>'simple' unicode conversions within md5_gen. Not utf8->unicode, but >>simple >>ASCII->unicode >> >>Fixed DOS formatting on DMD5_fmt.c >> >>Added __declspec(align(16)) for VC (properly #defined) for all MMX/SSE2 >>data. >> >>NT format properly works with PWdump files, in prepare function. Prior >>code >>only LM worked. >> >>Updated the PHPS 'thin' client to new md5_gen 'thin' code (needed since >>core >>changes) >> >>Wrote many 'thin' formats, but they will be released later as >individual >>patches. >> >>memory cleanup for 'tiny' memory routines. >> >>in mysqlSHA1, reduce size of SSE memset, and perform last SSE to not >>byte >>swap final. also new SSE SHA1 function. >> >>the --subformat command line switch has been depricated. Still there, >>due >>to nice --subform=list syntax >> >>Changed many #if defined into #ifdef and if !defined into #ifndef, >since >>I >>saw many $ifdef _MSC_VER in 1.7.7 which were #if defined() before. >> >>Bumped PLAINTEXT_LENGTH of ns_fmt from 15 to 25 >> >>Bumped PLAINTEXT_LENGTH of mssql05_fmt from 20 to 25 >> >>In mssql05_fmt, commented out setting the 2nd unicode bytes to 0. There >>already is a memset, so these are redeundant and not needed.
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.