[<prev] [next>] [day] [month] [year] [list]
Date: Sat, 7 May 2011 21:35:10 -0500
From: "jfoug" <jfoug@....net>
To: <john-dev@...ts.openwall.com>
Subject: RE: John core change patch (and md5-gen, etc) (was john-users)
Sorry, sent to wrong list :(
>-----Original Message-----
>From: jfoug Sent: Saturday, May 07, 2011 9:34 PM
>
>v-2 of this change just posted. This release has had a lot of back and
>forth testing (build testing and runtime detection testing). I feel
>very good about this release now. It should be ready to go. I placed the
>post-jumbo-1-md5-2 patch there. I have also built the diff against
>vanilla 1.7.7 and put it on the wiki also, calling it 1.7.7.jumbo-2-RC1-
>diff.
>
>I know Solar hates bottom quotes, but I have left most of the original
>post here, so people can see what was originally listed as included.
>The changes for the -v2 are these:
>
>bf_fmt.c init()
>phpassMD5_fmt_orig init() valid() added fmt_default_prepare to
>structure.
>PHPS_fmt_orig.c init() valid() added fmt_default_prepare to
>structure.
>SKEY_fmt.c valid() added fmt_default_prepare to structure.
>md5_gen_fmt.c Fixed numerous signed vs unsigned warnings
>md5_mmx.S Fixed data align bug (causing crashes)
>format.c $dummy% hash string was too short, and giving a
>'prepare' failure. Same 'shorter' check was already in loader.c
>
>there were no bugs found in the md5_gen code (which I am glad about).
>There were many warnings about signed vs unsigned char pointers, etc.
>Those have been cleaned up. There were only a few spots, since they were
>mostly in #defines, but dozens and dozens of places they were used. 3
>of the missing older style format are not included in a 'normal' build,
>but I updated them to the new format any way. The bf_fmt.c has it's
>init inside of an openmp which I do not build with, so it was missed.
>The crash bug in md5_mmx.S was never seen by me, because there was
>special #defines for MINGW32, so my size was already right. However,
>the other non MINGW builds cored for SSE2 builds under a x64 compiler.
>NOTE, the same data alignment is there for SHA1_mmx.S, but crashing was
>not observed there. The item missed in the format.c was due to reduced
>hash size of the dummy format. It is new with 1.7.7 and something I had
>not accounted for, since most of my build/testing was done on 1.7.6 and
>is quite old, done over the last year or so, even though only released
>in full today. The reduction from 13 bytes to 7 was already done in
>loader.c However, I needed the same check in format.c after the call to
>the new prepare() function.
>
>I am actually happy that so few issues turned up, and that they were
>very trivial over all, since this was a pretty intrusive change, and
>since the md5_gen code was about a 20-30% rewrite/new code. I think the
>core changes being release today should work well.
>
>There will soon be some patches release which provide 'thin' support for
>many formats. The patch will move the existing 'fat' format to a new
>name, and then drop in a thin format to the original file name. There
>are already a couple examples today, where the fat format is still
>hanging around, but the thin format has taken over (phpass, PHPS, etc).
>
>Also, there will be a patch sooner or later to get any changes needed to
>get the intrinsic code working properly and fully with md5_gen. I know
>the new (27) and (28) (OpenBSD md5) will not work properly on MD%_PARA
>builds. I am hopeful that is most of the changes, and that the other
>code 'still' works. But whatever is needed, will be added. There may
>also be some intrinsic stuff needed to get the SHA1 to use it in the
>md5_gen code (SSE2 is already supported, but it is a little slower than
>'native' SHA1).
>
>Jim.
>
>>-----Original Message-----
>>From: JimF Sent: Friday, May 06, 2011 7:12 PM
>>
>>This has been put on the wiki. It is actually 'based' as jumbo-2 (in
>>the
>>params.h, but that can be changed as needed). Here are the changes:
>>
>>If one of these 'not included', is the recent changes to the 'generic'
>>(md5_gen), and the format changes (along with bench/format/loader),
>then
>>hold up a little. I have been working hard on it. I have got these
>>things.
>>Many of these items were in the 1.7.6 patch. Many have been added
>since
>>that patch.
>>
>>New formats:
>>
>>md5_gen(18) | PostOffice MD5 (PO)
>>md5_gen(19) | Cisco PIX
>>md5_gen(20) | Cisco PIX (salted)
>>md5_gen(21) | HTTP Digest Access Auth
>>md5_gen(22) | md5(sha1($p))
>>md5_gen(23) | sha1(md5($p))
>>md5_gen(24) | sha1($p.$s)
>>md5_gen(25) | sha1($s.$p)
>>md5_gen(26) | sha1-raw
>>md5_gen(27) | OpenBSD MD5 (SSE working but SSE-para has not been added
>>to
>>these 2 formats yet)
>>md5_gen(28) | Apache MD5
>>
>>Significant changes to code in the md5_gen. I now try to use the
>>'md5_body'
>>code, both the MD5_X2 and 'single', depending upon is listed in arch.h
>>NOTE to do this, I had to expose some functions within md5_std.c (they
>>were
>>static). Also, I had to call the init function, for MD5_IMM builds
>>(that
>>issue mad me pull my hair out for a while). This was a big change, but
>>makes a big difference.
>>
>>md5_gen is now working/tested with big-endian systems (only a X2
>>actually
>>built and tested). However, all changes should work on an X1 build.
>>There
>>were numerous changes needed to get many of the formats working. Also,
>>many
>>of the better optimizations had to be done differently, so as to not
>>have to
>>call swap functions all over the place.
>>
>>The format structure has been changed. All format files have been
>>updated.
>>New method is 'prepare'. Also, the init and valid take a pointer to
>the
>>format structure being used.
>>
>>The md5 gen (from the interface john sees, the format structure), is
>now
>>a
>>'normal' format, and not a format that morphs itself. Thus when john
>>loads
>>the md5_gen, it will
>>
>>I have (again) gotten loader code updated (with the core changes).
>NOTE
>>,
>>there is one section I am not sure of. That section is the code that
>>fixes
>>a 'bugus' salt, or something like that.
>>
>>All 'special' code for md5_gen format has been removed from bench.c and
>>format.c
>>
>>All 'special' loader code for formats has been removed from loader.c.
>>It is
>>now contained within the prepare function of the *_fmt.c files.
>>
>>Changes made to md5-mmx.S and sha1-mmx.S (and the header files), to get
>>SSE2/MMX code working for VC builds. To do this, you still have to
>>build
>>with MinGW (to build the .S files, into .o files). The main changes
>are
>>additions of marshal functions usable by VC to properly setup the stack
>>and
>>registers, prior to jumping to the 'real' functions. All of this code
>is
>>properly #defined, so it will not impact any other build at all.
>>
>>New 'filter' in john.conf
>>
>>'simple' unicode conversions within md5_gen. Not utf8->unicode, but
>>simple
>>ASCII->unicode
>>
>>Fixed DOS formatting on DMD5_fmt.c
>>
>>Added __declspec(align(16)) for VC (properly #defined) for all MMX/SSE2
>>data.
>>
>>NT format properly works with PWdump files, in prepare function. Prior
>>code
>>only LM worked.
>>
>>Updated the PHPS 'thin' client to new md5_gen 'thin' code (needed since
>>core
>>changes)
>>
>>Wrote many 'thin' formats, but they will be released later as
>individual
>>patches.
>>
>>memory cleanup for 'tiny' memory routines.
>>
>>in mysqlSHA1, reduce size of SSE memset, and perform last SSE to not
>>byte
>>swap final. also new SSE SHA1 function.
>>
>>the --subformat command line switch has been depricated. Still there,
>>due
>>to nice --subform=list syntax
>>
>>Changed many #if defined into #ifdef and if !defined into #ifndef,
>since
>>I
>>saw many $ifdef _MSC_VER in 1.7.7 which were #if defined() before.
>>
>>Bumped PLAINTEXT_LENGTH of ns_fmt from 15 to 25
>>
>>Bumped PLAINTEXT_LENGTH of mssql05_fmt from 20 to 25
>>
>>In mssql05_fmt, commented out setting the 2nd unicode bytes to 0. There
>>already is a memset, so these are redeundant and not needed.
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
