[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date: Sat, 7 May 2011 21:33:59 -0500
From: "jfoug" <jfoug@....net>
To: <john-users@...ts.openwall.com>
Subject: RE: John core change patch (and md5-gen, etc)
v-2 of this change just posted. This release has had a lot of back and
forth testing (build testing and runtime detection testing). I feel very
good about this release now. It should be ready to go. I placed the
post-jumbo-1-md5-2 patch there. I have also built the diff against vanilla
1.7.7 and put it on the wiki also, calling it 1.7.7.jumbo-2-RC1-diff.
I know Solar hates bottom quotes, but I have left most of the original post
here, so people can see what was originally listed as included. The
changes for the -v2 are these:
bf_fmt.c init()
phpassMD5_fmt_orig init() valid() added fmt_default_prepare to structure.
PHPS_fmt_orig.c init() valid() added fmt_default_prepare to structure.
SKEY_fmt.c valid() added fmt_default_prepare to structure.
md5_gen_fmt.c Fixed numerous signed vs unsigned warnings
md5_mmx.S Fixed data align bug (causing crashes)
format.c $dummy% hash string was too short, and giving a
'prepare' failure. Same 'shorter' check was already in loader.c
there were no bugs found in the md5_gen code (which I am glad about). There
were many warnings about signed vs unsigned char pointers, etc. Those have
been cleaned up. There were only a few spots, since they were mostly in
#defines, but dozens and dozens of places they were used. 3 of the missing
older style format are not included in a 'normal' build, but I updated them
to the new format any way. The bf_fmt.c has it's init inside of an openmp
which I do not build with, so it was missed. The crash bug in md5_mmx.S
was never seen by me, because there was special #defines for MINGW32, so my
size was already right. However, the other non MINGW builds cored for SSE2
builds under a x64 compiler. NOTE, the same data alignment is there for
SHA1_mmx.S, but crashing was not observed there. The item missed in the
format.c was due to reduced hash size of the dummy format. It is new with
1.7.7 and something I had not accounted for, since most of my build/testing
was done on 1.7.6 and is quite old, done over the last year or so, even
though only released in full today. The reduction from 13 bytes to 7 was
already done in loader.c However, I needed the same check in format.c after
the call to the new prepare() function.
I am actually happy that so few issues turned up, and that they were very
trivial over all, since this was a pretty intrusive change, and since the
md5_gen code was about a 20-30% rewrite/new code. I think the core changes
being release today should work well.
There will soon be some patches release which provide 'thin' support for
many formats. The patch will move the existing 'fat' format to a new name,
and then drop in a thin format to the original file name. There are already
a couple examples today, where the fat format is still hanging around, but
the thin format has taken over (phpass, PHPS, etc).
Also, there will be a patch sooner or later to get any changes needed to get
the intrinsic code working properly and fully with md5_gen. I know the new
(27) and (28) (OpenBSD md5) will not work properly on MD%_PARA builds. I am
hopeful that is most of the changes, and that the other code 'still' works.
But whatever is needed, will be added. There may also be some intrinsic
stuff needed to get the SHA1 to use it in the md5_gen code (SSE2 is already
supported, but it is a little slower than 'native' SHA1).
Jim.
>-----Original Message-----
>From: JimF Sent: Friday, May 06, 2011 7:12 PM
>
>This has been put on the wiki. It is actually 'based' as jumbo-2 (in
>the
>params.h, but that can be changed as needed). Here are the changes:
>
>If one of these 'not included', is the recent changes to the 'generic'
>(md5_gen), and the format changes (along with bench/format/loader), then
>hold up a little. I have been working hard on it. I have got these
>things.
>Many of these items were in the 1.7.6 patch. Many have been added since
>that patch.
>
>New formats:
>
>md5_gen(18) | PostOffice MD5 (PO)
>md5_gen(19) | Cisco PIX
>md5_gen(20) | Cisco PIX (salted)
>md5_gen(21) | HTTP Digest Access Auth
>md5_gen(22) | md5(sha1($p))
>md5_gen(23) | sha1(md5($p))
>md5_gen(24) | sha1($p.$s)
>md5_gen(25) | sha1($s.$p)
>md5_gen(26) | sha1-raw
>md5_gen(27) | OpenBSD MD5 (SSE working but SSE-para has not been added
>to
>these 2 formats yet)
>md5_gen(28) | Apache MD5
>
>Significant changes to code in the md5_gen. I now try to use the
>'md5_body'
>code, both the MD5_X2 and 'single', depending upon is listed in arch.h
>NOTE to do this, I had to expose some functions within md5_std.c (they
>were
>static). Also, I had to call the init function, for MD5_IMM builds
>(that
>issue mad me pull my hair out for a while). This was a big change, but
>makes a big difference.
>
>md5_gen is now working/tested with big-endian systems (only a X2
>actually
>built and tested). However, all changes should work on an X1 build.
>There
>were numerous changes needed to get many of the formats working. Also,
>many
>of the better optimizations had to be done differently, so as to not
>have to
>call swap functions all over the place.
>
>The format structure has been changed. All format files have been
>updated.
>New method is 'prepare'. Also, the init and valid take a pointer to the
>format structure being used.
>
>The md5 gen (from the interface john sees, the format structure), is now
>a
>'normal' format, and not a format that morphs itself. Thus when john
>loads
>the md5_gen, it will
>
>I have (again) gotten loader code updated (with the core changes). NOTE
>,
>there is one section I am not sure of. That section is the code that
>fixes
>a 'bugus' salt, or something like that.
>
>All 'special' code for md5_gen format has been removed from bench.c and
>format.c
>
>All 'special' loader code for formats has been removed from loader.c.
>It is
>now contained within the prepare function of the *_fmt.c files.
>
>Changes made to md5-mmx.S and sha1-mmx.S (and the header files), to get
>SSE2/MMX code working for VC builds. To do this, you still have to
>build
>with MinGW (to build the .S files, into .o files). The main changes are
>additions of marshal functions usable by VC to properly setup the stack
>and
>registers, prior to jumping to the 'real' functions. All of this code is
>properly #defined, so it will not impact any other build at all.
>
>New 'filter' in john.conf
>
>'simple' unicode conversions within md5_gen. Not utf8->unicode, but
>simple
>ASCII->unicode
>
>Fixed DOS formatting on DMD5_fmt.c
>
>Added __declspec(align(16)) for VC (properly #defined) for all MMX/SSE2
>data.
>
>NT format properly works with PWdump files, in prepare function. Prior
>code
>only LM worked.
>
>Updated the PHPS 'thin' client to new md5_gen 'thin' code (needed since
>core
>changes)
>
>Wrote many 'thin' formats, but they will be released later as individual
>patches.
>
>memory cleanup for 'tiny' memory routines.
>
>in mysqlSHA1, reduce size of SSE memset, and perform last SSE to not
>byte
>swap final. also new SSE SHA1 function.
>
>the --subformat command line switch has been depricated. Still there,
>due
>to nice --subform=list syntax
>
>Changed many #if defined into #ifdef and if !defined into #ifndef, since
>I
>saw many $ifdef _MSC_VER in 1.7.7 which were #if defined() before.
>
>Bumped PLAINTEXT_LENGTH of ns_fmt from 15 to 25
>
>Bumped PLAINTEXT_LENGTH of mssql05_fmt from 20 to 25
>
>In mssql05_fmt, commented out setting the 2nd unicode bytes to 0. There
>already is a memset, so these are redeundant and not needed.
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
