Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [day] [month] [year] [list]
Date: Sat, 7 May 2011 21:35:10 -0500
From: "jfoug" <>
To: <>
Subject: RE: John core change patch (and md5-gen, etc)  (was john-users)

Sorry, sent to wrong list :(

>-----Original Message-----
>From: jfoug Sent: Saturday, May 07, 2011 9:34 PM
>v-2 of this change just posted.  This release has had a lot of back and
>forth testing (build testing and runtime detection testing).  I feel
>very good about this release now. It should be ready to go. I placed the
>post-jumbo-1-md5-2 patch there.   I have also built the diff against
>vanilla 1.7.7 and put it on the wiki also, calling it 1.7.7.jumbo-2-RC1-
>I know Solar hates bottom quotes, but I have left most of the original
>post here, so people can see what was originally listed as included.
>The changes for the -v2 are these:
>bf_fmt.c            init()
>phpassMD5_fmt_orig  init() valid() added fmt_default_prepare to
>PHPS_fmt_orig.c     init() valid() added fmt_default_prepare to
>SKEY_fmt.c          valid() added fmt_default_prepare to structure.
>md5_gen_fmt.c       Fixed numerous signed vs unsigned warnings
>md5_mmx.S           Fixed data align bug (causing crashes)
>format.c            $dummy% hash string was too short, and giving a
>'prepare' failure.  Same 'shorter' check was already in loader.c
>there were no bugs found in the md5_gen code (which I am glad about).
>There were many warnings about signed vs unsigned char pointers, etc.
>Those have been cleaned up. There were only a few spots, since they were
>mostly in #defines, but dozens and dozens of places they were used.  3
>of the missing older style format are not included in a 'normal' build,
>but I updated them to the new format any way.  The bf_fmt.c has it's
>init inside of an openmp which I do not build with, so it was missed.
>The crash bug in md5_mmx.S was never seen by me, because there was
>special #defines for MINGW32, so my size was already right.  However,
>the other non MINGW builds cored for SSE2 builds under a x64 compiler.
>NOTE, the same data alignment is there for SHA1_mmx.S, but crashing was
>not observed there.  The item missed in the format.c was due to reduced
>hash size of the dummy format.  It is new with 1.7.7 and something I had
>not accounted for, since most of my build/testing was done on 1.7.6 and
>is quite old, done over the last year or so, even though only released
>in full today.  The reduction from 13 bytes to 7 was already done in
>loader.c  However, I needed the same check in format.c after the call to
>the new prepare() function.
>I am actually happy that so few issues turned up, and that they were
>very trivial over all, since this was a pretty intrusive change, and
>since the md5_gen code was about a 20-30% rewrite/new code.  I think the
>core changes being release today should work well.
>There will soon be some patches release which provide 'thin' support for
>many formats.  The patch will move the existing 'fat' format to a new
>name, and then drop in a thin format to the original file name.  There
>are already a couple examples today, where the fat format is still
>hanging around, but the thin format has taken over (phpass, PHPS, etc).
>Also, there will be a patch sooner or later to get any changes needed to
>get the intrinsic code working properly and fully with md5_gen.  I know
>the new (27) and (28) (OpenBSD md5) will not work properly on MD%_PARA
>builds.  I am hopeful that is most of the changes, and that the other
>code 'still' works.  But whatever is needed, will be added.   There may
>also be some intrinsic stuff needed to get the SHA1 to use it in the
>md5_gen code (SSE2 is already supported, but it is a little slower than
>'native' SHA1).
>>-----Original Message-----
>>From: JimF Sent: Friday, May 06, 2011 7:12 PM
>>This has been put on the wiki.  It is actually 'based' as jumbo-2 (in
>>params.h, but that can be changed as needed). Here are the changes:
>>If one of these 'not included', is the recent changes to the 'generic'
>>(md5_gen), and the format changes (along with bench/format/loader),
>>hold up a little.  I have been working hard on it.  I have got these
>>Many of these items were in the 1.7.6 patch.  Many have been added
>>that patch.
>>New formats:
>>md5_gen(18) | PostOffice MD5 (PO)
>>md5_gen(19) | Cisco PIX
>>md5_gen(20) | Cisco PIX (salted)
>>md5_gen(21) | HTTP Digest Access Auth
>>md5_gen(22) | md5(sha1($p))
>>md5_gen(23) | sha1(md5($p))
>>md5_gen(24) | sha1($p.$s)
>>md5_gen(25) | sha1($s.$p)
>>md5_gen(26) | sha1-raw
>>md5_gen(27) | OpenBSD MD5  (SSE working but SSE-para has not been added
>>these 2 formats yet)
>>md5_gen(28) | Apache MD5
>>Significant changes to code in the md5_gen. I now try to use the
>>code, both the MD5_X2 and 'single', depending upon is listed in arch.h
>>NOTE to do this, I had to expose some functions within md5_std.c (they
>>static).  Also, I had to call the init function, for MD5_IMM builds
>>issue mad me pull my hair out for a while).  This was a big change, but
>>makes a big difference.
>>md5_gen is now working/tested with big-endian systems (only a X2
>>built and tested).  However, all changes should work on an X1 build.
>>were numerous changes needed to get many of the formats working. Also,
>>of the better optimizations had to be done differently, so as to not
>>have to
>>call swap functions all over the place.
>>The format structure has been changed.  All format files have been
>>New method is 'prepare'.  Also, the init and valid take a pointer to
>>format structure being used.
>>The md5 gen (from the interface john sees, the format structure), is
>>'normal' format, and not a format that morphs itself.  Thus when john
>>the md5_gen, it will
>>I have (again) gotten loader code updated (with the core changes).
>>there is one section I am not sure of.  That section is the code that
>>a 'bugus' salt, or something like that.
>>All 'special' code for md5_gen format has been removed from bench.c and
>>All 'special' loader code for formats has been removed from loader.c.
>>It is
>>now contained within the prepare function of the *_fmt.c files.
>>Changes made to md5-mmx.S and sha1-mmx.S (and the header files), to get
>>SSE2/MMX code working for VC builds.  To do this, you still have to
>>with MinGW (to build the .S files, into .o files).  The main changes
>>additions of marshal functions usable by VC to properly setup the stack
>>registers, prior to jumping to the 'real' functions. All of this code
>>properly #defined, so it will not impact any other build at all.
>>New 'filter' in john.conf
>>'simple' unicode conversions within md5_gen.  Not utf8->unicode, but
>>Fixed DOS formatting on DMD5_fmt.c
>>Added __declspec(align(16)) for VC (properly #defined) for all MMX/SSE2
>>NT format properly works with PWdump files, in prepare function. Prior
>>only LM worked.
>>Updated the PHPS 'thin' client to new md5_gen 'thin' code (needed since
>>Wrote many 'thin' formats, but they will be released later as
>>memory cleanup for 'tiny' memory routines.
>>in mysqlSHA1, reduce size of SSE memset, and perform last SSE to not
>>swap final. also new SSE SHA1 function.
>>the --subformat command line switch has been depricated.  Still there,
>>to nice --subform=list syntax
>>Changed many #if defined into #ifdef and if !defined into #ifndef,
>>saw many $ifdef _MSC_VER in 1.7.7 which were #if defined() before.
>>Bumped PLAINTEXT_LENGTH of ns_fmt from 15 to 25
>>Bumped PLAINTEXT_LENGTH of mssql05_fmt from 20 to 25
>>In mssql05_fmt, commented out setting the 2nd unicode bytes to 0. There
>>already is a memset, so these are redeundant and not needed.

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.