Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date: Tue, 27 Oct 2009 14:31:19 -0500
From: "JFoug" <>
To: <>
Subject: My patches (Performance, options, mingw, vc, generic md5, phpass, etc)

I have unified all of my released patches, into a single patch. This patch 
was derived from after jumbo-1 patch is applied

This can be found on the wiki at: 
(there is also an update to the md5-generic test files).

It is not just all of the prior patches packed into one, but it has pretty 
significant improvements to the generic MD5 code.

In the MD5, I have gotten things working as fast (if not faster), than the 
'native' functions.

Also, there is a way to link an existing format (or a new one), into the MD5 
generic code, as an 'engine' to do the work.  Thus, you can create a very 
'thin' format (only having 4 functions and a conversion function) to build, 
and those functions are pretty simple (validate, init (which now actually 
DOES something), getsalt and getbinary).   I have coded phpass and PHPS 
formats to use the generics.  There are many others which 'could' be done, 
but that is where I started.

There is now a way to build new formats into generic md5, WITHOUT having to 
add any code into john.  There is no parser, but lines can be added to 
john.conf that properly tells the md5 generic engine which 'primitive' 
functions to call, and provide 'testing' data.  The entire process is 
documented within the updated john.conf file (along with a handful of sample 
'formats').  There is also a doc/MD5_GENERIC readme file.   All formats 
under 1000 are reserved as john built-in's.  1000 and above are free to use 
(by adding them to john.conf).

There are many new command line options (as were in one of my prior 
patches).  However, there is another new option.  That option 
is --field-separator-char=c | \xHH   By default, john uses a ':' char (the 
colon).  This char is used in candidate files, in the john.pot, etc.   With 
this new option, you can specify (on the command line), to use a different 
character than the colon.  Thus, for formats which have a salt that uses the 
: char (or userID's with the : embedded), you can create an input file, 
replacing all of the 'separator' characters with a new character, and then 
on the command line, specify to john that this new character is the 
separator character to use.  NOTE this character also ends up in the 
john.pot file, AND lines in the john.pot file that do NOT use this separator 
(say some lines with a : character) will NOT be read as valid john.pot lines 
(when restoring).  There is one format, md5_gen(7) which requires usage of 
the --field-separator-char= switch.  That format is specfically for 
vBulletin format, with 3 byte salt, and all chars from space to ~ being 
valid to be in the salt (thus the : character is a valid salt char).  In the 
'test' file (which is md5_gen(7) format), the file requires 
ASCII 31, or --field-separator-char=\x19 to be used, or nothing is read from 
the file.

If there are any patch issues, compile issues (I know there are warnings 
building under OSX, but I do not get the warnings, so somone please list 
them), or of there are run time issues, then please follow up.

One BIG thing I saw in testing, was that the generic md5 (and the 'fast' 
MD5) are NOT working for MMX.  They work for generic x86 and for SSE2 
(tested only with MinGW, and with VC for x86 only, as that is all I have). 
So you you normally build with MMX, this patch should NOT be installed at 
this time.   I did have MMX (and SSE2) working at one time, so I will have 
to dig through changes, and see where it broke at, and why.  I have not done 
testing on using MMX, since it is really not that much improvement over 
generic, AND SSE2 is close to twice as fast as the MMX (when I had MMX 


Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.