Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date: Sun, 27 Apr 2014 9:44:33 -0400
From:  <>
Subject: Re: mmap()

>From an offlist request.  I thought this might be good information for the list, to see how cygwin, which is still limited to the diskIO constraints of Win32 (which SUCK) behaves. It does work with magnums mmap port (had to adjust some defines in the include section, but it was no big deal).  I have no idea how real-world ready this code is at the moment.  magnum has this in a separate tree in git, so I assume it is still being actively developed, and not ready for real world.

My testing:

Cygwin testing (on crappy laptop).  pw file is 1 rawmd5 at very end of rock-u passwords

Command line:
$ ../run/john in -form=raw-md5 -w=/dictionaries/RockYou_UserAccount-passwords.txt

$ wc /dictionaries/RockYou_UserAccount-passwords.txt
 7457743  7503103 70833938 /dictionaries/RockYou_UserAccount-passwords.txt

Testing with mmap:
Loaded 1 password hash (Raw-MD5 [MD5 128/128 SSE2 12x])
jucarius         (u0-RawMD5)
1g 0:00:00:02 DONE (2014-04-27 08:09) 0.4452g/s 3320Kp/s 3320Kc/s 3320KC/s 1616888..jucarius

Testing with bleeding (no memory buf)
1g 0:00:00:04 DONE (2014-04-27 08:14) 0.2157g/s 1609Kp/s 1609Kc/s 1609KC/s 1616888..jucarius

Testing with bleeding (memory buf large enough to hold entire file)
1g 0:00:00:01 DONE (2014-04-27 08:17) 0.5341g/s 3983Kp/s 3983Kc/s 3983KC/s 1616888..jucarius

With rules:  (Note, input hash busted, so the full rules would run)

Testing with mmap:
0g 0:00:01:04 DONE (2014-04-27 08:21) 0g/s 1950Kp/s 1950Kc/s 1950KC/s Ddiinnkkoing..Jucariusing
Testing with bleeding (no memory buf)
0g 0:00:03:56 DONE (2014-04-27 08:27) 0g/s 535200p/s 535200c/s 535200C/s Ddiinnkkoing..Jucariusing
Testing with bleeding (memory buf large enough to hold entire file)
0g 0:00:00:51 DONE (2014-04-27 08:22) 0g/s 2479Kp/s 2479Kc/s 2479KC/s Ddiinnkkoing..Jucariusing

For comparison:

$ ../run/john -test=5 -form=raw-md5
Benchmarking: Raw-MD5 [MD5 128/128 SSE2 12x]... DONE
Raw:    13875K c/s real, 13876K c/s virtual

So mmap was 83% of mem buffer for a flat run and 78.6% speed for rules.  But even the fastest run was only 28.7% the speed of bench, so we are currently sucking a lot of hot air with wordlist mode.


---- magnum <> wrote: 
> Could you try checking out the "mmap-array" branch and see if it builds 
> at all under Cygwin and/or VC or MinGW. It uses mmap() and I believe 
> Cygwin will handle it as-is, while non-Cygwin will need a wrapper that 
> is included but not tested. The wrapper is .c but is #included in 
> wordlist.c (when "windows but not cygwin") so inlined.
> So first question is, does it build at all? Second question is, how's 
> performance?
> I have tested following on OSX and Linux:
> 1. Rockyou list against a single NT hash
> 2. Rockyou list against a single NT hash + rules
> 3. Rockyou list against a single NT hash + rules and --mem=0
> all above cases with --fork=4
> In all cases this code is faster, or as fast. In some cases it's 50-200% 
> faster than current bleeding.
> There is also SSE2 code alternative for the mgetl() [which is a mmap 
> version of fgetl()] but it's commented out for now. It boosted Linux a 
> little but totally ruined performance on OSX. Actually that was probably 
> not because of OS but because of different CPUs. Some CPUs have a lot of 
> penalty for unaligned SSE2, others have more or less none. Anyway the 
> boost on Linux was so small I'll probably just scrap that part.
> magnum

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.