Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Mon, 21 Apr 2014 23:56:03 +0200
From: magnum <>
Subject: Re: mmap(2) in wordlist.c

On 2014-04-21 21:22, Solar Designer wrote:
> On Mon, Apr 21, 2014 at 02:34:13AM +0200, magnum wrote:
>> Is there any reason for not using Posix memory mapping for wordlist
>> mode's memory buffer in Jumbo? It should make it simpler and a lot more
>> effective for concurrent sessions including but not limited to forked
>> ones. Or is there a problem I fail to see?
> I think this is worth a try, while also keeping the present approach
> (for Windows, and for systems where it's faster).

That clashes with a hidden agenda though: A total rewrite of wordlist.c 
starting with core 1.8 version, and re-implementing Jumbo features as 
cleanly and KISS as possible. Unless real-world performance in some 
situations end up a lot worse than today's, I'd like to drop the old 
buffering altogether, including the --mem-file-size option which ideally 
should not be needed.

But it might be very hard to beat the current code. We have the once 
MPI-specific stuff that buffers "my" words only, now also applying to 
node/fork (btw this mitigates the 32x example I gave in the other post - 
but only for forked sessions). We also have non-consecutive dupes 
rejection applied before buffering so once everything is loaded, we 
don't need to leapfrog nor check for dupes - we just read next word from 
an array. Very effective, but also very ugly code because features were 
bolted on over time.

Atom's fgets-sse would obviously not speed up the current code a bit 
(when buffer is used) except at initial load. OTOH if I go with my 
latest thoughts about using a function similar to fgetl() except it 
reads the mapped buffer, it could/should use SSE. But it can probably 
never be as fast as the current buffer code except when we really 
benefit from not filling memory with duplicate copies of huge wordlists.

> I briefly experimented with read() vs. mmap() while developing popa3d,
> in 1998 or 1999.  At the time, read() turned out to be significantly
> faster, on Linux 2.0.x and ext2fs, when the read buffer size is properly
> tuned.  I didn't try this experiment in JtR.

What order of size was that at the time?


Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.