Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Mon, 21 Apr 2014 19:05:52 +0200
From: magnum <>
Subject: Re: mmap(2) in wordlist.c

On 2014-04-21 16:51, Rich Rumble wrote:
> On Sun, Apr 20, 2014 at 8:34 PM, magnum <> wrote:
>> Is there any reason for not using Posix memory mapping for wordlist mode's
>> memory buffer in Jumbo? It should make it simpler and a lot more effective
>> for concurrent sessions including but not limited to forked ones. Or is
>> there a problem I fail to see?

I started looking at this. The current implementation alters the buffer 
(turns CR/LF into NULLs) and we can't do this with a shared read-only 
buffer. It also creates a char array (which is 114 MB alone, for 
Rockyou) so you can address any given word with an index and use it just 
like that. This is not compatible with (so silently disabled for) 
external filters.

One solution would be to scrap that array and make something like 
fgetl() for the mapped buffer. This would sometimes be a tad slower than 
current solution but for some situations it would be a lot better. Think 
32 separate processes using the Rockyou list - all using the same 140 MB 
buffer instead of a total of 4 GB of duplicated data. And it would work 
just fine for external filters.

Theoretically, for node/fork/MPI we could leapfrog *pages* instead of 
words - that would be great for MPI. However, words would often span 
page boundaries and we should go with the KISS paradigm.

> Would cygwin translate/emulate that? Or would it need to "fall-back" to
> what's being done now, I would have no worldly idea, just first thing that
> sprang to mind for me.

We'd just make a trivial wrapper for Windows that turns mmap() into 
CreateFileMapping() - like this one (which is public domain):


Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.