[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Mon, 21 Apr 2014 19:05:52 +0200
From: magnum <john.magnum@...hmail.com>
To: john-dev@...ts.openwall.com
Subject: Re: mmap(2) in wordlist.c
On 2014-04-21 16:51, Rich Rumble wrote:
> On Sun, Apr 20, 2014 at 8:34 PM, magnum <john.magnum@...hmail.com> wrote:
>> Is there any reason for not using Posix memory mapping for wordlist mode's
>> memory buffer in Jumbo? It should make it simpler and a lot more effective
>> for concurrent sessions including but not limited to forked ones. Or is
>> there a problem I fail to see?
I started looking at this. The current implementation alters the buffer
(turns CR/LF into NULLs) and we can't do this with a shared read-only
buffer. It also creates a char array (which is 114 MB alone, for
Rockyou) so you can address any given word with an index and use it just
like that. This is not compatible with (so silently disabled for)
external filters.
One solution would be to scrap that array and make something like
fgetl() for the mapped buffer. This would sometimes be a tad slower than
current solution but for some situations it would be a lot better. Think
32 separate processes using the Rockyou list - all using the same 140 MB
buffer instead of a total of 4 GB of duplicated data. And it would work
just fine for external filters.
Theoretically, for node/fork/MPI we could leapfrog *pages* instead of
words - that would be great for MPI. However, words would often span
page boundaries and we should go with the KISS paradigm.
> Would cygwin translate/emulate that? Or would it need to "fall-back" to
> what's being done now, I would have no worldly idea, just first thing that
> sprang to mind for me.
We'd just make a trivial wrapper for Windows that turns mmap() into
CreateFileMapping() - like this one (which is public domain):
https://github.com/hwoarang/uClibc/blob/master-metag/utils/mmap-windows.c
magnum
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
