Date: Mon, 21 Apr 2014 19:05:52 +0200 From: magnum <john.magnum@...hmail.com> To: john-dev@...ts.openwall.com Subject: Re: mmap(2) in wordlist.c On 2014-04-21 16:51, Rich Rumble wrote: > On Sun, Apr 20, 2014 at 8:34 PM, magnum <john.magnum@...hmail.com> wrote: >> Is there any reason for not using Posix memory mapping for wordlist mode's >> memory buffer in Jumbo? It should make it simpler and a lot more effective >> for concurrent sessions including but not limited to forked ones. Or is >> there a problem I fail to see? I started looking at this. The current implementation alters the buffer (turns CR/LF into NULLs) and we can't do this with a shared read-only buffer. It also creates a char array (which is 114 MB alone, for Rockyou) so you can address any given word with an index and use it just like that. This is not compatible with (so silently disabled for) external filters. One solution would be to scrap that array and make something like fgetl() for the mapped buffer. This would sometimes be a tad slower than current solution but for some situations it would be a lot better. Think 32 separate processes using the Rockyou list - all using the same 140 MB buffer instead of a total of 4 GB of duplicated data. And it would work just fine for external filters. Theoretically, for node/fork/MPI we could leapfrog *pages* instead of words - that would be great for MPI. However, words would often span page boundaries and we should go with the KISS paradigm. > Would cygwin translate/emulate that? Or would it need to "fall-back" to > what's being done now, I would have no worldly idea, just first thing that > sprang to mind for me. We'd just make a trivial wrapper for Windows that turns mmap() into CreateFileMapping() - like this one (which is public domain): https://github.com/hwoarang/uClibc/blob/master-metag/utils/mmap-windows.c magnum
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.