Date: Mon, 21 Apr 2014 23:06:07 +0200 From: magnum <john.magnum@...hmail.com> To: john-dev@...ts.openwall.com Subject: Re: mmap(2) in wordlist.c On 2014-04-21 19:05, magnum wrote: > I started looking at this. The current implementation alters the buffer > (turns CR/LF into NULLs) and we can't do this with a shared read-only > buffer. It also creates a char array (which is 114 MB alone, for > Rockyou) so you can address any given word with an index and use it just > like that. This is not compatible with (so silently disabled for) > external filters. > > One solution would be to scrap that array and make something like > fgetl() for the mapped buffer. This would sometimes be a tad slower than > current solution but for some situations it would be a lot better. Think > 32 separate processes using the Rockyou list - all using the same 140 MB > buffer instead of a total of 4 GB of duplicated data. Actually it's even worse: Including the mentioned array (it's 109 MB rather than 114), rockyou in 32 processes would eat 7.7 GB with current code compared to a shared map, no array, totalling a mere 133 MB (the actual file size x1). magnum
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.