Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Mon, 21 Apr 2014 23:06:07 +0200
From: magnum <>
Subject: Re: mmap(2) in wordlist.c

On 2014-04-21 19:05, magnum wrote:
> I started looking at this. The current implementation alters the buffer
> (turns CR/LF into NULLs) and we can't do this with a shared read-only
> buffer. It also creates a char array (which is 114 MB alone, for
> Rockyou) so you can address any given word with an index and use it just
> like that. This is not compatible with (so silently disabled for)
> external filters.
> One solution would be to scrap that array and make something like
> fgetl() for the mapped buffer. This would sometimes be a tad slower than
> current solution but for some situations it would be a lot better. Think
> 32 separate processes using the Rockyou list - all using the same 140 MB
> buffer instead of a total of 4 GB of duplicated data.

Actually it's even worse: Including the mentioned array (it's 109 MB 
rather than 114), rockyou in 32 processes would eat 7.7 GB with current 
code compared to a shared map, no array, totalling a mere 133 MB (the 
actual file size x1).


Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.