Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Mon, 28 Feb 2011 16:17:45 +0300
From: Solar Designer <>
Subject: unique (was: GSoC 2011)

Bartavelle -

On Mon, Feb 28, 2011 at 11:13:02AM +0100, bartavelle wrote:
> I also wrote a quick hack that helps computing that kind of huge
> dictionnaries that works like "sort -u" except it doesn't sort, and is
> thus MUCH faster.

Isn't this exactly what JtR's "unique" program does?  Perhaps you just
need to increase UNIQUE_HASH_LOG and UNIQUE_BUFFER_SIZE in params.h -
e.g., set them to 24 and 0x40000000, respectively, to let unique use a
little over 1 GB of RAM.  Have you tried that?  Is your tool faster
(if unique is configured to have comparable memory usage)?

BTW, I made some optimizations to unique.c in what will become 1.7.7,
achieving speedups of 12% to 25% on my tests (that's with default
settings, without an increase in memory usage).

Thank you for your feedback (including the rest of it, which I intend to
comment on later; I just don't have time at the moment).


Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.