john-dev - unique using more than 2 GB RAM

Date: Sat, 28 Jan 2012 00:48:35 +0400
From: Solar Designer <solar@...nwall.com>
To: john-dev@...ts.openwall.com
Subject: unique using more than 2 GB RAM

magnum, Jim -

Here's a wish list item:

We should make "unique" able to use more than 2 GB of memory.  As it
turns out, "unique" at 2 GB is about twice slower than "sort -u -S 14G"
(on a 16 GB RAM machine), although of course this may vary by input
data.  Maybe "unique" should start using 40-bit offsets (good for up to
1 TB of RAM).  I am concerned that it will become slightly less
efficient (in terms of both speed and memory usage) when this new
functionality is not being made use of, though.

Alexander

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.