Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date: Sat, 28 Jan 2012 00:48:35 +0400
From: Solar Designer <>
Subject: unique using more than 2 GB RAM

magnum, Jim -

Here's a wish list item:

We should make "unique" able to use more than 2 GB of memory.  As it
turns out, "unique" at 2 GB is about twice slower than "sort -u -S 14G"
(on a 16 GB RAM machine), although of course this may vary by input
data.  Maybe "unique" should start using 40-bit offsets (good for up to
1 TB of RAM).  I am concerned that it will become slightly less
efficient (in terms of both speed and memory usage) when this new
functionality is not being made use of, though.


Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.