[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Sat, 12 Jan 2013 20:17:42 +0400
From: Solar Designer <solar@...nwall.com>
To: john-users@...ts.openwall.com
Subject: Re: parallel sort of text files under Windows
On Sat, Jan 12, 2013 at 08:48:46AM -0700, RB wrote:
> On Sat, Jan 12, 2013 at 8:18 AM, Solar Designer <solar@...nwall.com> wrote:
> > So if you're on a machine with e.g. only 4 GB RAM, the "unique" approach
> > is likely faster (since it's a simpler task). If you're on a machine
> > with e.g. 16 GB RAM or more, the "sort" approach is likely faster (since
> > it can use more RAM).
>
> This is actually a huge win with nearly any modern size of memory.
Of course, but at sizes below 2 GB "unique" is always faster than sort
(when using one CPU core).
sort becomes faster than "unique" (which is limited to 2 GB at most)
somewhere between 2 GB and 16 GB (I did not verify at what size exactly,
could be somewhere around 4 GB) or/and when you use multiple CPU cores
("unique" can't do that yet).
> -S and --parallel are some of my best friends.
Out of these two, -S makes a lot more difference than --parallel.
(Of course, you can use both.)
Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
