Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Tue, 15 Mar 2011 03:40:12 +0300
From: Solar Designer <>
Subject: Re: unique

On Mon, Feb 28, 2011 at 09:23:20AM -0600, jfoug wrote:
> Unique is nice, but you HAVE to bump up the params.h values a lot to make it
> work, on anything other than a tiny file.

You mean, to make it work "fast".  Because it does work on huge files
even with default settings, just slowly.

Also, I bumped up the defaults between 1.7.4 and  Starting with
that version, unique uses about 70 MB of RAM by default (previously, it
would use about 9 MB by default).  With these settings, unique is
reasonably usable on wordlists of up to a few hundred megabytes.  So
maybe your comment applies to a pre- version.

That said, I think a future version should have the memory buffer size
tunable from the command-line or/and in john.conf.  This has been on my
to-do for ages.

> I also hacked on this and added some things (but never released this code).
> I added a few extra command params.
>   -v 'verbose'  Lists some stuff.  If not verbose, it at least outputs # of
> lines read, and # output.


>   -inp=file     Allows file to be read, vs only reading from stdin.

What for?  Is it somehow more useful on Windows (IIRC, you're on Windows).

>   --2nd_uniq_file=file  Keeps any lines found in this file, from ending up
> in your output.
>   --2nd_uniq_file_only=file  Similar to prior, but assumes our input file is
> already unique, so does not unique against self, only against the 2nd file.

We need to come up with self-explanatory names for these options.

> Guess I should get these changes to Solar at some time, lol.  They do aide
> in wordlist maintainance.

Sure, please contribute - preferably upload a patch to the wiki.



Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.