Openwall GNU/*/Linux - a small security-enhanced Linux distro for servers
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Wed, 12 Sep 2018 14:07:04 +0200
From: "JohnyKrekan" <krekan@...nykrekan.com>
To: <john-users@...ts.openwall.com>
Subject: Re: good program for sorting large wordlists

My question now is not about sorting but about the wordlist which you would 
use for hash testing and is already saved on disk, the smaller (with all 
words lowercase) or the bigger (mixed). Is it better to let the program for 
example EWSA make the case modifications or use bigger one and disable all 
the case modifying rules.
Johny Krekan

----- Original Message ----- 
From: "Solar Designer" <solar@...nwall.com>
To: <john-users@...ts.openwall.com>
Sent: Wednesday, September 12, 2018 1:49 PM
Subject: Re: [john-users] good program for sorting large wordlists


> On Wed, Sep 12, 2018 at 01:14:22PM +0200, JohnyKrekan wrote:
>> Thanx for infos, after I have raised the memory sizes and the space for
>> temp, the sort went well. Iwas sorting it to know how many duplicates 
>> (when
>> ignoring the character case) are in the superwpa wordlist. The original
>> file size was approx 10.7 gb, after sorting it was 7.05 gb, so 4 gb was
>> taken by the same words with modified character case.
>
> It's a case where you don't need to sort.  You could use:
>
> ./unique -v output.lst < input.lst
>
> or e.g.:
>
> tr 'A-Z' 'a-z' < input.lst | ./unique -v output.lst
>
> Testing this on JtR's bundled password.lst:
>
> $ tr 'A-Z' 'a-z' < password.lst | ./unique output.lst
> Total lines read 3559 Unique lines written 3422
>
> If you're interested in sizes in bytes as well, use "ls -l" or "wc -c"
> on the two files.
>
> For tiny wordlists like password.lst, "sort -u" is more convenient in
> that it can output to a pipe, so you can do:
>
> $ tr 'A-Z' 'a-z' < password.lst | sort -u | wc -l
> 3422
>
> But for large wordlists "sort" may be slower, even with the "-S" and
> "--parallel" options.
>
> Alexander
>
> 

Powered by blists - more mailing lists

Your e-mail address:

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.