Openwall GNU/*/Linux - a small security-enhanced Linux distro for servers
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Date: Sun, 16 Sep 2018 12:51:54 -0700
From: jeff <jeff@...funit.com>
To: john-users@...ts.openwall.com
Subject: Re: good program for sorting large wordlists

On 9/16/2018 12:28, Albert Veli wrote:
> Hi!
>
> On Tue, Sep 11, 2018 at 5:19 PM JohnyKrekan <krekan@...nykrekan.com> wrote:
>
>> Hello, I would like to ask whether someone has experience with good tool
>> to sort large text files with possibilities such as gnu sort. I am using it
>> to sort wordlists but when I tried to sort 11 gb wordlist, it crashed while
>> writing final output file after writing around 7 gb of data  and did not
>> delete some temp files.
>
> If you don't succeed with other methods, one thing that has worked for me
> is splitting the wordlist into smaller parts and sorting each one
> individually. Then you can merge the sorted lists together using for
> instance mli2 from hashcat-utils. That will put the big list in sorted
> order. But the parts must be sorted first, before merging.
>
> This is only necessary for very large wordlists. Like in your case.
>
> PS I think there is a tool in hashcat-utils for splitting too. Don't
> remember the name. Maybe gate.
>
For whatever reason, I have a collection of large, sorted wordlists.
I have 9 over 10gb, and the biggest one is 123gb.

As mentioned above, I use split to split the files into manageable pieces.
I typically say something like 'split -l 100000000' or so.

I then use gnu sort on each piece.
gnu sort will sort files that are quite large, using temp files if needed.
It is still a good a good idea to have a reasonable amount of physical 
memory; my machine has 32 gb.

Then I take the sorted files and merge them using a program I wrote 
called multi-merge, which merges
one or more sorted files.

Then I use uniq on the sorted file, to remove duplicates.

This process can take awhile, but you will end up with a sorted, unique 
wordlist.

I also wrote a bunch of other programs to manipulate wordlists. In my 
experience, large wordlists often
contain quite a bit of junk, such as files with really long 
lines,sometimes 10k to over 100k bytes.
I have a program to truncate long lines, sample lines of big files, 
remove non-ascii lines, etc.
I also use emacs to look at the contents of files. It can edit 
multi-gigabyte files, though it is slow.
Sometimes long lines are many password separated by ',' or ';' or some 
other separator.

jeff

Powered by blists - more mailing lists

Your e-mail address:

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.