Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date: Sat, 14 May 2011 12:38:59 -0500
From: "JimF" <>
To: <>
Subject: Changes to john's unique program.

I am posting this noticication here ot john-dev, and not to john-users.  If 
it needs to also go to users, I will post it there also.

I have placed a patch to the unique.c (unique program)  This patch can be 
placed over 1.7.7-jumbo1 (or the jumbo-2-rc1).  It adds this to unique:

Changes to unique (different than original 1.7.7-jumbo1).  The unique code 
will be in the soon to be released jumbo-3-RCx patch.

1. added -v         screen outputs number of lines starting and finishing. 
(mingw builds always output this)

2. -inp=fname       Allows using without being a filter.  Fixes some Win32 
issued (embedded EOF, last line missing EOL, etc).  Note, even in the 
rockYou word list, there is an EOF character.

3. -ex_file=fname   A 2nd file that is also used to unique the data against. 
Only unique data in input file also NOT in this file will be written.

4. -ex_file_only=fn Same as -ex_file BUT assumes current data is already 
unique, thus does not perform unique on original input data.

5. -cut=num         Cut the lines to num bytes long (or shorter).  Also, 
the -ex_file data is cut when read, so it properly uniques data.

6. -cut=LM          This will cut words to 7 bytes, AND if longer than 7 
bytes, will grab the next 7 bytes.  Also, data is upcased.  NOTE, this 
operation is NOT performed on the -ex_file data.  To use -ex_file, it should 
already be in LM format.

7. -mem=num         A number from 13 to 25 (default john build is 20).  This 
controls size of hash buffer, and input building buffer.  Default of 20 is 
hash is 1 million items, and buffer of 64MB.  Each number up is 2x larger, 
each num down is 1/2 size.

Additional information:
The command line swiches need to be in full, so -cut=lm 
or -ex_file_o=filename are not valid.
The "=num" is a valid number. for the -mem=, it is [13,25]  for -cut= num 
must be in range of [1,LINE_BUFFER_SIZE] which is [1,1024]
fname (or fn) is to be replaced with the name of a file.
if using -mem, be careful not to use to large of a number, (i.e. more than 
1/2 your memory, or about that).  Crashes will likely occur, if using over 
this, and obviously speed will be greatly reduced (due to memory thrashing).

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.