Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [day] [month] [year] [list]
Date: Tue, 14 Aug 2012 00:41:02 +0400
From: Alexander Cherepanov <cherepan@...me.ru>
To: john-users@...ts.openwall.com
Subject: Alexander's writeup for CMIYC 2012

Hi!

This is the first time I participated in "Crack Me If You Can"
contest and as a member of john-users team I got much fun and a ton of
experience -- thank you, guys!

= Hardware

One home computer with Intel Core2 Quad CPU Q6600 @ 2.40GHz (4 cores,
4 threads) used for wordlists manipulations and for running scripts.

Two dual-cpu servers with Intel Xeon CPU X5670 @ 2.93GHz (24 cores,
48 threads) used for actual cracking.

= Software

John the Ripper (contest edition) plus some Perl and bash scripts.

= Preparation

Fortunately I found time to make some preparations before the contest.

1. I downloaded several full wikipedia dumps -- enwiki, dewiki,
eswiki, ruwiki -- and created wordlists (sorted by frequency) from
them approximately like this:

$ bunzip2 < enwiki-20120702-pages-articles.xml.bz2 | perl -lne 'print lc $& while /\w+/g' | sort | uniq -c | sort -rn | perl -pe 's/\s*\S+\s+//' | bzip2 > enwiki.lst.bz2

(not exact command line used). Tried to do the same with Gutenberg but
it was much bigger and slower and didn't finish in time.

2. I wrote a script to generate dates in various formats and a script
to calculate salts distribution for each format. Plus some other
scripts were sketched during PHD2012 and trial pseudo-contests.

3. I chose and installed ircd on our contest server.

= Contest: challenges

Challenges were time-critical so I started by looking into
challenges. IMHO it's very important to convert source files into the
form which john can load and do it as early as possible so that all
the team can start crack them instead of fighting with converting.

john readily supported encfs, odt, pdf, rar, ssh, truecrypt and zip --
seven, not bad, eh? Well, encfs2john and odf2john didn't run with the
python from my Debian Stable, zip2john failed on contest files,
pdf2john completely failed and truecrypt2john didn't build for
me. Only rar2john and ssh2john worked right away. Not that good
overall. Though encfs2john and odf2john happened to run on Debian
Unstable. (And JimF fixed zip2john and provided dumps for zip and
pdf. Then Dhiru added support for sxc files and provided corresponding
dumps.)

After these 4 dumps were ready (and uploaded for the team) I started
to crack them with default john settings, 10 threads each. And 1h31
later happened to be the first to crack encfs-easy. These jobs were
run for ~15h and only odt-easy was additionally cracked but I wasn't
the first to do it.

Then I tried to keep track which challenges were cracked by us and by
others (according to CMIYC tweeter), and update our team on it.

= Contest: password hashes

After challenges were maxed out I switched to hashes. First of all I
tried to convert all hashes to their canonical form. This suprisingly
took much more time then it was anticipated.

Then I started to adapt my scripts for this contest, improve them,
deal with bogus lines in .pot files etc. and ended up by being in
charge of all the scripts for processing .pot files on our contest
server. That was quite nervous especially when totally switching to
improved scripts 8 hours till the end of the contest:-)

Working on scripts I had almost no time to actually crack anything.
Nevertherless I managed to run several jobs. My workflow was like
this:

- run some attack against all types of hashes with the following
   script:

----------------------------------------------------------------------
#! /bin/bash

john=path/to/john

if [ "$1" = "" ]; then
     echo "Usage: ./run.sh <attack-name> <john-option> ..."
     echo "Example: ./run.sh enwiki-ru --wordlist=enwiki.lst --rules"
     exit 1
fi

name="$1"
shift

for file in hashes-*; do
     basename=`basename "$file"`
     format=`echo "$basename" | cut -d . -f 2`
     $john --session="$name.$basename" --pot="$name.$basename.pot" --format="$format" "$@" "$file" &
done
----------------------------------------------------------------------

   and let it run in the background;

- monitor it sometimes with "top", "wc -l *.pot" and the following
   script:

----------------------------------------------------------------------
#! /bin/bash

john=path/to/john

killall -HUP john
for file in *.rec; do
     format=`basename "$file" .rec`
     printf "%50s " "$format"
     $john --status="$format"
done
----------------------------------------------------------------------

- when fast hashes finish and only 4-6 running johns remain start the
   next attack;

- when time to stop an old job comes use some "ps | grep | cut | xargs kill".

I ran john with default settings, with the list of dates generated by
my script as a wordlist and with wordlists generated from wikipedia,
with and without rules. All attacks were against all hashes hence the
large overlap in cracks. The stats:

             |                                     Attacks                                     |
format      | dates     default   dewiki-ru enwiki    enwiki-ru eswiki    eswiki-ru ruwiki-ru | sum  uniq val  points
------------+---------------------------------------------------------------------------------+----------------------
bf          |    0/23h*    0/14h*    0/08h*    0/14h*    1/31h*    0/16h*    0/08h*    0/08h* |    1    1 5000   5000
sunmd5      |    0/---     0/---     0/---     0/---     0/---     0/---     0/08h*    0/08h* |    0    0 3000      0
sha512crypt |    0/23h*    0/14h*    0/08h*    1/14h*    4/31h*    0/16h*    0/08h*    0/08h* |    5    4 2000   8000
mscash2     |    0/23h*    0/14h*    0/08h*    1/14h*    2/31h*    1/16h*    0/08h*    1/08h* |    5    2  600   1200
md5         |    0/23h*    4/14h*   20/08h*   44/14h*   51/31h*   31/16h*   23/08h*   15/08h* |  188   71  500  35500
dynamic_28  |    0/23h*   11/14h*   25/08h*   42/14h*   51/31h*   36/16h*   23/08h*   16/08h* |  204   76  500  38000
dynamic_21  |    1/03h     6/14h*   59/08h*   41/14h*   45/31h*   65/11h    65/08h*   57/08h* |  339   83   12    996
des         |   13/01h   108/14h*  693/08h*  596/14h* 1178/31h*  685/09h   683/08h*  635/08h* | 4591 1292    9  11628
phps        |    1/00h    19/14h*  111/08h*   51/08h   164/31h*   91/02h   117/08h*  106/08h* |  660  173    8   1384
salted-sha1 |    2/02h    25/14h*  170/08h*   97/14h*  230/31h*  156/07h   162/08h*  145/08h* |  987  246    8   1968
oracle11    |    2/01h    20/14h*  105/08h*   67/14h*  157/31h*  101/04h   110/08h*  101/08h* |  663  168    7   1176
mscash      |    0/00h     5/14h*   86/08h*   41/06h   124/31h*   65/01h    77/08h*   69/08h* |  467  130    7    910
nt          |    4/00h   722/14h*  474/00h   188/00h   630/00h   303/00h   456/00h   422/00h  | 3199 1132    2   2264
raw-md5u    |    0/00h    65/14h*   57/00h    20/00h    73/00h    34/00h    49/00h    45/00h  |  343  117    2    234
raw-md4     |    1/00h   172/14h*  118/00h    48/00h   154/00h    69/00h   113/00h   103/00h  |  778  277    1    277
raw-md5     |    1/00h   136/14h*  121/00h    59/00h   161/00h    73/00h   114/00h   109/00h  |  774  233    1    233
raw-sha1    |    1/00h   272/14h*  253/00h   115/00h   353/00h   167/00h   250/00h   228/00h  | 1639  523    1    523
mysql-sha1  |    1/00h   116/14h*  144/00h    56/00h   193/00h   100/00h   142/00h   126/00h  |  878  261    1    261
nsldap      |    1/00h   218/14h*  214/00h    85/00h   281/00h   128/00h   207/00h   175/00h  | 1309  440    1    440
------------+---------------------------------------------------------------------------------+----------------------
total       |                                                                                 |      5229      109994

The time spent on each attack is shown after the slash with the star
meaning that the attack didn't complete and was aborted.

= Experience

This contest gave me a great deal of valuable experience in password
cracking and some general sysadmin experience.

Multilayered ssh tunneling worked well with autossh. rsync over ssh
turned out to be quite nice.

OTOH ircii in gnome-terminal was quirky and hard to use leading me to
miss some messages and post garbage sometimes. My apologies for that
to the team.

= Wishlist for Korelogic

Here are several suggestions/wishes for Korelogic.

- Clarify whether there are any 8-bit passwords in the contest and how
   to submit them.

- Describe how passwords for des are checked -- are 8-bit and long
   variants accepted as valid or not.

- Provide feedback on which passwords were not accepted as valid.

And from the table above it seems that phps hashes are somewhat
overpriced compared to salted-sha1 and oracle11.

= Thanks

I'd like to thank Laboratory of Algebraic Geometry, HSE, and its head
Alexey Zykin personally for letting me use laboratory's servers in
this contest.

Thanks to Korelogic for good organization of the contest, to Solar
Designer for trusting me to manage server scripts and to all the team
for making the experience so fascinating!

-- 
Alexander Cherepanov

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.