john-users - NT/unicode issue in current version

Openwall GNU/*/Linux 3.0 - a small security-enhanced Linux distro for servers
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date: Tue, 28 Sep 2010 01:00:14 +0200
From: "Magnum, P.I." <rawsmooth@...dband.net>
To: john-users@...ts.openwall.com
Subject: NT/unicode issue in current version

This really puzzles me:

The ISO-8859-1 (and ASCII) character code for '%' is 0x25 while the 
UTF-16LE character code is U+0025

The ISO-8859-1 character code for 'ü' (german u with diaeresis) is 0xFC 
while the UTF-16LE character code is U+00FC

I would think this means both should have the same chance of being 
cracked with bog standard John+jumbo when used in an NT hash. But the 
latter is not.

This could be a terminal encoding issue, but it is not. I use iconv back 
and forth, but it's edited out of the below for readability. I 
doublecheck what I'm doing using hexdump. Try it out yourself, 
preferably on an ISO-8859-1 terminal. The hashes are verified as what is 
being produced by Windows itself. I use john-1.7.6 and jumbo-7.

$ cat test.sam
ü:1000:aad3b435b51404eeaad3b435b51404ee:8bd6e4fb88e01009818749c5443ea712:
%:1001:aad3b435b51404eeaad3b435b51404ee:930ffb411991d600b1e691eea993deb9:

$ ./john test.sam -si -fo:NT
Loaded 2 password hashes with no different salts (NT MD4 [128/128 X2 
SSE2-16])
%                (%)
guesses: 1  time: 0:00:00:00 100.00% (ETA: Tue Sep 28 00:08:16 2010) 
c/s: 2658  trying: ü1923 - ü1900

$ echo ü | ./john test.sam -stdin -fo:NT
Loaded 1 password hash (NT MD4 [128/128 X2 SSE2-16])
guesses: 0  time: 0:00:00:00  c/s: 100  trying: ü

I even tried DumbForce to no avail. Why is it unable to crack the ü? Is 
it a bug? Is it the unintended result of some black magic ninja 
optimization somewhere? Or is it just me?

regards
magnum