Openwall GNU/*/Linux - a small security-enhanced Linux distro for servers
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Fri, 3 Jan 2014 03:00:38 +0100
From: Frank Dittrich <frank_dittrich@...mail.com>
To: john-dev@...ts.openwall.com
Subject: ./jtrts with linux-x86ö-64-native OMP build (was: OMP build failure)

On 01/03/2014 01:46 AM, magnum wrote:
> On 2014-01-03 01:37, Frank Dittrich wrote:
>> On 01/01/2014 07:39 PM, magnum wrote:
>>> Was the SAP issue fixed since then?
>>
>> Can't test it right now.
>> That was an OMP build, but OMP is broken for linux-x86-native and
>> linux-x86-64-native.
> 
> Ouch, that problem is fixed now and I just ran sapG through TS using OMP
> with no problems on well.

I checked out the old john and jtrts versions and verified the error was
still there. It was.

I also saw that the old version had failed self tests with OMP:
dynamic_14 (get_hash)
ecryptfs (get_hash)

And with --encoding=utf-8:
mscash2 (plaintext_length)

I must have missed that in August.

With latest bleeding-jumbo and jtrTestSuite and a linux-x86-64-native
OMP build (i5-4570 CPU) I get these failing self tests:

$ ./john --test=0 |LC_ALL=C grep -i fail
Will run 4 OpenMP threads
Testing: dynamic_38 [sha1($s.sha1($s.sha1($p))) (Wolt3BB) 128/128 AVX
480x4x2]... (4xOMP) FAILED (get_hash[0](13))
Testing: dynamic_107 [WHIRLPOOL(WHIRLPOOL($s).WHIRLPOOL($p)) 64/64
6144x1 OpenSSL]... (4xOMP) FAILED (get_hash[0](4))
Testing: dynamic_1027 [sha1(sha1(sha1($pass)))) (first 32 bytes) 128/128
AVX 480x4x2]... (4xOMP) FAILED (get_hash[0](13))
Testing: dynamic_1028 [sha1(sha1_raw($pass)) (first 32 bytes) 128/128
AVX 480x4x2]... (4xOMP) FAILED (get_hash[0](13))
Testing: dynamic_1501 [sha1($salt.sha1($pass) (Redmine) 128/128 AVX
480x4x2]... (4xOMP) FAILED (get_hash[0](13))
5 out of 354 tests have FAILED


$ ./john --list=build-info
Version: 1.8.0.2-bleeding-jumbo
Build: linux-x86-64-native OMP
Arch: 64-bit LE
$JOHN is ./
Format interface version: 11
Rec file version: REC4
Charset file version: CHR3
CHARSET_MIN: 1 (0x01)
CHARSET_MAX: 255 (0xff)
CHARSET_LENGTH: 24
Max. Markov mode level: 400
Max. Markov mode password length: 30
Compiler version: 4.8.1 20130909 [gcc-4_8-branch revision 202388]
gcc version: 4.8.1
OpenSSL library version: 01000105f
OpenSSL 1.0.1e 11 Feb 2013
NSS library version: 3.15.3.1
NSPR library version: 4.10.2
Kerberos version 5 support enabled


CPU: i5-4570


The test suite found just 4 failing selftests (dynamic_107 passed):

$ ./jtrts.pl
-------------------------------------------------------------------------------
- JtR-TestSuite (jtrts). Version 1.12.12, July 21, 2013.  By, Jim
Fougeron & others
- Testing:  John the Ripper password cracker, version
1.8.0.2-bleeding-jumbo_omp [linux-x86-64-native]
--------------------------------------------------------------------------------

John Jumbo build detected.
testing: john -test=0
Will run 4 OpenMP threads
Testing: dynamic_38 [sha1($s.sha1($s.sha1($p))) (Wolt3BB) 128/128 AVX
480x4x2]... (4xOMP) FAILED (get_hash[0](13))
Testing: dynamic_1027 [sha1(sha1(sha1($pass)))) (first 32 bytes) 128/128
AVX 480x4x2]... (4xOMP) FAILED (get_hash[0](13))
Testing: dynamic_1028 [sha1(sha1_raw($pass)) (first 32 bytes) 128/128
AVX 480x4x2]... (4xOMP) FAILED (get_hash[0](13))
Testing: dynamic_1501 [sha1($salt.sha1($pass) (Redmine) 128/128 AVX
480x4x2]... (4xOMP) FAILED (get_hash[0](13))
4 out of 354 tests have FAILED
testing: john -test=0 --encoding=utf8
Will run 4 OpenMP threads
testing: john -test=0 --encoding=koi8r
Will run 4 OpenMP threads

form=dynamic_0                    guesses: 1500 0:00:00:00 DONE  [PASSED]
.pot CHK:dynamic_0                guesses: 1500 0:00:00:00 DONE  [PASSED]
...
form=dynamic_37                   guesses: 1500 0:00:00:00 DONE  [PASSED]
.pot CHK:dynamic_37               guesses: 1500 0:00:00:00 DONE  [PASSED]
Use of uninitialized value $crack_xx[4] in substitution (s///) at
./jtrts.pl line 612.
Killed

At that point I had to kill the test suite - it was consuming more and
more memory...

I guess that
  $crack_xx[4] =~ s/100%/DONE/;
in jtrts.pl line 612 must be moved after the
    while (not defined $crack_xx[4]) { push (@...ck_xx, "unk"); }
a few lines further down...


I changed it like this:

diff --git a/jtrts.pl b/jtrts.pl
index b10a89b..9eed3a4 100755
--- a/jtrts.pl
+++ b/jtrts.pl
@@ -609,7 +609,7 @@ sub process {
                    }
                }
                # convert to legacy format
-               $crack_xx[4] =~ s/100%/DONE/;
+               if (defined $crack_xx[4]) { $crack_xx[4] =~ s/100%/DONE/; }
                while (not defined $crack_xx[1]) { push (@...ck_xx, "0"); }
                my $orig_crack_cnt = $crack_xx[1];
                ScreenOutSemi("\n");

and repeated the test


Now I got 5 failed self tests:

> ./jtrts.pl
-------------------------------------------------------------------------------
- JtR-TestSuite (jtrts). Version 1.12.12, July 21, 2013.  By, Jim
Fougeron & others
- Testing:  John the Ripper password cracker, version
1.8.0.2-bleeding-jumbo_omp [linux-x86-64-native]
--------------------------------------------------------------------------------

John Jumbo build detected.
testing: john -test=0
Will run 4 OpenMP threads
Testing: dynamic_38 [sha1($s.sha1($s.sha1($p))) (Wolt3BB) 128/128 AVX
480x4x2]... (4xOMP) FAILED (get_hash[0](13))
Testing: dynamic_87 [sha512(sha512($s).sha512($p)) 64/64 6144x1
sha2-OpenSSL]... (4xOMP) FAILED (get_hash[0](4))
Testing: dynamic_1027 [sha1(sha1(sha1($pass)))) (first 32 bytes) 128/128
AVX 480x4x2]... (4xOMP) FAILED (get_hash[0](13))
Testing: dynamic_1028 [sha1(sha1_raw($pass)) (first 32 bytes) 128/128
AVX 480x4x2]... (4xOMP) FAILED (get_hash[0](13))
Testing: dynamic_1501 [sha1($salt.sha1($pass) (Redmine) 128/128 AVX
480x4x2]... (4xOMP) FAILED (get_hash[0](13))
5 out of 354 tests have FAILED


So, the self test for dynamic_107 didn't fail here, but dynamic_87 did fail.
Could this depend on which thread finishes first/last?

When I repeatedly run
$ ./john --test=0 --format=dynamic,
dynamic_87 and dynamic_107 fail only sometimes, the other 4 formats fail
every time.
Out of 10 test runs, 87 failed 3 times, 107 failed 5 times.


form=dynamic_37                   guesses: 1500 0:00:00:00 DONE  [PASSED]
.pot CHK:dynamic_37               guesses: 1500 0:00:00:00 DONE  [PASSED]

form=dynamic_38                   guesses:    0 unk unk : Expected
count(s) (1500)  [!!!FAILED!!!]
Self test failed (get_hash[0](13))
sh: ./tst.pot: No such file or directory

So, the failing self test caused the .jtrts.pl hang before I changed
that line.

...

form=oracle11                     guesses:    0 0:00:00:13 DONE :
Expected count(s) (1500)  [!!!FAILED!!!]
.pot CHK:oracle11                 guesses:    0 unk unk  [PASSED]

This is the lotus85 FMT_NOT_EXACT problem.
Can we move Lotus85 after oracle11?
Or process the (plugin) formats twice? First those that donÄt produce
false positives, then the others?

And finally:

...
form=sapG-utf8                    guesses: 1500 0:00:00:00 DONE  [PASSED]
.pot CHK:sapG-utf8                guesses: 1500 0:00:00:00 DONE  [PASSED]

So, this problem is gone.


...
form=rar                          guesses:  130 0:00:00:26 DONE  [PASSED]
.pot CHK:rar                      guesses:  130 0:00:00:05 DONE  [PASSED]

Some tests had Errors. Performed 196 tests.2 errors
Time used was 704 seconds


Frank

Powered by blists - more mailing lists

Your e-mail address:

Powered by Openwall GNU/*/Linux - Powered by OpenVZ