Openwall GNU/*/Linux - a small security-enhanced Linux distro for servers
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date: Thu, 24 Oct 2013 14:01:44 +0200
From: Frank Dittrich <frank_dittrich@...mail.com>
To: john-dev@...ts.openwall.com
Subject: benchmark-unify and relbench adjustments

magnum, Solar, all,

changed format names and changed --test output require some adjustments
to increase the number of benchmarks which can be compared (between
bleeding and earlier versions, e.g., unstable.

Output for dynamic formats has been changed. Commit 197f067 moved the
expression from the format name into the algorithm section, i.e., after
the '['.
The expressions for the new formats dynamic_1400 and 1401 contain
"[...]", so that the "Benchmarking: " output now contains nested
"[...[...]...]".

Some formats have been renamed.

Several of the issues mentioned above have been addressed by the
attached patches (intended for bleeding, not for unstable).
Because I'd like these patches to be discussed here, I didn't create
pull requests.

0001-benchmark-unify-sort-format-name-mappings.patch just sorted the
format name mappings (the __DATA__ section) alphabetically (ignoring
case), so that it is easier to see which mappings had/have to be added,
adjusted, or removed (the dynamic changes allowed a few of the dynamic
mappings to be removed).
So, this patch contains no functional changes.
(A similar patch for unstable might be useful if we need to adjust
mappings for unstable as well, so far I didn't check this.)


0002-benchmark-unify-adjustments-for-dynamic-formats.patch deals with
the changes required for dynamic formats.
Because  commit 197f067 moved the expression out of the format name and
into the algorithm, it was easy to even convert the md5_gen(n) from
older jumbo versions (I tested 1.7.6-jumbo-12) to dynamic_n.


0003-minor-relbench-improvements-and-adjustments.patch contains some
relbench adjustments, also described in the commit message.

I suppressed some "Could not parse" errors:
-ignore "Benchmarking: .*\.\.\. FAILED"
-older john versions had "All \d+ tests have passed self-tests", now it
is "All \d+ formats passed self-tests" or "\d+ out of \d+ tests have FAILED"
(If at least 1 test failed in one of the input files, I'll print that
information, before printing the "Only in File [1|2]: ..." information.)

Instead of printing the individual benchmarks (those that exist only in
1 file, or the ratio in case -v has been used) in "random" order,
they'll now be sorted.

The previous relbench version suggested running benchmark-unify even if
there's no way this would increase the number of matching benchmarks.
(E.g., the old john version just had a "Raw" benchmark for a particular
format, while the new version has "Many salts" and "Only one salt" for
the same format.)
This has now been fixed.


Format names that changed between unstable and bleeding have not yet
been addressed.

Main reasons for not dealing with it right now:
-there are still several formats in unused (which should really be in a
separate subdirectory, e.g., broken) because they fail self test

-openssl-enc still fails self test: (promiscuos valid ($openssl$))

-some formats might still need to be renamed.
I'd suggest renaming "Office ..." to "MS Office ..." and
"M$ Cache Hash (DCC)" to "MS Cache Hash (DCC)", as well as
"M$ Cache Hash 2 (DCC2)" to "M$ Cache Hash 2 (DCC2)".


Finally, I don't know yet how to handle this change:

In addition to the format name, the format label has been included into
the output, at least if format label and format name differ.

This change has 2 major consequences:
-while relbench used to recognize different implementations of the same
hash algorithm in the same input file (and picked the fastest one for
comparison), this no longer works
-almost all format names of older john --test runs would need to be
mapped to the new output (<label>, <name>); this is neither an elegant
solution, nor does it work for formats with more than one implementation.

I was experimenting with another solution (this patch just checks an
environment variable instead of a new benchmark-unify option, but I can
provide a patch which introduces a --drop-labels option):

-----------------------------------------------------------------

diff --git a/run/benchmark-unify b/run/benchmark-unify
index febd4ea..d266b38 100755
--- a/run/benchmark-unify
+++ b/run/benchmark-unify
@@ -56,6 +56,9 @@ sub parse
 		if (defined($renamed{$name})) {
 			$name = $renamed{$name};
 		}
+		if ($drop_labels == 1) {
+			$name =~ s/^[^\s]*, (.*)/$1/;
+		}
 		print "Benchmarking: $name $end\n";
 	}
 	else {
@@ -65,6 +68,15 @@ sub parse

 $_ = '';

+# For now, just an environment variable, I might add a
+# ./benchmark-unify option --drop-labels[=0|=1] later
+if(defined $ENV{'JOHN_BENCHMARK_UNIFY_DROP_LABELS'}) {
+	$drop_labels = 1;
+}
+else {
+	$drop_labels = 0;
+}
+
 while(<DATA>) {
 	chomp;
 	($old_format, $new_format) = /^(.*)	(.*)$/;

-----------------------------------------------------------------

If JOHN_BENCHMARK_UNIFY_DROP_LABELS is defined, e.g., by using
$  JOHN_BENCHMARK_UNIFY_DROP_LABELS=1 ./benchmark-unify ...
then everything that looks like a format label will be dropped.

This will increase the number of benchmarks in bleeding that can be
compared with benchmarks in unstable, but it has unwanted side effects:

-the user would need to run benchmark-unify even on the --test output of
the latest john version (so far I always aimed at converting older
output to the newest, and keeping the newest output unchanged)

-in some cases, the format name will become (completely) meaningless
and/or ambiguous:

Fortigate, FortiOS
FortiOS

MSCHAPv2, C/R
C/R
mschapv2-naive, MSCHAPv2 C/R
MSCHAPv2 C/R
(so, these 2 implementations aren't recognized as the same hash format,
and just "C/R" is completely meaningless)

OpenVMS, Purdy
Purdy

WoWSRP, Battlenet
Battlenet

Clipperz, SRP
SRP

Drupal7, $S$ (x16385)
$S$ (x16385)

IKE, PSK
PSK

MongoDB, system / network
system / network

Office, 2007/2010 (SHA-1) / 2013 (SHA-512), with AES
2007/2010 (SHA-1) / 2013 (SHA-512), with AES

PBKDF2-HMAC-SHA256, rounds=12000
rounds=12000

PBKDF2-HMAC-SHA512, GRUB2 / OS X 10.8
GRUB2 / OS X 10.8

PST, custom CRC-32
custom CRC-32

RAdmin, v2.x
v2.x

LastPass, sniffed sessions
sniffed sessions

STRIP, Password Manager
Password Manager

Raw-SHA, "SHA-0"
"SHA-0"

Raw-SHA1-ng, (pwlen <= 15)
(pwlen <= 15)

Mozilla, key3.db
key3.db


Some of this has been discussed in the past:
http://thread.gmane.org/gmane.comp.security.openwall.john.devel/9522/focus=9688
http://www.openwall.com/lists/john-dev/2013/08/19/

But so far, no solution has been found.

I think the best option is to (optionally) remove the format labels in
benchmark-unify, even if this requires the user to run benchmark-unify
on newest john --test output (which wasn't necessary for older john
versions), and adjust the few format names which get meaningless if we
do so.
IMHO, we should try to have the same format name output (possibly after
removing the format label) for different implementations of the same
algorithm.
That includes OpenCL and CUDA implementations.
If someone wants to compare the performance of CPU formats and CPU
formats separately, this can be done by running
./john --test --format=cpu
./john --test --format=dynamic
./john --test --format=gpu
./john --test --format=opencl
./john --test --format=cuda

We need to come up with a solution before we can release bleeding as the
next john-1.8 jumbo version.

Frank

>>From 7cf49991d9083e780c3c6a32dca706fa3c71fecb Mon Sep 17 00:00:00 2001
From: Frank Dittrich <frank_dittrich@...mail.com>
Date: Wed, 23 Oct 2013 17:19:44 +0200
Subject: [PATCH 1/3] benchmark-unify: sort format name mappings

---
 run/benchmark-unify | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/run/benchmark-unify b/run/benchmark-unify
index 751ba3c..a49716b 100755
--- a/run/benchmark-unify
+++ b/run/benchmark-unify
@@ -101,39 +101,39 @@ bf-opencl, OpenBSD Blowfish (x32)	bcrypt-opencl ("$2a$05", 32 iterations)
 DIGEST-MD5	DIGEST-MD5 C/R
 dynamic_20: Cisco PIX (MD5 salted)	dynamic_20: Cisco ASA (MD5 salted)
 dynamic_38: sha1($s.sha1($s.($p))) (Wolt3BB)	dynamic_38: sha1($s.sha1($s.sha1($p))) (Wolt3BB)
-EPiServer SID Hashes	EPiServer SID salted SHA-1
 Eggdrop	Eggdrop Blowfish
+EPiServer SID Hashes	EPiServer SID salted SHA-1
 FreeBSD MD5	crypt-MD5
 generic crypt(3)	generic crypt(3) DES
 hmailserver	hMailServer salted SHA-256
 HTTP Digest access authentication	HTTP Digest access authentication MD5
 IPB2 MD5	Invision Power Board 2.x salted MD5
-KRB5 aes256-cts-hmac-sha1-96	Kerberos 5 db etype 18 aes256-cts-hmac-sha1-96
-KRB5 arcfour-hmac	Kerberos 5 db etype 23 rc4-hmac
 Kerberos v4 TGT	Kerberos v4 TGT DES
 Kerberos v5 TGT	Kerberos v5 TGT 3DES
+KRB5 aes256-cts-hmac-sha1-96	Kerberos 5 db etype 18 aes256-cts-hmac-sha1-96
+KRB5 arcfour-hmac	Kerberos 5 db etype 23 rc4-hmac
 Lotus5	Lotus Notes/Domino 5
-M$ Cache Hash	M$ Cache Hash MD4
 M$ Cache Hash 2 (DCC2)	M$ Cache Hash 2 (DCC2) PBKDF2-HMAC-SHA-1
+M$ Cache Hash	M$ Cache Hash MD4
+MediaWiki -- md5($s.'-'.md5($p))	MediaWiki md5($s.'-'.md5($p))
+More Secure Internet Password	Lotus Notes/Domino 6 More Secure Internet Password
+Mozilla SHA-1 3DES	Mozilla (key3.db) SHA-1 3DES
 MS Kerberos 5 AS-REQ Pre-Auth	Kerberos 5 AS-REQ Pre-Auth etype 23 md4, rc4-hmac-md5
 MS Kerberos 5 AS-REQ Pre-Auth MD4 MD5 RC4	Kerberos 5 AS-REQ Pre-Auth etype 23 md4, rc4-hmac-md5
-MS-SQL	MS SQL SHA-1
 MS-SQL05	MS SQL 2005 SHA-1
-MYSQL	MySQL
+MS-SQL	MS SQL SHA-1
 MYSQL_fast	MySQL
-MediaWiki -- md5($s.'-'.md5($p))	MediaWiki md5($s.'-'.md5($p))
-More Secure Internet Password	Lotus Notes/Domino 6 More Secure Internet Password
-Mozilla SHA-1 3DES	Mozilla (key3.db) SHA-1 3DES
-NT v2	NT MD4
+MYSQL	MySQL
 Netscape LDAP SHA	Netscape LDAP SHA-1
+NT v2	NT MD4
 ODF SHA-1 Blowfish	ODF SHA-1 Blowfish / SHA-256 AES
 Office 2007/2010 SHA-1/AES	Office 2007/2010 (SHA-1) / 2013 (SHA-512), with AES
-Oracle	Oracle 10 DES
 Oracle 11g	Oracle 11g SHA-1
-pdf	PDF MD5 SHA-2 RC4 / AES
+Oracle	Oracle 10 DES
 PDF MD5 RC4	PDF MD5 SHA-2 RC4 / AES
-PHPS -- md5(md5($pass).$salt)	PHPS md5(md5($pass).$salt)
+pdf	PDF MD5 SHA-2 RC4 / AES
 PHPass MD5	phpass MD5 ($P$9)
+PHPS -- md5(md5($pass).$salt)	PHPS md5(md5($pass).$salt)
 PIX MD5	Cisco PIX MD5
 pkzip	PKZIP
 rar	RAR3 SHA-1 AES (4 characters)
-- 
1.8.1.4



>>From 8abf94707513e78a94ef42286c8eaaf9ce76e1ca Mon Sep 17 00:00:00 2001
From: Frank Dittrich <frank_dittrich@...mail.com>
Date: Wed, 23 Oct 2013 22:51:54 +0200
Subject: [PATCH 2/3] benchmark-unify: adjustments for dynamic formats

Especially commit 197f067 required changes, because
	dynamic_1350 md5(md5($s.$p):$s); [128/128 SSE2 intrinsics 10x4x3]
etc. had to be converted to
	dynamic_1350 [md5(md5($s.$p):$s); 128/128 SSE2 intrinsics 10x4x3]
etc.
---
 run/benchmark-unify | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/run/benchmark-unify b/run/benchmark-unify
index a49716b..febd4ea 100755
--- a/run/benchmark-unify
+++ b/run/benchmark-unify
@@ -46,13 +46,13 @@
 sub parse
 {
 	chomp;
-	($name,$end) = /^Benchmarking: (.*[^ ]) +(\[.*\].*)$/;
+	# john-1.7.6-jumbo-12 still used md5_gen(n) instead of dynamic_n
+	s/^Benchmarking: +md5_gen\(([0-9]+)\):? +/Benchmarking: dynamic_$1: /;
+	s/^(Benchmarking: dynamic_[0-9]+):? ([^[]+)\[/$1 [$2/;
+	($name,$end) = /^Benchmarking: ([^\[]*[^ ]) +(\[.*\].*)$/;
 	if (defined($name) && defined($end)) {
-		$name =~ s/(dynamic_[0-9]+) /$1: /;
 		$name =~ s/\s+/ /g;
-		$name =~ s/\[/(/;
-		$name =~ s/\]/)/;
-
+		$end =~ s/\s+/ /g;
 		if (defined($renamed{$name})) {
 			$name = $renamed{$name};
 		}
@@ -99,8 +99,6 @@ while (<>) {
 __DATA__
 bf-opencl, OpenBSD Blowfish (x32)	bcrypt-opencl ("$2a$05", 32 iterations)
 DIGEST-MD5	DIGEST-MD5 C/R
-dynamic_20: Cisco PIX (MD5 salted)	dynamic_20: Cisco ASA (MD5 salted)
-dynamic_38: sha1($s.sha1($s.($p))) (Wolt3BB)	dynamic_38: sha1($s.sha1($s.sha1($p))) (Wolt3BB)
 Eggdrop	Eggdrop Blowfish
 EPiServer SID Hashes	EPiServer SID salted SHA-1
 FreeBSD MD5	crypt-MD5
-- 
1.8.1.4



>>From 911be062bb722a860ca760010907e5a29f7be38f Mon Sep 17 00:00:00 2001
From: Frank Dittrich <frank_dittrich@...mail.com>
Date: Wed, 23 Oct 2013 23:04:48 +0200
Subject: [PATCH 3/3] minor relbench improvements and adjustments

Suggest runnig benchmark-unify only if there really is a chance
that this might increase the number for benchmarks that can be compared
(e.g., not if the new john version just has "Many salts" and "Only one salt"
for a particular format instead of just "Raw" in the old version).

Suppress some of the "Could not parse" errors which had been caused by
minor changes in john's --test output.

Sort the list of formats that exist only in one file (and, for -v, the ratio).

Depending on the definition of a dynamic format, the --test output
might contain nested [ [ ... ] ], e.g., for dynamic_1400 and dynamic_1401:
Expression=sha1(unicode($p)) [Microsoft CREDHIST]
Expression=md5($u.\nskyper\n.$p) [Skype MD5]
So, parsing the "Benchmarking: " output had to be adjusted.
---
 run/relbench | 101 ++++++++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 89 insertions(+), 12 deletions(-)

diff --git a/run/relbench b/run/relbench
index ecd5d7d..02b48ac 100755
--- a/run/relbench
+++ b/run/relbench
@@ -1,8 +1,9 @@
 #!/usr/bin/perl -w
 #
-# John the Ripper benchmark output comparison tool, revision 4.1
+# John the Ripper benchmark output comparison tool, revision 4.2
 # Copyright (c) 2011 Solar Designer
-# Enhancements for revision 4.1, copyright (c) 2012 Frank Dittrich
+# Enhancements for revisions 4.1 and 4.2,
+# copyright (c) 2012-2013 Frank Dittrich
 # Redistribution and use in source and binary forms, with or without
 # modification, are permitted.
 # There's ABSOLUTELY NO WARRANTY, express or implied.
@@ -28,10 +29,6 @@
 # values will tend to deviate from 0.0 and 1.0, respectively.
 #
 
-$warned = 0;
-$onlyin1 = 0;
-$onlyin2 = 0;
-
 sub parse
 {
 	chomp;
@@ -43,7 +40,16 @@ sub parse
 		undef $kind; undef $real; undef $virtual;
 		return;
 	}
-	return if (/All \d+ tests have passed self-tests/);
+	if (/^(All|\d+ out of) \d+ (tests|formats)( have)? (passed self-tests|FAILED)/) {
+		($total) = /^All (\d+) (tests|formats)( have)? passed self-tests/;
+		if (!defined($total)) {
+			($failed, $total) =
+			    /^(\d+) out of (\d+) (tests|formats)( have)? FAILED/;
+		}
+		return;
+	}
+	return if /Benchmarking: .*\.\.\. FAILED/;
+
 	my $ok = 0;
 	if (defined($name)) {
 		($kind, $real, $reals, $virtual, $virtuals) =
@@ -65,12 +71,34 @@ sub parse
 			return;
 		}
 	} else {
-		($name) = /^\t?Benchmarking: (.+) \[.*\].* (DONE|Warning:.*)$/;
+		($name) = /^\t?Benchmarking: ([^\[]+) \[.*\].* (DONE|Warning:.*)$/;
 		$ok = defined($name);
 	}
 	print STDERR "Could not parse: $_\n" if (!$ok);
 }
 
+sub sort_benchmarks
+{
+	my $_;
+#	$a cmp $b;
+	$_ = "$a";
+	($name_a, $number_a, $benchmark_a) = /^(.*[^\d])(\d+):(.*)$/;
+	if(!defined($number_a)) {
+		$number_a = -1;
+		($name_a, $benchmark_a) = /^(.*):(.*)$/;
+	}
+	$_ = "$b";
+	($name_b, $number_b, $benchmark_b) = /^(.*[^\d])(\d+):(.*)$/;
+	if(!defined($number_b)) {
+		$number_b = -1;
+		($name_b, $benchmark_b) = /^(.*):(.*)$/;
+	}
+	if ($name_a ne $name_b) { return $name_a cmp $name_b }
+	elsif ($number_a != $number_b) { return $number_a <=> $number_b }
+	elsif ($benchmark_b eq "Short") { return 1 }
+	else { return $benchmark_a cmp $benchmark_b }
+}
+
 die "Usage: $0 [-v] BENCHMARK-FILE-1 BENCHMARK-FILE-2\n" if ($#ARGV != 1 && ($#ARGV != 2 || $ARGV[0] ne '-v'));
 
 if ($#ARGV != 1) {
@@ -83,10 +111,29 @@ if ($#ARGV != 1) {
 	$verbose = 0;
 }
 
+$warned = 0;
+$onlyin1 = 0;
+$onlyin2 = 0;
+$name1 = "";
+$name2 = "";
+$namesonlyin1 = 0;
+$namesonlyin2 = 0;
+
+$failed = 0;
+$total = 0;
 $_ = '';
 parse();
 while (<B1>) {
 	parse();
+	if (defined($name) && $name ne $name1) {
+		$name1 = $name;
+		if(defined($n1{$name1})) {
+			$n1{$name1} += 1;
+		}
+		else {
+			$n1{$name1} = 1;
+		}
+	}
 	next unless (defined($id));
 	if(defined($b1r{$id})) {
 		print STDERR "More than one benchmark for $id in file 1\n";
@@ -99,11 +146,25 @@ while (<B1>) {
 		$b1v{$id} = $virtual;
 }	}
 close(B1);
+if($failed != 0) {
+	print "File 1: $failed out of $total tests have FAILED\n"
+}
 
+$failed = 0;
+$total = 0;
 $_ = '';
 parse();
 while (<B2>) {
 	parse();
+	if (defined($name) && $name ne $name2) {
+		$name2 = $name;
+		if(defined($n2{$name2})) {
+			$n2{$name2} += 1;
+		}
+		else {
+			$n2{$name2} = 1;
+		}
+	}
 	next unless (defined($id));
 	if(defined($b2r{$id})) {
 		print STDERR "More than one benchmark for $id in file 2\n";
@@ -117,8 +178,24 @@ while (<B2>) {
 	}
 }
 close(B2);
+if($failed != 0) {
+	print "File 2: $failed out of $total tests have FAILED\n"
+}
+
+foreach $name (keys %n1) {
+	if (!defined($n2{$name})) {
+		$namesonlyin1 += 1;
+		next;
+	}
+}
+foreach $name (keys %n2) {
+	if (!defined($n1{$name})) {
+		$namesonlyin1 += 1;
+		next;
+	}
+}
 
-foreach $id (keys %b1r) {
+foreach $id (sort sort_benchmarks keys %b1r) {
 	if (!defined($b2r{$id})) {
 		print "Only in file 1: $id\n";
 		$onlyin1 += 1;
@@ -129,14 +206,14 @@ foreach $id (keys %b1r) {
 $minr = $maxr = $minv = $maxv = -1.0;
 $mr = $mv = 1.0;
 $n = 0;
-foreach $id (keys %b2r) {
+foreach $id (sort sort_benchmarks keys %b2r) {
 	if (!defined($b1r{$id})) {
 		print "Only in file 2: $id\n";
 		$onlyin2 += 1;
 		next;
 	}
 }
-foreach $id (keys %b2r) {
+foreach $id (sort sort_benchmarks keys %b2r) {
 	if (!defined($b1r{$id})) {
 		next;
 	}
@@ -155,7 +232,7 @@ foreach $id (keys %b2r) {
 		printf "Ratio:\t%.5f real, %.5f virtual\t$id\n", $kr, $kv;
 	}
 }
-if ($onlyin1 != 0 && $onlyin2 != 0) {
+if ($onlyin1 != 0 && $onlyin2 != 0 && $namesonlyin1 != 0 && $namesonlyin2 != 0) {
 	print STDERR "Converting the two benchmark files using benchmark-unify might\n";
 	print STDERR "increase the number of benchmarks which can be compared\n";
 }
-- 
1.8.1.4



Powered by blists - more mailing lists

Your e-mail address:

Powered by Openwall GNU/*/Linux - Powered by OpenVZ