john-users - Re: comparing two benchmarks

Openwall wordlists collection for password cracking (20+ languages)
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Mon, 7 Nov 2011 01:10:23 -0800 (PST)
From: firstname lastname <psykosonik_frequenz@...oo.com>
To: "john-users@...ts.openwall.com" <john-users@...ts.openwall.com>
Subject: Re: comparing two benchmarks

Thanks Alex for this Perl Script. Will go through the code and try to understand it. Also, will try out the script and see what it does! :)

NeonFlash



________________________________
From: Solar Designer <solar@...nwall.com>
To: john-users@...ts.openwall.com
Sent: Monday, November 7, 2011 7:54 AM
Subject: Re: [john-users] comparing two benchmarks

On Mon, Nov 07, 2011 at 05:50:24AM +0400, Solar Designer wrote:
> $ ./relbench.pl asis Os
> Geometric mean of 158:  0.914688 real, 0.915206 virtual
> Standard deviation:     0.966662 real, 0.962432 virtual
> 
> Thus, it appears that going from -O2 to -Os made things slower by 8.5%
> overall.  However, the standard deviation is large, meaning that things
> vary between individual benchmarks a lot.

While it is true that the standard deviation is relatively large on this
test, it is not as large as shown above - there was a bug in the script.
To correct it, right before these lines:

> $dr **= 0.5;
> $dv **= 0.5;

add:

$dr /= $n;
$dv /= $n;

Then we get:

$ ./relbench.pl asis Os
Geometric mean of 158:  0.914688 real, 0.915206 virtual
Standard deviation:     0.076904 real, 0.076567 virtual

For comparison, the standard deviation for equivalent benchmarks is
much lower:

$ ./relbench.pl asis asis-2
Geometric mean of 158:  1.000566 real, 1.002138 virtual
Standard deviation:     0.013746 real, 0.013566 virtual
$ ./relbench.pl Os Os-2
Geometric mean of 158:  1.002091 real, 1.001839 virtual
Standard deviation:     0.008966 real, 0.008853 virtual

Alexander