Date: Mon, 7 Nov 2011 05:50:24 +0400 From: Solar Designer <solar@...nwall.com> To: john-users@...ts.openwall.com Subject: comparing two benchmarks Hi, Attached is a Perl script I wrote to compare pairs of John the Ripper benchmarks, such as to see the overall effect of changes that are not limited to individual hash or cipher types - e.g., different compilers, optimization options, computers, or versions of John. With so many "formats" supported in -jumbo these days, it was difficult to see if such changes had an overall positive or overall negative effect. Of course, for typical uses of John you would care about individual benchmarks, but for some uses an overall comparison is needed - e.g. when creating binary packages of John for others to use or when using John as a benchmark for C compilers or CPUs rather than as a password cracker. ;-) For example, it was non-obvious to me whether gcc 4.6.2 produced faster x86-64 code as measured on Core 2 when compiling with -O2 or -Os. Some of John's benchmarks were faster with -O2, some with -Os. In 1.7.8-jumbo-7, there are as many as 158 individual benchmark outputs. Here's what I did to find out which of these options was faster overall: 1. Compiled John the Ripper 1.7.8-jumbo-7 as-is using the linux-x86-64 make target (it has -O2 in CFLAGS, among other things). 2. Ran "../run/john --test > asis". 3. Edited the Makefile replacing -O2 with -Os on the CFLAGS line, ran "make clean linux-x86-64". 4. Ran "../run/john --test > Os". 5. Ran the comparison script on the two files: $ ./relbench.pl asis Os Geometric mean of 158: 0.914688 real, 0.915206 virtual Standard deviation: 0.966662 real, 0.962432 virtual Thus, it appears that going from -O2 to -Os made things slower by 8.5% overall. However, the standard deviation is large, meaning that things vary between individual benchmarks a lot. 6. To double-check these results, I ran "../run/john --test > Os-2", reverted the change to Makefile, did "make clean linux-x86-64" again, and ran "../run/john --test > asis-2". 7. Comparing these additional files, I get: $ ./relbench.pl asis asis-2 Geometric mean of 158: 1.000566 real, 1.002138 virtual Standard deviation: 0.172785 real, 0.170517 virtual $ ./relbench.pl Os Os-2 Geometric mean of 158: 1.002091 real, 1.001839 virtual Standard deviation: 0.112707 real, 0.111287 virtual Notice consistent overall performance and much lower standard deviation when comparing what are supposed to be equivalent benchmarks. Now let's compare -O2 vs. -Os again, but using the extra files this time: $ ./relbench.pl asis-2 Os-2 Geometric mean of 158: 0.916083 real, 0.914933 virtual Standard deviation: 0.931147 real, 0.934134 virtual $ ./relbench.pl asis Os-2 Geometric mean of 158: 0.916601 real, 0.916890 virtual Standard deviation: 0.962342 real, 0.960655 virtual $ ./relbench.pl asis-2 Os Geometric mean of 158: 0.914171 real, 0.913253 virtual Standard deviation: 0.938063 real, 0.939076 virtual These are similar to what we saw originally, confirming that result: yes, we have an 8.5% overall slowdown for -Os. A drawback of the current approach is that all benchmark outputs are assigned the same weight, even though some are related to each other (e.g., many hashes are MD5-based) and the hashes and ciphers differ in popularity (so some are more important than others). But any other approach to this would be non-perfect as well. Alexander View attachment "relbench.pl" of type "text/plain" (1969 bytes)
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.