Date: Tue, 8 Nov 2011 02:00:05 +0400 From: Solar Designer <solar@...nwall.com> To: john-users@...ts.openwall.com Subject: Re: comparing two benchmarks On Mon, Nov 07, 2011 at 01:10:23AM -0800, firstname lastname wrote: > Thanks Alex for this Perl Script. Will go through the code and try to understand it. Also, will try out the script and see what it does! :) Attached is second revision of the script. The idea to calculate and report the usual standard deviation wasn't a very good one for two reasons. First, the value being reported was still not exactly standard deviation under the usual definition (even after my fix) because it was relative to the geometric mean rather than to the expected value (which is the same as the arithmetic mean when the weights are equal). Second, the usual standard deviation isn't a good measure to consider in this context (even if calculated properly). The script will now report the geometric standard deviation instead. It will also report several other measures. Sample output: $ ./relbench.pl asis asis Number of benchmarks: 158 Minimum: 1.00000 real, 1.00000 virtual Maximum: 1.00000 real, 1.00000 virtual Median: 1.00000 real, 1.00000 virtual Median absolute deviation: 0.00000 real, 0.00000 virtual Geometric mean: 1.00000 real, 1.00000 virtual Geometric standard deviation: 1.00000 real, 1.00000 virtual $ ./relbench.pl asis Os Number of benchmarks: 158 Minimum: 0.52807 real, 0.52807 virtual Maximum: 1.12544 real, 1.12007 virtual Median: 0.92055 real, 0.91886 virtual Median absolute deviation: 0.05072 real, 0.05383 virtual Geometric mean: 0.91469 real, 0.91521 virtual Geometric standard deviation: 1.09400 real, 1.09357 virtual The relevant concepts are explained here: http://en.wikipedia.org/wiki/Median http://en.wikipedia.org/wiki/Median_absolute_deviation http://en.wikipedia.org/wiki/Geometric_mean http://en.wikipedia.org/wiki/Geometric_standard_deviation Here's why we should in fact use the geometric mean (and not the arithmetic mean or some other mean): http://en.wikipedia.org/wiki/Geometric_mean#Properties "... This makes the geometric mean the only correct mean when averaging normalized results, that is results that are presented as ratios to reference values. This is the case when presenting computer performance with respect to a reference computer ..." "... using the arithmetic or harmonic mean would change the ranking of the results depending on what is used as a reference." "In all cases, the ranking given by the geometric mean stays the same as the one obtained with unnormalized values." We have to use normalized values because the hashes and ciphers being benchmarked have very different performance. As to the median and the median absolute deviation, these are more robust when there are outliers: http://en.wikipedia.org/wiki/Robust_statistics#Examples_of_robust_and_non-robust_statistics http://en.wikipedia.org/wiki/Outlier I recommend that we primarily use the geometric mean, but also look at the median as a sanity check. When they're very different from each other (not the case in sample output above), this indicates that there's at least one outlier that we may want to exclude from the script's input (and consider it separately). Here's an example: $ ./relbench.pl asis hacked Number of benchmarks: 158 Minimum: 1.00000 real, 1.00000 virtual Maximum: 1000.00000 real, 1.00000 virtual Median: 1.00000 real, 1.00000 virtual Median absolute deviation: 0.00000 real, 0.00000 virtual Geometric mean: 1.04469 real, 1.00000 virtual Geometric standard deviation: 1.72946 real, 1.00000 virtual In this case, I edited the output of one of the 158 benchmarks making it appear 1000 times faster (in terms of "real c/s" only). The geometric mean shows this as a 4.5% overall speedup and it also shows relatively high geometric standard deviation. The medians simply ignore it (as an outlier). Alexander View attachment "relbench.pl" of type "text/plain" (3181 bytes)
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.