Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Mon, 24 Oct 2016 13:53:01 -0700
From: Kees Cook <keescook@...omium.org>
To: David Windsor <dwindsor@...il.com>
Cc: "kernel-hardening@...ts.openwall.com" <kernel-hardening@...ts.openwall.com>, 
	"Reshetova, Elena" <elena.reshetova@...el.com>, Hans Liljestrand <ishkamiel@...il.com>
Subject: Re: HARDENED_ATOMIC benchmarks

On Sat, Oct 22, 2016 at 3:13 PM, David Windsor <dwindsor@...il.com> wrote:
> Hi,
>
> The following are the results of benchmarking HARDENED_ATOMIC.  The
> benchmarks performed were dbench and a timed Linux kernel compile using the
> Phoronix test suite [1] on a Linux VirtualBox guest.
>
> dbench was chosen specifically to gauge the performance penalty involved in
> heavy usage of struct file->f_count, as this is one of the hottest users of
> atomic_t/atomic_long_t.
>
> A small performance degradation was noticeable with HARDENED_ATOMIC enabled.
> The numbers are such that I'm unsure if this was due to the feature itself
> or to random line noise.  What follows is a summary of the benchmark
> results, then the results themselves.
>
> [1] http://www.phoronix-test-suite.com/
>
> HARDENED_ATOMIC Benchmarking Summary
> ===================================
>
> dbench
> ======
> CONFIG_HARDENED_ATOMIC not set: 109.79 MB/s
> CONFIG_HARDENED_ATOMIC set: 106.75 MB/s
> 2.8% slowdown
>
> Timed Linux kernel compile
> =====================
> CONFIG_HARDENED_ATOMIC not set: 504.12 seconds
> CONFIG_HARDENED_ATOMIC set: 504.81 seconds
> 0.01% slowdown
>
>
>
>
> HARDENED_ATOMIC Benchmark Results
> ===============================
>
> dbench
> ======
> HARDENED_ATOMIC disabled:
> dbenchDbench 4.0:
>     pts/dbench-1.0.0 [Client Count: 12]
>     Test 1 of 1
>     Estimated Trial Run Count:    3
>     Estimated Time To Completion: 1 Hour, 25 Minutes
>         Started Run 1 @ 08:48:01
>         Started Run 2 @ 09:00:04
>         Started Run 3 @ 09:12:07  [Std. Dev: 1.39%]
>
>     Test Results:
>         109.613
>         108.351
>         111.393
>
>     Average: 109.79 MB/s
>
>
> HARDENED_ATOMIC enabled:
> Dbench 4.0:
>     pts/dbench-1.0.0 [Client Count: 12]
>     Test 1 of 1
>     Estimated Trial Run Count:    3
>     Estimated Time To Completion: 1 Hour, 28 Minutes
>         Started Run 1 @ 06:48:33
>         Started Run 2 @ 07:00:37
>         Started Run 3 @ 07:12:40  [Std. Dev: 12.66%]
>         Started Run 4 @ 07:24:43  [Std. Dev: 10.94%]
>         Started Run 5 @ 07:36:46  [Std. Dev: 9.98%]
>         Started Run 6 @ 07:48:50  [Std. Dev: 9.30%]
>
>     Test Results:
>         87.0504
>         106.451
>         111.377
>         110.239
>         112.226
>         113.152
>
>     Average: 106.75 MB/s

Variation here is as large as the measured difference, so yeah, this
looks like it's mostly in the noise.

> Timed Linux Kernel Compile
> ======================
> HARDENED_ATOMIC disabled:
> Timed Linux Kernel Compilation 4.3:
>     pts/build-linux-kernel-1.6.0
>     Test 4 of 4
>     Estimated Trial Run Count:    3
>     Estimated Time To Completion: 55 Minutes
>     Estimated Trial Run Count:    3
>     Estimated Time To Completion: 55 Minutes
>         Running Pre-Test Script @ 02:39:33
>         Started Run 1 @ 02:40:06
>         Running Interim Test Script @ 02:48:39
>         Started Run 2 @ 02:48:45
>         Running Interim Test Script @ 02:57:07
>         Started Run 3 @ 02:57:14  [Std. Dev: 0.84%]
>         Running Post-Test Script @ 03:05:35
>
>     Test Results:
>         509.00912308693
>         501.88456201553
>         501.46281981468
>
>     Average: 504.12 Seconds
>
>
> HARDENED_ATOMIC enabled:
> Timed Linux Kernel Compilation 4.3:
>     pts/build-linux-kernel-1.6.0
>     Test 4 of 4
>     Estimated Trial Run Count:    3
>     Estimated Time To Completion: 45 Minutes
>         Running Pre-Test Script @ 05:26:55
>         Started Run 1 @ 05:27:30
>         Running Interim Test Script @ 05:36:01
>         Started Run 2 @ 05:36:08
>         Running Interim Test Script @ 05:44:32
>         Started Run 3 @ 05:44:38  [Std. Dev: 0.36%]
>         Running Post-Test Script @ 05:53:02
>
>     Test Results:
>         506.84527802467
>         504.29637002945
>         503.30119299889
>
>     Average: 504.81 Seconds

These have much smaller std deviations.

Thanks for the testing! The cover letter can likely get updated to
include the overview: no meaningfully measurable performance
difference. :)

-Kees

-- 
Kees Cook
Nexus Security

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.