Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Wed, 24 Feb 2021 10:54:17 -0600
From: ebiederm@...ssion.com (Eric W. Biederman)
To: kernel test robot <oliver.sang@...el.com>
Cc: Alexey Gladkov <gladkov.alexey@...il.com>,  0day robot <lkp@...el.com>,  LKML <linux-kernel@...r.kernel.org>,  lkp@...ts.01.org,  ying.huang@...el.com,  feng.tang@...el.com,  zhengjun.xing@...el.com,  io-uring@...r.kernel.org,  Kernel Hardening <kernel-hardening@...ts.openwall.com>,  Linux Containers <containers@...ts.linux-foundation.org>,  linux-mm@...ck.org,  Alexey Gladkov <legion@...nel.org>,  Andrew Morton <akpm@...ux-foundation.org>,  Christian Brauner <christian.brauner@...ntu.com>,  Jann Horn <jannh@...gle.com>,  Jens Axboe <axboe@...nel.dk>,  Kees Cook <keescook@...omium.org>,  Linus Torvalds <torvalds@...ux-foundation.org>,  Oleg Nesterov <oleg@...hat.com>
Subject: Re: d28296d248:  stress-ng.sigsegv.ops_per_sec -82.7% regression

kernel test robot <oliver.sang@...el.com> writes:

> Greeting,
>
> FYI, we noticed a -82.7% regression of stress-ng.sigsegv.ops_per_sec due to commit:
>
>
> commit: d28296d2484fa11e94dff65e93eb25802a443d47 ("[PATCH v7 5/7] Reimplement RLIMIT_SIGPENDING on top of ucounts")
> url: https://github.com/0day-ci/linux/commits/Alexey-Gladkov/Count-rlimits-in-each-user-namespace/20210222-175836
> base: https://git.kernel.org/cgit/linux/kernel/git/shuah/linux-kselftest.git next
>
> in testcase: stress-ng
> on test machine: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
> with following parameters:
>
> 	nr_threads: 100%
> 	disk: 1HDD
> 	testtime: 60s
> 	class: interrupt
> 	test: sigsegv
> 	cpufreq_governor: performance
> 	ucode: 0x42e
>
>
> In addition to that, the commit also has significant impact on the
> following tests:

Thank you.  Now we have a sense of where we need to test the performance
of these changes carefully.

Eric


> +------------------+-----------------------------------------------------------------------+
> | testcase: change | stress-ng: stress-ng.sigq.ops_per_sec -56.1% regression               |
> | test machine     | 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory |
> | test parameters  | class=interrupt                                                       |
> |                  | cpufreq_governor=performance                                          |
> |                  | disk=1HDD                                                             |
> |                  | nr_threads=100%                                                       |
> |                  | test=sigq                                                             |
> |                  | testtime=60s                                                          |
> |                  | ucode=0x42e                                                           |
> +------------------+-----------------------------------------------------------------------+
>
>
> If you fix the issue, kindly add following tag
> Reported-by: kernel test robot <oliver.sang@...el.com>
>
>
> Details are as below:
> -------------------------------------------------------------------------------------------------->
>
>
> To reproduce:
>
>         git clone https://github.com/intel/lkp-tests.git
>         cd lkp-tests
>         bin/lkp install                job.yaml  # job file is attached in this email
>         bin/lkp split-job --compatible job.yaml
>         bin/lkp run                    compatible-job.yaml
>
> =========================================================================================
> class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
>   interrupt/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-ivb-2ep1/sigsegv/stress-ng/60s/0x42e
>
> commit: 
>   4660d663b4 ("Reimplement RLIMIT_MSGQUEUE on top of ucounts")
>   d28296d248 ("Reimplement RLIMIT_SIGPENDING on top of ucounts")
>
> 4660d663b4207ce6 d28296d2484fa11e94dff65e93e 
> ---------------- --------------------------- 
>        fail:runs  %reproduction    fail:runs
>            |             |             |    
>          14:6         -217%           1:6     perf-profile.children.cycles-pp.error_entry
>          12:6         -179%           1:6     perf-profile.self.cycles-pp.error_entry
>          %stddev     %change         %stddev
>              \          |                \  
>  4.766e+08           -82.7%   82358308        stress-ng.sigsegv.ops
>    7942807           -82.7%    1372639        stress-ng.sigsegv.ops_per_sec
>      29408           -77.5%       6621 ± 61%  stress-ng.time.file_system_inputs
>       1566           +69.4%       2653        stress-ng.time.system_time
>       1274           -84.8%     193.22 ±  8%  stress-ng.time.user_time
>      12458 ±  5%     +37.2%      17097 ±  5%  numa-meminfo.node1.Active(anon)
>      51.41           +66.5%      85.59        iostat.cpu.system
>      41.17           -84.2%       6.50 ±  7%  iostat.cpu.user
>       3040 ±  4%     +37.9%       4193 ±  4%  numa-vmstat.node1.nr_active_anon
>       3040 ±  4%     +37.9%       4193 ±  4%  numa-vmstat.node1.nr_zone_active_anon
>      50.83           +67.2%      85.00        vmstat.cpu.sy
>      40.50           -85.6%       5.83 ± 11%  vmstat.cpu.us
>     225.33           -77.7%      50.33 ± 62%  vmstat.io.bi
>       7.00          -100.0%       0.00        vmstat.memory.buff
>      20735 ±  2%     -14.1%      17812 ±  5%  meminfo.Active
>      13506 ±  3%     +31.9%      17812 ±  5%  meminfo.Active(anon)
>       7228          -100.0%       0.00        meminfo.Active(file)
>      29308           +18.4%      34687 ±  2%  meminfo.Shmem
>     202067            -9.5%     182899        meminfo.VmallocUsed
>       0.01 ± 17%      -0.0        0.00 ± 10%  mpstat.cpu.all.iowait%
>       1.04            -0.1        0.92        mpstat.cpu.all.irq%
>       0.03 ±  8%      -0.0        0.02 ±  4%  mpstat.cpu.all.soft%
>      51.54           +35.6       87.17        mpstat.cpu.all.sys%
>      42.22           -35.6        6.66 ±  8%  mpstat.cpu.all.usr%
>       0.00 ± 70%    +191.7%       0.01 ± 26%  perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
>       0.00 ± 47%    +158.8%       0.01 ± 43%  perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
>       0.27 ± 25%     -55.4%       0.12 ± 37%  perf-sched.total_wait_and_delay.average.ms
>     202.17 ± 23%     -29.5%     142.50 ± 13%  perf-sched.total_wait_and_delay.count.ms
>       0.21 ± 18%     -69.3%       0.06 ± 58%  perf-sched.total_wait_time.average.ms
>       0.00 ± 70%    +191.7%       0.01 ± 26%  perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
>       8.17 ± 29%     -85.7%       1.17 ±125%  perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
>       0.00 ± 47%    +158.8%       0.01 ± 43%  perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.worker_thread.kthread.ret_from_fork
>       3.11 ± 11%     -76.1%       0.74 ±142%  perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
>       1790 ± 15%     -30.1%       1250 ± 13%  slabinfo.dmaengine-unmap-16.active_objs
>       1790 ± 15%     -30.1%       1250 ± 13%  slabinfo.dmaengine-unmap-16.num_objs
>     123.00          -100.0%       0.00        slabinfo.ext4_pending_reservation.active_objs
>     123.00          -100.0%       0.00        slabinfo.ext4_pending_reservation.num_objs
>       3619          -100.0%       0.00        slabinfo.f2fs_free_nid.active_objs
>       3619          -100.0%       0.00        slabinfo.f2fs_free_nid.num_objs
>      62865           -44.1%      35155        slabinfo.kmalloc-64.active_objs
>     984.33           -43.9%     552.50        slabinfo.kmalloc-64.active_slabs
>      63031           -43.9%      35389        slabinfo.kmalloc-64.num_objs
>     984.33           -43.9%     552.50        slabinfo.kmalloc-64.num_slabs
>     161.00 ±  9%     +67.1%     269.00 ±  7%  slabinfo.xfs_buf.active_objs
>     161.00 ±  9%     +67.1%     269.00 ±  7%  slabinfo.xfs_buf.num_objs
>       3399 ±  3%     +32.9%       4519 ±  4%  proc-vmstat.nr_active_anon
>       1806          -100.0%       0.00        proc-vmstat.nr_active_file
>       9333            +3.0%       9610        proc-vmstat.nr_mapped
>       7344           +18.4%       8698 ±  2%  proc-vmstat.nr_shmem
>      16319            -0.9%      16176        proc-vmstat.nr_slab_reclaimable
>      24399            -6.2%      22882        proc-vmstat.nr_slab_unreclaimable
>       3399 ±  3%     +32.9%       4519 ±  4%  proc-vmstat.nr_zone_active_anon
>       1806          -100.0%       0.00        proc-vmstat.nr_zone_active_file
>       3693 ± 80%     -80.1%     736.17 ± 61%  proc-vmstat.numa_hint_faults
>     293002            -2.7%     284991        proc-vmstat.numa_hit
>     249530            -3.3%     241180        proc-vmstat.numa_local
>       5007 ±111%     -90.5%     478.00 ± 81%  proc-vmstat.numa_pages_migrated
>      11443 ±  2%      -7.0%      10636 ±  4%  proc-vmstat.pgactivate
>     332528            -3.9%     319693        proc-vmstat.pgalloc_normal
>     249148 ±  2%      -4.1%     239053 ±  2%  proc-vmstat.pgfree
>       5007 ±111%     -90.5%     478.00 ± 81%  proc-vmstat.pgmigrate_success
>      14704           -77.5%       3310 ± 61%  proc-vmstat.pgpgin
>       0.00       +2.1e+105%       2095        proc-vmstat.pgpgout
>      13870 ± 10%     -55.1%       6227 ± 28%  softirqs.CPU0.RCU
>       9989 ±  3%     -62.2%       3775 ± 23%  softirqs.CPU1.RCU
>       8625 ± 13%     -76.1%       2061 ± 10%  softirqs.CPU10.RCU
>       7954 ± 15%     -65.9%       2709 ± 18%  softirqs.CPU14.RCU
>       9075 ± 14%     -78.7%       1929 ± 11%  softirqs.CPU17.RCU
>       8522 ± 13%     -76.7%       1985 ± 22%  softirqs.CPU18.RCU
>       9595 ±  7%     -63.3%       3522 ± 22%  softirqs.CPU2.RCU
>       8455 ± 11%     -74.5%       2152 ± 45%  softirqs.CPU20.RCU
>       8320 ± 12%     -76.7%       1939 ± 14%  softirqs.CPU21.RCU
>       8338 ± 13%     -71.7%       2359 ± 32%  softirqs.CPU23.RCU
>       8541 ± 12%     -75.5%       2089 ± 32%  softirqs.CPU26.RCU
>       9639 ± 20%     -79.5%       1976 ± 17%  softirqs.CPU28.RCU
>       9232 ± 13%     -78.0%       2026 ±  6%  softirqs.CPU30.RCU
>       7857 ± 17%     -68.9%       2446 ± 27%  softirqs.CPU34.RCU
>       8619 ± 11%     -75.8%       2081 ± 30%  softirqs.CPU36.RCU
>       9614 ±  3%     -74.3%       2469 ± 15%  softirqs.CPU4.RCU
>       8962 ± 10%     -77.9%       1981 ± 12%  softirqs.CPU41.RCU
>       9027 ± 12%     -78.6%       1932 ±  8%  softirqs.CPU42.RCU
>       9364 ± 12%     -76.5%       2197 ±  8%  softirqs.CPU44.RCU
>       8774 ± 13%     -75.5%       2147 ± 21%  softirqs.CPU47.RCU
>       8783 ± 12%     -76.0%       2105 ± 12%  softirqs.CPU5.RCU
>       9007 ±  9%     -75.8%       2177 ±  8%  softirqs.CPU6.RCU
>     417664 ±  7%     -72.8%     113621 ±  8%  softirqs.RCU
>      12708 ±  4%     +13.0%      14362 ±  2%  softirqs.TIMER
>      60500           -27.7%      43751        interrupts.CAL:Function_call_interrupts
>       1121           -26.9%     819.17 ±  3%  interrupts.CPU10.CAL:Function_call_interrupts
>       1561 ± 43%     -48.8%     800.00 ±  5%  interrupts.CPU11.CAL:Function_call_interrupts
>       1425 ±  6%     -25.3%       1065 ±  6%  interrupts.CPU12.CAL:Function_call_interrupts
>     166.17 ± 13%     -26.4%     122.33 ± 21%  interrupts.CPU13.RES:Rescheduling_interrupts
>       1402 ± 18%     -25.6%       1043 ± 22%  interrupts.CPU15.CAL:Function_call_interrupts
>     129.17 ± 50%     -42.5%      74.33 ±  4%  interrupts.CPU17.RES:Rescheduling_interrupts
>       1182 ±  9%     -31.0%     815.00 ±  2%  interrupts.CPU20.CAL:Function_call_interrupts
>       1120           -29.7%     787.17 ±  4%  interrupts.CPU21.CAL:Function_call_interrupts
>       1115 ±  3%     -28.2%     801.17        interrupts.CPU23.CAL:Function_call_interrupts
>       1169 ±  7%     -27.2%     851.33 ±  5%  interrupts.CPU24.CAL:Function_call_interrupts
>     177.33 ± 98%     -55.9%      78.17 ±  6%  interrupts.CPU25.RES:Rescheduling_interrupts
>       1142 ± 16%     -28.8%     813.00 ±  3%  interrupts.CPU27.CAL:Function_call_interrupts
>       1229 ± 18%     -33.3%     820.33 ±  4%  interrupts.CPU28.CAL:Function_call_interrupts
>       1124 ±  4%     -28.3%     806.17        interrupts.CPU29.CAL:Function_call_interrupts
>       1123 ±  3%     -28.5%     803.00        interrupts.CPU30.CAL:Function_call_interrupts
>       1127 ±  2%     -32.0%     766.67 ± 19%  interrupts.CPU31.CAL:Function_call_interrupts
>       1066 ±  8%     -22.3%     829.33 ±  7%  interrupts.CPU32.CAL:Function_call_interrupts
>       1109           -26.0%     820.50 ±  4%  interrupts.CPU34.CAL:Function_call_interrupts
>       1315 ± 22%     -37.7%     818.83 ±  2%  interrupts.CPU38.CAL:Function_call_interrupts
>       1164 ±  4%     -29.0%     827.00 ±  3%  interrupts.CPU39.CAL:Function_call_interrupts
>       5513 ± 35%     +13.1%       6237 ± 33%  interrupts.CPU39.NMI:Non-maskable_interrupts
>       5513 ± 35%     +13.1%       6237 ± 33%  interrupts.CPU39.PMI:Performance_monitoring_interrupts
>       1277 ± 23%     -36.1%     815.83        interrupts.CPU40.CAL:Function_call_interrupts
>      97.33 ± 28%     -25.7%      72.33 ±  6%  interrupts.CPU40.RES:Rescheduling_interrupts
>       1116           -24.8%     839.00 ± 11%  interrupts.CPU42.CAL:Function_call_interrupts
>       1130 ±  3%     -28.5%     808.67 ±  3%  interrupts.CPU43.CAL:Function_call_interrupts
>       1121           -29.8%     787.50 ±  4%  interrupts.CPU45.CAL:Function_call_interrupts
>       1119           -27.3%     813.83        interrupts.CPU46.CAL:Function_call_interrupts
>       1167 ±  6%     -28.0%     840.67        interrupts.CPU47.CAL:Function_call_interrupts
>       1667 ± 41%     -44.3%     928.67 ± 24%  interrupts.CPU5.CAL:Function_call_interrupts
>       1369 ± 24%     -39.5%     827.67 ±  3%  interrupts.CPU6.CAL:Function_call_interrupts
>      96.83 ± 25%     -23.1%      74.50 ±  2%  interrupts.CPU7.RES:Rescheduling_interrupts
>       1123           -28.1%     807.00        interrupts.CPU9.CAL:Function_call_interrupts
>       0.72 ±  5%    +107.4%       1.50 ±  2%  perf-stat.i.MPKI
>  8.023e+09           -13.5%  6.943e+09        perf-stat.i.branch-instructions
>       1.08 ±  2%      -0.5        0.55 ±  3%  perf-stat.i.branch-miss-rate%
>   71073978 ±  2%     -67.4%   23149119 ±  3%  perf-stat.i.branch-misses
>      29.33 ±  2%      +5.9       35.23        perf-stat.i.cache-miss-rate%
>    6189596          +120.2%   13628525        perf-stat.i.cache-misses
>   21228048           +82.4%   38714786        perf-stat.i.cache-references
>       3.36           +34.8%       4.53        perf-stat.i.cpi
>     109.52 ±  2%     -22.5%      84.92        perf-stat.i.cpu-migrations
>      22695           -56.5%       9882        perf-stat.i.cycles-between-cache-misses
>       1.15 ±  7%      -0.9        0.30 ±  4%  perf-stat.i.dTLB-load-miss-rate%
>  1.398e+08 ±  7%     -83.4%   23247225 ±  4%  perf-stat.i.dTLB-load-misses
>  1.154e+10           -34.4%  7.564e+09        perf-stat.i.dTLB-loads
>       1.17            -0.0        1.13        perf-stat.i.dTLB-store-miss-rate%
>  1.321e+08           -82.1%   23679743        perf-stat.i.dTLB-store-misses
>  1.071e+10           -81.3%  2.005e+09        perf-stat.i.dTLB-stores
>   41869658           -74.5%   10693569 ± 56%  perf-stat.i.iTLB-load-misses
>   19932113 ± 38%     -88.3%    2325708 ± 64%  perf-stat.i.iTLB-loads
>  3.945e+10           -25.9%  2.924e+10        perf-stat.i.instructions
>       1199 ±  4%    +182.7%       3389 ± 26%  perf-stat.i.instructions-per-iTLB-miss
>       0.31           -23.7%       0.24        perf-stat.i.ipc
>     634.71           -45.5%     345.81        perf-stat.i.metric.M/sec
>     166710 ± 10%   +3369.3%    5783625        perf-stat.i.node-load-misses
>     227268 ±  6%   +3063.1%    7188743        perf-stat.i.node-loads
>      48.41            -4.5       43.91        perf-stat.i.node-store-miss-rate%
>    5425859           -11.8%    4783945        perf-stat.i.node-store-misses
>    5599687            +6.1%    5943407        perf-stat.i.node-stores
>    7532204           -82.7%    1305118        perf-stat.i.page-faults
>       0.54          +146.0%       1.32        perf-stat.overall.MPKI
>       0.89 ±  2%      -0.6        0.33 ±  3%  perf-stat.overall.branch-miss-rate%
>      29.17 ±  2%      +6.0       35.20        perf-stat.overall.cache-miss-rate%
>       3.45           +35.0%       4.65        perf-stat.overall.cpi
>      21953           -54.5%       9979        perf-stat.overall.cycles-between-cache-misses
>       1.20 ±  7%      -0.9        0.31 ±  4%  perf-stat.overall.dTLB-load-miss-rate%
>       1.22            -0.1        1.17        perf-stat.overall.dTLB-store-miss-rate%
>     942.39          +245.5%       3255 ± 28%  perf-stat.overall.instructions-per-iTLB-miss
>       0.29           -25.9%       0.21        perf-stat.overall.ipc
>      42.24 ±  2%      +2.3       44.58        perf-stat.overall.node-load-miss-rate%
>      49.21            -4.6       44.59        perf-stat.overall.node-store-miss-rate%
>  7.894e+09           -13.5%   6.83e+09        perf-stat.ps.branch-instructions
>   69952381 ±  2%     -67.4%   22781972 ±  3%  perf-stat.ps.branch-misses
>    6093197          +120.1%   13409962        perf-stat.ps.cache-misses
>   20897937           +82.3%   38097787        perf-stat.ps.cache-references
>     107.78 ±  2%     -22.4%      83.62        perf-stat.ps.cpu-migrations
>  1.375e+08 ±  7%     -83.4%   22871450 ±  4%  perf-stat.ps.dTLB-load-misses
>  1.135e+10           -34.4%  7.442e+09        perf-stat.ps.dTLB-loads
>    1.3e+08           -82.1%   23295591        perf-stat.ps.dTLB-store-misses
>  1.054e+10           -81.3%  1.973e+09        perf-stat.ps.dTLB-stores
>   41193894           -74.5%   10519305 ± 56%  perf-stat.ps.iTLB-load-misses
>   19610606 ± 38%     -88.3%    2288293 ± 64%  perf-stat.ps.iTLB-loads
>  3.882e+10           -25.9%  2.876e+10        perf-stat.ps.instructions
>     164152 ± 10%   +3366.2%    5689843        perf-stat.ps.node-load-misses
>     223940 ±  6%   +3058.2%    7072454        perf-stat.ps.node-loads
>    5338769           -11.8%    4706549        perf-stat.ps.node-store-misses
>    5510338            +6.1%    5847491        perf-stat.ps.node-stores
>    7410609           -82.7%    1283937        perf-stat.ps.page-faults
>  2.454e+12           -25.9%  1.817e+12        perf-stat.total.instructions
>      33.68           -29.6        4.04        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
>      18.94           -16.5        2.46        perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
>      13.67           -12.3        1.42        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
>      12.27           -10.9        1.36 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
>       8.04            -7.0        1.05 ±  2%  perf-profile.calltrace.cycles-pp.__entry_text_start
>       6.06            -6.1        0.00        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack
>       6.02 ±  2%      -5.4        0.66 ±  2%  perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigaction.do_syscall_64.entry_SYSCALL_64_after_hwframe
>       5.51            -4.8        0.70        perf-profile.calltrace.cycles-pp.__setup_rt_frame.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
>       0.00            +0.6        0.65        perf-profile.calltrace.cycles-pp.inc_rlimit_ucounts_and_test.__sigqueue_alloc.__send_signal.force_sig_info_to_task.force_sig_fault
>      18.91           +27.0       45.89        perf-profile.calltrace.cycles-pp.irqentry_exit_to_user_mode.asm_exc_page_fault
>      15.00           +30.4       45.42        perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
>      14.66           +30.7       45.38        perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
>      11.73           +34.2       45.97        perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
>       8.85           +36.7       45.54        perf-profile.calltrace.cycles-pp.__bad_area_nosemaphore.exc_page_fault.asm_exc_page_fault
>       8.16           +37.2       45.40        perf-profile.calltrace.cycles-pp.force_sig_fault.__bad_area_nosemaphore.exc_page_fault.asm_exc_page_fault
>       8.04           +37.3       45.39        perf-profile.calltrace.cycles-pp.force_sig_info_to_task.force_sig_fault.__bad_area_nosemaphore.exc_page_fault.asm_exc_page_fault
>       6.53           +37.8       44.36        perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault
>       7.06           +38.0       45.05        perf-profile.calltrace.cycles-pp.__send_signal.force_sig_info_to_task.force_sig_fault.__bad_area_nosemaphore.exc_page_fault
>       4.28 ±  2%     +39.2       43.46        perf-profile.calltrace.cycles-pp.__sigqueue_free.get_signal.arch_do_signal_or_restart.exit_to_user_mode_prepare.irqentry_exit_to_user_mode
>       4.87 ±  2%     +39.9       44.81        perf-profile.calltrace.cycles-pp.__sigqueue_alloc.__send_signal.force_sig_info_to_task.force_sig_fault.__bad_area_nosemaphore
>       0.00           +42.7       42.72        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.put_ucounts.__sigqueue_free.get_signal
>       0.00           +43.0       42.99        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.put_ucounts.__sigqueue_free.get_signal.arch_do_signal_or_restart
>       0.00           +43.1       43.08        perf-profile.calltrace.cycles-pp.put_ucounts.__sigqueue_free.get_signal.arch_do_signal_or_restart.exit_to_user_mode_prepare
>       0.00           +43.5       43.52        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_ucounts.__sigqueue_alloc.__send_signal
>       0.00           +44.0       43.97        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_ucounts.__sigqueue_alloc.__send_signal.force_sig_info_to_task
>       0.00           +44.1       44.05        perf-profile.calltrace.cycles-pp.get_ucounts.__sigqueue_alloc.__send_signal.force_sig_info_to_task.force_sig_fault
>      31.89           +60.0       91.94        perf-profile.calltrace.cycles-pp.asm_exc_page_fault
>      33.84           -29.7        4.10        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
>      19.01           -16.5        2.48        perf-profile.children.cycles-pp.syscall_exit_to_user_mode
>      14.93           -13.3        1.59        perf-profile.children.cycles-pp.syscall_return_via_sysret
>      13.14           -11.7        1.43 ±  2%  perf-profile.children.cycles-pp.do_syscall_64
>      10.60            -9.3        1.27        perf-profile.children.cycles-pp.__entry_text_start
>       6.17 ±  2%      -5.5        0.68 ±  2%  perf-profile.children.cycles-pp.__x64_sys_rt_sigaction
>       5.55            -4.8        0.71 ±  2%  perf-profile.children.cycles-pp.__setup_rt_frame
>       3.63 ±  2%      -3.2        0.40 ±  2%  perf-profile.children.cycles-pp.__x64_sys_rt_sigprocmask
>       3.54            -3.2        0.32 ±  4%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
>       3.11 ±  3%      -2.8        0.35 ±  3%  perf-profile.children.cycles-pp._copy_from_user
>       2.91            -2.5        0.37 ±  2%  perf-profile.children.cycles-pp.copy_fpstate_to_sigframe
>       2.38            -2.1        0.28        perf-profile.children.cycles-pp.__irqentry_text_end
>       2.40 ±  4%      -2.1        0.30 ±  2%  perf-profile.children.cycles-pp.__might_fault
>       2.23            -1.9        0.32        perf-profile.children.cycles-pp.native_irq_return_iret
>       2.26 ±  2%      -1.9        0.35 ±  5%  perf-profile.children.cycles-pp.do_user_addr_fault
>       1.86            -1.6        0.27 ±  3%  perf-profile.children.cycles-pp.do_sigaction
>       1.75            -1.5        0.21 ±  3%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
>       1.67 ±  3%      -1.5        0.19 ±  3%  perf-profile.children.cycles-pp.__set_current_blocked
>       1.51 ±  2%      -1.3        0.18 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock_irq
>       1.33 ±  3%      -1.1        0.19        perf-profile.children.cycles-pp.copy_siginfo_to_user
>       1.24 ±  3%      -1.1        0.13 ±  3%  perf-profile.children.cycles-pp.recalc_sigpending
>       1.30 ±  4%      -1.1        0.18 ±  3%  perf-profile.children.cycles-pp._copy_to_user
>       1.23 ±  3%      -1.1        0.15 ±  3%  perf-profile.children.cycles-pp.___might_sleep
>       1.07 ±  4%      -0.9        0.12 ±  3%  perf-profile.children.cycles-pp.signal_setup_done
>       0.98 ±  4%      -0.9        0.11 ±  4%  perf-profile.children.cycles-pp.fpu__clear
>       0.93 ±  4%      -0.8        0.12 ±  8%  perf-profile.children.cycles-pp.__might_sleep
>       0.89            -0.8        0.14 ±  3%  perf-profile.children.cycles-pp.__clear_user
>       0.85 ±  4%      -0.7        0.10 ±  3%  perf-profile.children.cycles-pp.__set_task_blocked
>       0.81 ±  2%      -0.7        0.10 ±  8%  perf-profile.children.cycles-pp.sigprocmask
>       0.76 ± 11%      -0.7        0.07 ± 10%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
>       0.70 ±  2%      -0.6        0.06 ±  9%  perf-profile.children.cycles-pp.signal_wake_up_state
>       0.73            -0.6        0.13 ±  7%  perf-profile.children.cycles-pp.__perf_sw_event
>       0.50 ±  2%      -0.5        0.04 ± 44%  perf-profile.children.cycles-pp.complete_signal
>       0.50 ±  4%      -0.4        0.08 ± 11%  perf-profile.children.cycles-pp.___perf_sw_event
>       0.45            -0.4        0.06 ±  6%  perf-profile.children.cycles-pp.sync_regs
>       0.44            -0.4        0.07 ± 10%  perf-profile.children.cycles-pp.fixup_vdso_exception
>       0.35 ±  2%      -0.3        0.05        perf-profile.children.cycles-pp.prepare_signal
>       0.35 ±  3%      -0.3        0.08 ±  5%  perf-profile.children.cycles-pp.is_prefetch
>       0.33 ± 11%      -0.3        0.07 ±  6%  perf-profile.children.cycles-pp.kmem_cache_alloc
>       0.19 ±  4%      -0.1        0.05 ± 45%  perf-profile.children.cycles-pp.copy_from_kernel_nofault
>       0.33            +0.0        0.37 ±  3%  perf-profile.children.cycles-pp.kmem_cache_free
>       0.00            +0.4        0.36 ±  4%  perf-profile.children.cycles-pp.dec_rlimit_ucounts
>       0.00            +0.6        0.65        perf-profile.children.cycles-pp.inc_rlimit_ucounts_and_test
>      19.11           +26.8       45.91        perf-profile.children.cycles-pp.irqentry_exit_to_user_mode
>      15.69           +29.8       45.51        perf-profile.children.cycles-pp.exit_to_user_mode_prepare
>      14.70           +30.7       45.38        perf-profile.children.cycles-pp.arch_do_signal_or_restart
>      11.77           +34.2       45.99        perf-profile.children.cycles-pp.exc_page_fault
>       8.90           +36.7       45.55        perf-profile.children.cycles-pp.__bad_area_nosemaphore
>       8.17           +37.2       45.40        perf-profile.children.cycles-pp.force_sig_fault
>       8.10           +37.3       45.39        perf-profile.children.cycles-pp.force_sig_info_to_task
>       6.57           +37.8       44.38        perf-profile.children.cycles-pp.get_signal
>       7.11           +38.0       45.06        perf-profile.children.cycles-pp.__send_signal
>       4.28 ±  2%     +39.2       43.46        perf-profile.children.cycles-pp.__sigqueue_free
>       4.89 ±  2%     +39.9       44.82        perf-profile.children.cycles-pp.__sigqueue_alloc
>       0.00           +43.1       43.09        perf-profile.children.cycles-pp.put_ucounts
>       0.00           +44.1       44.05        perf-profile.children.cycles-pp.get_ucounts
>      31.95           +60.0       91.95        perf-profile.children.cycles-pp.asm_exc_page_fault
>       0.00           +86.2       86.24        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
>       0.32 ±  4%     +86.7       87.01        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
>      18.17           -15.8        2.36        perf-profile.self.cycles-pp.syscall_exit_to_user_mode
>      14.90           -13.3        1.58        perf-profile.self.cycles-pp.syscall_return_via_sysret
>      10.60            -9.3        1.27        perf-profile.self.cycles-pp.__entry_text_start
>       4.55 ±  2%      -4.5        0.04 ± 72%  perf-profile.self.cycles-pp.__sigqueue_alloc
>       4.09 ±  2%      -3.6        0.49 ±  2%  perf-profile.self.cycles-pp.irqentry_exit_to_user_mode
>       2.30 ±  5%      -2.1        0.19 ±  4%  perf-profile.self.cycles-pp.do_syscall_64
>       2.38            -2.1        0.28        perf-profile.self.cycles-pp.__irqentry_text_end
>       2.22            -1.9        0.32        perf-profile.self.cycles-pp.native_irq_return_iret
>       1.92 ±  8%      -1.8        0.16 ±  2%  perf-profile.self.cycles-pp.__x64_sys_rt_sigaction
>       1.72 ±  3%      -1.5        0.20 ±  5%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
>       1.71            -1.5        0.21 ±  4%  perf-profile.self.cycles-pp.copy_fpstate_to_sigframe
>       1.68            -1.5        0.20 ±  3%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
>       1.46 ±  4%      -1.3        0.12 ±  4%  perf-profile.self.cycles-pp.__x64_sys_rt_sigprocmask
>       1.46 ±  2%      -1.3        0.17 ±  3%  perf-profile.self.cycles-pp._raw_spin_lock_irq
>       1.21 ±  6%      -1.1        0.14 ±  5%  perf-profile.self.cycles-pp.__setup_rt_frame
>       1.21 ±  2%      -1.1        0.14 ±  3%  perf-profile.self.cycles-pp.___might_sleep
>       1.06 ±  4%      -1.0        0.09 ±  5%  perf-profile.self.cycles-pp.recalc_sigpending
>       1.01 ±  6%      -1.0        0.05        perf-profile.self.cycles-pp.asm_exc_page_fault
>       0.97 ±  4%      -0.9        0.10 ±  7%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
>       0.95 ±  2%      -0.8        0.12 ±  4%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
>       0.98            -0.8        0.15 ±  4%  perf-profile.self.cycles-pp.do_sigaction
>       0.84 ±  4%      -0.7        0.10 ±  6%  perf-profile.self.cycles-pp.__might_sleep
>       0.72 ±  8%      -0.6        0.09 ±  4%  perf-profile.self.cycles-pp._copy_from_user
>       0.71 ±  2%      -0.6        0.10 ±  7%  perf-profile.self.cycles-pp.do_user_addr_fault
>       0.65 ± 14%      -0.6        0.06 ± 11%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
>       0.68 ±  6%      -0.6        0.10 ±  3%  perf-profile.self.cycles-pp.__might_fault
>       0.80            -0.6        0.25 ±  3%  perf-profile.self.cycles-pp.get_signal
>       0.64 ±  4%      -0.6        0.08 ±  5%  perf-profile.self.cycles-pp.fpu__clear
>       0.57 ±  3%      -0.5        0.07 ±  7%  perf-profile.self.cycles-pp.__send_signal
>       0.53 ±  2%      -0.5        0.08 ±  6%  perf-profile.self.cycles-pp.__clear_user
>       0.49 ±  5%      -0.4        0.04 ± 45%  perf-profile.self.cycles-pp.arch_do_signal_or_restart
>       0.43            -0.4        0.06 ±  7%  perf-profile.self.cycles-pp.fixup_vdso_exception
>       0.42 ±  4%      -0.4        0.06 ±  7%  perf-profile.self.cycles-pp.___perf_sw_event
>       0.41            -0.4        0.05 ±  9%  perf-profile.self.cycles-pp.sync_regs
>       0.33 ±  3%      -0.3        0.04 ± 44%  perf-profile.self.cycles-pp.prepare_signal
>       0.31 ± 12%      -0.2        0.07 ±  7%  perf-profile.self.cycles-pp.kmem_cache_alloc
>       0.22 ±  3%      -0.1        0.10 ±  7%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
>       0.33            +0.0        0.36 ±  3%  perf-profile.self.cycles-pp.kmem_cache_free
>       0.00            +0.1        0.06 ±  9%  perf-profile.self.cycles-pp.get_ucounts
>       0.00            +0.1        0.06 ±  6%  perf-profile.self.cycles-pp.put_ucounts
>       0.00            +0.4        0.36 ±  4%  perf-profile.self.cycles-pp.dec_rlimit_ucounts
>       0.27 ±  4%      +0.5        0.77 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
>       0.00            +0.6        0.65        perf-profile.self.cycles-pp.inc_rlimit_ucounts_and_test
>       0.00           +86.2       86.24        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
>
>
>                                                                                 
>                              stress-ng.time.user_time                           
>                                                                                 
>   1400 +--------------------------------------------------------------------+   
>        |.++.+.+.++.+.+.++.+.++.+.+.++.+.+.++.+.++.+.+.++.+.++.+.+. +.+.+.++.|   
>   1200 |-+                                                        +         |   
>        |                                                                    |   
>   1000 |-+                                                                  |   
>        |                                                                    |   
>    800 |-+                                                                  |   
>        |                                                                    |   
>    600 |-+                                                                  |   
>        |                                                                    |   
>    400 |-+      O                                                           |   
>        |                                                                    |   
>    200 |-OO O O  O O O OO O OO O O OO O O OO O OO O O OO O OO               |   
>        |                                                                    |   
>      0 +--------------------------------------------------------------------+   
>                                                                                 
>                                                                                                                                                                 
>                             stress-ng.time.system_time                          
>                                                                                 
>   2800 +--------------------------------------------------------------------+   
>        |             O         O O    O   OO    O   O O  O O                |   
>   2600 |-OO O O  O O   OO O OO     OO   O    O O  O    O    O               |   
>        |                                                                    |   
>   2400 |-+      O                                                           |   
>        |                                                                    |   
>   2200 |-+                                                                  |   
>        |                                                                    |   
>   2000 |-+                                                                  |   
>        |                                                                    |   
>   1800 |-+                                                                  |   
>        |                                                                    |   
>   1600 |.++.+.+.++.+.    .+.++.+.+.++.+.+.++.+.++.+.+.++.+.+ .+.+.++.+.+.++.|   
>        |             +.++                                   +               |   
>   1400 +--------------------------------------------------------------------+   
>                                                                                 
>                                                                                                                                                                 
>                                  stress-ng.sigsegv.ops                          
>                                                                                 
>     5e+08 +-----------------------------------------------------------------+   
>           |.++.+.++.+.++.+.++.++.+.++.+.++.+.++.+.++.+.++.+.++.++.+.++.+.++.|   
>   4.5e+08 |-+                                                               |   
>     4e+08 |-+                                                               |   
>           |                                                                 |   
>   3.5e+08 |-+                                                               |   
>     3e+08 |-+                                                               |   
>           |                                                                 |   
>   2.5e+08 |-+                                                               |   
>     2e+08 |-+                                                               |   
>           |                                                                 |   
>   1.5e+08 |-+                                                               |   
>     1e+08 |-+                                                               |   
>           | OO O OO O OO O OO OO O OO O OO O OO O OO O OO O OO              |   
>     5e+07 +-----------------------------------------------------------------+   
>                                                                                 
>                                                                                                                                                                 
>                             stress-ng.sigsegv.ops_per_sec                       
>                                                                                 
>   8e+06 +-------------------------------------------------------------------+   
>         |                                                                   |   
>   7e+06 |-+                                                                 |   
>         |                                                                   |   
>   6e+06 |-+                                                                 |   
>         |                                                                   |   
>   5e+06 |-+                                                                 |   
>         |                                                                   |   
>   4e+06 |-+                                                                 |   
>         |                                                                   |   
>   3e+06 |-+                                                                 |   
>         |                                                                   |   
>   2e+06 |-+                                                                 |   
>         | OO O O OO O OO O OO O O OO O OO O OO O O OO O OO O O              |   
>   1e+06 +-------------------------------------------------------------------+   
>                                                                                 
>                                                                                 
> [*] bisect-good sample
> [O] bisect-bad  sample
>
> ***************************************************************************************************
> lkp-ivb-2ep1: 48 threads Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz with 112G memory
> =========================================================================================
> class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime/ucode:
>   interrupt/gcc-9/performance/1HDD/x86_64-rhel-8.3/100%/debian-10.4-x86_64-20200603.cgz/lkp-ivb-2ep1/sigq/stress-ng/60s/0x42e
>
> commit: 
>   4660d663b4 ("Reimplement RLIMIT_MSGQUEUE on top of ucounts")
>   d28296d248 ("Reimplement RLIMIT_SIGPENDING on top of ucounts")
>
> 4660d663b4207ce6 d28296d2484fa11e94dff65e93e 
> ---------------- --------------------------- 
>          %stddev     %change         %stddev
>              \          |                \  
>  4.176e+08           -56.1%  1.831e+08 ± 21%  stress-ng.sigq.ops
>    6959423           -56.1%    3051857 ± 21%  stress-ng.sigq.ops_per_sec
>  2.157e+08 ±  8%     -66.1%   73100172 ±  9%  stress-ng.time.involuntary_context_switches
>      14467            +1.4%      14674        stress-ng.time.minor_page_faults
>       4513            +1.2%       4569        stress-ng.time.percent_of_cpu_this_job_got
>       2150           +17.9%       2535 ±  3%  stress-ng.time.system_time
>     660.96           -53.1%     309.96 ± 30%  stress-ng.time.user_time
>  2.478e+08 ±  5%     -70.5%   73168836 ±  9%  stress-ng.time.voluntary_context_switches
>       7.70 ±  8%      -6.8%       7.18        iostat.cpu.idle
>      70.60           +16.6%      82.32 ±  3%  iostat.cpu.system
>      21.70           -51.6%      10.49 ± 29%  iostat.cpu.user
>      15660 ±  3%     +13.7%      17806 ±  3%  meminfo.Active
>      15660 ±  3%     +13.7%      17806 ±  3%  meminfo.Active(anon)
>      31878 ±  2%      +8.8%      34688 ±  2%  meminfo.Shmem
>     123900 ± 66%     -66.9%      41021 ± 30%  cpuidle.C1.usage
>  1.507e+08 ±  6%      +9.5%   1.65e+08 ±  4%  cpuidle.C6.time
>     681599 ± 38%     -72.8%     185137 ±113%  cpuidle.POLL.time
>     645871 ± 39%     -98.5%       9580 ± 75%  cpuidle.POLL.usage
>       1.78 ±  3%      -0.8        0.95        mpstat.cpu.all.irq%
>       0.03 ±  4%      -0.0        0.02 ± 27%  mpstat.cpu.all.soft%
>      70.90           +12.5       83.42 ±  3%  mpstat.cpu.all.sys%
>      22.35           -11.6       10.75 ± 29%  mpstat.cpu.all.usr%
>     825.33 ±  2%     -11.6%     729.67 ±  5%  slabinfo.file_lock_cache.active_objs
>     825.33 ±  2%     -11.6%     729.67 ±  5%  slabinfo.file_lock_cache.num_objs
>       1455 ± 11%     -32.4%     983.00 ± 13%  slabinfo.khugepaged_mm_slot.active_objs
>       1455 ± 11%     -32.4%     983.00 ± 13%  slabinfo.khugepaged_mm_slot.num_objs
>      69.67           +17.2%      81.67 ±  3%  vmstat.cpu.sy
>      21.00           -52.4%      10.00 ± 29%  vmstat.cpu.us
>    7139282 ±  7%     -68.5%    2251603 ±  9%  vmstat.system.cs
>     518916 ±  6%     -80.5%     101139 ±  4%  vmstat.system.in
>       8082 ± 51%     -68.3%       2565 ± 17%  softirqs.CPU10.SCHED
>       6261 ± 72%     -56.4%       2729 ± 14%  softirqs.CPU17.SCHED
>       6147 ± 66%     -64.3%       2195 ±  3%  softirqs.CPU25.SCHED
>      16334 ± 41%     -82.6%       2846 ± 24%  softirqs.CPU27.SCHED
>       6280 ± 56%     -61.9%       2394 ±  9%  softirqs.CPU39.SCHED
>       8248 ± 50%     -73.2%       2209 ±  8%  softirqs.CPU40.SCHED
>     228327 ±  9%     -46.3%     122665 ±  2%  softirqs.SCHED
>       3851 ±  4%     +13.8%       4381 ±  4%  proc-vmstat.nr_active_anon
>       9587            +2.8%       9855        proc-vmstat.nr_mapped
>       7943 ±  2%      +8.7%       8630 ±  2%  proc-vmstat.nr_shmem
>       3851 ±  4%     +13.8%       4381 ±  4%  proc-vmstat.nr_zone_active_anon
>     438.33 ±122%   +1463.6%       6853 ± 69%  proc-vmstat.numa_pages_migrated
>       8816 ±  4%     +11.5%       9827 ±  6%  proc-vmstat.pgactivate
>     291848            +1.7%     296689        proc-vmstat.pgalloc_normal
>     438.33 ±122%   +1463.6%       6853 ± 69%  proc-vmstat.pgmigrate_success
>     252.00 ± 19%     +30.8%     329.67 ±  7%  numa-vmstat.node0.nr_active_anon
>      44373 ± 23%     -64.0%      15967 ± 49%  numa-vmstat.node0.nr_anon_pages
>      46179 ± 21%     -61.8%      17619 ± 38%  numa-vmstat.node0.nr_inactive_anon
>     252.00 ± 19%     +30.8%     329.67 ±  7%  numa-vmstat.node0.nr_zone_active_anon
>      46179 ± 21%     -61.8%      17619 ± 38%  numa-vmstat.node0.nr_zone_inactive_anon
>       3678 ±  4%     +11.5%       4100 ±  4%  numa-vmstat.node1.nr_active_anon
>      13880 ± 74%    +206.7%      42565 ± 17%  numa-vmstat.node1.nr_anon_pages
>      16004 ± 59%    +181.1%      44988 ± 14%  numa-vmstat.node1.nr_inactive_anon
>       3678 ±  4%     +11.5%       4100 ±  4%  numa-vmstat.node1.nr_zone_active_anon
>      16004 ± 59%    +181.1%      44988 ± 14%  numa-vmstat.node1.nr_zone_inactive_anon
>       1007 ± 19%     +31.3%       1322 ±  7%  numa-meminfo.node0.Active
>       1007 ± 19%     +31.3%       1322 ±  7%  numa-meminfo.node0.Active(anon)
>      39280 ± 35%     -66.1%      13314 ± 74%  numa-meminfo.node0.AnonHugePages
>     177224 ± 24%     -63.9%      63933 ± 49%  numa-meminfo.node0.AnonPages
>     182948 ± 22%     -52.3%      87286 ± 59%  numa-meminfo.node0.AnonPages.max
>     184457 ± 21%     -61.8%      70532 ± 38%  numa-meminfo.node0.Inactive
>     184457 ± 21%     -61.8%      70532 ± 38%  numa-meminfo.node0.Inactive(anon)
>    1080630 ±  5%     -12.3%     947332 ±  3%  numa-meminfo.node0.MemUsed
>      14267 ±  4%     +17.0%      16695 ±  5%  numa-meminfo.node1.Active
>      14267 ±  4%     +17.0%      16695 ±  5%  numa-meminfo.node1.Active(anon)
>      56124 ± 74%    +202.8%     169963 ± 17%  numa-meminfo.node1.AnonPages
>      75025 ± 56%    +139.7%     179816 ± 14%  numa-meminfo.node1.AnonPages.max
>      64951 ± 59%    +176.3%     179452 ± 14%  numa-meminfo.node1.Inactive
>      64951 ± 59%    +176.3%     179452 ± 14%  numa-meminfo.node1.Inactive(anon)
>     941178 ±  5%     +14.6%    1078500 ±  3%  numa-meminfo.node1.MemUsed
>      51.14 ± 15%     +47.4%      75.37 ± 13%  sched_debug.cfs_rq:/.load_avg.avg
>     114.91 ± 14%     +38.2%     158.79 ± 13%  sched_debug.cfs_rq:/.load_avg.stddev
>       6.99 ± 70%    +352.8%      31.65 ± 27%  sched_debug.cfs_rq:/.removed.load_avg.avg
>      47.92 ± 70%    +153.1%     121.29 ± 12%  sched_debug.cfs_rq:/.removed.load_avg.stddev
>       2.38 ± 97%    +182.8%       6.74 ± 36%  sched_debug.cfs_rq:/.removed.runnable_avg.avg
>       2.38 ± 97%    +182.7%       6.73 ± 36%  sched_debug.cfs_rq:/.removed.util_avg.avg
>     356.50 ±  3%     -24.1%     270.50 ± 31%  sched_debug.cfs_rq:/.util_avg.min
>     290.54 ±  2%     +10.6%     321.30 ±  5%  sched_debug.cfs_rq:/.util_est_enqueued.avg
>     446696 ±  4%     +50.1%     670473 ±  7%  sched_debug.cpu.avg_idle.avg
>    1120651 ± 11%     +20.5%    1350114 ± 10%  sched_debug.cpu.avg_idle.max
>       2124 ± 10%     +43.0%       3038 ± 15%  sched_debug.cpu.avg_idle.min
>     708.23 ±  3%     -14.4%     606.42 ±  4%  sched_debug.cpu.clock_task.stddev
>     989.50 ±  8%     +12.0%       1108        sched_debug.cpu.curr->pid.min
>      21503 ± 71%    +114.0%      46014 ± 39%  sched_debug.cpu.max_idle_balance_cost.stddev
>    4583665 ±  7%     -68.5%    1443748 ±  9%  sched_debug.cpu.nr_switches.avg
>    6508139 ±  3%     -71.5%    1854831        sched_debug.cpu.nr_switches.max
>    2183394 ± 28%     -83.2%     367194 ± 87%  sched_debug.cpu.nr_switches.min
>    1135976 ± 11%     -69.9%     342489 ± 34%  sched_debug.cpu.nr_switches.stddev
>       0.03 ± 59%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
>       0.01 ± 12%     -36.4%       0.00 ± 26%  perf-sched.sch_delay.avg.ms.__sched_text_start.__sched_text_start.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
>       0.05 ± 57%    -100.0%       0.00        perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
>       0.19 ± 77%     -61.0%       0.07 ±  5%  perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
>       7.83 ±140%     -99.8%       0.01 ± 38%  perf-sched.sch_delay.max.ms.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
>       0.36 ± 18%     -64.4%       0.13 ± 75%  perf-sched.total_wait_and_delay.average.ms
>       0.24 ±  5%     -72.2%       0.07 ± 66%  perf-sched.total_wait_time.average.ms
>       1.38 ±  6%    -100.0%       0.00        perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
>       0.03 ± 41%     -61.6%       0.01 ± 80%  perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
>       0.01 ± 12%     -36.4%       0.00 ± 26%  perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
>       0.08 ± 17%     -30.3%       0.05 ±  3%  perf-sched.wait_and_delay.avg.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
>       2.00          -100.0%       0.00        perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.do_syslog.part.0
>       9.00           -81.5%       1.67 ± 28%  perf-sched.wait_and_delay.count.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
>       2.72 ±  8%    -100.0%       0.00        perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
>       0.80 ± 45%     -89.3%       0.09 ± 10%  perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
>       7.83 ±140%     -99.8%       0.01 ± 38%  perf-sched.wait_and_delay.max.ms.__sched_text_start.__sched_text_start.smpboot_thread_fn.kthread.ret_from_fork
>       1.35 ±  8%    -100.0%       0.00        perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
>       0.03 ± 42%     -66.3%       0.01 ± 82%  perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.do_task_dead.do_exit.do_group_exit
>       0.07 ± 24%     -27.6%       0.05 ±  3%  perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
>       3.21 ±  2%     -58.6%       1.33 ± 70%  perf-sched.wait_time.avg.ms.__sched_text_start.__sched_text_start.schedule_timeout.rcu_gp_kthread.kthread
>       2.69 ±  8%    -100.0%       0.00        perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.do_syslog.part.0
>       0.77 ± 50%     -89.7%       0.08 ± 10%  perf-sched.wait_time.max.ms.__sched_text_start.__sched_text_start.pipe_read.new_sync_read.vfs_read
>       6.12 ±  3%     -27.9%       4.42 ±  3%  perf-stat.i.MPKI
>  1.008e+10 ±  4%     -20.8%  7.989e+09 ±  2%  perf-stat.i.branch-instructions
>       1.55 ±  2%      -0.7        0.89        perf-stat.i.branch-miss-rate%
>  1.372e+08 ±  2%     -60.4%   54290819        perf-stat.i.branch-misses
>       5.24 ±  7%      +6.4       11.66 ±  5%  perf-stat.i.cache-miss-rate%
>   12610881 ±  8%     +30.8%   16498692 ±  4%  perf-stat.i.cache-misses
>  2.994e+08           -50.1%  1.495e+08 ±  2%  perf-stat.i.cache-references
>    7370989 ±  7%     -68.6%    2316288 ±  9%  perf-stat.i.context-switches
>       2.75 ±  4%     +37.7%       3.78 ±  2%  perf-stat.i.cpi
>      37935 ± 20%     -99.1%     344.33 ± 39%  perf-stat.i.cpu-migrations
>      10857 ±  9%     -24.6%       8184 ±  4%  perf-stat.i.cycles-between-cache-misses
>       1.33 ±  4%      -0.5        0.81 ± 15%  perf-stat.i.dTLB-load-miss-rate%
>  1.921e+08 ±  3%     -59.4%   78058438 ± 14%  perf-stat.i.dTLB-load-misses
>  1.375e+10 ±  4%     -32.6%   9.26e+09        perf-stat.i.dTLB-loads
>       0.51 ±  5%      +0.1        0.61 ±  9%  perf-stat.i.dTLB-store-miss-rate%
>   52874010           -55.6%   23456497 ± 13%  perf-stat.i.dTLB-store-misses
>  9.955e+09 ±  3%     -63.0%  3.687e+09 ±  4%  perf-stat.i.dTLB-stores
>   62946854 ±  4%     -59.7%   25392611 ±  3%  perf-stat.i.iTLB-load-misses
>   4.88e+10 ±  4%     -28.0%  3.514e+10        perf-stat.i.instructions
>       1071           +54.5%       1655 ± 10%  perf-stat.i.instructions-per-iTLB-miss
>       0.38 ±  3%     -26.8%       0.28 ±  3%  perf-stat.i.ipc
>       0.52 ±  7%     -70.6%       0.15 ± 11%  perf-stat.i.metric.K/sec
>     711.88 ±  3%     -38.1%     440.48        perf-stat.i.metric.M/sec
>      47.42            -1.7       45.75        perf-stat.i.node-load-miss-rate%
>    3468825 ± 20%    +118.3%    7571866 ±  5%  perf-stat.i.node-load-misses
>    3747717 ± 20%    +133.9%    8764955 ±  4%  perf-stat.i.node-loads
>      47.67            -2.6       45.11        perf-stat.i.node-store-miss-rate%
>    8290895 ±  4%     -28.2%    5954593 ±  6%  perf-stat.i.node-store-misses
>    8800840           -19.9%    7053271 ±  5%  perf-stat.i.node-stores
>       6.14 ±  3%     -30.8%       4.25        perf-stat.overall.MPKI
>       1.36 ±  2%      -0.7        0.68        perf-stat.overall.branch-miss-rate%
>       4.22 ± 10%      +6.8       11.05 ±  6%  perf-stat.overall.cache-miss-rate%
>       2.79 ±  4%     +38.7%       3.87        perf-stat.overall.cpi
>      10868 ±  8%     -23.9%       8267 ±  4%  perf-stat.overall.cycles-between-cache-misses
>       1.38 ±  4%      -0.5        0.84 ± 15%  perf-stat.overall.dTLB-load-miss-rate%
>       0.53 ±  5%      +0.1        0.63 ±  8%  perf-stat.overall.dTLB-store-miss-rate%
>     775.35           +78.9%       1386 ±  5%  perf-stat.overall.instructions-per-iTLB-miss
>       0.36 ±  4%     -28.0%       0.26        perf-stat.overall.ipc
>      48.07            -1.7       46.34        perf-stat.overall.node-load-miss-rate%
>      48.49            -2.7       45.76        perf-stat.overall.node-store-miss-rate%
>  9.922e+09 ±  4%     -20.8%  7.859e+09 ±  2%  perf-stat.ps.branch-instructions
>  1.351e+08 ±  2%     -60.4%   53438494        perf-stat.ps.branch-misses
>   12410866 ±  8%     +30.8%   16234283 ±  4%  perf-stat.ps.cache-misses
>  2.946e+08           -50.1%   1.47e+08 ±  2%  perf-stat.ps.cache-references
>    7252050 ±  7%     -68.6%    2278311 ±  9%  perf-stat.ps.context-switches
>      37323 ± 20%     -99.1%     338.80 ± 39%  perf-stat.ps.cpu-migrations
>   1.89e+08 ±  3%     -59.4%   76787292 ± 14%  perf-stat.ps.dTLB-load-misses
>  1.352e+10 ±  4%     -32.6%   9.11e+09        perf-stat.ps.dTLB-loads
>   52020930           -55.6%   23075080 ± 13%  perf-stat.ps.dTLB-store-misses
>  9.794e+09 ±  3%     -63.0%  3.627e+09 ±  4%  perf-stat.ps.dTLB-stores
>   61931461 ±  4%     -59.7%   24978175 ±  3%  perf-stat.ps.iTLB-load-misses
>  4.802e+10 ±  4%     -28.0%  3.457e+10        perf-stat.ps.instructions
>    3412948 ± 20%    +118.2%    7448345 ±  5%  perf-stat.ps.node-load-misses
>    3687448 ± 20%    +133.8%    8622075 ±  4%  perf-stat.ps.node-loads
>    8157655 ±  4%     -28.2%    5857930 ±  6%  perf-stat.ps.node-store-misses
>    8659916           -19.9%    6939197 ±  5%  perf-stat.ps.node-stores
>  3.035e+12 ±  4%     -27.9%  2.189e+12        perf-stat.total.instructions
>      82.33 ± 64%   +1322.3%       1171 ±127%  interrupts.36:PCI-MSI.2621442-edge.eth0-TxRx-1
>   17959753 ±  6%     -99.7%      45889 ±  2%  interrupts.CAL:Function_call_interrupts
>     103593 ±125%     -99.5%     569.33 ±  9%  interrupts.CPU0.CAL:Function_call_interrupts
>      45339 ±104%     -94.7%       2414 ± 30%  interrupts.CPU0.RES:Rescheduling_interrupts
>    1078794 ±112%     -99.9%       1002 ± 12%  interrupts.CPU1.CAL:Function_call_interrupts
>       6507 ± 26%     +21.7%       7922 ±  6%  interrupts.CPU1.NMI:Non-maskable_interrupts
>       6507 ± 26%     +21.7%       7922 ±  6%  interrupts.CPU1.PMI:Performance_monitoring_interrupts
>     505526 ±101%     -99.8%       1223 ± 46%  interrupts.CPU1.RES:Rescheduling_interrupts
>     791738 ± 80%     -99.9%     898.67 ± 13%  interrupts.CPU10.CAL:Function_call_interrupts
>     406363 ± 71%     -99.4%       2378 ± 80%  interrupts.CPU10.RES:Rescheduling_interrupts
>      46401 ±115%     -98.3%     789.00 ± 21%  interrupts.CPU11.CAL:Function_call_interrupts
>       5509 ± 35%     +25.8%       6931 ± 28%  interrupts.CPU11.NMI:Non-maskable_interrupts
>       5509 ± 35%     +25.8%       6931 ± 28%  interrupts.CPU11.PMI:Performance_monitoring_interrupts
>      27829 ±104%     -87.3%       3526 ± 77%  interrupts.CPU11.RES:Rescheduling_interrupts
>     290182 ±130%     -99.6%       1299 ± 24%  interrupts.CPU12.CAL:Function_call_interrupts
>     170757 ±136%     -98.3%       2886 ± 40%  interrupts.CPU12.RES:Rescheduling_interrupts
>     131940 ± 43%     -99.1%       1206 ± 37%  interrupts.CPU13.CAL:Function_call_interrupts
>      68303 ± 32%     -97.9%       1445 ±126%  interrupts.CPU13.RES:Rescheduling_interrupts
>     241043 ±116%     -99.5%       1319 ± 32%  interrupts.CPU14.CAL:Function_call_interrupts
>      98573 ± 93%     -97.0%       3005 ± 47%  interrupts.CPU14.RES:Rescheduling_interrupts
>     874074 ± 72%     -99.9%     875.33 ±  6%  interrupts.CPU15.CAL:Function_call_interrupts
>     350120 ± 49%     -99.9%     412.67 ± 52%  interrupts.CPU15.RES:Rescheduling_interrupts
>     577696 ± 23%     -99.9%     855.33 ±  3%  interrupts.CPU16.CAL:Function_call_interrupts
>       8263           -24.9%       6202 ± 23%  interrupts.CPU16.NMI:Non-maskable_interrupts
>       8263           -24.9%       6202 ± 23%  interrupts.CPU16.PMI:Performance_monitoring_interrupts
>     237815 ± 15%     -99.5%       1220 ± 53%  interrupts.CPU16.RES:Rescheduling_interrupts
>     686797 ±112%     -99.9%     933.67 ± 13%  interrupts.CPU17.CAL:Function_call_interrupts
>     317108 ±105%     -99.8%     536.00 ± 49%  interrupts.CPU17.RES:Rescheduling_interrupts
>     748163 ±122%     -99.9%     863.00 ±  5%  interrupts.CPU18.CAL:Function_call_interrupts
>     320920 ±112%     -99.5%       1469 ± 95%  interrupts.CPU18.RES:Rescheduling_interrupts
>     286835 ±121%     -99.7%     943.33 ± 15%  interrupts.CPU19.CAL:Function_call_interrupts
>     174455 ±113%     -98.7%       2326 ± 69%  interrupts.CPU19.RES:Rescheduling_interrupts
>     443377 ± 83%     -99.8%     944.33 ± 12%  interrupts.CPU2.CAL:Function_call_interrupts
>      57297 ± 99%     -98.5%     855.00 ±  6%  interrupts.CPU20.CAL:Function_call_interrupts
>      54090 ± 67%     -99.0%     535.33 ± 74%  interrupts.CPU20.RES:Rescheduling_interrupts
>      27584 ± 80%     -96.8%     890.33 ±  6%  interrupts.CPU21.CAL:Function_call_interrupts
>      27052 ± 82%     -94.3%       1539 ± 58%  interrupts.CPU21.RES:Rescheduling_interrupts
>      62804 ±104%     -97.8%       1362 ± 47%  interrupts.CPU22.CAL:Function_call_interrupts
>      27230 ± 84%     -92.6%       2002 ± 86%  interrupts.CPU22.RES:Rescheduling_interrupts
>     351930 ± 72%     -99.7%     966.00 ± 16%  interrupts.CPU23.CAL:Function_call_interrupts
>     136149 ± 62%     -98.1%       2565 ± 93%  interrupts.CPU23.RES:Rescheduling_interrupts
>     366644 ±138%     -99.8%     877.67 ±  5%  interrupts.CPU24.CAL:Function_call_interrupts
>     499203 ± 82%     -99.8%     862.33 ±  3%  interrupts.CPU25.CAL:Function_call_interrupts
>     298172 ± 70%     -99.2%       2317 ± 19%  interrupts.CPU25.RES:Rescheduling_interrupts
>      11844 ± 61%     -92.9%     838.33 ±  2%  interrupts.CPU26.CAL:Function_call_interrupts
>      82.33 ± 64%   +1322.3%       1171 ±127%  interrupts.CPU27.36:PCI-MSI.2621442-edge.eth0-TxRx-1
>    1841110 ± 49%    -100.0%     746.33 ± 26%  interrupts.CPU27.CAL:Function_call_interrupts
>       5501 ± 35%     +25.6%       6909 ± 28%  interrupts.CPU27.NMI:Non-maskable_interrupts
>       5501 ± 35%     +25.6%       6909 ± 28%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
>    1089819 ± 55%     -99.8%       1727 ± 80%  interrupts.CPU27.RES:Rescheduling_interrupts
>      74718 ±118%     -98.7%     942.00        interrupts.CPU28.CAL:Function_call_interrupts
>       5504 ± 35%     +25.6%       6912 ± 28%  interrupts.CPU28.NMI:Non-maskable_interrupts
>       5504 ± 35%     +25.6%       6912 ± 28%  interrupts.CPU28.PMI:Performance_monitoring_interrupts
>      67380 ±125%     -98.5%       1034 ± 37%  interrupts.CPU28.RES:Rescheduling_interrupts
>     367428 ± 77%     -99.7%       1054 ± 22%  interrupts.CPU29.CAL:Function_call_interrupts
>     239038 ± 82%     -99.3%       1767 ± 84%  interrupts.CPU29.RES:Rescheduling_interrupts
>     220744 ±117%     -99.5%       1046 ± 21%  interrupts.CPU3.CAL:Function_call_interrupts
>     191269 ±119%     -98.4%       3122 ± 15%  interrupts.CPU3.RES:Rescheduling_interrupts
>     247241 ± 81%     -99.6%     874.33 ±  5%  interrupts.CPU30.CAL:Function_call_interrupts
>     161878 ± 84%     -98.1%       3063 ± 37%  interrupts.CPU30.RES:Rescheduling_interrupts
>     144713 ±110%     -99.3%       1077 ± 23%  interrupts.CPU31.CAL:Function_call_interrupts
>       5502 ± 35%     +25.6%       6911 ± 28%  interrupts.CPU31.NMI:Non-maskable_interrupts
>       5502 ± 35%     +25.6%       6911 ± 28%  interrupts.CPU31.PMI:Performance_monitoring_interrupts
>      84227 ±100%     -96.4%       3013 ± 72%  interrupts.CPU31.RES:Rescheduling_interrupts
>     104510 ±109%     -99.2%     848.00 ±  4%  interrupts.CPU32.CAL:Function_call_interrupts
>       5515 ± 35%     +31.4%       7249 ± 20%  interrupts.CPU32.NMI:Non-maskable_interrupts
>       5515 ± 35%     +31.4%       7249 ± 20%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
>      70068 ±101%     -96.7%       2320 ±126%  interrupts.CPU32.RES:Rescheduling_interrupts
>      87671 ±131%     -99.0%     857.33 ±  2%  interrupts.CPU33.CAL:Function_call_interrupts
>      35054 ±127%     -96.5%       1242 ± 80%  interrupts.CPU33.RES:Rescheduling_interrupts
>      77336 ±124%     -98.7%     988.33 ± 21%  interrupts.CPU34.CAL:Function_call_interrupts
>      66106 ±123%     -96.0%       2630 ± 52%  interrupts.CPU34.RES:Rescheduling_interrupts
>     136901 ±122%     -99.1%       1278 ± 30%  interrupts.CPU35.CAL:Function_call_interrupts
>     731054 ± 69%     -99.9%       1046 ± 15%  interrupts.CPU36.CAL:Function_call_interrupts
>     376136 ± 70%     -99.6%       1429 ± 37%  interrupts.CPU36.RES:Rescheduling_interrupts
>     941442 ±105%     -99.9%     851.33 ±  7%  interrupts.CPU37.CAL:Function_call_interrupts
>     300885 ± 80%     -99.2%       2379 ± 57%  interrupts.CPU37.RES:Rescheduling_interrupts
>     242878 ±117%     -99.6%     995.00 ± 16%  interrupts.CPU38.CAL:Function_call_interrupts
>     702836 ± 69%     -99.9%     855.00 ±  3%  interrupts.CPU39.CAL:Function_call_interrupts
>     286450 ± 74%     -99.5%       1448 ± 70%  interrupts.CPU39.RES:Rescheduling_interrupts
>      37564 ± 55%     -96.0%       1505 ± 62%  interrupts.CPU4.CAL:Function_call_interrupts
>       6346 ± 26%     +24.0%       7870 ±  6%  interrupts.CPU4.NMI:Non-maskable_interrupts
>       6346 ± 26%     +24.0%       7870 ±  6%  interrupts.CPU4.PMI:Performance_monitoring_interrupts
>      24154 ± 65%     -94.0%       1444 ± 83%  interrupts.CPU4.RES:Rescheduling_interrupts
>     977103 ± 53%     -99.9%     837.33 ±  3%  interrupts.CPU40.CAL:Function_call_interrupts
>     512103 ± 71%     -99.7%       1579 ± 66%  interrupts.CPU40.RES:Rescheduling_interrupts
>     190023 ±117%     -99.6%     818.67 ±  2%  interrupts.CPU41.CAL:Function_call_interrupts
>     346179 ± 54%     -99.8%     829.00 ±  2%  interrupts.CPU42.CAL:Function_call_interrupts
>     183103 ± 48%     -99.5%     850.00 ± 54%  interrupts.CPU42.RES:Rescheduling_interrupts
>     371814 ±131%     -99.8%     863.33 ±  5%  interrupts.CPU43.CAL:Function_call_interrupts
>     158123 ±124%     -99.2%       1224 ± 71%  interrupts.CPU43.RES:Rescheduling_interrupts
>     578077 ± 61%     -99.9%     850.67 ±  3%  interrupts.CPU44.CAL:Function_call_interrupts
>     331471 ± 53%     -99.7%       1120 ± 91%  interrupts.CPU44.RES:Rescheduling_interrupts
>     172794 ±114%     -99.5%     838.33 ±  4%  interrupts.CPU45.CAL:Function_call_interrupts
>      89408 ±111%     -99.6%     326.00 ± 23%  interrupts.CPU45.RES:Rescheduling_interrupts
>      34858 ± 59%     -97.3%     947.00 ± 13%  interrupts.CPU46.CAL:Function_call_interrupts
>      22612 ± 53%     -87.6%       2794 ± 27%  interrupts.CPU46.RES:Rescheduling_interrupts
>      41349 ± 57%     -97.6%     996.67 ± 10%  interrupts.CPU47.CAL:Function_call_interrupts
>      27498 ± 54%     -93.7%       1725 ± 89%  interrupts.CPU47.RES:Rescheduling_interrupts
>      45388 ± 61%     -97.5%       1141 ± 57%  interrupts.CPU5.CAL:Function_call_interrupts
>      32500 ± 63%     -96.7%       1063 ± 95%  interrupts.CPU5.RES:Rescheduling_interrupts
>     534772 ±105%     -99.8%     843.33 ±  3%  interrupts.CPU6.CAL:Function_call_interrupts
>     259978 ± 94%     -98.5%       4008 ±  4%  interrupts.CPU6.RES:Rescheduling_interrupts
>     589336 ± 71%     -99.8%       1042 ± 25%  interrupts.CPU7.CAL:Function_call_interrupts
>     311886 ± 69%     -99.4%       1966 ± 66%  interrupts.CPU7.RES:Rescheduling_interrupts
>     363153 ± 95%     -99.8%     858.00 ±  5%  interrupts.CPU8.CAL:Function_call_interrupts
>     190915 ± 83%     -98.1%       3681 ± 94%  interrupts.CPU8.RES:Rescheduling_interrupts
>      78806 ± 46%     -98.9%     905.33 ±  7%  interrupts.CPU9.CAL:Function_call_interrupts
>      47384 ± 29%     -95.9%       1963 ± 58%  interrupts.CPU9.RES:Rescheduling_interrupts
>    9209745 ± 15%     -99.0%      95188 ±  9%  interrupts.RES:Rescheduling_interrupts
>      20.15 ±  4%     -14.2        5.98 ± 15%  perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
>      15.92 ±  4%     -11.9        3.98 ±  7%  perf-profile.calltrace.cycles-pp.signal_wake_up_state.__send_signal.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo
>      15.71 ±  4%     -11.8        3.95 ±  7%  perf-profile.calltrace.cycles-pp.try_to_wake_up.signal_wake_up_state.__send_signal.do_send_sig_info.kill_pid_info
>      13.68 ±  6%     -10.8        2.93 ± 10%  perf-profile.calltrace.cycles-pp.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
>      12.23 ± 26%     -10.4        1.80 ± 38%  perf-profile.calltrace.cycles-pp.group_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64
>      11.55 ± 27%     -10.0        1.56 ± 39%  perf-profile.calltrace.cycles-pp.security_task_kill.group_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo
>      11.33 ± 27%      -9.8        1.49 ± 40%  perf-profile.calltrace.cycles-pp.apparmor_task_kill.security_task_kill.group_send_sig_info.kill_pid_info.do_rt_sigqueueinfo
>      11.78 ±  5%      -9.3        2.52 ±  9%  perf-profile.calltrace.cycles-pp.schedule.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64
>      11.53 ±  5%      -9.1        2.47 ±  9%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait
>       8.27 ± 13%      -6.3        1.99 ±  7%  perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
>       7.32 ±  2%      -4.9        2.43 ± 25%  perf-profile.calltrace.cycles-pp.syscall_return_via_sysret
>       6.20 ± 13%      -4.7        1.53 ± 10%  perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
>       5.94 ± 13%      -4.5        1.46 ± 10%  perf-profile.calltrace.cycles-pp.__sched_text_start.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode.entry_SYSCALL_64_after_hwframe
>       6.32 ±  8%      -4.3        2.07 ±  6%  perf-profile.calltrace.cycles-pp.select_task_rq_fair.try_to_wake_up.signal_wake_up_state.__send_signal.do_send_sig_info
>       4.57 ± 30%      -4.1        0.44 ± 74%  perf-profile.calltrace.cycles-pp.aa_get_task_label.apparmor_task_kill.security_task_kill.group_send_sig_info.kill_pid_info
>       5.55 ±  8%      -3.7        1.90 ±  6%  perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.try_to_wake_up.signal_wake_up_state.__send_signal
>       4.43 ±  7%      -3.4        0.99 ±  6%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.signal_wake_up_state.__send_signal.do_send_sig_info
>       4.29 ±  7%      -3.3        0.95 ±  6%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.signal_wake_up_state.__send_signal
>       4.19 ±  6%      -3.3        0.92 ±  8%  perf-profile.calltrace.cycles-pp.dequeue_task_fair.__sched_text_start.schedule.schedule_hrtimeout_range_clock.do_sigtimedwait
>       4.60            -2.9        1.69 ± 28%  perf-profile.calltrace.cycles-pp.__entry_text_start
>       1.93 ± 16%      -1.6        0.37 ± 70%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__sched_text_start.schedule.exit_to_user_mode_prepare.syscall_exit_to_user_mode
>       2.40 ±  6%      -1.5        0.86 ±  6%  perf-profile.calltrace.cycles-pp.available_idle_cpu.select_idle_sibling.select_task_rq_fair.try_to_wake_up.signal_wake_up_state
>       0.00            +0.7        0.67        perf-profile.calltrace.cycles-pp.inc_rlimit_ucounts_and_test.__sigqueue_alloc.__send_signal.do_send_sig_info.kill_pid_info
>       0.67 ± 16%      +2.4        3.11 ± 53%  perf-profile.calltrace.cycles-pp.__lock_task_sighand.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo
>       0.42 ± 71%      +2.7        3.08 ± 53%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__lock_task_sighand.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo
>       0.00            +3.0        2.98 ± 54%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__lock_task_sighand.do_send_sig_info.kill_pid_info
>       0.00            +3.2        3.23 ± 51%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64
>       0.00            +3.3        3.29 ± 51%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
>      35.47 ±  7%      +9.5       45.02        perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe
>      34.17 ±  7%     +10.4       44.54        perf-profile.calltrace.cycles-pp.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe
>      33.33 ±  7%     +10.9       44.22        perf-profile.calltrace.cycles-pp.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe
>      81.67           +12.6       94.25        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
>      19.16 ±  4%     +19.5       38.64 ±  8%  perf-profile.calltrace.cycles-pp.__send_signal.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo
>      21.31 ±  6%     +20.5       41.77 ±  4%  perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
>      20.83 ±  3%     +21.5       42.33 ±  3%  perf-profile.calltrace.cycles-pp.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64
>      19.46 ±  5%     +21.9       41.31 ±  4%  perf-profile.calltrace.cycles-pp.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
>      59.72 ±  2%     +28.1       87.78 ±  2%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
>       3.78 ±  8%     +30.8       34.58 ±  9%  perf-profile.calltrace.cycles-pp.dequeue_signal.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe
>       3.07 ±  8%     +31.3       34.40 ±  9%  perf-profile.calltrace.cycles-pp.__dequeue_signal.dequeue_signal.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64
>       1.74 ±  9%     +32.1       33.86 ±  9%  perf-profile.calltrace.cycles-pp.__sigqueue_free.__dequeue_signal.dequeue_signal.do_sigtimedwait.__x64_sys_rt_sigtimedwait
>       1.94 ±  8%     +32.5       34.40 ±  8%  perf-profile.calltrace.cycles-pp.__sigqueue_alloc.__send_signal.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo
>       0.00           +32.8       32.81 ±  9%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.put_ucounts.__sigqueue_free.__dequeue_signal
>       0.00           +33.0       33.02 ±  8%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_ucounts.__sigqueue_alloc.__send_signal
>       0.00           +33.2       33.22 ±  9%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.put_ucounts.__sigqueue_free.__dequeue_signal.dequeue_signal
>       0.00           +33.3       33.32 ±  9%  perf-profile.calltrace.cycles-pp.put_ucounts.__sigqueue_free.__dequeue_signal.dequeue_signal.do_sigtimedwait
>       0.00           +33.5       33.48 ±  8%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_ucounts.__sigqueue_alloc.__send_signal.do_send_sig_info
>       0.00           +33.6       33.57 ±  8%  perf-profile.calltrace.cycles-pp.get_ucounts.__sigqueue_alloc.__send_signal.do_send_sig_info.kill_pid_info
>      20.22 ±  4%     -14.2        6.00 ± 15%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode
>      18.14 ±  8%     -14.1        4.06 ± 10%  perf-profile.children.cycles-pp.schedule
>      17.92 ±  7%     -14.0        3.96 ±  9%  perf-profile.children.cycles-pp.__sched_text_start
>      15.92 ±  4%     -11.9        3.98 ±  7%  perf-profile.children.cycles-pp.signal_wake_up_state
>      15.73 ±  4%     -11.8        3.96 ±  7%  perf-profile.children.cycles-pp.try_to_wake_up
>      13.71 ±  6%     -10.8        2.93 ± 10%  perf-profile.children.cycles-pp.schedule_hrtimeout_range_clock
>      12.24 ± 26%     -10.4        1.80 ± 38%  perf-profile.children.cycles-pp.group_send_sig_info
>      11.56 ± 27%     -10.0        1.56 ± 39%  perf-profile.children.cycles-pp.security_task_kill
>      11.35 ± 27%      -9.9        1.49 ± 40%  perf-profile.children.cycles-pp.apparmor_task_kill
>       8.45 ± 12%      -6.5        2.00 ±  7%  perf-profile.children.cycles-pp.exit_to_user_mode_prepare
>       8.04            -5.3        2.71 ± 25%  perf-profile.children.cycles-pp.syscall_return_via_sysret
>       6.34 ±  8%      -4.3        2.07 ±  6%  perf-profile.children.cycles-pp.select_task_rq_fair
>       4.58 ± 30%      -4.0        0.55 ± 36%  perf-profile.children.cycles-pp.aa_get_task_label
>       6.03            -4.0        2.05 ± 28%  perf-profile.children.cycles-pp.__entry_text_start
>       4.70 ±  6%      -3.7        0.99 ±  6%  perf-profile.children.cycles-pp.ttwu_do_activate
>       5.63 ±  8%      -3.7        1.93 ±  6%  perf-profile.children.cycles-pp.select_idle_sibling
>       4.57 ±  6%      -3.6        0.96 ±  6%  perf-profile.children.cycles-pp.enqueue_task_fair
>       4.23 ±  7%      -3.3        0.93 ±  8%  perf-profile.children.cycles-pp.dequeue_task_fair
>       3.78 ± 12%      -2.9        0.91 ± 11%  perf-profile.children.cycles-pp.pick_next_task_fair
>       3.44 ±  9%      -2.7        0.78 ±  9%  perf-profile.children.cycles-pp.switch_mm_irqs_off
>       3.07 ± 10%      -2.4        0.68 ±  9%  perf-profile.children.cycles-pp.update_load_avg
>       2.96 ±  3%      -2.3        0.69 ±  9%  perf-profile.children.cycles-pp.update_curr
>       2.35 ±  3%      -1.9        0.48 ±  6%  perf-profile.children.cycles-pp.enqueue_entity
>       2.26 ± 13%      -1.7        0.56 ± 10%  perf-profile.children.cycles-pp.load_new_mm_cr3
>       2.44 ±  6%      -1.6        0.86 ±  5%  perf-profile.children.cycles-pp.available_idle_cpu
>       1.82 ±  4%      -1.4        0.41 ±  7%  perf-profile.children.cycles-pp.dequeue_entity
>       1.93 ±  3%      -1.4        0.52 ± 32%  perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
>       1.63 ± 10%      -1.3        0.28 ±  8%  perf-profile.children.cycles-pp._raw_spin_lock
>       1.59 ± 13%      -1.3        0.30 ± 10%  perf-profile.children.cycles-pp.switch_fpu_return
>       1.63 ± 11%      -1.2        0.39 ± 10%  perf-profile.children.cycles-pp.reweight_entity
>       1.47 ±  7%      -1.2        0.31 ± 11%  perf-profile.children.cycles-pp.__switch_to
>       1.25 ± 12%      -1.0        0.27 ± 13%  perf-profile.children.cycles-pp.__switch_to_asm
>       1.21 ± 10%      -0.9        0.27 ± 13%  perf-profile.children.cycles-pp.set_next_entity
>       1.07 ±  7%      -0.9        0.13 ± 25%  perf-profile.children.cycles-pp.finish_task_switch
>       1.30 ±  4%      -0.9        0.41 ± 10%  perf-profile.children.cycles-pp._copy_from_user
>       1.11 ±  7%      -0.8        0.27 ± 12%  perf-profile.children.cycles-pp.update_rq_clock
>       1.04 ±  4%      -0.8        0.24 ± 15%  perf-profile.children.cycles-pp.ttwu_do_wakeup
>       0.99 ± 13%      -0.8        0.21 ± 14%  perf-profile.children.cycles-pp.hrtimer_start_range_ns
>       0.98 ±  4%      -0.8        0.22 ± 13%  perf-profile.children.cycles-pp.check_preempt_curr
>       1.03            -0.8        0.28 ±  4%  perf-profile.children.cycles-pp.recalc_sigpending
>       0.97 ± 12%      -0.7        0.22 ± 10%  perf-profile.children.cycles-pp.__update_load_avg_se
>       0.90 ±  8%      -0.7        0.18 ± 12%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
>       0.95 ± 10%      -0.7        0.23 ± 10%  perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
>       0.94 ± 33%      -0.7        0.23 ± 40%  perf-profile.children.cycles-pp.aa_may_signal
>       0.94 ±  6%      -0.7        0.23 ±  2%  perf-profile.children.cycles-pp.copy_siginfo_to_user
>       0.95 ±  6%      -0.7        0.27 ±  9%  perf-profile.children.cycles-pp.__might_fault
>       0.85 ±  5%      -0.7        0.19 ± 11%  perf-profile.children.cycles-pp.check_preempt_wakeup
>       0.83 ±  9%      -0.6        0.19 ±  7%  perf-profile.children.cycles-pp.___perf_sw_event
>       1.00 ±  2%      -0.6        0.36 ± 20%  perf-profile.children.cycles-pp.__copy_siginfo_from_user
>       0.79 ± 17%      -0.6        0.19 ± 11%  perf-profile.children.cycles-pp.put_prev_entity
>       0.80 ± 12%      -0.6        0.20 ±  8%  perf-profile.children.cycles-pp.pick_next_entity
>       0.82 ±  2%      -0.5        0.29 ± 24%  perf-profile.children.cycles-pp.__x64_sys_getpid
>       0.68 ±  2%      -0.5        0.16 ±  3%  perf-profile.children.cycles-pp.update_cfs_group
>       0.87 ± 17%      -0.5        0.38        perf-profile.children.cycles-pp.asm_call_sysvec_on_stack
>       0.53 ±  6%      -0.5        0.06 ± 16%  perf-profile.children.cycles-pp.complete_signal
>       0.58 ±  6%      -0.5        0.12 ± 10%  perf-profile.children.cycles-pp.sched_clock_cpu
>       0.55 ±  5%      -0.4        0.10 ± 12%  perf-profile.children.cycles-pp.hrtimer_try_to_cancel
>       0.79 ± 12%      -0.4        0.35 ±  7%  perf-profile.children.cycles-pp.cpumask_next_wrap
>       0.62 ± 18%      -0.4        0.19 ± 10%  perf-profile.children.cycles-pp._find_next_bit
>       0.71 ±  2%      -0.4        0.28 ± 34%  perf-profile.children.cycles-pp.__task_pid_nr_ns
>       0.62 ±  3%      -0.4        0.21 ± 30%  perf-profile.children.cycles-pp.__x64_sys_getuid
>       0.53 ± 14%      -0.4        0.14 ± 17%  perf-profile.children.cycles-pp.__calc_delta
>       0.59 ±  8%      -0.4        0.20 ± 32%  perf-profile.children.cycles-pp.check_kill_permission
>       0.50 ±  6%      -0.4        0.11 ± 11%  perf-profile.children.cycles-pp.sched_clock
>       0.50 ± 13%      -0.4        0.12 ± 10%  perf-profile.children.cycles-pp.copy_fpregs_to_fpstate
>       0.47 ±  6%      -0.4        0.09 ± 18%  perf-profile.children.cycles-pp.perf_tp_event
>       0.48 ±  6%      -0.4        0.10 ±  9%  perf-profile.children.cycles-pp.native_sched_clock
>       0.48 ±  3%      -0.4        0.11 ±  4%  perf-profile.children.cycles-pp.___might_sleep
>       0.74 ± 12%      -0.4        0.39        perf-profile.children.cycles-pp.kmem_cache_free
>       0.45 ±  8%      -0.3        0.13 ± 13%  perf-profile.children.cycles-pp._copy_to_user
>       0.36 ±  5%      -0.3        0.06 ±  8%  perf-profile.children.cycles-pp.__set_task_blocked
>       0.35 ± 22%      -0.3        0.07 ± 11%  perf-profile.children.cycles-pp.cpumask_next
>       0.38 ±  7%      -0.3        0.10 ± 14%  perf-profile.children.cycles-pp.ktime_get
>       0.46 ± 10%      -0.3        0.19 ± 36%  perf-profile.children.cycles-pp.__radix_tree_lookup
>       0.64 ±  4%      -0.3        0.37 ±  6%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
>       0.33 ± 16%      -0.3        0.06 ± 14%  perf-profile.children.cycles-pp.cpuacct_charge
>       0.53            -0.3        0.26 ± 33%  perf-profile.children.cycles-pp.send_signal
>       0.33 ± 16%      -0.3        0.07 ± 12%  perf-profile.children.cycles-pp.__wrgsbase_inactive
>       0.34 ±  8%      -0.2        0.09 ±  5%  perf-profile.children.cycles-pp.update_min_vruntime
>       0.39 ±  3%      -0.2        0.15 ± 12%  perf-profile.children.cycles-pp.copy_user_generic_unrolled
>       0.32 ±  6%      -0.2        0.07 ±  6%  perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
>       0.28 ±  2%      -0.2        0.04 ± 71%  perf-profile.children.cycles-pp.lock_hrtimer_base
>       0.30 ±  9%      -0.2        0.06 ±  7%  perf-profile.children.cycles-pp.set_next_buddy
>       0.35 ±  5%      -0.2        0.11 ± 36%  perf-profile.children.cycles-pp.syscall_enter_from_user_mode
>       0.30 ±  5%      -0.2        0.07 ± 17%  perf-profile.children.cycles-pp.rb_erase
>       0.32 ±  2%      -0.2        0.10 ±  9%  perf-profile.children.cycles-pp.__x86_retpoline_rax
>       0.30 ±  5%      -0.2        0.08 ± 14%  perf-profile.children.cycles-pp.__might_sleep
>       0.29 ±  7%      -0.2        0.07 ± 12%  perf-profile.children.cycles-pp.__clear_user
>       0.31 ± 14%      -0.2        0.09 ±  9%  perf-profile.children.cycles-pp.clear_buddies
>       0.25 ±  3%      -0.2        0.05        perf-profile.children.cycles-pp.recalc_sigpending_tsk
>       0.29 ± 12%      -0.2        0.09 ± 22%  perf-profile.children.cycles-pp.prepare_signal
>       0.30 ±  4%      -0.2        0.11 ± 34%  perf-profile.children.cycles-pp.from_kuid_munged
>       0.27 ±  4%      -0.2        0.09 ± 36%  perf-profile.children.cycles-pp.map_id_up
>       0.21 ± 21%      -0.2        0.03 ± 70%  perf-profile.children.cycles-pp.enqueue_hrtimer
>       0.22 ± 13%      -0.2        0.06 ±  8%  perf-profile.children.cycles-pp.get_timespec64
>       0.22 ± 17%      -0.2        0.06 ± 86%  perf-profile.children.cycles-pp.__cgroup_account_cputime
>       0.20 ±  4%      -0.2        0.04 ± 76%  perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare
>       0.20 ±  6%      -0.2        0.05        perf-profile.children.cycles-pp.__list_del_entry_valid
>       0.19 ±  7%      -0.2        0.04 ± 71%  perf-profile.children.cycles-pp.perf_trace_sched_switch
>       0.23 ±  8%      -0.1        0.08 ± 31%  perf-profile.children.cycles-pp.audit_signal_info
>       0.22 ±  5%      -0.1        0.08 ± 16%  perf-profile.children.cycles-pp.rcu_read_unlock_strict
>       0.17 ±  9%      -0.1        0.03 ± 70%  perf-profile.children.cycles-pp.read_tsc
>       0.18 ±  7%      -0.1        0.05 ± 72%  perf-profile.children.cycles-pp.audit_signal_info_syscall
>       0.17 ±  5%      -0.1        0.11 ±  4%  perf-profile.children.cycles-pp.kmem_cache_alloc
>       0.00            +0.5        0.51 ±  3%  perf-profile.children.cycles-pp.dec_rlimit_ucounts
>       0.00            +0.7        0.67 ±  2%  perf-profile.children.cycles-pp.inc_rlimit_ucounts_and_test
>       0.68 ± 16%      +2.4        3.11 ± 53%  perf-profile.children.cycles-pp.__lock_task_sighand
>       0.42 ±  6%      +2.9        3.30 ± 51%  perf-profile.children.cycles-pp._raw_spin_lock_irq
>      35.50 ±  7%      +9.5       45.03        perf-profile.children.cycles-pp.__x64_sys_rt_sigqueueinfo
>      34.19 ±  7%     +10.4       44.55        perf-profile.children.cycles-pp.do_rt_sigqueueinfo
>      33.35 ±  7%     +10.9       44.22        perf-profile.children.cycles-pp.kill_pid_info
>      81.79           +12.5       94.33        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
>      19.21 ±  4%     +19.4       38.65 ±  8%  perf-profile.children.cycles-pp.__send_signal
>      21.36 ±  6%     +20.4       41.78 ±  4%  perf-profile.children.cycles-pp.__x64_sys_rt_sigtimedwait
>      20.86 ±  3%     +21.5       42.34 ±  3%  perf-profile.children.cycles-pp.do_send_sig_info
>      19.50 ±  5%     +21.8       41.33 ±  4%  perf-profile.children.cycles-pp.do_sigtimedwait
>      59.88 ±  2%     +28.0       87.85 ±  2%  perf-profile.children.cycles-pp.do_syscall_64
>       3.81 ±  8%     +30.9       34.66 ±  9%  perf-profile.children.cycles-pp.dequeue_signal
>       3.09 ±  8%     +31.3       34.41 ±  9%  perf-profile.children.cycles-pp.__dequeue_signal
>       1.74 ±  9%     +32.1       33.87 ±  9%  perf-profile.children.cycles-pp.__sigqueue_free
>       1.95 ±  8%     +32.5       34.40 ±  8%  perf-profile.children.cycles-pp.__sigqueue_alloc
>       0.00           +33.3       33.32 ±  9%  perf-profile.children.cycles-pp.put_ucounts
>       0.00           +33.6       33.58 ±  8%  perf-profile.children.cycles-pp.get_ucounts
>       1.07 ± 11%     +68.8       69.87 ±  6%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
>       0.74 ± 39%     +71.3       72.04 ±  4%  perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
>      11.75 ±  2%      -7.8        3.95 ± 26%  perf-profile.self.cycles-pp.syscall_exit_to_user_mode
>       8.02            -5.3        2.70 ± 25%  perf-profile.self.cycles-pp.syscall_return_via_sysret
>       5.78 ± 25%      -5.1        0.71 ± 43%  perf-profile.self.cycles-pp.apparmor_task_kill
>       4.55 ± 30%      -4.0        0.53 ± 34%  perf-profile.self.cycles-pp.aa_get_task_label
>       6.02            -4.0        2.05 ± 28%  perf-profile.self.cycles-pp.__entry_text_start
>       1.79 ±  9%      -1.7        0.06 ± 13%  perf-profile.self.cycles-pp.__sigqueue_alloc
>       2.22 ±  9%      -1.7        0.50 ±  9%  perf-profile.self.cycles-pp.__sched_text_start
>       2.25 ± 13%      -1.7        0.55 ± 11%  perf-profile.self.cycles-pp.load_new_mm_cr3
>       2.40 ±  6%      -1.5        0.85 ±  6%  perf-profile.self.cycles-pp.available_idle_cpu
>       1.57 ± 12%      -1.3        0.30 ± 10%  perf-profile.self.cycles-pp.switch_fpu_return
>       1.74 ±  6%      -1.2        0.50 ± 11%  perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
>       1.79 ±  8%      -1.2        0.60 ±  6%  perf-profile.self.cycles-pp.select_idle_sibling
>       1.41 ±  7%      -1.1        0.30 ± 11%  perf-profile.self.cycles-pp.__switch_to
>       1.25 ±  3%      -1.0        0.27 ±  8%  perf-profile.self.cycles-pp.update_curr
>       1.23 ± 12%      -1.0        0.27 ± 13%  perf-profile.self.cycles-pp.__switch_to_asm
>       1.17 ±  7%      -0.9        0.23 ±  9%  perf-profile.self.cycles-pp.update_load_avg
>       1.15 ±  2%      -0.9        0.21 ±  5%  perf-profile.self.cycles-pp.switch_mm_irqs_off
>       1.13 ±  3%      -0.8        0.28 ±  8%  perf-profile.self.cycles-pp._raw_spin_lock
>       0.94 ± 13%      -0.8        0.14 ±  9%  perf-profile.self.cycles-pp.try_to_wake_up
>       1.06            -0.7        0.32 ± 28%  perf-profile.self.cycles-pp.do_syscall_64
>       0.93 ± 12%      -0.7        0.22 ± 12%  perf-profile.self.cycles-pp.__update_load_avg_se
>       0.93 ± 33%      -0.7        0.23 ± 40%  perf-profile.self.cycles-pp.aa_may_signal
>       0.90 ± 10%      -0.7        0.22 ±  9%  perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
>       0.77            -0.6        0.15 ±  5%  perf-profile.self.cycles-pp.recalc_sigpending
>       0.77 ± 13%      -0.6        0.20 ±  7%  perf-profile.self.cycles-pp.reweight_entity
>       0.74 ±  7%      -0.6        0.19 ± 11%  perf-profile.self.cycles-pp.update_rq_clock
>       0.67 ±  7%      -0.5        0.14 ±  6%  perf-profile.self.cycles-pp.___perf_sw_event
>       0.63 ±  7%      -0.5        0.11 ± 11%  perf-profile.self.cycles-pp.enqueue_entity
>       0.67 ±  3%      -0.5        0.15 ±  3%  perf-profile.self.cycles-pp.update_cfs_group
>       0.66 ±  9%      -0.5        0.16 ± 12%  perf-profile.self.cycles-pp.pick_next_task_fair
>       0.61 ± 13%      -0.5        0.12 ±  4%  perf-profile.self.cycles-pp.enqueue_task_fair
>       0.60 ±  5%      -0.5        0.11 ± 14%  perf-profile.self.cycles-pp.dequeue_task_fair
>       0.62 ±  8%      -0.5        0.14 ±  5%  perf-profile.self.cycles-pp.select_task_rq_fair
>       0.63 ±  6%      -0.5        0.16 ± 10%  perf-profile.self.cycles-pp.do_sigtimedwait
>       0.61 ± 18%      -0.4        0.19 ± 10%  perf-profile.self.cycles-pp._find_next_bit
>       0.68 ±  3%      -0.4        0.27 ± 33%  perf-profile.self.cycles-pp.__task_pid_nr_ns
>       0.54 ±  3%      -0.4        0.14 ±  6%  perf-profile.self.cycles-pp.__dequeue_signal
>       0.51            -0.4        0.12 ± 20%  perf-profile.self.cycles-pp.__send_signal
>       0.53 ± 14%      -0.4        0.14 ± 15%  perf-profile.self.cycles-pp.__calc_delta
>       0.49 ± 13%      -0.4        0.12 ± 10%  perf-profile.self.cycles-pp.copy_fpregs_to_fpstate
>       0.46 ±  7%      -0.4        0.09 ± 10%  perf-profile.self.cycles-pp.native_sched_clock
>       0.46 ±  3%      -0.4        0.10 ±  4%  perf-profile.self.cycles-pp.___might_sleep
>       0.50 ±  7%      -0.3        0.16 ± 23%  perf-profile.self.cycles-pp.exit_to_user_mode_prepare
>       0.50 ±  3%      -0.3        0.17 ± 29%  perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
>       0.44 ± 14%      -0.3        0.10 ± 16%  perf-profile.self.cycles-pp.schedule
>       0.42 ±  5%      -0.3        0.09 ±  9%  perf-profile.self.cycles-pp.finish_task_switch
>       0.43 ± 11%      -0.3        0.10 ±  9%  perf-profile.self.cycles-pp.pick_next_entity
>       0.41 ± 10%      -0.3        0.08 ±  5%  perf-profile.self.cycles-pp.__x64_sys_rt_sigtimedwait
>       0.44 ±  6%      -0.3        0.13 ±  6%  perf-profile.self.cycles-pp._copy_from_user
>       0.50 ±  3%      -0.3        0.20 ± 11%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
>       0.63 ± 20%      -0.3        0.35 ±  6%  perf-profile.self.cycles-pp.kmem_cache_free
>       0.37 ±  3%      -0.3        0.10 ± 12%  perf-profile.self.cycles-pp.check_preempt_wakeup
>       0.37 ±  7%      -0.3        0.10 ±  4%  perf-profile.self.cycles-pp.dequeue_entity
>       0.33 ± 15%      -0.3        0.06 ± 13%  perf-profile.self.cycles-pp.cpuacct_charge
>       0.46 ± 11%      -0.3        0.19 ± 36%  perf-profile.self.cycles-pp.__radix_tree_lookup
>       0.34 ±  3%      -0.3        0.07 ± 11%  perf-profile.self.cycles-pp._raw_spin_lock_irq
>       0.29 ±  7%      -0.3        0.04 ± 71%  perf-profile.self.cycles-pp.perf_tp_event
>       0.36 ± 11%      -0.3        0.10 ±  4%  perf-profile.self.cycles-pp.__might_fault
>       0.32 ±  7%      -0.2        0.08 ±  5%  perf-profile.self.cycles-pp.update_min_vruntime
>       0.31 ± 17%      -0.2        0.07 ± 12%  perf-profile.self.cycles-pp.__wrgsbase_inactive
>       0.37 ±  2%      -0.2        0.14 ± 13%  perf-profile.self.cycles-pp.copy_user_generic_unrolled
>       0.30 ±  4%      -0.2        0.07 ±  7%  perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime
>       0.26 ±  8%      -0.2        0.04 ± 71%  perf-profile.self.cycles-pp.set_next_buddy
>       0.29 ±  5%      -0.2        0.07 ± 17%  perf-profile.self.cycles-pp.rb_erase
>       0.28 ± 11%      -0.2        0.06 ± 14%  perf-profile.self.cycles-pp.set_next_entity
>       0.30 ±  4%      -0.2        0.09 ± 30%  perf-profile.self.cycles-pp.__x64_sys_getuid
>       0.39 ± 13%      -0.2        0.18 ±  9%  perf-profile.self.cycles-pp.cpumask_next_wrap
>       0.27 ±  5%      -0.2        0.07 ± 17%  perf-profile.self.cycles-pp.__might_sleep
>       0.31 ±  3%      -0.2        0.12 ± 28%  perf-profile.self.cycles-pp.__x64_sys_rt_sigqueueinfo
>       0.24            -0.2        0.05        perf-profile.self.cycles-pp.recalc_sigpending_tsk
>       0.28 ±  5%      -0.2        0.09 ± 36%  perf-profile.self.cycles-pp.syscall_enter_from_user_mode
>       0.27 ± 12%      -0.2        0.08 ± 24%  perf-profile.self.cycles-pp.prepare_signal
>       0.27 ± 13%      -0.2        0.08 ± 10%  perf-profile.self.cycles-pp.clear_buddies
>       0.26 ±  3%      -0.2        0.08 ± 12%  perf-profile.self.cycles-pp.__x86_retpoline_rax
>       0.21 ± 17%      -0.2        0.04 ± 71%  perf-profile.self.cycles-pp.put_prev_entity
>       0.26 ±  4%      -0.2        0.09 ± 36%  perf-profile.self.cycles-pp.map_id_up
>       0.23 ± 13%      -0.2        0.07 ± 11%  perf-profile.self.cycles-pp.schedule_hrtimeout_range_clock
>       0.19 ±  8%      -0.2        0.04 ± 70%  perf-profile.self.cycles-pp.ktime_get
>       0.19 ±  9%      -0.1        0.04 ± 71%  perf-profile.self.cycles-pp.perf_trace_sched_switch
>       0.18 ±  4%      -0.1        0.03 ± 70%  perf-profile.self.cycles-pp.__list_del_entry_valid
>       0.19 ±  8%      -0.1        0.05 ± 78%  perf-profile.self.cycles-pp.check_kill_permission
>       0.26            -0.1        0.13 ± 26%  perf-profile.self.cycles-pp.send_signal
>       0.21 ±  4%      -0.1        0.08 ± 26%  perf-profile.self.cycles-pp.__x64_sys_getpid
>       0.20 ±  2%      -0.1        0.06 ± 19%  perf-profile.self.cycles-pp.security_task_kill
>       0.17 ±  4%      -0.1        0.05 ± 78%  perf-profile.self.cycles-pp.kill_pid_info
>       0.16 ±  8%      -0.1        0.04 ± 71%  perf-profile.self.cycles-pp.rcu_read_unlock_strict
>       0.16 ± 10%      -0.1        0.05 ± 72%  perf-profile.self.cycles-pp.audit_signal_info_syscall
>       0.24 ±  8%      -0.1        0.14 ±  3%  perf-profile.self.cycles-pp.dequeue_signal
>       0.12 ±  4%      -0.1        0.04 ± 71%  perf-profile.self.cycles-pp.do_rt_sigqueueinfo
>       0.12 ± 11%      -0.1        0.04 ± 70%  perf-profile.self.cycles-pp.do_send_sig_info
>       0.15 ±  6%      -0.0        0.10        perf-profile.self.cycles-pp.kmem_cache_alloc
>       0.00            +0.1        0.07 ± 11%  perf-profile.self.cycles-pp.get_ucounts
>       0.00            +0.1        0.07 ±  6%  perf-profile.self.cycles-pp.put_ucounts
>       0.88 ±  7%      +0.2        1.04        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
>       0.00            +0.5        0.51 ±  3%  perf-profile.self.cycles-pp.dec_rlimit_ucounts
>       0.00            +0.7        0.67        perf-profile.self.cycles-pp.inc_rlimit_ucounts_and_test
>       0.74 ± 39%     +71.3       72.04 ±  4%  perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
>
>
>
>
>
> Disclaimer:
> Results have been estimated based on internal Intel analysis and are provided
> for informational purposes only. Any difference in system hardware or software
> design or configuration may affect actual performance.
>
>
> Thanks,
> Oliver Sang

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.