Openwall GNU/*/Linux - a small security-enhanced Linux distro for servers
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Wed, 9 Jan 2019 10:26:30 +0800
From: Jason Yan <yanaijie@...wei.com>
To: Kees Cook <keescook@...omium.org>, Jann Horn <jannh@...gle.com>
CC: Kernel Hardening <kernel-hardening@...ts.openwall.com>,
	<zhaohongjiang@...wei.com>, <miaoxie@...wei.com>, Li Bin
	<huawei.libin@...wei.com>, Wei Yongjun <weiyongjun1@...wei.com>
Subject: Re: [PATCH] usercopy: skip the check if not a real usercopy

Hi all,

On 2019/1/9 7:54, Kees Cook wrote:
> I would think we'd still want to be performing bounds-checking even in
> the kernel-to-kernel case. It seems like there is some other issue
> here? Why would the check stall?
>
> Can you find out from your build what your top-of-dump line this
> resolves to? (My various kernel version builds don't share the same
> function size, so this doesn't resolve for me...)
>
> $ ./scripts/faddr2line vmlinux __check_object_size+0x1f1/0x460
>
> Maybe it's getting stuck in loop doing stack frame walking? I'd expect
> that to show up in the backtrace though (since it's noinline).


It's very easy to reproduce in qemu using my config with v4.20. Please 
refer to the attachment.

I did some debug and found that check_object_size() did not stuck but
check_object_size() sometimes takes more than 30 milliseconds, and 
ftrace will call __probe_kernel_write() thousands of times, which makes 
the whole process stuck for more than 20 seconds.

[root@...alhost ~]# cat functiontracer.sh
ping 127.0.0.1 -i 0.01 > /dev/null &
sleep 1
echo function > /sys/kernel/debug/tracing/current_tracer
[root@...alhost ~]#
[root@...alhost ~]#
[root@...alhost ~]# sh functiontracer.sh
[   64.641771] hrtimer: interrupt took 2996981 ns
[   91.605642] watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [sh:363]
[   91.634624] Modules linked in:
[   91.642420] irq event stamp: 644434
[   91.656045] hardirqs last  enabled at (644433): [<ffffffff810074b3>] 
trace_hardirqs_on_thunk+0x1a/0x1c
[   91.685643] hardirqs last disabled at (644434): [<ffffffff810074cf>] 
trace_hardirqs_off_thunk+0x1a/0x1c
[   91.704044] softirqs last  enabled at (31910): [<ffffffff83a00ad1>] 
__do_softirq+0xad1/0xec6
[   91.719178] softirqs last disabled at (31889): [<ffffffff8125c61e>] 
irq_exit+0x36e/0x410
[   91.735142] CPU: 4 PID: 363 Comm: sh Not tainted 
4.20.0-514.55.6.9.x86_64 #73
[   91.749510] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS 1.10.2-2.fc27 04/01/2014
[   91.763355] RIP: 0010:__check_object_size+0x5/0x460
[   91.772696] Code: 94 06 01 e9 24 ff ff ff e8 58 44 b9 ff 48 c7 c5 e0 
61 d5 83 48 83 05 f9 d3 94 06 01 e9 19 ff ff ff 0f 1f 40 00 cc 8b 93 5b 
1e <41> 57 41 56 41 55 41 54 41 89 d5 55 53 48 89 f5 48 89 fb 48 83 ec
[   91.803047] RSP: 0018:ffff8880ba4cf948 EFLAGS: 00000202 ORIG_RAX: 
ffffffffffffff13
[   91.815877] RAX: 0000000000000005 RBX: ffffffff8727fce0 RCX: 
ffffffff817c3e05
[   91.827660] RDX: 0000000000000001 RSI: 0000000000000001 RDI: 
ffffffff8727fce0
[   91.838595] RBP: 0000000000000001 R08: 0000000000000000 R09: 
fffffbfff0e4ff9c
[   91.851927] R10: 0000000000000001 R11: fffffbfff0e4ff9c R12: 
00007ffffffff000
[   91.863474] R13: ffff8880014d6b90 R14: ffff8880ba41e000 R15: 
0000000000000001
[   91.876266] FS:  00007fc0d1974740(0000) GS:ffff88810ed00000(0000) 
knlGS:0000000000000000
[   91.889452] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   91.898713] CR2: 0000563a08d928d0 CR3: 00000000ba45c000 CR4: 
00000000000006e0
[   91.911489] Call Trace:
[   91.916210]  __probe_kernel_write+0x195/0x390
[   91.925149]  ftrace_write+0x67/0xa0
[   91.931678]  ftrace_replace_code+0x5b3/0xa30
[   91.941139]  ? ftrace_int3_handler+0x100/0x100
[   91.951361]  ftrace_modify_all_code+0x1f6/0x2e0
[   91.959368]  ? function_stack_trace_call+0x340/0x340
[   91.967523]  arch_ftrace_update_code+0x3a/0x70
[   91.976491]  ftrace_run_update_code+0x35/0xf0
[   91.984167]  ftrace_startup_enable+0x7a/0xa0
[   91.992115]  ftrace_startup+0x405/0x6a0
[   92.000624]  register_ftrace_function+0x76/0x150
[   92.009144]  function_trace_init+0x1bb/0x250
[   92.018110]  tracing_set_tracer+0x4af/0xa10
[   92.026486]  tracing_set_trace_write+0x40e/0x660
[   92.038413]  ? tracing_set_tracer+0xa10/0xa10
[   92.046526]  ? kasan_check_read+0x1d/0x30
[   92.053576]  ? do_raw_spin_unlock+0x6c/0x300
[   92.061467]  ? _raw_spin_unlock+0x44/0x70
[   92.070397]  ? do_anonymous_page+0x6d3/0x1030
[   92.081349]  ? tracing_set_tracer+0xa10/0xa10
[   92.088821]  __vfs_write+0x11b/0x880
[   92.095857]  ? kernel_read+0x150/0x150
[   92.104057]  ? __lock_acquire+0x925/0x1770
[   92.111109]  ? __lock_acquire+0x925/0x1770
[   92.119004]  ? pmd_alloc+0x140/0x140
[   92.125594]  ? __lock_is_held+0xe3/0x1a0
[   92.135659]  ? kasan_check_read+0x1d/0x30
[   92.143276]  ? rcu_read_lock_sched_held+0x1dd/0x210
[   92.153367]  ? rcu_sync_lockdep_assert+0xf0/0x190
[   92.170296]  ? __sb_start_write+0x1b3/0x3e0
[   92.189716]  vfs_write+0x210/0x640
[   92.197563]  ksys_write+0xe6/0x210
[   92.205248]  ? __x64_sys_read+0xe0/0xe0
[   92.212270]  ? trace_hardirqs_on_thunk+0x1a/0x1c
[   92.222093]  ? do_syscall_64+0x98/0x2b0
[   92.228907]  ? do_syscall_64+0x98/0x2b0
[   92.238538]  __x64_sys_write+0x94/0xe0
[   92.245679]  do_syscall_64+0x161/0x2b0
[   92.254452]  entry_SYSCALL_64_after_hwframe+0x49/0xbe
[   92.262656] RIP: 0033:0x7fc0d1a64ed8
[   92.269044] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 
00 f3 0f 1e fa 48 8d 05 45 78 0d 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 
05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55
[   92.298533] RSP: 002b:00007ffdc4882138 EFLAGS: 00000246 ORIG_RAX: 
0000000000000001
[   92.310119] RAX: ffffffffffffffda RBX: 0000000000000009 RCX: 
00007fc0d1a64ed8
[   92.322580] RDX: 0000000000000009 RSI: 0000558f3eb5ff90 RDI: 
0000000000000001
[   92.334280] RBP: 0000558f3eb5ff90 R08: 000000000000000a R09: 
00007fc0d1b37cc0
[   92.346608] R10: 000000000000000a R11: 0000000000000246 R12: 
00007fc0d1b38780
[   92.366134] R13: 0000000000000009 R14: 00007fc0d1b33740 R15: 
0000000000000009

[yanaijie@138 linux]$ ./scripts/faddr2line vmlinux 
__check_object_size+0x5/0x460
__check_object_size+0x5/0x460:
__check_object_size at mm/usercopy.c:254
[yanaijie@138 linux]$

View attachment "functiontrace.config" of type "text/plain" (177261 bytes)

Powered by blists - more mailing lists

Your e-mail address:

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.