[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Thu, 20 Dec 2018 20:25:33 +0100
From: Adam Zabrocki <pi3@....com.pl>
To: lkrg-users@...ts.openwall.com
Subject: Re: insmod: ERROR: could not insert module p_lkrg.ko:
No buffer space available
Hi,
Did you also enable CONFIG_ARCH_HAS_SYSCALL_WRAPPER? Can you share .config and
System.map files from your build?
Additionally, it would be useful if during initialization you enable more
verbose debugging like I have mentioed preivously:
> If LKRG fails initialization I'm suggesting to use at least number 4 for this
> parameter (e.g. # insmod p_lkrg.ko p_init_log_level=4).
> It will give more information about the root of the problem. If debug option
and share LKRG logs.
Thanks,
Ada,
On Thu, Dec 20, 2018 at 11:53:05AM +0100, bryn1u85 wrote:
> @Adam
>
> I recompiled kernel with enabled options KPROBE. After all i compiled lkgr.
> When i do a insomd im getting so weird output in dmesg:
>
> [ 5.115150] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
> [ 42.423441] p_lkrg: loading out-of-tree module taints kernel.
> [ 42.423830] p_lkrg: module verification failed: signature and/or
> required key missing - tainting kernel
> [ 42.425265] [p_lkrg] Loading LKRG...
> [ 247.029037] INFO: task kworker/1:0:18 blocked for more than 120 seconds.
> [ 247.031491] Tainted: G OE 4.19.10 #1
> [ 247.032083] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
>
> > [ 247.032666] kworker/1:0 D 0 18 2 0x80000000
> >
> [ 247.032682] Workqueue: events once_deferred
> [ 247.032683] Call Trace:
> [ 247.032689] ? __schedule+0x2b8/0x780
> [ 247.032691] schedule+0x2d/0x80
> [ 247.032692] schedule_preempt_disabled+0x5/0x10
> [ 247.032694] __mutex_lock.isra.8+0x199/0x4d0
> [ 247.032697] ? ttwu_do_wakeup+0x12/0xe0
> [ 247.032700] static_key_disable_cpuslocked+0x2a/0x70
> [ 247.032702] static_key_disable+0x11/0x20
> [ 247.032703] once_deferred+0x1a/0x30
> [ 247.032705] process_one_work+0x16a/0x2d0
> [ 247.032706] worker_thread+0x44/0x3e0
> [ 247.032708] kthread+0xee/0x120
> [ 247.032709] ? max_active_store+0x80/0x80
> [ 247.032710] ? kthread_bind+0x10/0x10
> [ 247.032712] ret_from_fork+0x35/0x40
> [ 247.032719] INFO: task kworker/1:1:70 blocked for more than 120 seconds.
> [ 247.033305] Tainted: G OE 4.19.10 #1
> [ 247.033897] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 247.034456] kworker/1:1 D 0 70 2 0x80000000
> [ 247.034460] Workqueue: events once_deferred
> [ 247.034460] Call Trace:
> [ 247.034463] ? __schedule+0x2b8/0x780
> [ 247.034464] ? __switch_to_asm+0x40/0x70
> [ 247.034465] schedule+0x2d/0x80
> [ 247.034466] schedule_preempt_disabled+0x5/0x10
> [ 247.034467] __mutex_lock.isra.8+0x199/0x4d0
> [ 247.034468] ? __switch_to_asm+0x34/0x70
> [ 247.034469] ? __switch_to_asm+0x34/0x70
> [ 247.034470] ? __switch_to_asm+0x40/0x70
> [ 247.034472] static_key_disable_cpuslocked+0x2a/0x70
> [ 247.034473] static_key_disable+0x11/0x20
> [ 247.034475] once_deferred+0x1a/0x30
> [ 247.034476] process_one_work+0x16a/0x2d0
> [ 247.034478] worker_thread+0x44/0x3e0
> [ 247.034479] kthread+0xee/0x120
> [ 247.034480] ? max_active_store+0x80/0x80
> [ 247.034481] ? kthread_bind+0x10/0x10
> [ 247.034482] ret_from_fork+0x35/0x40
> [ 247.034485] INFO: task kworker/4:2:186 blocked for more than 120 seconds.
> [ 247.035030] Tainted: G OE 4.19.10 #1
> [ 247.035574] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 247.036139] kworker/4:2 D 0 186 2 0x80000000
> [ 247.036143] Workqueue: events netstamp_clear
> [ 247.036144] Call Trace:
> [ 247.036145] ? __schedule+0x2b8/0x780
> [ 247.036146] ? __switch_to_asm+0x40/0x70
> [ 247.036147] schedule+0x2d/0x80
> [ 247.036148] schedule_preempt_disabled+0x5/0x10
> [ 247.036149] __mutex_lock.isra.8+0x199/0x4d0
> [ 247.036150] ? __switch_to_asm+0x34/0x70
> [ 247.036151] ? __switch_to_asm+0x34/0x70
> [ 247.036153] static_key_enable_cpuslocked+0x28/0x80
> [ 247.036154] static_key_enable+0x11/0x20
> [ 247.036155] process_one_work+0x16a/0x2d0
> [ 247.036156] worker_thread+0x44/0x3e0
> [ 247.036157] kthread+0xee/0x120
> [ 247.036159] ? max_active_store+0x80/0x80
> [ 247.036160] ? kthread_bind+0x10/0x10
> [ 247.036161] ret_from_fork+0x35/0x40
> [ 247.036163] INFO: task kworker/1:2:312 blocked for more than 120 seconds.
> [ 247.036745] Tainted: G OE 4.19.10 #1
> [ 247.037337] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 247.037948] kworker/1:2 D 0 312 2 0x80000000
> [ 247.037951] Workqueue: events once_deferred
> [ 247.037951] Call Trace:
> [ 247.037953] ? __schedule+0x2b8/0x780
> [ 247.037954] ? __switch_to_asm+0x40/0x70
> [ 247.037955] schedule+0x2d/0x80
> [ 247.037956] schedule_preempt_disabled+0x5/0x10
> [ 247.037957] __mutex_lock.isra.8+0x199/0x4d0
> [ 247.037958] ? __switch_to_asm+0x34/0x70
> [ 247.037960] ? __switch_to_asm+0x34/0x70
> [ 247.037961] ? __switch_to_asm+0x40/0x70
> [ 247.037962] static_key_disable_cpuslocked+0x2a/0x70
> [ 247.037964] static_key_disable+0x11/0x20
> [ 247.037965] once_deferred+0x1a/0x30
> [ 247.037966] process_one_work+0x16a/0x2d0
> [ 247.037967] worker_thread+0x44/0x3e0
> [ 247.037968] kthread+0xee/0x120
> [ 247.037969] ? max_active_store+0x80/0x80
> [ 247.037970] ? kthread_bind+0x10/0x10
> [ 247.037971] ret_from_fork+0x35/0x40
> [ 247.037979] INFO: task kworker/6:2:571 blocked for more than 120 seconds.
> [ 247.038583] Tainted: G OE 4.19.10 #1
> [ 247.039205] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 247.039845] kworker/6:2 D 0 571 2 0x80000000
> [ 247.039847] Workqueue: events kprobe_optimizer
> [ 247.039848] Call Trace:
> [ 247.039850] ? __schedule+0x2b8/0x780
> [ 247.039850] schedule+0x2d/0x80
> [ 247.039851] schedule_preempt_disabled+0x5/0x10
> [ 247.039852] __mutex_lock.isra.8+0x199/0x4d0
> [ 247.039855] ? synchronize_sched+0x55/0x80
> [ 247.039856] ? __call_rcu+0x280/0x280
> [ 247.039857] kprobe_optimizer+0x15b/0x290
> [ 247.039858] process_one_work+0x16a/0x2d0
> [ 247.039859] worker_thread+0x44/0x3e0
> [ 247.039860] kthread+0xee/0x120
> [ 247.039861] ? max_active_store+0x80/0x80
> [ 247.039862] ? kthread_bind+0x10/0x10
> [ 247.039863] ret_from_fork+0x35/0x40
> [ 247.039872] INFO: task insmod:5275 blocked for more than 120 seconds.
> [ 247.040500] Tainted: G OE 4.19.10 #1
> [ 247.041178] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 247.041845] insmod D 0 5275 4188 0x80000084
> [ 247.041846] Call Trace:
> [ 247.041848] ? __schedule+0x2b8/0x780
> [ 247.041849] schedule+0x2d/0x80
> [ 247.041850] schedule_preempt_disabled+0x5/0x10
> [ 247.041851] __mutex_lock.isra.8+0x199/0x4d0
> [ 247.041857] ? p_lkrg_fast_hash+0x24f/0x340 [p_lkrg]
> [ 247.041858] ? 0xffffffffc0693000
> [ 247.041861] p_create_database+0x1af/0x440 [p_lkrg]
> [ 247.041862] ? 0xffffffffc0693000
> [ 247.041864] p_lkrg_register+0xea/0x1000 [p_lkrg]
> [ 247.041867] do_one_initcall+0x47/0x1ac
> [ 247.041870] ? do_init_module+0x18/0x1e7
> [ 247.041871] ? kmem_cache_alloc+0x129/0x160
> [ 247.041873] do_init_module+0x50/0x1e7
> [ 247.041874] load_module+0x1801/0x1c80
> [ 247.041876] ? __symbol_put+0x50/0x50
> [ 247.041878] ? security_capable+0x3a/0x50
> [ 247.041879] __do_sys_finit_module+0x94/0xe0
> [ 247.041881] do_syscall_64+0x6f/0x2fc
> [ 247.041883] entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [ 247.041884] RIP: 0033:0x7faf389911c9
> [ 247.041890] Code: Bad RIP value.
> [ 247.041890] RSP: 002b:00007fff03ecca38 EFLAGS: 00000202 ORIG_RAX:
> 0000000000000139
> [ 247.041891] RAX: ffffffffffffffda RBX: 000000000063a240 RCX:
> 00007faf389911c9
> [ 247.041892] RDX: 0000000000000000 RSI: 000000000041a94e RDI:
> 0000000000000003
> [ 247.041892] RBP: 000000000041a94e R08: 0000000000000000 R09:
> 00007fff03eccbd8
> [ 247.041893] R10: 0000000000000003 R11: 0000000000000202 R12:
> 0000000000000000
> [ 247.041894] R13: 000000000063a210 R14: 0000000000000000 R15:
> 0000000000000000
> [ 369.908509] INFO: task kworker/1:0:18 blocked for more than 120 seconds.
> [ 369.910436] Tainted: G OE 4.19.10 #1
> [ 369.911172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 369.912150] kworker/1:0 D 0 18 2 0x80000000
> [ 369.912157] Workqueue: events once_deferred
> [ 369.912161] Call Trace:
> [ 369.912166] ? __schedule+0x2b8/0x780
> [ 369.912168] schedule+0x2d/0x80
> [ 369.912170] schedule_preempt_disabled+0x5/0x10
> [ 369.912171] __mutex_lock.isra.8+0x199/0x4d0
> [ 369.912175] ? ttwu_do_wakeup+0x12/0xe0
> [ 369.912178] static_key_disable_cpuslocked+0x2a/0x70
> [ 369.912179] static_key_disable+0x11/0x20
> [ 369.912181] once_deferred+0x1a/0x30
> [ 369.912183] process_one_work+0x16a/0x2d0
> [ 369.912184] worker_thread+0x44/0x3e0
> [ 369.912187] kthread+0xee/0x120
> [ 369.912188] ? max_active_store+0x80/0x80
> [ 369.912189] ? kthread_bind+0x10/0x10
> [ 369.912191] ret_from_fork+0x35/0x40
> [ 369.912198] INFO: task kworker/1:1:70 blocked for more than 120 seconds.
> [ 369.913554] Tainted: G OE 4.19.10 #1
> [ 369.914396] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 369.915130] kworker/1:1 D 0 70 2 0x80000000
> [ 369.915135] Workqueue: events once_deferred
> [ 369.915136] Call Trace:
> [ 369.915148] ? __schedule+0x2b8/0x780
> [ 369.915151] ? __switch_to_asm+0x40/0x70
> [ 369.915153] schedule+0x2d/0x80
> [ 369.915154] schedule_preempt_disabled+0x5/0x10
> [ 369.915155] __mutex_lock.isra.8+0x199/0x4d0
> [ 369.915157] ? __switch_to_asm+0x34/0x70
> [ 369.915158] ? __switch_to_asm+0x34/0x70
> [ 369.915159] ? __switch_to_asm+0x40/0x70
> [ 369.915161] static_key_disable_cpuslocked+0x2a/0x70
> [ 369.915163] static_key_disable+0x11/0x20
> [ 369.915164] once_deferred+0x1a/0x30
> [ 369.915166] process_one_work+0x16a/0x2d0
> [ 369.915167] worker_thread+0x44/0x3e0
> [ 369.915169] kthread+0xee/0x120
> [ 369.915170] ? max_active_store+0x80/0x80
> [ 369.915171] ? kthread_bind+0x10/0x10
> [ 369.915173] ret_from_fork+0x35/0x40
> [ 369.915176] INFO: task kworker/4:2:186 blocked for more than 120 seconds.
> [ 369.916475] Tainted: G OE 4.19.10 #1
> [ 369.917355] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 369.918123] kworker/4:2 D 0 186 2 0x80000000
> [ 369.918128] Workqueue: events netstamp_clear
> [ 369.918129] Call Trace:
> [ 369.918131] ? __schedule+0x2b8/0x780
> [ 369.918133] ? __switch_to_asm+0x40/0x70
> [ 369.918136] schedule+0x2d/0x80
> [ 369.918137] schedule_preempt_disabled+0x5/0x10
> [ 369.918139] __mutex_lock.isra.8+0x199/0x4d0
> [ 369.918140] ? __switch_to_asm+0x34/0x70
> [ 369.918142] ? __switch_to_asm+0x34/0x70
> [ 369.918143] static_key_enable_cpuslocked+0x28/0x80
> [ 369.918145] static_key_enable+0x11/0x20
> [ 369.918146] process_one_work+0x16a/0x2d0
> [ 369.918147] worker_thread+0x44/0x3e0
> [ 369.918149] kthread+0xee/0x120
> [ 369.918150] ? max_active_store+0x80/0x80
> [ 369.918151] ? kthread_bind+0x10/0x10
> [ 369.918153] ret_from_fork+0x35/0x40
> [ 369.918155] INFO: task kworker/1:2:312 blocked for more than 120 seconds.
> [ 369.919635] Tainted: G OE 4.19.10 #1
> [ 369.920421] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [ 369.921180] kworker/1:2 D 0 312 2 0x80000000
> [ 369.921184] Workqueue: events once_deferred
> [ 369.921184] Call Trace:
> [ 369.921187] ? __schedule+0x2b8/0x780
> [ 369.921189] ? __switch_to_asm+0x40/0x70
> [ 369.921190] schedule+0x2d/0x80
> [ 369.921192] schedule_preempt_disabled+0x5/0x10
> [ 369.921193] __mutex_lock.isra.8+0x199/0x4d0
> [ 369.921194] ? __switch_to_asm+0x34/0x70
> [ 369.921195] ? __switch_to_asm+0x34/0x70
> [ 369.921196] ? __switch_to_asm+0x40/0x70
> [ 369.921198] static_key_disable_cpuslocked+0x2a/0x70
> [ 369.921200] static_key_disable+0x11/0x20
> [ 369.921201] once_deferred+0x1a/0x30
> [ 369.921202] process_one_work+0x16a/0x2d0
> [ 369.921204] worker_thread+0x44/0x3e0
> [ 369.921205] kthread+0xee/0x120
> [ 369.921206] ? max_active_store+0x80/0x80
> [ 369.921207] ? kthread_bind+0x10/0x10
> [ 369.921209] ret_from_fork+0x35/0x40
>
>
> czw., 20 gru 2018 o 03:24 Adam Zabrocki <pi3@....com.pl> napisał(a):
>
> > Hi,
> >
> > I've just checked 4.19.10 kernel under Ubuntu:
> >
> > root@...-ubuntu:~/zzz/bypass/legit/new-mitigation/lkrg-main#
> > insmod output/p_lkrg.ko p_init_log_level=0
> > root@...-ubuntu:~/zzz/bypass/legit/new-mitigation/lkrg-main#
> > dmesg|tail
> > [ 951.305810] hv_balloon: Balloon request will be partially
> > fulfilled. Balloon floor reached.
> > [ 1251.307384] hv_balloon: Balloon request will be partially
> > fulfilled. Balloon floor reached.
> > [ 1454.690891] p_lkrg: loading out-of-tree module taints kernel.
> > [ 1454.690940] p_lkrg: module verification failed: signature
> > and/or required key missing - tainting kernel
> > [ 1454.692507] [p_lkrg] Loading LKRG...
> > [ 1455.286597] [p_lkrg] LKRG initialized successfully!
> > root@...-ubuntu:~/zzz/bypass/legit/new-mitigation/lkrg-main#
> > uname -a
> > Linux pi3-ubuntu 4.19.10-041910-generic #201812170433 SMP Mon Dec
> > 17 09:35:34 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
> > root@...-ubuntu:~/zzz/bypass/legit/new-mitigation/lkrg-main#
> >
> > so everything works fine. Kernel which I've used is this one:
> >
> > https://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19.10/
> >
> > I believe you have not standard configuration of your kernel (like
> > previously
> > you've enabled dynamic kernel module support but at the same time you've
> > disabled unloading of them - CONFIG_MODULE_UNLOAD=n).
> >
> > Nevertheless, I suspect you did not use CONFIG_ARCH_HAS_SYSCALL_WRAPPER
> > which
> > is enabled by default on all modern kernels. LKRG should correctly detect
> > that
> > but I've missed that you can still compile new kernels (4.17+) without
> > CONFIG_ARCH_HAS_SYSCALL_WRAPPER. I will try to address it soon.
> >
> > Until I do it, you can try to compile kernel with that CONFIG option or
> > manually change LKRG code. You can fix it by replacing following line of
> > code
> > in "lkrg-main/src/modules/exploit_detection/p_exploit_detection.h" file:
> >
> > - #if LINUX_VERSION_CODE >= KERNEL_VERSION(4,17,0)
> > + #if LINUX_VERSION_CODE >= KERNEL_VERSION(4,17,0) &&
> > defined(CONFIG_ARCH_HAS_SYSCALL_WRAPPER)
> >
> > Thanks,
> > Adam
> >
> > On Wed, Dec 19, 2018 at 11:35:27AM +0100, bryn1u85 wrote:
> > > My output.
> > >
> > > [root@...alhost lkrg-main]# dmesg | tail -20
> > > [ 3.836920] cryptd: max_cpu_qlen set to 1000
> > > [ 3.851179] AVX2 version of gcm_enc/dec engaged.
> > > [ 3.851180] AES CTR mode by8 optimization enabled
> > > [ 3.905981] EXT4-fs (vda1): mounted filesystem with ordered data mode.
> > > Opts: (null)
> > > [ 4.125758] Adding 7077884k swap on /dev/mapper/centos-swap.
> > > Priority:-2 extents:1 across:7077884k FS
> > > [ 4.757109] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
> > > [ 11.546012] random: crng init done
> > > [ 11.546017] random: 7 urandom warning(s) missed due to ratelimiting
> > > [ 296.265199] p_lkrg: loading out-of-tree module taints kernel.
> > > [ 296.265283] p_lkrg: module verification failed: signature and/or
> > > required key missing - tainting kernel
> > > [ 296.266610] [p_lkrg] Loading LKRG...
> > > [ 296.302774] [p_lkrg] Can't initialize exploit detection features!
> > > Exiting...
> > > [ 315.920095] [p_lkrg] Loading LKRG...
> > > [ 315.950728] [p_lkrg] Can't initialize exploit detection features!
> > > Exiting...
> > > [ 342.557674] [p_lkrg] Loading LKRG...
> > > [ 342.589567] [p_lkrg] Can't initialize exploit detection features!
> > > Exiting...
> > > [41040.104115] [p_lkrg] Loading LKRG...
> > > [41040.117676] [p_lkrg] [kretprobe] register_kretprobe() for
> > > <__x64_sys_execve> failed! [err=-38]
> > > [41040.118335] [p_lkrg] ERROR: Can't hook execve syscall :(
> > > [41040.139079] [p_lkrg] Can't initialize exploit detection features!
> > > Exiting...
> > > [root@...alhost lkrg-main]#
> > >
> > >
> > > śr., 19 gru 2018 o 03:51 Adam Zabrocki <pi3@....com.pl> napisał(a):
> > >
> > > > On Tue, Dec 18, 2018 at 09:03:44PM +0100, bryn1u85 wrote:
> > > > > hey guys,
> > > > >
> > > > > I recompiled kernel to kernel-4.19.10. After that im trying to run
> > lkrg
> > > > but
> > > > > im getting errors like below:
> > > > >
> > > > >
> > > > > [root@...alhost output]# insmod p_lkrg.ko
> > > > > insmod: ERROR: could not insert module p_lkrg.ko: No buffer space
> > > > available
> > > > > [root@...alhost output]#
> > > > >
> > > > > [ 97.954081] p_lkrg: loading out-of-tree module taints kernel.
> > > > > [ 97.954346] p_lkrg: module verification failed: signature and/or
> > > > > required key missing - tainting kernel
> > > > > [ 97.955845] [p_lkrg] Loading LKRG...
> > > > > [ 97.990086] [p_lkrg] Can't initialize exploit detection features!
> > > > > Exiting...
> > > > > [root@...alhost output]#
> > > > >
> > > > > What can i do in this situation ?
> > > > > Thanks !
> > > >
> > > > Hi,
> > > >
> > > > I believe you've already asked the same question a few times.
> > > > Alexander replied to you here:
> > > >
> > > > https://www.openwall.com/lists/lkrg-users/2018/12/06/1
> > > >
> > > > Additionally, that can be useful for you too:
> > > >
> > > >
> > > >
> > https://forums.gentoo.org/viewtopic-p-8247498.html?sid=72c22d571ef610bb77a41150177a2939#8247498
> > > >
> > > > In short:
> > > >
> > > > "For the future reference, if you would like to know why LKRG fails
> > > > initialization you can try this simple scenario:
> > > > LKRG module has a parameter p_init_log_level which defines default
> > > > log_level
> > > > which is going to be used during initialization. You can read more
> > about
> > > > log_level option (and in general about communication channel) here:
> > > >
> > > > https://openwall.info/wiki/p_lkrg/Examples#Communication-channel
> > > >
> > > > In short it might be a number between 0-4 or 0-6 (if debugging
> > compilation
> > > > was
> > > > used). If LKRG fails initialization I'm suggesting to use at least
> > > > number 4 for this parameter (e.g. # insmod p_lkrg.ko
> > p_init_log_level=4).
> > > > It
> > > > will give more information about the root of the problem. If debug
> > option
> > > > is
> > > > enabled number 5 and 6 is also available but you need to be carefully
> > > > using it
> > > > to not spam the kernel with too many logs."
> > > >
> > > > Thanks,
> > > > Adam
> > > >
> > > > --
> > > > pi3 (pi3ki31ny) - pi3 (at) itsec pl
> > > > http://pi3.com.pl
> > > >
> >
> > --
> > pi3 (pi3ki31ny) - pi3 (at) itsec pl
> > http://pi3.com.pl
> >
--
pi3 (pi3ki31ny) - pi3 (at) itsec pl
http://pi3.com.pl
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
