Openwall GNU/*/Linux - a small security-enhanced Linux distro for servers
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Thu, 20 Dec 2018 20:25:33 +0100
From: Adam Zabrocki <pi3@....com.pl>
To: lkrg-users@...ts.openwall.com
Subject: Re: insmod: ERROR: could not insert module p_lkrg.ko:
 No buffer space available

Hi,

Did you also enable CONFIG_ARCH_HAS_SYSCALL_WRAPPER? Can you share .config and 
System.map files from your build?

Additionally, it would be useful if during initialization you enable more 
verbose debugging like I have mentioed preivously:

> If LKRG fails initialization I'm suggesting to use at least number 4 for this
> parameter (e.g. # insmod p_lkrg.ko p_init_log_level=4).
> It will give more information about the root of the problem. If debug option

and share LKRG logs.

Thanks,
Ada,

On Thu, Dec 20, 2018 at 11:53:05AM +0100, bryn1u85 wrote:
> @Adam
> 
> I recompiled kernel with enabled options KPROBE. After all i compiled lkgr.
> When i do a insomd im getting so weird output in dmesg:
> 
> [    5.115150] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
> [   42.423441] p_lkrg: loading out-of-tree module taints kernel.
> [   42.423830] p_lkrg: module verification failed: signature and/or
> required key missing - tainting kernel
> [   42.425265] [p_lkrg] Loading LKRG...
> [  247.029037] INFO: task kworker/1:0:18 blocked for more than 120 seconds.
> [  247.031491]       Tainted: G           OE     4.19.10 #1
> [  247.032083] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> 
> > [  247.032666] kworker/1:0     D    0    18      2 0x80000000
> >
> [  247.032682] Workqueue: events once_deferred
> [  247.032683] Call Trace:
> [  247.032689]  ? __schedule+0x2b8/0x780
> [  247.032691]  schedule+0x2d/0x80
> [  247.032692]  schedule_preempt_disabled+0x5/0x10
> [  247.032694]  __mutex_lock.isra.8+0x199/0x4d0
> [  247.032697]  ? ttwu_do_wakeup+0x12/0xe0
> [  247.032700]  static_key_disable_cpuslocked+0x2a/0x70
> [  247.032702]  static_key_disable+0x11/0x20
> [  247.032703]  once_deferred+0x1a/0x30
> [  247.032705]  process_one_work+0x16a/0x2d0
> [  247.032706]  worker_thread+0x44/0x3e0
> [  247.032708]  kthread+0xee/0x120
> [  247.032709]  ? max_active_store+0x80/0x80
> [  247.032710]  ? kthread_bind+0x10/0x10
> [  247.032712]  ret_from_fork+0x35/0x40
> [  247.032719] INFO: task kworker/1:1:70 blocked for more than 120 seconds.
> [  247.033305]       Tainted: G           OE     4.19.10 #1
> [  247.033897] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [  247.034456] kworker/1:1     D    0    70      2 0x80000000
> [  247.034460] Workqueue: events once_deferred
> [  247.034460] Call Trace:
> [  247.034463]  ? __schedule+0x2b8/0x780
> [  247.034464]  ? __switch_to_asm+0x40/0x70
> [  247.034465]  schedule+0x2d/0x80
> [  247.034466]  schedule_preempt_disabled+0x5/0x10
> [  247.034467]  __mutex_lock.isra.8+0x199/0x4d0
> [  247.034468]  ? __switch_to_asm+0x34/0x70
> [  247.034469]  ? __switch_to_asm+0x34/0x70
> [  247.034470]  ? __switch_to_asm+0x40/0x70
> [  247.034472]  static_key_disable_cpuslocked+0x2a/0x70
> [  247.034473]  static_key_disable+0x11/0x20
> [  247.034475]  once_deferred+0x1a/0x30
> [  247.034476]  process_one_work+0x16a/0x2d0
> [  247.034478]  worker_thread+0x44/0x3e0
> [  247.034479]  kthread+0xee/0x120
> [  247.034480]  ? max_active_store+0x80/0x80
> [  247.034481]  ? kthread_bind+0x10/0x10
> [  247.034482]  ret_from_fork+0x35/0x40
> [  247.034485] INFO: task kworker/4:2:186 blocked for more than 120 seconds.
> [  247.035030]       Tainted: G           OE     4.19.10 #1
> [  247.035574] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [  247.036139] kworker/4:2     D    0   186      2 0x80000000
> [  247.036143] Workqueue: events netstamp_clear
> [  247.036144] Call Trace:
> [  247.036145]  ? __schedule+0x2b8/0x780
> [  247.036146]  ? __switch_to_asm+0x40/0x70
> [  247.036147]  schedule+0x2d/0x80
> [  247.036148]  schedule_preempt_disabled+0x5/0x10
> [  247.036149]  __mutex_lock.isra.8+0x199/0x4d0
> [  247.036150]  ? __switch_to_asm+0x34/0x70
> [  247.036151]  ? __switch_to_asm+0x34/0x70
> [  247.036153]  static_key_enable_cpuslocked+0x28/0x80
> [  247.036154]  static_key_enable+0x11/0x20
> [  247.036155]  process_one_work+0x16a/0x2d0
> [  247.036156]  worker_thread+0x44/0x3e0
> [  247.036157]  kthread+0xee/0x120
> [  247.036159]  ? max_active_store+0x80/0x80
> [  247.036160]  ? kthread_bind+0x10/0x10
> [  247.036161]  ret_from_fork+0x35/0x40
> [  247.036163] INFO: task kworker/1:2:312 blocked for more than 120 seconds.
> [  247.036745]       Tainted: G           OE     4.19.10 #1
> [  247.037337] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [  247.037948] kworker/1:2     D    0   312      2 0x80000000
> [  247.037951] Workqueue: events once_deferred
> [  247.037951] Call Trace:
> [  247.037953]  ? __schedule+0x2b8/0x780
> [  247.037954]  ? __switch_to_asm+0x40/0x70
> [  247.037955]  schedule+0x2d/0x80
> [  247.037956]  schedule_preempt_disabled+0x5/0x10
> [  247.037957]  __mutex_lock.isra.8+0x199/0x4d0
> [  247.037958]  ? __switch_to_asm+0x34/0x70
> [  247.037960]  ? __switch_to_asm+0x34/0x70
> [  247.037961]  ? __switch_to_asm+0x40/0x70
> [  247.037962]  static_key_disable_cpuslocked+0x2a/0x70
> [  247.037964]  static_key_disable+0x11/0x20
> [  247.037965]  once_deferred+0x1a/0x30
> [  247.037966]  process_one_work+0x16a/0x2d0
> [  247.037967]  worker_thread+0x44/0x3e0
> [  247.037968]  kthread+0xee/0x120
> [  247.037969]  ? max_active_store+0x80/0x80
> [  247.037970]  ? kthread_bind+0x10/0x10
> [  247.037971]  ret_from_fork+0x35/0x40
> [  247.037979] INFO: task kworker/6:2:571 blocked for more than 120 seconds.
> [  247.038583]       Tainted: G           OE     4.19.10 #1
> [  247.039205] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [  247.039845] kworker/6:2     D    0   571      2 0x80000000
> [  247.039847] Workqueue: events kprobe_optimizer
> [  247.039848] Call Trace:
> [  247.039850]  ? __schedule+0x2b8/0x780
> [  247.039850]  schedule+0x2d/0x80
> [  247.039851]  schedule_preempt_disabled+0x5/0x10
> [  247.039852]  __mutex_lock.isra.8+0x199/0x4d0
> [  247.039855]  ? synchronize_sched+0x55/0x80
> [  247.039856]  ? __call_rcu+0x280/0x280
> [  247.039857]  kprobe_optimizer+0x15b/0x290
> [  247.039858]  process_one_work+0x16a/0x2d0
> [  247.039859]  worker_thread+0x44/0x3e0
> [  247.039860]  kthread+0xee/0x120
> [  247.039861]  ? max_active_store+0x80/0x80
> [  247.039862]  ? kthread_bind+0x10/0x10
> [  247.039863]  ret_from_fork+0x35/0x40
> [  247.039872] INFO: task insmod:5275 blocked for more than 120 seconds.
> [  247.040500]       Tainted: G           OE     4.19.10 #1
> [  247.041178] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [  247.041845] insmod          D    0  5275   4188 0x80000084
> [  247.041846] Call Trace:
> [  247.041848]  ? __schedule+0x2b8/0x780
> [  247.041849]  schedule+0x2d/0x80
> [  247.041850]  schedule_preempt_disabled+0x5/0x10
> [  247.041851]  __mutex_lock.isra.8+0x199/0x4d0
> [  247.041857]  ? p_lkrg_fast_hash+0x24f/0x340 [p_lkrg]
> [  247.041858]  ? 0xffffffffc0693000
> [  247.041861]  p_create_database+0x1af/0x440 [p_lkrg]
> [  247.041862]  ? 0xffffffffc0693000
> [  247.041864]  p_lkrg_register+0xea/0x1000 [p_lkrg]
> [  247.041867]  do_one_initcall+0x47/0x1ac
> [  247.041870]  ? do_init_module+0x18/0x1e7
> [  247.041871]  ? kmem_cache_alloc+0x129/0x160
> [  247.041873]  do_init_module+0x50/0x1e7
> [  247.041874]  load_module+0x1801/0x1c80
> [  247.041876]  ? __symbol_put+0x50/0x50
> [  247.041878]  ? security_capable+0x3a/0x50
> [  247.041879]  __do_sys_finit_module+0x94/0xe0
> [  247.041881]  do_syscall_64+0x6f/0x2fc
> [  247.041883]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> [  247.041884] RIP: 0033:0x7faf389911c9
> [  247.041890] Code: Bad RIP value.
> [  247.041890] RSP: 002b:00007fff03ecca38 EFLAGS: 00000202 ORIG_RAX:
> 0000000000000139
> [  247.041891] RAX: ffffffffffffffda RBX: 000000000063a240 RCX:
> 00007faf389911c9
> [  247.041892] RDX: 0000000000000000 RSI: 000000000041a94e RDI:
> 0000000000000003
> [  247.041892] RBP: 000000000041a94e R08: 0000000000000000 R09:
> 00007fff03eccbd8
> [  247.041893] R10: 0000000000000003 R11: 0000000000000202 R12:
> 0000000000000000
> [  247.041894] R13: 000000000063a210 R14: 0000000000000000 R15:
> 0000000000000000
> [  369.908509] INFO: task kworker/1:0:18 blocked for more than 120 seconds.
> [  369.910436]       Tainted: G           OE     4.19.10 #1
> [  369.911172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [  369.912150] kworker/1:0     D    0    18      2 0x80000000
> [  369.912157] Workqueue: events once_deferred
> [  369.912161] Call Trace:
> [  369.912166]  ? __schedule+0x2b8/0x780
> [  369.912168]  schedule+0x2d/0x80
> [  369.912170]  schedule_preempt_disabled+0x5/0x10
> [  369.912171]  __mutex_lock.isra.8+0x199/0x4d0
> [  369.912175]  ? ttwu_do_wakeup+0x12/0xe0
> [  369.912178]  static_key_disable_cpuslocked+0x2a/0x70
> [  369.912179]  static_key_disable+0x11/0x20
> [  369.912181]  once_deferred+0x1a/0x30
> [  369.912183]  process_one_work+0x16a/0x2d0
> [  369.912184]  worker_thread+0x44/0x3e0
> [  369.912187]  kthread+0xee/0x120
> [  369.912188]  ? max_active_store+0x80/0x80
> [  369.912189]  ? kthread_bind+0x10/0x10
> [  369.912191]  ret_from_fork+0x35/0x40
> [  369.912198] INFO: task kworker/1:1:70 blocked for more than 120 seconds.
> [  369.913554]       Tainted: G           OE     4.19.10 #1
> [  369.914396] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [  369.915130] kworker/1:1     D    0    70      2 0x80000000
> [  369.915135] Workqueue: events once_deferred
> [  369.915136] Call Trace:
> [  369.915148]  ? __schedule+0x2b8/0x780
> [  369.915151]  ? __switch_to_asm+0x40/0x70
> [  369.915153]  schedule+0x2d/0x80
> [  369.915154]  schedule_preempt_disabled+0x5/0x10
> [  369.915155]  __mutex_lock.isra.8+0x199/0x4d0
> [  369.915157]  ? __switch_to_asm+0x34/0x70
> [  369.915158]  ? __switch_to_asm+0x34/0x70
> [  369.915159]  ? __switch_to_asm+0x40/0x70
> [  369.915161]  static_key_disable_cpuslocked+0x2a/0x70
> [  369.915163]  static_key_disable+0x11/0x20
> [  369.915164]  once_deferred+0x1a/0x30
> [  369.915166]  process_one_work+0x16a/0x2d0
> [  369.915167]  worker_thread+0x44/0x3e0
> [  369.915169]  kthread+0xee/0x120
> [  369.915170]  ? max_active_store+0x80/0x80
> [  369.915171]  ? kthread_bind+0x10/0x10
> [  369.915173]  ret_from_fork+0x35/0x40
> [  369.915176] INFO: task kworker/4:2:186 blocked for more than 120 seconds.
> [  369.916475]       Tainted: G           OE     4.19.10 #1
> [  369.917355] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [  369.918123] kworker/4:2     D    0   186      2 0x80000000
> [  369.918128] Workqueue: events netstamp_clear
> [  369.918129] Call Trace:
> [  369.918131]  ? __schedule+0x2b8/0x780
> [  369.918133]  ? __switch_to_asm+0x40/0x70
> [  369.918136]  schedule+0x2d/0x80
> [  369.918137]  schedule_preempt_disabled+0x5/0x10
> [  369.918139]  __mutex_lock.isra.8+0x199/0x4d0
> [  369.918140]  ? __switch_to_asm+0x34/0x70
> [  369.918142]  ? __switch_to_asm+0x34/0x70
> [  369.918143]  static_key_enable_cpuslocked+0x28/0x80
> [  369.918145]  static_key_enable+0x11/0x20
> [  369.918146]  process_one_work+0x16a/0x2d0
> [  369.918147]  worker_thread+0x44/0x3e0
> [  369.918149]  kthread+0xee/0x120
> [  369.918150]  ? max_active_store+0x80/0x80
> [  369.918151]  ? kthread_bind+0x10/0x10
> [  369.918153]  ret_from_fork+0x35/0x40
> [  369.918155] INFO: task kworker/1:2:312 blocked for more than 120 seconds.
> [  369.919635]       Tainted: G           OE     4.19.10 #1
> [  369.920421] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
> this message.
> [  369.921180] kworker/1:2     D    0   312      2 0x80000000
> [  369.921184] Workqueue: events once_deferred
> [  369.921184] Call Trace:
> [  369.921187]  ? __schedule+0x2b8/0x780
> [  369.921189]  ? __switch_to_asm+0x40/0x70
> [  369.921190]  schedule+0x2d/0x80
> [  369.921192]  schedule_preempt_disabled+0x5/0x10
> [  369.921193]  __mutex_lock.isra.8+0x199/0x4d0
> [  369.921194]  ? __switch_to_asm+0x34/0x70
> [  369.921195]  ? __switch_to_asm+0x34/0x70
> [  369.921196]  ? __switch_to_asm+0x40/0x70
> [  369.921198]  static_key_disable_cpuslocked+0x2a/0x70
> [  369.921200]  static_key_disable+0x11/0x20
> [  369.921201]  once_deferred+0x1a/0x30
> [  369.921202]  process_one_work+0x16a/0x2d0
> [  369.921204]  worker_thread+0x44/0x3e0
> [  369.921205]  kthread+0xee/0x120
> [  369.921206]  ? max_active_store+0x80/0x80
> [  369.921207]  ? kthread_bind+0x10/0x10
> [  369.921209]  ret_from_fork+0x35/0x40
> 
> 
> czw., 20 gru 2018 o 03:24 Adam Zabrocki <pi3@....com.pl> napisał(a):
> 
> > Hi,
> >
> > I've just checked 4.19.10 kernel under Ubuntu:
> >
> >         root@...-ubuntu:~/zzz/bypass/legit/new-mitigation/lkrg-main#
> > insmod output/p_lkrg.ko p_init_log_level=0
> >         root@...-ubuntu:~/zzz/bypass/legit/new-mitigation/lkrg-main#
> > dmesg|tail
> >         [  951.305810] hv_balloon: Balloon request will be partially
> > fulfilled. Balloon floor reached.
> >         [ 1251.307384] hv_balloon: Balloon request will be partially
> > fulfilled. Balloon floor reached.
> >         [ 1454.690891] p_lkrg: loading out-of-tree module taints kernel.
> >         [ 1454.690940] p_lkrg: module verification failed: signature
> > and/or required key missing - tainting kernel
> >         [ 1454.692507] [p_lkrg] Loading LKRG...
> >         [ 1455.286597] [p_lkrg] LKRG initialized successfully!
> >         root@...-ubuntu:~/zzz/bypass/legit/new-mitigation/lkrg-main#
> > uname -a
> >         Linux pi3-ubuntu 4.19.10-041910-generic #201812170433 SMP Mon Dec
> > 17 09:35:34 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
> >         root@...-ubuntu:~/zzz/bypass/legit/new-mitigation/lkrg-main#
> >
> > so everything works fine. Kernel which I've used is this one:
> >
> > https://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19.10/
> >
> > I believe you have not standard configuration of your kernel (like
> > previously
> > you've enabled dynamic kernel module support but at the same time you've
> > disabled unloading of them - CONFIG_MODULE_UNLOAD=n).
> >
> > Nevertheless, I suspect you did not use CONFIG_ARCH_HAS_SYSCALL_WRAPPER
> > which
> > is enabled by default on all modern kernels. LKRG should correctly detect
> > that
> > but I've missed that you can still compile new kernels (4.17+) without
> > CONFIG_ARCH_HAS_SYSCALL_WRAPPER. I will try to address it soon.
> >
> > Until I do it, you can try to compile kernel with that CONFIG option or
> > manually change LKRG code. You can fix it by replacing following line of
> > code
> > in "lkrg-main/src/modules/exploit_detection/p_exploit_detection.h" file:
> >
> > - #if LINUX_VERSION_CODE >= KERNEL_VERSION(4,17,0)
> > + #if LINUX_VERSION_CODE >= KERNEL_VERSION(4,17,0) &&
> > defined(CONFIG_ARCH_HAS_SYSCALL_WRAPPER)
> >
> > Thanks,
> > Adam
> >
> > On Wed, Dec 19, 2018 at 11:35:27AM +0100, bryn1u85 wrote:
> > > My output.
> > >
> > > [root@...alhost lkrg-main]# dmesg | tail -20
> > > [    3.836920] cryptd: max_cpu_qlen set to 1000
> > > [    3.851179] AVX2 version of gcm_enc/dec engaged.
> > > [    3.851180] AES CTR mode by8 optimization enabled
> > > [    3.905981] EXT4-fs (vda1): mounted filesystem with ordered data mode.
> > > Opts: (null)
> > > [    4.125758] Adding 7077884k swap on /dev/mapper/centos-swap.
> > > Priority:-2 extents:1 across:7077884k FS
> > > [    4.757109] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
> > > [   11.546012] random: crng init done
> > > [   11.546017] random: 7 urandom warning(s) missed due to ratelimiting
> > > [  296.265199] p_lkrg: loading out-of-tree module taints kernel.
> > > [  296.265283] p_lkrg: module verification failed: signature and/or
> > > required key missing - tainting kernel
> > > [  296.266610] [p_lkrg] Loading LKRG...
> > > [  296.302774] [p_lkrg] Can't initialize exploit detection features!
> > > Exiting...
> > > [  315.920095] [p_lkrg] Loading LKRG...
> > > [  315.950728] [p_lkrg] Can't initialize exploit detection features!
> > > Exiting...
> > > [  342.557674] [p_lkrg] Loading LKRG...
> > > [  342.589567] [p_lkrg] Can't initialize exploit detection features!
> > > Exiting...
> > > [41040.104115] [p_lkrg] Loading LKRG...
> > > [41040.117676] [p_lkrg] [kretprobe] register_kretprobe() for
> > > <__x64_sys_execve> failed! [err=-38]
> > > [41040.118335] [p_lkrg] ERROR: Can't hook execve syscall :(
> > > [41040.139079] [p_lkrg] Can't initialize exploit detection features!
> > > Exiting...
> > > [root@...alhost lkrg-main]#
> > >
> > >
> > > śr., 19 gru 2018 o 03:51 Adam Zabrocki <pi3@....com.pl> napisał(a):
> > >
> > > > On Tue, Dec 18, 2018 at 09:03:44PM +0100, bryn1u85 wrote:
> > > > > hey guys,
> > > > >
> > > > > I recompiled kernel to kernel-4.19.10. After that im trying to run
> > lkrg
> > > > but
> > > > > im getting errors like below:
> > > > >
> > > > >
> > > > > [root@...alhost output]# insmod p_lkrg.ko
> > > > > insmod: ERROR: could not insert module p_lkrg.ko: No buffer space
> > > > available
> > > > > [root@...alhost output]#
> > > > >
> > > > > [   97.954081] p_lkrg: loading out-of-tree module taints kernel.
> > > > > [   97.954346] p_lkrg: module verification failed: signature and/or
> > > > > required key missing - tainting kernel
> > > > > [   97.955845] [p_lkrg] Loading LKRG...
> > > > > [   97.990086] [p_lkrg] Can't initialize exploit detection features!
> > > > > Exiting...
> > > > > [root@...alhost output]#
> > > > >
> > > > > What can i do in this situation ?
> > > > > Thanks !
> > > >
> > > > Hi,
> > > >
> > > > I believe you've already asked the same question a few times.
> > > > Alexander replied to you here:
> > > >
> > > > https://www.openwall.com/lists/lkrg-users/2018/12/06/1
> > > >
> > > > Additionally, that can be useful for you too:
> > > >
> > > >
> > > >
> > https://forums.gentoo.org/viewtopic-p-8247498.html?sid=72c22d571ef610bb77a41150177a2939#8247498
> > > >
> > > > In short:
> > > >
> > > > "For the future reference, if you would like to know why LKRG fails
> > > > initialization you can try this simple scenario:
> > > > LKRG module has a parameter p_init_log_level which defines default
> > > > log_level
> > > > which is going to be used during initialization. You can read more
> > about
> > > > log_level option (and in general about communication channel) here:
> > > >
> > > > https://openwall.info/wiki/p_lkrg/Examples#Communication-channel
> > > >
> > > > In short it might be a number between 0-4 or 0-6 (if debugging
> > compilation
> > > > was
> > > > used). If LKRG fails initialization I'm suggesting to use at least
> > > > number 4 for this parameter (e.g. # insmod p_lkrg.ko
> > p_init_log_level=4).
> > > > It
> > > > will give more information about the root of the problem. If debug
> > option
> > > > is
> > > > enabled number 5 and 6 is also available but you need to be carefully
> > > > using it
> > > > to not spam the kernel with too many logs."
> > > >
> > > > Thanks,
> > > > Adam
> > > >
> > > > --
> > > > pi3 (pi3ki31ny) - pi3 (at) itsec pl
> > > > http://pi3.com.pl
> > > >
> >
> > --
> > pi3 (pi3ki31ny) - pi3 (at) itsec pl
> > http://pi3.com.pl
> >

-- 
pi3 (pi3ki31ny) - pi3 (at) itsec pl
http://pi3.com.pl

Powered by blists - more mailing lists

Your e-mail address:

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.