Openwall GNU/*/Linux - a small security-enhanced Linux distro for servers
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Thu, 20 Dec 2018 11:53:05 +0100
From: bryn1u85 <m.bryn1u@...il.com>
To: lkrg-users@...ts.openwall.com
Subject: Re: insmod: ERROR: could not insert module p_lkrg.ko: No
 buffer space available

@Adam

I recompiled kernel with enabled options KPROBE. After all i compiled lkgr.
When i do a insomd im getting so weird output in dmesg:

[    5.115150] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   42.423441] p_lkrg: loading out-of-tree module taints kernel.
[   42.423830] p_lkrg: module verification failed: signature and/or
required key missing - tainting kernel
[   42.425265] [p_lkrg] Loading LKRG...
[  247.029037] INFO: task kworker/1:0:18 blocked for more than 120 seconds.
[  247.031491]       Tainted: G           OE     4.19.10 #1
[  247.032083] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.

> [  247.032666] kworker/1:0     D    0    18      2 0x80000000
>
[  247.032682] Workqueue: events once_deferred
[  247.032683] Call Trace:
[  247.032689]  ? __schedule+0x2b8/0x780
[  247.032691]  schedule+0x2d/0x80
[  247.032692]  schedule_preempt_disabled+0x5/0x10
[  247.032694]  __mutex_lock.isra.8+0x199/0x4d0
[  247.032697]  ? ttwu_do_wakeup+0x12/0xe0
[  247.032700]  static_key_disable_cpuslocked+0x2a/0x70
[  247.032702]  static_key_disable+0x11/0x20
[  247.032703]  once_deferred+0x1a/0x30
[  247.032705]  process_one_work+0x16a/0x2d0
[  247.032706]  worker_thread+0x44/0x3e0
[  247.032708]  kthread+0xee/0x120
[  247.032709]  ? max_active_store+0x80/0x80
[  247.032710]  ? kthread_bind+0x10/0x10
[  247.032712]  ret_from_fork+0x35/0x40
[  247.032719] INFO: task kworker/1:1:70 blocked for more than 120 seconds.
[  247.033305]       Tainted: G           OE     4.19.10 #1
[  247.033897] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[  247.034456] kworker/1:1     D    0    70      2 0x80000000
[  247.034460] Workqueue: events once_deferred
[  247.034460] Call Trace:
[  247.034463]  ? __schedule+0x2b8/0x780
[  247.034464]  ? __switch_to_asm+0x40/0x70
[  247.034465]  schedule+0x2d/0x80
[  247.034466]  schedule_preempt_disabled+0x5/0x10
[  247.034467]  __mutex_lock.isra.8+0x199/0x4d0
[  247.034468]  ? __switch_to_asm+0x34/0x70
[  247.034469]  ? __switch_to_asm+0x34/0x70
[  247.034470]  ? __switch_to_asm+0x40/0x70
[  247.034472]  static_key_disable_cpuslocked+0x2a/0x70
[  247.034473]  static_key_disable+0x11/0x20
[  247.034475]  once_deferred+0x1a/0x30
[  247.034476]  process_one_work+0x16a/0x2d0
[  247.034478]  worker_thread+0x44/0x3e0
[  247.034479]  kthread+0xee/0x120
[  247.034480]  ? max_active_store+0x80/0x80
[  247.034481]  ? kthread_bind+0x10/0x10
[  247.034482]  ret_from_fork+0x35/0x40
[  247.034485] INFO: task kworker/4:2:186 blocked for more than 120 seconds.
[  247.035030]       Tainted: G           OE     4.19.10 #1
[  247.035574] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[  247.036139] kworker/4:2     D    0   186      2 0x80000000
[  247.036143] Workqueue: events netstamp_clear
[  247.036144] Call Trace:
[  247.036145]  ? __schedule+0x2b8/0x780
[  247.036146]  ? __switch_to_asm+0x40/0x70
[  247.036147]  schedule+0x2d/0x80
[  247.036148]  schedule_preempt_disabled+0x5/0x10
[  247.036149]  __mutex_lock.isra.8+0x199/0x4d0
[  247.036150]  ? __switch_to_asm+0x34/0x70
[  247.036151]  ? __switch_to_asm+0x34/0x70
[  247.036153]  static_key_enable_cpuslocked+0x28/0x80
[  247.036154]  static_key_enable+0x11/0x20
[  247.036155]  process_one_work+0x16a/0x2d0
[  247.036156]  worker_thread+0x44/0x3e0
[  247.036157]  kthread+0xee/0x120
[  247.036159]  ? max_active_store+0x80/0x80
[  247.036160]  ? kthread_bind+0x10/0x10
[  247.036161]  ret_from_fork+0x35/0x40
[  247.036163] INFO: task kworker/1:2:312 blocked for more than 120 seconds.
[  247.036745]       Tainted: G           OE     4.19.10 #1
[  247.037337] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[  247.037948] kworker/1:2     D    0   312      2 0x80000000
[  247.037951] Workqueue: events once_deferred
[  247.037951] Call Trace:
[  247.037953]  ? __schedule+0x2b8/0x780
[  247.037954]  ? __switch_to_asm+0x40/0x70
[  247.037955]  schedule+0x2d/0x80
[  247.037956]  schedule_preempt_disabled+0x5/0x10
[  247.037957]  __mutex_lock.isra.8+0x199/0x4d0
[  247.037958]  ? __switch_to_asm+0x34/0x70
[  247.037960]  ? __switch_to_asm+0x34/0x70
[  247.037961]  ? __switch_to_asm+0x40/0x70
[  247.037962]  static_key_disable_cpuslocked+0x2a/0x70
[  247.037964]  static_key_disable+0x11/0x20
[  247.037965]  once_deferred+0x1a/0x30
[  247.037966]  process_one_work+0x16a/0x2d0
[  247.037967]  worker_thread+0x44/0x3e0
[  247.037968]  kthread+0xee/0x120
[  247.037969]  ? max_active_store+0x80/0x80
[  247.037970]  ? kthread_bind+0x10/0x10
[  247.037971]  ret_from_fork+0x35/0x40
[  247.037979] INFO: task kworker/6:2:571 blocked for more than 120 seconds.
[  247.038583]       Tainted: G           OE     4.19.10 #1
[  247.039205] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[  247.039845] kworker/6:2     D    0   571      2 0x80000000
[  247.039847] Workqueue: events kprobe_optimizer
[  247.039848] Call Trace:
[  247.039850]  ? __schedule+0x2b8/0x780
[  247.039850]  schedule+0x2d/0x80
[  247.039851]  schedule_preempt_disabled+0x5/0x10
[  247.039852]  __mutex_lock.isra.8+0x199/0x4d0
[  247.039855]  ? synchronize_sched+0x55/0x80
[  247.039856]  ? __call_rcu+0x280/0x280
[  247.039857]  kprobe_optimizer+0x15b/0x290
[  247.039858]  process_one_work+0x16a/0x2d0
[  247.039859]  worker_thread+0x44/0x3e0
[  247.039860]  kthread+0xee/0x120
[  247.039861]  ? max_active_store+0x80/0x80
[  247.039862]  ? kthread_bind+0x10/0x10
[  247.039863]  ret_from_fork+0x35/0x40
[  247.039872] INFO: task insmod:5275 blocked for more than 120 seconds.
[  247.040500]       Tainted: G           OE     4.19.10 #1
[  247.041178] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[  247.041845] insmod          D    0  5275   4188 0x80000084
[  247.041846] Call Trace:
[  247.041848]  ? __schedule+0x2b8/0x780
[  247.041849]  schedule+0x2d/0x80
[  247.041850]  schedule_preempt_disabled+0x5/0x10
[  247.041851]  __mutex_lock.isra.8+0x199/0x4d0
[  247.041857]  ? p_lkrg_fast_hash+0x24f/0x340 [p_lkrg]
[  247.041858]  ? 0xffffffffc0693000
[  247.041861]  p_create_database+0x1af/0x440 [p_lkrg]
[  247.041862]  ? 0xffffffffc0693000
[  247.041864]  p_lkrg_register+0xea/0x1000 [p_lkrg]
[  247.041867]  do_one_initcall+0x47/0x1ac
[  247.041870]  ? do_init_module+0x18/0x1e7
[  247.041871]  ? kmem_cache_alloc+0x129/0x160
[  247.041873]  do_init_module+0x50/0x1e7
[  247.041874]  load_module+0x1801/0x1c80
[  247.041876]  ? __symbol_put+0x50/0x50
[  247.041878]  ? security_capable+0x3a/0x50
[  247.041879]  __do_sys_finit_module+0x94/0xe0
[  247.041881]  do_syscall_64+0x6f/0x2fc
[  247.041883]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[  247.041884] RIP: 0033:0x7faf389911c9
[  247.041890] Code: Bad RIP value.
[  247.041890] RSP: 002b:00007fff03ecca38 EFLAGS: 00000202 ORIG_RAX:
0000000000000139
[  247.041891] RAX: ffffffffffffffda RBX: 000000000063a240 RCX:
00007faf389911c9
[  247.041892] RDX: 0000000000000000 RSI: 000000000041a94e RDI:
0000000000000003
[  247.041892] RBP: 000000000041a94e R08: 0000000000000000 R09:
00007fff03eccbd8
[  247.041893] R10: 0000000000000003 R11: 0000000000000202 R12:
0000000000000000
[  247.041894] R13: 000000000063a210 R14: 0000000000000000 R15:
0000000000000000
[  369.908509] INFO: task kworker/1:0:18 blocked for more than 120 seconds.
[  369.910436]       Tainted: G           OE     4.19.10 #1
[  369.911172] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[  369.912150] kworker/1:0     D    0    18      2 0x80000000
[  369.912157] Workqueue: events once_deferred
[  369.912161] Call Trace:
[  369.912166]  ? __schedule+0x2b8/0x780
[  369.912168]  schedule+0x2d/0x80
[  369.912170]  schedule_preempt_disabled+0x5/0x10
[  369.912171]  __mutex_lock.isra.8+0x199/0x4d0
[  369.912175]  ? ttwu_do_wakeup+0x12/0xe0
[  369.912178]  static_key_disable_cpuslocked+0x2a/0x70
[  369.912179]  static_key_disable+0x11/0x20
[  369.912181]  once_deferred+0x1a/0x30
[  369.912183]  process_one_work+0x16a/0x2d0
[  369.912184]  worker_thread+0x44/0x3e0
[  369.912187]  kthread+0xee/0x120
[  369.912188]  ? max_active_store+0x80/0x80
[  369.912189]  ? kthread_bind+0x10/0x10
[  369.912191]  ret_from_fork+0x35/0x40
[  369.912198] INFO: task kworker/1:1:70 blocked for more than 120 seconds.
[  369.913554]       Tainted: G           OE     4.19.10 #1
[  369.914396] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[  369.915130] kworker/1:1     D    0    70      2 0x80000000
[  369.915135] Workqueue: events once_deferred
[  369.915136] Call Trace:
[  369.915148]  ? __schedule+0x2b8/0x780
[  369.915151]  ? __switch_to_asm+0x40/0x70
[  369.915153]  schedule+0x2d/0x80
[  369.915154]  schedule_preempt_disabled+0x5/0x10
[  369.915155]  __mutex_lock.isra.8+0x199/0x4d0
[  369.915157]  ? __switch_to_asm+0x34/0x70
[  369.915158]  ? __switch_to_asm+0x34/0x70
[  369.915159]  ? __switch_to_asm+0x40/0x70
[  369.915161]  static_key_disable_cpuslocked+0x2a/0x70
[  369.915163]  static_key_disable+0x11/0x20
[  369.915164]  once_deferred+0x1a/0x30
[  369.915166]  process_one_work+0x16a/0x2d0
[  369.915167]  worker_thread+0x44/0x3e0
[  369.915169]  kthread+0xee/0x120
[  369.915170]  ? max_active_store+0x80/0x80
[  369.915171]  ? kthread_bind+0x10/0x10
[  369.915173]  ret_from_fork+0x35/0x40
[  369.915176] INFO: task kworker/4:2:186 blocked for more than 120 seconds.
[  369.916475]       Tainted: G           OE     4.19.10 #1
[  369.917355] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[  369.918123] kworker/4:2     D    0   186      2 0x80000000
[  369.918128] Workqueue: events netstamp_clear
[  369.918129] Call Trace:
[  369.918131]  ? __schedule+0x2b8/0x780
[  369.918133]  ? __switch_to_asm+0x40/0x70
[  369.918136]  schedule+0x2d/0x80
[  369.918137]  schedule_preempt_disabled+0x5/0x10
[  369.918139]  __mutex_lock.isra.8+0x199/0x4d0
[  369.918140]  ? __switch_to_asm+0x34/0x70
[  369.918142]  ? __switch_to_asm+0x34/0x70
[  369.918143]  static_key_enable_cpuslocked+0x28/0x80
[  369.918145]  static_key_enable+0x11/0x20
[  369.918146]  process_one_work+0x16a/0x2d0
[  369.918147]  worker_thread+0x44/0x3e0
[  369.918149]  kthread+0xee/0x120
[  369.918150]  ? max_active_store+0x80/0x80
[  369.918151]  ? kthread_bind+0x10/0x10
[  369.918153]  ret_from_fork+0x35/0x40
[  369.918155] INFO: task kworker/1:2:312 blocked for more than 120 seconds.
[  369.919635]       Tainted: G           OE     4.19.10 #1
[  369.920421] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
[  369.921180] kworker/1:2     D    0   312      2 0x80000000
[  369.921184] Workqueue: events once_deferred
[  369.921184] Call Trace:
[  369.921187]  ? __schedule+0x2b8/0x780
[  369.921189]  ? __switch_to_asm+0x40/0x70
[  369.921190]  schedule+0x2d/0x80
[  369.921192]  schedule_preempt_disabled+0x5/0x10
[  369.921193]  __mutex_lock.isra.8+0x199/0x4d0
[  369.921194]  ? __switch_to_asm+0x34/0x70
[  369.921195]  ? __switch_to_asm+0x34/0x70
[  369.921196]  ? __switch_to_asm+0x40/0x70
[  369.921198]  static_key_disable_cpuslocked+0x2a/0x70
[  369.921200]  static_key_disable+0x11/0x20
[  369.921201]  once_deferred+0x1a/0x30
[  369.921202]  process_one_work+0x16a/0x2d0
[  369.921204]  worker_thread+0x44/0x3e0
[  369.921205]  kthread+0xee/0x120
[  369.921206]  ? max_active_store+0x80/0x80
[  369.921207]  ? kthread_bind+0x10/0x10
[  369.921209]  ret_from_fork+0x35/0x40


czw., 20 gru 2018 o 03:24 Adam Zabrocki <pi3@....com.pl> napisał(a):

> Hi,
>
> I've just checked 4.19.10 kernel under Ubuntu:
>
>         root@...-ubuntu:~/zzz/bypass/legit/new-mitigation/lkrg-main#
> insmod output/p_lkrg.ko p_init_log_level=0
>         root@...-ubuntu:~/zzz/bypass/legit/new-mitigation/lkrg-main#
> dmesg|tail
>         [  951.305810] hv_balloon: Balloon request will be partially
> fulfilled. Balloon floor reached.
>         [ 1251.307384] hv_balloon: Balloon request will be partially
> fulfilled. Balloon floor reached.
>         [ 1454.690891] p_lkrg: loading out-of-tree module taints kernel.
>         [ 1454.690940] p_lkrg: module verification failed: signature
> and/or required key missing - tainting kernel
>         [ 1454.692507] [p_lkrg] Loading LKRG...
>         [ 1455.286597] [p_lkrg] LKRG initialized successfully!
>         root@...-ubuntu:~/zzz/bypass/legit/new-mitigation/lkrg-main#
> uname -a
>         Linux pi3-ubuntu 4.19.10-041910-generic #201812170433 SMP Mon Dec
> 17 09:35:34 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
>         root@...-ubuntu:~/zzz/bypass/legit/new-mitigation/lkrg-main#
>
> so everything works fine. Kernel which I've used is this one:
>
> https://kernel.ubuntu.com/~kernel-ppa/mainline/v4.19.10/
>
> I believe you have not standard configuration of your kernel (like
> previously
> you've enabled dynamic kernel module support but at the same time you've
> disabled unloading of them - CONFIG_MODULE_UNLOAD=n).
>
> Nevertheless, I suspect you did not use CONFIG_ARCH_HAS_SYSCALL_WRAPPER
> which
> is enabled by default on all modern kernels. LKRG should correctly detect
> that
> but I've missed that you can still compile new kernels (4.17+) without
> CONFIG_ARCH_HAS_SYSCALL_WRAPPER. I will try to address it soon.
>
> Until I do it, you can try to compile kernel with that CONFIG option or
> manually change LKRG code. You can fix it by replacing following line of
> code
> in "lkrg-main/src/modules/exploit_detection/p_exploit_detection.h" file:
>
> - #if LINUX_VERSION_CODE >= KERNEL_VERSION(4,17,0)
> + #if LINUX_VERSION_CODE >= KERNEL_VERSION(4,17,0) &&
> defined(CONFIG_ARCH_HAS_SYSCALL_WRAPPER)
>
> Thanks,
> Adam
>
> On Wed, Dec 19, 2018 at 11:35:27AM +0100, bryn1u85 wrote:
> > My output.
> >
> > [root@...alhost lkrg-main]# dmesg | tail -20
> > [    3.836920] cryptd: max_cpu_qlen set to 1000
> > [    3.851179] AVX2 version of gcm_enc/dec engaged.
> > [    3.851180] AES CTR mode by8 optimization enabled
> > [    3.905981] EXT4-fs (vda1): mounted filesystem with ordered data mode.
> > Opts: (null)
> > [    4.125758] Adding 7077884k swap on /dev/mapper/centos-swap.
> > Priority:-2 extents:1 across:7077884k FS
> > [    4.757109] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
> > [   11.546012] random: crng init done
> > [   11.546017] random: 7 urandom warning(s) missed due to ratelimiting
> > [  296.265199] p_lkrg: loading out-of-tree module taints kernel.
> > [  296.265283] p_lkrg: module verification failed: signature and/or
> > required key missing - tainting kernel
> > [  296.266610] [p_lkrg] Loading LKRG...
> > [  296.302774] [p_lkrg] Can't initialize exploit detection features!
> > Exiting...
> > [  315.920095] [p_lkrg] Loading LKRG...
> > [  315.950728] [p_lkrg] Can't initialize exploit detection features!
> > Exiting...
> > [  342.557674] [p_lkrg] Loading LKRG...
> > [  342.589567] [p_lkrg] Can't initialize exploit detection features!
> > Exiting...
> > [41040.104115] [p_lkrg] Loading LKRG...
> > [41040.117676] [p_lkrg] [kretprobe] register_kretprobe() for
> > <__x64_sys_execve> failed! [err=-38]
> > [41040.118335] [p_lkrg] ERROR: Can't hook execve syscall :(
> > [41040.139079] [p_lkrg] Can't initialize exploit detection features!
> > Exiting...
> > [root@...alhost lkrg-main]#
> >
> >
> > śr., 19 gru 2018 o 03:51 Adam Zabrocki <pi3@....com.pl> napisał(a):
> >
> > > On Tue, Dec 18, 2018 at 09:03:44PM +0100, bryn1u85 wrote:
> > > > hey guys,
> > > >
> > > > I recompiled kernel to kernel-4.19.10. After that im trying to run
> lkrg
> > > but
> > > > im getting errors like below:
> > > >
> > > >
> > > > [root@...alhost output]# insmod p_lkrg.ko
> > > > insmod: ERROR: could not insert module p_lkrg.ko: No buffer space
> > > available
> > > > [root@...alhost output]#
> > > >
> > > > [   97.954081] p_lkrg: loading out-of-tree module taints kernel.
> > > > [   97.954346] p_lkrg: module verification failed: signature and/or
> > > > required key missing - tainting kernel
> > > > [   97.955845] [p_lkrg] Loading LKRG...
> > > > [   97.990086] [p_lkrg] Can't initialize exploit detection features!
> > > > Exiting...
> > > > [root@...alhost output]#
> > > >
> > > > What can i do in this situation ?
> > > > Thanks !
> > >
> > > Hi,
> > >
> > > I believe you've already asked the same question a few times.
> > > Alexander replied to you here:
> > >
> > > https://www.openwall.com/lists/lkrg-users/2018/12/06/1
> > >
> > > Additionally, that can be useful for you too:
> > >
> > >
> > >
> https://forums.gentoo.org/viewtopic-p-8247498.html?sid=72c22d571ef610bb77a41150177a2939#8247498
> > >
> > > In short:
> > >
> > > "For the future reference, if you would like to know why LKRG fails
> > > initialization you can try this simple scenario:
> > > LKRG module has a parameter p_init_log_level which defines default
> > > log_level
> > > which is going to be used during initialization. You can read more
> about
> > > log_level option (and in general about communication channel) here:
> > >
> > > https://openwall.info/wiki/p_lkrg/Examples#Communication-channel
> > >
> > > In short it might be a number between 0-4 or 0-6 (if debugging
> compilation
> > > was
> > > used). If LKRG fails initialization I'm suggesting to use at least
> > > number 4 for this parameter (e.g. # insmod p_lkrg.ko
> p_init_log_level=4).
> > > It
> > > will give more information about the root of the problem. If debug
> option
> > > is
> > > enabled number 5 and 6 is also available but you need to be carefully
> > > using it
> > > to not spam the kernel with too many logs."
> > >
> > > Thanks,
> > > Adam
> > >
> > > --
> > > pi3 (pi3ki31ny) - pi3 (at) itsec pl
> > > http://pi3.com.pl
> > >
>
> --
> pi3 (pi3ki31ny) - pi3 (at) itsec pl
> http://pi3.com.pl
>

Content of type "text/html" skipped

Powered by blists - more mailing lists

Your e-mail address:

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.