Follow @Openwall on Twitter for new release announcements and other news

Linux kernel remote logging: approaches, challenges, implementation

These are the slides of Solar Designer's talk at BSidesZagreb 2024. A video of the talk is available on YouTube.

This talk is based on research conducted for our Linux Kernel Runtime Guard (LKRG) project, which is a Linux kernel module that performs runtime integrity checking of the kernel and detection of security vulnerability exploits against the kernel. Delivery, storage, and processing of LKRG security events to/on a remote system is a natural extension of LKRG's functionality. Remote logging is also valuable on its own, including for troubleshooting and post-mortem analyses of (non-)security incidents, where the system's local logs might be unavailable, incomplete, or tampered with.

In this talk, I start by briefly examining pre-existing remote logging solutions and their suitability. Then I proceed to our own considerations and choices for transport and security protocols and software design, including many of the challenges and trade-offs encountered. Finally, I introduce and demonstrate the initial implementation in LKRG, released just in time for the talk, as well as its integration in Rocky Linux via the Security SIG package.

For the live demo (not seen on the slides), I used Valentina Palmiotti's (@chompie1337) exploit of an old vulnerability in the eBPF subsystem running current LKRG on a deliberately out-of-date Ubuntu VPS in New York, delivering logs to a VPS in Amsterdam running AlmaLinux 8.9 with Rocky Linux 8.9's SIG/Security package of lkrg-logger installed. The attack was detected and blocked (process killed before it could spawn a root shell), and LKRG messages promptly delivered to and logged on the other continent. And yes, we encourage and provide instructions for reuse of Rocky Linux SIG/Security packages on other Enterprise Linux distros.

This research and initial implementation have been sponsored by Binarly software supply chain security platform, whereas the public release, Rocky Linux integration, and this talk are due to my work at CIQ, the primary corporate sponsor of Rocky Linux.

Please click on the slides for higher-resolution versions. You can also download a PDF file with all of the slides (6 MB).

Slide 1 Slide 2 Slide 3 Slide 4 Slide 5 Slide 6 Slide 7 Slide 8 Slide 9 Slide 10 Slide 11 Slide 12 Slide 13 Slide 14 Slide 15 Slide 16 Slide 17 Slide 18 Slide 19 Slide 20 Slide 21 Slide 22 Slide 23 Slide 24 Slide 25 Slide 26 Slide 27 Slide 28 Slide 29 Slide 30

Quick Comment: