Date: Sat, 2 Oct 2021 14:41:34 +0300 From: Alexander Popov <alex.popov@...ux.com> To: Linus Torvalds <torvalds@...ux-foundation.org>, Petr Mladek <pmladek@...e.com> Cc: "Paul E. McKenney" <paulmck@...nel.org>, Jonathan Corbet <corbet@....net>, Andrew Morton <akpm@...ux-foundation.org>, Thomas Gleixner <tglx@...utronix.de>, Peter Zijlstra <peterz@...radead.org>, Joerg Roedel <jroedel@...e.de>, Maciej Rozycki <macro@...am.me.uk>, Muchun Song <songmuchun@...edance.com>, Viresh Kumar <viresh.kumar@...aro.org>, Robin Murphy <robin.murphy@....com>, Randy Dunlap <rdunlap@...radead.org>, Lu Baolu <baolu.lu@...ux.intel.com>, Kees Cook <keescook@...omium.org>, Luis Chamberlain <mcgrof@...nel.org>, Wei Liu <wl@....org>, John Ogness <john.ogness@...utronix.de>, Andy Shevchenko <andriy.shevchenko@...ux.intel.com>, Alexey Kardashevskiy <aik@...abs.ru>, Christophe Leroy <christophe.leroy@...roup.eu>, Jann Horn <jannh@...gle.com>, Greg Kroah-Hartman <gregkh@...uxfoundation.org>, Mark Rutland <mark.rutland@....com>, Andy Lutomirski <luto@...nel.org>, Dave Hansen <dave.hansen@...ux.intel.com>, Steven Rostedt <rostedt@...dmis.org>, Will Deacon <will.deacon@....com>, David S Miller <davem@...emloft.net>, Borislav Petkov <bp@...en8.de>, Kernel Hardening <kernel-hardening@...ts.openwall.com>, linux-hardening@...r.kernel.org, "open list:DOCUMENTATION" <linux-doc@...r.kernel.org>, Linux Kernel Mailing List <linux-kernel@...r.kernel.org>, notify@...nel.org Subject: Re: [PATCH] Introduce the pkill_on_warn boot parameter On 01.10.2021 22:59, Linus Torvalds wrote: > On Thu, Sep 30, 2021 at 2:15 AM Petr Mladek <pmladek@...e.com> wrote: >> >> Honestly, I am not sure if panic_on_warn() or the new pkill_on_warn() >> work as expected. I wonder who uses it in practice and what is >> the experience. > > Afaik, there are only two valid uses for panic-on-warn: > > (a) test boxes (particularly VM's) that are literally running things > like syzbot and want to report any kernel warnings > > (b) the "interchangeable production machinery" fail-fast kind of situation > > So in that (a) case, it's literally that you consider a warning to be > a failure case, and just want to stop. Very useful as a way to get > notified by syzbot that "oh, that assert can actually trigger". > > And the (b) case is more of a "we have 150 million machines, we expect > about a thousand of them to fail for any random reason any day > _anyway_ - perhaps simply due to hardware failure, and we'd rather > take a machine down quickly and then perhaps look at why only much > later when we have some pattern to the failures". > > You shouldn't expect panic-on-warn to ever be the case for any actual > production machine that _matters_. If it is, that production > maintainer only has themselves to blame if they set that flag. > > But yes, the expectation is that warnings are for "this can't happen, > but if it does, it's not necessarily fatal, I want to know about it so > that I can think about it". > > So it might be a case that you don't handle, but that isn't > necessarily _wrong_ to not handle. You are ok returning an error like > -ENOSYS for that case, for example, but at the same time you are "If > somebody uses this, we should perhaps react to it". > > In many cases, a "pr_warn()" is much better. But if you are unsure > just _how_ the situation can happen, and want a call trace and > information about what process did it, and it really is a "this > shouldn't ever happen" situation, a WARN_ON() or a WARN_ON_ONCE() is > certainly not wrong. > > So think of WARN_ON() as basically an assert, but an assert with the > intention to be able to continue so that the assert can actually be > reported. BUG_ON() and friends easily result in a machine that is > dead. That's unacceptable. > > And think of "panic-on-warn" as people who can deal with their own > problems. It's fundamentally not your issue. They took that choice, > it's their problem, and the security arguments are pure BS - because > WARN_ON() just shouldn't be something you can trigger anyway. Thanks, Linus. And what do you think about the proposed pkill_on_warn? Let me quote the rationale behind it. Currently, the Linux kernel provides two types of reaction to kernel warnings: 1. Do nothing (by default), 2. Call panic() if panic_on_warn is set. That's a very strong reaction, so panic_on_warn is usually disabled on production systems. >From a safety point of view, the Linux kernel misses a middle way of handling kernel warnings: - The kernel should stop the activity that provokes a warning, - But the kernel should avoid complete denial of service. >From a security point of view, kernel warning messages provide a lot of useful information for attackers. Many GNU/Linux distributions allow unprivileged users to read the kernel log (for various reasons), so attackers use kernel warning infoleak in vulnerability exploits. See the examples: https://a13xp0p0v.github.io/2021/02/09/CVE-2021-26708.html https://a13xp0p0v.github.io/2020/02/15/CVE-2019-18683.html https://googleprojectzero.blogspot.com/2018/09/a-cache-invalidation-bug-in-linux.html Let's introduce the pkill_on_warn parameter. If this parameter is set, the kernel kills all threads in a process that provoked a kernel warning. This behavior is reasonable from a safety point of view described above. It is also useful for kernel security hardening because the system kills an exploit process that hits a kernel warning. Linus, how do you see the proper way of handling WARN_ON() in kthreads if pkill_on_warn is enabled? Thanks! Best regards, Alexander
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.