Date: Mon, 1 May 2017 17:54:24 -0700 From: Kees Cook <keescook@...omium.org> To: Mathias Krause <minipli@...glemail.com> Cc: Daniel Cegiełka <daniel.cegielka@...il.com>, "kernel-hardening@...ts.openwall.com" <kernel-hardening@...ts.openwall.com> Subject: Re: It looks like there will be no more public versions of PaX and Grsec. On Mon, May 1, 2017 at 3:01 PM, Mathias Krause <minipli@...glemail.com> wrote: > On 27 April 2017 at 00:04, Kees Cook <keescook@...omium.org> wrote: >> On Wed, Apr 26, 2017 at 2:05 PM, Daniel Cegiełka >> <daniel.cegielka@...il.com> wrote: >>> https://grsecurity.net/passing_the_baton_faq.php >> >> Yeah, I'm sad to see them go. PaX Team was just last night helping >> with some details of PAX_REFCOUNT as it could appear in the upstream >> refcount_t API. I hope they'll still help out from time to time. >> >> It does underscore the critical need to upstream stuff, though. Forks >> of projects might disappear at any time. :( > > Now, since grsecurity and PaX went private, quite a few users > (including me) are left in the dark and are, to say the least, > slightly pissed. But not everybody seems to be barking at the right > tree. So I've a few comments and questions for the KSPP. First, let me adjust language here a bit to make things more clear. Talking about KSPP is like talking about net-dev: it is upstream, not a separate fork. It has a focus, like net-dev, but it is upstream. So there is no difference semantically between talking about KSPP and upstream. Additionally, while PaX Team, grsecurity, and ephox's work were technically separate development efforts (for example, just look at how PAX_USERCOPY differed between PaX and grsecurity), most people used the combination, which I'll just call grsecurity here. Both definitions will be helpful below... > I think the main reason for Brad and PaX Team to make their work > private is the increased amount of work KSSP has put on them without > providing any valuable work in return. They just don't want to be First, to think they didn't get valuable work in return from upstream is missing the forest for the trees. With every release of upstream, grsecurity would get tens of thousands of commits. If there wasn't benefit in these changes, grsecurity would never forward port to the latest upstream. The fact that it is a notable event that grsecurity has ceased updating their public patches is because people using grsecurity suddenly aren't getting the upstream changes, in addition to them not getting new grsecurity features. It is a totally false equivalency to say "upstream has [created work for|used code from] grsecurity without giving anything valuable in return". I'll come back to this later. Second, when HARDENED_USERCOPY was landing, I specifically asked grsecurity about the best way to make the upstreaming efforts easiest to deal with for them, since I knew there might be some work with their forward porting, and I didn't want to create undue work for them. They ignored me. I asked again a while later and was continued to be met with silence. Looking at the results in grsecurity, though, it's clear that they chose to integrate with upstream instead of maintaining a forked implementation. For example, the core PAX_USERCOPY logic lived in fs/exec.c. When I upstreamed it, I moved it to a place that made more sense in mm/usercopy.c. When grsecurity forward ported, they could have trivially left HARDENED_USERCOPY disabled, and kept PAX_USERCOPY using fs/exec.c, but they didn't. Since grsecurity has talked in the past about how their implementations are frequently arranged to make their forward porting easier, I think this is strong evidence that using upstream's implementation (with whatever further changes they wanted) was easier than keeping their own fork. This is further supported by grsecurity being paid by CII to upstream the gcc plugin infrastructure, where they used almost entirely the same paths, Kconfigs, and Makefile changes. So, given that grsecurity did not communicate at all about wanting forward porting made easier and that they've reused gcc plugins, hardened usercopy, etc, I think it's an entirely false claim that upstream is creating more work that normal forward porting. In some places, it even appears to be easier. Finally, even if you can somehow disregard the thousands of upstream changes benefiting grsecurity, and just want to look at the areas touched while upstreaming things from grsecurity, they have benefited from the upstreaming review by upstream finding bugs in various grsecurity features (some of which haven't even landed in upstream yet: Arnd Bergmann alone found tons of issues with the initify plugin and grsecurity fixed them). Hardened usercopy found bugs in grsecurity's slab implementation, gained strncpy_from_user() coverage, and triggered a massive consolidation of per-arch uaccess fixes which have found numerous upstream bugs that will now be fixed in grsecurity too. The current refcount_t work based on PAX_REFCOUNT uncovered a crash bug in grsecurity's implementation as well as a corner case in bounds checking. Upstreaming grsecurity features brings a huge amount of testing to bear on the code, which, like all the other things in upstream, grsecurity directly benefits from. And to top all of this off, while upstreaming the latent_entropy plugin, I noticed a English typo that was present in all the grsecurity gcc plugins, and when I sent them a trivial spelling fix patch for all their plugins (rather than just letting it stand and making work for them to catch it during forward porting and fix it everywhere else), they publicly mocked the patch (which they applied). So, no, unless you count fixing newly discovered bugs in their code, I again summarily reject the notion that upstreaming grsecurity features creates "more work for grsecurity without value in return". > forced to maintain and fix-up the variants of grsecurity/PaX features > KSPP lands upstream. And no, the work of KSPP does not make their life > easier, in fact, it makes it harder. Harder for two reasons: First, > the code does not end up verbatim, it's always changed, taken out of > context and "enhanced". Second is the loose of control over the code. > But let me elaborate those two a little further. See above. In many cases, they chose integration over maintaining a fork. In others, the two features coexist (for example, to various levels of needless redundancy, see arch/arm/mm/init.c's KERNEXEC overlap with RODATA). > The first point directly forces them to think through the upstream > incarnation of a feature, how it differs from their original one and, > for those parts, how to fix them up to work properly to fit their > needs, to fit their security requirements. Those changes generate a > lot of conflicts with their version of the code which takes time to > fiddle out. As happened for __ro_after_init, MEMORY_SANITIZE, > USERCOPY,... which only went in in reduced and modified form, > generating needless work on their side -- from their point of view. One of the costs of forking a project is dealing with forward porting. This problem is faced by all kinds of organizations (usually with small engineering teams), where some set of features is desired on top of Linux, and there are three choices for handling it, in ascending order of difficulty: - pick a kernel version, fork, develop features, use, abandon, repeat (see most IoT vendors) - pick a kernel version, fork, develop features, use, forward port to next kernel version, repeat (see most phone vendors) - develop features, upstream, use whatever future kernel desired (see most CPU vendors and distros) Upstreaming can be extremely time consuming. Grsecurity made it clear from long ago that they had no intention of upstreaming things because they wanted to use their time differently. This is entirely their choice, and one that, frankly, the vast majority of Linux forks make. But it means accepting the literally unending work of forward porting. That said, when I asked grsecurity if there I was any way they would accept payment to upstream things, they did briefly agree and upstreamed the gcc plugin infrastructure. So they were willing to upstream if paid, yet ultimately decided to stop, and to continue to forward port their patches. But now they made their work private, which, I should point out, doesn't change the amount of work for them, except maybe no longer getting bug reports from their users. To your specific examples, __ro_after_init is literally a one-line change: they just make in __read_only. This was some of my first attempts to make their forward porting work easy while upstream slowly incorporated features. The work around PAX_MEMORY_SANITIZE made the slab debug paths faster for everyone. I already talked about usercopy. So, neither of us can speak for grsecurity, but I reject your belief that upstream has somehow created needless work for them. Which brings me to a question I haven't seen anyone ask yet: why does grsecurity exist? If it was created to showcase strong security mechanisms to benefit Linux users, then why fight upstreaming? Why appear to abandon their users as soon as some of their work starts entering upstream? The number of users protected by these features has gone up by orders of magnitude, benefiting all Linux users, not just grsecurity's users. One could take the position that grsecurity desires to be differentiated from upstream so that they can point at how more protected grsecurity users are over upstream users, but as the gap shrinks, it will become harder to see the benefits. But that would seem to be still a long way down the road, given how time consuming upstreaming security features can be (regardless of whether they come from grsecurity or not). So if they just want to be better than upstream without caring about how many users are protected, then it's not hard to imagine grsecurity taking their patches private. I don't know that this is the actual rationale, but it really doesn't look to me like they want to protect as many Linux users as possible. And if protecting users isn't their goal, what is? > The second point, the loose of control over the code, is even worse. > Not getting any more conflicts when porting the grsecurity/PaX patch > to a new kernel release makes changes to code that used to live in > grsecurity/PaX probably go unnoticed. And, as it seem to be the case, > upstream developers are not always familiar with all the gory details > and might introduce weaknesses and bugs. This not only makes upstream > Linux less secure, it makes grsecurity and PaX less secure, too. You're speaking entirely in theoreticals, but I understand what you're trying to say. All forward porting runs this risk. Kernel internals change in ways that threaten even unchanged grsecurity features. To bring this down to earth, I would ask "how does grsecurity perform testing?" I can point to the many ways how upstream performs testing, including LKDTM for several of the security-sensitive features. As seen in the PAX_REFCOUNT porting work, the 4.9 grsecurity patch was clearly never actually tested since _any_ exercise of the PAX_REFCOUNT protection would Oops the kernel. I'm not saying they don't have tests, nor am I saying they didn't just make a one-time mistake, but I can point you directly to how upstream tests, how upstream developers find bugs in grsecurity features, and how things can improve once upstreamed (again, for example, the massive uaccess consolidation). As to familiarity with code, this is part of upstreaming. Upstreaming isn't just throwing a patch at the wall and running away: it needs to have an understandable and compelling changelog, it needs to clearly implemented and easy to maintain, etc. This differs greatly from grsecurity changes which were rarely commented, had nearly unusable changelogs, but only needed to be understood by a couple people. What would happen to grsecurity if one of their developers decided to give up on technology and go live in the woods? I would argue that features in upstream are significantly more resilient to bad things happening, but, we're now back where we started: a hard to prove opinion. > So I can understand why they've done this. Still, with the loose of > the public availability of the patch, Linux security has suffered a > lot. Not only is upstream far far away to reach a level that is > available today in grsecurity and PaX, no, it also won't benefit from > new developments any more. What a great achievement! :( Yeah, I'm quite disappointed too. When grsecurity talked at the first Linux Security Summit in 2010, they detailed a "ten year todo list" for upstream, itemizing all the features grsecurity had that upstream needed. Over the years I took them at their word that they wanted to see upstream improved but that they didn't have time to be bothered with upstreaming. And even finding money for upstreaming didn't really help things. And now that the needle has finally moved, and things are starting to actually land upstream, they go private. I thought they wanted to protect Linux users but not at the cost of upstreaming (which, again, I can certainly understand: it can be very hard). But now they've gone private. To me, that says I misunderstood their intentions from the very start. I'm glad to see that folks from the Hardened Gentoo project have started organizing to upstream grsecurity features too. I'm excited to find more people willing to risk their sanity and bring stuff upstream: https://wiki.gentoo.org/wiki/Hardened/Hardened_Kernel_Project > *sigh* > > I think the intention of the KSPP is good -- making vanilla Linux more > secure. But the way it does its work harms overall Linux security. It > does hurt mine, that's for sure! I know the value of grsecurity and I think I have thoroughly disproved this position. Something that I think is hard to see for people not involved in day-to-day upstream work is how very different the development workflows are between upstream and grsecurity. Upstream is normally evolutionary, doing things in easy to digest pieces, where as grsecurity could just land massive changes between releases. This makes it look like upstreaming is just landing tiny bits of features, instead of looking at the long view over time. Look at how __ro_after_init (while, yes, not being anywhere close to __read_only) has been slowly gaining ground in upstream. Look at how maybe one gcc plugin appears every couple of releases, or how usercopy continues to expand coverage. > PaX in particular and know how easy it still is to exploit a vanilla > Linux. Features like KERNEXEC and RAP make it almost impossible for an > attacker to abuse a memory corruption bug in the kernel. Many unnamed > features -- unnamed because not under an #ifdef -- make exploiting > use-after-free bugs much less interesting or reduce the race window > for TOCTTOU bugs involving user copy operations. None of this can be > found in vanilla Linux. Probably never will... These are all complaints to be made to grsecurity. They are the ones who provides those features originally, did not upstream them, and then took them away. As far as never finding them in upstream, I would remind you that grsecurity, while very impressive, is not the single source for all Linux kernel security improvements. For example, ARM created the PAN emulation code, HP is working on XPFO, etc. How much further along would upstream be with security features if you yourself had started sending patches like I've asked you repeatedly over the years? I wasn't being glib; I was being genuine. If you have time to criticize, you have time to write a patch. That said, I shouldn't give you too hard a time: you have 5 times the number of commits in the kernel than grsecurity has. > So, here's my list of questions for the KSPP: > 1/ When will I be able to switch to a vanilla Linux kernel that is > equivalently hardened as a grsecurity/PaX kernel used to be? How could anyone answer that question? I can't see the future. And besides, it's not like grsecurity was providing comprehensive protections. If you're using arm64, you might feel like you're already in a better position with upstream (got PAN emu, got hardened usercopy, still no RAP). If you're on x86_64, you might feel like you've very far (no SMAP emu, no RAP). Upstream's goal is protecting as many people as possible. > 2/ Who will maintain this code and how? > 3/ Who ensures the coverage and quality won't suffer for each new > kernel release? The upstream development community, just like everything else. I answered both of these already above. > Judging from the planed EOL for the v4.9 LTS kernel -- the last one a > grsecurity patch was publicly available for --, there are less than > two years to finish the work for item 1 to ensure a secure and smooth > transition from grsecurity to upstream Linux. Will the KSPP be able to > achieve this?... I have my doubts. Again, it depends on your perspective. And if you have doubts, then be part of the solution. What would you need to do to achieve your goal? > Even if so, item 3 will be a hard problem to solve as the Linux kernel > development model does not support such a tree-wide review point in > the release cycle. Assuming there are people able and willing to judge > and review the code base for required changes (e.g. atomic_t -> > refcount_t changes or function pointer cleanups for RAP), how would > those be able to simply get in those changes at, say, time of rc5? -- > without having to argue with a horde of maintainers of the individual > subsystems involved? This is an upstream development workflow problem that is especially frustrating for security work since it can regularly touch lots of areas, but it's one that I've been working to resolve. It's actually on my list of things to discuss at the next Kernel Summit, since making these wide changes is really irritating if it's going in via each maintainer. This doesn't have to be the case, but that's how I've been doing it lately since I can work on those pieces while waiting for reviews on other things. And in the case of refcount_t, the arduous work has provided some important discussions and reviews of existing refcounting in the kernel, so I can't say this workflow is entirely without merit. > Quite a lot of questions. The external patch solved them all by not > having to deal with upstream Linux development and not having the code > available until it's ready. But now it's gone and no adequate > replacement is on the horizon. What will KSPP do about it? I had a > reasonably secure kernel, now it's gone. :( Upstream was never in the business of providing the grsecurity patch, so you'll need to ask them why they abandoned their users. Upstream's work is unchanged. We'll still continue to work to bring security features into the kernel, and we'll continue to welcome new contributors. -Kees -- Kees Cook Pixel Security
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.