Date: Mon, 5 Jun 2017 14:46:33 +0200 From: Joakim Sindholt <opensource@...sha.com> To: musl@...ts.openwall.com, slade@...nam.net Subject: Re: ENOSYS/EOPNOTSUPP fallback? On Sun, Jun 04, 2017 at 09:22:27PM -0600, Benjamin Slade wrote: > I ran into what is perhaps a weird edge case. I'm running a system with > musl that uses a ZFS root fs. When I was trying to install some > flatpaks, I got an `fallocate` failure, with no `dd` fallback. Querying > the flatpak team, the fallback to `dd` seems to be something which glibc > does (and so the other components assume will be taken care). > > Here is the exchange regarding this issue: > https://github.com/flatpak/flatpak/issues/802 To quote the glibc source file linked in the bug: /* Minimize data transfer for network file systems, by issuing single-byte write requests spaced by the file system block size. (Most local file systems have fallocate support, so this fallback code is not used there.) */ /* NFS clients do not propagate the block size of the underlying storage and may report a much larger value which would still leave holes after the loop below, so we cap the increment at 4096. */ /* Write a null byte to every block. This is racy; we currently lack a better option. Compare-and-swap against a file mapping might address local races, but requires interposition of a signal handler to catch SIGBUS. */ Which leaves 2 massive bugs: 1) the leaving of unallocated gaps both because of the NFS thing but also because other file systems may work on entirely different principles that are not accounted for here and 2) overwriting data currently being written to the file as it's being forcibly allocated (which might be doing nothing, think deduplication). This is not a viable general solution and furthermore fallocate is mostly just an optimization hint. If it's a hard requirement of your software I would suggest implementing it in your file system. These operations can only be safely implemented in the kernel. An example: MyFS uses write time deduplication on unused blocks (and blocks with all zeroes fall under the umbrella of unused). Glibc starts its dance where it writes a zero byte to the beginning of each block it perceives and for now let's just say it has the right block size. MyFS just trashes these writes immediately without touching the disk and updates the size metadata which gets lazily written at some point. There's only 400k left on the disk and your fallocate of 16G will succeed and run exceptionally fast to boot, but it will have allocated nothing and your next write fails with ENOSPC. Another example: myutil has 2 threads running. One thread is constantly writing things to a file. The other thread sometimes writes large chunks of data to the file and so it hints the kernel to allocate these large chunks by calling fallocate, and only then taking the lock(s) held internally to synchronize the threads. The first thread finds it needs to update something in the section currently being fallocated by glibc's algorithm. Suddenly zero bytes appear at 4k intervals for no discernible reason, overwriting the data. Personally I would look into seeing to it that flatpak only uses fallocate as an optimization. The most reliable thing I can think of otherwise would be to do the locking necessary (if any) in the program and filling the entire target section of the file with data from /dev/urandom, but even that may fail spectacularly with transparent compression (albeit unlikely). Hope this was at least somewhat helpful.
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.