[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <ashD7YC3fDbm2wfC@raf.org>
Date: Fri, 9 Oct 2026 12:31:25 +1100
From: raf <musl@....org>
To: musl@...ts.openwall.com
Subject: Re: fnmatch: invalid multibyte sequence in the string is
taken as end of string ("" matches "\xff" in UTF-8 locales)
On Thu, Oct 08, 2026 at 10:00:56AM -0400, Rich Felker <dalias@...c.org> wrote:
> On Thu, Oct 08, 2026 at 02:27:31PM +1100, raf wrote:
> > On Tue, Oct 06, 2026 at 04:01:53AM +0000, "He, Zhibo" <zhibo.he25@...erial.ac.uk> wrote:
> >
> > > Hello,
> > >
> > > On musl 1.2.6 in a UTF-8 locale, fnmatch() returns 0 whenever the pattern
> > > is exhausted and the next bytes of the string form an invalid multibyte
> > > sequence. fnmatch("", "\xff", 0) returns 0, and so does
> > > fnmatch("admin", "admin\xff/../../etc/passwd", 0). glibc returns
> > > FNM_NOMATCH for both, and so does musl in the C locale.
> > >
> > > In src/regex/fnmatch.c, str_next() returns 0 at end of string and -1 for an
> > > illegal sequence. fnmatch_internal() tests k <= 0 after two of its
> > > str_next() calls, in the head loop and in the tail check, so -1 takes the
> > > end-of-string branch and the function returns 0 when the pattern is also at
> > > its end. With a star in the pattern only trailing bytes 0x80-0xBF reach the
> > > tail check: fnmatch("*.txt", "x.txt\x80\x80", 0) returns 0 and
> > > fnmatch("*.txt", "x.txt\xff", 0) does not. Both tests are unchanged on
> > > master (9b2d8a16).
> > >
> > > To reproduce:
> > >
> > > #include <fnmatch.h>
> > > #include <locale.h>
> > > #include <stdio.h>
> > > int main(void)
> > > {
> > > setlocale(LC_ALL, ""); /* C.UTF-8 on Alpine, LANG unset */
> > > printf("%d\n", fnmatch("", "\xff", 0)); /* 0 = match */
> > > printf("%d\n", fnmatch("admin", "admin\xff", 0)); /* 0 = match */
> > > printf("%d\n", fnmatch("admin", "admin\xc3\xa9", 0)); /* valid e-acute: 1 */
> > > printf("%d\n", fnmatch("*.txt", "x.txt\x80\x80", 0)); /* 0 = match */
> > > return 0;
> > > }
> > >
> > > On Alpine (musl 1.2.6-r2) with LANG unset this prints 0, 0, 1, 0. Under
> > > LC_ALL=C it prints 1, 1, 1, 1, and so does glibc 2.39 under C.UTF-8. On
> > > Alpine the over-match shows through glob(), busybox ash (case and pathname
> > > expansion) and PHP's fnmatch().
> > >
> > > raf's patch of 4 October 2025, "fnmatch: Make ? match binary/non-character
> > > byte (like * does)", changes the first of the two tests so that "?" accepts
> > > an invalid byte. It keeps the END case, and with it applied the program
> > > above still prints 0, 0, 1, 0.
> > >
> > > The patch below, against master, treats -1 as a mismatch at both sites.
> > > With the patched file linked against musl 1.2.6-r2 the program prints
> > > 1, 1, 1, 1, and fnmatch("*", "\xff", 0) still returns 0. Its first hunk
> > > changes the line that raf's patch changes. If "?" should match an invalid
> > > byte, the exception belongs in that k < 0 branch.
> > >
> > > --- a/src/regex/fnmatch.c
> > > +++ b/src/regex/fnmatch.c
> > > @@ -181,7 +181,9 @@
> > > break;
> > > default:
> > > k = str_next(str, n, &sinc);
> > > - if (k <= 0)
> > > + if (k < 0)
> > > + return FNM_NOMATCH;
> > > + if (!k)
> > > return (c==END) ? 0 : FNM_NOMATCH;
> > > str += sinc;
> > > n -= sinc;
> > > @@ -242,7 +244,7 @@
> > > c = pat_next(p, endpat-p, &pinc, flags);
> > > p += pinc;
> > > if ((k = str_next(s, endstr-s, &sinc)) <= 0) {
> > > - if (c != END) return FNM_NOMATCH;
> > > + if (c != END || k < 0) return FNM_NOMATCH;
> > > break;
> > > }
> > > s += sinc;
> > >
> > > Found by differential testing of musl, glibc and OpenBSD in an MSc project
> > > at Imperial College London.
> > >
> > > Zhibo He
> >
> > Hi Zhibo,
> >
> > That's interesting. I've made more changes to my local copy of musl's fnmatch()
> > so that it supports non-UTF8 bytes (not all filenames are UTF-8, and they're not
> > necessarily all even text). With my current version, your test program prints
> > 1, 1, 1, 1. Its str_next() never returns -1. A patch is attached.
>
> Patterns are specified to match characters, not arbitrary bytes.
> Putting byte strings that are not valid characters into filenames is
> generally not a supported usage (by musl or by the standards).
>
> The choice to have * match spans that potentially contain invalid
> bytes was made a long time ago based on several considerations:
>
> - Not spending time decoding every character to ensure it's valid when
> checking the match.
>
> - Because * doesn't match a specific number of characters, there were
> not really any weird corner cases that might have wrong behavior
> from ambiguity of what unit of input should count for a match. OTOH
> having ? or brackets match bogus bytes/partial characters has weird
> consequences like s1 matching p1 and s2 matching p2 but s1s2 not
> matching p1p2.
>
> - Avoiding gratuitous difficulty cleaning up after accidentally
> extracting an archive with bad encoding that wasn't interpreted
> properly during extraction.
>
> - Originally at the time it was written, not having a byte-based C
> locale that would allow doing these things just by temporarily
> setting LC_CTYPE=C.
>
> Trying to be clever with invalid byte sequences in UTF-8 rather than
> treating them as invalid is generally a recipe for bad things
> happening.
Thanks.
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
