[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <atVlku4vpgkABgl6@raf.org>
Date: Mon, 19 Oct 2026 11:34:26 +1100
From: raf <musl@....org>
To: musl@...ts.openwall.com
Subject: Re: fnmatch: invalid multibyte sequence in the string is
taken as end of string ("" matches "\xff" in UTF-8 locales)
On Sun, Oct 18, 2026 at 12:20:27PM -0400, Rich Felker <dalias@...c.org> wrote:
> On Sun, Oct 18, 2026 at 11:39:40PM +1100, raf wrote:
> > On Sat, Oct 17, 2026 at 11:42:42AM -0400, Rich Felker <dalias@...c.org> wrote:
> >
> > > On Sat, Oct 17, 2026 at 11:14:31PM +1100, raf wrote:
> > > > On Thu, Oct 15, 2026 at 07:49:06PM -0400, Rich Felker <dalias@...c.org> wrote:
> > > >
> > > > > On Thu, Oct 15, 2026 at 07:17:16PM -0400, Rich Felker wrote:
> > > > > > On Fri, Oct 16, 2026 at 09:40:27AM +1100, raf wrote:
> > > > > > > the only one. I'd expect glibc's fnmatch to fail, because
> > > > > > > it tries to match text as characters and then as individual
> > > > > > > bytes (e.g., ?? matches éé and é).
> > > > > >
> > > > > > What is your source for that? AFAICT it does the right thing,
> > > > > > operating on characters.
> > > > >
> > > > > OK, I looked up the glibc source and it's indeed doing this horribly
> > > > > wrong. If there are any bytes that don't parse as characters in the
> > > > > pattern or string, it switches the whole match operation to a
> > > > > byte-based one.
> > > > >
> > > > > This is.... bad.
> >
> > Actually, that sounds better (to me) than what I'm seeing.
> > At least it only switches to single-byte if the text
> > isn't encoded for the current user's locale. That presumably
> > indicates characters encoded for another user's different
> > locale. I'm seeing it do both every time.
> >
> > I just looked at the current 2.44 version and it looks as you say.
> >
> > /* Convert the strings into wide characters. Any conversion issue
> > fallback to the ascii version. */
> >
> > I don't know what version I had looked at earlier.
> > And maybe I misread it anyway.
> >
> > > > > Rich
> > > >
> > > > When I looked at the code (it was just a quick look),
> > > > it thought it looked as though it performed the search
> > > > as characters, and then as bytes. Or maybe that's just
> > > > the behaviour. It acts like that even when all the text
> > > > is valid UTF-8. On Debian12:
> > > >
> > > > > locale -a | grep en_AU
> > > > en_AU
> > > > en_AU.iso88591
> > > > en_AU.utf8
> > > > > touch é éé
> > > > > LC_ALL=en_AU.utf8 find -name '??'
> > > > ./é
> > > > ./éé
> > >
> > > I just ran this on a glibc box (Debian 11, glibc 2.31, findutils
> > > 4.8.0) and it behaved correctly, just matching the 2-character
> > > filename. So either you have something very weird going on, or this is
> > > a regression.
> >
> > I've got glibc 2.36. Apparently there was a bug where * wouldn't
> > match if there were any non-characters. That was fixed, and maybe the
> > fix lead to the bug I'm seeing. This bug was reported I think two years
> > ago. It looks like it might be better(?) in 2.44 than in 2.36.
> >
> > > Are you sure the environment you did the touch from is actually UTF-8
> > > (including correctly configured terminal) and that you're not putting
> > > bogus latin1 bytes in the filesystem? I can't think of a way that
> > > would give the behavior you're seeing, but clearly something weird is
> > > happening.
> > >
> > > Rich
> >
> > The filenames look correct:
> >
> > > echo $LANG
> > > en_AU.UTF-8
> > > touch é éé
> > > ls -1
> > é
> > éé
> > > ls -1 | od -c
> > 0000000 303 251 \n 303 251 303 251 \n
> > 0000010
> > > find -name '??'
> > ./é
> > ./éé
> > > find -name '?'
> > .
> > ./é
> > > find -name '????'
> > ./éé
>
> Thanks! We tracked it down on IRC and I've filed a bug report:
> https://sourceware.org/bugzilla/show_bug.cgi?id=34761
Thanks. That's great. It was reported three years ago, and now
that you have supplied them with a fix, they might act on it.
Thanks for making the world a better place!
But I do think the fallback is good for systems with multiple
encodings in use, or malicious non-text filenames trying to
evade detection. But it would be nice if that were never needed
(and in practice it probably never/rarely is). You probably can't
have that and a working ${foo%?} at the same time. But for a standard
fnmatch, it makes sense to support the needs of shells and normal
sane usage in preference to wierd cases. Perhaps the fallback could
be enabled only when requested with a new GNU-extension flag.
glibc has to cope with so many situations. It must be a nightmare
in there.
They say to never show your children how laws or sausages are made.
I think we can add standard c libraries to that list. :-)
(I don't really mean that)
cheers,
raf
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
