[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20261018162027.GY23438@brightrain.aerifal.cx>
Date: Sun, 18 Oct 2026 12:20:27 -0400
From: Rich Felker <dalias@...c.org>
To: musl@...ts.openwall.com
Subject: Re: fnmatch: invalid multibyte sequence in the string is
taken as end of string ("" matches "\xff" in UTF-8 locales)
On Sun, Oct 18, 2026 at 11:39:40PM +1100, raf wrote:
> On Sat, Oct 17, 2026 at 11:42:42AM -0400, Rich Felker <dalias@...c.org> wrote:
>
> > On Sat, Oct 17, 2026 at 11:14:31PM +1100, raf wrote:
> > > On Thu, Oct 15, 2026 at 07:49:06PM -0400, Rich Felker <dalias@...c.org> wrote:
> > >
> > > > On Thu, Oct 15, 2026 at 07:17:16PM -0400, Rich Felker wrote:
> > > > > On Fri, Oct 16, 2026 at 09:40:27AM +1100, raf wrote:
> > > > > > the only one. I'd expect glibc's fnmatch to fail, because
> > > > > > it tries to match text as characters and then as individual
> > > > > > bytes (e.g., ?? matches éé and é).
> > > > >
> > > > > What is your source for that? AFAICT it does the right thing,
> > > > > operating on characters.
> > > >
> > > > OK, I looked up the glibc source and it's indeed doing this horribly
> > > > wrong. If there are any bytes that don't parse as characters in the
> > > > pattern or string, it switches the whole match operation to a
> > > > byte-based one.
> > > >
> > > > This is.... bad.
>
> Actually, that sounds better (to me) than what I'm seeing.
> At least it only switches to single-byte if the text
> isn't encoded for the current user's locale. That presumably
> indicates characters encoded for another user's different
> locale. I'm seeing it do both every time.
>
> I just looked at the current 2.44 version and it looks as you say.
>
> /* Convert the strings into wide characters. Any conversion issue
> fallback to the ascii version. */
>
> I don't know what version I had looked at earlier.
> And maybe I misread it anyway.
>
> > > > Rich
> > >
> > > When I looked at the code (it was just a quick look),
> > > it thought it looked as though it performed the search
> > > as characters, and then as bytes. Or maybe that's just
> > > the behaviour. It acts like that even when all the text
> > > is valid UTF-8. On Debian12:
> > >
> > > > locale -a | grep en_AU
> > > en_AU
> > > en_AU.iso88591
> > > en_AU.utf8
> > > > touch é éé
> > > > LC_ALL=en_AU.utf8 find -name '??'
> > > ./é
> > > ./éé
> >
> > I just ran this on a glibc box (Debian 11, glibc 2.31, findutils
> > 4.8.0) and it behaved correctly, just matching the 2-character
> > filename. So either you have something very weird going on, or this is
> > a regression.
>
> I've got glibc 2.36. Apparently there was a bug where * wouldn't
> match if there were any non-characters. That was fixed, and maybe the
> fix lead to the bug I'm seeing. This bug was reported I think two years
> ago. It looks like it might be better(?) in 2.44 than in 2.36.
>
> > Are you sure the environment you did the touch from is actually UTF-8
> > (including correctly configured terminal) and that you're not putting
> > bogus latin1 bytes in the filesystem? I can't think of a way that
> > would give the behavior you're seeing, but clearly something weird is
> > happening.
> >
> > Rich
>
> The filenames look correct:
>
> > echo $LANG
> > en_AU.UTF-8
> > touch é éé
> > ls -1
> é
> éé
> > ls -1 | od -c
> 0000000 303 251 \n 303 251 303 251 \n
> 0000010
> > find -name '??'
> ./é
> ./éé
> > find -name '?'
> .
> ./é
> > find -name '????'
> ./éé
Thanks! We tracked it down on IRC and I've filed a bug report:
https://sourceware.org/bugzilla/show_bug.cgi?id=34761
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
