[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <atS-DFYc1ndOo8Re@raf.org>
Date: Sun, 18 Oct 2026 23:39:40 +1100
From: raf <musl@....org>
To: musl@...ts.openwall.com
Subject: Re: fnmatch: invalid multibyte sequence in the string is
taken as end of string ("" matches "\xff" in UTF-8 locales)
On Sat, Oct 17, 2026 at 11:42:42AM -0400, Rich Felker <dalias@...c.org> wrote:
> On Sat, Oct 17, 2026 at 11:14:31PM +1100, raf wrote:
> > On Thu, Oct 15, 2026 at 07:49:06PM -0400, Rich Felker <dalias@...c.org> wrote:
> >
> > > On Thu, Oct 15, 2026 at 07:17:16PM -0400, Rich Felker wrote:
> > > > On Fri, Oct 16, 2026 at 09:40:27AM +1100, raf wrote:
> > > > > the only one. I'd expect glibc's fnmatch to fail, because
> > > > > it tries to match text as characters and then as individual
> > > > > bytes (e.g., ?? matches éé and é).
> > > >
> > > > What is your source for that? AFAICT it does the right thing,
> > > > operating on characters.
> > >
> > > OK, I looked up the glibc source and it's indeed doing this horribly
> > > wrong. If there are any bytes that don't parse as characters in the
> > > pattern or string, it switches the whole match operation to a
> > > byte-based one.
> > >
> > > This is.... bad.
Actually, that sounds better (to me) than what I'm seeing.
At least it only switches to single-byte if the text
isn't encoded for the current user's locale. That presumably
indicates characters encoded for another user's different
locale. I'm seeing it do both every time.
I just looked at the current 2.44 version and it looks as you say.
/* Convert the strings into wide characters. Any conversion issue
fallback to the ascii version. */
I don't know what version I had looked at earlier.
And maybe I misread it anyway.
> > > Rich
> >
> > When I looked at the code (it was just a quick look),
> > it thought it looked as though it performed the search
> > as characters, and then as bytes. Or maybe that's just
> > the behaviour. It acts like that even when all the text
> > is valid UTF-8. On Debian12:
> >
> > > locale -a | grep en_AU
> > en_AU
> > en_AU.iso88591
> > en_AU.utf8
> > > touch é éé
> > > LC_ALL=en_AU.utf8 find -name '??'
> > ./é
> > ./éé
>
> I just ran this on a glibc box (Debian 11, glibc 2.31, findutils
> 4.8.0) and it behaved correctly, just matching the 2-character
> filename. So either you have something very weird going on, or this is
> a regression.
I've got glibc 2.36. Apparently there was a bug where * wouldn't
match if there were any non-characters. That was fixed, and maybe the
fix lead to the bug I'm seeing. This bug was reported I think two years
ago. It looks like it might be better(?) in 2.44 than in 2.36.
> Are you sure the environment you did the touch from is actually UTF-8
> (including correctly configured terminal) and that you're not putting
> bogus latin1 bytes in the filesystem? I can't think of a way that
> would give the behavior you're seeing, but clearly something weird is
> happening.
>
> Rich
The filenames look correct:
> echo $LANG
> en_AU.UTF-8
> touch é éé
> ls -1
é
éé
> ls -1 | od -c
0000000 303 251 \n 303 251 303 251 \n
0000010
> find -name '??'
./é
./éé
> find -name '?'
.
./é
> find -name '????'
./éé
cheers,
raf
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
