[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20261015231715.GU23438@brightrain.aerifal.cx>
Date: Thu, 15 Oct 2026 19:17:16 -0400
From: Rich Felker <dalias@...c.org>
To: musl@...ts.openwall.com
Subject: Re: fnmatch: invalid multibyte sequence in the string is
taken as end of string ("" matches "\xff" in UTF-8 locales)
On Fri, Oct 16, 2026 at 09:40:27AM +1100, raf wrote:
> On Wed, Oct 14, 2026 at 01:26:14PM +1100, raf <musl@....org> wrote:
>
> > On Fri, Oct 09, 2026 at 09:08:25AM -0400, Rich Felker <dalias@...c.org> wrote:
> >
> > > On Thu, Oct 08, 2026 at 02:27:31PM +1100, raf wrote:
> > > > That's interesting. I've made more changes to my local copy of musl's fnmatch()
> > > > so that it supports non-UTF8 bytes (not all filenames are UTF-8, and they're not
> > > > necessarily all even text). With my current version, your test program prints
> > > > 1, 1, 1, 1. Its str_next() never returns -1. A patch is attached.
> > >
> > > Getting back to the technical aspects of this, I think you should
> > > check how your patch makes ${foo%?} behave. I would expect shells
> > > implement this functionality by stepping back from the end of the
> > > string one byte at a time and calling fnmatch until they get a
> > > successful match. If this is the case, your patch will make them rip
> > > apart a final multibyte character and produce malformed output.
> > >
> > > Note that this does not happen with *. The shortest match of a lone *
> > > is always the empty string, so ${foo%*} is just $foo. And the longest
> > > match of a lone * is the full string. And if you have anything other
> > > than a * prior to the *, it will necessarily anchor to a character
> > > boundary -- as long as patterns matching a single character only match
> > > a single character and don't wrongly match a single byte of a
> > > multibyte character.
> > >
> > > The same should also apply from the beginning of a string using
> > > ${foo#?} etc.
> > >
> > > Rich
> >
> > Hi Rich, Thanks.
> >
> > I'm not using my modified version of fnmatch for a
> > shell, so the ${foo%?} syntax isn't possible to test.
> > I'm only using it for my rawhide (rh) program which is
> > an alternative to find(1).
> >
> > Glob patterns only appear in a single command line
> > argument that is the search criteria expression. It
> > doesn't get used in combination with all the
> > possibilities of shell syntax.
> >
> > Having said that, I did take care to make sure that,
> > when finding the final tailcnt chars of str, it would
> > only back up over a complete UTF-8 character, or over a
> > single non-UTF-8 byte, at a time.
> >
> > The real version only handles UTF-8 bytes, and is no
> > doubt faster as a result.
> >
> > But I needed it to work with more than just UTF-8.
> > Rawhide works on lots of operating systems, not just
> > UTF-8-only ones. And on most operating systems (except
> > Windows and macOS), filenames can contain bytes that
> > aren't valid text in the current user's locale's
> > encoding.
> >
> > By default, the local system implementation is used,
> > but several implementations seem to have bugs. That's
> > why I'm providing a modified musl implementation as an
> > alternative if the user requests it.
> >
> > cheers,
> > raf
> >
> > P.S. I'm looking forward to trying out your upcoming
> > fnmatch changes. But I need to finish something else
> > first.
>
> Hi again,
>
> Although I can't easily test ${abc%?} with my version,
> I'm sure it would do the wrong thing. And it's not
Thanks for testing! This might be a good thing to get added to os-test
and similar projects.
> the only one. I'd expect glibc's fnmatch to fail, because
> it tries to match text as characters and then as individual
> bytes (e.g., ?? matches éé and é).
What is your source for that? AFAICT it does the right thing,
operating on characters.
Setting LC_ALL from within the shell might or might not be updating
the locale the shell is processing text as; I don't recall what POSIX
says, but naively I would expect it not to update. You might want to
test invoking the shell from the start with LC_* set to what you want
the shell to see.
> And I'd expect the
> implementations on OpenBSD and NetBSD to fail, because they
> seem to ignore the locale completely, and always only match
> text as individual bytes (e.g., ?? matches é but not éé).
That's disappointing. I expected better at least from NetBSD.
> So I tested ${foo%?} on all my systems:
>
> # LC_ALL=en_AU.UTF-8; foo=abcé; echo ${foo%?}
> # LC_ALL=en_AU.utf8; foo=abcé; echo ${foo%?}
> # 1=abc 0=abc\xc3 (mostly)
> 1 freebsd sh
> 0 openbsd sh=ksh
> 0 netbsd ksh
> 0 netbsd sh (outputs ab - I don't understand this)
> 1 linux zsh
> 1 linux bash
> 0 linux sh=dash
> 1 solaris bash
> 1 solaris sh=ksh
> 1 solaris /usr/xpg4/bin/sh
> 1 macos zsh
> 1 macos bash
> 1 macos sh
> 1 cygwin bash
> 1 cygwin sh
>
> So dash is using glibc's fnmatch, but bash either doesn't use it,
> or it uses it differently to dash (e.g., finding the longest match
> rather than the first thing that matches).
%pat is always shortest match. %%pat is longest. So I don't think it
could be a shortest/longest distinction.
Do you have a full-featured busybox around? It would be interesting to
test its ash, which generally behaves a lot better than dash. I tried
running it on cfarm, but it looks like they have a bad busybox build
with locale omitted.
Rich
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
