[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <atNkqzNxObRZEgGM@raf.org>
Date: Sat, 17 Oct 2026 23:06:03 +1100
From: raf <musl@....org>
To: musl@...ts.openwall.com
Subject: Re: fnmatch: invalid multibyte sequence in the string is
taken as end of string ("" matches "\xff" in UTF-8 locales)
On Thu, Oct 15, 2026 at 07:17:16PM -0400, Rich Felker <dalias@...c.org> wrote:
> On Fri, Oct 16, 2026 at 09:40:27AM +1100, raf wrote:
> > On Wed, Oct 14, 2026 at 01:26:14PM +1100, raf <musl@....org> wrote:
> >
> > > On Fri, Oct 09, 2026 at 09:08:25AM -0400, Rich Felker <dalias@...c.org> wrote:
> > >
> > > > On Thu, Oct 08, 2026 at 02:27:31PM +1100, raf wrote:
> > > > > That's interesting. I've made more changes to my local copy of musl's fnmatch()
> > > > > so that it supports non-UTF8 bytes (not all filenames are UTF-8, and they're not
> > > > > necessarily all even text). With my current version, your test program prints
> > > > > 1, 1, 1, 1. Its str_next() never returns -1. A patch is attached.
> > > >
> > > > Getting back to the technical aspects of this, I think you should
> > > > check how your patch makes ${foo%?} behave. I would expect shells
> > > > implement this functionality by stepping back from the end of the
> > > > string one byte at a time and calling fnmatch until they get a
> > > > successful match. If this is the case, your patch will make them rip
> > > > apart a final multibyte character and produce malformed output.
> > > >
> > > > Note that this does not happen with *. The shortest match of a lone *
> > > > is always the empty string, so ${foo%*} is just $foo. And the longest
> > > > match of a lone * is the full string. And if you have anything other
> > > > than a * prior to the *, it will necessarily anchor to a character
> > > > boundary -- as long as patterns matching a single character only match
> > > > a single character and don't wrongly match a single byte of a
> > > > multibyte character.
> > > >
> > > > The same should also apply from the beginning of a string using
> > > > ${foo#?} etc.
> > > >
> > > > Rich
> > >
> > > Hi Rich, Thanks.
> > >
> > > I'm not using my modified version of fnmatch for a
> > > shell, so the ${foo%?} syntax isn't possible to test.
> > > I'm only using it for my rawhide (rh) program which is
> > > an alternative to find(1).
> > >
> > > Glob patterns only appear in a single command line
> > > argument that is the search criteria expression. It
> > > doesn't get used in combination with all the
> > > possibilities of shell syntax.
> > >
> > > Having said that, I did take care to make sure that,
> > > when finding the final tailcnt chars of str, it would
> > > only back up over a complete UTF-8 character, or over a
> > > single non-UTF-8 byte, at a time.
> > >
> > > The real version only handles UTF-8 bytes, and is no
> > > doubt faster as a result.
> > >
> > > But I needed it to work with more than just UTF-8.
> > > Rawhide works on lots of operating systems, not just
> > > UTF-8-only ones. And on most operating systems (except
> > > Windows and macOS), filenames can contain bytes that
> > > aren't valid text in the current user's locale's
> > > encoding.
> > >
> > > By default, the local system implementation is used,
> > > but several implementations seem to have bugs. That's
> > > why I'm providing a modified musl implementation as an
> > > alternative if the user requests it.
> > >
> > > cheers,
> > > raf
> > >
> > > P.S. I'm looking forward to trying out your upcoming
> > > fnmatch changes. But I need to finish something else
> > > first.
> >
> > Hi again,
> >
> > Although I can't easily test ${abc%?} with my version,
> > I'm sure it would do the wrong thing. And it's not
>
> Thanks for testing! This might be a good thing to get added to os-test
> and similar projects.
>
> > the only one. I'd expect glibc's fnmatch to fail, because
> > it tries to match text as characters and then as individual
> > bytes (e.g., ?? matches éé and é).
>
> What is your source for that? AFAICT it does the right thing,
> operating on characters.
My source ia just my own experiments. It's so wrong, that I
figure I must be doing something wrong, but I don't know what.
I'm doing the same on OpenBSD and NetBSD that I do everywhere
else but it just never works.
> Setting LC_ALL from within the shell might or might not be updating
> the locale the shell is processing text as; I don't recall what POSIX
> says, but naively I would expect it not to update. You might want to
> test invoking the shell from the start with LC_* set to what you want
> the shell to see.
openbsd$ echo $0
-ksh
openbsd$ locale -a | grep en_AU
en_AU.UTF-8
openbsd$ LC_ALL=en_AU.UTF-8 /bin/ksh
openbsd$ touch é éé
openbsd$ echo ??
é
openbsd$ echo ?
?
openbsd$ echo ????
éé
openbsd$ ^D
openbsd$
All my systems are a few years old, but I wouldn't think
that would be the problem:
openbsd$ uname -a
OpenBSD openbsd.raf.org 7.0 GENERIC#0 amd64
> > And I'd expect the
> > implementations on OpenBSD and NetBSD to fail, because they
> > seem to ignore the locale completely, and always only match
> > text as individual bytes (e.g., ?? matches é but not éé).
>
> That's disappointing. I expected better at least from NetBSD.
>
> > So I tested ${foo%?} on all my systems:
> >
> > # LC_ALL=en_AU.UTF-8; foo=abcé; echo ${foo%?}
> > # LC_ALL=en_AU.utf8; foo=abcé; echo ${foo%?}
> > # 1=abc 0=abc\xc3 (mostly)
> > 1 freebsd sh
> > 0 openbsd sh=ksh
> > 0 netbsd ksh
> > 0 netbsd sh (outputs ab - I don't understand this)
> > 1 linux zsh
> > 1 linux bash
> > 0 linux sh=dash
> > 1 solaris bash
> > 1 solaris sh=ksh
> > 1 solaris /usr/xpg4/bin/sh
> > 1 macos zsh
> > 1 macos bash
> > 1 macos sh
> > 1 cygwin bash
> > 1 cygwin sh
> >
> > So dash is using glibc's fnmatch, but bash either doesn't use it,
> > or it uses it differently to dash (e.g., finding the longest match
> > rather than the first thing that matches).
>
> %pat is always shortest match. %%pat is longest. So I don't think it
> could be a shortest/longest distinction.
>
> Do you have a full-featured busybox around? It would be interesting to
> test its ash, which generally behaves a lot better than dash. I tried
> running it on cfarm, but it looks like they have a bad busybox build
> with locale omitted.
I'm not sure. I have a busybox installed on Debian 12 which doesn't do well:
> busybox ash
BusyBox v1.35.0 (Debian 1:1.35.0-4+deb12u1+b1) built-in shell (ash)
Enter 'help' for a list of built-in commands.
raf:dev:/home/raf# echo $0
ash
raf:dev:/home/raf# locale -a |grep en_AU
en_AU
en_AU.iso88591
en_AU.utf8
raf:dev:/home/raf# LC_ALL=en_AU.UTF-8; foo=abcé; echo ${foo%?}
abc�
> Rich
cheers,
raf
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
