Follow @Openwall on Twitter for new release announcements and other news
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <20261119163417.GE5716@brightrain.aerifal.cx>
Date: Thu, 19 Nov 2026 11:34:17 -0500
From: Rich Felker <dalias@...c.org>
To: zhaotq@....edu.cn, musl@...ts.openwall.com
Subject: Re: musl libc 1.2.5: titlecase characters violate POSIX
 lower/upper mutual exclusion

On Thu, Nov 19, 2026 at 04:11:11PM +0100, Szabolcs Nagy wrote:
> * Rich Felker <dalias@...c.org> [2026-11-19 09:06:38 -0500]:
> > On Thu, Nov 19, 2026 at 01:29:21PM +0800, zhaotq@....edu.cn wrote:
> > > Hello,
> > > 
> > > We found a POSIX character-classification defect in musl 1.2.5: some
> > > characters are classified as both lower and upper. The probe selects C.UTF-8
> > > before testing so that the non-ASCII wide characters are valid arguments
> > > under POSIX.
> > > 
> > > musl implements:
> > > 
> > >     int iswlower(wint_t wc) { return towupper(wc) != wc; }
> > >     int iswupper(wint_t wc) { return towlower(wc) != wc; }
> > > 
> > > Some Unicode titlecase characters, including U+01C5 and U+01C8, have case
> > > mappings in both directions. Consequently musl returns nonzero from both
> > > functions for those code points.
> > > 
> > > POSIX.1-2024 XBD 7.3.1 marks lower and upper as mutually exclusive in its
> > > "Valid Character Class Combinations" table. XSH iswlower() and iswupper()
> > > require nonzero results only for members of the respective class. Returning
> > > nonzero from both functions for U+01C5 and U+01C8 therefore means at least
> > > one classification is incorrect.
> > > 
> > > The probe also tests U+1F88, which musl classifies only as upper; that value
> > > does not demonstrate the violation. This finding does not require all
> > > titlecase characters to return zero from both functions.
> > 
> > This probably needs consideration for what the right fix, if anything,
> > is. The normative text you're looking at is about requirements on the
> > locale definition format put into the localedef utility, and may not
> > even apply to an implementation-defined character classification; this
> > probably requires some rules-lawyering to figure out.
> > 
> > Does it matter anyway though? Maybe application programmers deem these
> > invariants they should be able to rely on.
> 
> current REG_ICASE handling only adds upper/lower version of a
> char if it is islower/isupper, otherwise during exec only the
> original char is tried. (this logic comes from tre)
> 
> so musl regex is not case-insensitive wrt these titlecase chars.

Yeah, I think the current REG_ICASE logic is wrong/inconsistent with
respect to Unicode.

fnmatch does it the other direction but is probably also wrong:
folding the character from the haystack to include an opposite-case
variant and matching either the original or the variant to the
pattern.

Neither handle many-to-one mappings right. Titlecase is just a special
case of the problem.

The easiest fix is probably dropping case handling of titlecase
entirely (making them alphabetic but caseless and unmapped by case
transformation) since this makes the regex and fnmatch algorithms
valid (I think) but I'm not sure it's the best. It would also agree
with POSIX without change but would disagree with Unicode intent.

I'm not sure if Unicode intent is relevant. Titlecase seems to be made
up of 2 classes of characters. The first doesn't matter; it's <compat>
characters that should never actually be used and that just exist for
round-tripping some legacy encodings that had pairs of spacing
characters together as one. The second however is precombined forms
with Greek combining iota, which apparently becomes spacing
(prosgegrammeni) with capital letters. I think uppercasing the
combined form would necessarily be a 1-to-2 character mapping, which
is not representable in single-case mappings anyway and which is not
represented in the base UCD. This is sufficiently "broken" from the
start that it might be better not to even try representing it, than to
have things like U+1F88 mapping to U+1F80 under towlower, but no
towupper map for either of them.

Whatever we decide, this would be a change to go in during the next or
a later release cycle, after locale merge.

Rich

Powered by blists - more mailing lists

Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.