[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20260911193909.GA30289@brightrain.aerifal.cx>
Date: Fri, 11 Sep 2026 15:39:12 -0400
From: Rich Felker <dalias@...c.org>
To: musl@...ts.openwall.com
Subject: Collation testing - UCA test vectors pass
As described in the thread "Collation implementation is functional",
collation testing began last week. The test vectors being used are
those referenced at:
https://www.unicode.org/Public/UCA/16.0.0/CollationTest.html
Specifically, the CLDR version corresponding to the CLDR root data, is
what I'm using, and the file
CollationTest_CLDR_NON_IGNORABLE.txt
which corresponds to no tailoring, as collation tables are
untailored(*) and we do not have locale-generation tooling setup for
tailoring at the moment.
The first run of testing was largely successful but produced a number
of failures. These broke down into:
- localedef bug in the table production clobbering entries for
characters sequnces that are a prefix of a contraction (fixed in
commits f166483ff7 and ebeac8adbf).
- root data not actually matching the expected order because it
contains extra case information in the high bits of tertiary
weights, which needs to be stripped for the "untailored" (*)
ordering (fixup applied in commit 6b384b8679).
- localedef not having yet implemented support for parsing collation
rules that reference other collation elements for some or all of
their weights (added in commit cd337e231b).
- localedef's hard-coded radical-stroke weight assignment lead byte
not being compatible with the newer uca root data used for testing
(changed in commit e7b8e72f96, but needs change to be read from data
rather than hard-coded).
- root data containing bogus entries not corresponding to real
characters but intended for internal use by icu. (filtered out by
commit 5ccc49058a).
- null character and utf16 surrogate code units appearing in test
vector table (fixed by filtering these out in the test, in
musl-uca-draft repor, by commits 9799e793f4 and 2ee48d6710, since
these values cannot appear in C strings of multibyte characters).
- lack of support in the collation runtime code for matching
discontiguous matches, of which I was not aware at the time of
writing (added to musl-uca-draft repo by commits 5a46fe8e23,
16dcb1f256, and 19cef85f55).
All were fixed as part of the testing process, and at present, the
test vector file passes with no errors.
Unless otherwise noted, the above commit ids are for the
musl-locale-tools-draft repo. Only the last class of failure was a
deficiency in the actual collation runtime code; the rest were all
bugs or "impedance-mismatch" between the table generation tooling and
the uca root data file.
So at this point I deem the implementation good with regard to the
official conformance tests.
This is part of the locale support overhaul project, funded by NLnet
and the NGI Zero Core Fund.
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
