[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20260831220338.GT23438@brightrain.aerifal.cx>
Date: Mon, 31 Aug 2026 18:03:38 -0400
From: Rich Felker <dalias@...c.org>
To: musl@...ts.openwall.com
Subject: Re: Collation data binary format conclusions
On Wed, Aug 26, 2026 at 06:49:43PM -0400, Rich Felker wrote:
> Specifics of encoding. First, the multilevel table tree keys:
>
> /3/0: count of weight levels and array of per-weight-level flags
> /3/1: collation mappings table [added lead byte for multi-CEs]
> /3/2: header dictionary table [no changes]
I'm adding one additional tree to amend this for implicit weights.
Since the assignment of implicit weights should really be tied to the
locale data as built, not changes that might happen separately to the
collation implementation, it makes sense to just store the
rules/parameters here rather than to hard-code them in libc.
/3/3: subtable of implicit weight rules, with keys beginning at 0 and
running sequentially.
/3/3/{0,1,2,...}: individual rule parameters
The format of rule parameters is:
start of range (3 byte unicode codepoint)
end of range (3 byte unicode codepoint)
subtracted offset (3 byte unicode codepoint)
mapping rule (1 byte, must be 1)
base lead byte value (1 byte)
prefix offset (1 byte)
prefix length (1 byte)
suffix offset (1 byte)
suffix length (1 byte)
The contents for current root collation data with legacy implicit
ideographic weights are:
/3/3/0: 003400 004DBF 000000 01 86 0F 04 13 02 FD 03 01 01 05 05
/3/3/1: 020000 03FFFF 000000 01 86 0F 04 13 02 FD 03 01 01 05 05
/3/3/2: 004E00 009FFF 000000 01 7E 0F 04 13 02 FD 03 01 01 05 05
/3/3/3: 00FAE0 00FA29 000000 01 7E 0F 04 13 02 FD 03 01 01 05 05
/3/3/4: 000000 10FFFF 000000 01 01 0F 05 14 02 FD 04 01 01 E0 05 05
** ** ** **
(The starred columns are not data but offsets to the bytes that
follow.)
This maps ideographic characters with implicit weights to the pattern:
[ xx yy zz, 05, 05 ]
where zz is 1......0 filling in the dots from the low 6 bits from the
codepoint value, yy is 1....... taking the next 7 bits from the
codepoint value, and zz is either 7E (core) or 86 (extended) plus the
remaining high bits.
It maps all other (unassigned/unmapped) codepoints to the pattern:
[ E0 xx yy zz, 05, 05 ]
where zz and yy are as above, but xx is 01 plus the remaining high
bits (yielding xx in the range 01 to 88). This gives implicit mappings
with primary weights in the range:
E0 01 80 80 (U+0000)
...
E0 88 FF FE (U+10FFFF)
In all of the above, the 05 bytes in the secondary and tertiary
weights are just the common/default weight at that level. These would
be changed if using different root data with different common weights,
and a quaternary or higher level could be added. There is no need for
implicit weights to ever vary beyond the primary weight level.
The above ruleset is for a locale using legacy ideographic implicit
weights. In a locale using radical-stroke order for ideographic
characters, it may be appropriate to use just the single rule:
/3/3/0: 000000 10FFFF 000000 01 01 0F 05 14 02 FD 04 01 01 E0 05 05
to put any new/unassigned ideographic characters together with
everything else unassigned. Or one may choose instead to apply the
legacy implicit ideographic rules for characters without
radical-stroke order data, which can be done with a ruleset like:
/3/3/0: 020000 03FFFF 000000 01 01 0F 05 14 02 FD 04 01 01 DF 05 05
/3/3/1: 000000 10FFFF 000000 01 01 0F 05 14 02 FD 04 01 01 E0 05 05
This would use 4-byte primary weights for the unmapped ideographic
characters, but sort them in a separate leady byte before all of the
other unmapped/unassigned characters.
The rules under /3/3 are to be processed in order as a linear search
for the first matching range. Implicit rules are only searched/applied
at all in the absence of a matching explicit mapping. The last rule
should always cover the entire Unicode codespace so that any
unassigned codepoints are caught and given an implicit mapping.
The reason for the linear search is that there are only intended to be
enough ranges to implement the UCA implicit mapping rules, not enough
to benefit from any fancier data structure. The reason rules always
spread the bits across 3 bytes is that 3 bytes suffice to cover the
entire "20.1 bit" Unicode codespace, and that ranges small enough to
fit entirely in 2 bytes do not warrant handling by implicit rules;
they are small enough to do explicitly, and do not appear in the UCA
implicit rules anyway.
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
