[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <20260902143234.GW23438@brightrain.aerifal.cx>
Date: Wed, 2 Sep 2026 10:32:34 -0400
From: Rich Felker <dalias@...c.org>
To: musl@...ts.openwall.com
Subject: Re: Collation data binary format conclusions
On Mon, Aug 31, 2026 at 06:03:38PM -0400, Rich Felker wrote:
> On Wed, Aug 26, 2026 at 06:49:43PM -0400, Rich Felker wrote:
> > Specifics of encoding. First, the multilevel table tree keys:
> >
> > /3/0: count of weight levels and array of per-weight-level flags
> > /3/1: collation mappings table [added lead byte for multi-CEs]
> > /3/2: header dictionary table [no changes]
>
> I'm adding one additional tree to amend this for implicit weights.
> Since the assignment of implicit weights should really be tied to the
> locale data as built, not changes that might happen separately to the
> collation implementation, it makes sense to just store the
> rules/parameters here rather than to hard-code them in libc.
>
> /3/3: subtable of implicit weight rules, with keys beginning at 0 and
> running sequentially.
>
> /3/3/{0,1,2,...}: individual rule parameters
>
> The format of rule parameters is:
>
> start of range (3 byte unicode codepoint)
> end of range (3 byte unicode codepoint)
> subtracted offset (3 byte unicode codepoint)
> mapping rule (1 byte, must be 1)
> base lead byte value (1 byte)
> prefix offset (1 byte)
> prefix length (1 byte)
> suffix offset (1 byte)
> suffix length (1 byte)
Minor adjustment: let's make the codepoint values 4-byte rather than
3-byte so compilers can easily use a load(+bswap) in archs with
misaligned load support. The high byte must of course be 0.
> The contents for current root collation data with legacy implicit
> ideographic weights are:
>
> /3/3/0: 003400 004DBF 000000 01 86 0F 04 13 02 FD 03 01 01 05 05
> /3/3/1: 020000 03FFFF 000000 01 86 0F 04 13 02 FD 03 01 01 05 05
> /3/3/2: 004E00 009FFF 000000 01 7E 0F 04 13 02 FD 03 01 01 05 05
> /3/3/3: 00FAE0 00FA29 000000 01 7E 0F 04 13 02 FD 03 01 01 05 05
> /3/3/4: 000000 10FFFF 000000 01 01 0F 05 14 02 FD 04 01 01 E0 05 05
> ** ** ** **
The encoding written by hand here for the CE did not match the actual
spec, which uses cumulative sizes. Instead of FD 03 01 01 it should be
FD 03 04 05.
And of course offsets need to be adjusted for the above 3-byte to
4-byte change.
Rich
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
