Date: Wed, 25 Dec 2019 21:13:54 -0500 From: Rich Felker <dalias@...c.org> To: Florian Weimer <fw@...eb.enyo.de> Cc: JeanHeyd Meneide <phdofthehouse@...il.com>, musl@...ts.openwall.com Subject: Re: [ Guidance ] Potential New Routines; Requesting Help On Wed, Dec 25, 2019 at 09:07:05PM +0100, Florian Weimer wrote: > * JeanHeyd Meneide: > > > I hope this e-mail finds you doing well this Holiday Season! I am > > interested in developing a few fast routines for text encoding for > > musl after the positive reception of a paper for the C Standard > > related to fast conversion routines: > > > > https://thephd.github.io/vendor/future_cxx/papers/source/C%20-%20Efficient%20Character%20Conversions.html > > I'm somewhat concerned that the C multibyte functions are too broken > to be useful. There is a at least one widely implemented character > set (Big5 as specified for HTML5) which does not fit the model implied > by the standard. Big5 does not have shift states, but a C > implementation using UTF-32 for wchar_t has to pretend it has because > correct conversion from Unicode to Big5 needs lookahead and cannot be > performed one character at a time. I don't think this can be modeled with shift states. C explicitly forbids a stateful wchar_t encoding/multi-wchar_t-characters. Shift states would be meaningful for the other direction. In any case I don't think it really matters. There are no existing implementations with this version of Big5 (with the offending HKSCS characters included) as the locale charset, since it can't work, and there really is no good reason to be adding *new* locale encodings. The reason we (speaking of the larger community; musl doesn't) have non-UTF-8 locales is legacy compatibility for users who need or insist on keeping them. If there really is an insistence on using this version of Big5, the characters should be added to Unicode as <compat> characters so that there's an unambiguous one-to-one correspondence, and the people who care about it working should take responsibility for doing that. > This would at least affect the proposed c8rtomb function. > > I posted a brief review of the problematic charsets in glibc here: > > <https://sourceware.org/ml/libc-alpha/2019-05/msg00079.html> I've read it but seemingly not in enough detail to gather what parts are relevant to this conversation. Rich
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.