Date: Thu, 26 Dec 2019 00:43:45 -0500 From: JeanHeyd Meneide <phdofthehouse@...il.com> To: Rich Felker <dalias@...c.org> Cc: Florian Weimer <fw@...eb.enyo.de>, musl@...ts.openwall.com Subject: Re: [ Guidance ] Potential New Routines; Requesting Help Dear Rich Felker and Florien Weimer, Thank you for taking an interest in this! On Wed, Dec 25, 2019 at 9:14 PM Rich Felker <dalias@...c.org> wrote: > > On Wed, Dec 25, 2019 at 09:07:05PM +0100, Florian Weimer wrote: > > > > I'm somewhat concerned that the C multibyte functions are too broken > > to be useful. There is a at least one widely implemented character > > set (Big5 as specified for HTML5) which does not fit the model implied > > by the standard. Big5 does not have shift states, but a C > > implementation using UTF-32 for wchar_t has to pretend it has because > > correct conversion from Unicode to Big5 needs lookahead and cannot be > > performed one character at a time. > > I don't think this can be modeled with shift states. C explicitly > forbids a stateful wchar_t encoding/multi-wchar_t-characters. Shift > states would be meaningful for the other direction. > > In any case I don't think it really matters. There are no existing > implementations with this version of Big5 (with the offending HKSCS > characters included) as the locale charset, since it can't work, and > there really is no good reason to be adding *new* locale encodings. > The reason we (speaking of the larger community; musl doesn't) have > non-UTF-8 locales is legacy compatibility for users who need or insist > on keeping them. I have no intentions on adding new locale-based charsets (and I absolutely agree that we should not be adding anymore). That being said, I want to focus on the main part of this, which is the ability to model existing encodings which have both/either shift states and/or multi-character expansions. It is true wchar_t is invariably busted. There is no way a wide character string can be multi-unit: that shipped sailed when wchar_t was specified as is, and when it was codified in various APIs such as mbtowc/wctomb and friends. I will, however, note that the paper specifically wants to add the Restartable versions of "single unit" wc and mb to/from functions. The reason I chose the restartable forms is because in C2x a defect report was accepted that clarified the original intent of the single-character functions with respect to their R versions: > "After discussion, the committee concluded that mbstate was already specified to handle this case, and as such the second interpretation is intended. The committee believes that there is an underspecification, and solicited a further paper from the author along the lines of the second option. Although not discussed a Suggested Technical Corrigendum can be found at N2040." -- WG14, April 2016, http://www.open-std.org/jtc1/sc22/wg14/www/docs/n2059.htm#dr_488 This means that while wcto* and *towc functions are broken, the restartable versions need not be. By returning a value of "0" (e.g., for c8rtomb) or returning a value of "-3" (e.g., for mbrtoc8), we can write out multiple characters based on the input and any data stored in mbstate_t. This allows us to handle e.g. UTF-16 for c16, multi-conversions for c8, and more. My understanding is that for one of the referenced encodings in the linked mailing post (TSCII), this would cover that use case. My understanding is also that for some Big5 encodings it would -- as you stated -- treat ambiguous leading sequences as a shift state, accumulate data in the mbstate_t, and then write out data if the sequence is made unambiguous by having further data provided to the next call. Is my understanding incorrect? Is there an implementation limitation I am missing here? I would hate to do this the wrong way and make the encoding situation even worse: my goal is to absolutely ensure we can transition legacy encodings to statically-known encodings. Sincerely, JeanHeyd Meneide
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.