Date: Fri, 11 Nov 2022 16:02:23 +0100 From: Florian Weimer <fweimer@...hat.com> To: Rich Felker <dalias@...c.org> Cc: musl@...ts.openwall.com Subject: Re: Choice of wchar_t mapping for non-ASCII bytes in the POSIX locale * Rich Felker: > On Thu, Nov 10, 2022 at 09:07:53AM +0100, Florian Weimer wrote: >> It has come to my attention that musl uses the range 0xDF80…0xDFFF to >> cover the entire byte range: >> >> /* Arbitrary encoding for representing code units instead of characters. */ >> #define CODEUNIT(c) (0xdfff & (signed char)(c)) >> #define IS_CODEUNIT(c) ((unsigned)(c)-0xdf80 < 0x80) >> >> There is a very similar surrogate character mapping for undecodable >> UTF-8 bytes, suggested here: >> >> <https://web.archive.org/web/20090830064219/http://mail.nl.linux.org/linux-utf8/2000-07/msg00040.html> >> >> It uses 0xDC80…0xDCFF. This has been picked up by various >> implementations, including Python. >> >> Is there a reason why musl picked a different surrogate mapping here? >> Isn't it similar enough to the UTF-8 hack that it makes sense to pick >> the same range? > > I'll have to look back through archives to see what the motivations > for the particular range were -- I seem to recall there being some. > But I think the more important thing here is the *lack* of any > motivation to align with anything else. The values here are explicitly > *not* intended for use in any sort of information interchange. They're > invalid codes that are not Unicode scalar values, and the only reason > they exist at all is to make application-internal (or even > implementation-internal, in the case of regex/glob/etc.) > round-tripping work in the byte-based C locale while avoiding > assigning character properties to the bytes or inadvertently handling > them in a way that might facilitate pretending they're just latin1. For glibc, we are doing this because POSIX requires this for the C (POSIX) locale. It's now required to use a single-byte character set with wchar_t mappings for all bytes. Previously, I had hoped to transition to UTF-8 by default (possibly with a surrogate-escape encoding like Python's). I guess as an alternative, we could just use the Latin-1 mapping. Why hasn't musl done this? Because it would promote the idea that the world is Latin-1? > Aside from that, I'm not sure how closely "invalid non-UTF-8 bytes > that appeared in a stream expected to be UTF-8" and "bytes of what's > expected to be valid UTF-8 being treated bytewise for processing by > user request" are related. I think those two are fairly similar? But “fake single-byte character set due to POSIX mandate” is different? Thanks, Florian
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.