[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date: Mon, 30 Dec 2019 13:53:45 -0500
From: JeanHeyd Meneide <phdofthehouse@...il.com>
To: Rich Felker <dalias@...c.org>
Cc: Florian Weimer <fw@...eb.enyo.de>, musl@...ts.openwall.com
Subject: Re: [ Guidance ] Potential New Routines; Requesting Help
On Mon, Dec 30, 2019 at 12:28 PM Rich Felker <dalias@...c.org> wrote:
> I think you misunderstood my remarks here. I was not talking about
> invention of new charsets (which we seem to agree should not happen),
> but making it possible to use existing legacy charsets which were
> previously not usable as a locale's encoding due to limitations of the
> C APIs. I see making that possible as counter-productive. It does not
> serve to let users keep doing something they were already doing
> (compatibility), only do to something newly backwards.
My goal is to allow developers to go from an encoding they do not
control fully (the multibyte encoding) to an encoding they know and
can reason about in their program (c8, for example). This is why I am
providing the mb -> cNN and wc -> cNN functions in both
single-character and string forms. The hope is to make it easy to go
from a statically known encoding (modulo difficulties from
__STD_C_UTF16/32__ not being defined) to the platform encoding, and
vice-versa, using the same style of functions like mb(s)(r)towc(s) and
wc(s)(r)tomb(s).
> > ... I will, however, note that the paper
> > specifically wants to add the Restartable versions of "single unit" wc
> > and mb to/from functions.
>
> I don't follow. mbrtowc and wcrtomb already exist and have since at
> least C99.
Apologies, I meant doing wc <-> cNN and mb <-> cNN!
> > ...
> >
> > This means that while wcto* and *towc functions are broken, the
>
> I don't see them as broken. They support every encoding that has ever
> worked in the past as the encoding for a locale (tautologically). The
> only way they're "broken" is if you want to add new locale encodings
> that weren't previously supportable.
Apologies; this was in reference to wide characters given a not
UTF-32 interpretation on certain platforms like Windows and certain
flavors of IBM. They chose 16 bits, which can't accommodate Unicode
without needing multiple wchar_t. Unfortunately, this means that they
were really out of luck before DR488 was accepted: they had no means
to return multiple wchar_t for characters outside the 16-bit maximum.
With DR488, restartable functions have the potential to convert out
properly (albeit, the DR was only applied to char16_t functions, so
while I have a hope and a wish we can fix it for their platforms it
might not work out for the wcto* and *towc functions anyways).
char16_t functions, though, should offer those platforms a better
way out (though not a perfect one: they'll need to rely on platform
knowledge and perform some casts).
> ...
>
> Conversion of arbitrary encodings other than the one in use by the
> locale requires a different API that takes encodings by name or some
> other identifier. The standard (POSIX) API for this is iconv, which
> has plenty of limitations of its own, some the same as what you've
> identified.
Absolutely agreed! I just want the ones that the platform controls
(wide character and multibyte character encodings) to have correct,
simple paths to static encodings that can be used for more rigorous
text processing.
Sincerely,
JeanHeyd Meneide
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
