Date: Mon, 30 Dec 2019 13:39:10 -0500 From: JeanHeyd Meneide <phdofthehouse@...il.com> To: Rich Felker <dalias@...c.org> Cc: musl@...ts.openwall.com Subject: Re: [ Guidance ] Potential New Routines; Requesting Help On Mon, Dec 30, 2019 at 12:31 PM Rich Felker <dalias@...c.org> wrote: > > ... > This is interesting, but I'm trying to understand the motivation. > > If __STDC_ISO_10646__ is defined, wchar_t is UTF-32/UCS-4, and the > proposed functions are just the identity (for the c32 ones) and > UTF-16/32 conversion. > > If it's not defined, you have the same problem as the current mb/cNN > functions: there's no reason to believe arbitrary Unicode characters > can round-trip through wchar_t any better than they can through > multibyte characters. In fact on such implementations it's likely that > wchar_t meanings are locale-dependent and just a remapping of the > byte/multibyte characters. I'm sorry, I'll try to phrase it as best as I can. The issue I and others have with the lack of cNNtowc is that, if we are to write standards-compliant C, the only way to do that transformation from, for example, char16_t data to wchar_t portably is: c16rtomb -> multibyte data -> mbrtowc The problem with such a conversion sequence is that there are many legacy encodings and this causes bugs on many user's machines. Text representable in both char16_t and wchar_t is lost in the middle: due to the middle not handling it, putting us in a place where we lose of data going to and from wchar_t to char16_t. This has been frustrating for a number of users who try to rely on the standard, only to have to write the above conversions sequence and fail. Thus, providing a direct function with no intermediates results in a better Standard C experience. A minor but still helpful secondary motivation is in giving people on certain long-standing platforms a way out. By definition, UTF16 does not work with wchar_t, so I am explicitly told that wchar_t for a platform like .e.g Windows is UCS-2 (the non-multi-unit version of UTF-16 that was deprecated a while ago) is wrong when using the Standard Library if I want real Unicode Support. Library developers tell me to rely on platform-specific APIs. The "use MultiByteToWideChar" or "use ICU" or "use this AIX-specific function", makes it much less of a Standard way to handle text: hence, the paper to the WG14 C Committee. The restartable versions of the single-character functions and the bulk conversion functions give ways for implementations locked to behaving like the deprecated UCS-2, 16-bit-single-unit-encoding a way out, and also allow us to have lossless data conversion. This reasoning might be a little bit "overdone" for libraries like musl and glibc who got wchar_t right (thank you!), but part of standardizing these things means I have to account for implementations that have been around longer than I have been alive. :) Does that make sense? > What situation do you envision where the proposed functions let you > reliably do something that's not already possible? My understanding is that libraries such as musl are "blessed" as distributions of the Standard Library, and that they can access system information that makes it possible for them to utilize what the current "wchar_t encoding" is in a way normal, regular developers cannot. Specifically, in the generic external implementation I have been working on, I have a number of #ifdef to check for, say, IBM machines, then check if they are specifically under zh/tw or even jp locales, because they deploy a wchar_t in these scenarios that is neither UTF16 or UTF32 (but instead a flavor of one of the GB encodings and Japanese encodings); otherwise, IBM uses UTF16/UCS-2 for wchar_t in i686 and UTF-32 for wchar_t in x86_64 for certain machines. I also check for what happens on Windows under various settings as well. Doing this as an external library is hard, because there is no way I can control the knobs for such reliably, but that a Standard Library distribution would have access to that information (since they are providing such functions already). So, for example, musl -- being the C library -- controls how the wchar_t should behave (modulo compiler intervention) for its wide character functions. Similarly, glibc would know what to do for its platforms, and IBM would know what to do for its platforms, and so on and so forth. Each distribution would provide behavior in coordination with their platform. Is this incorrect? Am I assuming a level of standard library <-> vendor relation/cooperation that does not exist? > > While I have a basic implementation, I would like to use some > > processor and compiler intrinsics to make it faster and make sure my > > first contribution meets both quality and speed standards for a C > > library. > > > > Is there a place in the codebase I can look to for guidance on > > how to handle intrinsics properly within musl libc? If there is > > already infrastructure and common idioms in place, I would rather use > > that then starting to spin up my own. > > I'm not sure what you mean by intrinsics or why you're looking for > them but I guess you're thinking of something as a performance > optimization? musl favors having code in straight simple C except when > there's a strong reason (known bottleneck in existing real-world > software -- things like memcpy, strlen, etc.) to do otherwise. The > existing mb/wc code is slightly "vectorized" (see mbsrtowcs) but doing > so was probably a mistake. The motivation came along with one of the > early motivations for musl: not making UTF-8 a major performance > regression like it was in glibc. But it turned out the bigger issue > was the performance of character-at-a-time and byte-at-a-time > conversions, not bulk conversion. My experience so far is that the character-at-a-time functions can cause severe performance penalties for external users, especially if the library is dynamically linked. If the C standard provides the bulk-conversion functions, performance would increase drastically for users desiring bulk conversion (because they do not have to write a loop around a dynamically-loaded function call to do conversions one-at-a-time). I am glad that musl has had similar experience, and would like to make the bulk functions available in musl too! My asking about intrinsics and such was that I have some optimizations using hand-vectorized instructions for some bulk cases. I will be more than happy to just contribute regular and readable plain C, though, and then revisit such functions if it turns out that vectorization with SIMD and other instructions for various platforms turns out to be worth it. My initial hunch is that it is, but I'm more than happy to focus on correctness first, extreme performance (maybe) later. > If we do adopt these functions, the right way to do it would be using > them to refactor the existing c16/c32 functions. Basically, for > example, the bulk of c16rtomb would become c16rtowc, and c16rtomb > would be replaced with a call to c16rtowc followed by wctomb. And the > string ones can all be simple loop wrappers. I would be more than happy to write the implementation as such! Most of the wchar_t functions will be very easy since musl and glibc chose the right wchar_t. (Talking to other vendors is going to be a much, much more difficult conversation...) Best Wishes, JeanHeyd Meneide
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.