Date: Tue, 1 Sep 2015 02:32:35 -0400 From: Rich Felker <dalias@...c.org> To: musl@...ts.openwall.com Subject: First feedback on new C locale problems So far I've gotten 2 reports of things breaking from the new C locale. The first was in Alpine: gpg-connect-agent: conversion from 'utf-8' to 'UTF-8-CODE-UNITS' not available and turned out to be caused by --disable-nls omitting setlocale, leading to nl_langinfo(CODESET) requesting the C locale's codeset name. This could be fixed by making iconv support "UTF-8-CODE-UNITS" and do something reasonable with it, but the second issue was uglier. In Void Linux: help2man: Unknown encoding 'UTF-8-CODE-UNITS' at /usr/bin/help2man line 56. (https://github.com/voidlinux/void-packages/issues/2425) Oddly this one did not affect Alpine, for the same reason the first one did: Alpine has gettext support turned off, and help2man omits the offending code: http://anonscm.debian.org/cgit/users/bod/help2man.git/tree/help2man.PL?id=9ce0caa4cf164261ddde3fe987a260f5ba0dd558#n117 (which is overriding the system locale with "C" by default) when gettext support is disabled. Being that this is Perl code and it's passing the charset name to Perl's conversion functions, we can't just work around this by adding a new charset alias to iconv. Note that fixing the broken programs to call setlocale properly and honor the user's locale would make them work in the normal case, but they would break again if the user explicitly invoked them with LC_CTYPE=C. Anyway, what I suspect is that we're going to find a fair number of programs are calling nl_langinfo(CODESET) without actually having set the locale properly. As long as they're not using multibyte functions in libc to process text, failing to have called setlocale is not such a bad thing; they can do character processing themselves if they know the intended encoding, using iconv or native UTF-8 code or whatever. And the situation we've got right now is that, despite best efforts not to impact users who don't intentionally _try_ to get a byte-based C locale, this functionality is causing actual regressions in musl's promise of "always UTF-8". What I'd like to do to fix it is just always return "UTF-8" for nl_langinfo(CODESET) regardless of locale (rather than returning "UTF-8-CODE-UNITS" when in C locale). POSIX places no requirements on nl_langinfo that would preclude this, and it seems like it would restore the desired properties and fix all the regressions. I don't mind leaving it as-is for a little bit while we discuss this though; hopefully we'll turn up some more interesting application bugs to get fixed. Rich
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.