Date: Fri, 1 Apr 2016 22:46:25 -0400 From: Assaf Gordon <assafgordon@...il.com> To: Rich Felker <dalias@...c.org> Cc: musl@...ts.openwall.com Subject: Re: Possible bug in setlocale upon invalid LC_ALL value Hello Rich, thank you for the prompt and detailed response. > On Apr 1, 2016, at 20:58, Rich Felker <dalias@...c.org> wrote: > > On Fri, Apr 01, 2016 at 08:47:01PM -0400, Assaf Gordon wrote: >> I think I've encountered a problem in musl, where using setlocale with invalid locale name returns the invalid locale instead of a known locale. > > This is intentional. All locale names are valid under musl, and those > which don't have any particular definition are just aliases for > C.UTF-8. I will suggest a minor fix to GNU coreutils to accommodate for this current implementation. > The alternative would be that UTF-8 support breaks whenever > LC_* vars are set but locales are not installed/configured, which > would pretty much _always_ be the case when running a static-linked > standalone binary on a non-musl-based system (where LC_* are probably > set to something the main host libc recognizes). > > One possibility if this behavior is problematic would be to only > consider names without their own definitions as aliases for C.UTF-8 > when MUSL_LOCPATH is not set. However I think we'd need to see a > strong motivation for doing that, since it seems like it would be > worse behavior in some ways, especially when using LC_MESSAGES set to > a language for which you don't have a locale installed. I'm not an expert about locales to argue one way or the other. Naively, I would think that this is somewhat problematic, because a best-behaving program (one that checks set locale's return code for errors) has no way to warn the user that he/she used an invalid locale. Perhaps a work-around would be to handle it this way: if an invalid (non-existing) locale is given in LC_* env vars, setlocale(LC_ALL,"") should return NULL (indicating an error), then all other invocations of setlocale(LC_*,NULL) would return the "C.UTF-8" indicator. This would allow detecting the error, but not affect further processing (if invalid locales are already an alias to C.UTF-8). This seems to match other OSes/libcs which return fixed "C" in such cases. The reason for such check is that it is common user mistake to specify non-existing locales, then be confused by the seemingly incorrect results. Allowing a program to detect incorrect locales is a good mitigation. I'll side-step the non-UTF-8 locales (which would be a problem in the current musl auto-aliasing to UTF-8), and show one possible case where silent aliasing leads to incorrect results. consider the following UTF-8 string: M N Ñ O P Y Z Æ Ø Å (which includes Spanish eñe and the last three letters in the Swedish alphabet). When sorting with locale-aware programs, different locales should give different collation orders (e.g. es_ES.UTF-8 vs sv_FI.UTF-8). To reproduce: A='\116\n\303\221\n\117\n\120\n\131\n\132\n\303\205\n\303\204\n\303\226\n' printf "$A" | LC_ALL=sv_FI.UTF-8 sort printf "$A" | LC_ALL=es_ES.UTF-8 sort If a user has a typo in the locale name (e.g. sv_SV.UTF-8), there's no way for a program to detect it, and he will get unexpected ordered results. GNU coreutils' 'sort' program added a --debug option to help user diagnose such issues. On Linux with glibc, this will be the output: $ printf "$A" | LC_ALL=es_ES.UTF-8 sort --debug > /dev/null sort: using ‘es_ES.UTF-8’ sorting rules $ printf "$A" | LC_ALL=sv_FI.UTF-8 sort --debug > /dev/null sort: using ‘sv_FI.UTF-8’ sorting rules $ printf "$A" | LC_ALL=sv_SV.UTF-8 sort --debug > /dev/null sort: using simple byte comparison $ printf "$A" | LC_ALL=foobar sort --debug > /dev/null sort: using simple byte comparison The last two messages ("simple byte") is the hint that the locale is invalid, and sort will does not use it. On Alpine (linux + musl), there's no way to detect such case: $ printf "$A" | LC_ALL=sv_FI.UTF-8 gsort --debug > /dev/null gsort: using ‘sv_FI.UTF-8’ sorting rules $ printf "$A" | LC_ALL=sv_SV.UTF-8 gsort --debug > /dev/null gsort: using ‘sv_SV.UTF-8’ sorting rules $ printf "$A" | LC_ALL=foobar gsort --debug > /dev/null gsort: using ‘foobar’ sorting rules regards, - assaf
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.