Date: Sun, 22 Apr 2012 16:41:03 -0400 From: Rich Felker <dalias@...ifal.cx> To: musl@...ts.openwall.com Subject: Help establishing wctype character classes Hi, I'm looking for some help establishing the proper definitions (in terms of Unicode) for the different wctype character classes (isw*() functions) in musl. Some are relatively easy and already done: - Spaces: Unicode White_Space property, minus non-breaking spaces (which lack "space" semantics; that's their whole point) and script-specific, non-blank-glyph characters that Unicode oddly calls "spaces". - Control: The traditional C0/C1 control codes, plus some bidi and embedding control stuff. (Basically, anything that can't be printed literally in plain text and expected to work.) - Printable: Any valid codepoint that's not control. This broad definition is to avoid the situation where an application refuses to print valid text in the end user's language because it contains characters added to Unicode after the libc/app was published. It's assumed that the vast majority of, if not all, new characters added to Unicode will be non-control characters and non-problematic to print. - Graphic: Any printable character that's not a space. This definition is provided by the C standard. - Digit: Exactly the ASCII decimal digits '0'...'9' and nothing else. This definition is provided by the C standard. If anyone has objections to the above definitions, I'd be happy to hear them, but what I'm most concerned about is the remaining classes, specifically alphabetic and punctuation. Here are some tentative partial definitions: - Alphabetic: Derived property Alphabetic from Unicode's DerivedCoreProperties.txt. Not yet in musl. - Punctutation: Currently defined in musl as Unicode character classes P* and Sm, plus Pattern_Syntax from PropList.txt. However, these definitions omit a number of characters which should probably be in one or the other of these two classes, in particular at least: - All Unicode class N* (numeric) characters, like superscripts, vulgar fractions, Roman numerals, etc. - Combining diacritic characters that are not themselves letters. and I'm unclear on what should be done with them. Especially for numeric/digit-like characters, they cannot be classified as "digits" (per ISO C rules), and calling them letters may make more sense; glibc on the other hand considers them punctuation. As for combining diacritics, considering them letters if their intended usage is to combine onto letters would be convenient for handling NFD text with regular expressions (/[[:alpha:]]+/ will match full words even if the diacritics are decomposed), but I suspect it will also make some cases behave worse. Before anybody suggests just dumping what glibc does and copying it, glibc is horribly incorrect on a number of characters in ways that break internationalized usage. For instance, all Tibetan subjoined letter characters are wrongly classified as punctuation rather than as letters; this will break usage of regular expressions like /[[:alpha:]]+/ with affected scripts/languages. Rich
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.