[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date: Sun, 22 Apr 2012 16:41:03 -0400
From: Rich Felker <dalias@...ifal.cx>
To: musl@...ts.openwall.com
Subject: Help establishing wctype character classes
Hi,
I'm looking for some help establishing the proper definitions (in
terms of Unicode) for the different wctype character classes (isw*()
functions) in musl. Some are relatively easy and already done:
- Spaces: Unicode White_Space property, minus non-breaking spaces
(which lack "space" semantics; that's their whole point) and
script-specific, non-blank-glyph characters that Unicode oddly calls
"spaces".
- Control: The traditional C0/C1 control codes, plus some bidi and
embedding control stuff. (Basically, anything that can't be printed
literally in plain text and expected to work.)
- Printable: Any valid codepoint that's not control. This broad
definition is to avoid the situation where an application refuses to
print valid text in the end user's language because it contains
characters added to Unicode after the libc/app was published. It's
assumed that the vast majority of, if not all, new characters added
to Unicode will be non-control characters and non-problematic to
print.
- Graphic: Any printable character that's not a space. This definition
is provided by the C standard.
- Digit: Exactly the ASCII decimal digits '0'...'9' and nothing else.
This definition is provided by the C standard.
If anyone has objections to the above definitions, I'd be happy to
hear them, but what I'm most concerned about is the remaining classes,
specifically alphabetic and punctuation. Here are some tentative
partial definitions:
- Alphabetic: Derived property Alphabetic from Unicode's
DerivedCoreProperties.txt. Not yet in musl.
- Punctutation: Currently defined in musl as Unicode character classes
P* and Sm, plus Pattern_Syntax from PropList.txt.
However, these definitions omit a number of characters which should
probably be in one or the other of these two classes, in particular at
least:
- All Unicode class N* (numeric) characters, like superscripts, vulgar
fractions, Roman numerals, etc.
- Combining diacritic characters that are not themselves letters.
and I'm unclear on what should be done with them. Especially for
numeric/digit-like characters, they cannot be classified as "digits"
(per ISO C rules), and calling them letters may make more sense; glibc
on the other hand considers them punctuation. As for combining
diacritics, considering them letters if their intended usage is to
combine onto letters would be convenient for handling NFD text with
regular expressions (/[[:alpha:]]+/ will match full words even if the
diacritics are decomposed), but I suspect it will also make some cases
behave worse.
Before anybody suggests just dumping what glibc does and copying it,
glibc is horribly incorrect on a number of characters in ways that
break internationalized usage. For instance, all Tibetan subjoined
letter characters are wrongly classified as punctuation rather than as
letters; this will break usage of regular expressions like
/[[:alpha:]]+/ with affected scripts/languages.
Rich
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
