Date: Fri, 13 Dec 2013 14:46:40 -0500 From: Rich Felker <dalias@...ifal.cx> To: musl@...ts.openwall.com Subject: Re: validation of utf-8 strings passed as system call arguments On Fri, Dec 13, 2013 at 11:57:54AM -0700, writeonce@...ipix.org wrote: > There's no way to convert between UTF-8 and UTF-16 without > parsing/decoding the UTF-8, which includes validating it for free if > your parser is written properly. Failure to validate would lead to all > sorts of bugs, many of them dangerous, including things like treating > strings not containing '/', '\', ':', '.', etc. as if they contained > those characters, resulting in directory escape vulnerabilities. > > Absolutely, and this is something that I am checking anyway. But there is > also the special case where an ill-formed utf-8 byte sequence can still > result in a valid code point, which can then be safely converted to > utf-16. These cases, which are generally known as the problem of the "non > shortest form," pertain to byte sequences that used to be valid before > Unicode version 3.1, but are now forbidden, hence table 3-7 of the current > (6.2) standard. What I was saying is that you don't have this problem if you're parsing/decoding UTF-8 correctly. And parsing it correctly is not harder/slower than doing it the way that results in misinterpreting illegal sequences as "non shortest form" for other characters. A good treatment of the subject (and near-optimal implementation) is here: http://bjoern.hoehrmann.de/utf-8/decoder/dfa/ My implementation in musl is based on the same ideas (UTF-8 decoding as a state machine rather than complex conditionals) but I reduced the size of the state from two ints to just one and reduced the size of the state table significantly by essentially encoding the transitions and partial character values into the state values. If you're making UTF-8 to UTF-16 conversions to feed to the Windows kernel filesystem code, I'd do them at the last possible opportunity before passing the strings to the kernel, and just generate a fake error equivalent to "file does not exist" or "invalid filename" if the conversion encounters any illegal sequences. Rich
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.