Date: Fri, 13 Dec 2013 13:23:14 -0700 From: <writeonce@...ipix.org> To: musl@...ts.openwall.com Subject: RE: validation of utf-8 strings passed as system call arguments On 12/13/2013 02:46 PM, Rich Felker wrote: > On Fri, Dec 13, 2013 at 11:57:54AM -0700, writeonce@...ipix.org > wrote: >> There's no way to convert between UTF-8 and UTF-16 without >> parsing/decoding the UTF-8, which includes validating it for free >> if your parser is written properly. Failure to validate would lead >> to all sorts of bugs, many of them dangerous, including things like >> treating strings not containing '/', '\', ':', '.', etc. as if they >> contained those characters, resulting in directory escape >> vulnerabilities. >> >> Absolutely, and this is something that I am checking anyway. But >> there is also the special case where an ill-formed utf-8 byte >> sequence can still result in a valid code point, which can then be >> safely converted to utf-16. These cases, which are generally known >> as the problem of the "non shortest form," pertain to byte >> sequences that used to be valid before Unicode version 3.1, but are >> now forbidden, hence table 3-7 of the current (6.2) standard. > > What I was saying is that you don't have this problem if you're > parsing/decoding UTF-8 correctly. And parsing it correctly is not > harder/slower than doing it the way that results in misinterpreting > illegal sequences as "non shortest form" for other characters. A > good treatment of the subject (and near-optimal implementation) is > here: > > http://bjoern.hoehrmann.de/utf-8/decoder/dfa/ > > My implementation in musl is based on the same ideas (UTF-8 decoding > as a state machine rather than complex conditionals) but I reduced > the size of the state from two ints to just one and reduced the size > of the state table significantly by essentially encoding the > transitions and partial character values into the state values. Thanks for the tips and reference. Once everything else is working I'll certainly switch to a method that follows either your, or Hoehrmann's optimization (which I'll admittedly need more than a few minutes to understand...) For the time being I am leaving the set of conditionals that follows the standard and table 3-7, as that is very easy to implement. And with the target strings being relative shortness, hopefully this won't even bear any real performance consequences. > > If you're making UTF-8 to UTF-16 conversions to feed to the Windows > kernel filesystem code, I'd do them at the last possible opportunity > before passing the strings to the kernel, and just generate a fake > error equivalent to "file does not exist" or "invalid filename" if > the conversion encounters any illegal sequences. Indeed, that is exactly how I am doing this. zg > > Rich
Powered by blists - more mailing lists
Confused about mailing lists and their use? Read about mailing lists on Wikipedia and check out these guidelines on proper formatting of your messages.