[<prev] [next>] [day] [month] [year] [list]
Date: Fri, 13 Dec 2013 13:23:14 -0700
From: <writeonce@...ipix.org>
To: musl@...ts.openwall.com
Subject: RE: validation of utf-8 strings passed as system call arguments
On 12/13/2013 02:46 PM, Rich Felker wrote:
> On Fri, Dec 13, 2013 at 11:57:54AM -0700, writeonce@...ipix.org
> wrote:
>> There's no way to convert between UTF-8 and UTF-16 without
>> parsing/decoding the UTF-8, which includes validating it for free
>> if your parser is written properly. Failure to validate would lead
>> to all sorts of bugs, many of them dangerous, including things like
>> treating strings not containing '/', '\', ':', '.', etc. as if they
>> contained those characters, resulting in directory escape
>> vulnerabilities.
>>
>> Absolutely, and this is something that I am checking anyway. But
>> there is also the special case where an ill-formed utf-8 byte
>> sequence can still result in a valid code point, which can then be
>> safely converted to utf-16. These cases, which are generally known
>> as the problem of the "non shortest form," pertain to byte
>> sequences that used to be valid before Unicode version 3.1, but are
>> now forbidden, hence table 3-7 of the current (6.2) standard.
>
> What I was saying is that you don't have this problem if you're
> parsing/decoding UTF-8 correctly. And parsing it correctly is not
> harder/slower than doing it the way that results in misinterpreting
> illegal sequences as "non shortest form" for other characters. A
> good treatment of the subject (and near-optimal implementation) is
> here:
>
> http://bjoern.hoehrmann.de/utf-8/decoder/dfa/
>
> My implementation in musl is based on the same ideas (UTF-8 decoding
> as a state machine rather than complex conditionals) but I reduced
> the size of the state from two ints to just one and reduced the size
> of the state table significantly by essentially encoding the
> transitions and partial character values into the state values.
Thanks for the tips and reference. Once everything else is working I'll
certainly switch to a method that follows either your, or Hoehrmann's
optimization (which I'll admittedly need more than a few minutes to
understand...) For the time being I am leaving the set of conditionals
that follows the standard and table 3-7, as that is very easy to
implement. And with the target strings being relative shortness,
hopefully this won't even bear any real performance consequences.
>
> If you're making UTF-8 to UTF-16 conversions to feed to the Windows
> kernel filesystem code, I'd do them at the last possible opportunity
> before passing the strings to the kernel, and just generate a fake
> error equivalent to "file does not exist" or "invalid filename" if
> the conversion encounters any illegal sequences.
Indeed, that is exactly how I am doing this.
zg
>
> Rich
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
