reading invalid text data, other encodings

[JeffBezanson]

We probably need to change 1 or 2 behaviors when reading invalid data or unknown encodings. There are two cases: Char, and things like readuntil/readline.

readuntil can be reasonably defined in terms of bytes: just read everything until a certain value. This is good because then you can at least get the data without explicit support for every encoding. Currently we might return an invalid UTF8String, from which you can get the (unaltered) data. I don't know whether that is the best approach. Maybe there should be a lower-level routine that returns a byte array. We also need functions that do the same for different fixed-width encodings (16-bit, 32-bit).

Reading a Char I don't think can be done reasonably without knowing the encoding. The best immediate change I can think of is to give an error for invalid data while trying to read a UTF-8-encoded Char.

[JeffBezanson]

We could have readuntil(io, c) where the type of c determines what kind of units to read --- byte at a time, Char at a time, Int16 at a time, etc.

[pao]

UTF-8 is a variable-width encoding, though, right? If I'm trying to read up to a letter "l"/0x6C in a text that also happens to contain the simplified Chinese character ma3 ("horse"), which is 马/0x9A6C, that might be a problem if I'm not decoding it as UTF-8.

Does there need to be a fast byte-oriented readuntil() and a Unicode-aware readuntil()?

(Note: I do not know any Chinese. This just happens to be a frequent example character on Language Log, and also contains a byte in the low ASCII range.)

[StefanKarpinski]

I really think we need a layered approach here where you can get raw bytes and then build strings from those. Searching for a character without specifying an encoding makes no sense. Auto-detecting the encoding by default seems like it would be sufficient. It would ideally be done lazily in the sense that you could start with ASCII and promote a stream to Latin-1 or UTF-8 if and when a character that gives a clue is encountered (i.e. invalid UTF-8 implies Latin-1).

[JeffBezanson]

Yes, I was proposing a low-level readuntil that reads int8, int16, etc., since those can be useful independent of any text-related issues. Text reading functions can be built on top. Using readuntil with a Char delimiter is a different case. It has to know the encoding up front. But, for example, reading up to an ASCII character from a UTF-8 stream can use the byte-based readuntil.

[pao]

The point of the above Chinese character digression was that reading up to an ASCII character from a UTF-8 stream cannot use the byte-based readuntil, since ASCII characters appear in the middle of UTF-8 byte sequences which represent non-ASCII characters.

[nolta]

On Fri, Dec 21, 2012 at 12:45 AM, pao notifications@github.com wrote:

The point of the above Chinese character digression was that reading up to
an ASCII character from a UTF-8 stream cannot use the byte-based readuntil,
since ASCII characters appear in the middle of UTF-8 byte sequences which
represent non-ASCII characters.

"马" is [e9,a9,ac] in utf8.

-Mike

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1792#issuecomment-11603393.

[JeffBezanson]

In fact I believe searching for ascii characters in this way was one of the
major design considerations for utf-8.
On Dec 21, 2012 12:55 AM, "Mike Nolta" notifications@github.com wrote:

On Fri, Dec 21, 2012 at 12:45 AM, pao notifications@github.com wrote:

The point of the above Chinese character digression was that reading up
to
an ASCII character from a UTF-8 stream cannot use the byte-based
readuntil,
since ASCII characters appear in the middle of UTF-8 byte sequences
which
represent non-ASCII characters.

"马" is [e9,a9,ac] in utf8.

-Mike

—
Reply to this email directly or view it on GitHub<
https://github.com/JuliaLang/julia/issues/1792#issuecomment-11603393>.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1792#issuecomment-11603512.

[Keno]

@pao I think you might be confusing the unicode codepoint with the actual utf-8 encoding

[pao]

Apparently so. Nothing to see here, move along. Argh Unicode, so useful yet complicated.

[JeffBezanson]

Come to think of it, is there a good reason not to have Latin1String instead of ASCIIString? It is also a subset of unicode, and gives us more characters for almost no additional effort. It seems like a better default in case invalid utf-8 is encountered, since every byte array is valid latin-1.

[StefanKarpinski]

ASCII is a sub-encoding of UTF-8 whereas Latin-1 is not. Thus, you can concatenate ASCII and UTF-8 data and the result is the UTF-8 encoding of the input strings. With Latin-1 this doesn't work; instead you have to transcode the Latin-1 data first, expanding high bytes to UTF-8 two-bytes and then concatenate the data. You may recall that we originally had Latin1String and UTF8String but changed to ASCIIString for exactly these reasons.

[JeffBezanson]

Ok, maybe we should just add a function to convert latin-1 to utf-8.
On Dec 28, 2012 6:11 PM, "Stefan Karpinski" notifications@github.com
wrote:

ASCII is a sub-encoding of UTF-8 whereas Latin-1 is not. Thus, you can
concatenate ASCII and UTF-8 data and the result is the UTF-8 encoding of
the concatenation of the inputs. With Latin-1, this doesn't work; instead
you have to transcode the Latin-1 data first, expanding high bytes to UTF-8
two-byte characters before concatenating the data. You may recall that we
originally had Latin1String and UTF8String but changed to ASCIIString for
exactly these reasons.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1792#issuecomment-11744046.

[StefanKarpinski]

That doesn't really seem to solve the kinds of problems we've been seeing. Mostly we've been having problems with UTF-8 strings that contain invalid UTF-8 data like 0xff bytes (which is not the start of any valid UTF-8 character). Since 346bf5e we handle this pretty well, but regexes still don't handle it. For readuntil(io, c::Char) where io is UTF-8 encoded and c is an ASCII character, we should search for uint8(c) instead because that will a) be faster, b) handle cases where the UTF-8 data is invalid reasonably, and c) be close to working if the data is really Latin-1 that we're misinterpreting as UTF-8.

[JeffBezanson]

We already support reading until an int8; I will just reshuffle the
interface a bit. Adding the ability to interpret as latin1 is orthogonal to
the other stuff. Reading latin1 as invalid utf-8 is one of our problems.
On Dec 28, 2012 6:38 PM, "Stefan Karpinski" notifications@github.com
wrote:

That doesn't really seem to solve the kinds of problems we've been seeing.
Mostly we've been having problems with UTF-8 strings that contain invalid
UTF-8 data like 0xff bytes (which is not the start of any valid UTF-8
character). Since 346bf5ehttps://github.com/JuliaLang/julia/commit/346bf5eb2ffb279bf3e2b4102ffd7e7dfd9b34acwe handle this pretty well, but regexes still don't handle it. For readuntil(io,
c::Char) where io is UTF-8 encoded and c is an ASCII character, we should
search for uint8(c) instead because that will a) be faster, b) handle
cases where the UTF-8 data is invalid reasonably, and c) be close to
working if the data is really Latin-1 that we're misinterpreting as UTF-8.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1792#issuecomment-11744443.

[StefanKarpinski]

To make things really smooth, we probably need to do some degree of encoding autodetection. UTF-32 and UTF-16 are easy to autodetect; Latin-1 and UTF-8 are unfortunately both common and hard to distinguish. I was thinking that the way to go would be to start out with ASCII and promote the encoding whenever something that is unequivocally Latin-1 or UTF-8 occurs, which implies a design where the encoding of a stream can change on the fly.