`mb_detect_encoding()` detects UTF-8 emoji byte sequence as ISO-8859-1 since PHP 8.1

[filecage]

Description

I have a piece of code that tries to normalise the encoding of incoming strings using mb_detect_encoding(). While upgrading to PHP 8.1, I've noticed that a test which ensures that an urlencoded UTF-8 sequence (party hat emoji) now fails. It all comes down to a behaviour change of mb_detect_encoding() when passing UTF-8 and ISO-8859-1 (in which ever order) as $encodings in PHP 8.1.

I know that the way this method works (or the way that determining the enconding of a string works at all) can not be 100% realiable, so I'd also agree on not classifying this as a bug. However, it is an undocumented behaviour change introduced in 8.1 that might break existing code as it did with mine.

I assume that this change has been introduced with 28b346b.

Example

See https://3v4l.org/RgdfE

<?php
echo mb_detect_encoding('🥳', ['UTF-8', 'ISO-8859-1']);

Resulted in this output:

ISO-8859-1

But I expected this output instead:

UTF-8

PHP Version

8.1.1

Operating System

No response

[alexdowad]

Hi, @filecage, and thanks for much for this report. Hope to look into it very soon.

[alexdowad]

Hmm. One point to start:

Legacy versions of PHP return UTF-8, not because they are smart enough to tell that the string is actually UTF-8 text, but because you put UTF-8 first in the list of candidate encodings. If you put ISO-8859-1 first, then all versions return ISO-8859-1.

The string is actually valid under both encodings; if interpreted as ISO-8859-1, it's an accented "o", followed by an accented "Y", followed by a Yen sign, followed by a superscript digit 3. A funny combination, but not invalid.

The current detection code treats all codepoints higher than U+FFFF as "rare", and penalizes candidate encodings under which the string contains such codepionts. That is what is making it prefer ISO-8859-1 over UTF-8.

@nikic Do you think I need to teach it that some emoji over U+FFFF are not "rare"?

Another point... no matter what we do, encoding detection will always be very, very unreliable for short strings. If you want to auto-detect text encoding with any level of reliability, you really need to give it more input text to work with.

Even so, let us think a bit about this issue of emoji. It is a fact that some emoji are used by a lot of people, and they do occupy ranges of Unicode codepoints over U+FFFF.

[filecage]

Thanks a lot @alexdowad for looking into this!

Legacy versions of PHP return UTF-8, not because they are smart enough to tell that the string is actually UTF-8 text, but because you put UTF-8 first in the list of candidate encodings. If you put ISO-8859-1 first, then all versions return ISO-8859-1.

When I saw that the output was different, I did not think that previous versions might be accidentally returning the correct results.

Even so, let us think a bit about this issue of emoji. It is a fact that some emoji are used by a lot of people, and they do occupy ranges of Unicode codepoints over U+FFFF.

I haven't yet had a chance to look deeply into it but I could imagine that if we're somehow able to reliably detect emojis in a string, it's a very strong indicator for a string to be UTF-8, even for short strings. It might not be worth the effort though.

[jurihahn]

I have the same problem, if text starts with emoji, it's not detected as UTF-8 (I am using Docker php:8.1.6-fpm)

[titanz35]

I have a smiliar problem too, with a custom test suit in PhpUnit that throw us an error.

The test check if mb_detect_encoding find the correct encoding with an UTF-8 file content, with a BOM header.

You can see the simplify test here :
https://3v4l.org/83vWZ

The problem is present in php 8.1 and 8.2 too.

[alexdowad]

@titanz35's problem has nothing to do with emoji; rather, it's about detecting encoding for strings which use a BOM. I have just prepared a fix for that.

[alexdowad]

@filecage, I am just amending mb_detect_encoding so it accepts the most basic emoji like "smiling face" and "frowning face". But I have seen that your test case is actually U+1F973 FACE WITH PARTY HORN AND PARTY HAT.

Do you think that qualifies as a 'common' and thus expected character to be found in input strings for mb_detect_encoding?

[filecage]

@alexdowad according to unicode, U+1F973 is categorized as basic emoji: https://unicode.org/Public/emoji/15.0/emoji-sequences.txt

When it comes to what I'd expect from mb_detect_encoding(), I would probably go with: either all or nothing. I know that that's a tough one, but I assume that people (including myself) won't expect mb_detect_encoding() to work with with some, but not all.

Just out of personal interest, what would be a good reason not to include all unicode standardized emojis? Is it that it would slow down the detection itself or that we have no reliable way of having an exhaustive list of existing emojis? Or is it something else?