Flag emojis for England, Scotland, and Wales get turned into black flags

[duxovni]

Steps to reproduce

1. Start composing a message
2. Open the emoji picker
3. Go to the very end of the flags section, where 🏴󠁧󠁢󠁥󠁮󠁧󠁿 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🏴󠁧󠁢󠁷󠁬󠁳󠁿 are
4. Click one of those emojis

Alternatively:

1. Start composing a message
2. Paste in a 🏴󠁧󠁢󠁥󠁮󠁧󠁿, 🏴󠁧󠁢󠁳󠁣󠁴󠁿, or 🏴󠁧󠁢󠁷󠁬󠁳󠁿 emoji from the clipboard

Outcome

What did you expect?

The emoji are displayed properly in the message composer and in the timeline.

What happened instead?

All three of these emoji appear as black flags in the message composer and in the timeline after sending.

It still takes 7 backspace keystrokes to delete them in the composer, which suggests that all of the unicode codepoints are still present, but for some reason only the initial U+1F3F4 Black Flag is displayed, and the remaining codepoints are hidden rather than being combined into the flag properly. The emoji also display properly in "View Source".

I've reproduced this behavior on

• develop.element.io in Firefox
• Element Desktop 1.10.11 on NixOS Unstable
• Element Desktop 1.10.12 on Arch Linux

My guess is this is an unrelated issue from #16884:

• This is reproducible across platforms and between browser/desktop, while the trans flag is only broken in Arch Linux's Element Desktop package
• The 🏴󠁧󠁢󠁥󠁮󠁧󠁿 🏴󠁧󠁢󠁳󠁣󠁴󠁿 🏴󠁧󠁢󠁷󠁬󠁳󠁿 emojis display properly in the emoji picker and in View Source, whereas the trans flag in Arch Desktop is broken into its constituent glyphs everywhere.
• As mentioned, these emoji take 7 backspace keystrokes to delete because the joiner characters aren't being combined properly. The broken trans flag still only takes one keystroke to delete, which suggests it's more an issue around fonts/display.

Operating system

No response

Browser information

No response

URL for webapp

No response

Application version

No response

Homeserver

No response

Will you send logs?

No

[simpleAndrew]

Hey there!

Any plans to get this fixed somehow? :)

[Lamby777]

Hey, I don't know what this project is, so this probably isn't very helpful, but it's worth mentioning since nobody so far has brought up Windows. I found this thread from a Google search when I noticed Flagpedia was showing all the emojis as 2-letter character codes except for England, Scotland, and Wales, which showed up as a black flag. Pretty sure this is a Windows thing, since they refuse to add flag emojis.

[Half-Shot]

I suspect this is due to us splitting emoji's in messages into mx_emoji span tags, and doing a very poor job of complex emoji. https://github.com/matrix-org/matrix-react-sdk/blob/7d0c68aa2922ba6e2d0134ab0999698323bb1f27/src/HtmlUtils.tsx#LL54C1-L57C74 seems to find emoji by regex, and in practice it seems to break things like 🏴󠁧󠁢󠁷󠁬󠁳󠁿. Room previews and the emoji picker actually do render this fine, so the flag isn't broken entirely, it's just the way we process it.

[t3chguy]

We can probably use lodash's split which has proper grapheme splitting

[Johennes]

Looks like we are already doing that (https://github.com/matrix-org/matrix-react-sdk/blob/7d0c68aa2922ba6e2d0134ab0999698323bb1f27/src/HtmlUtils.tsx#L474) but lodash itself can't properly handle that emoji.

for (const char of split("🏴󠁧󠁢󠁷󠁬󠁳󠁿", "")) {
    console.log(`char: ${char}`)
    console.log(`codepoint: ${char.codePointAt(0).toString(16)}`);
}

...

char: 🏴 
codepoint: 1f3f4 
char: 󠁧 
codepoint: e0067 
char: 󠁢 
codepoint: e0062 
char: 󠁷 
codepoint: e0077 
char: 󠁬 
codepoint: e006c 
char: 󠁳 
codepoint: e0073 
char: 󠁿 
codepoint: e007f

Those are all the the code points in "🏴󠁧󠁢󠁷󠁬󠁳󠁿", just not split correctly.