New room list: Rooms go missing sometimes (lock breaking + sticky mishandling)

[turt2live]

see rageshakes

[jaller94]

Got this message that looked suspicious and was in the JS console for a room that had disappeared:
!redactedRoomId:server is reportedly new but is already known - assuming TagChange instead

https://github.com/matrix-org/riot-web-rageshakes/issues/3146

[jaller94]

• Reloading the tab resolves the issue.
• Sometimes, clicking on another room that's in my Favourites makes all missing rooms reappear.
• So far I only noticed this happening to my Favourites.
• So far I only noticed this happening to rooms (not DM rooms).
• So far I have not noticed a correlation to visible messages inside the room.
• So far 1 or 2 rooms have disappeared at the same time out of my 4 favourites.
• I'm also logged in with another Riot Web 1.6.8 (non-develop) version and the old Riot Android.

[jaller94]

If you've never see this (Bridge Crew was missing until I clicked on a room):

It reappeared with me clicking on a room. Other interactions like resizing the group or opening the settings of another room don't cause the missing room to disappear.

[turt2live]

So there's two common tracks here:

1. When accepting an invite, the room can be duplicated temporarily in two places. Switching away causes it to disappear.
2. Sometimes favourites just go missing. I think this is caused by the invite being accepted on a different device, and the sticky room getting lost.
   • Worth noting that while 100% of the reports are with favourites, I think that's just popularity bias rather than a problem with a specific sublist.

... now to verify these theories.

[turt2live]

So in @jaller94's rageshakes (3138, 3145, 3146) there's two rooms (!WC and !BK) which are flipping between visible and invisible, and it looks like they were caused by case 2: something happening for the room to be sticky and thus go missing for an external update like an invite or something. There's no apparent cause in the logs, so this is purely speculation based on the regeneration milliseconds prior to !WC is reportedly new but is already known - assuming TagChange instead, which then triggers a regeneration which dropped the room due to being sticky.

Bugs so far:

1. External updates manipulate the sticky room, causing NewRoom updates to trigger and regenerate the list.
   • When the list regenerates, it doesn't see the temporarily unstickied room and drops it for some reason, possibly because of a timing conflict with the sticky room still being set.
   • Because two rooms went missing, the switching between them caused the other to go missing due to the algorithm hitting the panic path and regenerating the entire list.

[turt2live]

3144 is the same as above.

3143 is paired with 3144, and the randomness appears to be caused by the room list desynchronizing more and more from reality.

3137 is reported as favourites getting smaller each click, but it appears to be the case described above. Something caused an update to fire and then with every click the sticky room desynchronized further and further.

3147 is the same as above.

---

3140 confirms the theory of invites causing duplication and thus stuckness. The room doesn't appear until 12:53 in the log (presumably the invite was received in a prior session), and when it does it is about accepting it. The js-sdk quickly sends us some timeline updates which rightfully get ignored until the room is handled as new (successfully), where it then received some decryption events. The decryption events were only 40ms after the room was added, so presumably the decryption process was just a tad bit slower than the room list. The room list correctly handles the updates, firing those updates for the room list to render.

The room appears fine until 13:10 when the room suddenly turns into "No tags known for !Hah", an indicator of some sort of desync. The log appears in the context of a filter in progress (not active from the looks of it, just statically waiting), which means that it might have bypassed a lock somewhere.

The room never recovered from the tags warning, which means that the room list store knew that the algorithm knew the room existed, but for some reason it eluded the tagging cache, causing it to go missing every time it tried to update.

3141 is a similar invite issue to this, with the same cause.

---

Additional bugs:

1. When filtering, we appear to by bypassing a lock which can cause random (potentially sticky) rooms to disappear. There's also evidence in the log to suggest that we are simply not locking when updates are received and while a filter is being recalculated (ie: 2 rooms being updated) which may cause a tag desync.