Limited message history unless I increase federated room backfill limit

[s0div]

Description:

I tried creating a new server and setting up federation between the old and the new servers. When a user is invited to a federated room, they don’t see the full room history — it seems that federation only provides a limited amount of history (about 100 messages). The room’s history visibility is set to “since the point in time this option was selected”, so I’m not referring to rooms where history visibility is set to “since they were invited” or “since they joined.”

After checking the Synapse source code, specifically synapse/handlers/federation.py, I found that the function try_backfill has a hardcoded limit=100.

synapse/synapse/handlers/federation.py

Lines 446 to 540 in dc7f01f

	async def try_backfill(domains: StrCollection) -> bool:
	# TODO: Should we try multiple of these at a time?
	# Number of contacted remote homeservers that have denied our backfill
	# request with a 4xx code.
	denied_count = 0
	# Maximum number of contacted remote homeservers that can deny our
	# backfill request with 4xx codes before we give up.
	max_denied_count = 5
	for dom in domains:
	# We don't want to ask our own server for information we don't have
	if self.is_mine_server_name(dom):
	continue
	try:
	await self._federation_event_handler.backfill(
	dom, room_id, limit=100, extremities=extremities_to_request
	)
	# If this succeeded then we probably already have the
	# appropriate stuff.
	# TODO: We can probably do something more intelligent here.
	return True
	except NotRetryingDestination as e:
	logger.info("_maybe_backfill_inner: %s", e)
	continue
	except FederationDeniedError:
	logger.info(
	"_maybe_backfill_inner: Not attempting to backfill from %s because the homeserver is not on our federation whitelist",
	dom,
	)
	continue
	except (SynapseError, InvalidResponseError) as e:
	logger.info("Failed to backfill from %s because %s", dom, e)
	continue
	except HttpResponseException as e:
	if 400 <= e.code < 500:
	logger.warning(
	"Backfill denied from %s because %s [%d/%d]",
	dom,
	e,
	denied_count,
	max_denied_count,
	)
	denied_count += 1
	if denied_count >= max_denied_count:
	return False
	continue
	logger.info("Failed to backfill from %s because %s", dom, e)
	continue
	except CodeMessageException as e:
	if 400 <= e.code < 500:
	logger.warning(
	"Backfill denied from %s because %s [%d/%d]",
	dom,
	e,
	denied_count,
	max_denied_count,
	)
	denied_count += 1
	if denied_count >= max_denied_count:
	return False
	continue
	logger.info("Failed to backfill from %s because %s", dom, e)
	continue
	except RequestSendFailed as e:
	logger.info("Failed to get backfill from %s because %s", dom, e)
	continue
	except Exception as e:
	logger.exception("Failed to backfill from %s because %s", dom, e)
	continue
	return False
	# If we have the `processing_start_time`, then we can make an
	# observation. We wouldn't have the `processing_start_time` in the case
	# where `_maybe_backfill_inner` is recursively called to find any
	# backfill points regardless of `current_depth`.
	if processing_start_time is not None:
	processing_end_time = self.clock.time_msec()
	backfill_processing_before_timer.labels(
	**{SERVER_NAME_LABEL: self.server_name}
	).observe((processing_end_time - processing_start_time) / 1000)
	success = await try_backfill(likely_domains)
	if success:
	return True
	# TODO: we could also try servers which were previously in the room, but
	# are no longer.
	return False

Potential solutions

Would it be possible to change this limit through homeserver.yaml? Or is there a specific reason why this value is hardcoded?

[MadLittleMods]

@s0div The root of the problem likely isn't the limit=100 on backfill.

To explain what's happening, when the client uses the /messages endpoint, Synapse checks if it's approaching a gap in the history, and if so, will backfill more events from the federation to fill in the response. This means your client can paginate the history all the way up to the beginning of the room as it will backfill over and over for each /messages request as you go back in time. Synapse-wise, this works fine for new room joins regardless of the limit used in /messages or backfill, see matrix-org/complement#816 for concrete Complement tests for this scenario.

But there is a similar scenario where you re-join a room that you were previously joined to that does have problems. In this case, if you make a /messages?limit=500 request (where the limit query parameter is larger than the backfill limit=100 we use within Synapse), Synapse will backfill the limit=100 and return those 100 new events + all of the old history leaving an invisible gap in between. So your client goes from 100 new events, [gap], the old history from when you were previously joined. This is what MSC3871: Gappy timelines is trying to address. This is a hole in the spec as there is no way for a homeserver indicate gaps to the client so they can paginate the gap and cause the homeserver to backfill more.

In terms of why you're missing history. Which join scenario are we dealing with (compare to what I explained above)? What client are you using? (Element Web has bugs around this for example)

[s0div]

No, I am not rejoining the same room, it is a fresh room join. I also tried to join the same room with another user, and I am still not seeing the full message history and experiencing the same issue.

As I understand from your answer, by default synapse tries to sync the full message history during the initial sync when a user first joins a room, but this does not seem to work in my case.

The room I am trying to join is not newly created—it has a history of messages. Also, the room has a retention policy and is encrypted, if that helps.

I am having this issue on both the web and desktop apps.

[MadLittleMods]

I am having this issue on both the web and desktop apps.

I assume Element Web and Desktop? Both are using the same codebase (desktop is just a wrapped version of the web app).

Element Web has a multitude timeline bugs that could be causing this issue.

@s0div To narrow down a client vs homeserver issue, we would need to inspect the network requests happening. Specifically the /messages requests for the room in question. What would be obvious is if you scrolled up to the gap and no /messages requests were made (client issue).

When it happens, you could Submit debug logs in Element using the /rageshake command (include a link to this issue). I'm not sure if the logs will have enough detail or not but I'll assume the Element Web team can diagnose if they look at them. Otherwise, it will be on you to narrow things down further.

[s0div]

It happened in every room with every user. If it helps, I tried increasing the backfill limit in source code in my testing environment, and it loaded the full room history. If I change it back, some parts of the history have disappeared.

[MadLittleMods]

@s0div Please share a set of Synapse logs that include all of the /messages requests before your change and scrollback until you get stuck. And then a fresh set of logs after you make your change and scrollback into the history.

These kind of log lines should be included:

2025-09-30 19:30:47,899 - synapse.access.http.8817 - 521 - INFO - GET-121 - 127.0.0.1 - 8817 - {@anon-20250930_193014-38:127.0.0.1:8839} Processed request: 0.039sec/0.000sec (0.011sec, 0.002sec) (0.005sec/0.019sec/20) 3221B 200 "GET /_matrix/client/v3/rooms/!TGmGiCHpooInlGVpAZ:127.0.0.1:8839/messages?from=s123_10_0_1_1_1_1_85_0_1_2&dir=b&access_token=<redacted> HTTP/1.1" "Perl + Net::Async::HTTP/0.50" [7 dbevts]

[s0div]

I am sharing exported logs using /rageshake (scrolled up till I see no messages):
log-2.txt

[MadLittleMods]

Which room ID did you reproduce the problem in?

# `awk '!seen[$0]++'` is giving the unique list (`uniq` only works on adjacent lines being the same)
$ cat ~/Downloads/log-2.txt | grep /messages | sed -nE 's/.*\/([^\/]*?)\/messages.*/\1/p' | awk '!seen[$0]++'
!hAplNFVtMiCazgzwpW%3Aoldserver
!QhXfDRqUnLYdRfAJUo%3Aoldserver
!LuOVJTAkgcZFogwzwK%3Aoldserver
!rFtNXaBoDQcHRLBmVG%3Aoldserver
!BdvJOjGkVmOCDlDFen%3Aoldserver
!hNTCeedoanJCYBQzhC%3Aoldserver
!EFbDTocSuhzsTFoeBG%3Aoldserver
!UPAfBdynNkXhMrkqWO%3Aoldserver
!TgNrOYnDhMllIYHmdn%3Aoldserver
!smuBOrHVhNylzLqHpp%3Aoldserver
!BjqgssxqxXDzXEPYFZ%3Aoldserver
!jkFNdYJRHXAXoTYltG%3Anewserver.com
!ldETomPcsNaCkTzkzJ%3Aoldserver
!TIXfAvTNRYVBFJoMbC%3Aoldserver
!IvpARcSjaficWRTwQC%3Aoldserver

[s0div]

In this !ldETomPcsNaCkTzkzJ%3Aoldserver room.