Main Admin Dashboard Page on SaaS Instance (/admin/users) Is Slow To Load

[jimmybrancaccio]

Bug description

In the past 7 days or so the main page in the admin dashboard (/admin/users) has been much slower to than previously. Prior to about a week ago the page would load within 2-5 seconds. At this time it takes ~30 seconds for the page to load. While the page is loading searching is not possible.

Steps to reproduce

1. Clear your web browsers cache.
2. Visit https://gitpod.io/admin/users in your web browser while keeping an eye on the time.
3. Observe the extended time it takes for the page to complete loading and for searching to be possible.

https://www.loom.com/share/957360c6c38043839fd9c79ffe079520

Workspace affected

N/A

Expected behavior

I would expect that the page loads in a timely manner. It's possible it's pulling additional user data to display the rows of users on the page which is causing this additional delay. The page will likely need to be profiled to get a better understand of what is causing it to load so slowly.

Example repository

N/A

Anything else?

N/A

[jldec]

I also noticed that starting a search while the intial open-page query is still in progress will cause the search input to be lost and the search results to be replaced by the default (empty) results of the intial page load.

[jldec]

scheduled as this makes our admin dashboard painful to use.

[easyCZ]

When the page loads, it makes a request for

{"jsonrpc":"2.0","id":10,"method":"adminGetUsers","params":{"searchTerm":"","limit":50,"orderBy":"creationDate","offset":0,"orderDir":"desc"}}

Which takes around ~21 seconds.

Subsequent query for a term "easy"

{"jsonrpc":"2.0","id":11,"method":"adminGetUsers","params":{"searchTerm":"easy","limit":50,"orderBy":"creationDate","offset":0,"orderDir":"desc"}}

Takes ~7 seconds.

Removing the search term and searching again makes

{"jsonrpc":"2.0","id":12,"method":"adminGetUsers","params":{"searchTerm":"","limit":50,"orderBy":"creationDate","offset":0,"orderDir":"desc"}}

And takes ~21 seconds again.

This is caused by the underlying DB query being slow.

[easyCZ]

^ 90th percentile of the API call

From metrics, we can see that indeed this used to perform better and started happening around Sept 22 - metrics

[easyCZ]

This looks to be introduced with this commit - 29c88a7

[easyCZ]

Related to #13225