Default low items-per-page `limit` leads to bad performance

[gjoseph92]

While converting some stackstac examples to pystac-client for gjoseph92/stackstac#81, I noticed some queries that were substantially slower with pystac-client than the equivalent query with sat-search. Not as bad as stac-utils/pystac#546, but 4x slower:

import satsearch
import pystac_client

%%time
items = satsearch.Search(
    url="https://earth-search.aws.element84.com/v0",
    intersects=dict(type="Point", coordinates=[-106, 35.7]),
    collections=["sentinel-s2-l2a-cogs"],
    datetime="2019-01-01/2020-01-01"
).items()
# CPU times: user 110 ms, sys: 31.9 ms, total: 142 ms
# Wall time: 2.23 s

%%time
items = pystac_client.Client.open(
    "https://earth-search.aws.element84.com/v0"
).search(
    intersects=dict(type="Point", coordinates=[-106, 35.7]),
    collections=["sentinel-s2-l2a-cogs"],
    datetime="2019-01-01/2020-01-01"
).get_all_items()
# CPU times: user 334 ms, sys: 59.5 ms, total: 394 ms
# Wall time: 7.05 s

Profiling, I realized pystack-client is fetching substantially more pages. sat-search sets limit=10000 by default for you, so this search required just a single request. Whereas pystac-client does not specify a limit by default, and ends up getting only 10 elements per page. Indeed, setting limit=10000 for pystack-client gives the same performance:

%%time
items = pystac_client.Client.open(
    "https://earth-search.aws.element84.com/v0"
).search(
    intersects=dict(type="Point", coordinates=[-106, 35.7]),
    collections=["sentinel-s2-l2a-cogs"],
    datetime="2019-01-01/2020-01-01",
    limit=10000,
).get_all_items()
# CPU times: user 125 ms, sys: 22.9 ms, total: 148 ms
# Wall time: 1.59 s

I think we should set a default value for limit in pystac-client, to something high (like 10,000). It makes things work out-of-the-box better, and from an API compliance perspective I think it's a valid choice.

Justification

The docstring for limit in pystac-client says

The maximum number of items to return per page. Defaults to None, which falls back to the limit set by the service.

This is slightly misleading: it uses the default client limit set by the service, not the service's own limit for the maximum number of items it's willing to send per page.

Looking at the STAC API spec:

Parameter	Type	Source API	Description
limit	integer	OAFeat	The maximum number of results to return (page size). Defaults to 10

Hence where the 10 comes from.
If we read about limits in the OGC Features API spec (OAFeat) [emphasis mine]:

So (using the default/maximum values of 10/10000 from the OpenAPI fragment in requirement /req/core/fc-limit-definition):

• If you ask for 10, you will get 0 to 10 (as requested) and if there are more, a next link;
• If you don’t specify a limit, you will get 0 to 10 (default) and if there are more, a next link;
• If you ask for 50000, you might get up to 10000 (server-limited) and if there are more, a next link;
• If you follow the next link from the previous response, you might get up to 10000 additional features and if there are more, a next link.

Basically, the limit parameter is meant to protect the client from getting too much data at once. The server will protect itself from processing too much. So pystac-client should set limit to the maximum number of items it's reasonably willing to process at once. Of course this could vary based on the data source, but I feel like >10 is a reasonable default. But I don't know what the policy is in pystac-client around setting defaults that aren't derived from the STAC spec.

[matthewhanson]

It's important to point out the difference between limit and max_items parameters here to pystac-client.

limit is a STAC API parameter and controls the number of items per page. In practice, based on the server, there is a cap to this number. Once the response gets too large an error may be generated. And by default, each server defaults to a different number. Earth-search defaults to 10, which definitely results in a large number of pages for large requests, and MS Planetary Computer defaults to 250, which works well if not a bit large if you are browsing and using a JSON renderer. At some point this number can become too large for an API, which will lead to an error due to payload size or API timeout. For stac-server this number is currently somewhere between 250 and 1000...maybe more but depends on the size of the individual items. For CMR-STAC this max # is more like 250, as anything larger can lead to a timeout.

max_items is not a STAC API param, it is specific to pystac-client, and controls the total number of items that will be fetched and is unrelated to limit or the total number of pages.

Note that in sat-search the limit parameter is actually the equivalent of max_items and page_limit is the same as limit.

I agree that pystac-client should default to something sensible in the case where the default server limit is not set, but it also should not risk having something too large. I'm setting it to 100 for now in the 0.3.1 release, but think it could probably be set larger.