feat: Add specialized `ApifyRequestQueueClientSimple` #573

Pijukatel · 2025-08-27T14:09:20Z

Description

Create two variants of the ApifyRequestQueueClient:
- ApifyRequestQueueClientFull - current version that supports multiple producers/consumers and locking of requests. More Apify API calls, higher API usage -> more expensive, slower.
- ApifyRequestQueueClientSimple - new constrained client for self-consumer and multiple constrained producers. (Detailed constraints in the docs). Fewer Apify API calls, lower API usage -> cheaper, faster.
Most of the ApifyRequestQueueClient tests were moved away from actor-based tests, so that they can be parametrized for both versions of the ApifyRequestQueueClients and to make local debugging easier.

Usage:

RequestQueue with full client:
await RequestQueue.open(storage_client=ApifyStorageClient(simple_request_queue=False))
RequestQueue with simple(default) client:
await RequestQueue.open(storage_client=ApifyStorageClient())

Stats difference:

The full client is doing significantly more API calls and regarding the API usage it is doing 50% more RequestQueue writes and also more RequestQueue reads.

Example rq related stats for crawler started with 1000 requests:
ApifyRequestQueueClientFull:
API calls: 2123
API usage: {'readCount': 1000, 'writeCount': 3000, 'deleteCount': 0, 'headItemReadCount': 0, 'storageBytes': 104035}

ApifyRequestQueueClientSimple:
API calls: 1059
API usage: {'readCount': 3, 'writeCount': 2000, 'deleteCount': 0, 'headItemReadCount': 14, 'storageBytes': 103826}

Issues

Part of: Add specialized/parametrized RequestQueueClients #513

Migrate most actor based tests to normal force cloud rq tests (for future parametrization of the Apify clients)

vdusek

high-level things

vdusek · 2025-09-19T11:29:31Z

src/apify/storage_clients/_apify/_request_queue_client_simple.py

+    - Only one client is consuming the request queue at the time.
+    - Multiple producers can put requests to the queue, but their forefront requests are not guaranteed to be handled
+      so quickly as this client does not aggressively fetch the forefront and relies on local head estimation.
+    - Requests are only added to the queue, never deleted. (Marking as handled is ok.)


Requests are only added to the queue, never deleted. (Marking as handled is ok.)

? I don't get it.

Well API has delete endpoint. We do not expose it in RQ, but if someone is calling it while we use this client to work on that RQ, then it will have unpredictable behavior.
https://docs.apify.com/api/v2/request-queue-request-delete

It is not a normal use case, but better to be explicit about it

vdusek · 2025-09-19T11:31:01Z

src/apify/storage_clients/_apify/_request_queue_client_simple.py

+    - Multiple producers can put requests to the queue, but their forefront requests are not guaranteed to be handled
+      so quickly as this client does not aggressively fetch the forefront and relies on local head estimation.
+    - Requests are only added to the queue, never deleted. (Marking as handled is ok.)
+    - Other producers can add new requests, but not modify existing ones (otherwise caching can miss the updates)


Other producers can add new requests, but not modify existing ones (otherwise caching can miss the updates)

Modify existing ones? What do you mean by that?

API has update endpoint. If someone (other producers) are updating existing requests and this client has already cached the requests locally, then the client will use the outdated request.
https://docs.apify.com/api/v2/request-queue-request-put

It is not a normal use case, but better to be explicit about it

vdusek · 2025-09-19T11:33:55Z

docs/04_upgrading/upgrading_to_v3.md

+```python
+from apify.storages import RequestQueue
+from apify.storage_clients import ApifyStorageClient
+
+async def main():
+    # Full client
+    rq_full = await RequestQueue.open(storage_client=ApifyStorageClient(simple_request_queue=False))
+    # Default optimized client
+    rq_simple = await RequestQueue.open(storage_client=ApifyStorageClient())
+```


We should use an example with Actor here.

I thought about that, but it is just wrapping it with Actor:
I can add it, but it felt like pointless clutter

vdusek · 2025-09-19T11:34:26Z

tests/integration/conftest.py

+        # Reset the Actor class state.
+        apify._actor.Actor.__wrapped__.__class__._is_any_instance_initialized = False  # type: ignore[attr-defined]
+        apify._actor.Actor.__wrapped__.__class__._is_rebooting = False  # type: ignore[attr-defined]


Why do we need this now?

We don't need it, but I saw a warning log in some tests and realized that we do not isolate the tests so well because is_any_instance_initialized and _is_rebooting were leaking from the previous tests

This warning could be observed in tests
WARN Repeated Actor initialization detected - this is non-standard usage, proceed with care

vdusek · 2025-09-19T11:47:10Z

src/apify/storage_clients/_apify/_storage_client.py

+    def __init__(self, *, simple_request_queue: bool = True) -> None:
+        """Initialize the Apify storage client.
+
+        Args:
+            simple_request_queue: If True, the `create_rq_client` will always return `ApifyRequestQueueClientSimple`,
+                if false it will return `ApifyRequestQueueClientFull`. Simple client is suitable for single consumer
+                scenarios and makes less API calls. Full client is suitable for multiple consumers scenarios at the
+                cost of higher API usage
+        """


As we discussed on Slack, I'm more for:

def __init__(self, *, access: Literal['single', 'shared' = 'single') -> None:

also taking into account potential RQ v3...

vdusek · 2025-09-19T11:48:41Z

src/apify/storage_clients/_apify/_storage_client.py

+from ._request_queue_client_full import ApifyRequestQueueClientFull
+from ._request_queue_client_simple import ApifyRequestQueueClientSimple


And the naming of these internal classes...

If we go with the access literal parameter, maybe we can use:

ApifyRequestQueueSingleClient

ApifyRequestQueueSharedClient

vdusek · 2025-09-19T11:58:22Z

tests/integration/conftest.py

-@pytest.fixture
-async def request_queue_force_cloud(apify_token: str, monkeypatch: pytest.MonkeyPatch) -> AsyncGenerator[RequestQueue]:
+@pytest.fixture(params=[False, True])
+async def default_request_queue_apify(


why not just request_queue_apify?

vdusek · 2025-09-19T12:01:46Z

tests/integration/test_request_queue.py

Wait, you are removing the Actor tests (!?).

With the current framework, the integration tests are full end-to-end tests running on the platform as Actors.

What you're proposing changes that (!).

I think there are several levels of integration tests.

level - API

These are, for example, RequestQueue specific tests that do not mock API and are doing real API calls.

level - Platform

Actor tests that are integration of Apify platform and also API

level - Crawlee on Platform

Crawler running in Actor. Test integration of Crawlee with Apify Platform and API

What I have done to most RQ specific tests is to move them from level 2 and 3 to level 1. I think we should have more tests on lower levels and fewer tests on higher levels

Pijukatel added 5 commits August 26, 2025 13:26

Draft for tests

8e2f5d4

Updated draft

1d869a4

Try to use list_head

08df986

Locks not needed with in_progress

6131fff

Add alternate client

553663a

github-actions bot assigned Pijukatel Aug 27, 2025

github-actions bot added this to the 122nd sprint - Tooling team milestone Aug 27, 2025

github-actions bot added the t-tooling Issues with this label are in the ownership of the tooling team. label Aug 27, 2025

Pijukatel changed the title ~~No locking queue~~ feat: No locking queue Aug 27, 2025

Pijukatel added 2 commits August 28, 2025 13:03

WIP

eadab26

Find the chacing problem.

249f8f5

Migrate most actor based tests to normal force cloud rq tests (for future parametrization of the Apify clients)

Pijukatel changed the title ~~feat: No locking queue~~ feat: Add specialized ApifyRequestQueueClientSimple Aug 28, 2025

Pijukatel requested a review from vdusek August 28, 2025 13:09

Merge remote-tracking branch 'origin/master' into no-locking-queue

4ada123

github-actions bot added the tested Temporary label used only programatically for some analytics. label Aug 28, 2025

Pijukatel requested a review from janbuchar August 28, 2025 13:15

Pijukatel added 4 commits August 28, 2025 15:51

Wip changes

10e0652

Add init cache test, update upgrading guide

359c46e

Merge remote-tracking branch 'origin/master' into no-locking-queue

ce090c0

Finalize change and add few more tests

b511011

Pijukatel marked this pull request as ready for review September 19, 2025 08:57

Pijukatel added 2 commits September 19, 2025 11:06

Merge remote-tracking branch 'origin/master' into no-locking-queue

fb32861

Remove unnecesary methods from the specialized client

7ec13ef

Pijukatel mentioned this pull request Sep 19, 2025

Investigate caching options in ApifyRequestQueueClient #550

Open

vdusek requested changes Sep 19, 2025

View reviewed changes

Pijukatel added 4 commits September 19, 2025 16:14

Merge remote-tracking branch 'origin/master' into no-locking-queue

10bc7e2

Rename default_request_queue_apify

7712410

Use single and shared literals and rename the RQ client classes

e63f546

Merge remote-tracking branch 'origin/master' into no-locking-queue

ffa70ff

Pijukatel requested a review from vdusek September 19, 2025 14:58

		from ._request_queue_client_full import ApifyRequestQueueClientFull
		from ._request_queue_client_simple import ApifyRequestQueueClientSimple

feat: Add specialized ApifyRequestQueueClientSimple #573

Are you sure you want to change the base?

feat: Add specialized ApifyRequestQueueClientSimple #573

Uh oh!

Conversation

Pijukatel commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Usage:

Stats difference:

Issues

Uh oh!

vdusek left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Pijukatel Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Pijukatel Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

feat: Add specialized `ApifyRequestQueueClientSimple` #573

feat: Add specialized `ApifyRequestQueueClientSimple` #573

Pijukatel commented Aug 27, 2025 •

edited

Loading

Pijukatel Sep 19, 2025 •

edited

Loading

Pijukatel Sep 19, 2025 •

edited

Loading