eliminate duplicates in cache invalidation

[dbu]

because of other bugs, i ended up refreshing the same url 100 times. this pretty much knocks out apache. oviously my application should be more careful, and when i ask to invalidate 100 different urls i don't see how we could prevent this. however, i think it would be nice if we could eliminate duplicates and only refresh / invalidate each of them once. but as we store requests, this is a bit more tricky than just using array_unique.

[staabm]

Would it make sense to throttle the invalidation requests?

[ddeboer]

eliminate duplicates and only refresh / invalidate each of them once

We could store the queued requests hashed by URL and headers, as I think multiple invalidation requests to the same URL but with different headers should still be possible.

throttle the invalidation requests

Makes sense to me, but wouldn’t be strictly necessary if we remove duplicates (hopefully).

[dbu]

i like the idea of hashing by all things that make the requests distinct. its just nice if the client can be lazy with this. and yeah, just the url would not be enough, most notable for everything BAN. i will have a look.

about throttling, i am not so sure. if the multi request curl allows to set a maximum of parallel requests, we could use that (but this would probably be configured by passing in an accordingly configured guzzle client, no?). anything on our side seems too much to me.

in a really heavy setup, one could use a message queue to queue requests and then have workers that work on that queue. or simply invalidate / ban and queue refresh requests into a message queue. both could be solved with special proxy clients. but that seems out of scope for this bundle to me, as it would depend on the message queue used and when you have such a big setup you do want to fine-tune anyways.