Cache seems to increase to infinity

[alexander-schranz]

We are using the following cache configuration in sulu:

Configuration.yaml

    sessions:
        default:
            backend:
                caches:
                    meta: doctrine_phpcr.meta_cache_provider
                    nodes: doctrine_phpcr.nodes_cache_provider
        live:
            backend:
                caches:
                    meta: doctrine_phpcr.meta_cache_provider
                    nodes: doctrine_phpcr.nodes_cache_provider

services:
    doctrine_phpcr.meta_cache_provider:
        class: Symfony\Component\Cache\DoctrineProvider
        public: false
        arguments:
            - '@doctrine_phpcr.meta_cache_pool'

    doctrine_phpcr.nodes_cache_provider:
        class: Symfony\Component\Cache\DoctrineProvider
        public: false
        arguments:
            - '@doctrine_phpcr.nodes_cache_pool'

framework:
    cache:
        pools:
            doctrine_phpcr.meta_cache_pool:
                adapter: cache.app

            doctrine_phpcr.nodes_cache_pool:
                adapter: cache.app

We did have currently problem that the cache increases and is going to get very big, after our redis was on 8gb, i flushed the whole cache after some weeks its again on 2gb:

After analysing the cache keys I find a lot keys for the same node (~50) e.g.:

sQtmeIwJkU:%5Bnodes%3A_%2Fcmf%2Fsulu%2Fcontents%2Fthemenseiten%2Fernaehrungsformen%2Fcholesterinarme-ernaehrung%2C_default_live%5D%5B87%5D
sQtmeIwJkU:%5Bnodes%3A_%2Fcmf%2Fsulu%2Fcontents%2Fthemenseiten%2Fernaehrungsformen%2Fcholesterinarme-ernaehrung%2C_default_live%5D%5B218%5D
sQtmeIwJkU:%5Bnodes%3A_%2Fcmf%2Fsulu%2Fcontents%2Fthemenseiten%2Fernaehrungsformen%2Fcholesterinarme-ernaehrung%2C_default_live%5D%5B456%5D
sQtmeIwJkU:%5Bnodes%3A_%2Fcmf%2Fsulu%2Fcontents%2Fthemenseiten%2Fernaehrungsformen%2Fcholesterinarme-ernaehrung%2C_default_live%5D%5B463%5D
...

Decoded the keys looks like this:

sQtmeIwJkU:[nodes:_/cmf/sulu/contents/themenseiten/ernaehrungsformen/cholesterinarme-ernaehrung,_default_live][87]
sQtmeIwJkU:[nodes:_/cmf/sulu/contents/themenseiten/ernaehrungsformen/cholesterinarme-ernaehrung,_default_live][218]
sQtmeIwJkU:[nodes:_/cmf/sulu/contents/themenseiten/ernaehrungsformen/cholesterinarme-ernaehrung,_default_live][456]
sQtmeIwJkU:[nodes:_/cmf/sulu/contents/themenseiten/ernaehrungsformen/cholesterinarme-ernaehrung,_default_live][463]
...

What is the last part [number], we are using in this project doctrine dbal adapter so no versioning is active here so not sure what the number could represent here. Does somebody have any hint what this number could represent or how to avoid that multiple entries for the same node are created.

[alexander-schranz]

Finally found out what happened:

The Doctrine CacheProvider does prefix the cache id with a namespace name and prefix with a namespace version see:

https://github.com/doctrine/cache/blob/13e3381b25847283a91948d04640543941309727/lib/Doctrine/Common/Cache/CacheProvider.php#L174-L179

This version is increased when calling CacheProvider::deleteAll function:

https://github.com/doctrine/cache/blob/13e3381b25847283a91948d04640543941309727/lib/Doctrine/Common/Cache/CacheProvider.php#L153-L165

This deleteAll function is called in the CachedClient of the Jackalope CachedClient implementation:

https://github.com/jackalope/jackalope-doctrine-dbal/blob/e2dbc73cc55492a3b0b0e756ed1bfb79443bd03f/src/Jackalope/Transport/DoctrineDBAL/CachedClient.php#L64

Seems like clearCaches is called in the CachedClient at some places e.g. move nodes or delete properties.

This currently create a lot of new doctrine cache namespaces and that makes that old entries are never removed and as they never expire the cache increases to infinity.

[alexander-schranz]

At current state I think the best would be to add a default_lifetime to the nodes cache configuration, to avoid that cache entries never expire.

E.g.:

            doctrine_phpcr.nodes_cache_pool:
                adapter: cache.app
                default_lifetime: 604800 # 1 week cache lifetime

What do you think? /cc @dbu @danrot @chirimoya

[dbu]

oh, i was not aware doctrine cache does that. this is a bit stupid, at least in your scenario, it could just delete the cache entries it seems... i guess to work around that, a cache lifetime configuration could help.

if you run out of space with a redis that you use as a cache, you should also look at your redis configuration though: redis can be configured like a database where it keeps things until they expire/are deleted. or it can act as an actual cache, with strategies to throw out things when it runs out of space.

for this use case, the cache should be configured to kick out least recently used entries when a threshold is exceeded. that would probably make the cache always consume the amount of memory you configured, but it would no longer need manual interaction.

is the lifetime already configurable and you suggest adding it to the recipe? if so, i agree its a good idea as a failsafe. somewhere in the documentation we might also mention the thing i explained about using redis as a cache.

[alexander-schranz]

@dbu thank you for your response. I think the redis is currently running under default configuration.

I think the configuration you are talking about in redis would be:

maxmemory 100mb
maxmemory-policy allkeys-lru

Seems totally make sense in our case, as I think the instance is not used for something else then a cache or php sessions.

I think also its good to have maybe a default cachelifetime configured as a failsafe. In the phpcr docs I could find nothing about redis cache.

[dbu]

yeah that redis config looks good, allkeys-lru is what you want. for the amount of memory, it depends of course on the size of your site and usage pattern. too low, and it will kick out thing that would still be useful, too high makes you waste memory again...

i think the right place to talk about that would be in the doc of the redis adapter of doctrine cache. if you want to do a PR to them to mention that, i think it would be a good idea.
and a PR on our recipe with the default_lifetime (and a short comment explaining the issue and saying that ideally you configure your cache to evict things when it runs out of memory and remove the ttl)

[alexander-schranz]

@dbu created a PR for the symfony cache redis adapter: symfony/symfony-docs#14683
hope this helps some people.

[dbu]

awesome! is it ok to close the issue here?

[alexander-schranz]

sure we can close it from my side.