Proposal for some consumer processes improvements

Author: hostep

design-documents/asynchronous-operations/consumer-processes-improvements.md

@@ -0,0 +1,87 @@
+# Improvements to message queue consumer processes
+
+## The current situation
+
+Currently all defined message queue consumers are getting spawned by a cronjob called `consumers_runner`. You can optionally choose to disable this entirely or only allow specific consumers to run based on some config settings in the `app/etc/env.php` file (see [docs](https://devdocs.magento.com/guides/v2.3/config-guide/mq/manage-message-queues.html#configuration)).  
+This `consumers_runner` job first looks around on the server(s) to see if there are already running consumer processes for the same queue. If that's the case it won't spawn a new one, but if no consumer process is found for a certain queue, it will spawn one. By default it will listen for maximum 10.000 messages and if those 10.000 messages have been handled, it will kill itself. The next time the cronjob `consumers_runner` triggers, it will spawn a new process.
+
+There is also a poison pill feature in Magento, which is basically just a random hash stored in the database in the table `queue_poison_pill`. If some configuration value gets changed using the Magento backend, the poison pill version is changed. 
+The running consumer processes check on this version, but only when a new message is in the queueu. If a consumer discovers a new message in the queue, it first checks if the poison pill version has changed. If the version wasn't changed, the message will be handled. If the version did get changed, the consumer will kill itself and the cronjob will spawn a new one on its next run and only then will handle that message.
+
+Magento 2.3.0 Open Source shipped with the message queue system and made use of it when using the bulk api import feature.  
+And with Magento version 2.3.2 some already existing features were converted to make use of the message queue system. These are tasks which are triggered using the backend of Magento which can potentially take a while to execute. In order to prevent the webserver or php-fpm to run against a timeout, it was chosen to send these tasks to the message queue system and let these tasks get executed asynchronously. Some examples are:
+
+- Generating coupon codes
+- Mass editing products
+- Exporting data
+- ...
+
+Currently in Magento 2.3.2, 4 consumer processes get spawned by the cron system. Each of these processes take memory and cpu and regularly queries the database (or RabbitMQ if that broker is being used) to see if new messages are available and then process them.
+
+
+## Problem 1: not enough options to keep consumers under control
+
+### The problem
+
+There is currently too little control over these consumer processes.  
+What if people only very irregularly use one of these features mentioned above, let's say they only once a year export all their products to check their inventory levels.  
+Then you have some consumer process sitting there, doing nothing for 364 days in the year, wasting precious cpu cycles and taking up precious memory, until finally once a year the shopowner decides to execute a certain task it can execute.
+
+There is also the potential problem that the consumer process will take up more and more memory due to some memory leak in the code it executes.
+
+### The suggested solution
+
+The suggestion is to give more control per consumer to the shopowner or developers managing the shop.  
+We could add some additional options to the consumer processes to keep them more under control. Currently there is one limit available: `max-messages`. If that number of messages gets processed, the consumer will kill itself.  
+I'd like to suggest some other limits which we can set:
+
+- `max-idle-time`: if no message was being handled in xx seconds, then kill yourself
+- `max-time`: after xx total seconds, and after you are done with handling the current message, kill yourself
+- `memory-limit`: if xx MB's of memory is being taken up by the process and after you are done with handling the current message, kill yourself (a way to work around potential memory leaks)
+
+Next to these limits, a configurable sleep time might be nice:
+
+- `sleep`: xx milliseconds to sleep before checking if a new message is available (currently this is [hardcoded to 1 second](https://github.com/magento/magento2/blob/2.3.2/lib/internal/Magento/Framework/MessageQueue/CallbackInvoker.php#L59))
+
+I'd also like to see an option defined on the consumer, but being used by the `consumers_runner` cronjob:
+
+- `only-spawn-when-message-available`: the idea is that the `consumers_runner` job checks the queue before spawning a consumer, to see if there is actually a message pending in the queue. If there is one, then go ahead and spawn a consumer (only if one isn't already running). If there isn't a message in the queue, then don't spawn a consumer.
+
+### Some options combined
+
+The problem outlined above, where a specific consumer only needs to run very infrequently could be solved by combining the options:
+
+- `only-spawn-when-message-available`
+- `max-idle-time`
+
+The consumer will only spawn when it is needed, and it will kill itself when it wasn't active for a certain period.  
+That should save some precious server resources.
+
+### Making these options configurable and have some defaults
+
+These options should be configurable per consumer type.
+Some sensible defaults could be set in the [`queue_consumer.xml`](https://devdocs.magento.com/guides/v2.3/extension-dev-guide/message-queues/config-mq.html#queueconsumerxml) file for some of these options.  
+Next to that, developers or shopowners should be able to override these values per consumer type. At least being able to override them in the `app/etc/env.php` file would be nice, but a backend interface for making these things configurable would also be very nice (but can maybe be done in a later phase?).
+
+### Credits
+
+Some of these ideas where taken from the [`symfony/messenger` component](https://github.com/symfony/messenger/blob/3d65f22f9a56f6475c19999fdbc3a897cefc8900/Command/ConsumeMessagesCommand.php#L77-L80).
+
+## Problem 2: deployment problems
+
+### The problem
+
+When deploying a new version of Magento/third party/custom code to a server, we can potentially run against a problem where consumers are still using old code loaded in memory. This is not only the code the consumer is running itself, but might also be Magento core code itself for the consumer processes themselves (which might change when upgrading Magento to a newer version).
+
+Also when using a capistrano-style deployment where you have a symlinked a `current` directory pointing to a certain release directory, after a new deploy the running consumers will still supposedly run from the old release directory referencing a `.pid` file containing its process id in a directory which is in the old release directory. The cronjob `consumers_runner` will go searching for that file in the new release directory and won't find it. Causing a second consumer to start up, even though the old one is still running. (This is probably already fixed by this unreleased commit: [MC-18477](https://github.com/magento/magento2/commit/1d9e07b218c7c8ad1f05706828cb2dd47d2d2d58))
+
+### The suggested solution
+
+The suggestion here would be to give deployment scripts the option to signal the running consumer proccesses to handle their current messages and then kill themselves as soon as possible.  
+We already have the poison pill functionality which could be used here, but the problem with that one is that it will only be checked when a new message appears in the queue. If no messages appear in the queue during a deploy, and we only update the poison pill version, the consumer will still keep running because it doesn't check for the poison pill version until a new message appears.  
+So I would suggest some kind of dummy message to be created in each queue which only purpose is for the consumers to check the poison pill version and that message then should get removed again from the queue and nothing should be done with it.
+
+I'm seeing this as some new command being added to `bin/magento` (`queue:consumers:suicide` ?) which does both these things:
+
+- updates the poison pill version
+- sends a dummy message to all queues