Skip to content

Commit ba36873

Browse files
buskamuzabuskamuza
authored
Merge pull request #171 from knowj/cron-upgrade
Background Processes - Draft 1
2 parents 3c69e2c + ffb4a2a commit ba36873

File tree

1 file changed

+171
-0
lines changed

1 file changed

+171
-0
lines changed

‎design-documents/background-processes/background-processes-proposal.md

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
# Overview
2+
The cron is one of the most business critical components of Magento, offloading resource
3+
intensive tasks from the user request into the backend and allowing scheduling long running
4+
processes.
5+
6+
The business critical element of the cron needs better supporting in the foundation of the module and
7+
supporting documentation giving better control to developers, technology providers and eCommerce managers.
8+
9+
In addition to the cron, certain processes should be moved away from the cron architecture to a service
10+
based architecture leveraging AMQP.
11+
12+
# Problem
13+
The cron implementation in it's current existence doesn't provide the business critical foundation
14+
or visibility required for a fully functioning eCommerce system.
15+
16+
# Solution - Business Critical Foundation
17+
The overall approach of how magento handled backend/scheduled tasks needs to be reviewed with a
18+
*business critical* foundation in mind. This involves a re-architecture of the cron itself
19+
and a move toward a service based AMQP
20+
21+
1. Separation
22+
2. Visibility
23+
3. Education
24+
25+
## 1. Separation
26+
Separation of business critical and none business critical backend processes as part of the core
27+
architecture of Magento. This should be multi threaded "out the box" running business critical processes
28+
in their own threads (Emails, Indexes).
29+
30+
At the foundation no rogue process should ever prevent a business critical operation.
31+
32+
- Multi thread core processes out the box (e.g. workers)
33+
- Process timeouts (Default X minutes and configurable by job)
34+
- Retry limits
35+
- Exponential back off
36+
37+
The architectural goal for separation is service isolation using the Cron to schedule and message queues
38+
for all processes with scalable workers for execution.
39+
40+
### Cron
41+
The current cronjobs that run within the core of Magento are:
42+
- sales_send_order_emails
43+
- sales_send_order_invoice_emails
44+
- sales_send_order_shipment_emails
45+
- sales_send_order_creditmemo_emails
46+
- sales_clean_quotes
47+
- sales_clean_orders
48+
- aggregate_sales_report_order_data
49+
- aggregate_sales_report_invoiced_data
50+
- aggregate_sales_report_refunded_data
51+
- aggregate_sales_report_bestsellers_data
52+
- sales_grid_order_async_insert
53+
- sales_grid_order_invoice_async_insert
54+
- sales_grid_order_shipment_async_insert
55+
- sales_grid_order_creditmemo_async_insert
56+
- catalog_index_refresh_price
57+
- catalog_product_flat_indexer_store_cleanup
58+
- catalog_product_outdated_price_values_cleanup
59+
- catalog_product_frontend_actions_flush
60+
- catalog_product_attribute_value_synchronize
61+
- indexer_reindex_all_invalid
62+
- indexer_update_all_views
63+
- indexer_clean_all_changelogs
64+
- persistent_clear_expired
65+
- security_clean_admin_expired_sessions
66+
- security_clean_password_reset_request_event_records
67+
- magento_newrelicreporting_cron
68+
- mysqlmq_clean_messages
69+
- outdated_authentication_failures_cleanup
70+
- expired_tokens_cleanup
71+
- catalog_product_alert
72+
- sitemap_generate
73+
- backend_clean_cache
74+
- aggregate_sales_report_shipment_data
75+
- currency_rates_update
76+
- captcha_delete_old_attempts
77+
- captcha_delete_expired_images
78+
- paypal_fetch_settlement_reports
79+
- messagequeue_clean_outdated_locks
80+
- consumers_runner
81+
- consumerConsumerName
82+
- bulk_cleanup
83+
- catalogrule_apply_all
84+
- newsletter_send_all
85+
- system_backup
86+
- visitor_clean
87+
- analytics_subscribe
88+
- analytics_update
89+
- analytics_collect_data
90+
- aggregate_sales_report_coupons_data
91+
92+
This excludes 3rd party bundled extensions such as DotDigital.
93+
94+
## 2. Visibility
95+
On Magento 1 we had the AOE Scheduler which provided us a considerably enhanced user experience. This
96+
module isn't available for Magento 2. Allowing webmasters and developers to clearly see what processes
97+
are running, scheduled and complete gives them greater control and reduces support requests and increases
98+
confidence in the platform.
99+
100+
- Admin visible timeline of scheduled tasks (Scheduled, in progress, complete, errored)
101+
- Ability to schedule, delete, remove, pause specific tasks.
102+
- Cleanup of redundant processes to reduce noise
103+
- Logging and Alerting
104+
105+
## 3. Education
106+
- Technical documentation for effectively implementing schedules into Magento modules
107+
- Technical documentation for when to use cron vs AMQP
108+
- Setup documentation for multi threaded implementation
109+
110+
# Examples
111+
112+
## Example 1 - Rogue process
113+
A process that connects to an API such as a payment provider could end up getting stuck in a loop
114+
and blocking any other process from progressing currently there is no mechanic in place to
115+
a: Allow other business critical processes to continue aside from defining multiple crontab processes in groups
116+
b: Report or stop the rogue process from continuing until manually spotted
117+
c: No visibility of the processes aside from checking the database
118+
d: No reporting or alerting of the above condition
119+
120+
# Staged Approach
121+
The overhaul of the background processing within Magento is an large project with high impact to the
122+
current public API's and implementation of the current Magento Cron.
123+
124+
## Stage 1 - Framework
125+
Create/expand the framework for AMQP and Worker management.
126+
127+
1. Review queue interfaces
128+
2. Standardise data structure
129+
3. Error handling, logging and alerting
130+
4. Review worker interfaces
131+
5. Review worker management
132+
133+
### Problems to solve:
134+
1. Worker management
135+
How to manage workers at from simple to expert levels of complexity. Currently the cron spawns workers
136+
with (from the looks of it) no visibility over the threads that are spawned. We also need to factor
137+
in completely multi-node infrastructure.
138+
The aim should be to provide a highly scalable implementation from a highly scaled environment to a
139+
single server infrastructure with simple platform implementation.
140+
141+
- Potential use of Kubernetes: https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/
142+
143+
## Stage 1 - Dashboard
144+
1. Visibility of each queue and number of items in each queue
145+
2. Visibility of any errors
146+
147+
## Stage 2 - Transactional Emails
148+
Migrate the following cron tasks into the new queue based implementation
149+
- sales_send_order_emails
150+
- sales_send_order_invoice_emails
151+
- sales_send_order_shipment_emails
152+
- sales_send_order_creditmemo_emails
153+
154+
## Stage 3 - Indexers
155+
- sales_grid_order_async_insert
156+
- sales_grid_order_invoice_async_insert
157+
- sales_grid_order_shipment_async_insert
158+
- sales_grid_order_creditmemo_async_insert
159+
- catalog_index_refresh_price
160+
- catalog_product_flat_indexer_store_cleanup
161+
- catalog_product_outdated_price_values_cleanup
162+
- catalog_product_frontend_actions_flush
163+
- catalog_product_attribute_value_synchronize
164+
- indexer_reindex_all_invalid
165+
- indexer_update_all_views
166+
- indexer_clean_all_changelogs
167+
168+
## Additional
169+
170+
- Content Staging
171+
- Content staging product team required at this point

0 commit comments

Comments
 (0)