Avoids endless loop of indexers being marked as invalid #29196

hostep · 2020-07-19T09:30:34Z

Description (*)

Hi

This was discovered on a particular project where we noticed that the indexers catalogrule_rule and catalog_category_product are constantly being marked as Reindex required.

The cronjob indexer_reindex_all_invalid is running and is supposed to fix this, but that doesn't happen.
If however, I stop the cronjobs, and manually execute bin/magento indexer:reindex catalogrule_rule catalog_category_product we see a different behavior and only catalog_category_product is then marked as Reindex required.. If I then execute either bin/magento indexer:reindex catalog_category_product or let the cronjob indexer_reindex_all_invalid take over again, everything is getting back to a stable state and all indexers are marked as Ready.

So this means that the execution of the indexer_reindex_all_invalid cronjob and bin/magento indexer:reindex {all-invalid-indexers} are behaving differently which isn't correct in my opinion.

We've seen this happening with Magento OS 2.3.5, and I was able to reproduce it on the 2.4-develop branch (using commit f39a2e0)

I currently only have a way to reproduce it by using the database of this particular project and we also need a particular module from Amasty installed as well (the Improved Layered Navigation module version 2.14.3).
But in my opinion it's not a bug in the Amasty module but a problem in core Magento since we see a difference in behavior in how invalid indexers are reindexed if you do it manually or if the cronjob executes it.

After searching around a bit I noticed a particular commit which was only applied to the manual reindexing command: MAGETWO-51540: Linked indexers catalog_category_product and catalog_product_cateogry do not set valid status after shell run.

So I now introduced these same changes to the code which is used by the indexer_reindex_all_invalid cronjob and after testing these changes on both our project (on Magento 2.3.5) as on my test on the 2.4-develop branch, it looks like it fixes the issue.

In my opinion, this is a pretty important bugfix, because this bug causes the server of this particular project to have a high CPU load because of the constant reindexing which happens with every cron run which is not necessary.

In my opinion, we should also try to refactor the code so both bin/magento indexer:reindex as the cronjob indexer_reindex_all_invalid should use the same code so we can avoid in the future where one fix is applied to one part and forgotten about in the other part.
I've currently chosen not to refactor this so it's easier to review these changes, and maybe somebody can follow up in a later PR to refactor this?

Related Pull Requests

Fixed Issues (if relevant)

None that I could find

Manual testing scenarios (*)

Have magerun2 available to manually execute cronjobs (https://files.magerun.net/)
Setup a clean Magento
Install Amasty's Improved Layered Navigation module version 2.14.3
Use a specific database from our project (sorry, I can't publicly share this database)
Run bin/magento setup:upgrade
Run bin/magento indexer:reindex so all indexers are marked as valid
Invalidate the two specific 2 indexers: bin/magento indexer:reset catalogrule_rule catalog_category_product
Look at the indexers statuses: bin/magento indexer:status (the two indexers should be marked as invalid)
Run php n98-magerun2.phar sys:cron:run indexer_reindex_all_invalid
Look at the indexers statuses: bin/magento indexer:status (the two indexers are still marked as invalid)
Run php n98-magerun2.phar sys:cron:run indexer_reindex_all_invalid
Look at the indexers statuses: bin/magento indexer:status (the two indexers are still marked as invalid)
Run bin/magento indexer:reindex catalogrule_rule catalog_category_product
Look at the indexers statuses: bin/magento indexer:status (only one indexers is marked as invalid: catalog_category_product)
Run php n98-magerun2.phar sys:cron:run indexer_reindex_all_invalid
Look at the indexers statuses: bin/magento indexer:status (All indexers are now finally marked as valid)

With this PR applied, after step 8, we can jump immediately to step 13. This is probably still not 100% fixing the problem since I would expect all indexers to be marked as valid after this, but maybe that Amasty module is intervening here, not exactly sure without more deeper debugging.

If somebody needs the database from this project, I'm willing to provide it to you in private via Slack for example, but it can contain confidential information, so please I'm asking internal Magento devs to not abuse the information in the database.

Thanks!

Questions or comments

Maybe a reproducible case is written down in internal ticket MAGETWO-51540 which you can try to reproduce using the indexer_reindex_all_invalid cronjob?

Contribution checklist (*)

Pull request has a meaningful description of its purpose
All commits are accompanied by meaningful commit messages
All new or changed code is covered with unit/integration tests (if applicable)
All automated tests passed successfully (all builds are green)

Resolved issues:

resolves [Issue] Avoids endless loop of indexers being marked as invalid #29478: Avoids endless loop of indexers being marked as invalid
resolves Refactor Magento_Indexer module so that the "indexer_reindex_all_invalid" cronjob and "bin/magento indexer:reindex" command use the same code #29297: Refactor Magento_Indexer module so that the "indexer_reindex_all_invalid" cronjob and "bin/magento indexer:reindex" command use the same code (done by @engcom-Golf, thanks for that!)

m2-assistant · 2020-07-19T09:30:36Z

Hi @hostep. Thank you for your contribution
Here is some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

@magento give me test instance - deploy test instance based on PR changes
@magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

@magento run all tests - run or re-run all required tests against the PR changes
@magento run <test-build(s)> - run or re-run specific test build(s)
For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names. Allowed build names are:

Database Compare
Functional Tests CE
Functional Tests EE,
Functional Tests B2B
Integration Tests
Magento Health Index
Sample Data Tests CE
Sample Data Tests EE
Sample Data Tests B2B
Static Tests
Unit Tests
WebAPI Tests

You can find more information about the builds here

ℹ️ Please run only needed test builds instead of all when developing. Please run all test builds before sending your PR for review.

For more details, please, review the Magento Contributor Guide documentation.

⚠️ According to the Magento Contribution requirements, all Pull Requests must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of Pull Requests happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

hostep · 2020-07-19T09:31:25Z

@magento run all tests

hostep · 2020-07-19T11:37:50Z

@magento run Static Tests,Unit Tests

hostep · 2020-07-19T11:39:08Z

@magento run Unit Tests

hostep · 2020-07-19T12:23:45Z

@magento run all tests

rogyar · 2020-07-26T12:50:04Z

Hi @hostep. That's a good point, thank you for your collaboration. Additionally, I would create a separate service i.e. validateSharedIndex and put the logic of getIndexerIdsBySharedIndex and validateSharedIndex there. So we can reuse the newly created class for the indexer processor and IndexerReindexCommand.
But you may do it in a separate PR after the current one is merged if you wish.

rogyar · 2020-07-26T12:51:00Z

@magento run WebAPI Tests

hostep · 2020-07-27T19:42:53Z

@rogyar: thanks for the feedback!

I agree we should refactor this, therefore I created #29297 to follow this up.
I'll try to work on that issue when I find some more time (currently pretty busy with other non-Magento stuff)

hostep · 2020-07-28T12:22:33Z

Just some extra information regarding the server impact we noticed due to this change.
We deployed this solution to a production environment on 07/22, the results are quite noticeable in our case.

Server load dropped significantly:

Write operations (disk i/o) dropped significantly:

We see more Varnish hits (presumably because caches get flushed a lot less frequently):

hostep · 2020-07-28T13:13:33Z

And some further info for the people interested.
I took a look at all our projects which run on Magento 2.3.x (a total of 28 shops, ranging from versions 2.3.1 to 2.3.5) and noticed 3 shops which all have this behavior. All 3 shops are running on version 2.3.5. So it looks like some change in 2.3.5 amplified the chance of this problem happening.
If I have to guess it's probably MC-30569: Shared indexers have invalid status after full reindex (on 2.4 a similar change exists as MC-30568: Shared indexers have invalid status after full reindex)

magento-engcom-team · 2020-07-31T15:14:35Z

Hi @rogyar, thank you for the review.
ENGCOM-7929 has been created to process this Pull Request

VladimirZaets · 2020-08-11T13:38:46Z

@magento create issue

…ronjob

hostep · 2020-08-14T06:27:57Z

@magento run all tests

gabrieldagama

Hi @hostep, thanks for your pull request, please follow a small comment regards your changes (I think it may be important). Also, I strongly recommend avoiding the code duplication between this class and IndexerReindexCommand, by duplicating the code we are leaving a gap that could result in these 2 classes having different behavior again, what do you think? Thanks.

gabrieldagama · 2020-09-03T09:16:50Z

app/code/Magento/Indexer/Model/Processor.php

+    private function validateSharedIndex(string $sharedIndex): self
+    {
+        if (empty($sharedIndex)) {
+            throw new \InvalidArgumentException(


I guess it would be important to handle this exception on the reindexAllInvalid, otherwise, the whole command would keep failing on a specific indexer and not move forward. Also, I would recommend logging the exception msg.

You are probably right, but this falls out of scope of this PR, it might fit better in the refactoring step which should happen when resolving #29297

Sorry for being so strict, but I want to this bug to be fixed as soon as possible and not being held back by "code quality" requirements. That can happen later. The refactoring will probably not take that much time, but rewriting all the automated tests will take a significant amount of time and I don't have a lot of free time at the moment. Sorry!

magento-engcom-team · 2020-10-12T10:22:42Z

Hi @rogyar, thank you for the review.
ENGCOM-7929 has been created to process this Pull Request

engcom-Foxtrot · 2020-10-16T11:12:32Z

@hostep hello! I cannot reproduce the issue you are fixing in this PR. Can you please provide some additional information?

engcom-Foxtrot

Cannot reproduce by provided test steps.

Pull Request state was updated. Re-review required.

engcom-Foxtrot · 2020-10-16T13:11:00Z

QA passed.

…29196

m2-assistant · 2020-10-24T10:37:47Z

Hi @hostep, thank you for your contribution!
Please, complete Contribution Survey, it will take less than a minute.
Your feedback will help us to improve contribution process.

hostep · 2020-10-24T13:49:58Z

Thanks for merging and for the extra assistance by @engcom-Golf & @engcom-Foxtrot!

collymore · 2020-10-31T00:33:37Z

This issue was happening for us on a Magento 2.3.5 store, I've pushed the core PR code to a module/composer package for anyone that needs it on earlier versions https://github.com/develodesign/magento-2-indexer-loop-fix

ghost added the Progress: pending review label Jul 19, 2020

magento-engcom-team added Component: Indexer Release Line: 2.4 labels Jul 19, 2020

Avoids indefinite loop of indexers being marked as invalid.

f0c6453

hostep force-pushed the improve-indexer_reindex_all_invalid-cronjob branch from 758a885 to f0c6453 Compare July 19, 2020 11:36

hostep changed the title ~~Avoids indefinite loop of indexers being marked as invalid.~~ Avoids endless loop of indexers being marked as invalid Jul 22, 2020

rogyar self-assigned this Jul 26, 2020

rogyar added the Auto-Tests: Covered All changes in Pull Request is covered by auto-tests label Jul 26, 2020

hostep mentioned this pull request Jul 27, 2020

Refactor Magento_Indexer module so that the "indexer_reindex_all_invalid" cronjob and "bin/magento indexer:reindex" command use the same code #29297

Closed

5 tasks

rogyar added the Award: bug fix label Jul 31, 2020

rogyar approved these changes Jul 31, 2020

View reviewed changes

ghost added Progress: ready for testing and removed Progress: pending review labels Jul 31, 2020

m2-assistant bot mentioned this pull request Aug 11, 2020

[Issue] Avoids endless loop of indexers being marked as invalid #29478

Closed

4 tasks

Merge branch '2.4-develop' into improve-indexer_reindex_all_invalid-c…

07d4ac6

…ronjob

sdzhepa added the Triage: Dev.Experience Issue related to Developer Experience and needs help with Triage to Confirm or Reject it label Aug 11, 2020

gabrieldagama suggested changes Sep 3, 2020

View reviewed changes

ghost added Progress: ready for testing and removed Progress: review labels Oct 12, 2020

ghost added Progress: extended testing Progress: ready for testing and removed Progress: ready for testing Progress: extended testing labels Oct 12, 2020

engcom-Foxtrot requested changes Oct 16, 2020

View reviewed changes

ghost added Progress: needs update and removed Progress: ready for testing labels Oct 16, 2020

engcom-Foxtrot approved these changes Oct 16, 2020

View reviewed changes

ghost added Progress: ready for testing Progress: extended testing and removed Progress: needs update Progress: ready for testing labels Oct 16, 2020

ghost added Progress: accept and removed Progress: extended testing labels Oct 16, 2020

engcom-Foxtrot added the QA: Ready to add to Regression Scope Should be analyzed and added to Regression Testing Scope(if applicable) label Oct 16, 2020

engcom-Alfa added QA: Added to Regression Scope Scenario was analysed and added to Regression Testing Scope and removed QA: Ready to add to Regression Scope Should be analyzed and added to Regression Testing Scope(if applicable) labels Oct 16, 2020

rogyar added the Award: category of expertise label Oct 19, 2020

magento-engcom-team pushed a commit that referenced this pull request Oct 24, 2020

ENGCOM-7929: Avoids endless loop of indexers being marked as invalid #…

c1e136b

…29196

magento-engcom-team merged commit 2b99ec4 into magento:2.4-develop Oct 24, 2020

magento-engcom-team added the Event: MageCONF CD 2020 label Oct 24, 2020

Avoids endless loop of indexers being marked as invalid #29196

Avoids endless loop of indexers being marked as invalid #29196

Uh oh!

Conversation

hostep commented Jul 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description (*)

Related Pull Requests

Fixed Issues (if relevant)

Manual testing scenarios (*)

Questions or comments

Contribution checklist (*)

Resolved issues:

Uh oh!

m2-assistant bot commented Jul 19, 2020

Uh oh!

hostep commented Jul 19, 2020

Uh oh!

hostep commented Jul 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hostep commented Jul 19, 2020

Uh oh!

hostep commented Jul 19, 2020

Uh oh!

rogyar commented Jul 26, 2020

Uh oh!

rogyar commented Jul 26, 2020

Uh oh!

hostep commented Jul 27, 2020

Uh oh!

hostep commented Jul 28, 2020

Uh oh!

hostep commented Jul 28, 2020

Uh oh!

magento-engcom-team commented Jul 31, 2020

Uh oh!

VladimirZaets commented Aug 11, 2020

Uh oh!

hostep commented Aug 14, 2020

Uh oh!

gabrieldagama left a comment

Choose a reason for hiding this comment

Uh oh!

gabrieldagama Sep 3, 2020

Choose a reason for hiding this comment

Uh oh!

hostep Sep 3, 2020

Choose a reason for hiding this comment

Uh oh!

magento-engcom-team commented Oct 12, 2020

Uh oh!

engcom-Foxtrot commented Oct 16, 2020

Uh oh!

engcom-Foxtrot left a comment

Choose a reason for hiding this comment

Uh oh!

engcom-Foxtrot commented Oct 16, 2020

Uh oh!

m2-assistant bot commented Oct 24, 2020

Uh oh!

hostep commented Oct 24, 2020

Uh oh!

collymore commented Oct 31, 2020

Uh oh!

Uh oh!

hostep commented Jul 19, 2020 •

edited

Loading

hostep commented Jul 19, 2020 •

edited

Loading