Skip to content

Stopping old queue consumers during upgrade #31495

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 0 commits into from
Closed

Stopping old queue consumers during upgrade #31495

wants to merge 0 commits into from

Conversation

swnsma
Copy link
Contributor

@swnsma swnsma commented Dec 31, 2020 •
edited by engcom-Oscar
Loading

Description (*)

Put poison pill after each 'setup:upgrade' execution.

Fixed Issues (if relevant)

  1. Fixes Queue consumers #23540

Steps for testing:

  1. Check "version" value in "queue_poison_pill" table in Magento instance database.
  2. Upgrade Magento instance.
  3. Verify "version" value in "queue_poison_pill" table has changed in Magento instance database.
  4. Execute queue:consumers:restart command.
  5. Verify "version" value in "queue_poison_pill" table has changed in Magento instance database.

Contribution checklist (*)

  • Pull request has a meaningful description of its purpose
  • All commits are accompanied by meaningful commit messages
  • All new or changed code is covered with unit/integration tests (if applicable)
  • All automated tests passed successfully (all builds are green)

@m2-assistant
Copy link

m2-assistant bot commented Dec 31, 2020

Hi @swnsma. Thank you for your contribution
Here is some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

  • @magento give me test instance - deploy test instance based on PR changes
  • @magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

  • @magento run all tests - run or re-run all required tests against the PR changes
  • @magento run <test-build(s)> - run or re-run specific test build(s)
    For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names. Allowed build names are:

  1. Database Compare
  2. Functional Tests CE
  3. Functional Tests EE,
  4. Functional Tests B2B
  5. Integration Tests
  6. Magento Health Index
  7. Sample Data Tests CE
  8. Sample Data Tests EE
  9. Sample Data Tests B2B
  10. Static Tests
  11. Unit Tests
  12. WebAPI Tests

You can find more information about the builds here

ℹ️ Please run only needed test builds instead of all when developing. Please run all test builds before sending your PR for review.

For more details, please, review the Magento Contributor Guide documentation.

⚠️ According to the Magento Contribution requirements, all Pull Requests must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of Pull Requests happens in the queue order. If you want to speed up the delivery of your contribution, please join the Community Contributions Triage session to discuss the appropriate ticket.

🎥 You can find the recording of the previous Community Contributions Triage on the Magento Youtube Channel

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

@magento-engcom-team magento-engcom-team added Component: MessageQueue Release Line: 2.4 Partner: Vaimo Pull Request is created by partner Vaimo partners-contribution Pull Request is created by Magento Partner labels Dec 31, 2020
@hostep
Copy link
Contributor

hostep commented Jan 2, 2021 •
edited
Loading

This also kind-of fixes magento/community-features#181 and problem 2 mentioned over here I think.

Not sure if we still need a dedicated bin/magento command as well, or if doing this in setup:upgrade is enough. But for me, it would be enough.

@swnsma
Copy link
Contributor Author

swnsma commented Jan 4, 2021

@magento run all tests

@sidolov sidolov added Priority: P2 A defect with this priority could have functionality issues which are not to expectations. Severity: S2 Major restrictions or short-term circumventions are required until a fix is available. labels Jan 4, 2021
@coderimus
Copy link
Contributor

@magento run Functional Tests B2B, Functional Tests CE, Functional Tests EE, Integration Tests, Magento Health Index

@coderimus coderimus self-assigned this Jan 6, 2021
@coderimus
Copy link
Contributor

@magento run Functional Tests B2B, Functional Tests CE, Magento Health Index

@coderimus
Copy link
Contributor

Hello @swnsma
Thank you for the fix! As for me, it is a really great solution which removes headache for everyone who uses the MQ in the Magento. I think that the approach you provided is applicable and it uses in other core modules also, as I can see. No need to have another CI command in that case 👍

I approve this PR in a comment for now but we have the Health Index failed and without assistance from the core team we are not able to move forward so let's call @sivaschenko and @sidolov for help :) Please, assist me with the next step: just approve it and move for testing or we need to do something else in case of failed Health Index with the Difference with mainline detected message?

@sivaschenko
Copy link
Member

Thanks for the pull request @swnsma !

I am moving this pull request to "To Apporve" status to process internal approval.

Approval ticket: https://jira.corp.magento.com/browse/MC-40323

cc @sidolov

@magento-engcom-team
Copy link
Contributor

Hi @coderimus, thank you for the review.
ENGCOM-8607 has been created to process this Pull Request
✳️ @coderimus, could you please add one of the following labels to the Pull Request?

Label Description
Auto-Tests: Covered All changes in Pull Request is covered by auto-tests
Auto-Tests: Not Covered Changes in Pull Request requires coverage by auto-tests
Auto-Tests: Not Required Changes in Pull Request does not require coverage by auto-tests

@ihor-sviziev
Copy link
Contributor

@swnsma good job! Really good improvement! The only one thing I would like to see there additionally - test coverage. Can we cover running setup:upgrade command in kind of integration test and compare poison pill value before & after?

@coderimus
Copy link
Contributor

Hi @ihor-sviziev
Agree with your test request and we definitely need this. But in that case the PR, as I understood, will pass internal approval from the core team and if everything fine we can deliver it and create tests.

@JeroenBoersma
Copy link
Member

I have one con on this solution, we don't want to run setup:upgrade --keep-generated on every release, only for those who add or update things in the database... This is checked with setup:db:status and app:config:status if these report a non-change we just do a symlink change and have near zero downtime.

So, this solution still doesn't fix this behavior because then the old runners keep running...

A simple solution would be having the possibility to also run this from commandline, then we would ran that if db and app are up-to-date

@hostep
Copy link
Contributor

hostep commented Jan 10, 2021

Ah yes good point @JeroenBoersma, a dedicated command line option would still be useful in this case next to the automatic setup:upgrade renewal.

@magento-automated-testing
Copy link

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

@ihor-sviziev
Copy link
Contributor

ihor-sviziev commented Jun 21, 2022 •
edited
Loading

Hi @engcom-Alfa,
I saw that you tried to move to "accept" status in #31495 (comment)
Does it mean we got the approval for the changes?

PS: I moved this PR back to "pending approval", as that was the last status before resolving conflict, and the changes still look good to me

@engcom-Alfa
Copy link
Contributor

Hi @engcom-Alfa, I saw that you tried to move to "accept" status in #31495 (comment) Does it mean we got the approval for the changes?

PS: I moved this PR back to "pending approval", as that was the last status before resolving conflict, and the changes still look good to me

Sure, will cross check and process it accordingly.
Thanks for the info :)

@engcom-Alfa
Copy link
Contributor

@magento run Functional Tests B2B, Magento Health Index

@magento-automated-testing
Copy link

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

@engcom-Alfa
Copy link
Contributor

@magento run Functional Tests B2B

@magento-automated-testing
Copy link

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

@PascalBrouwers
Copy link
Contributor

@sidolov @sivaschenko can we get this merged?

@ihor-sviziev
Copy link
Contributor

also pinging @ishakhsuvarov

@engcom-Hotel
Copy link
Contributor

Raised new internal approval request JIRA ticket on the adobe portal for Health Index test failure. Waiting for the approvals.

@magento-automated-testing
Copy link

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

@ishakhsuvarov
Copy link
Contributor

@ihor-sviziev I am working to push this ticket forward

@ishakhsuvarov
Copy link
Contributor

@swnsma @ihor-sviziev This actually got merged, however last commit with 2.4 sync did not get included in the internal merge, thus creating this confusiong situation.

Not sure why github thinks it's "closed" and not "merged" but there is 0 files and commits in the changeset.

Sorry for the confusion.

@swnsma
Copy link
Contributor Author

swnsma commented Dec 18, 2022

@ishakhsuvarov, thank you!
After so many months this seems to be Christmas Miracle! :)

@swnsma swnsma deleted the poison-pill branch December 18, 2022 17:05
@fritzmg
Copy link

fritzmg commented Jan 16, 2023

Since the exact commits are lost (or may be someone has a link?) - was this actually already released, or will this be in the next release?

@hostep
Copy link
Contributor

hostep commented Jan 16, 2023

Yes, it will be included in Magento 2.4.6 (it's included in 2.4.6-beta1 at least)

And I've found these commits: https://github.com/magento/magento2/search?q=%22magento%2Fmagento2%2323540%22&type=commits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vasylmalanka vasylmalanka vasylmalanka requested changes

@ihor-sviziev ihor-sviziev ihor-sviziev approved these changes

@coderimus coderimus coderimus approved these changes

@ishakhsuvarov ishakhsuvarov ishakhsuvarov approved these changes

@sidolov sidolov Awaiting requested review from sidolov

Assignees

@ihor-sviziev ihor-sviziev

@coderimus coderimus

@ishakhsuvarov ishakhsuvarov

@vasylmalanka vasylmalanka

@engcom-Foxtrot engcom-Foxtrot

Labels
Auto-Tests: Covered All changes in Pull Request is covered by auto-tests Award: advanced Award: special achievement Award: test coverage Component: MessageQueue Partner: Vaimo Pull Request is created by partner Vaimo Partner: vConnect partners-contribution Pull Request is created by Magento Partner Priority: P2 A defect with this priority could have functionality issues which are not to expectations. QA: Added to Regression Scope Scenario was analysed and added to Regression Testing Scope Release Line: 2.4 Severity: S2 Major restrictions or short-term circumventions are required until a fix is available.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Queue consumers