Bugfix: Retry Handler to Prevent Unbounded Retries while trying to Mitigate YUM Update Errors

[rirakshi-dev]

Issue

Bug 30347555: [AzGPS][Linux][Guest] Limit number of retries on Known Errors - YUM - RHEL

This bugfix task was created due to a customer reported issue.

What's Broken:

1. Yum Update hits the following error: Failed to download metadata for repo 'EfficiOS-RHEL7-x86-64': repomd.xml parser error: Parse error at line: 33 (xmlParseStartTag: invalid element name
2. Inside YumPackageManager.py
   a. try_mitigate_issues_if_any() invokes self.check_known_issues_and_attempt_fix() to self-mitigate the issue.
   b. self.check_known_issues_and_attempt_fix() finds the error signature, "Failed to download metadata for repo" in its list, and invokes "sudo yum update -y --disablerepo='*' --enablerepo='*microsoft*'"
   c. This command executes successfully, and try_mitigate_issues_if_any() performs a check to validate if this fixed the failure.
   d. However, "Failed to download metadata for repo" error is again thrown.
   e. try_mitigate_issues_if_any() invokes itself recursively, to resolve this issue.
3. Since there is no upper bound on the # of retry attempts that can be made to fix the issue, this goes into an unbounded loop, ultimately resulting in timeout of the extension.
4. Customer Impact: Auto Assessment gets stuck while showing ‘In Progress’ on the portal.

What's the Fix:

• Retry Limit Control: Added a retry_count and max_retries mechanism to prevent infinite loops by limiting the number of attempts to fix errors.

• Tracking Seen Errors: Introduced a seen_errors set to track previously encountered errors (out) and preemptively exit the retry loop to avoid retrying fixes for the same error repeatedly.

• Duplicate Error Detection: If the current error (out) is already in seen_errors, the function logs the issue as a customer environment error and exits gracefully.

Unit Tests:

Added 4 unit tests.

Result:

How to interpret the above?
1. Output error received to try_mitigate_issues_if_any() .
2. Tries to fix the error in self.check_known_issues_and_attempt_fix() via disablerepo command.
3. Performs post mitigation check, hits an error & recursively invokes try_mitigate_issues_if_any() with this error.
4. Error signature is already present in seen_errors, logs error message (highlighted in yellow) & exits.

FAQ: Why is MAX_RETRY_ATTEMPTS_FOR_ERROR_MITIGATION = 10 (A very high value)?

Answer: TL;DR - To give self.check_known_issues_and_attempt_fix() a chance to fix things on its own, especially if the error stack is large.

For example,
Error 1 -> Try to mitigate, which raises Error 2 -> call self.check_known_issues_and_attempt_fix() , which raises Error 3 -> call self.check_known_issues_and_attempt_fix() , which raises Error 4 -> call self.check_known_issues_and_attempt_fix() , which raises Error 5 -> call self.check_known_issues_and_attempt_fix() , after which there are no other errors.

If we set MAX_RETRY_ATTEMPTS_FOR_ERROR_MITIGATION to a lower value like 3, it will pre-emptively exit the loop, although there might be no fault in the first place.

If we see the same error again and again, the check against seen_errors should exit the loop as expected.

[Copilot]

Pull Request Overview

This PR fixes a bug in the YUM package manager where unbounded retries during error mitigation caused the extension to timeout. Key changes include:

• Adding a retry limit mechanism with retry_count and max_retries to prevent infinite recursion.
• Introducing a seen_errors set to avoid duplicate error mitigation attempts.
• Adding 4 new unit tests to verify that the retry limit is enforced.

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
src/core/tests/library/LegacyEnvLayerExtensions.py	Adds a test scenario for issue mitigation retries
src/core/tests/Test_YumPackageManager.py	Adds 4 units tests
src/core/src/package_managers/YumPackageManager.py	Updates error mitigation logic with retry control and error logging
src/core/src/bootstrap/Constants.py	Introduces the MAX_RETRY_ATTEMPTS_FOR_ERROR_MITIGATION constant

[rirakshi-dev]

@microsoft-github-policy-service agree company="Microsoft"

@microsoft-github-policy-service agree company="Microsoft"

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.51%. Comparing base (5faac3e) to head (42ba8b2).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #303      +/-   ##
==========================================
+ Coverage   92.49%   92.51%   +0.02%     
==========================================
  Files         100      100              
  Lines       16304    16351      +47     
==========================================
+ Hits        15080    15127      +47     
  Misses       1224     1224              

Flag	Coverage Δ
python27	92.51% <100.00%> (+0.02%)	⬆️
python39	92.51% <100.00%> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[feng-j678]

the pr description is awesome and very detailed