pySCG: add testing framework for code examples and docs #1011
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…d docs Add pytest-based testing framework with 765 test cases to validate Python code examples and Markdown documentation structure. Features: - Python validation: syntax, deprecation warnings, expected output - Markdown validation: required sections, tables, section order, code refs - Link validation: internal links, index table, code file references - Smart CI/CD: tests only changed CWE directories on PRs (~1 min) - Full suite: runs all tests on pushes to main (~3 min) - Multi-version: Python 3.9-3.14 testing with tox - Clean reporting: generate_issue_report.py for actionable summaries - Auto cleanup: removes test artifacts after test runs CI/CD Integration: - GitHub Actions workflow with matrix strategy - Lychee link checker (internal links in CI, external available locally) - Coverage reporting to Codecov - Fast execution with uv package manager Documentation: - Comprehensive tests/README.md with usage examples - Updated CONTRIBUTING.md with testing workflow - KNOWN_ISSUES.md documents 44 files with pre-existing issues The framework identifies issues in 44 files (31 documentation files with missing sections/tables/order issues, 13 Python files with timeouts/warnings) documented in KNOWN_ISSUES.md. These pre-existing issues can be addressed in follow-up PRs. All new contributions will be validated automatically. Closes ossf#1010 Signed-off-by: tommcd <[email protected]>
- Add blank lines around lists, headings, and code blocks (MD022, MD031, MD032) - Standardize list style to dashes (MD004) - Add language specifications to code blocks (MD040) - Remove trailing spaces (MD009) - Add trailing newline to KNOWN_ISSUES.md (MD047) - Standardize bold style to underscores (MD050) These errors were caught by GitHub Super-Linter but not by local auto-fix. All 7 files now pass markdownlint validation. Signed-off-by: tommcd <[email protected]>
tommcd
force-pushed
the
feature/python-testing-framework
branch
from
59d5ee7 to
32ebdf7
Compare
Change readme reference from 'README.md' to 'tests/README.md' to match actual file location and fix build error in CI. Signed-off-by: tommcd <[email protected]>
Fix bash script that generates pytest -k filter to properly handle single directory case. Trim trailing spaces before joining with ' or ' to avoid invalid pytest expressions like 'CWE-703 or '. Signed-off-by: tommcd <[email protected]>
tommcd
changed the title
Mark CWE-390 files with EXPECTED_TIMEOUT to skip timeout tests. These files intentionally contain infinite loops/blocking operations for educational purposes as documented in KNOWN_ISSUES.md. Signed-off-by: tommcd <[email protected]>
Fix typo 'exammple01.py' -> 'example01.py' in markdown link. Signed-off-by: tommcd <[email protected]>
Mark CWE-798/compliant01.py with EXPECTED_FAILURE for deprecated Path() kwargs usage. This is a known issue documented in KNOWN_ISSUES.md that will be addressed in a follow-up PR. Signed-off-by: tommcd <[email protected]>
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. Thanks for integrating Codecov - We've got you covered ☂️ |
Remove invalid 'exist_ok=True' parameter from Path() constructor. This parameter belongs to Path.mkdir(), not Path(). The invalid usage triggers DeprecationWarning in Python 3.12+ and will cause errors in Python 3.14+. The file is created by open() on the next line, so exist_ok was never needed here. This is a bug fix with no behavior change. Signed-off-by: tommcd <[email protected]>
Add critical test to ensure README inlined code blocks match actual Python files (after stripping SPDX headers and test markers). This test catches a common documentation error where: - Code is updated but README isn't updated - SPDX headers are incorrectly included in inlined code - Test markers (EXPECTED_TIMEOUT/EXPECTED_FAILURE) appear in docs Currently identifies 18 files with mismatches (pre-existing issues). These should be fixed in follow-up PRs. Signed-off-by: tommcd <[email protected]>
Update generate_issue_report.py to detect and report inlined code mismatches. Regenerated KNOWN_ISSUES.md showing 18 files where README inlined code doesn't match actual Python files. Signed-off-by: tommcd <[email protected]>
Fix markdown linting error MD047. Signed-off-by: tommcd <[email protected]>
Fix MD012 markdown linting error. Signed-off-by: tommcd <[email protected]>
Fix generate_issue_report.py to add trailing newline when saving. Signed-off-by: tommcd <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add pytest-based testing framework with 783 test cases to validate Python code examples and Markdown documentation structure.
Features:
New: Inlined Code Validation
CI/CD Integration:
Documentation:
The framework identifies issues in 53 files (35 documentation files with missing sections/tables/order/inlined code issues, 11 Python files with timeouts, 1 with deprecation warning) documented in KNOWN_ISSUES.md. These pre-existing issues can be addressed in follow-up PRs. All new contributions will be validated automatically.
Closes #1010