Use test containers for CI. #16
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add wait_for_tidb function that waits for TiDB to be ready before tests start. Uses extended timeout (240s) for 6.1.x and 6.5.x versions which need longer startup time, matching the behavior in tidb-playground.sh. This fixes test timeouts where tests were trying to connect before TiDB was ready to accept connections.
Rename 'Cache PD/TiKV Docker image' to 'Pull and cache PD/TiKV Docker image' to match the TiDB step naming and clarify that these steps both pull and cache the images.
Spike investigation for replacing Makefile + Docker setup with Testcontainers: Benefits: - Pure Go solution (no shell scripts, no Makefile) - Automatic container lifecycle management - Built-in readiness checks (no manual wait loops) - Parallel test execution support - Better error messages and debugging - Simpler CI/CD (just 'go test') Includes: - Comprehensive spike document with examples - Proof of concept helper code - Migration strategy - Performance comparison - Trade-offs analysis This is exploratory - not ready for production use yet.
Testcontainers Go automatically works with Podman without special configuration. Updated spike document to include: - Podman benefits (rootless, daemonless, better security) - Automatic detection (works transparently) - Configuration options (CONTAINER_HOST env var) - Podman-specific considerations for TiDB (use manual multi-container instead of Compose module) - Testing Podman compatibility Podman is a drop-in replacement for Docker, so tests should work without code changes when Podman is available.
Built working POC for replacing Makefile + Docker with Testcontainers: Implementation: - testcontainers_helper.go: Helper functions for starting MySQL containers using GenericContainer (compatible with Go 1.21) - data_source_databases_testcontainers_test.go: Example test showing Testcontainers usage - Uses build tag 'testcontainers' so old tests still work Features: - Automatic container lifecycle management - Works with Docker and Podman (auto-detection) - Sets environment variables automatically - Cleanup on test completion - Compatible with existing test infrastructure Usage: go test -tags=testcontainers -v ./mysql/... -run TestAccDataSourceDatabases_WithTestcontainers Dependencies: - testcontainers-go v0.30.0 (compatible with Go 1.21) - Uses GenericContainer instead of MySQL module for compatibility This is a proof of concept - ready for testing and evaluation.
The testcontainers-go dependency uses go-m1cpu which generates harmless CGO compiler warnings. Added: 1. .testcontainers-build-flags.sh: Script to suppress warnings 2. Updated TESTCONTAINERS_POC.md: Documentation about warnings 3. Note in testcontainers_helper.go: Explanation of warnings These warnings don't affect functionality - they're from CGO code in a third-party package. Users can suppress them with CGO_CFLAGS if desired.
Upgrades: - testcontainers-go: v0.30.0 → v0.38.0 (latest compatible with Go 1.23) - Includes bug fixes and improvements New test file: - resource_database_testcontainers_test.go: Converted TestAccDatabase and TestAccDatabase_collationChange to use Testcontainers This demonstrates: - Multiple test functions in one file using Testcontainers - Import state testing with Testcontainers - PreConfig hooks work correctly with Testcontainers - CheckDestroy functions work with Testcontainers
This commit migrates the test infrastructure from Makefile-based Docker management to testcontainers, providing a more maintainable and portable testing solution. Key changes: - Add TiDB testcontainers support with multi-container cluster setup - Implement startTiDBCluster() for PD, TiKV, and TiDB coordination - Add shared TiDB cluster management in TestMain - Support all TiDB versions: 6.1.7, 6.5.12, 7.1.6, 7.5.7, 8.1.2, 8.5.3 - Convert CI workflow to use testcontainers - Replace Makefile targets with direct go test commands - Simplify workflow by removing Docker Buildx caching and mysql-client - Use matrix strategy with db_type and db_version for all database types - Reduce workflow complexity by ~100 lines - Create Go-based test runner (scripts/test-runner.go) - Replace bash script with Go implementation for better maintainability - Add parallelism support via PARALLEL environment variable - Add formatted table output using tablewriter library - Support MySQL, Percona, MariaDB, and TiDB test matrices - Improve test infrastructure - Rename TESTCONTAINERS_MYSQL_IMAGE to DOCKER_IMAGE for clarity - Add shared container pattern with TestMain for efficiency - Suppress MySQL driver "unexpected EOF" log messages - Add mysql_no_login plugin installation attempt with graceful fallback - Fix TestAccUser_auth to skip when plugin unavailable - Update dependencies - Upgrade testcontainers-go to v0.40.0 - Add tablewriter library for formatted output - Add testcontainers-go/network for TiDB multi-container support Benefits: - Simpler CI/CD: No complex Makefile coordination needed - Better isolation: Testcontainers handles container lifecycle - Faster execution: Shared containers reduce startup overhead - Easier maintenance: Pure Go implementation, no shell scripts - Consistent approach: All database types use same testcontainers pattern Test coverage: - MySQL: 5.6, 5.7, 8.0 - Percona: 5.7, 8.0 - MariaDB: 10.3, 10.8, 10.10 - TiDB: 6.1.7, 6.5.12, 7.1.6, 7.5.7, 8.1.2, 8.5.3
- Renamed GNUmakefile to Makefile (GNU Make is standard on Linux/macOS) - Replaced all Docker-based test targets with testcontainers-based implementations - Removed manual Docker container management (docker run, wait loops, cleanup) - Updated testversion, testpercona, testmariadb, testtidb targets to use testcontainers - Added help target as default that shows all available tasks with descriptions - Added ## comments to all targets for help system - Kept testrdsdb targets unchanged (they require actual RDS instances)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.