[feat] Adding support for PostgreSQL storage via SQLAlchemy #3567
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…e-only behaviour and schema layout. - Introduce _ConnectionStrategy class and _SqliteConnector sub-classe to encapsulate URL building, engine creation, and per-dialect kwargs. - Build schema with SQLAlchemy Core (MetaData, Table, Column), preserving table/column names and the index_testcases_time index. - Convert INSERT, SELECT, DELETE queries to Core expressions. - Enable SQLite foreign keys via PRAGMA foreign_keys=ON on each connection (matches previous cascade semantics). - Replace raw sqlite3 connection/locking with SQLAlchemy transactional contexts (engine.begin()) - Drop legacy file lock code and direct sqlite3.connect() calls. - Remove uuids_sql raw string building in remove paths; use SQLAlchemy's in_() instead.
- Introduce _PostgresConnector and wire backend='postgresql' in
StorageBackend.create().
- Read connection parameters from site config: storage/0/postgresql_driver, postgresql_host, postgresql_port,
postgresql_db, postgresql_conn_timeout and credentials from env: RFM_POSTGRES_USER / RFM_POSTGRES_PASSWORD.
- Pass connect_timeout via DBAPI connect args.
- Extend config schema (schemas/config.json) with the new Postgres options
and add sensible defaults (driver=psycopg2, conn_timeout=60).
- Keep SQLite as-is; no behavior change for existing users.
- Add psycopg2-binary==2.9.8 to requirements.txt to provide the PG driver.
- Add ConnectionStrategy.json_column_type property (default Text, overridden to JSONB for Postgres). - Use this type in schema so Postgres stores sessions.json_blob as JSONB. - Normalize JSONB values in _decode_sessions() by serializing non-strings to JSON strings for consistent downstream parsing. - CLI: display backend info depending on storage backend (SQLite file path vs PostgreSQL host:port/db). - write report as encoded JSON if postgresql
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #3567 +/- ##
============================================
- Coverage 91.22% 69.17% -22.05%
============================================
Files 62 60 -2
Lines 13525 13482 -43
============================================
- Hits 12338 9326 -3012
- Misses 1187 4156 +2969 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
mredenti
changed the title
mredenti
changed the title
vkarak
added
request for enhancement
reporting
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces SQLAlchemy Core to abstract database interactions in Reframe's storage backend, adding support for PostgreSQL alongside SQLite. It serves as a proof-of-concept to gather feedback on this approach before adding comprehensive tests and documentation. I have manually tested PostgreSQL with
psycopg2for schema creation, storage, and CLI output. Full PostgreSQL tests intest_storage.pywill follow approval.Motivation
Supporting multiple SQL dialects (SQLite, PostgreSQL, MySQL) is complex due to differences in:
psycopg2orpg8000)SQLAlchemy Core unifies these interactions, improving maintainability and extensibility.
Next Steps (pending feedback)
Note Users can replace
psycopg2with their preferred PostgreSQL driver (e.g.,pg8000) by updatingrequirements.txtandpostgresql_driverin the system configuration. Suggestions on test cases or PostgreSQL edge cases are welcome.