diff --git a/README.md b/README.md index a1b1da6..bcbd778 100644 --- a/README.md +++ b/README.md @@ -69,9 +69,10 @@ The server starts on port 8080 by default. Use `--address :PORT` to customize. **What happens when the server starts:** 1. Loads configuration from the specified YAML file -2. Immediately fetches registry data from the configured source -3. Starts background sync coordinator for automatic updates -4. Serves MCP Registry API endpoints on the configured address +2. Runs database migrations automatically (if database is configured) +3. Immediately fetches registry data from the configured source +4. Starts background sync coordinator for automatic updates +5. Serves MCP Registry API endpoints on the configured address For detailed configuration options and examples, see the [examples/README.md](examples/README.md). @@ -83,7 +84,7 @@ The `thv-registry-api` CLI provides the following commands: # Start the API server thv-registry-api serve --config config.yaml [--address :8080] -# Run database migrations +# Manually run database migrations thv-registry-api migrate up --config config.yaml [--yes] thv-registry-api migrate down --config config.yaml --num-steps N [--yes] @@ -198,8 +199,10 @@ The server optionally supports PostgreSQL database connectivity for storing regi |-------|------|----------|---------|-------------| | `host` | string | Yes | - | Database server hostname or IP address | | `port` | int | Yes | - | Database server port | -| `user` | string | Yes | - | Database username | +| `user` | string | Yes | - | Database username for normal operations | | `passwordFile` | string | No* | - | Path to file containing the database password | +| `migrationUser` | string | No | `user` | Database username for running migrations (should have elevated privileges) | +| `migrationPasswordFile` | string | No | `passwordFile` | Path to file containing the migration user's password | | `database` | string | Yes | - | Database name | | `sslMode` | string | No | `require` | SSL mode (`disable`, `require`, `verify-ca`, `verify-full`) | | `maxOpenConns` | int | No | `25` | Maximum number of open connections to the database | @@ -210,7 +213,9 @@ The server optionally supports PostgreSQL database connectivity for storing regi #### Password Security -The server supports secure password management with the following priority order: +The server supports secure password management with separate credentials for normal operations and migrations. + +**Normal Operations Password (for `user`):** 1. **Password File** (Recommended for production): - Set `passwordFile` to the path of a file containing only the password @@ -231,7 +236,29 @@ The server supports secure password management with the following priority order thv-registry-api serve --config config.yaml ``` +**Migration User Password (for `migrationUser`):** + +1. **Migration Password File**: + - Set `migrationPasswordFile` to the path of a file containing the migration user's password + - Falls back to `passwordFile` if not specified + - Example: + ```yaml + database: + migrationUser: db_migrator + migrationPasswordFile: /secrets/db-migration-password + ``` + +2. **Environment Variable**: + - Set `THV_DATABASE_MIGRATION_PASSWORD` environment variable + - Falls back to `THV_DATABASE_PASSWORD` if not specified + - Example: + ```bash + export THV_DATABASE_MIGRATION_PASSWORD="migration-user-password" + thv-registry-api serve --config config.yaml + ``` + **Security Best Practices:** +- Use separate users for migrations (with elevated privileges) and normal operations (read-only or limited) - Never commit passwords directly in configuration files - Use password files with restricted permissions (e.g., `chmod 400`) - In Kubernetes, mount passwords from Secrets @@ -252,9 +279,52 @@ Tune these values based on your workload: #### Database Migrations -The server includes built-in database migration commands to manage the database schema. +The server includes built-in database migration support to manage the database schema. + +**Automatic migrations on startup:** + +When you start the server with `serve`, database migrations run automatically if database configuration is present in your config file. This ensures your database schema is always up to date. + +The only thing necessary is granting the role `toolhive_registry_server` to +the database user you want, for example + +```sql +BEGIN; -**Running migrations with CLI:** +DO $$ + BEGIN + IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'toolhive_registry_server') THEN + CREATE ROLE toolhive_registry_server; + END IF; + END +$$; + +DO $$ + BEGIN + IF NOT EXISTS (SELECT FROM pg_user WHERE usename = 'thvr_user') THEN + CREATE USER thvr_user WITH PASSWORD 'thvr_user_pass'; + END IF; + END +$$; + +GRANT toolhive_registry_server TO thvr_user; + +COMMIT; +``` + +To help with that, we plan to add a `prime` subcommand that does just that +in an idempotent fashion with username and password provided by the user. + +Once done, you start the server as follows + +```bash +# Migrations run automatically when database is configured +thv-registry-api serve --config examples/config-database-dev.yaml +``` + +**Manual migration commands (optional):** + +You can also run migrations manually using the CLI commands: ```bash # Apply all pending migrations @@ -285,8 +355,7 @@ task migrate-down CONFIG=examples/config-database-dev.yaml NUM_STEPS=1 1. **Configure database**: Create a config file with database settings (see [examples/config-database-dev.yaml](examples/config-database-dev.yaml)) 2. **Set password**: Either set `THV_DATABASE_PASSWORD` env var or use `passwordFile` in config -3. **Run migrations**: Use `migrate up` to apply schema changes -4. **Start server**: Run `serve` command with the same config file +3. **Start server**: Run `serve` command - migrations will run automatically **Example: Local development setup** @@ -302,10 +371,7 @@ docker run -d --name postgres \ # 2. Set password environment variable export THV_DATABASE_PASSWORD="devpassword" -# 3. Run migrations -task migrate-up CONFIG=examples/config-database-dev.yaml - -# 4. Start the server +# 3. Start the server (migrations run automatically) thv-registry-api serve --config examples/config-database-dev.yaml ``` @@ -316,12 +382,7 @@ thv-registry-api serve --config examples/config-database-dev.yaml echo "your-secure-password" > /run/secrets/db_password chmod 400 /run/secrets/db_password -# 2. Run migrations (using passwordFile from config) -thv-registry-api migrate up \ - --config examples/config-database-prod.yaml \ - --yes - -# 3. Start the server +# 2. Start the server (migrations run automatically) thv-registry-api serve --config examples/config-database-prod.yaml ``` @@ -597,12 +658,12 @@ docker run -v $(pwd)/examples:/config \ ### Docker Compose -A complete Docker Compose setup is provided in the repository root that includes PostgreSQL, automatic migrations, and the API server. +A complete Docker Compose setup is provided in the repository root that includes PostgreSQL and the API server with automatic migrations. **Quick start:** ```bash -# Start all services (PostgreSQL + migrations + API) +# Start all services (PostgreSQL + API with automatic migrations) docker-compose up # Run in detached mode @@ -620,25 +681,27 @@ docker-compose down -v **Architecture:** -The docker-compose.yaml includes three services: +The docker-compose.yaml includes two services: 1. **postgres** - PostgreSQL 18 database server -2. **migrate** - One-time migration service (runs schema migrations) -3. **registry-api** - Main API server +2. **registry-api** - Main API server (runs migrations automatically on startup) **Service startup flow:** ``` -postgres (healthy) → migrate (completes) → registry-api (starts) +postgres (healthy) → registry-api (runs migrations, then starts) ``` **Configuration:** - Config file: `examples/config-docker.yaml` - Sample data: `examples/registry-sample.json` -- Database password: Set via `THV_DATABASE_PASSWORD` environment variable in docker-compose.yaml +- Database passwords: Set via environment variables in docker-compose.yaml + - `THV_DATABASE_PASSWORD`: Application user password + - `THV_DATABASE_MIGRATION_PASSWORD`: Migration user password The setup demonstrates: -- Database-backed registry storage -- Automatic schema migrations on startup +- Database-backed registry storage with separate users for migrations and operations +- Automatic schema migrations on startup using elevated privileges +- Normal operations using limited database privileges (principle of least privilege) - File-based data source (for demo purposes) - Proper service dependencies and health checks @@ -663,14 +726,23 @@ To use your own registry data: **Database access:** +The Docker Compose setup creates three database users: +- `registry`: Superuser (for administration) +- `db_migrator`: Migration user with schema modification privileges +- `db_app`: Application user with limited data access privileges + To connect to the PostgreSQL database directly: ```bash -# Using psql +# As superuser (for administration) docker exec -it toolhive-registry-postgres psql -U registry -d registry -# Using environment variables from compose +# As application user +docker exec -it toolhive-registry-postgres psql -U db_app -d registry + +# From host machine PGPASSWORD=registry_password psql -h localhost -U registry -d registry +PGPASSWORD=app_password psql -h localhost -U db_app -d registry ``` ## Integration with ToolHive diff --git a/cmd/thv-registry-api/app/migrate_down.go b/cmd/thv-registry-api/app/migrate_down.go index f76d3ca..c419e8c 100644 --- a/cmd/thv-registry-api/app/migrate_down.go +++ b/cmd/thv-registry-api/app/migrate_down.go @@ -43,10 +43,10 @@ func runMigrateDown(cmd *cobra.Command, _ []string) error { return fmt.Errorf("database configuration is required") } - // Get connection string - connString, err := cfg.Database.GetConnectionString() + // Get migration connection string (uses migration user if configured) + connString, err := cfg.Database.GetMigrationConnectionString() if err != nil { - return fmt.Errorf("failed to get database connection string: %w", err) + return fmt.Errorf("failed to get migration connection string: %w", err) } // Prompt user for confirmation if not using --yes flag @@ -103,8 +103,9 @@ func parseMigrateDownFlags(cmd *cobra.Command) (uint, bool, string, error) { } func confirmMigrationDown(numSteps uint, dbCfg *config.DatabaseConfig) bool { - logger.Warnf("WARNING: This will revert %d migration(s) from database: %s@%s:%d/%s", - numSteps, dbCfg.User, dbCfg.Host, dbCfg.Port, dbCfg.Database) + migrationUser := dbCfg.GetMigrationUser() + logger.Warnf("WARNING: This will revert %d migration(s) from database: %s@%s:%d/%s (as user: %s)", + numSteps, migrationUser, dbCfg.Host, dbCfg.Port, dbCfg.Database, migrationUser) logger.Warnf("WARNING: This operation may result in DATA LOSS") fmt.Print("Are you sure you want to continue? Type 'yes' to proceed: ") var response string diff --git a/cmd/thv-registry-api/app/migrate_up.go b/cmd/thv-registry-api/app/migrate_up.go index 1e2f5ef..bf54b37 100644 --- a/cmd/thv-registry-api/app/migrate_up.go +++ b/cmd/thv-registry-api/app/migrate_up.go @@ -46,16 +46,19 @@ func runMigrateUp(cmd *cobra.Command, _ []string) error { return fmt.Errorf("database configuration is required") } - // Get connection string - connString, err := cfg.Database.GetConnectionString() + // Get migration connection string (uses migration user if configured) + connString, err := cfg.Database.GetMigrationConnectionString() if err != nil { - return fmt.Errorf("failed to get database connection string: %w", err) + return fmt.Errorf("failed to get migration connection string: %w", err) } + // Get the migration user for display + migrationUser := cfg.Database.GetMigrationUser() + // Prompt user if not using --yes flag if !yes { - logger.Infof("About to apply migrations to database: %s@%s:%d/%s", - cfg.Database.User, cfg.Database.Host, cfg.Database.Port, cfg.Database.Database) + logger.Infof("About to apply migrations to database: %s@%s:%d/%s (as user: %s)", + migrationUser, cfg.Database.Host, cfg.Database.Port, cfg.Database.Database, migrationUser) fmt.Print("Continue? (yes/no): ") var response string if _, err := fmt.Scanln(&response); err != nil { diff --git a/cmd/thv-registry-api/app/serve.go b/cmd/thv-registry-api/app/serve.go index 670010e..9375153 100644 --- a/cmd/thv-registry-api/app/serve.go +++ b/cmd/thv-registry-api/app/serve.go @@ -8,12 +8,14 @@ import ( "syscall" "time" + "github.com/jackc/pgx/v5" "github.com/spf13/cobra" "github.com/spf13/viper" "github.com/stacklok/toolhive/pkg/logger" "sigs.k8s.io/controller-runtime/pkg/log" "sigs.k8s.io/controller-runtime/pkg/log/zap" + "github.com/stacklok/toolhive-registry-server/database" registryapp "github.com/stacklok/toolhive-registry-server/internal/app" "github.com/stacklok/toolhive-registry-server/internal/config" ) @@ -28,6 +30,8 @@ The server requires a configuration file (--config) that specifies: - Sync policy and filtering rules - All other operational settings +If database configuration is present, migrations will run automatically on startup. + See examples/ directory for sample configurations.`, RunE: runServe, } @@ -73,6 +77,14 @@ func runServe(_ *cobra.Command, _ []string) error { logger.Infof("Loaded configuration from %s (registry: %s, %d registries configured)", configPath, cfg.GetRegistryName(), len(cfg.Registries)) + // Run database migrations if database is configured + if cfg.Database != nil { + logger.Infof("Database configuration found, running migrations...") + if err := runMigrations(ctx, cfg); err != nil { + return fmt.Errorf("failed to run database migrations: %w", err) + } + } + // Build application using the builder pattern address := viper.GetString("address") app, err := registryapp.NewRegistryApp( @@ -101,3 +113,61 @@ func runServe(_ *cobra.Command, _ []string) error { // Graceful shutdown return app.Stop(defaultGracefulTimeout) } + +// runMigrations executes database migrations on startup +func runMigrations(ctx context.Context, cfg *config.Config) error { + // Get migration connection string (uses migration user if configured) + connString, err := cfg.Database.GetMigrationConnectionString() + if err != nil { + return fmt.Errorf("failed to get migration connection string: %w", err) + } + + // Log which user is running migrations + migrationUser := cfg.Database.GetMigrationUser() + logger.Infof("Running migrations as user: %s", migrationUser) + + // Connect to database + conn, err := pgx.Connect(ctx, connString) + if err != nil { + return fmt.Errorf("failed to connect to database: %w", err) + } + defer func() { + if closeErr := conn.Close(ctx); closeErr != nil { + logger.Errorf("Error closing database connection: %v", closeErr) + } + }() + + tx, err := conn.BeginTx( + ctx, + pgx.TxOptions{ + IsoLevel: pgx.Serializable, + AccessMode: pgx.ReadWrite, + }, + ) + if err != nil { + return fmt.Errorf("failed to begin transaction: %w", err) + } + defer func() { + if err := tx.Rollback(ctx); err != nil { + logger.Errorf("Error rolling back transaction: %v", err) + } + }() + + // Run migrations + logger.Infof("Applying database migrations...") + if err := database.MigrateUp(ctx, tx.Conn()); err != nil { + return fmt.Errorf("failed to apply migrations: %w", err) + } + + // Get and log current version + version, dirty, err := database.GetVersion(connString) + if err != nil { + logger.Warnf("Unable to get migration version: %v", err) + } else if dirty { + logger.Warnf("Database is in a dirty state at version %d", version) + } else { + logger.Infof("Database migrations completed successfully. Current version: %d", version) + } + + return nil +} diff --git a/cmd/thv-registry-api/app/serve_test.go b/cmd/thv-registry-api/app/serve_test.go new file mode 100644 index 0000000..6aa5c56 --- /dev/null +++ b/cmd/thv-registry-api/app/serve_test.go @@ -0,0 +1,136 @@ +package app + +import ( + "context" + "fmt" + "net/url" + "os" + "testing" + + "github.com/jackc/pgx/v5" + "github.com/stretchr/testify/require" + + "github.com/stacklok/toolhive-registry-server/database" + "github.com/stacklok/toolhive-registry-server/internal/config" +) + +// setupBenchmarkDB sets up a test database container for benchmarking +// Returns a config and cleanup function +func setupBenchmarkDB(t *testing.T) (*config.Config, string, func()) { + t.Helper() + + ctx := context.Background() + db, cleanupFunc := database.SetupTestDBContainer(t, ctx) + + // Get connection details from the database connection + connStr := db.Config().ConnString() + parsedURL, err := url.Parse(connStr) + require.NoError(t, err) + + tx, err := db.BeginTx(ctx, pgx.TxOptions{ + IsoLevel: pgx.Serializable, + AccessMode: pgx.ReadWrite, + }) + require.NoError(t, err) + defer tx.Commit(ctx) + + // Extract connection details + host := parsedURL.Hostname() + port := 5432 + if parsedURL.Port() != "" { + _, err := fmt.Sscanf(parsedURL.Port(), "%d", &port) + require.NoError(t, err) + } + + dbName := "testdb" + + user := "appuser" + password := "apppass" + + // Create the toolhive_registry_server role if it doesn't exist + // (This role is normally created by migrations, but we need it for the test user) + // Use DO block to check existence first since CREATE ROLE IF NOT EXISTS may not work in all contexts + _, err = tx.Conn().Exec(ctx, ` + DO $$ + BEGIN + IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'toolhive_registry_server') THEN + CREATE ROLE toolhive_registry_server; + END IF; + END + $$; + `) + require.NoError(t, err) + + // Create the application user with the given password + _, err = tx.Conn().Exec(ctx, fmt.Sprintf( + "CREATE USER %s WITH PASSWORD '%s'", + pgx.Identifier{user}.Sanitize(), + password, + )) + require.NoError(t, err) + + // Grant the toolhive_registry_server role to the user + _, err = tx.Conn().Exec(ctx, fmt.Sprintf( + "GRANT toolhive_registry_server TO %s", + pgx.Identifier{user}.Sanitize(), + )) + require.NoError(t, err) + + adminUser := "testuser" + adminPassword := "testpass" + os.Setenv("THV_DATABASE_MIGRATION_PASSWORD", adminPassword) + + // Create config with database settings + cfg := &config.Config{ + Database: &config.DatabaseConfig{ + Host: host, + Port: port, + User: user, + MigrationUser: adminUser, + Database: dbName, + SSLMode: "disable", + }, + Registries: []config.RegistryConfig{ + // Add a minimal registry config to satisfy validation + { + Name: "test", + Format: "toolhive", + File: &config.FileConfig{ + Path: "./examples/registry-sample.json", + }, + SyncPolicy: &config.SyncPolicyConfig{ + Interval: "1h", + }, + }, + }, + } + + return cfg, connStr, func() { + cleanupFunc() + os.Unsetenv("THV_DATABASE_MIGRATION_PASSWORD") + } +} + +// TestRunMigrations tests the runMigrations function +func TestRunMigrations(t *testing.T) { + t.Parallel() + ctx := context.Background() + + // Set up database container once for all iterations + cfg, connStr, cleanupFunc := setupBenchmarkDB(t) + t.Cleanup(cleanupFunc) + + // Run migrations. This was tested with 100 instances without issues. + for i := range 3 { + t.Run(fmt.Sprintf("migrations-instance-%d", i), func(t *testing.T) { + t.Parallel() + err := runMigrations(ctx, cfg) + require.NoError(t, err) + + version, dirty, err := database.GetVersion(connStr) + require.NoError(t, err) + require.False(t, dirty) + require.Equal(t, uint(1), version) + }) + } +} diff --git a/database/migrations/000001_init.down.sql b/database/migrations/000001_init.down.sql index cdb62dd..538f360 100644 --- a/database/migrations/000001_init.down.sql +++ b/database/migrations/000001_init.down.sql @@ -13,3 +13,26 @@ DROP TYPE IF EXISTS registry_type; DROP TYPE IF EXISTS transport; DROP TYPE IF EXISTS remote_transport; DROP TYPE IF EXISTS icon_theme; + +-- Revoke permissions on existing tables from application role if it exists +-- This removes the privileges granted in the up migration +REVOKE SELECT ON ALL TABLES IN SCHEMA public FROM toolhive_registry_server; +REVOKE INSERT ON ALL TABLES IN SCHEMA public FROM toolhive_registry_server; +REVOKE UPDATE ON ALL TABLES IN SCHEMA public FROM toolhive_registry_server; +REVOKE DELETE ON ALL TABLES IN SCHEMA public FROM toolhive_registry_server; + +-- Revoke permissions on existing sequences from application role +REVOKE USAGE ON ALL SEQUENCES IN SCHEMA public FROM toolhive_registry_server; +REVOKE SELECT ON ALL SEQUENCES IN SCHEMA public FROM toolhive_registry_server; + +-- Revoke default permissions on future tables from application role +ALTER DEFAULT PRIVILEGES IN SCHEMA public REVOKE SELECT ON TABLES FROM toolhive_registry_server; +ALTER DEFAULT PRIVILEGES IN SCHEMA public REVOKE INSERT ON TABLES FROM toolhive_registry_server; +ALTER DEFAULT PRIVILEGES IN SCHEMA public REVOKE UPDATE ON TABLES FROM toolhive_registry_server; +ALTER DEFAULT PRIVILEGES IN SCHEMA public REVOKE DELETE ON TABLES FROM toolhive_registry_server; + +-- Revoke default permissions on future sequences from application role +ALTER DEFAULT PRIVILEGES IN SCHEMA public REVOKE USAGE ON SEQUENCES FROM toolhive_registry_server; +ALTER DEFAULT PRIVILEGES IN SCHEMA public REVOKE SELECT ON SEQUENCES FROM toolhive_registry_server; + +DROP ROLE IF EXISTS toolhive_registry_server; diff --git a/database/migrations/000001_init.up.sql b/database/migrations/000001_init.up.sql index 7602c86..82f8a04 100644 --- a/database/migrations/000001_init.up.sql +++ b/database/migrations/000001_init.up.sql @@ -109,3 +109,32 @@ CREATE TABLE mcp_server_icon ( theme icon_theme, -- NULL means 'any' theme. PRIMARY KEY (server_id, source_uri, mime_type, theme) -- Unclear if mime_type or theme should be part of the PK ); + +DO $$ +BEGIN + IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'toolhive_registry_server') THEN + CREATE ROLE toolhive_registry_server; + END IF; +END +$$; + +-- Grant permissions on existing tables to application role if it exists +-- This allows normal operations with limited privileges +GRANT SELECT ON ALL TABLES IN SCHEMA public TO toolhive_registry_server; +GRANT INSERT ON ALL TABLES IN SCHEMA public TO toolhive_registry_server; +GRANT UPDATE ON ALL TABLES IN SCHEMA public TO toolhive_registry_server; +GRANT DELETE ON ALL TABLES IN SCHEMA public TO toolhive_registry_server; + +-- Grant permissions on existing sequences to application role +GRANT USAGE ON ALL SEQUENCES IN SCHEMA public TO toolhive_registry_server; +GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO toolhive_registry_server; + +-- Grant permissions on future tables to application role +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO toolhive_registry_server; +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT INSERT ON TABLES TO toolhive_registry_server; +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT UPDATE ON TABLES TO toolhive_registry_server; +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT DELETE ON TABLES TO toolhive_registry_server; + +-- Grant permissions on future sequences to application role +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT USAGE ON SEQUENCES TO toolhive_registry_server; +ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON SEQUENCES TO toolhive_registry_server; diff --git a/docker-compose.yaml b/docker-compose.yaml index edaccac..f3525bb 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -1,11 +1,15 @@ # ToolHive Registry API Server - Docker Compose Configuration # # Services: -# 1. postgres - PostgreSQL 18 database +# 1. postgres - PostgreSQL 18 database with separate users for migrations and operations # 2. registry-api - Main API server (runs migrations on startup) # # Startup flow: -# postgres (healthy) → registry-api (runs migrations then starts) +# postgres (healthy) → registry-api (runs migrations as db_migrator, then operates as db_app) +# +# Database Users: +# - db_migrator: Migration user with elevated privileges (CREATE, ALTER, DROP) +# - db_app: Application user with limited privileges (SELECT, INSERT, UPDATE, DELETE) # # Migrations are embedded in the binary and run automatically on startup. @@ -22,6 +26,7 @@ services: - "5432:5432" volumes: - postgres_data:/var/lib/postgresql/data + - ./docker/postgres-init.sh:/docker-entrypoint-initdb.d/init.sh:ro healthcheck: test: ["CMD-SHELL", "pg_isready -U registry"] interval: 5s @@ -47,7 +52,8 @@ services: command: ["serve", "--config", "/config.yaml", "--address", ":8080"] environment: - LOG_LEVEL=debug - - THV_DATABASE_PASSWORD=registry_password + - THV_DATABASE_PASSWORD=app_password # Application user password + - THV_DATABASE_MIGRATION_PASSWORD=migration_password # Migration user password networks: - registry-network restart: unless-stopped diff --git a/docker/dockerfile-compose b/docker/dockerfile-compose index f8df82b..f03dfa0 100644 --- a/docker/dockerfile-compose +++ b/docker/dockerfile-compose @@ -30,18 +30,16 @@ GOOS=linux go build -ldflags "${LDFLAGS}" -o main ./cmd/thv-registry-api/main.go # Use minimal base image to package the binary FROM index.docker.io/library/alpine@sha256:4b7ce07002c69e8f3d704a9c5d6fd3053be500b7f1c69fc0d80990c2ad8dd412 -COPY --from=builder /workspace/main / -COPY docker/entrypoint.sh /entrypoint.sh +COPY --from=builder /workspace/main /thv-registry-api COPY LICENSE /licenses/LICENSE # Create working directory and data directory with proper permissions -# Make entrypoint script executable RUN mkdir -p /app/data && \ - chown -R 1001:1001 /app && \ - chmod +x /entrypoint.sh + chown -R 1001:1001 /app WORKDIR /app USER 1001 -ENTRYPOINT ["/entrypoint.sh"] +# Migrations run automatically when the server starts with "serve" command +ENTRYPOINT ["/thv-registry-api"] diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh deleted file mode 100644 index a06f9b8..0000000 --- a/docker/entrypoint.sh +++ /dev/null @@ -1,45 +0,0 @@ -#!/bin/sh -set -e - -# Function to run migrations -run_migrations() { - echo "Running database migrations..." - /main migrate up --config "$1" --yes - echo "Migrations completed successfully" -} - -# Check if the first argument is "serve" -if [ "$1" = "serve" ]; then - # Save all original arguments for later - ORIGINAL_ARGS="$@" - - # Extract config path from arguments - CONFIG_PATH="" - shift # Remove "serve" from arguments - - # Parse arguments to find --config - while [ $# -gt 0 ]; do - case "$1" in - --config) - CONFIG_PATH="$2" - shift 2 - ;; - *) - shift - ;; - esac - done - - # Run migrations if config path is found - if [ -n "$CONFIG_PATH" ]; then - run_migrations "$CONFIG_PATH" - else - echo "Warning: No config file specified, skipping migrations" - fi - - # Start the server with all original arguments - exec /main $ORIGINAL_ARGS -else - # For other commands (like migrate, version), just pass through - exec /main "$@" -fi diff --git a/docker/postgres-init.sh b/docker/postgres-init.sh new file mode 100755 index 0000000..e7cc2cd --- /dev/null +++ b/docker/postgres-init.sh @@ -0,0 +1,24 @@ +#!/bin/sh +set -e + +# This script runs automatically when the PostgreSQL container is initialized +# It creates separate database users for migrations and normal operations + +psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL + -- Create migration user with elevated privileges for schema changes + CREATE USER db_migrator WITH PASSWORD 'migration_password'; + GRANT ALL PRIVILEGES ON DATABASE $POSTGRES_DB TO db_migrator; + GRANT ALL PRIVILEGES ON SCHEMA public TO db_migrator; + + -- Create application user with limited privileges for normal operations + CREATE USER db_app WITH PASSWORD 'app_password'; + GRANT CONNECT ON DATABASE $POSTGRES_DB TO db_app; + GRANT USAGE ON SCHEMA public TO db_app; + + -- Note: Table-level permissions will be granted automatically after migrations + -- via ALTER DEFAULT PRIVILEGES (done in migration files) +EOSQL + +echo "Database users created successfully:" +echo " - db_migrator: Migration user with elevated privileges" +echo " - db_app: Application user with limited privileges" diff --git a/docs/cli/thv-registry-api_serve.md b/docs/cli/thv-registry-api_serve.md index 13d77da..9976f82 100644 --- a/docs/cli/thv-registry-api_serve.md +++ b/docs/cli/thv-registry-api_serve.md @@ -22,6 +22,8 @@ The server requires a configuration file (--config) that specifies: - Sync policy and filtering rules - All other operational settings +If database configuration is present, migrations will run automatically on startup. + See examples/ directory for sample configurations. ``` diff --git a/examples/config-database-dev.yaml b/examples/config-database-dev.yaml index 5011ada..f73a6d6 100644 --- a/examples/config-database-dev.yaml +++ b/examples/config-database-dev.yaml @@ -11,8 +11,9 @@ # # Usage: # export THV_DATABASE_PASSWORD="devpassword" -# thv-registry-api migrate up --config examples/config-database-dev.yaml # thv-registry-api serve --config examples/config-database-dev.yaml +# +# Note: Database migrations run automatically when the server starts # Registry name/identifier registryName: dev-registry diff --git a/examples/config-database-prod.yaml b/examples/config-database-prod.yaml index 2f17cad..f4bb598 100644 --- a/examples/config-database-prod.yaml +++ b/examples/config-database-prod.yaml @@ -19,11 +19,10 @@ # 4. SSL certificates configured if using verify-ca or verify-full # # Usage: -# # Run migrations -# thv-registry-api migrate up --config examples/config-database-prod.yaml -# -# # Start server +# # Start server (migrations run automatically) # thv-registry-api serve --config examples/config-database-prod.yaml +# +# Note: Database migrations run automatically when the server starts # Registry name/identifier registryName: production-registry diff --git a/examples/config-database-separate-users.yaml b/examples/config-database-separate-users.yaml new file mode 100644 index 0000000..96027e0 --- /dev/null +++ b/examples/config-database-separate-users.yaml @@ -0,0 +1,81 @@ +# Example configuration demonstrating separate database users for migrations and operations +# +# This configuration shows how to use different database users: +# - Migration user: Has elevated privileges (CREATE, ALTER, DROP) for schema changes +# - Regular user: Has limited privileges (SELECT, INSERT, UPDATE, DELETE) for normal operations +# +# Security Benefits: +# - Migrations run with elevated privileges only when needed +# - Normal operations run with minimal required privileges +# - Reduces attack surface and follows principle of least privilege +# +# Setup Instructions: +# 1. Create PostgreSQL users with appropriate privileges: +# ```sql +# -- Migration user with schema modification privileges +# CREATE USER db_migrator WITH PASSWORD 'migration_password'; +# GRANT ALL PRIVILEGES ON DATABASE toolhive_registry TO db_migrator; +# GRANT ALL PRIVILEGES ON SCHEMA public TO db_migrator; +# +# -- Regular user with limited privileges +# CREATE USER db_app WITH PASSWORD 'app_password'; +# GRANT CONNECT ON DATABASE toolhive_registry TO db_app; +# GRANT USAGE ON SCHEMA public TO db_app; +# -- After migrations, grant specific table permissions: +# GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO db_app; +# GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO db_app; +# ``` +# +# 2. Create password files or set environment variables: +# ```bash +# # Option 1: Password files (recommended for production) +# echo "app_password" > /secrets/db-password +# echo "migration_password" > /secrets/db-migration-password +# chmod 400 /secrets/db-* +# +# # Option 2: Environment variables (for development) +# export THV_DATABASE_PASSWORD="app_password" +# export THV_DATABASE_MIGRATION_PASSWORD="migration_password" +# ``` +# +# 3. Start the server (migrations run automatically with migration user): +# thv-registry-api serve --config examples/config-database-separate-users.yaml + +# Registry name/identifier +registryName: secure-registry + +# Data source configuration +source: + type: git + format: toolhive + git: + repository: https://github.com/stacklok/toolhive.git + branch: main + path: pkg/registry/data/registry.json + +# Automatic synchronization policy +syncPolicy: + interval: "30m" + +# PostgreSQL database configuration with separate users +database: + # Database connection details + host: localhost + port: 5432 + database: toolhive_registry + sslMode: require + + # Regular user for normal operations (limited privileges) + user: db_app + passwordFile: /secrets/db-password + # Or use environment variable: THV_DATABASE_PASSWORD + + # Migration user for schema changes (elevated privileges) + migrationUser: db_migrator + migrationPasswordFile: /secrets/db-migration-password + # Or use environment variable: THV_DATABASE_MIGRATION_PASSWORD + + # Connection pool settings + maxOpenConns: 25 + maxIdleConns: 5 + connMaxLifetime: "5m" diff --git a/examples/config-docker.yaml b/examples/config-docker.yaml index cc6b2ba..0141233 100644 --- a/examples/config-docker.yaml +++ b/examples/config-docker.yaml @@ -12,11 +12,10 @@ # # The docker-compose setup includes: # 1. postgres - PostgreSQL 18 database server -# 2. migrate - One-time migration service (runs schema migrations) -# 3. registry-api - Main API server +# 2. registry-api - Main API server (runs migrations automatically on startup) # # Database password is provided via THV_DATABASE_PASSWORD environment variable -# set in the docker-compose.yaml file for both migrate and registry-api services. +# set in the docker-compose.yaml file for the registry-api service. # Registry name/identifier registryName: docker-registry @@ -35,9 +34,10 @@ syncPolicy: # Check for file changes every 5 minutes interval: "5m" -# PostgreSQL database configuration -# The password is provided via THV_DATABASE_PASSWORD environment variable -# This is set in docker-compose.yaml for both the migrate and registry-api services +# PostgreSQL database configuration with separate users for migrations and operations +# This demonstrates the principle of least privilege: +# - Migration user (db_migrator) has elevated privileges for schema changes +# - Application user (db_app) has limited privileges for normal operations database: # Use service name from docker-compose as hostname host: postgres @@ -45,11 +45,17 @@ database: # PostgreSQL default port port: 5432 - # Database credentials (must match POSTGRES_USER in docker-compose.yaml) - user: registry + # Application user with limited privileges (SELECT, INSERT, UPDATE, DELETE) + # Password from THV_DATABASE_PASSWORD environment variable + user: db_app - # Password is read from THV_DATABASE_PASSWORD environment variable - # Do not set passwordFile in Docker Compose - use environment variables instead + # Migration user with elevated privileges (CREATE, ALTER, DROP) + # Password from THV_DATABASE_MIGRATION_PASSWORD environment variable + migrationUser: db_migrator + + # Passwords are read from environment variables (set in docker-compose.yaml): + # - THV_DATABASE_PASSWORD: Application user password + # - THV_DATABASE_MIGRATION_PASSWORD: Migration user password # Database name (must match POSTGRES_DB in docker-compose.yaml) database: registry diff --git a/internal/config/config.go b/internal/config/config.go index a5dd06d..0982552 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -355,7 +355,7 @@ type DatabaseConfig struct { // Port is the database server port Port int `yaml:"port"` - // User is the database username + // User is the database username for normal operations User string `yaml:"user"` // PasswordFile is the path to a file containing the database password @@ -363,6 +363,15 @@ type DatabaseConfig struct { // The file should contain only the password with optional trailing whitespace PasswordFile string `yaml:"passwordFile,omitempty"` + // MigrationUser is the database username for running migrations + // This user should have elevated privileges for schema changes + // If not specified, falls back to User + MigrationUser string `yaml:"migrationUser,omitempty"` + + // MigrationPasswordFile is the path to a file containing the migration user's password + // If not specified, falls back to PasswordFile + MigrationPasswordFile string `yaml:"migrationPasswordFile,omitempty"` + // Database is the database name Database string `yaml:"database"` @@ -379,7 +388,7 @@ type DatabaseConfig struct { ConnMaxLifetime string `yaml:"connMaxLifetime,omitempty"` } -// GetPassword returns the database password using the following priority: +// GetPassword returns the database password for normal operations using the following priority: // 1. Read from PasswordFile if specified // 2. Read from THV_DATABASE_PASSWORD environment variable // @@ -410,8 +419,48 @@ func (d *DatabaseConfig) GetPassword() (string, error) { ) } -// GetConnectionString builds a PostgreSQL connection string with proper password handling. -// The password is URL-escaped to handle special characters safely. +// GetMigrationPassword returns the migration user password using the following priority: +// 1. Read from MigrationPasswordFile if specified +// 2. Read from THV_DATABASE_MIGRATION_PASSWORD environment variable +// 3. Fall back to GetPassword() (regular user password) +// +// The password from file will have leading/trailing whitespace trimmed. +func (d *DatabaseConfig) GetMigrationPassword() (string, error) { + // Priority 1: Read from migration password file if specified + if d.MigrationPasswordFile != "" { + // Use filepath.Clean to prevent path traversal attacks + cleanPath := filepath.Clean(d.MigrationPasswordFile) + + data, err := os.ReadFile(cleanPath) + if err != nil { + return "", fmt.Errorf("failed to read migration password from file %s: %w", d.MigrationPasswordFile, err) + } + + // Trim whitespace (including newlines) from file content + password := strings.TrimSpace(string(data)) + return password, nil + } + + // Priority 2: Check migration-specific environment variable + if envPassword := os.Getenv("THV_DATABASE_MIGRATION_PASSWORD"); envPassword != "" { + return envPassword, nil + } + + // Priority 3: Fall back to regular password + return d.GetPassword() +} + +// GetMigrationUser returns the username for migrations. +// If MigrationUser is specified, returns that; otherwise falls back to User. +func (d *DatabaseConfig) GetMigrationUser() string { + if d.MigrationUser != "" { + return d.MigrationUser + } + return d.User +} + +// GetConnectionString builds a PostgreSQL connection string for normal operations +// with proper password handling. The password is URL-escaped to handle special characters safely. func (d *DatabaseConfig) GetConnectionString() (string, error) { password, err := d.GetPassword() if err != nil { @@ -439,6 +488,35 @@ func (d *DatabaseConfig) GetConnectionString() (string, error) { return connString, nil } +// GetMigrationConnectionString builds a PostgreSQL connection string for migrations +// using the migration user credentials. The password is URL-escaped to handle special characters safely. +func (d *DatabaseConfig) GetMigrationConnectionString() (string, error) { + password, err := d.GetMigrationPassword() + if err != nil { + return "", err + } + + sslMode := d.SSLMode + if sslMode == "" { + sslMode = "require" + } + + // URL-escape the password to handle special characters + escapedPassword := url.QueryEscape(password) + + connString := fmt.Sprintf( + "postgres://%s:%s@%s:%d/%s?sslmode=%s", + d.GetMigrationUser(), + escapedPassword, + d.Host, + d.Port, + d.Database, + sslMode, + ) + + return connString, nil +} + // LoadConfig loads and parses configuration from a YAML file func LoadConfig(opts ...Option) (*Config, error) { loaderCfg := &loaderConfig{} diff --git a/internal/service/db/impl.go b/internal/service/db/impl.go index 3f59f5c..67793ca 100644 --- a/internal/service/db/impl.go +++ b/internal/service/db/impl.go @@ -87,30 +87,34 @@ func (s *dbService) ListServers( ctx context.Context, opts ...service.Option[service.ListServersOptions], ) ([]*upstreamv0.ServerJSON, error) { - options := &service.ListServersOptions{} + options := &service.ListServersOptions{ + Limit: 50, // default limit + } for _, opt := range opts { if err := opt(options); err != nil { return nil, err } } - decoded, err := base64.StdEncoding.DecodeString(options.Cursor) - if err != nil { - return nil, fmt.Errorf("invalid cursor format: %w", err) - } - nextTime, err := time.Parse(time.RFC3339, string(decoded)) - if err != nil { - return nil, fmt.Errorf("invalid cursor format: %w", err) - } - params := sqlc.ListServersParams{ - Next: &nextTime, Size: int64(options.Limit), } if options.RegistryName != nil { params.RegistryName = options.RegistryName } + if options.Cursor != "" { + decoded, err := base64.StdEncoding.DecodeString(options.Cursor) + if err != nil { + return nil, fmt.Errorf("invalid cursor format: %w", err) + } + nextTime, err := time.Parse(time.RFC3339, string(decoded)) + if err != nil { + return nil, fmt.Errorf("invalid cursor format: %w", err) + } + params.Next = &nextTime + } + // Note: this function fetches a list of servers. In case no records are // found, the called function should return an empty slice as it's // customary in Go.