Run migrations automatically on app startup

README.md

@@ -69,9 +69,10 @@ The server starts on port 8080 by default. Use `--address :PORT` to customize.
 
 **What happens when the server starts:**
 1. Loads configuration from the specified YAML file
-2. Immediately fetches registry data from the configured source
-3. Starts background sync coordinator for automatic updates
-4. Serves MCP Registry API endpoints on the configured address
+2. Runs database migrations automatically (if database is configured)
+3. Immediately fetches registry data from the configured source
+4. Starts background sync coordinator for automatic updates
+5. Serves MCP Registry API endpoints on the configured address
 
 For detailed configuration options and examples, see the [examples/README.md](examples/README.md).
 
@@ -83,7 +84,7 @@ The `thv-registry-api` CLI provides the following commands:
 # Start the API server
 thv-registry-api serve --config config.yaml [--address :8080]
 
-# Run database migrations
+# Manually run database migrations
 thv-registry-api migrate up --config config.yaml [--yes]
 thv-registry-api migrate down --config config.yaml --num-steps N [--yes]
 
@@ -198,8 +199,10 @@ The server optionally supports PostgreSQL database connectivity for storing regi
 |-------|------|----------|---------|-------------|
 | `host` | string | Yes | - | Database server hostname or IP address |
 | `port` | int | Yes | - | Database server port |
-| `user` | string | Yes | - | Database username |
+| `user` | string | Yes | - | Database username for normal operations |
 | `passwordFile` | string | No* | - | Path to file containing the database password |
+| `migrationUser` | string | No | `user` | Database username for running migrations (should have elevated privileges) |
+| `migrationPasswordFile` | string | No | `passwordFile` | Path to file containing the migration user's password |
 | `database` | string | Yes | - | Database name |
 | `sslMode` | string | No | `require` | SSL mode (`disable`, `require`, `verify-ca`, `verify-full`) |
 | `maxOpenConns` | int | No | `25` | Maximum number of open connections to the database |
@@ -210,7 +213,9 @@ The server optionally supports PostgreSQL database connectivity for storing regi
 
 #### Password Security
 
-The server supports secure password management with the following priority order:
+The server supports secure password management with separate credentials for normal operations and migrations.
+
+**Normal Operations Password (for `user`):**
 
 1. **Password File** (Recommended for production):
    - Set `passwordFile` to the path of a file containing only the password
@@ -231,7 +236,29 @@ The server supports secure password management with the following priority order
      thv-registry-api serve --config config.yaml
      ```
 
+**Migration User Password (for `migrationUser`):**
+
+1. **Migration Password File**:
+   - Set `migrationPasswordFile` to the path of a file containing the migration user's password
+   - Falls back to `passwordFile` if not specified
+   - Example:
+     ```yaml
+     database:
+       migrationUser: db_migrator
+       migrationPasswordFile: /secrets/db-migration-password
+     ```
+
+2. **Environment Variable**:
+   - Set `THV_DATABASE_MIGRATION_PASSWORD` environment variable
+   - Falls back to `THV_DATABASE_PASSWORD` if not specified
+   - Example:
+     ```bash
+     export THV_DATABASE_MIGRATION_PASSWORD="migration-user-password"
+     thv-registry-api serve --config config.yaml
+     ```
+
 **Security Best Practices:**
+- Use separate users for migrations (with elevated privileges) and normal operations (read-only or limited)
 - Never commit passwords directly in configuration files
 - Use password files with restricted permissions (e.g., `chmod 400`)
 - In Kubernetes, mount passwords from Secrets
@@ -252,9 +279,52 @@ Tune these values based on your workload:
 
 #### Database Migrations
 
-The server includes built-in database migration commands to manage the database schema.
+The server includes built-in database migration support to manage the database schema.
+
+**Automatic migrations on startup:**
+
+When you start the server with `serve`, database migrations run automatically if database configuration is present in your config file. This ensures your database schema is always up to date.
+
+The only thing necessary is granting the role `toolhive_registry_server` to
+the database user you want, for example
+
+```sql
+BEGIN;
 
-**Running migrations with CLI:**
+DO $$
+  BEGIN
+    IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'toolhive_registry_server') THEN
+      CREATE ROLE toolhive_registry_server;
+    END IF;
+  END
+$$;
+
+DO $$
+  BEGIN
+    IF NOT EXISTS (SELECT FROM pg_user WHERE usename = 'thvr_user') THEN
+      CREATE USER thvr_user WITH PASSWORD 'thvr_user_pass';
+    END IF;
+  END
+$$;
+
+GRANT toolhive_registry_server TO thvr_user;
+
+COMMIT;
+```
+
+To help with that, we plan to add a `prime` subcommand that does just that
+in an idempotent fashion with username and password provided by the user.
+
+Once done, you start the server as follows
+
+```bash
+# Migrations run automatically when database is configured
+thv-registry-api serve --config examples/config-database-dev.yaml
+```
+
+**Manual migration commands (optional):**
+
+You can also run migrations manually using the CLI commands:
 
 ```bash
 # Apply all pending migrations
@@ -285,8 +355,7 @@ task migrate-down CONFIG=examples/config-database-dev.yaml NUM_STEPS=1
 
 1. **Configure database**: Create a config file with database settings (see [examples/config-database-dev.yaml](examples/config-database-dev.yaml))
 2. **Set password**: Either set `THV_DATABASE_PASSWORD` env var or use `passwordFile` in config
-3. **Run migrations**: Use `migrate up` to apply schema changes
-4. **Start server**: Run `serve` command with the same config file
+3. **Start server**: Run `serve` command - migrations will run automatically
 
 **Example: Local development setup**
 
@@ -302,10 +371,7 @@ docker run -d --name postgres \
 # 2. Set password environment variable
 export THV_DATABASE_PASSWORD="devpassword"
 
-# 3. Run migrations
-task migrate-up CONFIG=examples/config-database-dev.yaml
-
-# 4. Start the server
+# 3. Start the server (migrations run automatically)
 thv-registry-api serve --config examples/config-database-dev.yaml
 ```
 
@@ -316,12 +382,7 @@ thv-registry-api serve --config examples/config-database-dev.yaml
 echo "your-secure-password" > /run/secrets/db_password
 chmod 400 /run/secrets/db_password
 
-# 2. Run migrations (using passwordFile from config)
-thv-registry-api migrate up \
-  --config examples/config-database-prod.yaml \
-  --yes
-
-# 3. Start the server
+# 2. Start the server (migrations run automatically)
 thv-registry-api serve --config examples/config-database-prod.yaml
 ```
 
@@ -597,12 +658,12 @@ docker run -v $(pwd)/examples:/config \
 
 ### Docker Compose
 
-A complete Docker Compose setup is provided in the repository root that includes PostgreSQL, automatic migrations, and the API server.
+A complete Docker Compose setup is provided in the repository root that includes PostgreSQL and the API server with automatic migrations.
 
 **Quick start:**
 
 ```bash
-# Start all services (PostgreSQL + migrations + API)
+# Start all services (PostgreSQL + API with automatic migrations)
 docker-compose up
 
 # Run in detached mode
@@ -620,25 +681,27 @@ docker-compose down -v
 
 **Architecture:**
 
-The docker-compose.yaml includes three services:
+The docker-compose.yaml includes two services:
 1. **postgres** - PostgreSQL 18 database server
-2. **migrate** - One-time migration service (runs schema migrations)
-3. **registry-api** - Main API server
+2. **registry-api** - Main API server (runs migrations automatically on startup)
 
 **Service startup flow:**
 ```
-postgres (healthy) → migrate (completes) → registry-api (starts)
+postgres (healthy) → registry-api (runs migrations, then starts)
 ```
 
 **Configuration:**
 
 - Config file: `examples/config-docker.yaml`
 - Sample data: `examples/registry-sample.json`
-- Database password: Set via `THV_DATABASE_PASSWORD` environment variable in docker-compose.yaml
+- Database passwords: Set via environment variables in docker-compose.yaml
+  - `THV_DATABASE_PASSWORD`: Application user password
+  - `THV_DATABASE_MIGRATION_PASSWORD`: Migration user password
 
 The setup demonstrates:
-- Database-backed registry storage
-- Automatic schema migrations on startup
+- Database-backed registry storage with separate users for migrations and operations
+- Automatic schema migrations on startup using elevated privileges
+- Normal operations using limited database privileges (principle of least privilege)
 - File-based data source (for demo purposes)
 - Proper service dependencies and health checks
 
@@ -663,14 +726,23 @@ To use your own registry data:
 
 **Database access:**
 
+The Docker Compose setup creates three database users:
+- `registry`: Superuser (for administration)
+- `db_migrator`: Migration user with schema modification privileges
+- `db_app`: Application user with limited data access privileges
+
 To connect to the PostgreSQL database directly:
 
 ```bash
-# Using psql
+# As superuser (for administration)
 docker exec -it toolhive-registry-postgres psql -U registry -d registry
 
-# Using environment variables from compose
+# As application user
+docker exec -it toolhive-registry-postgres psql -U db_app -d registry
+
+# From host machine
 PGPASSWORD=registry_password psql -h localhost -U registry -d registry
+PGPASSWORD=app_password psql -h localhost -U db_app -d registry
 ```
 
 ## Integration with ToolHive

cmd/thv-registry-api/app/migrate_down.go

@@ -43,10 +43,10 @@ func runMigrateDown(cmd *cobra.Command, _ []string) error {
 		return fmt.Errorf("database configuration is required")
 	}
 
-	// Get connection string
-	connString, err := cfg.Database.GetConnectionString()
+	// Get migration connection string (uses migration user if configured)
+	connString, err := cfg.Database.GetMigrationConnectionString()
 	if err != nil {
-		return fmt.Errorf("failed to get database connection string: %w", err)
+		return fmt.Errorf("failed to get migration connection string: %w", err)
 	}
 
 	// Prompt user for confirmation if not using --yes flag
@@ -103,8 +103,9 @@ func parseMigrateDownFlags(cmd *cobra.Command) (uint, bool, string, error) {
 }
 
 func confirmMigrationDown(numSteps uint, dbCfg *config.DatabaseConfig) bool {
-	logger.Warnf("WARNING: This will revert %d migration(s) from database: %s@%s:%d/%s",
-		numSteps, dbCfg.User, dbCfg.Host, dbCfg.Port, dbCfg.Database)
+	migrationUser := dbCfg.GetMigrationUser()
+	logger.Warnf("WARNING: This will revert %d migration(s) from database: %s@%s:%d/%s (as user: %s)",
+		numSteps, migrationUser, dbCfg.Host, dbCfg.Port, dbCfg.Database, migrationUser)
 	logger.Warnf("WARNING: This operation may result in DATA LOSS")
 	fmt.Print("Are you sure you want to continue? Type 'yes' to proceed: ")
 	var response string

cmd/thv-registry-api/app/migrate_up.go

@@ -46,16 +46,19 @@ func runMigrateUp(cmd *cobra.Command, _ []string) error {
 		return fmt.Errorf("database configuration is required")
 	}
 
-	// Get connection string
-	connString, err := cfg.Database.GetConnectionString()
+	// Get migration connection string (uses migration user if configured)
+	connString, err := cfg.Database.GetMigrationConnectionString()
 	if err != nil {
-		return fmt.Errorf("failed to get database connection string: %w", err)
+		return fmt.Errorf("failed to get migration connection string: %w", err)
 	}
 
+	// Get the migration user for display
+	migrationUser := cfg.Database.GetMigrationUser()
+
 	// Prompt user if not using --yes flag
 	if !yes {
-		logger.Infof("About to apply migrations to database: %s@%s:%d/%s",
-			cfg.Database.User, cfg.Database.Host, cfg.Database.Port, cfg.Database.Database)
+		logger.Infof("About to apply migrations to database: %s@%s:%d/%s (as user: %s)",
+			migrationUser, cfg.Database.Host, cfg.Database.Port, cfg.Database.Database, migrationUser)
 		fmt.Print("Continue? (yes/no): ")
 		var response string
 		if _, err := fmt.Scanln(&response); err != nil {

cmd/thv-registry-api/app/serve.go

@@ -8,12 +8,14 @@ import (
 	"syscall"
 	"time"
 
+	"github.com/jackc/pgx/v5"
 	"github.com/spf13/cobra"
 	"github.com/spf13/viper"
 	"github.com/stacklok/toolhive/pkg/logger"
 	"sigs.k8s.io/controller-runtime/pkg/log"
 	"sigs.k8s.io/controller-runtime/pkg/log/zap"
 
+	"github.com/stacklok/toolhive-registry-server/database"
 	registryapp "github.com/stacklok/toolhive-registry-server/internal/app"
 	"github.com/stacklok/toolhive-registry-server/internal/config"
 )
@@ -28,6 +30,8 @@ The server requires a configuration file (--config) that specifies:
 - Sync policy and filtering rules
 - All other operational settings
 
+If database configuration is present, migrations will run automatically on startup.
+
 See examples/ directory for sample configurations.`,
 	RunE: runServe,
 }
@@ -73,6 +77,14 @@ func runServe(_ *cobra.Command, _ []string) error {
 	logger.Infof("Loaded configuration from %s (registry: %s, %d registries configured)",
 		configPath, cfg.GetRegistryName(), len(cfg.Registries))
 
+	// Run database migrations if database is configured
+	if cfg.Database != nil {
+		logger.Infof("Database configuration found, running migrations...")
+		if err := runMigrations(ctx, cfg); err != nil {
+			return fmt.Errorf("failed to run database migrations: %w", err)
+		}
+	}
+
 	// Build application using the builder pattern
 	address := viper.GetString("address")
 	app, err := registryapp.NewRegistryApp(
@@ -101,3 +113,61 @@ func runServe(_ *cobra.Command, _ []string) error {
 	// Graceful shutdown
 	return app.Stop(defaultGracefulTimeout)
 }
+
+// runMigrations executes database migrations on startup
+func runMigrations(ctx context.Context, cfg *config.Config) error {
+	// Get migration connection string (uses migration user if configured)
+	connString, err := cfg.Database.GetMigrationConnectionString()
+	if err != nil {
+		return fmt.Errorf("failed to get migration connection string: %w", err)
+	}
+
+	// Log which user is running migrations
+	migrationUser := cfg.Database.GetMigrationUser()
+	logger.Infof("Running migrations as user: %s", migrationUser)
+
+	// Connect to database
+	conn, err := pgx.Connect(ctx, connString)
+	if err != nil {
+		return fmt.Errorf("failed to connect to database: %w", err)
+	}
+	defer func() {
+		if closeErr := conn.Close(ctx); closeErr != nil {
+			logger.Errorf("Error closing database connection: %v", closeErr)
+		}
+	}()
+
+	tx, err := conn.BeginTx(
+		ctx,
+		pgx.TxOptions{
+			IsoLevel:   pgx.Serializable,
+			AccessMode: pgx.ReadWrite,
+		},
+	)
+	if err != nil {
+		return fmt.Errorf("failed to begin transaction: %w", err)
+	}
+	defer func() {
+		if err := tx.Rollback(ctx); err != nil {
+			logger.Errorf("Error rolling back transaction: %v", err)
+		}
+	}()
+
+	// Run migrations
+	logger.Infof("Applying database migrations...")
+	if err := database.MigrateUp(ctx, tx.Conn()); err != nil {
+		return fmt.Errorf("failed to apply migrations: %w", err)
+	}
+
+	// Get and log current version
+	version, dirty, err := database.GetVersion(connString)
+	if err != nil {
+		logger.Warnf("Unable to get migration version: %v", err)
+	} else if dirty {
+		logger.Warnf("Database is in a dirty state at version %d", version)
+	} else {
+		logger.Infof("Database migrations completed successfully. Current version: %d", version)
+	}
+
+	return nil
+}