postgres-startup init fail

[AlexisDesbonnets]

Overview

When I create a postgres cluster, the instance crash on initialization. The bug happened only on kubernetes 1.22.

Environment

• Platform: Kubernetes on ovh managed kubernetes service
• Platform Version: 1.22.2
• PGO Image Tag: ubi8-5.0.4-0
• Postgres Version: 13.5
• Storage: csi-cinder-classic

Steps to Reproduce

REPRO

1. Install postgres-operator example with customize
2. Install the postgres hippo cluster defined in the example

EXPECTED

1. The postgres cluster starts correctely.

ACTUAL

The postgres-startup container failed to init the cluster

Initializing ...
::postgres-operator: uid::26
::postgres-operator: gid::26
::postgres-operator: postgres path::/usr/pgsql-13/bin/postgres
::postgres-operator: postgres version::postgres (PostgreSQL) 13.5
::postgres-operator: config directory::/pgdata/pg13
::postgres-operator: data directory::/pgdata/pg13
install: cannot change permissions of ‘/pgdata/pg13’: No such file or directory

Events:
  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Warning  FailedScheduling        40s                default-scheduler        0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.
  Normal   Scheduled               38s                default-scheduler        Successfully assigned postgres-operator/hippo-instance1-mzs4-0 to node-b4422cc5-9dc5-4116-a4d5-a194f6171309
  Normal   SuccessfulAttachVolume  35s                attachdetach-controller  AttachVolume.Attach succeeded for volume "ovh-managed-kubernetes-uo1sfr-pvc-729b89ad-c0dd-48f9-847b-36e36d8cc8e6"
  Normal   Pulled                  30s                kubelet                  Successfully pulled image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:centos8-13.5-0" in 1.260460237s
  Normal   Pulled                  28s                kubelet                  Successfully pulled image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:centos8-13.5-0" in 1.232337581s
  Normal   Pulling                 15s (x3 over 32s)  kubelet                  Pulling image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:centos8-13.5-0"
  Normal   Created                 14s (x3 over 30s)  kubelet                  Created container postgres-startup
  Normal   Started                 14s (x3 over 30s)  kubelet                  Started container postgres-startup
  Normal   Pulled                  14s                kubelet                  Successfully pulled image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres:centos8-13.5-0" in 1.226427728s
  Warning  BackOff                 13s (x3 over 27s)  kubelet                  Back-off restarting failed container

[jkatz]

Does your storage class require you to set up any supplemental groups, e.g.:

https://access.crunchydata.com/documentation/postgres-operator/v5/references/crd/#postgresclusterspec

There are some more notes about supplemental groups in the 5.0.3 release notes:

https://access.crunchydata.com/documentation/postgres-operator/v5/releases/5.0.3/

[Venryx]

Just wanted to mention that I'm hitting the same issue, on OVH Managed Kubernetes as well.

Environment

Platform: Kubernetes on ovh managed kubernetes service
Platform Version: 1.22.2
PGO Image Tag: ubi8-5.0.1-0
Postgres Version: 13.3
Storage: csi-cinder-high-speed

Logs (from postgres-startup container)

[postgres-startup] Initializing ...
[postgres-startup] ::postgres-operator: uid::26
[postgres-startup] ::postgres-operator: gid::26 65534
[postgres-startup] ::postgres-operator: postgres path::/usr/pgsql-13/bin/postgres
[postgres-startup] ::postgres-operator: postgres version::postgres (PostgreSQL) 13.3
[postgres-startup] ::postgres-operator: config directory::/pgdata/pg13
[postgres-startup] ::postgres-operator: data directory::/pgdata/pg13
[postgres-startup] install: cannot change permissions of ‘/pgdata/pg13’: No such file or directory

I have been trying to resolve this issue for about an hour now, without much progress. (I am not very experienced with the details of the postgres-operator, or even kubernetes in general, but am trying to learn)

Note that I just (earlier this night) updated my OVH Kubernetes cluster from 1.20 to 1.22, so as the OP said, this is likely the cause of the issue. (though I can't rule out that there are other contributing causes/changes, as I started the reset+update to try to get around a persistent-volume not-mounting issue I was having).

[Venryx]

Okay, I reset + changed-version my cluster to 1.21 (1.21.5-0), and the error above is no longer being hit:

[postgres-startup] Initializing ...
[postgres-startup] ::postgres-operator: uid::26
[postgres-startup] ::postgres-operator: gid::26 65534
[postgres-startup] ::postgres-operator: postgres path::/usr/pgsql-13/bin/postgres
[postgres-startup] ::postgres-operator: postgres version::postgres (PostgreSQL) 13.3
[postgres-startup] ::postgres-operator: config directory::/pgdata/pg13
[postgres-startup] ::postgres-operator: data directory::/pgdata/pg13
[K8s EVENT: Pod debate-map-repo-host-0 (ns: postgres-operator)] Successfully pulled image "registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:centos8-2.33-1" in 37.671615103s
[K8s EVENT: Pod debate-map-repo-host-0 (ns: postgres-operator)] Pulling image "registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:centos8-2.33-1"
[nss-wrapper-init] nss_wrapper: user exists
[nss-wrapper-init] nss_wrapper: group exists
[nss-wrapper-init] nss_wrapper: environment configured
[K8s EVENT: Pod debate-map-instance1-hfj5-0 (ns: postgres-operator)] Successfully pulled image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha:centos8-13.3-1" in 20.861051117s
[K8s EVENT: Pod debate-map-repo-host-0 (ns: postgres-operator)] Successfully pulled image "registry.developers.crunchydata.com/crunchydata/crunchy-pgbackrest:centos8-2.33-1" in 7.632752328s
[K8s EVENT: Pod debate-map-instance1-hfj5-0 (ns: postgres-operator)] Pulling image "registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha:centos8-13.3-1"
Server listening on 0.0.0.0 port 2022.
Server listening on :: port 2022.
[...]

So something about version 1.22 of Kubernetes is indeed causing the issue. (EDIT: Apparently it's more complicated than just this version change, as v1.21 also has issues [on a different cloud provider] for some of the commenters below.)

I spent a good 15 minutes searching through the v1.22+ changelog here, but did not notice anything likely to be the cause of the issue. Perhaps there is something that OVH changed internally about their provisioning for v1.22 nodes, beyond the changes to Kubernetes itself?

Anyway, if others are having the same issue (and are not using OVH Managed Kubernetes), please comment here so we can narrow down the source of the problem.

[jkatz]

In the Operator log on startup (the pgo-... Pod), could you see if there are any lines that indicate what spec.openshift is set to?

[jkatz]

I'm fairly confident this is similar to an issue that prompted #2897 and should be fixed in that.

The interim solution is to set the following in your PostgresCluster spec:

spec:
  openshift: false

[Leedwing]

I just wanted to mention that I'm hitting the same issue, on Linode Managed Kubernetes service (LKE)as well. On both Kubernetes version 1.21.1 and 1.22.2

Environment
Platform: Kubernetes on Linode managed kubernetes service (LKE)
Platform Version: 1.21.1 and 1.22.2
PGO Image Tag: ubi8-5.0.3-0
Postgres Version: 13.4
Storage: csi-cinder-high-speed

Logs (from postgres-startup container)

Initializing ...
::postgres-operator: uid::26
::postgres-operator: gid::26
::postgres-operator: postgres path::/usr/pgsql-13/bin/postgres
::postgres-operator: postgres version::postgres (PostgreSQL) 13.4
::postgres-operator: config directory::/pgdata/pg13
::postgres-operator: data directory::/pgdata/pg13
install: cannot change permissions of ‘/pgdata/pg13’: No such file or directory

I tried with the suggestions of @jkatz configuring PostgresCluster like this

spec:
  openshift: false

and then

spec:
  supplementalGroups:
    - 65534

and then with both

spec:
  openshift: false
  supplementalGroups:
    - 65534

But nothing worked. Any idea?

[jisaitua]

I'm in the same situation as @Leedwing here: LKE (I tested 1.21 and 1.22) using both specifications (openshit and/or supplementalGroups), but neither worked.

[Leedwing]

Update

I deployed the postgres-operator-examples on DigitalOcean Kubernetes Service v1.20.11 and v1.21.1

Environment
Platform: DigitalOcean Kubernetes service
Platform Version: 1.20.11 and 1.21.5
PGO Image Tag: ubi8-5.0.3-0
Postgres Version: 13.4
Storage: csi-cinder-high-speed

AND EVERYTHING WORKED FINE!

I would say that on LKE, the issue might be caused by some internal behavior and probably related to permissions against block storage. Because also the monitoring component shipped with postgres-operator-examples that uses block storage cannot start on LKE due to the same kind of error: permission denied

• Grafana

GF_PATHS_DATA='/data/grafana/data' is not writable.
You may have issues with file permissions, more information here: http://docs.grafana.org/installation/docker/#migrate-to-v51-or-later
...
...
service init failed: failed to connect to database: mkdir /data/grafana: permission denied

• Prometheus

level=error ts=2021-12-27T19:36:47.147Z caller=query_logger.go:87 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
panic: Unable to create mmap-ed active query log

• Alertmanager

level=error ts=2021-12-27T19:41:09.010Z caller=nflog.go:365 component=nflog msg="Running maintenance failed" err="open /alertmanager/nflog.73fec05415888652: permission denied"
level=info ts=2021-12-27T19:41:09.011Z caller=silence.go:389 component=silences msg="Running maintenance failed" err="open /alertmanager/silences.195c39cdee14e24: permission denied

[jisaitua]

I installed it a few weeks ago on LKE (1.21) and it was working (and I was surprised how easy it was), so I guess the problem started due to some changes in Postgres Operator.

[Leedwing]

@jisaitua I also got it working few weeks ago on LKE (1.21). It was also working on kubernetes 1.21.5 shipped with docker desktop locally and is still working.
I've checked the change log https://developers-linode.netlify.app/changelog/cloud-manager/ of Linode and noticed that they have done some update related to block storage (i guess this might be the cause the permission issue).
Another point why i guess the change might be at Linode side is that, i got CNAME Records created with the kubernetes External DNS instead of A Records as it was few weeks ago. And with the same scripts everything is working as expected on DigitalOcean.

PS: i'am using the same postgres-operator-examples as few weeks ago. I didn't pull the new changes.

[jisaitua]

PS: i'am using the same postgres-operator-examples as few weeks ago. I didn't pull the new changes.

@Leedwing Good point. If that's the case, then the problem looks like a Linode problem. Maybe @jkatz can help to find some way to solve it.