From 5a310903af3f4949351af37abb5ce384b9326e6c Mon Sep 17 00:00:00 2001 From: Chris Stephen Date: Wed, 10 Sep 2025 08:12:22 -0300 Subject: [PATCH 1/4] Add docs on setting server data retention policies --- .../operator/pages/data-retention.adoc | 90 +++++++++++++++++++ .../operator/pages/data-retention.adoc | 90 +++++++++++++++++++ 2 files changed, 180 insertions(+) create mode 100644 docs/server-admin-4.7/modules/operator/pages/data-retention.adoc create mode 100644 docs/server-admin-4.8/modules/operator/pages/data-retention.adoc diff --git a/docs/server-admin-4.7/modules/operator/pages/data-retention.adoc b/docs/server-admin-4.7/modules/operator/pages/data-retention.adoc new file mode 100644 index 0000000000..cdd59f60c2 --- /dev/null +++ b/docs/server-admin-4.7/modules/operator/pages/data-retention.adoc @@ -0,0 +1,90 @@ += Data Retention in Server +:page-platform: Server v4.8, Server Admin +:page-description: Learn how to configure data retention policies for MongoDB, PostgreSQL, and object storage buckets in your CircleCI server installation. +:experimental: + +[#background] +== Background + +You can set up retention policies for both Mongo and Postgres to clean up data older than *n* days. The following sections outline the step-by-step process for implementing these retention policies in your server environment. + +[#setting-postgres-retention] +== Step 1: Setting a Retention Period for Postgres + +. Connect to a REPL session in one of the frontend pods by running the following commands: ++ +[source,bash] +---- +kubectl exec -it -- /bin/bash +---- ++ +Then, connect to the REPL: ++ +[source,bash] +---- +lein repl :connect 6005 +---- + +. Once connected, the current setting can be verified using the following command: ++ +[source,clojure] +---- +(circle.http.api.admin-commands/get-setting :wfc-workflow-deletion-retention-period) +---- + +. The retention period can be set as needed (the example below sets it to 90 days): ++ +[source,clojure] +---- +(circle.http.api.admin-commands/set-setting :wfc-workflow-deletion-retention-period 90) +---- + +. The deletion interval can be verified by running: ++ +[source,clojure] +---- +(circle.http.api.admin-commands/get-setting :wfc-workflow-deletion-interval) +---- ++ +By default, the interval is set to `0`. This value must be updated to a number greater than `0` for WFC deletion to run every *n* seconds. For example, the following command sets it to 1000 seconds: ++ +[source,clojure] +---- +(circle.http.api.admin-commands/set-setting :wfc-workflow-deletion-interval 1000) +---- + +. In instances with significant data volumes, additional `workflows_conductor_event_consumer` replicas may be required to ensure deletion progresses smoothly until it aligns with the configured retention period. + +. The WFC event consumer pod logs can be checked to verify that deletion is progressing without errors. + +. The oldest `created_at` date for a job can be verified to ensure alignment with the retention period using the following command: ++ +[source,bash] +---- +kubectl exec postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD psql -U "postgres" -d "conductor_production" -c "SELECT * FROM public.jobs ORDER BY created_at ASC LIMIT 2;"' +---- + +[#setting-mongodb-retention] +== Step 2: Setting a Retention Period for MongoDB (Action Logs) + +Retention limits for action logs can be configured in the same REPL session using the following commands: + +[source,clojure] +---- +(circle.http.api.admin-commands/set-setting :delete-old-builds.retention-limit-days 180) +(circle.http.api.admin-commands/set-setting :delete-old-action-logs.enabled true) +---- + +[#setting-s3-lifecycle-policies] +== Step 3: Set Up Lifecycle Policies for the S3 Bucket + +After configuring retention limits for your MongoDB and PostgresDB objects, you can also apply object expiry policies to your S3 or GCS buckets. These policies typically expire objects at n+1 days, where n is the retention period set for your databases. + +NOTE: **Object Storage Paths** + +If a retention policy of `n` days is configured for both MongoDB and Postgres data, you can set `n+1` for **all objects** in your S3/GCS buckets to expire at `n+1` days. This ensures alignment between database retention and object storage retention. + +CAUTION: **Audit Logs** + +Be sure to configure exceptions for critical paths, such as `audit-logs/*`, in accordance with your organization's compliance or audit requirements. Objects under these paths should not be expired by default. + +WARNING: **Risk of Irreversible Data Loss** + +Incorrect lifecycle settings may result in data being removed earlier than expected and without recovery options. CircleCI bears no responsibility or liability for any data loss resulting from lifecycle configurations applied to your object storage buckets. diff --git a/docs/server-admin-4.8/modules/operator/pages/data-retention.adoc b/docs/server-admin-4.8/modules/operator/pages/data-retention.adoc new file mode 100644 index 0000000000..cdd59f60c2 --- /dev/null +++ b/docs/server-admin-4.8/modules/operator/pages/data-retention.adoc @@ -0,0 +1,90 @@ += Data Retention in Server +:page-platform: Server v4.8, Server Admin +:page-description: Learn how to configure data retention policies for MongoDB, PostgreSQL, and object storage buckets in your CircleCI server installation. +:experimental: + +[#background] +== Background + +You can set up retention policies for both Mongo and Postgres to clean up data older than *n* days. The following sections outline the step-by-step process for implementing these retention policies in your server environment. + +[#setting-postgres-retention] +== Step 1: Setting a Retention Period for Postgres + +. Connect to a REPL session in one of the frontend pods by running the following commands: ++ +[source,bash] +---- +kubectl exec -it -- /bin/bash +---- ++ +Then, connect to the REPL: ++ +[source,bash] +---- +lein repl :connect 6005 +---- + +. Once connected, the current setting can be verified using the following command: ++ +[source,clojure] +---- +(circle.http.api.admin-commands/get-setting :wfc-workflow-deletion-retention-period) +---- + +. The retention period can be set as needed (the example below sets it to 90 days): ++ +[source,clojure] +---- +(circle.http.api.admin-commands/set-setting :wfc-workflow-deletion-retention-period 90) +---- + +. The deletion interval can be verified by running: ++ +[source,clojure] +---- +(circle.http.api.admin-commands/get-setting :wfc-workflow-deletion-interval) +---- ++ +By default, the interval is set to `0`. This value must be updated to a number greater than `0` for WFC deletion to run every *n* seconds. For example, the following command sets it to 1000 seconds: ++ +[source,clojure] +---- +(circle.http.api.admin-commands/set-setting :wfc-workflow-deletion-interval 1000) +---- + +. In instances with significant data volumes, additional `workflows_conductor_event_consumer` replicas may be required to ensure deletion progresses smoothly until it aligns with the configured retention period. + +. The WFC event consumer pod logs can be checked to verify that deletion is progressing without errors. + +. The oldest `created_at` date for a job can be verified to ensure alignment with the retention period using the following command: ++ +[source,bash] +---- +kubectl exec postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD psql -U "postgres" -d "conductor_production" -c "SELECT * FROM public.jobs ORDER BY created_at ASC LIMIT 2;"' +---- + +[#setting-mongodb-retention] +== Step 2: Setting a Retention Period for MongoDB (Action Logs) + +Retention limits for action logs can be configured in the same REPL session using the following commands: + +[source,clojure] +---- +(circle.http.api.admin-commands/set-setting :delete-old-builds.retention-limit-days 180) +(circle.http.api.admin-commands/set-setting :delete-old-action-logs.enabled true) +---- + +[#setting-s3-lifecycle-policies] +== Step 3: Set Up Lifecycle Policies for the S3 Bucket + +After configuring retention limits for your MongoDB and PostgresDB objects, you can also apply object expiry policies to your S3 or GCS buckets. These policies typically expire objects at n+1 days, where n is the retention period set for your databases. + +NOTE: **Object Storage Paths** + +If a retention policy of `n` days is configured for both MongoDB and Postgres data, you can set `n+1` for **all objects** in your S3/GCS buckets to expire at `n+1` days. This ensures alignment between database retention and object storage retention. + +CAUTION: **Audit Logs** + +Be sure to configure exceptions for critical paths, such as `audit-logs/*`, in accordance with your organization's compliance or audit requirements. Objects under these paths should not be expired by default. + +WARNING: **Risk of Irreversible Data Loss** + +Incorrect lifecycle settings may result in data being removed earlier than expected and without recovery options. CircleCI bears no responsibility or liability for any data loss resulting from lifecycle configurations applied to your object storage buckets. From 5a26cfb1d883c5e7c3bd2d24111fb1e5c64fa492 Mon Sep 17 00:00:00 2001 From: Chris Stephen Date: Wed, 10 Sep 2025 08:35:11 -0300 Subject: [PATCH 2/4] Address linting errors --- .../modules/operator/pages/data-retention.adoc | 12 ++++++------ .../modules/operator/pages/data-retention.adoc | 12 ++++++------ 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/server-admin-4.7/modules/operator/pages/data-retention.adoc b/docs/server-admin-4.7/modules/operator/pages/data-retention.adoc index cdd59f60c2..8887ce8c04 100644 --- a/docs/server-admin-4.7/modules/operator/pages/data-retention.adoc +++ b/docs/server-admin-4.7/modules/operator/pages/data-retention.adoc @@ -1,4 +1,4 @@ -= Data Retention in Server += Data retention in server :page-platform: Server v4.8, Server Admin :page-description: Learn how to configure data retention policies for MongoDB, PostgreSQL, and object storage buckets in your CircleCI server installation. :experimental: @@ -6,10 +6,10 @@ [#background] == Background -You can set up retention policies for both Mongo and Postgres to clean up data older than *n* days. The following sections outline the step-by-step process for implementing these retention policies in your server environment. +You can set up retention policies for both Mongo and PostgreSQL to clean up data older than *n* days. The following sections outline the step-by-step process for implementing these retention policies in your server environment. [#setting-postgres-retention] -== Step 1: Setting a Retention Period for Postgres +== Step 1: Setting a retention period for PostgreSQL . Connect to a REPL session in one of the frontend pods by running the following commands: + @@ -65,7 +65,7 @@ kubectl exec postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD psql -U "postg ---- [#setting-mongodb-retention] -== Step 2: Setting a Retention Period for MongoDB (Action Logs) +== Step 2: Setting a retention period for MongoDB (action logs) Retention limits for action logs can be configured in the same REPL session using the following commands: @@ -76,12 +76,12 @@ Retention limits for action logs can be configured in the same REPL session usin ---- [#setting-s3-lifecycle-policies] -== Step 3: Set Up Lifecycle Policies for the S3 Bucket +== Step 3: Set up lifecycle policies for the S3 Bucket After configuring retention limits for your MongoDB and PostgresDB objects, you can also apply object expiry policies to your S3 or GCS buckets. These policies typically expire objects at n+1 days, where n is the retention period set for your databases. NOTE: **Object Storage Paths** + -If a retention policy of `n` days is configured for both MongoDB and Postgres data, you can set `n+1` for **all objects** in your S3/GCS buckets to expire at `n+1` days. This ensures alignment between database retention and object storage retention. +If a retention policy of `n` days is configured for both MongoDB and PostgreSQL data, you can set `n+1` for **all objects** in your S3/GCS buckets to expire at `n+1` days. This ensures alignment between database retention and object storage retention. CAUTION: **Audit Logs** + Be sure to configure exceptions for critical paths, such as `audit-logs/*`, in accordance with your organization's compliance or audit requirements. Objects under these paths should not be expired by default. diff --git a/docs/server-admin-4.8/modules/operator/pages/data-retention.adoc b/docs/server-admin-4.8/modules/operator/pages/data-retention.adoc index cdd59f60c2..8887ce8c04 100644 --- a/docs/server-admin-4.8/modules/operator/pages/data-retention.adoc +++ b/docs/server-admin-4.8/modules/operator/pages/data-retention.adoc @@ -1,4 +1,4 @@ -= Data Retention in Server += Data retention in server :page-platform: Server v4.8, Server Admin :page-description: Learn how to configure data retention policies for MongoDB, PostgreSQL, and object storage buckets in your CircleCI server installation. :experimental: @@ -6,10 +6,10 @@ [#background] == Background -You can set up retention policies for both Mongo and Postgres to clean up data older than *n* days. The following sections outline the step-by-step process for implementing these retention policies in your server environment. +You can set up retention policies for both Mongo and PostgreSQL to clean up data older than *n* days. The following sections outline the step-by-step process for implementing these retention policies in your server environment. [#setting-postgres-retention] -== Step 1: Setting a Retention Period for Postgres +== Step 1: Setting a retention period for PostgreSQL . Connect to a REPL session in one of the frontend pods by running the following commands: + @@ -65,7 +65,7 @@ kubectl exec postgresql-0 -- sh -c 'PGPASSWORD=$POSTGRES_PASSWORD psql -U "postg ---- [#setting-mongodb-retention] -== Step 2: Setting a Retention Period for MongoDB (Action Logs) +== Step 2: Setting a retention period for MongoDB (action logs) Retention limits for action logs can be configured in the same REPL session using the following commands: @@ -76,12 +76,12 @@ Retention limits for action logs can be configured in the same REPL session usin ---- [#setting-s3-lifecycle-policies] -== Step 3: Set Up Lifecycle Policies for the S3 Bucket +== Step 3: Set up lifecycle policies for the S3 Bucket After configuring retention limits for your MongoDB and PostgresDB objects, you can also apply object expiry policies to your S3 or GCS buckets. These policies typically expire objects at n+1 days, where n is the retention period set for your databases. NOTE: **Object Storage Paths** + -If a retention policy of `n` days is configured for both MongoDB and Postgres data, you can set `n+1` for **all objects** in your S3/GCS buckets to expire at `n+1` days. This ensures alignment between database retention and object storage retention. +If a retention policy of `n` days is configured for both MongoDB and PostgreSQL data, you can set `n+1` for **all objects** in your S3/GCS buckets to expire at `n+1` days. This ensures alignment between database retention and object storage retention. CAUTION: **Audit Logs** + Be sure to configure exceptions for critical paths, such as `audit-logs/*`, in accordance with your organization's compliance or audit requirements. Objects under these paths should not be expired by default. From 34a5c6d8c89bad048cefc138e41496fc809314dc Mon Sep 17 00:00:00 2001 From: Chris Stephen Date: Thu, 11 Sep 2025 10:54:28 -0300 Subject: [PATCH 3/4] Update navigation items with new pages --- docs/server-admin-4.7/modules/ROOT/nav.adoc | 1 + docs/server-admin-4.8/modules/ROOT/nav.adoc | 1 + 2 files changed, 2 insertions(+) diff --git a/docs/server-admin-4.7/modules/ROOT/nav.adoc b/docs/server-admin-4.7/modules/ROOT/nav.adoc index 255f5b336f..bf4b055e4f 100644 --- a/docs/server-admin-4.7/modules/ROOT/nav.adoc +++ b/docs/server-admin-4.7/modules/ROOT/nav.adoc @@ -34,6 +34,7 @@ ** xref:operator:managing-orbs.adoc[Managing orbs] ** xref:operator:manage-virtual-machines-with-machine-provisioner.adoc[Manage virtual machines with machine provisioner] ** xref:operator:configuring-external-services.adoc[Configuring external services] +** xref:operator:data-retention.adoc[Data retention in server] ** xref:operator:expanding-internal-database-volumes.adoc[Expanding internal database volumes] ** xref:operator:managing-load-balancers.adoc[Managing load balancers] ** xref:operator:user-authentication.adoc[User authentication] diff --git a/docs/server-admin-4.8/modules/ROOT/nav.adoc b/docs/server-admin-4.8/modules/ROOT/nav.adoc index 9ccf85e118..25badc9f27 100644 --- a/docs/server-admin-4.8/modules/ROOT/nav.adoc +++ b/docs/server-admin-4.8/modules/ROOT/nav.adoc @@ -34,6 +34,7 @@ ** xref:operator:managing-orbs.adoc[Managing orbs] ** xref:operator:manage-virtual-machines-with-machine-provisioner.adoc[Manage virtual machines with machine provisioner] ** xref:operator:configuring-external-services.adoc[Configuring external services] +** xref:operator:data-retention.adoc[Data retention in server] ** xref:operator:expanding-internal-database-volumes.adoc[Expanding internal database volumes] ** xref:operator:managing-load-balancers.adoc[Managing load balancers] ** xref:operator:user-authentication.adoc[User authentication] From 85d7ed10669f72c1c0c121ac939eeaf1bc943092 Mon Sep 17 00:00:00 2001 From: Chris Stephen Date: Thu, 11 Sep 2025 11:03:02 -0300 Subject: [PATCH 4/4] Bump up warning in data retention doc --- .../modules/operator/pages/data-retention.adoc | 6 +++--- .../modules/operator/pages/data-retention.adoc | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/server-admin-4.7/modules/operator/pages/data-retention.adoc b/docs/server-admin-4.7/modules/operator/pages/data-retention.adoc index 8887ce8c04..0792bac899 100644 --- a/docs/server-admin-4.7/modules/operator/pages/data-retention.adoc +++ b/docs/server-admin-4.7/modules/operator/pages/data-retention.adoc @@ -78,6 +78,9 @@ Retention limits for action logs can be configured in the same REPL session usin [#setting-s3-lifecycle-policies] == Step 3: Set up lifecycle policies for the S3 Bucket +WARNING: **Risk of Irreversible Data Loss** + +Incorrect lifecycle settings may result in data being removed earlier than expected and without recovery options. CircleCI bears no responsibility or liability for any data loss resulting from lifecycle configurations applied to your object storage buckets. + After configuring retention limits for your MongoDB and PostgresDB objects, you can also apply object expiry policies to your S3 or GCS buckets. These policies typically expire objects at n+1 days, where n is the retention period set for your databases. NOTE: **Object Storage Paths** + @@ -85,6 +88,3 @@ If a retention policy of `n` days is configured for both MongoDB and PostgreSQL CAUTION: **Audit Logs** + Be sure to configure exceptions for critical paths, such as `audit-logs/*`, in accordance with your organization's compliance or audit requirements. Objects under these paths should not be expired by default. - -WARNING: **Risk of Irreversible Data Loss** + -Incorrect lifecycle settings may result in data being removed earlier than expected and without recovery options. CircleCI bears no responsibility or liability for any data loss resulting from lifecycle configurations applied to your object storage buckets. diff --git a/docs/server-admin-4.8/modules/operator/pages/data-retention.adoc b/docs/server-admin-4.8/modules/operator/pages/data-retention.adoc index 8887ce8c04..0792bac899 100644 --- a/docs/server-admin-4.8/modules/operator/pages/data-retention.adoc +++ b/docs/server-admin-4.8/modules/operator/pages/data-retention.adoc @@ -78,6 +78,9 @@ Retention limits for action logs can be configured in the same REPL session usin [#setting-s3-lifecycle-policies] == Step 3: Set up lifecycle policies for the S3 Bucket +WARNING: **Risk of Irreversible Data Loss** + +Incorrect lifecycle settings may result in data being removed earlier than expected and without recovery options. CircleCI bears no responsibility or liability for any data loss resulting from lifecycle configurations applied to your object storage buckets. + After configuring retention limits for your MongoDB and PostgresDB objects, you can also apply object expiry policies to your S3 or GCS buckets. These policies typically expire objects at n+1 days, where n is the retention period set for your databases. NOTE: **Object Storage Paths** + @@ -85,6 +88,3 @@ If a retention policy of `n` days is configured for both MongoDB and PostgreSQL CAUTION: **Audit Logs** + Be sure to configure exceptions for critical paths, such as `audit-logs/*`, in accordance with your organization's compliance or audit requirements. Objects under these paths should not be expired by default. - -WARNING: **Risk of Irreversible Data Loss** + -Incorrect lifecycle settings may result in data being removed earlier than expected and without recovery options. CircleCI bears no responsibility or liability for any data loss resulting from lifecycle configurations applied to your object storage buckets.