Reduce Concourse WG Costs

[beyhan]

This is a tracking issue for all cost reduction activities in the Concourse WG.

As discussed during the TOC meeting on 3rd of June 2025 we agreed on the following activities:

• WG leads should get Ternary access
• Make a report for your Working Group that you should track
• 1 month from now at next meeting, report an idea (implemented or not) for how you will reduce costs
• Contribute to the cost optimization guide Create Automtion Cost Optimization Guide #1185
• Implement cost optimiztion ideas Create Automtion Cost Optimization Guide #1185
• Explore the cost optimization ideas suggested in Ternary

GCP Compute:

ID	Recommendation Type	Resource	Sub Account	Status	Created On	Est. Monthly Savings
7d64144d-a437-4f21-bcdb-5bdce798401d	Zombie Resource	smoke-welcome-lionfish	cf-concourse-production	New	06/01/2025	76
8f3c39b9-dc4b-46b8-b7e7-5b2beda898ea	Unused Static IP Address	cf-concourse-production (us-central1)	cf-concourse-production	New	04/24/2025	72
cb322db4-c22d-4d65-b2bc-177561abaf5e	VM Rightsizing	vm-c81a06dd-48b1-4e0e-45c5-cc7ef14d7fd1	cf-concourse-production	New	04/18/2025	66
64581a54-0d10-4674-b908-3b71ad607e25	VM Rightsizing	cpa-gcp-ops-manager-vm	cf-concourse-production	New	04/18/2025	54
13608587-6dc2-411e-867e-74fbf14f13eb	Zombie Resource	vm-9579c89b-f87a-44c4-705a-a6a3896ff746	cf-concourse-production	New	04/18/2025	47
be3842c4-24b1-48d7-99e0-85b3f878266b	VM Rightsizing	smoke-welcome-lionfish	cf-concourse-production	New	05/30/2025	33
603a100d-db0e-4bef-a2ba-b093f4e26c3f	VM Rightsizing	smoke-polished-shiner	cf-concourse-production	New	04/18/2025	33
03b86c1f-84d9-4a29-9d1b-58ae8e3913fa	VM Rightsizing	vm-ce9a411b-1ea0-4547-62df-8593c6dcb541	cf-concourse-production	New	04/18/2025	25
7830717e-e1c1-4c7b-bbb2-dab2a0c1d2a8	VM Rightsizing	vm-9579c89b-f87a-44c4-705a-a6a3896ff746	cf-concourse-production	New	04/18/2025	24
c96b5cf8-be22-4073-a376-333ded94f3a0	VM Rightsizing	vm-eb41412c-1dc4-4a7c-50ed-d6c4676a899b	cf-concourse-production	New	04/18/2025	22
faa1450c-b046-4ef0-8ebb-c74c038ffaa3	VM Rightsizing	vm-0b854582-6ed2-4957-58b6-9f17f4c3f46e	cf-concourse-production	New	04/18/2025	18
cd405f55-fe66-41af-a48f-a29d79ce1af3	Unused Static IP Address	cf-concourse-production (global)	cf-concourse-production	New	04/24/2025	14
c55c3d80-71fe-4604-9963-5d39334534d8	Unattached Storage	gke-k8s-topgun-f0a7d4a-pvc-08b0456c-be8e-487f-b6ae-adc7d37ae007	cf-concourse-production	New	04/18/2025	12
5b10c172-cb38-42a5-a51a-c0a6d05b1f91	Zombie Resource	vm-2b659945-ce0f-44be-7c55-bf71dc9367a0	cf-concourse-production	New	04/18/2025	11
378af92d-0b71-4bb7-829d-b5747052f1d5	Zombie Resource	vm-2ad9ef4c-d905-48c3-7229-b45a119228d7	cf-concourse-production	New	05/31/2025	11
ec927fbb-46a2-4cd1-90b2-49b0caccdb35	Zombie Resource	vm-f201483c-dc9a-425e-7fc9-61b3bd070610	cf-concourse-production	New	06/01/2025	11

GCP GKE:

ID	Recommendation Type	Resource	Sub Account	Status	Created On	Est. Monthly Savings
6859147c-1636-4b1a-a5c2-7b4bda16a7b2	Nodepool Utilization	cf-concourse-production	k8s-topgun	ci-workers	cf-concourse-production	New
ea52b588-9f11-46b4-af09-6a1fe6c14d5a	Nodepool Utilization	cf-concourse-production	production	ci-workers	cf-concourse-production	New
edd8f636-0f6c-4169-a7b0-d72b2438fcc5	Nodepool Utilization	cf-concourse-production	k8s-topgun	ubuntu	cf-concourse-production	New
a679c01e-a794-488a-859b-aca9b59739a2	Nodepool Utilization	cf-concourse-production	k8s-topgun	cos	cf-concourse-production	New
d5f96f1f-d4a0-4a64-aafa-46b126e82d92	Nodepool Utilization	cf-concourse-production	production	ci-pr-workers	cf-concourse-production	New

GCP Storage:

ID	Recommendation Type	Resource	Sub Account	Status	Created On	Est. Monthly Savings
cacd2a84-f091-44ee-8117-97fc3ff7c373	Object Lifecycle Policy	cryogenics-artifacts	cf-concourse-production	New	04/29/2025	23
21680377-62d9-436d-8a3b-14e42de972e7	Object Lifecycle Policy	topgun-blobstore	cf-concourse-production	New	04/29/2025	22
97ac387b-b7e0-4ed1-9ab8-2ff3109fe5d2	Object Lifecycle Policy	concourse-artifacts	cf-concourse-production	New	04/29/2025	7
23d465d7-b698-449e-b574-ecfd5c438f33	Object Lifecycle Policy	concourse-bosh-blobs	cf-concourse-production	New	04/29/2025	6

[taylorsilva]

Derek and I did the following:

• Deleted all orphaned VM's
• Deleted all unattached storage volumes
• Deleted all stale/unused GCP buckets

We're not at end of month at the time of writing this, but projecting out, it looks like our GCP project cost will be down ~40% this month.

May	June (Projected)
$8,534.68	$5,141.04

The orphaned VM's and storage disks came from CI tests. There's some work that can be down there to ensure these kinds of resources don't get orphaned for long after tests are done running.

[beyhan]

New optimisation recommendations based on PointFive tooling as discussed during the TOC meeting 8th of July:

account_name	name	recommendations	current_cost	potential_savings	resource_name	resource_id	resource_type	region
CF-Concourse-Production	Inactive Cloud SQL Instance	Shut down or delete the instance, creating a snapshot first if necessary.	136.11	136.11	vault-f7a988b3	//sqladmin.googleapis.com/projects/cf-concourse-production/instances/vault-f7a988b3	GCPSQLInstance	us-central1
CF-Concourse-Production	Missing Autoclass on GCS Bucket	Activate Autoclass for this storage bucket.	0.01	0.0050656	bosh-topgun-bbl-state	//storage.googleapis.com/projects/_/buckets/bosh-topgun-bbl-state	GCPStorageBucket	us-central1
CF-Concourse-Production	Missing Autoclass on GCS Bucket	Activate Autoclass for this storage bucket.	0.01	0.0048704	concourse-media-assets	//storage.googleapis.com/projects/_/buckets/concourse-media-assets	GCPStorageBucket	us
CF-Concourse-Production	Suboptimal Compute VM Machine Type	Switch to the more cost-effective machine type based on your usage.	45.08	45.08	vm-9579c89b-f87a-44c4-705a-a6a3896ff746	//compute.googleapis.com/projects/562923869489/zones/us-central1-a/instances/1078594556032842115	GCPComputeInstance	us-central1
CF-Concourse-Production	Inactive GCS Bucket	Delete the bucket if it is no longer in use, ensuring data is backed up beforehand.	0.27	0.271415	artifacts.cf-concourse-production.appspot.com	//storage.googleapis.com/projects/_/buckets/artifacts.cf-concourse-production.appspot.com	GCPStorageBucket	us
CF-Concourse-Production	Suboptimal Compute VM Machine Type	Switch to the more cost-effective machine type based on your usage.	45.48	45.08	vm-eb41412c-1dc4-4a7c-50ed-d6c4676a899b	//compute.googleapis.com/projects/562923869489/zones/us-central1-a/instances/3154970085530138991	GCPComputeInstance	us-central1
CF-Concourse-Production	Underutilized Cloud SQL Instance	Switch to the more cost-effective Cloud SQL instance tier db-custom-1-5120	136.11	83.26	vault-f7a988b3	//sqladmin.googleapis.com/projects/cf-concourse-production/instances/vault-f7a988b3	GCPSQLInstance	us-central1
CF-Concourse-Production	Missing Autoclass on GCS Bucket	Activate Autoclass for this storage bucket.	214.66	171.73	concourse-bosh-blobs	//storage.googleapis.com/projects/_/buckets/concourse-bosh-blobs	GCPStorageBucket	us
CF-Concourse-Production	Missing Autoclass on GCS Bucket	Activate Autoclass for this storage bucket.	0.02	0.0120736	us.artifacts.cf-concourse-production.appspot.com	//storage.googleapis.com/projects/_/buckets/us.artifacts.cf-concourse-production.appspot.com	GCPStorageBucket	us
CF-Concourse-Production	Missing Autoclass on GCS Bucket	Activate Autoclass for this storage bucket.	1.01	0.807356	concourse-charts-enterprise	//storage.googleapis.com/projects/_/buckets/concourse-charts-enterprise	GCPStorageBucket	us
CF-Concourse-Production	Suboptimal Compute VM Machine Type	Switch to the more cost-effective machine type based on your usage.	380.94	133.12	windows-worker-ci	//compute.googleapis.com/projects/562923869489/zones/us-central1-a/instances/4308519407001959349	GCPComputeInstance	us-central1
CF-Concourse-Production	Missing Autoclass on GCS Bucket	Activate Autoclass for this storage bucket.	220.2	176.16	concourse-artifacts	//storage.googleapis.com/projects/_/buckets/concourse-artifacts	GCPStorageBucket	us-central1
CF-Concourse-Production	Underutilized Cloud SQL Instance	Switch to the more cost-effective Cloud SQL instance tier db-custom-1-5120	159.36	83.26	ci-23286c2d	//sqladmin.googleapis.com/projects/cf-concourse-production/instances/ci-23286c2d	GCPSQLInstance	us-central1
CF-Concourse-Production	Suboptimal Compute VM Machine Type	Switch to the more cost-effective machine type based on your usage.	45.08	45.08	vm-ce9a411b-1ea0-4547-62df-8593c6dcb541	//compute.googleapis.com/projects/562923869489/zones/us-central1-a/instances/6140083342500034975	GCPComputeInstance	us-central1
CF-Concourse-Production	Missing Autoclass on GCS Bucket	Activate Autoclass for this storage bucket.	12.8	10.24	concourse-charts	//storage.googleapis.com/projects/_/buckets/concourse-charts	GCPStorageBucket	us
CF-Concourse-Production	Suboptimal Compute VM Machine Type	Switch to the more cost-effective machine type based on your usage.	45.08	45.08	vm-0b854582-6ed2-4957-58b6-9f17f4c3f46e	//compute.googleapis.com/projects/562923869489/zones/us-central1-a/instances/5939921854882557783	GCPComputeInstance	us-central1
CF-Concourse-Production	Inactive GCS Bucket	Delete the bucket if it is no longer in use, ensuring data is backed up beforehand.	0.01	0.009009	concourse-customer-patches	//storage.googleapis.com/projects/_/buckets/concourse-customer-patches	GCPStorageBucket	us
CF-Concourse-Production	Missing Autoclass on GCS Bucket	Activate Autoclass for this storage bucket.	0.01	0.0072072	concourse-customer-patches	//storage.googleapis.com/projects/_/buckets/concourse-customer-patches	GCPStorageBucket	us
CF-Concourse-Production	Missing Autoclass on GCS Bucket	Activate Autoclass for this storage bucket.	24.02	19.22	topgun-blobstore	//storage.googleapis.com/projects/_/buckets/topgun-blobstore	GCPStorageBucket	us
CF-Concourse-Production	Missing Autoclass on GCS Bucket	Activate Autoclass for this storage bucket.	28.53	22.83	cryogenics-artifacts	//storage.googleapis.com/projects/_/buckets/cryogenics-artifacts	GCPStorageBucket	us
CF-Concourse-Production	Inactive GCS Bucket	Delete the bucket if it is no longer in use, ensuring data is backed up beforehand.	0.02	0.015092	us.artifacts.cf-concourse-production.appspot.com	//storage.googleapis.com/projects/_/buckets/us.artifacts.cf-concourse-production.appspot.com	GCPStorageBucket	us
CF-Concourse-Production	Suboptimal Compute VM Machine Type	Switch to the more cost-effective machine type based on your usage.	11.42	11.27	vm-34f9d486-15ec-44a3-6377-28e761a3d80d	//compute.googleapis.com/projects/562923869489/zones/us-central1-f/instances/4292495377926247901	GCPComputeInstance	us-central1
CF-Concourse-Production	Inactive GCS Bucket	Delete the bucket if it is no longer in use, ensuring data is backed up beforehand.	0.01	0.006332	bosh-topgun-bbl-state	//storage.googleapis.com/projects/_/buckets/bosh-topgun-bbl-state	GCPStorageBucket	us-central1
CF-Concourse-Production	Inactive GCS Bucket	Delete the bucket if it is no longer in use, ensuring data is backed up beforehand.	1.49	1.49	concourse-branch-artifacts	//storage.googleapis.com/projects/_/buckets/concourse-branch-artifacts	GCPStorageBucket	us-central1
CF-Concourse-Production	Inactive GCS Bucket	Delete the bucket if it is no longer in use, ensuring data is backed up beforehand.	28.53	28.53	cryogenics-artifacts	//storage.googleapis.com/projects/_/buckets/cryogenics-artifacts	GCPStorageBucket	us
CF-Concourse-Production	Missing Autoclass on GCS Bucket	Activate Autoclass for this storage bucket.	1.49	1.19	concourse-branch-artifacts	//storage.googleapis.com/projects/_/buckets/concourse-branch-artifacts	GCPStorageBucket	us-central1
CF-Concourse-Production	Suboptimal Compute VM Machine Type	Switch to the more cost-effective machine type based on your usage.	11.29	11.27	vm-2b659945-ce0f-44be-7c55-bf71dc9367a0	//compute.googleapis.com/projects/562923869489/zones/us-central1-a/instances/3272167820231462550	GCPComputeInstance	us-central1
CF-Concourse-Production	Inactive GCS Bucket	Delete the bucket if it is no longer in use, ensuring data is backed up beforehand.	1.09	1.09	concourse-prod-backup	//storage.googleapis.com/projects/_/buckets/concourse-prod-backup	GCPStorageBucket	us
CF-Concourse-Production	Inactive GCS Bucket	Delete the bucket if it is no longer in use, ensuring data is backed up beforehand.	1.01	1.01	concourse-charts-enterprise	//storage.googleapis.com/projects/_/buckets/concourse-charts-enterprise	GCPStorageBucket	us
CF-Concourse-Production	Inactive GCS Bucket	Delete the bucket if it is no longer in use, ensuring data is backed up beforehand.	0.02	0.020764	mp-concourse-data	//storage.googleapis.com/projects/_/buckets/mp-concourse-data	GCPStorageBucket	us
CF-Concourse-Production	Missing Autoclass on GCS Bucket	Activate Autoclass for this storage bucket.	0.27	0.217132	artifacts.cf-concourse-production.appspot.com	//storage.googleapis.com/projects/_/buckets/artifacts.cf-concourse-production.appspot.com	GCPStorageBucket	us
CF-Concourse-Production	Missing Autoclass on GCS Bucket	Activate Autoclass for this storage bucket.	0.02	0.0166112	mp-concourse-data	//storage.googleapis.com/projects/_/buckets/mp-concourse-data	GCPStorageBucket	us

[taylorsilva]

I've stopped the DB instance vault-f7a988b3. I thought that was what our main CI instance's Vault was using as the backend. Dug into its config and found out we're actually using GCS as the Vault storage backend. I'm not sure what is using this db then. It might have been leftover when we migrated Vault from the DB to the GCS bucket? It does show 3-4 active connections to the DB.

Turning it off and we'll see if anything breaks after a week or two.

[taylorsilva]

I made a change to some of our bosh CI jobs. They previously would create a deployment and then leave the deployment running forever. Later builds of the job would recreate the deployment, but a vm was always left running.

After concourse/ci@cc2d4dc, the job now cleans up the deployment if the job succeeds. This should slightly reduce costs again for us.