Deploying a Notify Development Environment

Target audience

This is an overview of the steps needed to configure and deploy a Notify development environment using the deploy pipeline. The intended audience is developers with existing knowledge of the Notify deploy pipeline, detailed here. An understand of AWS' Parameter Store and how to log into AWS via the GDS CLI.

Before you begin

• Pull the latest version of these three repos:

  • notifications-credentials
  • notifications-aws
  • notifications-aws-account-wide-terraform

• You will need an up-to-date version of:

  • GDS-CLI
  • Terraform using tfenv (v1.7.5 and above), instructions for macOS.

• Find a development environment that is not in use by checking the #govuk-notify-infrastructure-team slack channel topic

• Update the #govuk-notify-infrastructure-team Slack channel topic to claim the development environment you are going to use.

• Make sure you are on the VPN (you will need this for concourse and set-pipeline-vars)

• Check you have the $PASSWORD_STORE_DIR environment variable set by running echo $PASSWORD_STORE_DIR, pointing to the local directory containing notifications-credentials. If you do not have this set, add this line to your ~/.zshrc file and load a new terminal window:

• All actions on this page are run against the notify-deploy AWS account.

# ~/.zshrc
export PASSWORD_STORE_DIR=/full/path/to/notifications-credentials

Deploy the latest account-wide terraform to ensure concourse worker permissions are up-to-date

From your terminal navigate to your notifications-aws-account-wide-terraform repo directory. Run the following command where <env> is formatted like "dev-c":

gds aws notify-<env>-admin -- make <env> apply

In most cases it is unlikely to make any changes, but is worth checking before applying.

Run upload-credentials to set environment specific variables needed by the app at runtime

From your terminal navigate to your notifications-aws/scripts/upload-credentials dir. Run the following command, where <env> is formatted like "dev-c":

AWS_PAGER="" gds aws notify-<env>-admin -- ./upload-credentials.sh <env>

There is no output if successful and will upload dev environment-specific credentials needed for the deployment. These update the parameter store of eu-west-1 in the specified account.

Overview of running a dev env

In concourse each development environment has a concourse team and within that team a concourse pipeline called deploy-notify. This makes up the deployment pipeline for that whole environment. The pipeline consists of four tabs:

• deploy
• operator
• pack-bag
• destroy

To deploy an environment you initially run the pack-bag job. Select the deploy job, then trigger new build. Once complete, this triggers the start-deploy job in the deploy tab of the deploy-notify pipeline.

To destroy an environment you can run the destroy job. When you do, don't forget to update the #govuk-notify-infrastructure-team slack channel topic to show the dev environment is free for the next person.

To perform pipeline control tasks you can use jobs in the operator tab, for example if you need to forcibly release the pipeline lock if it has not been relinquished by a failed job (care must be taken to ensure no jobs are running that have legitimately taken the lock).

Controlling inputs to the deploy-notify pipeline

The deploy-notify pipeline is a single pipeline for the whole of Notify. The versions of the different apps which make up a deployment are dictated by the resources that are input to the pack-bag job.

Pinning

You should initially pin the versions of each resource while you are developing a feature, as this ensures that changes to the latest versions of other apps do not interfere with the effects of changes you are working on.

Each resource input to the pack-bag job is sourced from an AWS Parameter Store (SSM) variable, specifically in the notify-deploy AWS account in the London region eu-west-2.

To quickly pin all resources in an environment:

fly --target notify-dev-<env> login --team-name dev-<env> --concourse-url https://concourse.notify.tools/ -b
fly -t notify-dev-<env> login -b  

cd path/to/notifications-aws/concourse/deploy-notify
make <env> pin-bag-resources

Resetting a dev env's parameter store values

1. In your Terminal, navigate to your notifications-aws/concourse/deploy-notify directory.
2. Use the following make target, where <env> is formatted like "dev-c":

AWS_PAGER="" gds aws notify-deploy-devsecrets -- make <env> set-pipeline-vars

There may be some errors relating to missing items in the password store. This is normal, as these are creds used by other builds.

Setting your development branches/images

There is now a script to help you set up a development environment with custom branches/docker images, see the README for full details.

There are 2 resource types you can use as pack bag inputs:

1. Docker resources - the apps, stubs and images that make up Notify.

2. Git resources - Code stored in a git repo; in our case this is only notifications-aws.

To amend Parameter Store values you will need to use use the `devsecrets` role.

gds aws notify-deploy-devsecrets -l

Warning

Resources in concourse appear with dashes"-, but their corresponding resource paths in SSM use underscores _ .e.g notifications-api becomes notifications_api.

Updating Git resources

There are 2 ways:

Via Packbag

1. Make the changes to your branch, push the commit(s) and create a PR.
2. Go to eu-west-2's parameter store and find the parameter /concourse/concourse/pipelines/dev-<a-f>/deploy-notify/notifications_aws_repo_branch. Change its value from main to the name of your branch.
3. Unpin the notifications-aws resource. This resource will be pulled from the branch you set in parameter store.
4. Manually trigger the packbag resource. This will add the update resource to the packbag.
5. This should then cause the rest of the pipeline to auto apply.

In the console this looks like

Manual Pipeline Changes

Caution

This method is quicker to test changes, but has a larger scope for user errors.

1. Make the changes to the resources in notifications-aws you want to.
2. cd path/to/concourse/deploy-notify/deploy-notify.yml.j2
3. Remove the render-pipeline job. This will remove this job from the deployment pipeline when you run it. Necessary as that job would apply the pipeline config which is set in the branch you've set in parameter store (main by default) - not your local pipeline file's config.
4. Manually update the pipeline make dev-<a-f> set-pipeline.
5. Check your changes and apply if happy apply them
6. Trigger a new build of the start-deploy job. You should see the render-pipeline job has disappeared.

Update docker resource

Docker resources refer to the apps, stubs and images that make up Notify.

Background

When working on a branch, changes to docker resources are built into a 'demo image' which allow the changes to be tested before merged into main. Once merged, docker resources are built into 'final images' in the final-images-builds Team in Concourse.

When demo images are built they are pushed toeu-west-1's ECR under the resource name with -demo appended e.g. if you want to test changes to the notifications-api docker resource, then the demo image will be pushed to the notifications-api-demo ECR repo.

They are pushed with a tag in the following format pr-<pr-number>-production-YYYY-MM-DD-HH-MM-SS-<commit-SHA> e.g.
pr-4524-production-2025-07-30-13.57.09-5195542d0250201cfdbc9f5b5a576964aaa8c05a

Tag for final images are in the format YYYY-MM-DD-HH-MM-SS-<commit-SHA>

Updating

Parameters will be in this format and both must be updated

• /concourse/concourse/pipelines/<env-name>/deploy-notify/<docker-resource-name>_repo_name e.g.
  /concourse/concourse/pipelines/dev-d/deploy-notify/notifications_api_repo_name

and

• /concourse/concourse/pipelines/<env-name>/deploy-notify/<docker-resource-name>_repo_tag_regex e.g.
  /concourse/concourse/pipelines/dev-d/deploy-notify/notifications_api_repo_tag_regex

For all images which match the _tag_regex, the latest version is used.

Note

The _repo_name parameter refers to the name the ECR repo, not the Git repo.

1. Go to eu-west-2's parameter store.
2. Find the _repo_name parameter for your docker resource and change the value to append -demo e.g. notifications-api becomes notifications-api-demo.
3. Find the tag_regex parameter for your docker resource and change the value to ^pr-<PR number>-.* where PR number is the number of your PR which triggered the build of the demo image e.g. ^pr-4532-.*
4. Unpin the docker resource and you should see an image which matches the repo_name and tag_regex
5. Manually run the pack bag job which should trigger a deploy of the new image.

You can confirm your new image will be deployed by seeing an update to the task_definition in the deploy job, deploy-infra-terraform task.

In this console this look like

You can ignore this error, the devsecrets account has limited permissions intentionally:

Running your development deployment

Return to concourse and select the team for your dev environment. In the pack-bag tab you can now click into that resource and verify that the latest hash of the commit (if a git-repo resource) or hash of the target image in ECR is correct. It may take a few minutes for concourse to detect changes to its SSM source variables and retrieve the correct resources:

Once you have set up the required inputs to your pack-bag and pinned the versions of every resource that you are not working on, you can start the pack-bag job to begin a deployment of the dev environment.

As the deployment progresses through the jobs of the deploy pipeline the same deployment-bag is fetched by each job, assuring that the same versions of each app are played together. You must allow the jobs to run in sequence before you can assume that the latest deployment is available (e.g. don't force the tests job to run before the deploy job has completed).

If a job fails for transient reasons you can rerun it by clicking into the job and clicking "trigger a new build". Because of the way the deployment-bag is consumed, it will run idempotently as long as the same deployment-bag is still available (if you have modified variables in SSM since deployment you may find old deployment bags are no longer available).

Unpinning

Resources should be unpinned in your env when you have finished.

To quickly pin all resources in an environment:

fly --target notify-dev-<env> login --team-name dev-<env> --concourse-url https://concourse.notify.tools/ -b
fly -t notify-dev-<env> login -b  

cd path/to/notifications-aws/concourse/deploy-notify
make <env> unpin-bag-resources

Playing with your dev-env

You have now set up your dev env 🎉 . You can now access your custom Notify app at:

https://www.<env>.notify-<env>.space

Creating a new service

Each time a dev env is brought build anew (not woken) all services within that env must be created. This can only be done via the console.

Creating accounts and sending notifications from your dev environment

Important note. The instructions in this sub section do not apply to dev-livemail. In dev-livemail SES is in production mode and can send emails to any address.

The Simple Email Service (SES) in the dev environments is rate-limited and running in sandbox mode meaning you can only send email to verified identities. This is an email addresses that you've proven that you own.

Therefore to sign up for a Notify account or send email notifications to yourself via Notify in a dev environment, you must make sure your email address is configured as a verified identity in SES.

SES sandbox identities are not managed via terraform, and should be manually added to each dev environment that you are using this is a one-time operation. To do this, log in to the dev environment

gds aws notify-<env>-admin -l

From the AWS console, choose Amazon Simple Email Service (SES) using the services search menu. Then click Identities under the Configuration menu on the left hand menu. Here you will see a list of identities set up in SES.

Use the "Create Identify" button to add a new email address (not domain) to the list that SES is allowed to send to. SES will send a verification message with a magic link to the email address that you configured to verify you are able to access the address. Once an identity is verified, the dev environment will be able to send emails to that address.

You can also do this with the CLI

gds aws notify-dev-<>-admin -s  # open a subshell
aws ses verify-email-identity --email-address <email address> --region eu-west-1
exit  # exit the subshell

Making changes to the pipeline itself: set-pipeline

The pipeline itself is defined as a jinja2 template deploy-notify.yml.j2 at notifications-aws/concourse/deploy-notify.

The template is built using a script called render-pipeline.sh.

You can make amendments to how the pipeline is constructed by setting Jinja2 variables then running the set-pipeline make target. You can see what Jinja2 variables are available by looking in the render-pipeline.sh script.

To do this firstly login to concourse using the fly CLI tool where <env> is like "dev-c".

fly -t notify-<env> login

Then set a Jinja variable before running the make target from the same directory (notifications-aws/concourse/deploy-notify):

JINJA_BOOL_include_continuity_tests=1 make <env> set-pipeline

Viewing in concourse you will now see the continuity test job appears as part of the pipeline:

Shutting down the Dev environment overnight

"Shutting down" currently means destroying. This can be done by going to the destroy tab of the pipeline and triggering a new "start-destroy" job. Anything you have not deployed to this env via Terraform (i.e. notifications-aws) will be lost. Other jobs need to be finished before running destroy (it shares the pipeline lock).

After the destroy is complete go back to the pack bag and unpin the components (so the environment is ready for the next person to configure).

Don't forget to edit the slack channel topic on #govuk-notify-infrastructure-team to show that the dev environment that has been destroyed is now available for reuse.

Dev environments' overnight behaviour

Sleeping

At 1900 each night the dev envs are put to 'sleep' to cost save. You can manually put your env to sleep by going to the operator tab > sleep-environment > trigger a new build.

Your env will not be usable once sleeping.

Waking

Sleeping envs are not automatically woken the next day. You must manually do this by going to the operator tab > wake-environment > trigger a new build.

Troubleshooting

Waiting to release lock

If you find that the deployment seems to hang at the start-deploy job, specifically on the step acquiring the pipeline-lock, you may need to forcibly unlock the pipeline using the force-unlock-pipeline job under the operator tab (⚠️take care that no jobs are running that have legitimately taken the lock).

Terraform apply fails or Terraform Termination

This could be because of a time out, or something being wrong in the dependency graph. You should initially re run the job with the same inputs.

TargetNotConnected

This is an intermittent bug and the job should be re run with the same inputs.

nothing to commit, working tree clean

This means there are no changes to the packbag

No emails are coming though

Your email address may not be registered with SES. See Creating accounts and sending notifications from your dev environment section.

waiting for ECS Service create: timeout while waiting for state to become 'tfSTABLE'

Documented in detail here, indicates you're trying to deploy an image which no longer exists in ECR.