Unable to provision EC2 instances: "Tag key must have a value"

[julian-perge]

Describe the bug

If runner_tags is empty, the amazonec2-tags has a leading comma that causes the API request to AWS to fail with the error message mentioned in the title. See JSON request below for the error.

If runner_tags is empty, output has leading comma since the sed command doesn't remove it in gitlab-runner.tftpl:

"amazonec2-tags=,__PARENT_TAG__",

If runner_tags is not empty, or the leading comma is manually removed by going into the agent EC2 instance and editing /etc/gitlab-runners/config.toml, the error no longer shows up:

# module var.runner_tags
  runner_tags = {
    "testing" = "new runner who dis"
  }

# output from user_data with tags
"amazonec2-tags=testing,new runner who dis,__PARENT_TAG__",

Manual edit that works for us after agents were provisioned and running:

# before
"amazonec2-tags=,gitlab-runner-parent-id,runner-123123",

# after
"amazonec2-tags=gitlab-runner-parent-id,runner-123123",

To Reproduce

Steps to reproduce the behavior:

1. Apply a module with empty runner_tags
2. Wait for amazon to provision the runner
3. See error in CloudTrail logs

Expected behavior

The API requests are successful and the runner is registered, and does not show any errors in CloudTrail or CloudWatch logs about being able to provision the runners.

Additional context

The weird and difficult part about this issue, is that this only showed up 3-4 weeks after we switched to Spot Fleets.

• We switched to Spot Fleets around 2023-07-12, and today was the first time we encountered this specific error.
• We even destroyed and applied fresh modules with no other changes, and this error kept showing up.

Redacted request JSON to AWS

{
    "eventVersion": "1.08",
    "userIdentity": {
        // redacted
    },
    "eventTime": "2023-08-09T17:31:22Z",
    "eventSource": "ec2.amazonaws.com",
    "eventName": "CreateFleet",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "#####",
    "userAgent": "aws-sdk-go/1.44.153 (go1.12.9; linux; amd64)",
    "errorCode": "Client.InvalidTagKey.Malformed",
    "errorMessage": "Tag key must have a value",
    "requestParameters": {
        "CreateFleetRequest": {
            "TargetCapacitySpecification": {
                "DefaultTargetCapacityType": "spot",
                "TotalTargetCapacity": 1
            },
            "Type": "instant",
            "SpotOptions": {
                "AllocationStrategy": "price-capacity-optimized",
                "MaxTotalPrice": 0.5
            },
            "LaunchTemplateConfigs": {
                "LaunchTemplateSpecification": {
                    "LaunchTemplateName": "gitlab-runner-agent-worker-20230809172259696900000029",
                    "Version": "$Latest"
                },
                "Overrides": [
                    {
                        "tag": 1,
                        "SubnetId": "subnet-#####",
                        "InstanceType": "t3.large"
                    },
                    {
                        "tag": 2,
                        "SubnetId": "subnet-#####",
                        "InstanceType": "t3.large"
                    },
                    {
                        "tag": 3,
                        "SubnetId": "subnet-#####",
                        "InstanceType": "t3.large"
                    }
                ],
                "tag": 1
            },
            "TagSpecification": [
                {
                    "ResourceType": "instance",
                    "tag": 1,
                    "Tag": [
                        {
                            "Value": "runner-#####",
                            "tag": 1,
                            "Key": "Name"
                        },
                        {
                            "Value": "gitlab-runner-parent-id",
                            "tag": 2,
                            "Key": ""
                        }
                    ]
                },
                {
                    "ResourceType": "volume",
                    "tag": 2,
                    "Tag": [
                        {
                            "Value": "runner-#####",
                            "tag": 1,
                            "Key": "Name"
                        },
                        {
                            "Value": "gitlab-runner-parent-id",
                            "tag": 2,
                            "Key": ""
                        }
                    ]
                },
                {
                    "ResourceType": "network-interface",
                    "tag": 3,
                    "Tag": [
                        {
                            "Value": "runner-#####",
                            "tag": 1,
                            "Key": "Name"
                        },
                        {
                            "Value": "gitlab-runner-parent-id",
                            "tag": 2,
                            "Key": ""
                        }
                    ]
                },
                {
                    "ResourceType": "fleet",
                    "tag": 4,
                    "Tag": [
                        {
                            "Value": "runner-#####",
                            "tag": 1,
                            "Key": "Name"
                        },
                        {
                            "Value": "gitlab-runner-parent-id",
                            "tag": 2,
                            "Key": ""
                        }
                    ]
                }
            ]
        }
    }
}

[kayman-mk]

Seems a little bit strange as you should always set the var.environment which is also added to the tags. So runner_tags should never be empty.

[julian-perge]

Thank you for the response and quick PR @kayman-mk ! We do indeed set var.environment in our module for clarification.