✨ IPv6 support for self-managed clusters #5603

tthvo · 2025-07-30T21:03:14Z

What type of PR is this?
/kind feature

What this PR does / why we need it:

As of today, CAPA supports IPv6 on EKS, but not self-managed clusters. Thus, these changes bring IPv6 support for self-managed clusters, specifically single-stack IPv6 (no dualstack support yet)

Which issue(s) this PR fixes:

Fixes #2420 (part 2 for self-managed cluster, part 1 covers EKS)

Special notes for your reviewer:

API server: By default, API LB restrict access to only over IPv6 (overwrites allowed). You need a host with IPv6 to run CAPI/CAPA.
EC2 instance type: Only nitro-based instance type can support IPv6.
CNI: The CNI plugin needs to support IPV6. I include a sample manifest in test/e2e/data/cni/calico_ipv6.yaml. Calico does not support IPv6 with "IP-in-IP" so we need to use VXLAN.

Checklist:

Release note:

Enable IPv6 support for self-managed kubernetes clusters

linux-foundation-easycla · 2025-07-30T21:03:20Z

The committers listed above are authorized under a signed CLA.

✅ login: tthvo / name: Thuan Vo (17f3816, b910741, 7b3846e, b11e1fc, aa0221e, b60ecb1, e27ccf3, cbddc1a, 55bb29f, cc1fed5, 01b3c12, b0363f4, 32551c3, ace87c9, f6cdcc9, 6b3cded, c11fd51, 543f7d3, 4171752, 3abd5a1)

k8s-ci-robot · 2025-07-30T21:03:21Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign nrb for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2025-07-30T21:03:23Z

Welcome @tthvo!

It looks like this is your first PR to kubernetes-sigs/cluster-api-provider-aws 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/cluster-api-provider-aws has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2025-07-30T21:03:24Z

Hi @tthvo. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tthvo · 2025-07-30T21:23:21Z

/cc @nrb @sadasu @patrickdillon

I am not yet sure what to do with e2e tests or if there are any existing ones for IPv6 clusters...I leave it as an pending TODO.

k8s-ci-robot · 2025-07-30T21:23:24Z

@tthvo: GitHub didn't allow me to request PR reviews from the following users: sadasu, patrickdillon.

Note that only kubernetes-sigs members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @nrb @sadasu @patrickdillon

I am not yet sure what to do with e2e tests or if there are any existing ones for IPv6 clusters...I leave it as an pending TODO.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tthvo · 2025-07-30T21:33:01Z

A quick preview of kubectl get for an IPv6 self-managed cluster can be found here. So far, nodes, pods and services are assigned the expected IPv6 family IPs 😄

damdo · 2025-08-04T09:45:03Z

/ok-to-test

damdo · 2025-08-04T09:46:01Z

/assign @mtulio

Asking you for a review Marco as I know you have been working on this downstream

api/v1beta2/defaults.go

docs/book/src/topics/ipv6-enabled-cluster.md

sadasu · 2025-09-10T20:38:15Z

pkg/cloud/services/network/routetables.go

 	}
 }

+func (s *Service) getNat64PrivateRoute(natGatewayID string) *ec2.CreateRouteInput {


Question: I don't know what makes this a private route. I see getNatGatewayPrivateRoute() also implemented similarly.

Oh, the "private" here means that the route is added to the route table associated with a private subnet.

Now that I think about it more, this route should be added for both private and public subnets (i.e. public subnet only install) to allow IPv6-only workloads in those subnets to reach IPv4-only internet services.

~~Now that I think about it more, this route should be added for both private and public subnets (i.e. public subnet only install)~~

Sorry, I wanted to correct myself 😓 In public-subnet-only install, there are no NATs being created. Thus, we should not add a route to NAT for NAT64.

In order words, this NAT64 is only applicable for private subnets as it is currently.

tthvo · 2025-09-19T04:44:25Z

/retest-required

tthvo · 2025-09-30T00:51:48Z

Status: As of Sep 29, 2025, the only tested scenarios in the PR are:

Managed dualstack VPC/subnets with single-stack IPv6 pod/service networking.
Managed dualstack VPC/subnets with dualstack-IPv6-primary pod/service networking.
BYO dualstack VPC/subnets with single-stack IPv6 pod/service networking.
BYO dualstack VPC/subnets with dualstack-IPv6-primary pod/service networking.

Todo: I am currently exploring options/changes to support dualstack-ipv4-primary and also testing a couple more scenarios (e.g. dualstack public subnets and ipv4-only subnets, edge subnets). Will update once done 👀

(AWSCluster resource)

AWS requires that when registering targets by instance ID for an IPv6 target group, the targets must have an assigned primary IPv6 address. Note: The default subnets managed by CAPA are already set up to assign IPv6 addresses to newly created ENIs.

The httpProtocolIPv6 field enables or disables the IPv6 endpoint of the instance metadata service. The SDK only applies this field if httpEndpoint is enabled. When running on single-stack IPv6, pods only have IPv6, thus requiring an IPv6 endpoint to query IMDS as IPv4 network is unreachable.

In the case where egress-only-internet-gateway is deleted, CAPA reconcilation loop will create a new one. Thus, CAPA needs to modify the routes to point to the new eigw ID.

This allows IPv6-only workloads to reach IPv4-only services. AWS supports this via NAT64/DNS64. More details: https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-nat64-dns64.html

The API for DescribeEgressOnlyInternetGateways does not support attachment.vpc-id filter. Thus, the call will return all available eigw. Consequences: - CAPA incorrectly selects an unintended eigw for use. Leading to route creation failure since the eigw belongs to a different VPC. - CAPA incorrectly destroys all eigw of all VPCs. This is very catastrophic as it can break other workloads. This commit changes the filter to use cluster tag instead. Additional safeguard is also included to check if the eigw is truly attached the VPC.

CAPA handles icmpv6 as a protocol number 58. AWS accepts protocol number when creating rules. However, describing a rule from AWS API returns the protocol name, thus causing CAPA to not recognize it and fail.

…ices For IPv4, we have field NodePortIngressRuleCidrBlocks that specifies the allowed source IPv4 CIDR for node NodePort services on port 30000-32767. This extends that field to also accept IPv6 source CIDRs.

We need an option to configure IPv6 source CIDRs for SSH ingress rule of the bastion host. This extends the field allowedCIDRBlocks to also accepts IPv6 CIDR blocks.

When creating a bastion host for an IPv6 cluster, the instance has both public IPv4 and IPv6. Thus, we need to report them in the cluster status if any. This also adds an additional print column to display that bastion IPv6.

This is a minimal template set to install an IPv6-enabled cluster. Both the controlplane and worker nodes must use nitro-based instance type (with IPv6 support).

This is a set of customized calico CNI manifests to support single-stack IPv6 cluster. Note that VXLAN is used since IP-in-IP currently only supports IPv4. References: - https://docs.tigera.io/calico/latest/networking/ipam/ipv6#ipv6 - https://docs.tigera.io/calico/latest/getting-started/kubernetes/self-managed-onprem/config-options#switching-from-ip-in-ip-to-vxlan - https://docs.tigera.io/calico/latest/networking/configuring/vxlan-ipip

This combines existing docs for IPv6 EKS clusters with non-EKS ones, and also properly register the topic page into the documentation TOC.

Validation for specified VPC and subnet CIDRs is added for early feedback from the webhook. There are already existing checks for bastion and nodePort CIDRs.

The following is added: - [BYO VPC] Mention the required route when enabling DNS64. - [BYO VPC] Mention that CAPA only utilizes the IPv6 aspect of the dual stack VPC.

There is a brief period where the IPv6 CIDR is not yet associated with the subnets. Thus, when CAPA creates the default dualstack subnets, it should wait until the IPv6 CIDR is associated before proceeding. If not, CAPA will misinterprete the subnet as non-IPv6 and proceed its reconcilation. The consequence is that CAPA will skip creating a route to eigw. Route to eigw for destination "::/0" to eigw is required for EC2 instance time sync on start-up.

…ined When AWSCluster.spec.network.vpc.ipv6 is non-nil, most handlers in CAPA treats it as "adding" IPv6 capabilities on top of IPv4 infrastructure. Except security group ingress rules for API LB. This commit aligns the API LB SG handler with the rest of the code base. These rules can be overriden in the AWSCluster LB spec to allow only IPv6 CIDRs if needed.

The field isIpv6 is set to true if and only if the subnet has an associated IPv6 CIDR. This means the VPC is also associated with an IPv6 CIDR.

The field targetGroupIPType is added to the loadbalancer spec to allow configuring ip address type of target group for API load balancers. This field is not applicable to Classic Load Balancers (CLB). This commit also defines a new network status field to determine the ip type of API load balancers.

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. labels Jul 30, 2025

k8s-ci-robot requested review from AndiDog and damdo July 30, 2025 21:03

k8s-ci-robot added needs-priority cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jul 30, 2025

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 30, 2025

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jul 30, 2025

k8s-ci-robot requested a review from nrb July 30, 2025 21:23

tthvo force-pushed the singestack-ipv6 branch from ddacd99 to 13e4379 Compare July 31, 2025 00:37

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 4, 2025

k8s-ci-robot assigned mtulio Aug 4, 2025

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 4, 2025

tthvo force-pushed the singestack-ipv6 branch from 13e4379 to 8816d87 Compare August 5, 2025 18:09

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 5, 2025

tthvo force-pushed the singestack-ipv6 branch from 8816d87 to 5dd2888 Compare August 5, 2025 18:11

nrb reviewed Aug 5, 2025

View reviewed changes

api/v1beta2/defaults.go Outdated Show resolved Hide resolved

nrb reviewed Aug 5, 2025

View reviewed changes

docs/book/src/topics/ipv6-enabled-cluster.md Outdated Show resolved Hide resolved

sadasu reviewed Sep 10, 2025

View reviewed changes

tthvo force-pushed the singestack-ipv6 branch 3 times, most recently from deeb032 to 437f428 Compare September 19, 2025 04:17

tthvo mentioned this pull request Sep 24, 2025

🐛 fix: use cluster tag key to list managed egress-only internet gateway #5684

Open

5 tasks

tthvo force-pushed the singestack-ipv6 branch from 437f428 to 43d6ec8 Compare September 30, 2025 00:27

tthvo force-pushed the singestack-ipv6 branch from 43d6ec8 to c250c5c Compare October 1, 2025 22:48

tthvo added 19 commits October 1, 2025 15:59

validations: allow IPv6 configurations for unmanaged clusters

e27ccf3

(AWSCluster resource)

routing: ensure routes to eigw are up to date

17f3816

In the case where egress-only-internet-gateway is deleted, CAPA reconcilation loop will create a new one. Thus, CAPA needs to modify the routes to point to the new eigw ID.

subnets: configure default subnets to use NAT64/DNS64

b910741

This allows IPv6-only workloads to reach IPv4-only services. AWS supports this via NAT64/DNS64. More details: https://docs.aws.amazon.com/vpc/latest/userguide/nat-gateway-nat64-dns64.html

securitygroup: ensure icmpv6 is supported

b11e1fc

CAPA handles icmpv6 as a protocol number 58. AWS accepts protocol number when creating rules. However, describing a rule from AWS API returns the protocol name, thus causing CAPA to not recognize it and fail.

securitygroup: allow setting allowed IPv6 CIDR for node NodePort serv…

b60ecb1

…ices For IPv4, we have field NodePortIngressRuleCidrBlocks that specifies the allowed source IPv4 CIDR for node NodePort services on port 30000-32767. This extends that field to also accept IPv6 source CIDRs.

securitygroup: allow configuring IPv6 source CIDRs for bastion SSH

ace87c9

We need an option to configure IPv6 source CIDRs for SSH ingress rule of the bastion host. This extends the field allowedCIDRBlocks to also accepts IPv6 CIDR blocks.

crd: add IPv6 of bastion host to cluster status

cc1fed5

When creating a bastion host for an IPv6 cluster, the instance has both public IPv4 and IPv6. Thus, we need to report them in the cluster status if any. This also adds an additional print column to display that bastion IPv6.

template: manifest templates for IPv6-enabled cluster

543f7d3

This is a minimal template set to install an IPv6-enabled cluster. Both the controlplane and worker nodes must use nitro-based instance type (with IPv6 support).

docs: add documentations for enabling IPv6 in non-eks clusters

f6cdcc9

This combines existing docs for IPv6 EKS clusters with non-EKS ones, and also properly register the topic page into the documentation TOC.

validations: validate vpc and subnet CIDR

c11fd51

Validation for specified VPC and subnet CIDRs is added for early feedback from the webhook. There are already existing checks for bastion and nodePort CIDRs.

docs: update doc for enabling ipv6

b0363f4

The following is added: - [BYO VPC] Mention the required route when enabling DNS64. - [BYO VPC] Mention that CAPA only utilizes the IPv6 aspect of the dual stack VPC.

cni: document the requirement for calico ipv6 support

32551c3

crd: clarify isIpv6 field on subnet spec

4171752

The field isIpv6 is set to true if and only if the subnet has an associated IPv6 CIDR. This means the VPC is also associated with an IPv6 CIDR.

tthvo force-pushed the singestack-ipv6 branch from c250c5c to 4171752 Compare October 1, 2025 23:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

✨ IPv6 support for self-managed clusters #5603

✨ IPv6 support for self-managed clusters #5603

tthvo commented Jul 30, 2025

Uh oh!

linux-foundation-easycla bot commented Jul 30, 2025 •

edited

Loading

Uh oh!

k8s-ci-robot commented Jul 30, 2025

Uh oh!

k8s-ci-robot commented Jul 30, 2025

Uh oh!

k8s-ci-robot commented Jul 30, 2025

Uh oh!

tthvo commented Jul 30, 2025

Uh oh!

k8s-ci-robot commented Jul 30, 2025

Uh oh!

tthvo commented Jul 30, 2025

Uh oh!

damdo commented Aug 4, 2025

Uh oh!

damdo commented Aug 4, 2025

Uh oh!

Uh oh!

Uh oh!

sadasu Sep 10, 2025

Uh oh!

tthvo Sep 17, 2025

Uh oh!

tthvo Sep 18, 2025

Uh oh!

tthvo commented Sep 19, 2025

Uh oh!

tthvo commented Sep 30, 2025

Uh oh!

Uh oh!

✨ IPv6 support for self-managed clusters #5603

Are you sure you want to change the base?

✨ IPv6 support for self-managed clusters #5603

Conversation

tthvo commented Jul 30, 2025

Uh oh!

linux-foundation-easycla bot commented Jul 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Jul 30, 2025

Uh oh!

k8s-ci-robot commented Jul 30, 2025

Uh oh!

k8s-ci-robot commented Jul 30, 2025

Uh oh!

tthvo commented Jul 30, 2025

Uh oh!

k8s-ci-robot commented Jul 30, 2025

Uh oh!

tthvo commented Jul 30, 2025

Uh oh!

damdo commented Aug 4, 2025

Uh oh!

damdo commented Aug 4, 2025

Uh oh!

Uh oh!

Uh oh!

sadasu Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

tthvo Sep 17, 2025

Choose a reason for hiding this comment

Uh oh!

tthvo Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

tthvo commented Sep 19, 2025

Uh oh!

tthvo commented Sep 30, 2025

Uh oh!

Uh oh!

linux-foundation-easycla bot commented Jul 30, 2025 •

edited

Loading