Skip to content

✨ IPv6 support for self-managed clusters #5603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 20 commits into
base: main
Choose a base branch
from
+24,114 −9,359
Open

✨ IPv6 support for self-managed clusters #5603

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
e27ccf3
validations: allow IPv6 configurations for unmanaged clusters
tthvo Jul 22, 2025
7b3846e
ec2: enable primary IPv6 on ENI for EC2 instances
tthvo Jul 22, 2025
aa0221e
ec2: support option HTTPProtocolIPv6 for EC2 IMDS
tthvo Jul 22, 2025
17f3816
routing: ensure routes to eigw are up to date
tthvo Jul 22, 2025
b910741
subnets: configure default subnets to use NAT64/DNS64
tthvo Jul 23, 2025
cbddc1a
eigw: use cluster tag key to list managed egress-only internet gateway
tthvo Jul 23, 2025
b11e1fc
securitygroup: ensure icmpv6 is supported
tthvo Jul 23, 2025
b60ecb1
securitygroup: allow setting allowed IPv6 CIDR for node NodePort serv…
tthvo Jul 28, 2025
ace87c9
securitygroup: allow configuring IPv6 source CIDRs for bastion SSH
tthvo Jul 28, 2025
cc1fed5
crd: add IPv6 of bastion host to cluster status
tthvo Jul 30, 2025
543f7d3
template: manifest templates for IPv6-enabled cluster
tthvo Jul 29, 2025
55bb29f
cni: customized calico manifests for single-stack IPv6
tthvo Jul 29, 2025
f6cdcc9
docs: add documentations for enabling IPv6 in non-eks clusters
tthvo Jul 29, 2025
c11fd51
validations: validate vpc and subnet CIDR
tthvo Aug 5, 2025
b0363f4
docs: update doc for enabling ipv6
tthvo Aug 6, 2025
32551c3
cni: document the requirement for calico ipv6 support
tthvo Aug 8, 2025
01b3c12
subnets: wait till IPv6 CIDR is associated with subnets
tthvo Sep 19, 2025
6b3cded
sg: allow both ipv4 and ipv6 cidrs to API LB if vpc ipv6 block is def…
tthvo Sep 29, 2025
4171752
crd: clarify isIpv6 field on subnet spec
tthvo Jul 29, 2025
3abd5a1
api: add spec field to configure target group ipType
tthvo Oct 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions api/v1beta1/awscluster_conversion.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ func (src *AWSCluster) ConvertTo(dstRaw conversion.Hub) error {
dst.Status.Bastion.HostID = restored.Status.Bastion.HostID
dst.Status.Bastion.CapacityReservationPreference = restored.Status.Bastion.CapacityReservationPreference
dst.Status.Bastion.CPUOptions = restored.Status.Bastion.CPUOptions
dst.Status.Bastion.IPv6Address = restored.Status.Bastion.IPv6Address
}
dst.Spec.Partition = restored.Spec.Partition

Expand Down Expand Up @@ -155,6 +156,7 @@ func (src *AWSCluster) ConvertTo(dstRaw conversion.Hub) error {
func restoreControlPlaneLoadBalancerStatus(restored, dst *infrav1.LoadBalancer) {
dst.ARN = restored.ARN
dst.LoadBalancerType = restored.LoadBalancerType
dst.LoadBalancerIPAddressType = restored.LoadBalancerIPAddressType
dst.ELBAttributes = restored.ELBAttributes
dst.ELBListeners = restored.ELBListeners
dst.Name = restored.Name
Expand Down Expand Up @@ -192,6 +194,7 @@ func restoreControlPlaneLoadBalancer(restored, dst *infrav1.AWSLoadBalancerSpec)
dst.Scheme = restored.Scheme
dst.CrossZoneLoadBalancing = restored.CrossZoneLoadBalancing
dst.Subnets = restored.Subnets
dst.TargetGroupIPType = restored.TargetGroupIPType
}

// ConvertFrom converts the v1beta1 AWSCluster receiver to a v1beta1 AWSCluster.
Expand Down
4 changes: 1 addition & 3 deletions api/v1beta1/network_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,6 @@ type SubnetSpec struct {

// IPv6CidrBlock is the IPv6 CIDR block to be used when the provider creates a managed VPC.
// A subnet can have an IPv4 and an IPv6 address.
// IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
// +optional
IPv6CidrBlock string `json:"ipv6CidrBlock,omitempty"`

Expand All @@ -260,8 +259,7 @@ type SubnetSpec struct {
// +optional
IsPublic bool `json:"isPublic"`

// IsIPv6 defines the subnet as an IPv6 subnet. A subnet is IPv6 when it is associated with a VPC that has IPv6 enabled.
// IPv6 is only supported in managed clusters, this field cannot be set on AWSCluster object.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is possible to have an IPv6-enabled VPC that contains IPv4 subnets. In that case, I would assume that this field would be unset or explicitly set to false i.e even of IPv6 enabled VPCs, IPV4 subnets would be the default? Could you please confirm if that is the case and also update description to reflect what the default would be?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So far, I only consider the "happy" default path that vpc and subnets are dual-stack. Let me add this to my list of questions to confirm. Will get back asap.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: I updated the CRD description to:

IsIPv6 defines the subnet as an IPv6 subnet.
A subnet is IPv6 when it is associated with an IPv6 CIDR.

This should mean that IsIPv6 reflects the state of the subnet (i.e. not depending on the VPC). So, an Ipv4 subnet will have sn.IsIPv6==false as expected even in a dualstack VPC.

Commit: 43d6ec8

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That being said, CAPA will describe the subnet and update the field to reflect the correct state (i.e. IPv6 or not, depending on whether there is an associated IPv6 CIDR)

Describing subnets:

cluster-api-provider-aws/pkg/cloud/services/network/subnets.go

Lines 405 to 411 in f3c2166

spec.CidrBlock = aws.ToString(ec2sn.CidrBlock)
for _, set := range ec2sn.Ipv6CidrBlockAssociationSet {
if set.Ipv6CidrBlockState.State == types.SubnetCidrBlockStateCodeAssociated {
spec.IPv6CidrBlock = aws.ToString(set.Ipv6CidrBlock)
spec.IsIPv6 = true
}
}

Deep-copy subnet state to spec:

cluster-api-provider-aws/pkg/cloud/services/network/subnets.go

Lines 136 to 137 in f3c2166

// Update subnet spec with the existing subnet details
existingSubnet.DeepCopyInto(sub)

// IsIPv6 defines the subnet as an IPv6 subnet. A subnet is IPv6 when it is associated with an IPv6 CIDR.
// +optional
IsIPv6 bool `json:"isIpv6,omitempty"`

Expand Down
4 changes: 3 additions & 1 deletion api/v1beta1/zz_generated.conversion.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

23 changes: 21 additions & 2 deletions api/v1beta2/awscluster_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,8 +152,9 @@ type Bastion struct {

// AllowedCIDRBlocks is a list of CIDR blocks allowed to access the bastion host.
// They are set as ingress rules for the Bastion host's Security Group (defaults to 0.0.0.0/0).
// If the cluster has IPv6 enabled, defaults to ::/0 and 0.0.0.0/0.
// +optional
AllowedCIDRBlocks []string `json:"allowedCIDRBlocks,omitempty"`
AllowedCIDRBlocks CidrBlocks `json:"allowedCIDRBlocks,omitempty"`

// InstanceType will use the specified instance type for the bastion. If not specified,
// Cluster API Provider AWS will use t3.micro for all regions except us-east-1, where t2.micro
Expand Down Expand Up @@ -252,6 +253,15 @@ type AWSLoadBalancerSpec struct {
// PreserveClientIP lets the user control if preservation of client ips must be retained or not.
// If this is enabled 6443 will be opened to 0.0.0.0/0.
PreserveClientIP bool `json:"preserveClientIP,omitempty"`

// TargetGroupIPType sets the IP address type for the target group.
// Valid values are ipv4 and ipv6. If not specified, defaults to ipv4 unless
// the VPC has IPv6 enabled, in which case it defaults to ipv6.
// This applies to the API server target group.
// This field cannot be set if LoadBalancerType is classic or disabled.
// +kubebuilder:validation:Enum=ipv4;ipv6
// +optional
TargetGroupIPType *TargetGroupIPType `json:"targetGroupIPType,omitempty"`
}

// AdditionalListenerSpec defines the desired state of an
Expand All @@ -271,6 +281,14 @@ type AdditionalListenerSpec struct {
// HealthCheck sets the optional custom health check configuration to the API target group.
// +optional
HealthCheck *TargetGroupHealthCheckAdditionalSpec `json:"healthCheck,omitempty"`

// TargetGroupIPType sets the IP address type for the target group.
// Valid values are ipv4 and ipv6. If not specified, defaults to ipv4 unless
// the VPC has IPv6 enabled, in which case it defaults to ipv6.
// This field cannot be set if LoadBalancerType is classic or disabled.
// +kubebuilder:validation:Enum=ipv4;ipv6
// +optional
TargetGroupIPType *TargetGroupIPType `json:"targetGroupIPType,omitempty"`
}

// AWSClusterStatus defines the observed state of AWSCluster.
Expand Down Expand Up @@ -323,7 +341,8 @@ type S3Bucket struct {
// +kubebuilder:printcolumn:name="Ready",type="string",JSONPath=".status.ready",description="Cluster infrastructure is ready for EC2 instances"
// +kubebuilder:printcolumn:name="VPC",type="string",JSONPath=".spec.network.vpc.id",description="AWS VPC the cluster is using"
// +kubebuilder:printcolumn:name="Endpoint",type="string",JSONPath=".spec.controlPlaneEndpoint",description="API Endpoint",priority=1
// +kubebuilder:printcolumn:name="Bastion IP",type="string",JSONPath=".status.bastion.publicIp",description="Bastion IP address for breakglass access"
// +kubebuilder:printcolumn:name="Bastion IP",type="string",JSONPath=".status.bastion.publicIp",description="Bastion IPv4 address for breakglass access"
// +kubebuilder:printcolumn:name="Bastion IPv6",type="string",JSONPath=".status.bastion.ipv6Address",description="Bastion IPv6 address for breakglass access"
// +k8s:defaulter-gen=true

// AWSCluster is the schema for Amazon EC2 based Kubernetes Cluster API.
Expand Down
90 changes: 83 additions & 7 deletions api/v1beta2/awscluster_webhook.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,14 @@ func (r *AWSCluster) validateControlPlaneLoadBalancerUpdate(oldlb, newlb *AWSLoa
)
}
}

// TargetGroupIPType is immutable after creation.
if !cmp.Equal(oldlb.TargetGroupIPType, newlb.TargetGroupIPType) {
allErrs = append(allErrs,
field.Forbidden(field.NewPath("spec", "controlPlaneLoadBalancer", "targetGroupIPType"),
"field is immutable and cannot be changed after target group creation"),
)
}
}

return allErrs
Expand Down Expand Up @@ -301,16 +309,35 @@ func (r *AWSCluster) validateSSHKeyName() field.ErrorList {

func (r *AWSCluster) validateNetwork() field.ErrorList {
var allErrs field.ErrorList
if r.Spec.NetworkSpec.VPC.IsIPv6Enabled() {
allErrs = append(allErrs, field.Invalid(field.NewPath("ipv6"), r.Spec.NetworkSpec.VPC.IPv6, "IPv6 cannot be used with unmanaged clusters at this time."))

vpcSpec := r.Spec.NetworkSpec.VPC
vpcField := field.NewPath("spec", "network", "vpc")
if vpcSpec.CidrBlock != "" {
if _, _, err := net.ParseCIDR(vpcSpec.CidrBlock); err != nil {
allErrs = append(allErrs, field.Invalid(vpcField.Child("cidrBlock"), vpcSpec.CidrBlock, "VPC CIDR block is invalid"))
}
}
for _, subnet := range r.Spec.NetworkSpec.Subnets {
if subnet.IsIPv6 || subnet.IPv6CidrBlock != "" {
allErrs = append(allErrs, field.Invalid(field.NewPath("subnets"), r.Spec.NetworkSpec.Subnets, "IPv6 cannot be used with unmanaged clusters at this time."))
if vpcSpec.IPv6 != nil && vpcSpec.IPv6.CidrBlock != "" {
if _, _, err := net.ParseCIDR(vpcSpec.IPv6.CidrBlock); err != nil {
allErrs = append(allErrs, field.Invalid(vpcField.Child("ipv6", "cidrBlock"), vpcSpec.IPv6.CidrBlock, "VPC IPv6 CIDR block is invalid"))
}
}

subnetField := field.NewPath("spec", "network", "subnets")
for i, subnet := range r.Spec.NetworkSpec.Subnets {
if subnet.ZoneType != nil && subnet.IsEdge() {
if subnet.ParentZoneName == nil {
allErrs = append(allErrs, field.Invalid(field.NewPath("subnets"), r.Spec.NetworkSpec.Subnets, "ParentZoneName must be set when ZoneType is 'local-zone'."))
allErrs = append(allErrs, field.Invalid(subnetField.Index(i).Child("parentZoneName"), subnet.ParentZoneName, "ParentZoneName must be set when ZoneType is 'local-zone'."))
}
}
if subnet.CidrBlock != "" {
if _, _, err := net.ParseCIDR(subnet.CidrBlock); err != nil {
allErrs = append(allErrs, field.Invalid(subnetField.Index(i).Child("cidrBlock"), subnet.CidrBlock, "subnet CIDR block is invalid"))
}
}
if subnet.IPv6CidrBlock != "" {
if _, _, err := net.ParseCIDR(subnet.IPv6CidrBlock); err != nil {
allErrs = append(allErrs, field.Invalid(subnetField.Index(i).Child("ipv6CidrBlock"), subnet.IPv6CidrBlock, "subnet IPv6 CIDR block is invalid"))
}
}
}
Expand Down Expand Up @@ -350,10 +377,15 @@ func (r *AWSCluster) validateNetwork() field.ErrorList {

secondaryCidrBlocks := r.Spec.NetworkSpec.VPC.SecondaryCidrBlocks
secondaryCidrBlocksField := field.NewPath("spec", "network", "vpc", "secondaryCidrBlocks")
for _, cidrBlock := range secondaryCidrBlocks {
for i, cidrBlock := range secondaryCidrBlocks {
if r.Spec.NetworkSpec.VPC.CidrBlock != "" && r.Spec.NetworkSpec.VPC.CidrBlock == cidrBlock.IPv4CidrBlock {
allErrs = append(allErrs, field.Invalid(secondaryCidrBlocksField, secondaryCidrBlocks, fmt.Sprintf("AWSCluster.spec.network.vpc.secondaryCidrBlocks must not contain the primary AWSCluster.spec.network.vpc.cidrBlock %v", r.Spec.NetworkSpec.VPC.CidrBlock)))
}
if cidrBlock.IPv4CidrBlock != "" {
if _, _, err := net.ParseCIDR(cidrBlock.IPv4CidrBlock); err != nil {
allErrs = append(allErrs, field.Invalid(secondaryCidrBlocksField.Index(i).Child("ipv4CidrBlock"), cidrBlock.IPv4CidrBlock, "secondary VPC CIDR block is invalid"))
}
}
}

return allErrs
Expand Down Expand Up @@ -443,6 +475,33 @@ func (r *AWSCluster) validateControlPlaneLBs() (admission.Warnings, field.ErrorL
if r.Spec.ControlPlaneLoadBalancer.DisableHostsRewrite {
allErrs = append(allErrs, field.Invalid(field.NewPath("spec", "controlPlaneLoadBalancer", "disableHostsRewrite"), r.Spec.ControlPlaneLoadBalancer.DisableHostsRewrite, "cannot disable hosts rewrite if the LoadBalancer reconciliation is disabled"))
}

if r.Spec.ControlPlaneLoadBalancer.TargetGroupIPType != nil {
allErrs = append(allErrs, field.Invalid(field.NewPath("spec", "controlPlaneLoadBalancer", "targetGroupIPType"), r.Spec.ControlPlaneLoadBalancer.TargetGroupIPType, "cannot set target group IP type if the LoadBalancer reconciliation is disabled"))
}
}

if r.Spec.ControlPlaneLoadBalancer != nil {
basePath := field.NewPath("spec", "controlPlaneLoadBalancer")
if r.Spec.ControlPlaneLoadBalancer.TargetGroupIPType != nil {
allErrs = append(allErrs, r.validateTargetGroupIPType(basePath.Child("targetGroupIPType"), r.Spec.ControlPlaneLoadBalancer.TargetGroupIPType, r.Spec.ControlPlaneLoadBalancer)...)
}
for i, listener := range r.Spec.ControlPlaneLoadBalancer.AdditionalListeners {
if listener.TargetGroupIPType != nil {
allErrs = append(allErrs, r.validateTargetGroupIPType(basePath.Child("additionalListeners").Index(i).Child("targetGroupIPType"), listener.TargetGroupIPType, r.Spec.ControlPlaneLoadBalancer)...)
}
}
}
if r.Spec.SecondaryControlPlaneLoadBalancer != nil {
basePath := field.NewPath("spec", "secondaryControlPlaneLoadBalancer")
if r.Spec.SecondaryControlPlaneLoadBalancer.TargetGroupIPType != nil {
allErrs = append(allErrs, r.validateTargetGroupIPType(basePath.Child("targetGroupIPType"), r.Spec.SecondaryControlPlaneLoadBalancer.TargetGroupIPType, r.Spec.SecondaryControlPlaneLoadBalancer)...)
}
for i, listener := range r.Spec.SecondaryControlPlaneLoadBalancer.AdditionalListeners {
if listener.TargetGroupIPType != nil {
allErrs = append(allErrs, r.validateTargetGroupIPType(basePath.Child("additionalListeners").Index(i).Child("targetGroupIPType"), listener.TargetGroupIPType, r.Spec.SecondaryControlPlaneLoadBalancer)...)
}
}
}

return allWarnings, allErrs
Expand All @@ -464,3 +523,20 @@ func (r *AWSCluster) validateIngressRules(path *field.Path, rules []IngressRule)
}
return allErrs
}

// validateTargetGroupIPType validates that the target group IP type is compatible
// with the load balancer type and VPC configuration.
func (r *AWSCluster) validateTargetGroupIPType(path *field.Path, targetGroupIPType *TargetGroupIPType, lbSpec *AWSLoadBalancerSpec) field.ErrorList {
var allErrs field.ErrorList

if targetGroupIPType != nil {
if lbSpec.LoadBalancerType == LoadBalancerTypeClassic {
allErrs = append(allErrs, field.Invalid(path, targetGroupIPType, "targetGroupIPType cannot be used with classic load balancer types"))
}
if TargetGroupIPTypeIPv6.Equals(targetGroupIPType) && !r.Spec.NetworkSpec.VPC.IsIPv6Enabled() {
allErrs = append(allErrs, field.Invalid(path, targetGroupIPType, "targetGroupIPType IPv6 requires IPv6 to be enabled on the VPC. Set spec.network.vpc.ipv6 to enable IPv6"))
}
}

return allErrs
}
Loading