[ci] fix k8s integration tests flakiness

[pkoutsovasilis]

What does this PR do?

This PR introduces the following changes:

1. Vendor external Kubernetes artifacts for testing:

   • Kube-state-metrics (KSM) Helm subchart is now vendored to eliminate remote fetches during k8s integration tests.
   • Rendered manifests from the kustomize configuration are vendored to remove the runtime dependency on kustomize.
   • The kube-stack Helm chart is also vendored to avoid network pulls.

   To support this, a new mage target called integration:buildKubernetesTestData was added. This target is now invoked as a prerequisite by:

   • integration:testKubernetes
   • integration:testKubernetesMatrix
   • integration:testKubernetesSingle

   This aims to prevent CI failures from GitHub/network issues and addresses #8319.

2. Fix decoding of Beats-style API Keys in K8s tests:

   • PR #7754 introduced a regression that broke Beats-style API key generation required by the Helm and Kustomize k8s integration tests.
   • This PR patches the decoding logic, and highlights the need to validate event ingestion in all integration tests.

3. Fix %CA_TRUSTED% environment variable injection in Kustomize tests:

   • Recent failures revealed malformed ca_trusted_fingerprint values caused by %CA_TRUSTED% placeholders not being overridden.
   • This PR sets CA_TRUSTED to an empty value in the relevant test environments to ensure expected TLS behavior.

   {"log.level":"info","@timestamp":"2025-06-18T02:40:53.068Z","message":"'ca_trusted_fingerprint' set, looking for matching fingerprints","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"log.logger":"tls","log.origin":{"file.line":180,"file.name":"tlscommon/tls_config.go","function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.trustRootCA"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
   {"log.level":"error","@timestamp":"2025-06-18T02:40:53.068Z","message":"Error dialing decode 'ca_trusted_fingerprint': encoding/hex: invalid byte: U+0025 '%'","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"ecs.version":"1.6.0","log.logger":"esclientleg","log.origin":{"file.line":39,"file.name":"transport/logging.go","function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func2"},"service.name":"filebeat","network.transport":"tcp","server.address":"767b2b9229dd4bd098dab11b95e74c64.us-west2.gcp.elastic-cloud.com:443","ecs.version":"1.6.0"}
   

4. Increase memory limits for Elastic Agent in Kustomize-based tests:

   • Multiple OOMKilled errors were observed (example 1, example 2).
   • A bump in memory limits is applied specifically for the Kustomize scenario, where all inputs run on a single DaemonSet pod, potentially spawning more Beat subprocesses.

Why is it important?

• Ensures Kubernetes integration tests are resilient to GitHub outages and network instability by removing external dependencies.
• Fixes broken functionality introduced in previous PRs related to API key handling.
• Prevents TLS misconfiguration caused by unintended environment variable injections.
• Addresses out-of-memory crashes that reduce test reliability and increase CI flakiness.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

None expected. All changes are isolated to the Kubernetes integration test setup and do not affect runtime or user configurations.

How to test this PR locally

mage integration:buildKubernetesTestData

EXTERNAL=true SNAPSHOT=true PACKAGES=docker DOCKER_VARIANTS=basic PLATFORMS=linux/arm64 mage package

INSTANCE_PROVISIONER="kind" TEST_PLATFORMS="kubernetes/arm64/1.33.0/basic" mage integration:TestKubernetes

Related issues

• Closes #8319

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[pchila]

One question and a small nitpick on some test path (both are non-blocking)
LGTM otherwise

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 1196468

Failed CI Steps

• fips:x86_64:sudo-false:fleet

History

• 💚 Build #22718 succeeded b2d39c8
• 💛 Build #22677 was flaky d00a1b8
• 💔 Build #22663 failed 9e27cf0

cc @pkoutsovasilis

[pchila]

After some testing with @pkoutsovasilis , it seems that we can vendor our helm dependencies in uncompressed form: this means that instead of including deploy/helm/elastic-agent/charts/kube-state-metrics-5.30.1.tgz we can include the uncompressed deploy/helm/elastic-agent/charts/kube-state-metrics directory (the same is valid for the testing/integration/k8s/testdata/opentelemetry-kube-stack-0.3.9.tgz helm chart in testdata.

This has the benefit of not having to include a binary file in our git changes and we can more clearly see what changes when chart version gets bumped.

@pkoutsovasilis could you please add a commit with such a change ?

[pchila]

Even if it increases the number of new files, I prefer the exploded charts to having a .tgz committed in git.
It's a shame for the lint GH action not to support diffs over 20k lines for PRs so it cannot filter the linter violations only to the modified files but this should not be a recurring problem.

[swiatekm]

LGTM. It's unfortunate that we need to vendor all this code, and in an ideal world I'd prefer if these artifacts were cached locally on our CI runners instead. But this should work and probably won't be too burdensome to maintain.

Left some questions and nitpicks that shouldn't block merging.

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[github-actions]

@Mergifyio backport 8.17 8.18 8.19 9.0

[mergify]

backport 8.17 8.18 8.19 9.0

✅ Backports have been created

• #8621 [8.17] (backport #8575) [ci] fix k8s integration tests flakiness has been created for branch 8.17 but encountered conflicts
• #8622 [8.18] (backport #8575) [ci] fix k8s integration tests flakiness has been created for branch 8.18 but encountered conflicts
• #8623 [8.19] (backport #8575) [ci] fix k8s integration tests flakiness has been created for branch 8.19 but encountered conflicts
• #8624 [9.0] (backport #8575) [ci] fix k8s integration tests flakiness has been created for branch 9.0 but encountered conflicts