[otel] Implement EDOT diagnostics extension

[VihasMakwana]

What does this PR do?

This PR add a new elasticdiagnsotics extension. This new extension will be used in hybrid mode and will perform diagnostics for running otel collector. It will:

• Acts as a registrar and keeps track of common diagnostic hooks.
• Collects profiles using runtime/pprof.
• Collects internal telemetry exposed by the OTeL Collector.
• Implements the extensioncapabilities.ConfigWatcher interface and stores the latest configuration of the running collector.
• Listens for diagnostic requests and provides diagnostic data.

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

None that i can think of. I have to do some final testing on windows to verify this.

Related issues

• Relates [beat_receiver] manager interface beats#44604

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b edot-diagnostics upstream/edot-diagnostics
git merge upstream/main
git push upstream edot-diagnostics

[mergify]

This pull request does not have a backport label. Could you fix it @VihasMakwana? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[VihasMakwana]

@blakerouse @cmacknz @swiatekm This up for review now!

[elastic-sonarqube]

Quality Gate failed

Failed conditions
24.1% Coverage on New Code (required ≥ 40%)

See analysis details on SonarQube

[swiatekm]

LGTM overall, had some more comments but they aren't blocking.

[blakerouse]

This looks good and is ready to merge. I do have one request for an added comment.

[blakerouse]

Thanks for adding the comment. Looks good.

[ebeahan]

v1.33.0:amd64:basic,slim,complete,service,elastic-otel-collector is timing out in CI to obtain a BK agent, but we don't see this as a blocker. Force merging.

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 51808e4

Failed CI Steps

History

• 💔 Build #27798 failed dd87904
• 💔 Build #27753 failed f0d4b4c
• 💔 Build #27697 failed b377c31
• 💔 Build #27696 failed 4d8212a
• 💔 Build #27666 failed 4eca3ba

cc @VihasMakwana