[observability] most basic OpenTelemetry integration into MCK

[nammn]

Summary

This pull request introduces OpenTelemetry tracing support to the MongoDB Kubernetes Operator and its related components. Key changes include the integration of OpenTelemetry libraries, the addition of tracing configuration, and updates to ensure trace propagation across the application. These changes enhance observability and debugging capabilities.

In our CI suite this means we will have the following kind of traces:

trace_id: abc123

                         ┌────────────────────┐
                         │     Evergreen      │
                         │  span_id: ROOT     │
                         │  parent_id: none   │
                         └─────────┬──────────┘
                                   │
        ┌──────────────────────────┼─────────────────────────┐
        ▼                          ▼                         ▼
┌──────────────┐         ┌────────────────┐         ┌────────────────────┐
│   E2E Test   │         │   Operator     │         │     (Other…)       │
│ span_id: A1  │         │ span_id: B1    │         │                    │
│ parent: ROOT │         │ parent: ROOT   │         │                    │
└──────┬───────┘         └──────┬─────────┘         └────────────────────┘
       │                        │
       ▼                        ▼
┌──────────────┐         ┌────────────────────┐
│ E2E Function │         │   Reconcile Loop   │
│ span_id: A2  │         │   span_id: B2      │
│ parent: A1   │         │   parent: B1       │
└──────────────┘         └────────────────────┘

OpenTelemetry Integration:

• Tracing in main.go:
  • Added OpenTelemetry setup in the main function, including trace and span ID extraction from environment variables and the creation of a root span for the operator. Tracing context is propagated across controllers and shutdown processes are handled gracefully.
• Telemetry in pkg/telemetry/client.go: <--- this is good to know if we happen to make a change and happen to send to prod atlas
  • Added a span to the SendEventWithRetry function to capture telemetry events and include the Atlas base URL as a span attribute.

Helm Chart Updates:

• Operator configuration:
  • Added OpenTelemetry-specific environment variables (OTEL_TRACE_ID, OTEL_PARENT_ID, OTEL_EXPORTER_OTLP_ENDPOINT) to the operator's deployment template. ([helm_chart/templates/operator.yamlR83-R90](https://github.com/mongodb/mongodb-kubernetes/pull/93/files#diff-5d2e377a6806023ca9eff60be4d7e5cd879803de2bd3800b630f479f8728f322R83-R90))
  • Introduced OpenTelemetry configuration options (enabled, traceID, parentID, collectorEndpoint) in the Helm chart's values.yaml.

Dependency Updates:

• Go module dependencies:
  • Added OpenTelemetry-related libraries (otel, otel/sdk, otel/trace, etc.) to go.mod.

Proof of Work

• e.g. patch

• generated traces in our ci: Link
  

Checklist

• Have you linked a jira ticket and/or is the ticket in the title?
• Have you checked whether your jira ticket required DOCSP changes?
• Have you checked for release_note changes?

Reminder (Please remove this when merging)

• Please try to Approve or Reject Changes the PR, keep PRs in review as short as possible
• Our Short Guide for PRs: Link
• Remember the following Communication Standards - use comment prefixes for clarity:
  • blocking: Must be addressed before approval.
  • follow-up: Can be addressed in a later PR or ticket.
  • q: Clarifying question.
  • nit: Non-blocking suggestions.
  • note: Side-note, non-actionable. Example: Praise
  • --> no prefix is considered a question

[nammn]

updated the service-name for better querying

[nammn]

we should remove the operator to test teardown at the end of the test-run + it enables us to ensure traces are exported in time

[Copilot]

Pull Request Overview

This PR introduces basic OpenTelemetry tracing support into the MongoDB Kubernetes Operator to improve observability and debugging. Key changes include the integration of OpenTelemetry libraries and configuration in main.go and pkg/telemetry, updates to Helm chart values and templates for trace propagation, and accompanying changes in deployment and test scripts to ensure correct trace information is passed.

Reviewed Changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
scripts/funcs/operator_deployment	Adds OpenTelemetry environment variable handling to operator deployment configuration.
scripts/funcs/kubernetes	Updates operator naming and adjusts deployment deletion commands.
scripts/evergreen/e2e/e2e.sh	Introduces cluster diagnostics and cleanup functions for enhanced troubleshooting.
scripts/evergreen/deployments/test-app/templates/mongodb-enterprise-tests.yaml	Modifies environment variables to set the correct service name in traces.
pkg/telemetry/*	Implements tracing setup, span creation, and propagation via new OpenTelemetry integrations.
pipeline.py	Updates comment on trace flags to reflect sampling behavior.
main.go	Integrates tracing setup with graceful shutdown and propagates trace context to controllers and webhooks.
helm_chart/*	Adds OpenTelemetry configuration options to Helm chart values and templates.
go.mod	Introduces OpenTelemetry dependencies.
docker/mongodb-kubernetes-tests/tests/conftest.py	Replaces print with logger and corrects a spelling mistake in a comment.
LICENSE-THIRD-PARTY	Updates third-party dependency list to include new OpenTelemetry libraries.