[observability] most basic OpenTelemetry integration into MCK

[nammn]

Summary

This pull request introduces OpenTelemetry tracing support to the MongoDB Kubernetes Operator and its related components. Key changes include the integration of OpenTelemetry libraries, the addition of tracing configuration, and updates to ensure trace propagation across the application. These changes enhance observability and debugging capabilities.

In our CI suite this means we will have the following kind of traces:

trace_id: abc123

                         ┌────────────────────┐
                         │     Evergreen      │
                         │  span_id: ROOT     │
                         │  parent_id: none   │
                         └─────────┬──────────┘
                                   │
        ┌──────────────────────────┼─────────────────────────┐
        ▼                          ▼                         ▼
┌──────────────┐         ┌────────────────┐         ┌────────────────────┐
│   E2E Test   │         │   Operator     │         │     (Other…)       │
│ span_id: A1  │         │ span_id: B1    │         │                    │
│ parent: ROOT │         │ parent: ROOT   │         │                    │
└──────┬───────┘         └──────┬─────────┘         └────────────────────┘
       │                        │
       ▼                        ▼
┌──────────────┐         ┌────────────────────┐
│ E2E Function │         │   Reconcile Loop   │
│ span_id: A2  │         │   span_id: B2      │
│ parent: A1   │         │   parent: B1       │
└──────────────┘         └────────────────────┘

OpenTelemetry Integration:

• Tracing in main.go:
  • Added OpenTelemetry setup in the main function, including trace and span ID extraction from environment variables and the creation of a root span for the operator. Tracing context is propagated across controllers and shutdown processes are handled gracefully.
• Telemetry in pkg/telemetry/client.go: <--- this is good to know if we happen to make a change and happen to send to prod atlas
  • Added a span to the SendEventWithRetry function to capture telemetry events and include the Atlas base URL as a span attribute.

Helm Chart Updates:

• Operator configuration:
  • Added OpenTelemetry-specific environment variables (OTEL_TRACE_ID, OTEL_PARENT_ID, OTEL_EXPORTER_OTLP_ENDPOINT) to the operator's deployment template. ([helm_chart/templates/operator.yamlR83-R90](https://github.com/mongodb/mongodb-kubernetes/pull/93/files#diff-5d2e377a6806023ca9eff60be4d7e5cd879803de2bd3800b630f479f8728f322R83-R90))
  • Introduced OpenTelemetry configuration options (enabled, traceID, parentID, collectorEndpoint) in the Helm chart's values.yaml.

Dependency Updates:

• Go module dependencies:
  • Added OpenTelemetry-related libraries (otel, otel/sdk, otel/trace, etc.) to go.mod.

Proof of Work

• e.g. patch

• generated traces in our ci: Link
  

Checklist

• Have you linked a jira ticket and/or is the ticket in the title?
• Have you checked whether your jira ticket required DOCSP changes?
• Have you checked for release_note changes?

Reminder (Please remove this when merging)

• Please try to Approve or Reject Changes the PR, keep PRs in review as short as possible
• Our Short Guide for PRs: Link
• Remember the following Communication Standards - use comment prefixes for clarity:
  • blocking: Must be addressed before approval.
  • follow-up: Can be addressed in a later PR or ticket.
  • q: Clarifying question.
  • nit: Non-blocking suggestions.
  • note: Side-note, non-actionable. Example: Praise
  • --> no prefix is considered a question

[nammn]

updated the service-name for better querying

[nammn]

we should remove the operator to test teardown at the end of the test-run + it enables us to ensure traces are exported in time

[lsierant]

This is a very, very nice change and I would like to have it included. But as of right now I think we should discuss this PR a bit more before deciding to merge. There are too many end-user implications to just LGTM it now.

[nammn]

we have one operator now - this is a cleanup

[nammn]

cleanup

[nammn]

this was not running before and therefore we had no flushing

[nammn]

let's wait until all things are flushed before we stop. PoW is in the PR description - you can see the trace of the oeprator

[nammn]

pow:

(venv) ~/projects/ops-manager-kubernetes git:[traces-operator]
helm template helm_chart | rg OTEL
(venv) ~/projects/ops-manager-kubernetes git:[traces-operator]
helm template --set operator.opentelemetry.tracing.enabled=true --set operator.opentelemetry.tracing.traceID=your-trace-id --set operator.opentelemetry.tracing.parentID=your-parent-id --set operator.opentelemetry.tracing.collectorEndpoint=http://jaeger:14268/api/traces helm_chart | rg OTEL
            - name: OTEL_TRACE_ID
            - name: OTEL_PARENT_ID
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
(venv) ~/projects/ops-manager-kubernetes git:[traces-operator]
helm template --set operator.opentelemetry.tracing.enabled=true --set operator.opentelemetry.tracing.traceID=your-trace-id --set operator.opentelemetry.tracing.parentID=your-parent-id --set operator.opentelemetry.tracing.collectorEndpoint=http://jaeger:14268/api/traces helm_chart | rg OTEL -C 2
                fieldRef:
                  fieldPath: metadata.namespace
            - name: OTEL_TRACE_ID
              value: "your-trace-id"
            - name: OTEL_PARENT_ID
              value: "your-parent-id"
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://jaeger:14268/api/traces"
            - name: WATCH_NAMESPACE

[Copilot]

Pull Request Overview

Introduces basic OpenTelemetry tracing into the MongoDB Kubernetes Operator (MCK) and related scripts, enabling trace propagation from CI tests through operator spans.

• Initializes tracing in main.go using OTEL env vars and creates a root operator span.
• Instruments core telemetry functions (trace.go, configmap.go, collector.go, client.go) with spans and attributes.
• Propagates OTEL settings through Helm charts, shell scripts, and test configurations.

Reviewed Changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
scripts/funcs/operator_deployment	Adds parsing of OTEL env vars to Helm values
scripts/funcs/kubernetes	Updates default operator name and cleanup/uninstall logic
scripts/evergreen/e2e/e2e.sh	Refactors cluster diagnostics and OpenShift cleanup sequences
scripts/evergreen/deployments/test-app/templates/mongodb-enterprise-tests.yaml	Sets OTEL_SERVICE_NAME and pytest --trace-parent flags
pkg/telemetry/trace.go	Implements SetupTracingFromParent with OTLP exporter
pkg/telemetry/configmap.go	Wraps ConfigMap creation in a tracing span
pkg/telemetry/collector.go	Adds a span around RunTelemetry
pkg/telemetry/client.go	Adds a span in SendEventWithRetry with Atlas base URL
pipeline.py	Clarifies trace_flags comment
main.go	Hooks up tracing setup, root span, and tracer shutdown
helm_chart/templates/operator.yaml	Injects OTEL env vars into the operator deployment
go.mod	Adds OpenTelemetry dependencies
docker/mongodb-kubernetes-tests/tests/conftest.py	Uses logger.debug instead of print and fixes a typo
LICENSE-THIRD-PARTY	Updates third-party license entries for new dependencies

Comments suppressed due to low confidence (1)

pkg/telemetry/client.go:118

• No unit tests cover the new OpenTelemetry instrumentation in SendEventWithRetry; consider adding tests to validate span creation and attribute setting.

	_, span := TRACER.Start(ctx, "SendEventWithRetry")

[Copilot]

[nitpick] Consider calling span.RecordError(err) inside the error branch of the ConfigMap creation to capture failures in the trace.

[nammn]

KUBE_ENVIRONMENT_NAME="dev-openshift-cluster"

if [[ "${KUBE_ENVIRONMENT_NAME}" == *openshift* ]]; then
  echo "Contains openshift"
else
  echo "Does NOT contain openshift"
fi

-> Contains openshift

[nammn]

for some reason it was duplicated

[lsierant]

Thanks for the changes! LGTM!