fix: Resolve uv.lock conflicting dependencies with semgrep

[jonpspri]

Summary

This PR resolves issue #1230 by addressing dependency resolution conflicts between semgrep and OpenTelemetry observability exporter packages. The root cause was that semgrep and various opentelemetry-exporter-* packages had incompatible dependency versions that prevented uv from successfully resolving the dependency tree.

Root Cause

The semgrep package and OpenTelemetry exporter packages (opentelemetry-exporter-jaeger, opentelemetry-exporter-zipkin, opentelemetry-exporter-otlp-proto-grpc) had conflicting dependencies that created an unresolvable dependency graph in uv.lock.

Solution

This PR restructures dependency management to eliminate the conflicts:

1. Makefile improvements

• Updated venv target to check if uv is already installed system-wide before installing it
• Modified test target to use uv run pytest instead of installing pytest separately
• Fixed semgrep target to use uvx semgrep to run semgrep in an isolated environment, avoiding dependency conflicts
• Removed syntax error in lint-smart target (extra fi statement)

2. pyproject.toml restructuring

• Moved development dependencies from [project.optional-dependencies] to [dependency-groups] (uv's preferred format)
• Removed semgrep from dev dependencies - now run via uvx which isolates its dependencies
• Simplified observability extras to only include core OpenTelemetry packages (opentelemetry-api, opentelemetry-sdk)
• Removed conflicting exporter extras: observability-jaeger, observability-zipkin, and observability-all
• Users now install specific exporter backends separately as needed
• Removed dev-all extras group

3. Documentation updates (observability.md)

• Expanded backend options with detailed descriptions of Arize Phoenix, Jaeger, and Grafana Tempo
• Added clear guidance on exporter backend installation per platform
• Clarified that only one exporter backend should be installed at a time to avoid conflicts
• Improved formatting and structure for better readability

4. uv.lock regeneration

• Updated lock file to reflect new dependency structure
• Successfully resolved all dependency conflicts

Test Plan

• Run uv lock to verify dependency resolution succeeds
• Run make venv to verify virtual environment creation works
• Run make test to verify tests run with uv run pytest
• Run make semgrep to verify semgrep runs via uvx in isolation
• Verify observability features still work with core OpenTelemetry packages
• Verify no dependency conflicts in uv.lock

Impact

This change transitions towards using uv/uvx for dependency management while maintaining backward compatibility with existing workflows. By running semgrep via uvx and simplifying observability extras, users can now install the gateway and its optional features without encountering dependency conflicts.

Closes #1230

[Copilot]

Pull Request Overview

This PR resolves dependency conflicts between semgrep and OpenTelemetry observability packages by restructuring dependency management and adopting uv/uvx for isolated execution.

• Restructured pyproject.toml to use uv's [dependency-groups] format and removed conflicting observability exporter extras
• Updated Makefile to use uvx semgrep for isolated execution and uv run pytest for testing
• Enhanced observability documentation with detailed backend descriptions and installation guidance

Reviewed Changes

Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.

File	Description
pyproject.toml	Restructured dependency management, moved dev dependencies to [dependency-groups], removed conflicting observability exporters, commented out semgrep dependency
docs/docs/manage/observability/observability.md	Expanded backend descriptions, added installation instructions per backend, clarified mutual exclusivity of exporters
Makefile	Updated to use uvx semgrep for isolated execution, uv run pytest for testing, and conditional uv installation

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[jonpspri]

Looking at what's changing the pytest configuration -- it has to do with removing the "dev" packaging from the optional packages. There's a good reason for that -- the list of dev dependencies doesn't ship with the production package. The down side is the pytest environment needs a different approach to installing dev/test dependencies.

[jonpspri]

@crivetimihai Are you able to trigger a re-run of the one failed job? It looks like the failure was from resource contention to me.

[jonpspri]

Yep... All reviews passed.

[jonpspri]

Rebase and push with a tweak to get that '|| true' that Copilot so loves back into the codebase.

[jonpspri]

Sorry for all the pushes... my branch got polluted with some other things I was working on.

[crivetimihai]

@jonpspri is this rebased and ready to merge, or does it have other files that should be a separate PR?

[jonpspri]

@crivetimihai It should be GTG. Everything in it now is related to resolving the semgrep/observability version conflicts.