Add Ourlogs

[aldy505]

Problem Statement

Product Discussion: getsentry/sentry#86804
JS SDK docs: getsentry/sentry-javascript#15916

Solution Brainstorm

The way we'll tell you how to configure this should be versioned (learned this the hard way from User Feedback feature). Grab the version you're currently on (or if yours is higher than any of these, choose the highest one).

• Setup guide for 25.2.0
• Setup guide for 25.3.0
• Setup guide for 25.4.0

[aldy505]

Setup Guide for 25.2.0

Modify sentry.conf.py:

SENTRY_FEATURES = # ...
  ) +
  # Ourlogs
  (
    "organizations:ourlogs-enabled",
    "organizations:ourlogs-ingestion",
    "projects:ourlogs-breadcrumb-extraction",
  )

Modify docker-compose.yml

  snuba-ourlogs-consumer:
    <<: *snuba_defaults
    command: rust-consumer --storage ourlogs --consumer-group ourlogs_group --auto-offset-reset=latest --max-batch-time-ms 1000 --no-strict-offset-reset --use-rust-processor
    profiles:
      - feature-complete

Proof:

[aldy505]

Setup Guide for 25.3.0

Modify sentry/sentry.conf.py:

SENTRY_FEATURES = # ...
  ) +
  # Ourlogs
  (
    "organizations:ourlogs-enabled",
    "organizations:ourlogs-ingestion",
    "projects:ourlogs-breadcrumb-extraction",
  )

Modify sentry/config.yml:

relay.ourlogs-breadcrumb-extraction.sample-rate: 1.0
relay.ourlogs-ingestion.sample-rate: 1.0

Modify docker-compose.yml

  snuba-ourlogs-consumer:
    <<: *snuba_defaults
    command: rust-consumer --storage eap_items_log --consumer-group ourlogs_group --auto-offset-reset=latest --max-batch-time-ms 1000 --no-strict-offset-reset --use-rust-processor
    profiles:
      - feature-complete

Proof:

[aldy505]

Setup Guide for 25.4.0

Date of writing: 2025-04-17

Modify sentry/sentry.conf.py:

SENTRY_FEATURES = # ...
  ) +
  # Ourlogs
  (
        "organizations:ourlogs-enabled",
        "organizations:ourlogs-ingestion",
        "organizations:ourlogs-stats",
        "organizations:ourlogs-graph",
        "organizations:ourlogs-visualize-sidebar",
        "organizations:ourlogs-dashboards",
        "organizations:ourlogs-alerts",
        "organizations:ourlogs-live-refresh",
        "projects:ourlogs-breadcrumb-extraction",
  )

Modify sentry/config.yml:

relay.ourlogs-breadcrumb-extraction.sample-rate: 1.0
relay.ourlogs-ingestion.sample-rate: 1.0

Modify docker-compose.yml

  snuba-ourlogs-consumer:
    <<: *snuba_defaults
    command: rust-consumer --storage eap_items_log --consumer-group ourlogs_group --auto-offset-reset=latest --max-batch-time-ms 1000 --no-strict-offset-reset --use-rust-processor
    profiles:
      - feature-complete

[radalin]

I have a self hosted sentry which I bumped to 25.5 today and tried to initialize the logs stuff. I have added the mentioned stuff in the code and restarted sentry. I have been able to see the Logs section. I have bumped the python sentry sdk to newest version and setup logger, however I was not able to see the logs. Then I have created a sentry.io account and sent the logs there and it worked. When I checked the logs in the snubar-ourlogs-consumer I noticed the following error:

snuba-ourlogs-consumer-1  | 2025-05-20 18:30:29,561 Initializing Snuba...
snuba-ourlogs-consumer-1  | 2025-05-20 18:30:36,751 Snuba initialization took 7.190729629714042s
snuba-ourlogs-consumer-1  | 2025-05-20 18:30:37,738 Initializing Snuba...

snuba-ourlogs-consumer-1  | 2025-05-20 18:30:45,037 Snuba initialization took 7.299777054227889s
snuba-ourlogs-consumer-1  | Usage: snuba rust-consumer [OPTIONS]
snuba-ourlogs-consumer-1  | Try 'snuba rust-consumer --help' for help.
snuba-ourlogs-consumer-1  | 
snuba-ourlogs-consumer-1  | Error: Invalid value for '--storage': 'eap_items_log' is not one of 'errors', 'eap_items_span', 'uptime_monitor_checks', 'eap_items', 'profiles', 'profile_chunks', 'replays', 'search_issues', 'querylog', 'transactions', 'functions_raw', 'spans', 'outcomes_raw', 'metrics_raw', 'groupassignees', 'generic_metrics_sets_raw', 'generic_metrics_distributions_raw', 'generic_metrics_counters_raw', 'generic_metrics_gauges_raw', 'group_attributes', 'groupedmessages'.

I have encountered the same error with "ourlogs" value for the storage option. After setting the settings as mentioned, I have also ran the "install.sh" command but the error didn't change

Can you confirm if those values are correct for 25.5? Or maybe there is something that needs to be done to initialize the snuba storage?

[radalin]

I have done a bit more digging and based on this commit: getsentry/snuba@77472c2 it seems the dedicated storage for logs is removed and merged with spans and therefore new storage name is "eap_items" instead of eap_items_log. So for 25.5 you should use the below docker compose addition:

  snuba-ourlogs-consumer:
    <<: *snuba_defaults
    command: rust-consumer --storage eap_items --consumer-group ourlogs_group --auto-offset-reset=latest --max-batch-time-ms 1000 --no-strict-offset-reset --use-rust-processor
    profiles:
      - feature-complete