Histograms are exported empty if no observations are recorded during a reading interval #3277
Description
Describe your environment
Python Version: 3.11.3
Installed dependencies:
opentelemetry-distro==0.38b0
opentelemetry-exporter-otlp==1.17.0
Running on Amazon Linux 2 with pyenv.
Steps to reproduce
Here is a minimal reproducible code:
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.metrics import set_meter_provider
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import (
ConsoleMetricExporter,
PeriodicExportingMetricReader,
)
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
import time
exporter = OTLPMetricExporter(endpoint="http://localhost:4318/v1/metrics")
console = ConsoleMetricExporter()
reader = PeriodicExportingMetricReader(exporter, export_interval_millis=30_000)
reader_console = PeriodicExportingMetricReader(console, export_interval_millis=30_000)
provider = MeterProvider(
resource=Resource.create({SERVICE_NAME: "otel-test"}),
metric_readers=[reader, reader_console],
)
set_meter_provider(provider)
meter = provider.get_meter("my-meter")
histogram = meter.create_histogram("my_histogram")
counter = meter.create_counter("my_counter")
for i in range(60):
histogram.record(5, {"attribute": "value"})
counter.add(1, {"attribute": "value_counter"})
# Notice how we are going to collect several times the aggregation before a new observation is recorded in the instruments.
time.sleep(600)What is the expected behaviour?
I would expect that histograms are properly exported, even if they are not receiving updates. I'm not sure what is the behaviour defined in the spec (i.e.: should not export if there are no observations or should export, even if there are no observations). Regardless, we should not export invalid content.
What is the actual behaviour?
When no observations are recorded in the histogram between two collections by the PeriodicExportingMetricReader, the second export will be problematic. The exported histogram will have no data as we can see in the output bellow from the ConsoleMetricExporter
python otel_manual.py
{
"resource_metrics": [
{
"resource": {
"attributes": {
"telemetry.sdk.language": "python",
"telemetry.sdk.name": "opentelemetry",
"telemetry.sdk.version": "1.17.0",
"service.name": "otel-test"
},
"schema_url": ""
},
"scope_metrics": [
{
"scope": {
"name": "my-meter",
"version": null,
"schema_url": ""
},
"metrics": [
{
"name": "my_histogram",
"description": "",
"unit": "",
"data": {
"data_points": [
{
"attributes": {
"attribute": "value"
},
"start_time_unix_nano": 1682118709187296460,
"time_unix_nano": 1682118739187659996,
"count": 1,
"sum": 5,
"bucket_counts": [
0,
1,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
"explicit_bounds": [
0.0,
5.0,
10.0,
25.0,
50.0,
75.0,
100.0,
250.0,
500.0,
750.0,
1000.0,
2500.0,
5000.0,
7500.0,
10000.0
],
"min": 5,
"max": 5
}
],
"aggregation_temporality": 2
}
},
{
"name": "my_counter",
"description": "",
"unit": "",
"data": {
"data_points": [
{
"attributes": {
"attribute": "value_counter"
},
"start_time_unix_nano": 1682118709187344532,
"time_unix_nano": 1682118739187659996,
"value": 1
}
],
"aggregation_temporality": 2,
"is_monotonic": true
}
}
],
"schema_url": ""
}
],
"schema_url": ""
}
]
}
{
"resource_metrics": [
{
"resource": {
"attributes": {
"telemetry.sdk.language": "python",
"telemetry.sdk.name": "opentelemetry",
"telemetry.sdk.version": "1.17.0",
"service.name": "otel-test"
},
"schema_url": ""
},
"scope_metrics": [
{
"scope": {
"name": "my-meter",
"version": null,
"schema_url": ""
},
"metrics": [
{
"name": "my_histogram",
"description": "",
"unit": "",
"data": {
"data_points": [],
"aggregation_temporality": 2
}
},
{
"name": "my_counter",
"description": "",
"unit": "",
"data": {
"data_points": [
{
"attributes": {
"attribute": "value_counter"
},
"start_time_unix_nano": 1682118709187344532,
"time_unix_nano": 1682118769193444639,
"value": 1
}
],
"aggregation_temporality": 2,
"is_monotonic": true
}
}
],
"schema_url": ""
}
],
"schema_url": ""
}
]
}
{
"resource_metrics": [
{
"resource": {
"attributes": {
"telemetry.sdk.language": "python",
"telemetry.sdk.name": "opentelemetry",
"telemetry.sdk.version": "1.17.0",
"service.name": "otel-test"
},
"schema_url": ""
},
"scope_metrics": [
{
"scope": {
"name": "my-meter",
"version": null,
"schema_url": ""
},
"metrics": [
{
"name": "my_histogram",
"description": "",
"unit": "",
"data": {
"data_points": [],
"aggregation_temporality": 2
}
},
{
"name": "my_counter",
"description": "",
"unit": "",
"data": {
"data_points": [
{
"attributes": {
"attribute": "value_counter"
},
"start_time_unix_nano": 1682118709187344532,
"time_unix_nano": 1682118799194444986,
"value": 1
}
],
"aggregation_temporality": 2,
"is_monotonic": true
}
}
],
"schema_url": ""
}
],
"schema_url": ""
}
]
}
The export operation that happens immediately after a new observation will be successful. The main issue here is exporting invalid data to OTLP receivers because this will blow the collector down the line.
For instance, this is what the collector receives. The histogram is not even recognized as histogram.
collector-1 | 2023-04-21T23:13:23.416Z info ResourceMetrics #0
collector-1 | Resource SchemaURL:
collector-1 | Resource attributes:
collector-1 | -> telemetry.sdk.language: Str(python)
collector-1 | -> telemetry.sdk.name: Str(opentelemetry)
collector-1 | -> telemetry.sdk.version: Str(1.17.0)
collector-1 | -> service.name: Str(otel-test)
collector-1 | ScopeMetrics #0
collector-1 | ScopeMetrics SchemaURL:
collector-1 | InstrumentationScope my-meter
collector-1 | Metric #0
collector-1 | Descriptor:
collector-1 | -> Name: my_histogram
collector-1 | -> Description:
collector-1 | -> Unit:
collector-1 | -> DataType: Empty
collector-1 | Metric #1
collector-1 | Descriptor:
collector-1 | -> Name: my_counter
collector-1 | -> Description:
collector-1 | -> Unit:
collector-1 | -> DataType: Sum
collector-1 | -> IsMonotonic: true
collector-1 | -> AggregationTemporality: Cumulative
collector-1 | NumberDataPoints #0
collector-1 | Data point attributes:
collector-1 | -> attribute: Str(value_counter)
collector-1 | StartTimestamp: 2023-04-21 23:11:49.187322766 +0000 UTC
collector-1 | Timestamp: 2023-04-21 23:13:23.414505953 +0000 UTC
collector-1 | Value: 1
Additional context
Add any other context about the problem here.