[tracing] add OTEL quartz autoinstrumentation #7192

chojomok · 2025-07-07T19:06:25Z

Summary of changes

Added a new QuartzDiagnosticObserver to enable OTEL instrumentation from Quartz.
- v3 uses a Diagnostic Observer to make Activities.
- v4 uses ActivitySource.
  - v4 snapshot generated from version 4.0.0-preview-20250718-1741 of the library
Integration is supported for v3.1 of Quartz.
Requires DD_TRACE_OTEL_ENABLED=true for tracing to work.
Currently not available for .NET framework due to our DiagnosticObserver infrastructure not being compatible with .NET Framework. Will look into expanding support in the future.
Changes to Activity5
- Added setters to the following properties
  - DisplayName
  - Kind - this is private but with bytecode we are able to call the setter.

Reason for change

This change enables Datadog APM to trace Quartz.NET job scheduling and execution events, providing observability for background job processing. It expands instrumentation coverage for .NET applications using Quartz, addressing a gap in our supported integrations.

Implementation details

We create an Diagnostic Observer to listen on "Quartz" so the built in OTEL Instrumentation can be used and transformed into a Datadog Span.

Test coverage

Tests from v3.1 of Quartz to the latest version
Tests for upcoming v4 from the prerelease is available and tested on locally.

Other details

Activity Kind Setter

We used the same implementation as the opentelemetry-dotnet-contrib repo which can be found in this code snippet
since we don't have access to the classes as the contrib repo, we used bytecode to avoid casting issues.

dd-trace-dotnet-ci-bot · 2025-07-07T20:04:22Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7192) - mean (72ms)  : 70, 73
     .   : milestone, 72,
    master - mean (73ms)  : 71, 75
     .   : milestone, 73,

    section Baseline
    This PR (7192) - mean (68ms)  : 66, 70
     .   : milestone, 68,
    master - mean (70ms)  : 67, 73
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (7192) - mean (1,045ms)  : 1009, 1080
     .   : milestone, 1045,
    master - mean (1,050ms)  : 1011, 1090
     .   : milestone, 1050,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7192) - mean (106ms)  : 105, 108
     .   : milestone, 106,
    master - mean (108ms)  : 106, 109
     .   : milestone, 108,

    section Baseline
    This PR (7192) - mean (106ms)  : 104, 108
     .   : milestone, 106,
    master - mean (107ms)  : 104, 109
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (7192) - mean (741ms)  : 718, 763
     .   : milestone, 741,
    master - mean (747ms)  : 727, 767
     .   : milestone, 747,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7192) - mean (100ms)  : 99, 101
     .   : milestone, 100,
    master - mean (102ms)  : 100, 104
     .   : milestone, 102,

    section Baseline
    This PR (7192) - mean (99ms)  : 97, 101
     .   : milestone, 99,
    master - mean (101ms)  : 98, 104
     .   : milestone, 101,

    section CallTarget+Inlining+NGEN
    This PR (7192) - mean (770ms)  : 725, 816
     .   : milestone, 770,
    master - mean (785ms)  : 750, 819
     .   : milestone, 785,

gantt
    title Execution time (ms) FakeDbCommand (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7192) - mean (93ms)  : 91, 94
     .   : milestone, 93,
    master - mean (93ms)  : 92, 95
     .   : milestone, 93,

    section Baseline
    This PR (7192) - mean (92ms)  : 89, 94
     .   : milestone, 92,
    master - mean (93ms)  : 91, 96
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    This PR (7192) - mean (656ms)  : 644, 668
     .   : milestone, 656,
    master - mean (666ms)  : 651, 682
     .   : milestone, 666,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7192) - mean (207ms)  : 198, 216
     .   : milestone, 207,
    master - mean (213ms)  : 199, 226
     .   : milestone, 213,

    section Baseline
    This PR (7192) - mean (206ms)  : 196, 215
     .   : milestone, 206,
    master - mean (206ms)  : 193, 218
     .   : milestone, 206,

    section CallTarget+Inlining+NGEN
    This PR (7192) - mean (1,208ms)  : 1154, 1261
     .   : milestone, 1208,
    master - mean (1,225ms)  : 1148, 1301
     .   : milestone, 1225,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7192) - mean (298ms)  : 281, 315
     .   : milestone, 298,
    master - mean (295ms)  : 277, 312
     .   : milestone, 295,

    section Baseline
    This PR (7192) - mean (295ms)  : 279, 310
     .   : milestone, 295,
    master - mean (298ms)  : 278, 318
     .   : milestone, 298,

    section CallTarget+Inlining+NGEN
    This PR (7192) - mean (984ms)  : 939, 1028
     .   : milestone, 984,
    master - mean (983ms)  : 938, 1028
     .   : milestone, 983,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7192) - mean (301ms)  : 285, 316
     .   : milestone, 301,
    master - mean (305ms)  : 287, 323
     .   : milestone, 305,

    section Baseline
    This PR (7192) - mean (300ms)  : 286, 314
     .   : milestone, 300,
    master - mean (302ms)  : 281, 322
     .   : milestone, 302,

    section CallTarget+Inlining+NGEN
    This PR (7192) - mean (1,027ms)  : 980, 1074
     .   : milestone, 1027,
    master - mean (1,042ms)  : 985, 1098
     .   : milestone, 1042,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 8) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Bailout
    This PR (7192) - mean (289ms)  : 273, 305
     .   : milestone, 289,
    master - mean (292ms)  : 271, 312
     .   : milestone, 292,

    section Baseline
    This PR (7192) - mean (288ms)  : 272, 304
     .   : milestone, 288,
    master - mean (291ms)  : 270, 311
     .   : milestone, 291,

    section CallTarget+Inlining+NGEN
    This PR (7192) - mean (908ms)  : 818, 999
     .   : milestone, 908,
    master - mean (932ms)  : 840, 1024
     .   : milestone, 932,

pr-commenter · 2025-07-07T20:25:40Z

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmarks for #7192 compared to master:

1 benchmarks are faster, with geometric mean 2.040
1 benchmarks are slower, with geometric mean 2.440
5 benchmarks have fewer allocations
12 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	10.9μs	55.3ns	271ns	0	0	0	5.49 KB
master	`StartStopWithChild`	netcoreapp3.1	13.6μs	69.5ns	348ns	0	0	0	5.68 KB
master	`StartStopWithChild`	net472	22.5μs	124ns	734ns	0.939	0.209	0	6.04 KB
#7192	`StartStopWithChild`	net6.0	10.7μs	54.1ns	242ns	0	0	0	5.5 KB
#7192	`StartStopWithChild`	netcoreapp3.1	13.2μs	63.7ns	263ns	0	0	0	5.69 KB
#7192	`StartStopWithChild`	net472	22.8μs	127ns	853ns	1.04	0.347	0.116	6.03 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7192

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces‑net472	3.31 KB	3.35 KB	46 B	1.39%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	924μs	455ns	1.76μs	2.71 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	1.02ms	1.17μs	4.39μs	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	1.2ms	306ns	1.19μs	3.31 KB
#7192	`WriteAndFlushEnrichedTraces`	net6.0	934μs	78.5ns	304ns	2.71 KB
#7192	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	1.02ms	69.1ns	259ns	2.7 KB
#7192	`WriteAndFlushEnrichedTraces`	net472	1.2ms	68.4ns	256ns	3.35 KB

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7192

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net472	194.25 KB	246.48 KB	52.23 KB	26.89%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net472	197.76 KB	249.99 KB	52.23 KB	26.41%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑netcoreapp3.1	174.18 KB	206.2 KB	32.02 KB	18.38%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑netcoreapp3.1	177.6 KB	209.62 KB	32.02 KB	18.03%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody‑net6.0	172.07 KB	198.49 KB	26.42 KB	15.35%
Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody‑net6.0	175.58 KB	202 KB	26.41 KB	15.04%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`AllCycleSimpleBody`	net6.0	320μs	1.46μs	5.83μs	0	0	172.07 KB
master	`AllCycleSimpleBody`	netcoreapp3.1	452μs	289ns	1.12μs	0	0	174.18 KB
master	`AllCycleSimpleBody`	net472	433μs	146ns	567ns	30.2	0	194.25 KB
master	`AllCycleMoreComplexBody`	net6.0	327μs	171ns	640ns	0	0	175.58 KB
master	`AllCycleMoreComplexBody`	netcoreapp3.1	501μs	1.01μs	3.9μs	0	0	177.6 KB
master	`AllCycleMoreComplexBody`	net472	439μs	131ns	508ns	30.2	0	197.76 KB
master	`ObjectExtractorSimpleBody`	net6.0	327ns	1.83ns	10.8ns	0	0	280 B
master	`ObjectExtractorSimpleBody`	netcoreapp3.1	398ns	2.17ns	11.7ns	0	0	272 B
master	`ObjectExtractorSimpleBody`	net472	301ns	0.154ns	0.577ns	0.044	0	281 B
master	`ObjectExtractorMoreComplexBody`	net6.0	6.41μs	8.81ns	34.1ns	0	0	3.78 KB
master	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	7.76μs	37.6ns	155ns	0	0	3.69 KB
master	`ObjectExtractorMoreComplexBody`	net472	6.77μs	2.82ns	10.5ns	0.577	0	3.8 KB
#7192	`AllCycleSimpleBody`	net6.0	334μs	1.77μs	8.68μs	0	0	198.49 KB
#7192	`AllCycleSimpleBody`	netcoreapp3.1	479μs	1.73μs	6.25μs	0	0	206.2 KB
#7192	`AllCycleSimpleBody`	net472	452μs	89ns	345ns	37.9	2.23	246.48 KB
#7192	`AllCycleMoreComplexBody`	net6.0	344μs	457ns	1.77μs	0	0	202 KB
#7192	`AllCycleMoreComplexBody`	netcoreapp3.1	495μs	2.38μs	9.5μs	0	0	209.62 KB
#7192	`AllCycleMoreComplexBody`	net472	457μs	140ns	504ns	37.9	2.23	249.99 KB
#7192	`ObjectExtractorSimpleBody`	net6.0	327ns	0.244ns	0.944ns	0	0	280 B
#7192	`ObjectExtractorSimpleBody`	netcoreapp3.1	402ns	2.25ns	14.6ns	0	0	272 B
#7192	`ObjectExtractorSimpleBody`	net472	301ns	0.0471ns	0.182ns	0.0439	0	281 B
#7192	`ObjectExtractorMoreComplexBody`	net6.0	6.26μs	32.7ns	157ns	0	0	3.78 KB
#7192	`ObjectExtractorMoreComplexBody`	netcoreapp3.1	7.91μs	6.8ns	26.3ns	0	0	3.69 KB
#7192	`ObjectExtractorMoreComplexBody`	net472	6.65μs	0.924ns	3.2ns	0.598	0	3.8 KB

Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EncodeArgs`	net6.0	77μs	206ns	772ns	0	32.4 KB
master	`EncodeArgs`	netcoreapp3.1	96.8μs	228ns	882ns	0	32.4 KB
master	`EncodeArgs`	net472	110μs	7.64ns	29.6ns	4.93	32.5 KB
master	`EncodeLegacyArgs`	net6.0	145μs	17.6ns	68ns	0	2.14 KB
master	`EncodeLegacyArgs`	netcoreapp3.1	196μs	195ns	753ns	0	2.14 KB
master	`EncodeLegacyArgs`	net472	264μs	38.1ns	143ns	0	2.17 KB
#7192	`EncodeArgs`	net6.0	78.7μs	366ns	1.42μs	0	32.4 KB
#7192	`EncodeArgs`	netcoreapp3.1	99.4μs	25.2ns	94.2ns	0	32.4 KB
#7192	`EncodeArgs`	net472	109μs	14.6ns	56.7ns	4.92	32.5 KB
#7192	`EncodeLegacyArgs`	net6.0	145μs	55ns	213ns	0	2.14 KB
#7192	`EncodeLegacyArgs`	netcoreapp3.1	196μs	199ns	745ns	0	2.14 KB
#7192	`EncodeLegacyArgs`	net472	260μs	18.4ns	68.9ns	0	2.17 KB

Benchmarks.Trace.Asm.AppSecWafBenchmark - Slower ⚠️ Fewer allocations 🎉

Slower ⚠️ in #7192

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack‑netcoreapp3.1	2.440	300,701.07	733,739.06

Faster 🎉 in #7192

Benchmark	base/diff	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark‑netcoreapp3.1	2.040	853,087.08	418,280.71

Fewer allocations 🎉 in #7192

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark‑netcoreapp3.1	4.51 KB	4.48 KB	-28 B	-0.62%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`RunWafRealisticBenchmark`	net6.0	403μs	147ns	570ns	4.55 KB
master	`RunWafRealisticBenchmark`	netcoreapp3.1	852μs	2.37μs	8.54μs	4.51 KB
master	`RunWafRealisticBenchmark`	net472	432μs	61.7ns	239ns	4.66 KB
master	`RunWafRealisticBenchmarkWithAttack`	net6.0	291μs	84.5ns	327ns	2.24 KB
master	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	301μs	90.3ns	338ns	2.22 KB
master	`RunWafRealisticBenchmarkWithAttack`	net472	318μs	47ns	182ns	2.29 KB
#7192	`RunWafRealisticBenchmark`	net6.0	400μs	96.8ns	362ns	4.55 KB
#7192	`RunWafRealisticBenchmark`	netcoreapp3.1	418μs	241ns	932ns	4.48 KB
#7192	`RunWafRealisticBenchmark`	net472	440μs	73.9ns	276ns	4.66 KB
#7192	`RunWafRealisticBenchmarkWithAttack`	net6.0	291μs	114ns	426ns	2.24 KB
#7192	`RunWafRealisticBenchmarkWithAttack`	netcoreapp3.1	688μs	13.9μs	130μs	2.22 KB
#7192	`RunWafRealisticBenchmarkWithAttack`	net472	319μs	71.2ns	276ns	2.29 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`SendRequest`	net6.0	59.9μs	244ns	1.03μs	14.52 KB
master	`SendRequest`	netcoreapp3.1	72.2μs	337ns	1.35μs	17.42 KB
master	`SendRequest`	net472	0.00142ns	0.000876ns	0.00339ns	0 b
#7192	`SendRequest`	net6.0	60.6μs	226ns	815ns	14.52 KB
#7192	`SendRequest`	netcoreapp3.1	70.5μs	51.8ns	194ns	17.42 KB
#7192	`SendRequest`	net472	0.00088ns	0.000443ns	0.00172ns	0 b

Benchmarks.Trace.CharSliceBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7192

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑netcoreapp3.1	0 b	1 B	1 B	∞
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑net6.0	2 B	4 B	2 B	100.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net6.0	3 B	4 B	1 B	33.33%

Fewer allocations 🎉 in #7192

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice‑net472	73 B	0 b	-73 B	-100.00%
Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool‑net472	46 B	0 b	-46 B	-100.00%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`OriginalCharSlice`	net6.0	1.86ms	5.02μs	18.1μs	0	640.06 KB
master	`OriginalCharSlice`	netcoreapp3.1	2.1ms	988ns	3.7μs	0	640 KB
master	`OriginalCharSlice`	net472	2.73ms	9.36μs	36.2μs	100	641.95 KB
master	`OptimizedCharSlice`	net6.0	1.35ms	216ns	838ns	0	2 B
master	`OptimizedCharSlice`	netcoreapp3.1	1.69ms	500ns	1.93μs	0	1 B
master	`OptimizedCharSlice`	net472	1.97ms	893ns	3.46μs	0	73 B
master	`OptimizedCharSliceWithPool`	net6.0	888μs	249ns	930ns	0	3 B
master	`OptimizedCharSliceWithPool`	netcoreapp3.1	840μs	41.1ns	154ns	0	0 b
master	`OptimizedCharSliceWithPool`	net472	1.13ms	186ns	670ns	0	46 B
#7192	`OriginalCharSlice`	net6.0	1.91ms	3.02μs	11.7μs	0	640.01 KB
#7192	`OriginalCharSlice`	netcoreapp3.1	2.1ms	4.8μs	16.6μs	0	640 KB
#7192	`OriginalCharSlice`	net472	2.66ms	329ns	1.27μs	100	641.95 KB
#7192	`OptimizedCharSlice`	net6.0	1.41ms	372ns	1.39μs	0	4 B
#7192	`OptimizedCharSlice`	netcoreapp3.1	1.7ms	187ns	723ns	0	1 B
#7192	`OptimizedCharSlice`	net472	2.08ms	335ns	1.3μs	0	0 b
#7192	`OptimizedCharSliceWithPool`	net6.0	838μs	26ns	97.4ns	0	4 B
#7192	`OptimizedCharSliceWithPool`	netcoreapp3.1	878μs	204ns	789ns	0	1 B
#7192	`OptimizedCharSliceWithPool`	net472	1.25ms	327ns	1.27μs	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7192

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0	41.88 KB	42.71 KB	826 B	1.97%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	644μs	504ns	1.82μs	0	41.88 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	787μs	3.15μs	12.2μs	0	41.79 KB
master	`WriteAndFlushEnrichedTraces`	net472	815μs	1.02μs	3.82μs	4.46	56.04 KB
#7192	`WriteAndFlushEnrichedTraces`	net6.0	652μs	1.2μs	4.51μs	0	42.71 KB
#7192	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	723μs	4.24μs	41.1μs	0	41.84 KB
#7192	`WriteAndFlushEnrichedTraces`	net472	901μs	5.14μs	38.8μs	4.46	55.96 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.83μs	8.56ns	33.1ns	0	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	2.45μs	11.2ns	43.5ns	0	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.73μs	2.99ns	11.6ns	0.151	0.0138	987 B
#7192	`ExecuteNonQuery`	net6.0	1.92μs	9.08ns	35.2ns	0	0	1.02 KB
#7192	`ExecuteNonQuery`	netcoreapp3.1	2.65μs	8.63ns	33.4ns	0	0	1.02 KB
#7192	`ExecuteNonQuery`	net472	2.8μs	3.33ns	12.5ns	0.154	0.014	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.76μs	0.929ns	3.47ns	0	1.03 KB
master	`CallElasticsearch`	netcoreapp3.1	2.22μs	10.5ns	43.5ns	0	1.03 KB
master	`CallElasticsearch`	net472	3.58μs	6.88ns	26.6ns	0.162	1.04 KB
master	`CallElasticsearchAsync`	net6.0	1.75μs	7.94ns	31.8ns	0	1.01 KB
master	`CallElasticsearchAsync`	netcoreapp3.1	2.4μs	10.9ns	43.8ns	0	1.08 KB
master	`CallElasticsearchAsync`	net472	3.79μs	6.55ns	25.4ns	0.171	1.1 KB
#7192	`CallElasticsearch`	net6.0	1.7μs	1.96ns	7.58ns	0	1.03 KB
#7192	`CallElasticsearch`	netcoreapp3.1	2.24μs	10ns	37.4ns	0	1.03 KB
#7192	`CallElasticsearch`	net472	3.6μs	2.37ns	9.17ns	0.163	1.04 KB
#7192	`CallElasticsearchAsync`	net6.0	1.77μs	8.92ns	40.9ns	0	1.01 KB
#7192	`CallElasticsearchAsync`	netcoreapp3.1	2.49μs	12.9ns	62.1ns	0	1.08 KB
#7192	`CallElasticsearchAsync`	net472	3.84μs	2.74ns	10.6ns	0.172	1.1 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.83μs	8.77ns	35.1ns	0	952 B
master	`ExecuteAsync`	netcoreapp3.1	2.28μs	3.85ns	14.9ns	0	952 B
master	`ExecuteAsync`	net472	2.57μs	1.88ns	7.3ns	0.141	915 B
#7192	`ExecuteAsync`	net6.0	1.75μs	6.76ns	26.2ns	0	952 B
#7192	`ExecuteAsync`	netcoreapp3.1	2.3μs	5.63ns	21.8ns	0	952 B
#7192	`ExecuteAsync`	net472	2.61μs	3.43ns	13.3ns	0.142	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	7.02μs	23.1ns	89.4ns	0	2.36 KB
master	`SendAsync`	netcoreapp3.1	8.42μs	30.8ns	119ns	0	2.9 KB
master	`SendAsync`	net472	12.2μs	6.88ns	26.7ns	0.487	3.18 KB
#7192	`SendAsync`	net6.0	6.86μs	32ns	120ns	0	2.36 KB
#7192	`SendAsync`	netcoreapp3.1	8.75μs	15.7ns	58.8ns	0	2.9 KB
#7192	`SendAsync`	net472	12.3μs	7.06ns	27.3ns	0.491	3.18 KB

Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed ✔️ More allocations ⚠️

More allocations ⚠️ in #7192

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0	259.34 KB	282.11 KB	22.78 KB	8.78%

Fewer allocations 🎉 in #7192

Benchmark	Base Allocated	Diff Allocated	Change	Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1	277.64 KB	276.13 KB	-1.51 KB	-0.54%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑netcoreapp3.1	45.55 KB	42.64 KB	-2.91 KB	-6.39%

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`StringConcatBenchmark`	net6.0	42.3μs	247ns	2.02μs	43.78 KB
master	`StringConcatBenchmark`	netcoreapp3.1	54.1μs	887ns	8.69μs	45.55 KB
master	`StringConcatBenchmark`	net472	57.8μs	129ns	483ns	57.34 KB
master	`StringConcatAspectBenchmark`	net6.0	476μs	2.08μs	7.49μs	259.34 KB
master	`StringConcatAspectBenchmark`	netcoreapp3.1	565μs	2.86μs	13.4μs	277.64 KB
master	`StringConcatAspectBenchmark`	net472	410μs	2.29μs	15.9μs	278.53 KB
#7192	`StringConcatBenchmark`	net6.0	41.7μs	232ns	1.43μs	43.93 KB
#7192	`StringConcatBenchmark`	netcoreapp3.1	46.6μs	226ns	933ns	42.64 KB
#7192	`StringConcatBenchmark`	net472	56.9μs	151ns	521ns	57.34 KB
#7192	`StringConcatAspectBenchmark`	net6.0	480μs	2.3μs	8.89μs	282.11 KB
#7192	`StringConcatAspectBenchmark`	netcoreapp3.1	512μs	2.05μs	7.94μs	276.13 KB
#7192	`StringConcatAspectBenchmark`	net472	404μs	2.26μs	14.6μs	278.53 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.58μs	12.6ns	52.1ns	0	1.7 KB
master	`EnrichedLog`	netcoreapp3.1	3.49μs	7.67ns	29.7ns	0	1.7 KB
master	`EnrichedLog`	net472	4.04μs	4.43ns	17.2ns	0.243	1.64 KB
#7192	`EnrichedLog`	net6.0	2.55μs	1.67ns	6.45ns	0	1.7 KB
#7192	`EnrichedLog`	netcoreapp3.1	3.44μs	13.6ns	52.7ns	0	1.7 KB
#7192	`EnrichedLog`	net472	3.96μs	5.07ns	19.6ns	0.256	1.64 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Allocated
master	`EnrichedLog`	net6.0	124μs	291ns	1.13μs	4.31 KB
master	`EnrichedLog`	netcoreapp3.1	131μs	289ns	1.12μs	4.31 KB
master	`EnrichedLog`	net472	170μs	370ns	1.43μs	4.52 KB
#7192	`EnrichedLog`	net6.0	125μs	158ns	590ns	4.31 KB
#7192	`EnrichedLog`	netcoreapp3.1	130μs	404ns	1.51μs	4.31 KB
#7192	`EnrichedLog`	net472	171μs	393ns	1.52μs	4.52 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	5.07μs	6.23ns	23.3ns	0	2.26 KB
master	`EnrichedLog`	netcoreapp3.1	6.61μs	15.2ns	59ns	0	2.26 KB
master	`EnrichedLog`	net472	7.52μs	12.6ns	49ns	0.297	2.08 KB
#7192	`EnrichedLog`	net6.0	5.05μs	2.39ns	8.94ns	0	2.26 KB
#7192	`EnrichedLog`	netcoreapp3.1	6.6μs	16.2ns	62.9ns	0	2.26 KB
#7192	`EnrichedLog`	net472	7.43μs	8.16ns	31.6ns	0.296	2.08 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	2μs	8.79ns	34ns	0	1.2 KB
master	`SendReceive`	netcoreapp3.1	2.58μs	8ns	31ns	0	1.2 KB
master	`SendReceive`	net472	3.12μs	5.05ns	19.6ns	0.185	1.2 KB
#7192	`SendReceive`	net6.0	1.97μs	9.52ns	38.1ns	0	1.2 KB
#7192	`SendReceive`	netcoreapp3.1	2.6μs	8.41ns	32.6ns	0	1.2 KB
#7192	`SendReceive`	net472	3.17μs	2.94ns	11.4ns	0.189	1.2 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	4.16μs	4.47ns	17.3ns	0	1.58 KB
master	`EnrichedLog`	netcoreapp3.1	5.65μs	9.21ns	35.7ns	0	1.63 KB
master	`EnrichedLog`	net472	6.68μs	5.91ns	22.9ns	0.301	2.03 KB
#7192	`EnrichedLog`	net6.0	4.07μs	4.09ns	15.3ns	0	1.58 KB
#7192	`EnrichedLog`	netcoreapp3.1	5.49μs	11.9ns	46ns	0	1.63 KB
#7192	`EnrichedLog`	net472	6.45μs	10.1ns	39.3ns	0.317	2.03 KB

Benchmarks.Trace.SpanBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	743ns	3.65ns	16.3ns	0	576 B
master	`StartFinishSpan`	netcoreapp3.1	921ns	4.29ns	17.1ns	0	576 B
master	`StartFinishSpan`	net472	901ns	0.299ns	1.08ns	0.0914	578 B
master	`StartFinishScope`	net6.0	921ns	2.09ns	8.1ns	0	696 B
master	`StartFinishScope`	netcoreapp3.1	1.15μs	6.12ns	31.2ns	0	696 B
master	`StartFinishScope`	net472	1.14μs	0.759ns	2.94ns	0.101	658 B
#7192	`StartFinishSpan`	net6.0	748ns	3.38ns	13.1ns	0	576 B
#7192	`StartFinishSpan`	netcoreapp3.1	914ns	4.87ns	25.3ns	0	576 B
#7192	`StartFinishSpan`	net472	903ns	0.886ns	3.43ns	0.0905	578 B
#7192	`StartFinishScope`	net6.0	903ns	0.9ns	3.49ns	0	696 B
#7192	`StartFinishScope`	netcoreapp3.1	1.11μs	3.08ns	11.1ns	0	696 B
#7192	`StartFinishScope`	net472	1.09μs	1.03ns	3.86ns	0.103	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	1.04μs	2.28ns	8.84ns	0	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	1.41μs	6.48ns	25.1ns	0	696 B
master	`RunOnMethodBegin`	net472	1.36μs	0.788ns	3.05ns	0.102	658 B
#7192	`RunOnMethodBegin`	net6.0	1.02μs	0.845ns	3.27ns	0	696 B
#7192	`RunOnMethodBegin`	netcoreapp3.1	1.34μs	6.41ns	24.8ns	0	696 B
#7192	`RunOnMethodBegin`	net472	1.4μs	0.975ns	3.65ns	0.105	658 B

datadog-datadog-prod-us1 · 2025-08-07T21:28:11Z

⚠️ Tests

⚠️ Warnings

🧪 5 Tests failed

FlakyRetriesWithExceptionReplay from Datadog.Trace.ClrProfiler.IntegrationTests.CI.XUnitRetriesTests (Datadog)
The sample did not exit in 600000ms. Memory dump taken: False. Killing process.
FlakyRetriesWithExceptionReplay from Datadog.Trace.ClrProfiler.IntegrationTests.CI.XUnitRetriesTests (Datadog) (✨ Fix with BitsAI)
The sample did not exit in 600000ms. Memory dump taken: True. Killing process.
SubmitsTraces from Datadog.Trace.ClrProfiler.IntegrationTests.QuartzTests (Datadog)
application not found: D:\a\_work\1\s\artifacts\bin\Samples.Quartz\release_net7.0\Samples.Quartz.dll
View all

ℹ️ Info

❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 94984e4 | Docs | Was this helpful? Give us feedback!}

I need to figure out how to to replace this from what's in master and then generate it all over agian

change to match master

bouwkast

Nice!

I have a couple of suggestions to just add some more protections. Requesting changes for that, but feel free to re-request review afterwards

I vaguely remember the tests being slow for this (or was that Hangfire?) did we resolve that?

tracer/src/Datadog.Trace/Activity/ActivityListener.cs

tracer/src/Datadog.Trace/Activity/Handlers/QuartzActivityHandler.cs

tracer/test/test-applications/integrations/Samples.Quartz/Program.cs

tracer/build/PackageVersionsGeneratorDefinitions.json

…ram.cs Co-authored-by: Steven Bouwkamp <[email protected]>

Co-authored-by: Steven Bouwkamp <[email protected]>

…er.cs Co-authored-by: Steven Bouwkamp <[email protected]>

chojomok · 2025-09-25T20:30:28Z

I vaguely remember the tests being slow for this (or was that Hangfire?) did we resolve that?

@bouwkast
yeah that was for hangfire - quartz is working fine with the tests. The sample app stops once all the scheduled jobs are finished running.

Co-authored-by: Steven Bouwkamp <[email protected]>

bouwkast

Nice!

I think just one small thing to fix the sample to build again - I think it is actually left over stuff maybe for testing with OpenTelemetry? Not 100% sure though.

But after that gets fixed and the tests pass I think we are good.

tracer/test/test-applications/integrations/Samples.Quartz/Program.cs

…ram.cs Co-authored-by: Steven Bouwkamp <[email protected]>

chojomok · 2025-09-29T19:55:19Z

there were some additional issues with netcoreapp snapshots.
Since we're only updating IActivity5, netcoreapp 3.1 spans are very lack luster but that's fine for now until we get complaints to look into updating the feature.

## Summary of changes * Added a new `QuartzDiagnosticObserver` to enable OTEL instrumentation from Quartz. * v3 uses a Diagnostic Observer to make Activities. * v4 uses ActivitySource. * v4 snapshot generated from version `4.0.0-preview-20250718-1741` of the library * Integration is supported for `v3.1` of Quartz. * Requires `DD_TRACE_OTEL_ENABLED=true` for tracing to work. * Currently not available for .NET framework due to our DiagnosticObserver infrastructure not being compatible with .NET Framework. Will look into expanding support in the future. * Changes to Activity5 * Added setters to the following properties * `DisplayName` * `Kind` - this is private but with bytecode we are able to call the setter. ## Reason for change This change enables Datadog APM to trace Quartz.NET job scheduling and execution events, providing observability for background job processing. It expands instrumentation coverage for .NET applications using Quartz, addressing a gap in our supported integrations. ## Implementation details * We create an Diagnostic Observer to listen on "Quartz" so the built in OTEL Instrumentation can be used and transformed into a Datadog Span. ## Test coverage * Tests from v3.1 of Quartz to the latest version * Tests for upcoming v4 from the prerelease is available and tested on locally. ## Other details ### Activity Kind Setter * We used the same implementation as the `opentelemetry-dotnet-contrib` repo which can be found in this [code snippet](https://github.com/open-telemetry/opentelemetry-dotnet-contrib/blob/d22fdd94a54d64b311620c4c8e8f49e652f790b2/src/Shared/ActivityInstrumentationHelper.cs#L10) * since we don't have access to the classes as the contrib repo, we used bytecode to avoid casting issues.  --------- Co-authored-by: Andrew Lock <[email protected]> Co-authored-by: Steven Bouwkamp <[email protected]> Co-authored-by: Zach Montoya <[email protected]>

chojomok force-pushed the mohamammad/quarz-integration branch 2 times, most recently from c6cc14c to 47c3554 Compare August 8, 2025 18:00

chojomok added 22 commits August 8, 2025 14:39

hangfire sample app

1edb97f

add otel tracing to sample app

1cd2c26

send the otel traces to the agent

1471305

create Diagnostic Listener for Quartz

beb58a5

try duckstruct to populate args

fafaa42

update span metdata

3513730

add additional span tags to match otel

7b0c45b

fix sample app

8835199

setting up master quartz

732c136

prep prerelease quartz

7e2bc8e

creating quartz integration tests

e21d895

add quartz snapshot

06616cf

create versioned snapshots

8ece2ec

try to get packageversion as suffix

1c75b66

setup test to take variable packageversion (results in no package)

e881934

Update QuartzTestsV4.verified.txt

ae19e00

update version logic

ab5f430

generate proper package versions

ff412e7

clean build after merge

010a4ce

running generatepackage

9b83d82

I need to figure out how to to replace this from what's in master and then generate it all over agian

remove our custom instrumentation and add #nullable to listener

6800615

Update PackageVersionsLatestMajors.g.props

838618e

change to match master

chojomok force-pushed the mohamammad/quarz-integration branch from 55a2343 to 838618e Compare August 8, 2025 18:58

chojomok added 2 commits August 8, 2025 15:03

copy masters PackageVersionsLatestMinors.g.props

9041083

copy masters packageminor

bb3274a

bouwkast self-requested a review September 25, 2025 19:10

bouwkast requested changes Sep 25, 2025

View reviewed changes

tracer/src/Datadog.Trace/Activity/ActivityListener.cs Show resolved Hide resolved

tracer/src/Datadog.Trace/Activity/Handlers/QuartzActivityHandler.cs Outdated Show resolved Hide resolved

tracer/test/test-applications/integrations/Samples.Quartz/Program.cs Outdated Show resolved Hide resolved

bouwkast reviewed Sep 25, 2025

View reviewed changes

tracer/build/PackageVersionsGeneratorDefinitions.json Outdated Show resolved Hide resolved

chojomok and others added 3 commits September 25, 2025 16:27

Update tracer/test/test-applications/integrations/Samples.Quartz/Prog…

9844589

…ram.cs Co-authored-by: Steven Bouwkamp <[email protected]>

Update tracer/build/PackageVersionsGeneratorDefinitions.json

bbf8a5f

Co-authored-by: Steven Bouwkamp <[email protected]>

Update tracer/src/Datadog.Trace/Activity/Handlers/QuartzActivityHandl…

f8efc23

…er.cs Co-authored-by: Steven Bouwkamp <[email protected]>

Update tracer/src/Datadog.Trace/Activity/ActivityListener.cs

eacf82c

Co-authored-by: Steven Bouwkamp <[email protected]>

bouwkast self-requested a review September 26, 2025 19:50

bouwkast approved these changes Sep 26, 2025

View reviewed changes

tracer/test/test-applications/integrations/Samples.Quartz/Program.cs Outdated Show resolved Hide resolved

chojomok and others added 6 commits September 26, 2025 15:54

Update tracer/test/test-applications/integrations/Samples.Quartz/Prog…

ff7fb1b

…ram.cs Co-authored-by: Steven Bouwkamp <[email protected]>

add coreapp3.1 snapshots

a460bd4

Merge branch 'master' into mohamammad/quarz-integration

b898efb

Merge branch 'master' into mohamammad/quarz-integration

feca4f3

fixed typo for netcoreapp in quartz test

f290118

fix snapshot location

25d549f

chojomok and others added 6 commits September 29, 2025 16:34

update mockactivity5.displayname to have a setter

f5c76e0

package version is null for windows, add check for netcoreapp

775e67d

Merge branch 'master' into mohamammad/quarz-integration

f0c729d

fix QuartzDiagnosticObserver.cs(58,25): error DDDUCK001

7c705c4

directives .netcoreapp3x instead of just 31

dc6b2e4

fix unreachable code

94984e4

bouwkast mentioned this pull request Oct 2, 2025

Ignore the test filter for RunWindowsAzureFunctionsTests #7611

Closed

chojomok merged commit 15e1ddd into master Oct 3, 2025
156 checks passed

chojomok deleted the mohamammad/quarz-integration branch October 3, 2025 13:52

github-actions bot added this to the vNext-v3 milestone Oct 3, 2025

bouwkast added area:automatic-instrumentation Automatic instrumentation managed C# code (Datadog.Trace.ClrProfiler.Managed) type:new-feature labels Oct 6, 2025

[tracing] add OTEL quartz autoinstrumentation #7192

[tracing] add OTEL quartz autoinstrumentation #7192

Uh oh!

Conversation

chojomok commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

Activity Kind Setter

Uh oh!

dd-trace-dotnet-ci-bot bot commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Execution-Time Benchmarks Report ⏱️

Uh oh!

pr-commenter bot commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Benchmarks Report for benchmark platform 🐌

Benchmark details

Raw results

More allocations ⚠️ in #7192

Raw results

More allocations ⚠️ in #7192

Raw results

Raw results

Slower ⚠️ in #7192

Faster 🎉 in #7192

Fewer allocations 🎉 in #7192

Raw results

Raw results

More allocations ⚠️ in #7192

Fewer allocations 🎉 in #7192

Raw results

More allocations ⚠️ in #7192

Raw results

Raw results

Raw results

Raw results

Raw results

More allocations ⚠️ in #7192

Fewer allocations 🎉 in #7192

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Uh oh!

datadog-datadog-prod-us1 bot commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

bouwkast left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chojomok commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bouwkast left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

chojomok commented Sep 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

chojomok commented Jul 7, 2025 •

edited

Loading

dd-trace-dotnet-ci-bot bot commented Jul 7, 2025 •

edited

Loading

pr-commenter bot commented Jul 7, 2025 •

edited

Loading

datadog-datadog-prod-us1 bot commented Aug 7, 2025 •

edited

Loading

chojomok commented Sep 25, 2025 •

edited

Loading