Skip to content

Conversation

stefanosiano
Copy link
Member

📜 Description

Update Android benchmark configurations to include newer API levels and devices
increased number of benchmark runs to increase accuracy
increased passing threshold from 5% to 5.5%

#skip-changelog

💡 Motivation and Context

Nightly benchmarks keep failing due to missing devices from saucelabs.
Also, they rarely fail for surpassing the threshold, but that's most likely a false positive. I increased the benchmark runs to reduce this possibility. The con is that the test takes a couple minutes longer

💚 How did you test it?

📝 Checklist

  • I added tests to verify the changes.
  • No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled.
  • I updated the docs if needed.
  • I updated the wizard if needed.
  • Review from the native team if needed.
  • No breaking change or entry added to the changelog.
  • No breaking change for hybrid SDKs or communicated to hybrid SDKs.

🔮 Next steps

…nd devices

increased number of benchmark runs to increase accuracy
increased passing threshold from 5% to 5.5%
@stefanosiano stefanosiano marked this pull request as ready for review August 14, 2025 09:45
Copy link
Contributor

github-actions bot commented Aug 14, 2025

Performance metrics 🚀

  Plain With Sentry Diff
Startup time 463.50 ms 508.30 ms 44.80 ms
Size 1.58 MiB 2.10 MiB 530.95 KiB

Previous results on branch: stefanosiano/fix-benchmarks

Startup times

Revision Plain With Sentry Diff
844ef43 405.83 ms 429.35 ms 23.51 ms

App size

Revision Plain With Sentry Diff
844ef43 1.58 MiB 2.10 MiB 530.96 KiB

@stefanosiano stefanosiano enabled auto-merge (squash) August 14, 2025 11:44
@stefanosiano stefanosiano merged commit ee747ae into main Aug 14, 2025
39 of 40 checks passed
@stefanosiano stefanosiano deleted the stefanosiano/fix-benchmarks branch August 14, 2025 11:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants