-
Notifications
You must be signed in to change notification settings - Fork 469
ci(system-tests): run serverless system-tests #15481
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
ci(system-tests): run serverless system-tests #15481
Conversation
|
|
3ec3616 to
243f81d
Compare
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 247 ± 3 ms. The average import time from base is: 248 ± 3 ms. The import time difference between this PR and base is: -1.5 ± 0.1 ms. Import time breakdownThe following import paths have grown:
|
Performance SLOsComparing candidate florentin.labelle/APPSEC-60128/run-serverless-system-tests (0644ba6) with baseline main (ae9b6a7) 📈 Performance Regressions (3 suites)📈 iastaspects - 118/118✅ add_aspectTime: ✅ 0.404µs (SLO: <10.000µs 📉 -96.0%) vs baseline: +1.0% Memory: ✅ 40.167MB (SLO: <41.500MB -3.2%) vs baseline: +4.9% ✅ add_inplace_aspectTime: ✅ 0.406µs (SLO: <10.000µs 📉 -95.9%) vs baseline: +1.1% Memory: ✅ 40.305MB (SLO: <41.500MB -2.9%) vs baseline: +5.0% ✅ add_inplace_noaspectTime: ✅ 0.316µs (SLO: <10.000µs 📉 -96.8%) vs baseline: -2.6% Memory: ✅ 40.265MB (SLO: <41.500MB -3.0%) vs baseline: +5.1% ✅ add_noaspectTime: ✅ 0.277µs (SLO: <10.000µs 📉 -97.2%) vs baseline: -1.2% Memory: ✅ 40.147MB (SLO: <41.500MB -3.3%) vs baseline: +4.6% ✅ bytearray_aspectTime: ✅ 1.354µs (SLO: <10.000µs 📉 -86.5%) vs baseline: -0.6% Memory: ✅ 40.265MB (SLO: <41.500MB -3.0%) vs baseline: +5.0% ✅ bytearray_extend_aspectTime: ✅ 1.498µs (SLO: <10.000µs 📉 -85.0%) vs baseline: -0.5% Memory: ✅ 40.265MB (SLO: <41.500MB -3.0%) vs baseline: +5.2% ✅ bytearray_extend_noaspectTime: ✅ 0.611µs (SLO: <10.000µs 📉 -93.9%) vs baseline: ~same Memory: ✅ 40.088MB (SLO: <41.500MB -3.4%) vs baseline: +4.3% ✅ bytearray_noaspectTime: ✅ 0.479µs (SLO: <10.000µs 📉 -95.2%) vs baseline: -1.2% Memory: ✅ 40.265MB (SLO: <41.500MB -3.0%) vs baseline: +5.1% ✅ bytes_aspectTime: ✅ 1.293µs (SLO: <10.000µs 📉 -87.1%) vs baseline: +0.8% Memory: ✅ 40.088MB (SLO: <41.500MB -3.4%) vs baseline: +4.8% ✅ bytes_noaspectTime: ✅ 0.492µs (SLO: <10.000µs 📉 -95.1%) vs baseline: -1.4% Memory: ✅ 40.225MB (SLO: <41.500MB -3.1%) vs baseline: +4.9% ✅ bytesio_aspectTime: ✅ 1.357µs (SLO: <10.000µs 📉 -86.4%) vs baseline: +2.2% Memory: ✅ 40.128MB (SLO: <41.500MB -3.3%) vs baseline: +4.6% ✅ bytesio_noaspectTime: ✅ 0.500µs (SLO: <10.000µs 📉 -95.0%) vs baseline: +0.8% Memory: ✅ 40.204MB (SLO: <41.500MB -3.1%) vs baseline: +4.3% ✅ capitalize_aspectTime: ✅ 0.741µs (SLO: <10.000µs 📉 -92.6%) vs baseline: +0.9% Memory: ✅ 40.265MB (SLO: <41.500MB -3.0%) vs baseline: +4.9% ✅ capitalize_noaspectTime: ✅ 0.434µs (SLO: <10.000µs 📉 -95.7%) vs baseline: -1.1% Memory: ✅ 40.069MB (SLO: <41.500MB -3.4%) vs baseline: +4.5% ✅ casefold_aspectTime: ✅ 0.736µs (SLO: <10.000µs 📉 -92.6%) vs baseline: +0.9% Memory: ✅ 40.167MB (SLO: <41.500MB -3.2%) vs baseline: +4.6% ✅ casefold_noaspectTime: ✅ 0.367µs (SLO: <10.000µs 📉 -96.3%) vs baseline: -1.2% Memory: ✅ 40.107MB (SLO: <41.500MB -3.4%) vs baseline: +4.8% ✅ decode_aspectTime: ✅ 0.724µs (SLO: <10.000µs 📉 -92.8%) vs baseline: -0.3% Memory: ✅ 40.245MB (SLO: <41.500MB -3.0%) vs baseline: +5.1% ✅ decode_noaspectTime: ✅ 0.415µs (SLO: <10.000µs 📉 -95.9%) vs baseline: -0.9% Memory: ✅ 40.245MB (SLO: <41.500MB -3.0%) vs baseline: +5.0% ✅ encode_aspectTime: ✅ 0.713µs (SLO: <10.000µs 📉 -92.9%) vs baseline: +0.9% Memory: ✅ 40.343MB (SLO: <41.500MB -2.8%) vs baseline: +5.0% ✅ encode_noaspectTime: ✅ 0.407µs (SLO: <10.000µs 📉 -95.9%) vs baseline: +1.7% Memory: ✅ 40.166MB (SLO: <41.500MB -3.2%) vs baseline: +4.7% ✅ format_aspectTime: ✅ 3.411µs (SLO: <10.000µs 📉 -65.9%) vs baseline: -0.2% Memory: ✅ 40.147MB (SLO: <41.500MB -3.3%) vs baseline: +4.7% ✅ format_map_aspectTime: ✅ 3.542µs (SLO: <10.000µs 📉 -64.6%) vs baseline: -0.9% Memory: ✅ 40.265MB (SLO: <41.500MB -3.0%) vs baseline: +5.0% ✅ format_map_noaspectTime: ✅ 0.768µs (SLO: <10.000µs 📉 -92.3%) vs baseline: -0.5% Memory: ✅ 40.206MB (SLO: <41.500MB -3.1%) vs baseline: +4.7% ✅ format_noaspectTime: ✅ 0.594µs (SLO: <10.000µs 📉 -94.1%) vs baseline: -1.1% Memory: ✅ 40.126MB (SLO: <41.500MB -3.3%) vs baseline: +4.5% ✅ index_aspectTime: ✅ 0.362µs (SLO: <10.000µs 📉 -96.4%) vs baseline: ~same Memory: ✅ 40.088MB (SLO: <41.500MB -3.4%) vs baseline: +4.9% ✅ index_noaspectTime: ✅ 0.278µs (SLO: <10.000µs 📉 -97.2%) vs baseline: -0.9% Memory: ✅ 40.186MB (SLO: <41.500MB -3.2%) vs baseline: +4.8% ✅ join_aspectTime: ✅ 1.344µs (SLO: <10.000µs 📉 -86.6%) vs baseline: +1.4% Memory: ✅ 40.226MB (SLO: <41.500MB -3.1%) vs baseline: +5.0% ✅ join_noaspectTime: ✅ 0.489µs (SLO: <10.000µs 📉 -95.1%) vs baseline: -1.4% Memory: ✅ 40.206MB (SLO: <41.500MB -3.1%) vs baseline: +4.8% ✅ ljust_aspectTime: ✅ 2.898µs (SLO: <20.000µs 📉 -85.5%) vs baseline: 📈 +12.3% Memory: ✅ 40.146MB (SLO: <41.500MB -3.3%) vs baseline: +4.3% ✅ ljust_noaspectTime: ✅ 0.403µs (SLO: <10.000µs 📉 -96.0%) vs baseline: -0.8% Memory: ✅ 40.205MB (SLO: <41.500MB -3.1%) vs baseline: +5.0% ✅ lower_aspectTime: ✅ 2.308µs (SLO: <10.000µs 📉 -76.9%) vs baseline: +2.3% Memory: ✅ 40.107MB (SLO: <41.500MB -3.4%) vs baseline: +4.4% ✅ lower_noaspectTime: ✅ 0.368µs (SLO: <10.000µs 📉 -96.3%) vs baseline: -0.2% Memory: ✅ 40.244MB (SLO: <41.500MB -3.0%) vs baseline: +4.8% ✅ lstrip_aspectTime: ✅ 2.270µs (SLO: <20.000µs 📉 -88.6%) vs baseline: +1.0% Memory: ✅ 40.186MB (SLO: <41.500MB -3.2%) vs baseline: +5.0% ✅ lstrip_noaspectTime: ✅ 0.382µs (SLO: <10.000µs 📉 -96.2%) vs baseline: +0.2% Memory: ✅ 40.186MB (SLO: <41.500MB -3.2%) vs baseline: +4.8% ✅ modulo_aspectTime: ✅ 1.047µs (SLO: <10.000µs 📉 -89.5%) vs baseline: +4.8% Memory: ✅ 40.147MB (SLO: <41.500MB -3.3%) vs baseline: +4.7% ✅ modulo_aspect_for_bytearray_bytearrayTime: ✅ 1.550µs (SLO: <10.000µs 📉 -84.5%) vs baseline: -1.3% Memory: ✅ 40.187MB (SLO: <41.500MB -3.2%) vs baseline: +5.0% ✅ modulo_aspect_for_bytesTime: ✅ 0.976µs (SLO: <10.000µs 📉 -90.2%) vs baseline: -0.5% Memory: ✅ 40.029MB (SLO: <41.500MB -3.5%) vs baseline: +4.4% ✅ modulo_aspect_for_bytes_bytearrayTime: ✅ 1.254µs (SLO: <10.000µs 📉 -87.5%) vs baseline: +2.1% Memory: ✅ 40.128MB (SLO: <41.500MB -3.3%) vs baseline: +4.6% ✅ modulo_noaspectTime: ✅ 0.633µs (SLO: <10.000µs 📉 -93.7%) vs baseline: +0.6% Memory: ✅ 40.127MB (SLO: <41.500MB -3.3%) vs baseline: +4.4% ✅ replace_aspectTime: ✅ 4.878µs (SLO: <10.000µs 📉 -51.2%) vs baseline: -1.2% Memory: ✅ 40.186MB (SLO: <41.500MB -3.2%) vs baseline: +5.0% ✅ replace_noaspectTime: ✅ 0.464µs (SLO: <10.000µs 📉 -95.4%) vs baseline: +0.9% Memory: ✅ 40.165MB (SLO: <41.500MB -3.2%) vs baseline: +4.9% ✅ repr_aspectTime: ✅ 0.911µs (SLO: <10.000µs 📉 -90.9%) vs baseline: +0.5% Memory: ✅ 40.265MB (SLO: <41.500MB -3.0%) vs baseline: +4.9% ✅ repr_noaspectTime: ✅ 0.418µs (SLO: <10.000µs 📉 -95.8%) vs baseline: -0.7% Memory: ✅ 40.146MB (SLO: <41.500MB -3.3%) vs baseline: +4.6% ✅ rstrip_aspectTime: ✅ 1.953µs (SLO: <20.000µs 📉 -90.2%) vs baseline: +0.9% Memory: ✅ 40.343MB (SLO: <41.500MB -2.8%) vs baseline: +5.3% ✅ rstrip_noaspectTime: ✅ 0.383µs (SLO: <10.000µs 📉 -96.2%) vs baseline: +1.6% Memory: ✅ 40.126MB (SLO: <41.500MB -3.3%) vs baseline: +4.5% ✅ slice_aspectTime: ✅ 0.494µs (SLO: <10.000µs 📉 -95.1%) vs baseline: -0.2% Memory: ✅ 40.167MB (SLO: <41.500MB -3.2%) vs baseline: +5.1% ✅ slice_noaspectTime: ✅ 0.445µs (SLO: <10.000µs 📉 -95.6%) vs baseline: +0.2% Memory: ✅ 40.225MB (SLO: <41.500MB -3.1%) vs baseline: +4.7% ✅ stringio_aspectTime: ✅ 1.538µs (SLO: <10.000µs 📉 -84.6%) vs baseline: +0.8% Memory: ✅ 40.284MB (SLO: <41.500MB -2.9%) vs baseline: +5.0% ✅ stringio_noaspectTime: ✅ 0.719µs (SLO: <10.000µs 📉 -92.8%) vs baseline: -0.2% Memory: ✅ 40.264MB (SLO: <41.500MB -3.0%) vs baseline: +4.7% ✅ strip_aspectTime: ✅ 2.238µs (SLO: <20.000µs 📉 -88.8%) vs baseline: +0.2% Memory: ✅ 40.146MB (SLO: <41.500MB -3.3%) vs baseline: +4.3% ✅ strip_noaspectTime: ✅ 0.385µs (SLO: <10.000µs 📉 -96.2%) vs baseline: -0.7% Memory: ✅ 40.068MB (SLO: <41.500MB -3.5%) vs baseline: +4.6% ✅ swapcase_aspectTime: ✅ 2.515µs (SLO: <10.000µs 📉 -74.9%) vs baseline: +3.0% Memory: ✅ 40.067MB (SLO: <41.500MB -3.5%) vs baseline: +3.9% ✅ swapcase_noaspectTime: ✅ 0.531µs (SLO: <10.000µs 📉 -94.7%) vs baseline: -1.9% Memory: ✅ 40.324MB (SLO: <41.500MB -2.8%) vs baseline: +5.1% ✅ title_aspectTime: ✅ 2.635µs (SLO: <10.000µs 📉 -73.6%) vs baseline: 📈 +10.1% Memory: ✅ 40.185MB (SLO: <41.500MB -3.2%) vs baseline: +4.1% ✅ title_noaspectTime: ✅ 0.503µs (SLO: <10.000µs 📉 -95.0%) vs baseline: -0.2% Memory: ✅ 40.028MB (SLO: <41.500MB -3.5%) vs baseline: +4.4% ✅ translate_aspectTime: ✅ 3.332µs (SLO: <10.000µs 📉 -66.7%) vs baseline: +0.8% Memory: ✅ 40.225MB (SLO: <41.500MB -3.1%) vs baseline: +4.5% ✅ translate_noaspectTime: ✅ 1.039µs (SLO: <10.000µs 📉 -89.6%) vs baseline: -1.0% Memory: ✅ 40.246MB (SLO: <41.500MB -3.0%) vs baseline: +4.9% ✅ upper_aspectTime: ✅ 2.300µs (SLO: <10.000µs 📉 -77.0%) vs baseline: +1.9% Memory: ✅ 40.106MB (SLO: <41.500MB -3.4%) vs baseline: +4.4% ✅ upper_noaspectTime: ✅ 0.370µs (SLO: <10.000µs 📉 -96.3%) vs baseline: +1.0% Memory: ✅ 40.284MB (SLO: <41.500MB -2.9%) vs baseline: +4.9% 📈 iastaspectsospath - 24/24✅ ospathbasename_aspectTime: ✅ 5.162µs (SLO: <10.000µs 📉 -48.4%) vs baseline: 📈 +25.4% Memory: ✅ 40.265MB (SLO: <41.000MB 🟡 -1.8%) vs baseline: +5.1% ✅ ospathbasename_noaspectTime: ✅ 1.085µs (SLO: <10.000µs 📉 -89.2%) vs baseline: ~same Memory: ✅ 40.108MB (SLO: <41.000MB -2.2%) vs baseline: +4.4% ✅ ospathjoin_aspectTime: ✅ 6.163µs (SLO: <10.000µs 📉 -38.4%) vs baseline: +0.1% Memory: ✅ 40.206MB (SLO: <41.000MB 🟡 -1.9%) vs baseline: +4.7% ✅ ospathjoin_noaspectTime: ✅ 2.270µs (SLO: <10.000µs 📉 -77.3%) vs baseline: -0.4% Memory: ✅ 40.147MB (SLO: <41.000MB -2.1%) vs baseline: +4.7% ✅ ospathnormcase_aspectTime: ✅ 3.427µs (SLO: <10.000µs 📉 -65.7%) vs baseline: +0.3% Memory: ✅ 40.226MB (SLO: <41.000MB 🟡 -1.9%) vs baseline: +4.8% ✅ ospathnormcase_noaspectTime: ✅ 0.572µs (SLO: <10.000µs 📉 -94.3%) vs baseline: -1.2% Memory: ✅ 40.265MB (SLO: <41.000MB 🟡 -1.8%) vs baseline: +5.0% ✅ ospathsplit_aspectTime: ✅ 4.759µs (SLO: <10.000µs 📉 -52.4%) vs baseline: ~same Memory: ✅ 40.206MB (SLO: <41.000MB 🟡 -1.9%) vs baseline: +4.9% ✅ ospathsplit_noaspectTime: ✅ 1.597µs (SLO: <10.000µs 📉 -84.0%) vs baseline: -0.7% Memory: ✅ 40.265MB (SLO: <41.000MB 🟡 -1.8%) vs baseline: +4.8% ✅ ospathsplitdrive_aspectTime: ✅ 3.663µs (SLO: <10.000µs 📉 -63.4%) vs baseline: +0.5% Memory: ✅ 40.364MB (SLO: <41.000MB 🟡 -1.6%) vs baseline: +5.2% ✅ ospathsplitdrive_noaspectTime: ✅ 0.697µs (SLO: <10.000µs 📉 -93.0%) vs baseline: -0.4% Memory: ✅ 40.265MB (SLO: <41.000MB 🟡 -1.8%) vs baseline: +4.9% ✅ ospathsplitext_aspectTime: ✅ 4.541µs (SLO: <10.000µs 📉 -54.6%) vs baseline: +0.3% Memory: ✅ 40.206MB (SLO: <41.000MB 🟡 -1.9%) vs baseline: +4.7% ✅ ospathsplitext_noaspectTime: ✅ 1.379µs (SLO: <10.000µs 📉 -86.2%) vs baseline: -1.1% Memory: ✅ 40.187MB (SLO: <41.000MB 🟡 -2.0%) vs baseline: +4.7% 📈 telemetryaddmetric - 29/29✅ 1-count-metric-1-timesTime: ✅ 3.378µs (SLO: <20.000µs 📉 -83.1%) vs baseline: 📈 +15.4% Memory: ✅ 34.760MB (SLO: <35.500MB -2.1%) vs baseline: +4.9% ✅ 1-count-metrics-100-timesTime: ✅ 200.797µs (SLO: <220.000µs -8.7%) vs baseline: -1.2% Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.9% ✅ 1-distribution-metric-1-timesTime: ✅ 3.307µs (SLO: <20.000µs 📉 -83.5%) vs baseline: +0.4% Memory: ✅ 34.780MB (SLO: <35.500MB -2.0%) vs baseline: +4.9% ✅ 1-distribution-metrics-100-timesTime: ✅ 216.696µs (SLO: <230.000µs -5.8%) vs baseline: ~same Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +4.9% ✅ 1-gauge-metric-1-timesTime: ✅ 2.160µs (SLO: <20.000µs 📉 -89.2%) vs baseline: -1.1% Memory: ✅ 34.918MB (SLO: <35.500MB 🟡 -1.6%) vs baseline: +5.2% ✅ 1-gauge-metrics-100-timesTime: ✅ 135.684µs (SLO: <150.000µs -9.5%) vs baseline: -0.3% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ 1-rate-metric-1-timesTime: ✅ 3.083µs (SLO: <20.000µs 📉 -84.6%) vs baseline: -0.1% Memory: ✅ 34.741MB (SLO: <35.500MB -2.1%) vs baseline: +4.9% ✅ 1-rate-metrics-100-timesTime: ✅ 213.331µs (SLO: <250.000µs 📉 -14.7%) vs baseline: -0.8% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.1% ✅ 100-count-metrics-100-timesTime: ✅ 20.273ms (SLO: <22.000ms -7.9%) vs baseline: ~same Memory: ✅ 34.741MB (SLO: <35.500MB -2.1%) vs baseline: +4.7% ✅ 100-distribution-metrics-100-timesTime: ✅ 2.273ms (SLO: <2.300ms 🟡 -1.2%) vs baseline: ~same ✅ 100-gauge-metrics-100-timesTime: ✅ 1.408ms (SLO: <1.550ms -9.2%) vs baseline: +0.2% Memory: ✅ 34.760MB (SLO: <35.500MB -2.1%) vs baseline: +4.9% ✅ 100-rate-metrics-100-timesTime: ✅ 2.234ms (SLO: <2.550ms 📉 -12.4%) vs baseline: +0.8% Memory: ✅ 34.819MB (SLO: <35.500MB 🟡 -1.9%) vs baseline: +5.0% ✅ flush-1-metricTime: ✅ 4.531µs (SLO: <20.000µs 📉 -77.3%) vs baseline: -1.1% Memory: ✅ 35.114MB (SLO: <35.500MB 🟡 -1.1%) vs baseline: +4.9% ✅ flush-100-metricsTime: ✅ 175.051µs (SLO: <250.000µs 📉 -30.0%) vs baseline: +0.7% Memory: ✅ 35.154MB (SLO: <35.500MB 🟡 -1.0%) vs baseline: +5.0% ✅ flush-1000-metricsTime: ✅ 2.191ms (SLO: <2.500ms 📉 -12.4%) vs baseline: +0.2% Memory: ✅ 36.038MB (SLO: <36.500MB 🟡 -1.3%) vs baseline: +5.1% 🟡 Near SLO Breach (15 suites)🟡 coreapiscenario - 10/10 (1 unstable)
|
20d8758 to
e61562d
Compare
Description
Run datadog-lambda-python system tests in the pipeline of dd-trace-py, to catch changes that would break features inside datadog_lambda early.
Testing
Risks
Additional Notes