Skip to content

Conversation

@filzrev
Copy link
Contributor

@filzrev filzrev commented Jun 20, 2025

This PR update package dependencies and update benchmark relating code.
to test System.CommandLine 2.0.0-beta5.

Before this PR merged

Method Mean Error StdDev Ratio RatioSD Allocated Alloc Ratio
'ConsoleAppFramework v5' 1.731 ms 0.1185 ms 0.1365 ms 1.01 0.11 400 B 1.00
CliFx 26.083 ms 0.4402 ms 0.5069 ms 15.15 1.13 62000 B 155.00
System.CommandLine 32.463 ms 3.0543 ms 3.5173 ms 18.86 2.42 18720 B 46.80
Cocona.Lite 49.657 ms 1.5002 ms 1.7277 ms 28.85 2.30 60072 B 150.18
Spectre.Console.Cli 60.221 ms 2.9316 ms 3.3760 ms 34.98 3.16 66016 B 165.04
Cocona 143.379 ms 5.7622 ms 6.6357 ms 83.29 7.08 466880 B 1,167.20

After this PR merged

Method Mean Error StdDev Ratio RatioSD Completed Work Items Allocated Alloc Ratio
'ConsoleAppFramework v5' 1.737 ms 0.1390 ms 0.1601 ms 1.01 0.12 - - NA
System.CommandLine 25.508 ms 1.6135 ms 1.8581 ms 14.80 1.63 - 11008 B NA
CliFx 27.410 ms 1.9893 ms 2.2908 ms 15.90 1.87 - 60712 B NA
Cocona.Lite 52.075 ms 1.9009 ms 2.1890 ms 30.21 2.83 1.0000 59384 B NA
Spectre.Console.Cli 62.090 ms 2.4840 ms 2.8606 ms 36.02 3.44 - 65760 B NA
Cocona 148.965 ms 7.1488 ms 8.2326 ms 86.43 8.65 3.0000 466128 B NA

Note: Completed Work Items columns are displayed. Because ThreadingDiagnoser is added..

@@ -1,23 +1,15 @@
// This benchmark project is based on CliFx.Benchmarks.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.editorconfig is recently modified to use utf8-bom.
It need to apply this rule to all cs files by dotnet format command on another PR.

@filzrev
Copy link
Contributor Author

filzrev commented Jun 20, 2025

I've compared to README's benchmark results that is measured on 2024/5/31.
But result is apparently different for System.CommandLine

I don't know what cause this performance differences.

I also tried v5.0.0 version benchmarks but results are almost same.

It's reproduced by using v5.0.0 tag version of benchmark code. (that using System.CommandLine 2.0.0-beta1.20071.2)

v5.0.0 benchmark results

Method Mean Error Ratio Allocated Alloc Ratio
'ConsoleAppFramework v5' 1.173 ms NA 1.00 400 B 1.00
CliFx 26.977 ms NA 23.00 62000 B 155.00
Cocona.Lite 49.620 ms NA 42.30 60072 B 150.18
Spectre.Console.Cli 62.151 ms NA 52.98 66016 B 165.04
Cocona 147.717 ms NA 125.92 420520 B 1,051.30
System.CommandLine 257.581 ms NA 219.57 40888 B 102.22

@neuecc
Copy link
Member

neuecc commented Jun 23, 2025

Thank you, checking the latest is also important from a fairness perspective.

@neuecc neuecc merged commit 9ced84e into Cysharp:master Jun 23, 2025
1 check passed
@filzrev
Copy link
Contributor Author

filzrev commented Jun 23, 2025

Thanks for merging this PR.

checking the latest is also important from a fairness perspective.

If possible, please update README.md benchmark result images also.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants