Solution Execution Time Comparison

[phil-the-frog]

I suggest add a small statistic indication of the solutions' avg execute speed in the "Solutions" menu.
And when you click the "Compare with your solution" button, it will compare your solution and the other solutions execute speeds. I feel this will make room for improvement in people ideas of optimization.

┆Issue is synchronized with this Clickup by Unito

[10XL]

You can fork a solution(which creates a kumite) and compare the performance of arbitrary code/solutions. Here's an example in ruby(the output isn't displaying correctly, but you can fork and run the tests yourself.)

[nais86]

@10XL,
you can do that yes, but maybe the suggestion is about showing the execution time, and maybe also the memory used, in every posted solution; a Performance Rating, if you will.
Clever solutions, or one-line solutions, would then have a clear Performance Rating that can be compared to other solutions.

This would add a new dimension to the Codewars platform.
Experienced users could then submit their solutions aimed at improving certain performance metrics (execution time, memory used, etc).
Which could lead to more in-depth discussions, learning, etc.

[phil-the-frog]

@esoNais
Yes, you hit it right on the head. Maybe performance statistics would be useful too. Like, documenting average execution time/memory used and then comparing your solution to the average for that kata would be useful as well.

[Voileexperiments]

Of course, then the problem is that you need to run the code a lot of times to get good results, which are often over 12 seconds.

(Statistics 101: To get a ±0.1% uncertainty, you need to run the code 10^6 times)

Also, how is runtime defined? Best case? Average case? Worse case? Mean or Median or quartiles? And how do you define an "average case"? What does a "random input" even mean for some problems?

[phil-the-frog]

@Voileexperiments
Is 0.1% uncertainty really necessary? I'd be perfectly comfortable with even +-10%, only because it's impractical to implement the means to reach 0.1%. I personally only want the Mean of the solution times. If anyone wants anything else they will have to consider the issues of implementing such a solution.

[kevin-lindsay-1]

I second this idea; I think that "scoring" could be implemented similarly to how the Zachtronics games are scored. In the Zachtronics games, generally there are 3 scores that you shoot for, those being execution time/cycles, SLOC, and "resources" (in the games there is typically a resource management system of some kind).

Their programming games are:
TIS-100
Shenzhen I/O
EXAPUNKS

These games are great because the scoring system pushes you to compete against yourself, and I've had a few instances where the scores I've met for certain problems aren't even on the chart because so few people have optimized as heavily, which was exhilarating.

[Olangu]

Just want to second this.

It's horrible to try to weigh your own solutions with other peoples solutions based only on how "beautiful" the code is.

In most cases you need to optimize either on footprint, memory usage or speed. We need those statistics on the solutions! Especially when learning a new language you can't really have a gut feeling for the performance on different ways to solve a problem.

BTW, Is there any way to check the full tests that run on the problems?

[Steffan153]

@Olangu

BTW, Is there any way to check the full tests that run on the problems?

Yes. You can click fork and view the tests, open a new translation and check the tests, or click "view test cases" on the discourse page.

[heribertolugo]

i found this page because i was searching for this feature.. @Olangu is exactly correct. sometimes clever code, a perl looking one liner might perform best, but may also perform worst. using built in library tools (such as .net linq) may hurt or improve performance as well.

performance (speed) rating and memory usage rating should be auto calculated for each solution. and should be completely separate from users clicking like on "clever" or "best practice". many users may not even have a solid understanding of what "best practice" or "clever" would actually mean in real world scenarios. these 2 are relative to the audience, rather than to actual standards or veterans.

having performance (speed) rating and memory usage rating would be a much more accurate way to gauge ones owns code and see others that ranked higher. actually giving users a tool to use for improvement in both coding and understanding.

i created my own performance test on my local vs 10 other solutions. i was actually surprised on how well some did. i used a sample data set of 25 and ran each set in each iteration. i found that 500 or less usually gave same similar. 1000 to 10000 seemed to produce similar. anything over that, up to 500000 seemed to produce a third set of results. while the ones which performed great at over 1000, performed good below that - there were some that out performed by magnitudes but snailed over 1000.

so i would think a median of a similar approach would be reliable. or even just a flat number like 100000 iterations for different sample data sets.

nobody is expecting physics equations for performance accuracy. just some kind of benchmark for assessing a fair performance ranking.

[barakplasma]

In 2016, this was Codewars's ( @jhoffner ) response to a similar request: #651 (comment)

( TL:DR; "The result are just too inconsistent to rely on and if its not reliable, there is no point to having it." )

I agree with @Philsop and @heribertolugo that performance results don't need to be very accurate to be useful as a way of sorting unrated solutions.

[FArekkusu]

performance results don't need to be very accurate to be useful as a way of sorting unrated solutions

@barakplasma the execution time is not just "inaccurate" or "inconsistent", it's completely unrepresentative of the real solution efficiency. A more or less accurate way to tell which one is faster is by checking how far they can go without timing out which would mean manually editing the test cases to run the function thousands of times, or to generate huge inputs (whichever approach makes sense for that particular case). Remaining fair when all the inputs for different solutions are different (due to randomness) is impossible, moreover sometimes the runtime will be jumping like crazy between runs because a solution can be optimized to efficiently handle one sort of data, but lose by a big margin for a different sort of data.

Everybody wants to have "execution time measurement" but nobody knows what an impossible feat it is to implement it in a remotely sensible way.

[NathanHazout]

Hi,
I can't claim to know the inner workings of the current architecture of CodeWars, however I just want to point out that the described behavior is what LeetCode manages to do pretty well.
In LeetCode as well problems and tests are user-provided, just like in CodeWars.

Below is a screenshot of what it looks like once you've submitted the solution.
Notice the "You are here!", and you can also click on any of the bar in the chart to see a sample solution that received that score.