Add features to facilitate regression testing #17
Description
I'd like to add some features to make it easier to compare benchmark results across commits (among other conveniences). Some items on the todo list:
- Define benchmarks in a tree (e.g. nested arrays)
- Write results to a file
- Load results from a file
- Compare results
- Tuned benchmarks which attempt to finish execution in a reasonable amount of time.
A very basic version of tuning is demoed in https://github.com/JordanMartinez/purescript-arrays/pull/1
Some of these capabilities are similar to what's available in https://github.com/hdgarrood/purescript-benchotron
There will likely be dependency issues with allowing all of these features to work with core libraries (similar to the circular dependency issues with quickcheck). Maybe we could resolve these circular dependencies by having tests and benchmarks exist as a secondary project in the same repo. If this discussion already happened, could you point me to it?
Inspired some of the features available in these two Julia packages:
https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/doc/manual.md
https://github.com/JuliaCI/PkgBenchmark.jl