Add performance tests

[vr009]

Added benchmarks of large Select and Replace.
Added a new target in Makefile for running benchmark tests.
Added a new step in ci for running benchmark.
Added description to the README for how to run benchmark tests.

Added calls of b.ResetTimer() to all Benchmark tests before the bench loop.
That call clears all counters of allocs, timers etc. in internal counter.
The preparation before bench loop affected perf results.

Fixes #122

[ligurio]

• BENCH_REFERENCE_REPO - I would rename it to GO_TARANTOOL_URL for clarity
• if .. then .. fi is bulky, I would replace it with operator for test condition and command itself
• you do not need to change dir just for making a git clone
• Cloning the repository into ${BENCH_PATH} for generating benchmark reference file - target just clones a repository, we don't know here what for we do it. I would remove "for generating benchmark reference file"

Please consider my changes:

diff --git a/Makefile b/Makefile
index ca1fb95..40ff068 100644
--- a/Makefile
+++ b/Makefile
@@ -9,8 +9,9 @@ TEST_PATH ?= ${PROJECT_DIR}/...
 BENCH_FILE := ${BENCH_PATH}/bench.txt
 REFERENCE_FILE := ${BENCH_PATH}/reference.txt
 BENCH_FILES := ${REFERENCE_FILE} ${BENCH_FILE}
-BENCH_REFERENCE_REPO := ${BENCH_PATH}/go-tarantool
 BENCH_OPTIONS := -bench=. -run=^Benchmark -benchmem -benchtime=${DURATION} -count=${COUNT}
+GO_TARANTOOL_URL := https://github.com/tarantool/go-tarantool
+GO_TARANTOOL_DIR := ${BENCH_PATH}/go-tarantool
 
 .PHONY: clean
 clean:
@@ -83,11 +84,11 @@ ${BENCH_FILE} bench:
 		| tee ${BENCH_FILE}
 	benchstat ${BENCH_FILE}
 
-${BENCH_REFERENCE_REPO}:
-	@echo "Cloning the repository into ${BENCH_PATH} for generating benchmark reference file"
-	cd ${BENCH_PATH} && if ! [ -e go-tarantool ]; then git clone https://github.com/tarantool/go-tarantool.git; fi
+${GO_TARANTOOL_DIR}:
+	@echo "Cloning the repository into ${GO_TARANTOOL_DIR}"
+	[ ! -e ${GO_TARANTOOL_DIR} ] && git clone ${GO_TARANTOOL_URL} ${GO_TARANTOOL_DIR}
 
-${REFERENCE_FILE}: ${BENCH_REFERENCE_REPO}
+${REFERENCE_FILE}: ${GO_TARANTOOL_DIR}
 	@echo "Running benchmark tests from master for using results in bench-diff target"
 	cd ${BENCH_PATH}/go-tarantool && git pull && go test ${BENCH_PATH}/go-tarantool/... ${BENCH_OPTIONS} 2>&1 \
 		| tee ${REFERENCE_FILE}

[vr009]

Fixed. Thank you for review!

[ligurio]

Thanks for changes!

LGTM after processing my comment about target for cloning repository.

[Totktonada]

I played around the benchmarking toolkit and have some feedback.

Documentation

I would add a sections regarding benchmarking. I mean, let's add a header for the section to easily spot it visually.

I would start from a quickstart (make bench-deps, make bench, make bench-diff) and then would describe options and show examples how to set them.

I would also add some recommendations how to achieve stable results. I briefly written my suggestions below.

Before any judgments, verify whether results are stable on given host and how large the noise. Run make bench-diff without changes and look on the report. Several times.

There are suggestions how to achieve best results:

• Close all background applications, especially web browser. Look at top (htop, atop, ...) and if something bubbles there, close it.
• Disable cron daemon.
• Disable TurboBoost and set fixed frequency.

  • If you're using intel_pstate frequency driver (it is usually default):

    Disable TurboBoost:

    $ echo 0 > /sys/devices/system/cpu/intel_pstate/no_turbo

    Set fixed frequency: not sure it is possible.

  • If you're using acpi-cpufreq driver:

    Ensure you actually don't use intel_pstate:

    $ grep -o 'intel_pstate=\w\+' /proc/cmdline
    intel_pstate=disable
    $ cpupower -c all frequency-info | grep driver:
      driver: acpi-cpufreq
      <...>

    Disable TurboBoost:

    $ echo 0 > /sys/devices/system/cpu/cpufreq/boost

    Set fixed frequency:

    $ cpupower -c all frequency-set -g userspace
    $ cpupower -c all frequency-set -f 1.80GHz # adjust for your CPU

Also I would note that benchstat should be in PATH. I need the following command to start using the Makefile:

$ export PATH="/home/${USER}/go/bin:${PATH}"

Usability

I met several problems.

1. I had received an error, because my current directory contains symlinks.

   $ pwd
   /home/alex/p/t/go-tarantool
   $ make bench
   Running benchmark tests from the current branch
   go test /home/alex/projects/tarantool-meta/go-tarantool/... -bench=. -run=^Benchmark -benchmem -benchtime=3s -count=5 2>&1 \
    | tee /home/alex/projects/tarantool-meta/go-tarantool/bench/bench.txt
   tee: /home/alex/projects/tarantool-meta/go-tarantool/bench/bench.txt: No such file or directory
   pattern /home/alex/projects/tarantool-meta/go-tarantool/...: directory /home/alex/projects/tarantool-meta/go-tarantool is outside module root (/home/alex/p/t/go-tarantool)
   make: *** [Makefile:84: bench] Error 1
   $ cd $(realpath .)
   $ pwd
   /home/alex/projects/tarantool-meta/go-tarantool
   $ make bench # now it works

2. I had received an error with custom BENCH_PATH with make bench:

   $ make bench BENCH_PATH=bench29
   Running benchmark tests from the current branch
   go test /home/alex/projects/tarantool-meta/go-tarantool/... -bench=. -run=^Benchmark -benchmem -benchtime=3s -count=5 2>&1 \
    | tee bench29/bench.txt
   tee: bench29/bench.txt: No such file or directory
   <...>

   I expect that the directory will be created on demand.

3. Custom BENCH_PATH with make bench-diff:

   $ make bench-diff BENCH_PATH=bench29
   <...>
   tee: bench29/reference.txt: No such file or directory
   go: warning: "bench29/go-tarantool/..." matched no packages
   no packages to test
   make: *** [Makefile:94: bench29/reference.txt] Error 1

   However the following works:

   $ make bench-diff BENCH_PATH=$(realpath bench29)

Benchmarking results

Before I disbled TurboBoost I received stable degradation for on make bench-diff disregarding reference/HEAD order. I mean that the second result was always worse. It is why I leave the recommendations above and suggest to include something like this into the document.

But when I disabled TurboBoost, I received 1-2% noise, which looks suitable for comparisons. There are rare spikes, so we anyway should run the comparison several times to ensure that results are good. It is okay for the manual tool.

My resolution is that the tooling is suitable for manual comparisons.

Future thoughts

It would be nice to look around unit-test-like benchmarks / microbenchmarks, which would test parts of the connector without actual interation with tarantool. I think they will be more sensitive. I would not make it a separate task, but would experiment around in the scope of some relevant work.

[Totktonada]

The changes are okay for me in general.

I propose to look on my feedback (summarized above), fix simple problems, leave warnings in CONTRIBUTING.md on hard problems and land it to master. Feel free to proceed without further reviews.