go/benchmark only takes timer measurement as parameter to increase b.N

[joesonw]

What version of Go are you using (go version)?

$ go version
go version go1.14.2 darwin/amd64

Does this issue reproduce with the latest release?

I don't know. AFAIK, this is not a bug, so in theory, YES.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPRIVATE=""
GOTMPDIR=""
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/b4/t4v6zpln4rb16cvg747dz3j80000gn/T/go-build515811079=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I'm trying to benchmark a buffer library. I shall test write and read separately. But I need to write data to the buffer before read.

func BenchmarkXXX(b *testing.B) {
    buf := NewSomeBuffer()

    b.Run("Write", func (b *testing.B) {
        for i := 0; i < b.N; i++ {
            buf.Write(make([]byte, SOME_SIZE))
        }
    })

    b.Run("Write", func (b *testing.B) {
        for i := 0; i < b.N; i++ {
            buf.Write(make([]byte, SOME_SIZE))
        }

        b.ResetTimer() // Here is what went wrong

        for i := 0; i < b.N; i++ {
            buf.Read(make([]byte, SOME_SIZE))
        }
    })
}

What did you expect to see?

Hence in this scenario, read is much faster that write. It should ran similar amount of samples.

What did you see instead?

But, because b.ResetTimer undid all measurements. When go/benchmark trying to workout b.N values, It only measured the amount of time to read, It ended up timed out because of the enormous amount of data to write. Say 1,000,000 samples when test write alone, and 5,000,000 samples when test read

[davecheney]

Use b.StopTimer/b.StartTimer.

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For asking questions, see:

• Stack Overflow with questions tagged "go"

• The Go Forum, a web-based forum

• Gophers Slack, use the invite app for access. The #general channel is a good starting point.

• Go Community on Hashnode with questions and posts tagged with "go"

• The golang-nuts mailing list

• IRC channel #go-nuts on Freenode

[joesonw]

@davecheney

It produces the same issue. When it predicates the next b.N, it only takes account of the timer run duration, not the actual function run duration.

go/src/testing/benchmark.go

Line 125 in c5d7f2f

b.start = time.Now()

go/src/testing/benchmark.go

Line 135 in c5d7f2f

b.duration += time.Since(b.start)

go/src/testing/benchmark.go

Lines 296 to 324 in c5d7f2f

	d := b.benchTime.d
	for n := int64(1); !b.failed && b.duration < d && n < 1e9; {
	last := n
	// Predict required iterations.
	goalns := d.Nanoseconds()
	prevIters := int64(b.N)
	prevns := b.duration.Nanoseconds()
	if prevns <= 0 {
	// Round up, to avoid div by zero.
	prevns = 1
	}
	// Order of operations matters.
	// For very fast benchmarks, prevIters ~= prevns.
	// If you divide first, you get 0 or 1,
	// which can hide an order of magnitude in execution time.
	// So multiply first, then divide.
	n = goalns * prevIters / prevns
	// Run more iterations than we think we'll need (1.2x).
	n += n / 5
	// Don't grow too fast in case we had timing errors previously.
	n = min(n, 100*last)
	// Be sure to run at least one more than last time.
	n = max(n, last+1)
	// Don't run more than 1e9 times. (This also keeps n in int range on 32 bit platforms.)
	n = min(n, 1e9)
	b.runN(int(n))
	}
	}
	b.result = BenchmarkResult{b.N, b.duration, b.bytes, b.netAllocs, b.netBytes, b.extra}

[davecheney]

Thank you for raising this issue. This doesn't appear to be a bug in Go. We only use the issue tracker for tracking bugs and proposals. Please see the resources I mentioned above for appropriate places to ask questions. Thank you.