runtime: allocated pages below zero on openbsd/riscv64

[4a6f656c]

#!watchflakes
post <- log ~ `fatal error: allocated pages below zero` && goos == "openbsd" && goarch == "riscv64"

The openbsd/riscv64 port is regularly failing with errors like:

runtime: inUse=0 npages=4
fatal error: allocated pages below zero?

https://build.golang.org/log/bb95405e34149982611770caf632d58e9bc50ff4
https://build.golang.org/log/e5fb11bcae4d4dab31c13f7e271852e39ae5a6df

This seems particularly reproducible (although still intermittent) via the TestArenaCollision test:

$ ../bin/go tool dist test -run runtime:cpu124                                          
                                                                                                                  
##### Test execution environment.                                                                                 
# GOARCH: riscv64                                                                                                 
# CPU:                                                                                                            
# GOOS: openbsd                                                                                                   
# OS Version: OpenBSD 7.4 OpenBSD 7.4 (GENERIC.MP) #423: Tue Oct  3 07:33:24 MDT 2023     deraadt@riscv64.openbsd.org:/usr/src/sys/arch/riscv64/compile/GENERIC.MP  riscv64
                                                                                                                  
##### GOMAXPROCS=2 runtime -cpu=1,2,4 -quick                                                                      
--- FAIL: TestArenaCollision (0.26s)                                                                              
    malloc_test.go:284: === RUN   TestArenaCollision                                                              
            malloc_test.go:301: reserved [0x60000000, 0x64000000)                                                 
            malloc_test.go:301: reserved [0x54000000, 0x58000000)                                                 
            malloc_test.go:301: reserved [0xa0000000, 0xa4000000)
            malloc_test.go:301: reserved [0x70000000, 0x74000000)                                                 
            malloc_test.go:301: reserved [0xc8000000, 0xcc000000)                                                 
        runtime: inUse=0 npages=1                                                                                 
        fatal error: allocated pages below zero?

inUse always appears to be zero, while npages is some small positive value.

It would seem that this is specific to openbsd/riscv64 and possibly even a kernel/mmap related problem.

/cc @mknyszek any pointers in tracking this down would be greatly appreciated.

[mknyszek]

Hm, unfortunately this doesn't seem like it would be a kernel/mmap problem. This is a check that's failing when one or more pages are freed back to the page heap. What's happening is that some of the bookkeeping maintained per 4 MiB aligned address range indicates that the region is totally free, so it doesn't make sense to try and free memory back to it. I think it implies some kind of corruption of the data structure, or a data race. It could also mean that when the check isn't firing, some pages are getting allocated twice (which could lead to other memory corruption symptoms).

It may just be reproducing in TestArenaCollision because it allocates a whole bunch of memory and likely triggers a GC (causing other memory to be freed, since all the memory it allocates stays live indefinitely).

If this is straightforward to reproduce, my first thought would be to pepper debuglog calls around the various allocs and frees to the page heap and make sure nothing is being allocated (and thus later possibly freed) twice.

If you haven't used debuglog before, it's a very handy low-overhead tool for identifying what actually happened in the runtime. Because it's low overhead, it maintains a low chance of perturbing bugs involving race conditions and tends to reproduce real thread orderings. You can enable it by passing -tags debuglog to go test or go build. You use it in the runtime by calling dlog().s("string example").u64(5).end() from almost anywhere in the runtime. It'll print the log when the program crashes. (If it's not doing so, find some reasonable place to call printDebugLog.) The only place you might run into trouble is trying to call it from signal handlers. It's safe, but you may run out of nosplit stack space. Don't forget to call .end()!

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `fatal error: allocated pages below zero` && goos == "openbsd" && goarch == "riscv64"

2023-09-28 17:55 openbsd-riscv64-jsing sys@2964e1e4 go@3a69dcdc go/types [build] (log)

runtime: inUse=0 npages=3
fatal error: allocated pages below zero?

runtime stack:
runtime.throw({0x91b6fa?, 0x0?})
	runtime/panic.go:1018 +0x44 fp=0x123f5f310 sp=0x123f5f2e8 pc=0x53b6c
runtime.(*scavChunkData).free(0x0?, 0x3, 0x45350?)
	runtime/mgcscavenge.go:1341 +0xd8 fp=0x123f5f330 sp=0x123f5f310 pc=0x3daf0
runtime.(*scavengeIndex).free(0x10a8cc0, 0x146, 0xc7, 0x3)
	runtime/mgcscavenge.go:1154 +0xa4 fp=0x123f5f368 sp=0x123f5f330 pc=0x3d71c
...
runtime.gcMarkTermination()
	runtime/mgc.go:1101 +0x510 fp=0x50041ef8 sp=0x50041d18 pc=0x33050
runtime.gcMarkDone()
	runtime/mgc.go:941 +0x25c fp=0x50041f48 sp=0x50041ef8 pc=0x32a0c
runtime.gcBgMarkWorker()
	runtime/mgc.go:1445 +0x374 fp=0x50041fd8 sp=0x50041f48 pc=0x33d84
runtime.goexit({})
	runtime/asm_riscv64.s:521 +0x4 fp=0x50041fd8 sp=0x50041fd8 pc=0x89ea4
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1237 +0x2c

2023-09-28 17:55 openbsd-riscv64-jsing sys@2964e1e4 go@68a12a80 runtime [build] (log)

runtime: inUse=0 npages=4
fatal error: allocated pages below zero?

runtime stack:
runtime.throw({0x9169de?, 0x440009c0?})
	/home/joel/src/go-openbsd-riscv64-bootstrap/src/runtime/panic.go:837 +0x44 fp=0xb64d82f8 sp=0xb64d82d0 pc=0x5318c
runtime.(*scavChunkData).free(0x74838?, 0x4, 0xb64d8360?)
	/home/joel/src/go-openbsd-riscv64-bootstrap/src/runtime/mgcscavenge.go:1381 +0xd8 fp=0xb64d8318 sp=0xb64d82f8 pc=0x3d288
runtime.(*scavengeIndex).free(...)
	/home/joel/src/go-openbsd-riscv64-bootstrap/src/runtime/mgcscavenge.go:1159
...
	/home/joel/src/go-openbsd-riscv64-bootstrap/src/runtime/proc.go:3812 +0x68 fp=0xb64d86d0 sp=0xb64d8698 pc=0x5dca0
runtime.newstack()
	/home/joel/src/go-openbsd-riscv64-bootstrap/src/runtime/stack.go:1071 +0x2b0 fp=0xb64d88b8 sp=0xb64d86d0 pc=0x6e968
runtime.morestack()
	/home/joel/src/go-openbsd-riscv64-bootstrap/src/runtime/asm_riscv64.s:204 +0x58 fp=0xb64d88c0 sp=0xb64d88b8 pc=0x87698

goroutine 175 [running]:
	goroutine running on other thread; stack unavailable
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/joel/src/go-openbsd-riscv64-bootstrap/src/runtime/mgc.go:1237 +0x2c

2023-10-04 10:08 openbsd-riscv64-jsing go@0074125c encoding/gob.TestLargeSlice (log)

runtime: inUse=0 npages=1
fatal error: allocated pages below zero?

runtime stack:
runtime.throw({0x1c2d9c?, 0x64821cfb?})
	/home/gopher/build/go/src/runtime/panic.go:1018 +0x44 fp=0x4cbaaad8 sp=0x4cbaaab0 pc=0x46ddc
runtime.(*scavChunkData).free(0x320048?, 0x1, 0x5eca0?)
	/home/gopher/build/go/src/runtime/mgcscavenge.go:1341 +0xd8 fp=0x4cbaaaf8 sp=0x4cbaaad8 pc=0x34218
runtime.(*scavengeIndex).free(0x346760, 0x1df, 0x87, 0x1)
	/home/gopher/build/go/src/runtime/mgcscavenge.go:1154 +0xa4 fp=0x4cbaab30 sp=0x4cbaaaf8 pc=0x33e44
...
	/home/gopher/build/go/src/encoding/gob/decode.go:1234 +0xfc fp=0xd3a1ddf8 sp=0xd3a1dd20 pc=0x130d6c
encoding/gob.(*Decoder).DecodeValue(0x70074000, {0x17d480?, 0x70119800?, 0x70114a00?})
	/home/gopher/build/go/src/encoding/gob/decoder.go:229 +0x1a0 fp=0xd3a1de70 sp=0xd3a1ddf8 pc=0x131d70
encoding/gob.(*Decoder).Decode(0x70074000, {0x17d480?, 0x70119800?})
	/home/gopher/build/go/src/encoding/gob/decoder.go:204 +0x144 fp=0xd3a1dec0 sp=0xd3a1de70 pc=0x131b94
encoding/gob.testEncodeDecode(0x7022d6c0, {0x17d480, 0x701197e8}, {0x17d480, 0x70119800})
	/home/gopher/build/go/src/encoding/gob/codec_test.go:1559 +0x288 fp=0xd3a1df30 sp=0xd3a1dec0 pc=0x14a358
encoding/gob.TestLargeSlice.func1(0x0?)
	/home/gopher/build/go/src/encoding/gob/codec_test.go:1579 +0xcc fp=0xd3a1df78 sp=0xd3a1df30 pc=0x1604bc
testing.tRunner(0x7022d6c0, 0x1cd0f8)

2023-10-04 15:11 openbsd-riscv64-jsing go@a9036396 runtime.TestFakeTime (log)

--- FAIL: TestFakeTime (16.20s)
    crash_test.go:152: running /home/gopher/build/go/bin/go build -o /home/gopher/build/tmp/go-build1023398732/testfaketime_-tags=faketime.exe -tags=faketime
    time_test.go:31: building testfaketime [-tags=faketime]: exit status 1
        # runtime
        runtime: inUse=0 npages=5
        fatal error: allocated pages below zero?

        runtime stack:
        runtime.throw({0x91b6fa?, 0x74df4?})
        	runtime/panic.go:1018 +0x44 fp=0x649e2470 sp=0x649e2448 pc=0x53b6c
...
        	runtime/mgc.go:203 +0x2c fp=0xe40417d8 sp=0xe40417c8 pc=0x319f4
        runtime.goexit({})
        	runtime/asm_riscv64.s:521 +0x4 fp=0xe40417d8 sp=0xe40417d8 pc=0x89ea4
        created by runtime.gcenable in goroutine 1
        	runtime/mgc.go:203 +0x70

        goroutine 16 [running]:
        	goroutine running on other thread; stack unavailable
        created by runtime.gcBgMarkStartWorkers in goroutine 1
        	runtime/mgc.go:1237 +0x2c

2023-10-04 16:05 openbsd-riscv64-jsing go@0c64ebce cmd/link.TestMachOBuildVersion (log)

--- FAIL: TestMachOBuildVersion (15.45s)
    link_test.go:377: [/home/gopher/build/go/bin/go build -ldflags=-linkmode=internal -o /home/gopher/build/tmp/TestMachOBuildVersion1940749510/001/main /home/gopher/build/tmp/TestMachOBuildVersion1940749510/001/main.go]: exit status 1:
        # runtime
        runtime: inUse=0 npages=4
        fatal error: allocated pages below zero?

        runtime stack:
        runtime.throw({0x91b6fa?, 0x106bba0?})
        	runtime/panic.go:1018 +0x44 fp=0x36af2bf0a8 sp=0x36af2bf080 pc=0x53b6c
        runtime.(*scavChunkData).free(0x750c0?, 0x4, 0xaf2bf110?)
...
        runtime.gcMarkTermination()
        	runtime/mgc.go:1101 +0x510 fp=0x6003c6f8 sp=0x6003c518 pc=0x33050
        runtime.gcMarkDone()
        	runtime/mgc.go:941 +0x25c fp=0x6003c748 sp=0x6003c6f8 pc=0x32a0c
        runtime.gcBgMarkWorker()
        	runtime/mgc.go:1445 +0x374 fp=0x6003c7d8 sp=0x6003c748 pc=0x33d84
        runtime.goexit({})
        	runtime/asm_riscv64.s:521 +0x4 fp=0x6003c7d8 sp=0x6003c7d8 pc=0x89ea4
        created by runtime.gcBgMarkStartWorkers in goroutine 1
        	runtime/mgc.go:1237 +0x2c

2023-10-04 16:50 openbsd-riscv64-jsing go@3a69dcdc cmd/go.TestGoListExport (log)

vcs-test.golang.org rerouted to http://127.0.0.1:19379
https://vcs-test.golang.org rerouted to https://127.0.0.1:36129
go test proxy running at GOPROXY=http://127.0.0.1:24112/mod
--- FAIL: TestGoListExport (17.46s)
    go_test.go:1227: running testgo [list -f {{.Export}} strings]
    go_test.go:1231: running testgo [list -export -f {{.Export}} strings]
    go_test.go:1231: standard error:
    go_test.go:1231: # runtime
        runtime: inUse=0 npages=4
        fatal error: allocated pages below zero?
...
        traceback: unexpected SPWRITE function runtime.morestack
        runtime.morestack()
        	runtime/asm_riscv64.s:204 +0x58 fp=0x6a921c60 sp=0x6a921c58 pc=0x87d30

        goroutine 179 [running]:
        	goroutine running on other thread; stack unavailable
        created by runtime.gcBgMarkStartWorkers in goroutine 1
        	runtime/mgc.go:1237 +0x2c

    go_test.go:1231: go [list -export -f {{.Export}} strings] failed unexpectedly in /home/gopher/build/go/src/cmd/go: exit status 1

2023-10-04 22:08 openbsd-riscv64-jsing net@5d5a036a go@bc150700 runtime [build] (log)

runtime: inUse=0 npages=5
fatal error: allocated pages below zero?

runtime stack:
runtime.throw({0x72caf7?, 0x6a14c?})
	../../go/src/runtime/panic.go:837 +0x44 fp=0x11c941ad0 sp=0x11c941aa8 pc=0x4a744
runtime.(*scavChunkData).free(0x0?, 0x5, 0x4f1ec?)
	../../go/src/runtime/mgcscavenge.go:1381 +0xd8 fp=0x11c941af0 sp=0x11c941ad0 pc=0x376d0
runtime.(*scavengeIndex).free(0xd46ca0, 0x1c2, 0x8, 0x5)
	../../go/src/runtime/mgcscavenge.go:1159 +0xa4 fp=0x11c941b28 sp=0x11c941af0 pc=0x371fc
...
runtime.sweepone()
	../../go/src/runtime/mgcsweep.go:393 +0x140 fp=0x70045780 sp=0x70045728 pc=0x38590
runtime.bgsweep(0x0?)
	../../go/src/runtime/mgcsweep.go:301 +0x14c fp=0x700457c8 sp=0x70045780 pc=0x382ec
runtime.gcenable.func1()
	../../go/src/runtime/mgc.go:203 +0x2c fp=0x700457d8 sp=0x700457c8 pc=0x2c7dc
runtime.goexit()
	../../go/src/runtime/asm_riscv64.s:521 +0x4 fp=0x700457d8 sp=0x700457d8 pc=0x7e434
created by runtime.gcenable in goroutine 1
	../../go/src/runtime/mgc.go:203 +0x70

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `fatal error: allocated pages below zero` && goos == "openbsd" && goarch == "riscv64"

2023-10-04 22:08 openbsd-riscv64-jsing net@5d5a036a go@008dabcd cmd/link/internal/ld [build] (log)

runtime: inUse=0 npages=4
fatal error: allocated pages below zero?

runtime stack:
runtime.throw({0x91b723?, 0x400000?})
	runtime/panic.go:1018 +0x44 fp=0x3901bba360 sp=0x3901bba338 pc=0x53b6c
runtime.(*scavChunkData).free(0x13afb5b00?, 0x4, 0x45350?)
	runtime/mgcscavenge.go:1341 +0xd8 fp=0x3901bba380 sp=0x3901bba360 pc=0x3daf0
runtime.(*scavengeIndex).free(0x10a8ce0, 0x1ea, 0x7, 0x4)
	runtime/mgcscavenge.go:1154 +0xa4 fp=0x3901bba3b8 sp=0x3901bba380 pc=0x3d71c
...
runtime.gcMarkTermination()
	runtime/mgc.go:1101 +0x510 fp=0x78041ef8 sp=0x78041d18 pc=0x33050
runtime.gcMarkDone()
	runtime/mgc.go:941 +0x25c fp=0x78041f48 sp=0x78041ef8 pc=0x32a0c
runtime.gcBgMarkWorker()
	runtime/mgc.go:1445 +0x374 fp=0x78041fd8 sp=0x78041f48 pc=0x33d84
runtime.goexit({})
	runtime/asm_riscv64.s:521 +0x4 fp=0x78041fd8 sp=0x78041fd8 pc=0x89ea4
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1237 +0x2c

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `fatal error: allocated pages below zero` && goos == "openbsd" && goarch == "riscv64"

2023-10-05 19:45 openbsd-riscv64-jsing go@10da3b64 cmd/link/internal/ld [build] (log)

runtime: inUse=0 npages=4
fatal error: allocated pages below zero?

runtime stack:
runtime.throw({0x8cb7f1?, 0x63?})
	runtime/panic.go:1018 +0x44 fp=0x8fb5c550 sp=0x8fb5c528 pc=0x4b024
runtime.(*scavChunkData).free(0x48?, 0x4, 0x310?)
	runtime/mgcscavenge.go:1341 +0xd8 fp=0x8fb5c570 sp=0x8fb5c550 pc=0x37df0
runtime.(*scavengeIndex).free(0x1038ce0, 0x1fc, 0x1c9, 0x4)
	runtime/mgcscavenge.go:1154 +0xa4 fp=0x8fb5c5a8 sp=0x8fb5c570 pc=0x37a1c
...
runtime.newstack()
	runtime/stack.go:1070 +0x2b0 fp=0x8fb5cb08 sp=0x8fb5c968 pc=0x655d8
traceback: unexpected SPWRITE function runtime.morestack
runtime.morestack()
	runtime/asm_riscv64.s:204 +0x58 fp=0x8fb5cb10 sp=0x8fb5cb08 pc=0x7c808

goroutine 38 [running]:
	goroutine running on other thread; stack unavailable
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1237 +0x2c

2023-10-06 15:42 openbsd-riscv64-jsing go@28f4ea16 cmd/compile/internal/ssagen [build] (log)

runtime: inUse=0 npages=1
fatal error: allocated pages below zero?

runtime stack:
runtime.throw({0x91b9cf?, 0x0?})
	runtime/panic.go:1018 +0x44 fp=0x17492fd20 sp=0x17492fcf8 pc=0x53c34
runtime.(*scavChunkData).free(0x0?, 0x1, 0x45428?)
	runtime/mgcscavenge.go:1341 +0xd8 fp=0x17492fd40 sp=0x17492fd20 pc=0x3dbc8
runtime.(*scavengeIndex).free(0x10a8e20, 0x140, 0x8f, 0x1)
	runtime/mgcscavenge.go:1154 +0xa4 fp=0x17492fd78 sp=0x17492fd40 pc=0x3d7f4
...
runtime.park_m(0x518c5a00?)
	runtime/proc.go:3752 +0x1ac fp=0x1749302a0 sp=0x174930248 pc=0x5e0dc
traceback: unexpected SPWRITE function runtime.mcall
runtime.mcall()
	runtime/asm_riscv64.s:291 +0x40 fp=0x1749302b0 sp=0x1749302a0 pc=0x87ef0

goroutine 360 [running]:
	goroutine running on other thread; stack unavailable
created by cmd/compile/internal/gc.compileFunctions.func3 in goroutine 14
	cmd/compile/internal/gc/compile.go:163 +0x204

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `fatal error: allocated pages below zero` && goos == "openbsd" && goarch == "riscv64"

2023-10-05 16:33 openbsd-riscv64-jsing net@88194ad8 go@10da3b64 cmd/compile/internal/ssa [build] (log)

runtime: inUse=0 npages=1
fatal error: allocated pages below zero?

runtime stack:
runtime.throw({0x72caf7?, 0x32?})
	../../go/src/runtime/panic.go:837 +0x44 fp=0x323b62bb20 sp=0x323b62baf8 pc=0x4a744
runtime.(*scavChunkData).free(0x1aa0?, 0x1, 0x354?)
	../../go/src/runtime/mgcscavenge.go:1381 +0xd8 fp=0x323b62bb40 sp=0x323b62bb20 pc=0x376d0
runtime.(*scavengeIndex).free(0xd46cc0, 0x1c7, 0xaf, 0x1)
	../../go/src/runtime/mgcscavenge.go:1159 +0xa4 fp=0x323b62bb78 sp=0x323b62bb40 pc=0x371fc
...
runtime.sweepone()
	../../go/src/runtime/mgcsweep.go:393 +0x140 fp=0x70047780 sp=0x70047728 pc=0x38590
runtime.bgsweep(0x0?)
	../../go/src/runtime/mgcsweep.go:301 +0x14c fp=0x700477c8 sp=0x70047780 pc=0x382ec
runtime.gcenable.func1()
	../../go/src/runtime/mgc.go:203 +0x2c fp=0x700477d8 sp=0x700477c8 pc=0x2c7dc
runtime.goexit()
	../../go/src/runtime/asm_riscv64.s:521 +0x4 fp=0x700477d8 sp=0x700477d8 pc=0x7e434
created by runtime.gcenable in goroutine 1
	../../go/src/runtime/mgc.go:203 +0x70

2023-10-05 16:33 openbsd-riscv64-jsing net@88194ad8 go@008dabcd runtime [build] (log)

runtime: inUse=0 npages=4
fatal error: allocated pages below zero?

runtime stack:
runtime.throw({0x8cb7f1?, 0xca214370?})
	runtime/panic.go:1018 +0x44 fp=0x33e2bf3a70 sp=0x33e2bf3a48 pc=0x4b024
runtime.(*scavChunkData).free(0x29018?, 0x4, 0x656e6c18?)
	runtime/mgcscavenge.go:1341 +0xd8 fp=0x33e2bf3a90 sp=0x33e2bf3a70 pc=0x37df0
runtime.(*scavengeIndex).free(0x1038ce0, 0x1ce, 0x1f1, 0x4)
	runtime/mgcscavenge.go:1154 +0xa4 fp=0x33e2bf3ac8 sp=0x33e2bf3a90 pc=0x37a1c
...
runtime.newstack()
	runtime/stack.go:1070 +0x2b0 fp=0x33e2bf4028 sp=0x33e2bf3e88 pc=0x655d8
traceback: unexpected SPWRITE function runtime.morestack
runtime.morestack()
	runtime/asm_riscv64.s:204 +0x58 fp=0x33e2bf4030 sp=0x33e2bf4028 pc=0x7c808

goroutine 172 [running]:
	goroutine running on other thread; stack unavailable
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1237 +0x2c

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `fatal error: allocated pages below zero` && goos == "openbsd" && goarch == "riscv64"

2023-09-28 17:55 openbsd-riscv64-jsing sys@2964e1e4 go@10da3b64 runtime [build] (log)

runtime: inUse=0 npages=4
fatal error: allocated pages below zero?

runtime stack:
runtime.throw({0x91b723?, 0x480841a0?})
	runtime/panic.go:1018 +0x44 fp=0xe9983c88 sp=0xe9983c60 pc=0x53b6c
runtime.(*scavChunkData).free(0x750c0?, 0x4, 0xe9983cf0?)
	runtime/mgcscavenge.go:1341 +0xd8 fp=0xe9983ca8 sp=0xe9983c88 pc=0x3daf0
runtime.(*scavengeIndex).free(...)
	runtime/mgcscavenge.go:1154
...
runtime.gcMarkTermination()
	runtime/mgc.go:1101 +0x510 fp=0x480466f8 sp=0x48046518 pc=0x33050
runtime.gcMarkDone()
	runtime/mgc.go:941 +0x25c fp=0x48046748 sp=0x480466f8 pc=0x32a0c
runtime.gcBgMarkWorker()
	runtime/mgc.go:1445 +0x374 fp=0x480467d8 sp=0x48046748 pc=0x33d84
runtime.goexit({})
	runtime/asm_riscv64.s:521 +0x4 fp=0x480467d8 sp=0x480467d8 pc=0x89ea4
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1237 +0x2c

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `fatal error: allocated pages below zero` && goos == "openbsd" && goarch == "riscv64"

2023-10-06 21:01 openbsd-riscv64-jsing go@be3d5fb6 runtime [build] (log)

runtime: inUse=0 npages=2
fatal error: allocated pages below zero?

runtime stack:
runtime.throw({0x9169de?, 0x0?})
	/home/joel/src/go-openbsd-riscv64-bootstrap/src/runtime/panic.go:837 +0x44 fp=0x12653de28 sp=0x12653de00 pc=0x5318c
runtime.(*scavChunkData).free(0x0?, 0x2, 0x44b20?)
	/home/joel/src/go-openbsd-riscv64-bootstrap/src/runtime/mgcscavenge.go:1381 +0xd8 fp=0x12653de48 sp=0x12653de28 pc=0x3d288
runtime.(*scavengeIndex).free(0x10a64c0, 0x1f7, 0x106, 0x2)
	/home/joel/src/go-openbsd-riscv64-bootstrap/src/runtime/mgcscavenge.go:1159 +0xa4 fp=0x12653de80 sp=0x12653de48 pc=0x3cdb4
...
runtime.gcMarkTermination()
	/home/joel/src/go-openbsd-riscv64-bootstrap/src/runtime/mgc.go:1101 +0x510 fp=0x7c03def8 sp=0x7c03dd18 pc=0x32648
runtime.gcMarkDone()
	/home/joel/src/go-openbsd-riscv64-bootstrap/src/runtime/mgc.go:941 +0x25c fp=0x7c03df48 sp=0x7c03def8 pc=0x32004
runtime.gcBgMarkWorker()
	/home/joel/src/go-openbsd-riscv64-bootstrap/src/runtime/mgc.go:1445 +0x378 fp=0x7c03dfd8 sp=0x7c03df48 pc=0x33388
runtime.goexit()
	/home/joel/src/go-openbsd-riscv64-bootstrap/src/runtime/asm_riscv64.s:521 +0x4 fp=0x7c03dfd8 sp=0x7c03dfd8 pc=0x8980c
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/home/joel/src/go-openbsd-riscv64-bootstrap/src/runtime/mgc.go:1237 +0x2c

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `fatal error: allocated pages below zero` && goos == "openbsd" && goarch == "riscv64"

2023-10-06 20:57 openbsd-riscv64-jsing go@b72bbaeb cmd/go.TestScript (log)

vcs-test.golang.org rerouted to http://127.0.0.1:37962
https://vcs-test.golang.org rerouted to https://127.0.0.1:15767
go test proxy running at GOPROXY=http://127.0.0.1:33421/mod
2023/10/07 15:18:31 http: TLS handshake error from 127.0.0.1:22669: EOF
2023/10/07 15:18:31 http: TLS handshake error from 127.0.0.1:6836: read tcp 127.0.0.1:15767->127.0.0.1:6836: read: connection reset by peer
--- FAIL: TestScript (0.55s)
    --- FAIL: TestScript/clean_cache_n (36.50s)
        script_test.go:132: 2023-10-07T04:16:21Z
        script_test.go:134: $WORK=/home/gopher/build/tmp/cmd-go-test-3516719434/tmpdir1256145971/clean_cache_n1824977704
        script_test.go:156: 
...
            fatal error: allocated pages below zero?

            runtime stack:
            runtime.throw({0x94b72b?, 0x800?})
            	runtime/panic.go:1018 +0x44 fp=0x8c7d2e90 sp=0x8c7d2e68 pc=0x53cac
            runtime.(*scavChunkData).free(0x40160?, 0x5, 0x45440?)
            	runtime/mgcscavenge.go:1341 +0xd8 fp=0x8c7d2eb0 sp=0x8c7d2e90 pc=0x3dbe0
            runtime.(*scavengeIndex).free(0x1109200, 0x21c, 0x1d, 0x5)
            	runtime/mgcscavenge.go:1154 +0xa4 fp=0x8c7d2ee8 sp=0x8c7d2eb0 pc=0x3d80c
            runtime.(*pageAlloc).free(...)
...
            runtime.gcMarkTermination()
            	runtime/mgc.go:1127 +0x538 fp=0x84e8c6f8 sp=0x84e8c518 pc=0x33168
            runtime.gcMarkDone()
            	runtime/mgc.go:941 +0x25c fp=0x84e8c748 sp=0x84e8c6f8 pc=0x32afc
            runtime.gcBgMarkWorker()
            	runtime/mgc.go:1445 +0x374 fp=0x84e8c7d8 sp=0x84e8c748 pc=0x33e74
            runtime.goexit({})
            	runtime/asm_riscv64.s:521 +0x4 fp=0x84e8c7d8 sp=0x84e8c7d8 pc=0x89fbc
            created by runtime.gcBgMarkStartWorkers in goroutine 1
            	runtime/mgc.go:1237 +0x2c

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `fatal error: allocated pages below zero` && goos == "openbsd" && goarch == "riscv64"

2023-10-05 16:33 openbsd-riscv64-jsing net@88194ad8 go@8b6e0e6e runtime [build] (log)

runtime: inUse=0 npages=4
fatal error: allocated pages below zero?

runtime stack:
runtime.throw({0x94ba2d?, 0x440841a0?})
	runtime/panic.go:1018 +0x44 fp=0x6eb1b438 sp=0x6eb1b410 pc=0x53c94
runtime.(*scavChunkData).free(0x751e8?, 0x4, 0x6eb1b4a0?)
	runtime/mgcscavenge.go:1341 +0xd8 fp=0x6eb1b458 sp=0x6eb1b438 pc=0x3dbe0
runtime.(*scavengeIndex).free(...)
	runtime/mgcscavenge.go:1154
...
runtime.gcMarkTermination()
	runtime/mgc.go:1101 +0x510 fp=0x4441cef8 sp=0x4441cd18 pc=0x33140
runtime.gcMarkDone()
	runtime/mgc.go:941 +0x25c fp=0x4441cf48 sp=0x4441cef8 pc=0x32afc
runtime.gcBgMarkWorker()
	runtime/mgc.go:1445 +0x374 fp=0x4441cfd8 sp=0x4441cf48 pc=0x33e74
runtime.goexit({})
	runtime/asm_riscv64.s:521 +0x4 fp=0x4441cfd8 sp=0x4441cfd8 pc=0x89fcc
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1237 +0x2c

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `fatal error: allocated pages below zero` && goos == "openbsd" && goarch == "riscv64"

2023-10-06 20:48 openbsd-riscv64-jsing go@ad76a98d cmd/compile/internal/ssa [build] (log)

runtime: inUse=0 npages=58
fatal error: allocated pages below zero?

runtime stack:
runtime.throw({0x310bd6?, 0x1?})
	runtime/panic.go:1018 +0x44 fp=0x8d419ba0 sp=0x8d419b78 pc=0x45294
runtime.(*scavChunkData).free(0x0?, 0x3a, 0x1d8?)
	runtime/mgcscavenge.go:1341 +0xd8 fp=0x8d419bc0 sp=0x8d419ba0 pc=0x341d0
runtime.(*scavengeIndex).free(0x57af40, 0x30f, 0xcd, 0x3a)
	runtime/mgcscavenge.go:1154 +0xa4 fp=0x8d419bf8 sp=0x8d419bc0 pc=0x33dfc
...
	cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:405 +0x34 fp=0x7be79fd8 sp=0x7be79fc8 pc=0x23d80c
runtime.goexit({})
	runtime/asm_riscv64.s:521 +0x4 fp=0x7be79fd8 sp=0x7be79fd8 pc=0x74894
created by cmd/vendor/golang.org/x/tools/go/analysis/unitchecker.run.func3 in goroutine 14
	cmd/vendor/golang.org/x/tools/go/analysis/unitchecker/unitchecker.go:402 +0x6c

goroutine 17 [running]:
	goroutine running on other thread; stack unavailable
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1237 +0x2c

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `fatal error: allocated pages below zero` && goos == "openbsd" && goarch == "riscv64"

2023-10-05 16:33 openbsd-riscv64-jsing net@88194ad8 go@b01cb72e net/http [build] (log)

runtime: inUse=0 npages=4
fatal error: allocated pages below zero?

runtime stack:
runtime.throw({0x94ba2d?, 0x9?})
	runtime/panic.go:1018 +0x44 fp=0x175b4da30 sp=0x175b4da08 pc=0x53c94
runtime.(*scavChunkData).free(0xb6b20b00?, 0x4, 0x45440?)
	runtime/mgcscavenge.go:1341 +0xd8 fp=0x175b4da50 sp=0x175b4da30 pc=0x3dbe0
runtime.(*scavengeIndex).free(0x1109200, 0x1bf, 0x54, 0x4)
	runtime/mgcscavenge.go:1154 +0xa4 fp=0x175b4da88 sp=0x175b4da50 pc=0x3d80c
...
runtime.gcMarkTermination()
	runtime/mgc.go:1101 +0x510 fp=0x6c382ef8 sp=0x6c382d18 pc=0x33140
runtime.gcMarkDone()
	runtime/mgc.go:941 +0x25c fp=0x6c382f48 sp=0x6c382ef8 pc=0x32afc
runtime.gcBgMarkWorker()
	runtime/mgc.go:1445 +0x374 fp=0x6c382fd8 sp=0x6c382f48 pc=0x33e74
runtime.goexit({})
	runtime/asm_riscv64.s:521 +0x4 fp=0x6c382fd8 sp=0x6c382fd8 pc=0x89fcc
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1237 +0x2c

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `fatal error: allocated pages below zero` && goos == "openbsd" && goarch == "riscv64"

2023-10-06 17:02 openbsd-riscv64-jsing go@b455e239 runtime.TestArenaCollision (log)

--- FAIL: TestArenaCollision (0.15s)
    malloc_test.go:284: === RUN   TestArenaCollision
            malloc_test.go:301: reserved [0x48000000, 0x4c000000)
            malloc_test.go:301: reserved [0x11c000000, 0x120000000)
            malloc_test.go:301: reserved [0x78000000, 0x7c000000)
        runtime: inUse=3 npages=4
        fatal error: allocated pages below zero?

        runtime stack:
        runtime.throw({0x422a44?, 0x899c8?})
...
        runtime.bgsweep(0x0?)
        	/home/gopher/build/go/src/runtime/mgcsweep.go:308 +0x1c4 fp=0x4403efc8 sp=0x4403ef80 pc=0x38d04
        runtime.gcenable.gowrap1()
        	/home/gopher/build/go/src/runtime/mgc.go:203 +0x2c fp=0x4403efd8 sp=0x4403efc8 pc=0x2ce74
        runtime.goexit({})
        	/home/gopher/build/go/src/runtime/asm_riscv64.s:521 +0x4 fp=0x4403efd8 sp=0x4403efd8 pc=0x87b8c
        created by runtime.gcenable in goroutine 1
        	/home/gopher/build/go/src/runtime/mgc.go:203 +0x70

        (exit status exit status 2)

2023-10-06 17:02 openbsd-riscv64-jsing go@b455e239 runtime.TestArenaCollision (log)

--- FAIL: TestArenaCollision (0.26s)
    malloc_test.go:284: === RUN   TestArenaCollision
            malloc_test.go:301: reserved [0x50000000, 0x54000000)
            malloc_test.go:301: reserved [0xbc000000, 0xc0000000)
            malloc_test.go:301: reserved [0xb0000000, 0xb4000000)
            malloc_test.go:301: reserved [0x164000000, 0x168000000)
            malloc_test.go:301: reserved [0x16c000000, 0x170000000)
        --- PASS: TestArenaCollision (0.06s)
        runtime: inUse=0 npages=1
        fatal error: allocated pages below zero?
...
        traceback: unexpected SPWRITE function runtime.morestack
        runtime.morestack()
        	/home/gopher/build/go/src/runtime/asm_riscv64.s:204 +0x58 fp=0x55018530 sp=0x55018528 pc=0x85a18

        goroutine 19 [running]:
        	goroutine running on other thread; stack unavailable
        created by runtime.gcBgMarkStartWorkers in goroutine 18
        	/home/gopher/build/go/src/runtime/mgc.go:1237 +0x2c

        (exit status exit status 2)

— watchflakes

[gopherbot]

Found new dashboard test flakes for:

#!watchflakes
post <- log ~ `fatal error: allocated pages below zero` && goos == "openbsd" && goarch == "riscv64"

2023-10-06 20:50 openbsd-riscv64-jsing sys@8ccaaf02 go@561bf045 runtime [build] (log)

runtime: inUse=0 npages=4
fatal error: allocated pages below zero?

runtime stack:
runtime.throw({0x94b88b?, 0x0?})
	runtime/panic.go:1018 +0x44 fp=0xc83b1250 sp=0xc83b1228 pc=0x53c2c
runtime.(*scavChunkData).free(0x0?, 0x4, 0x453e8?)
	runtime/mgcscavenge.go:1341 +0xd8 fp=0xc83b1270 sp=0xc83b1250 pc=0x3dbb0
runtime.(*scavengeIndex).free(0x1109300, 0x3ed, 0xc0, 0x4)
	runtime/mgcscavenge.go:1154 +0xa4 fp=0xc83b12a8 sp=0xc83b1270 pc=0x3d7dc
...
runtime.gcMarkTermination()
	runtime/mgc.go:1101 +0x50c fp=0xf8046ef8 sp=0xf8046d18 pc=0x33124
runtime.gcMarkDone()
	runtime/mgc.go:941 +0x25c fp=0xf8046f48 sp=0xf8046ef8 pc=0x32ae4
runtime.gcBgMarkWorker()
	runtime/mgc.go:1445 +0x374 fp=0xf8046fd8 sp=0xf8046f48 pc=0x33e54
runtime.goexit({})
	runtime/asm_riscv64.s:521 +0x4 fp=0xf8046fd8 sp=0xf8046fd8 pc=0x89ebc
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1237 +0x2c

2023-10-06 20:50 openbsd-riscv64-jsing sys@8ccaaf02 go@f711892a runtime [build] (log)

runtime: inUse=0 npages=2
fatal error: allocated pages below zero?

runtime stack:
runtime.throw({0x8fb94f?, 0x1?})
	runtime/panic.go:1018 +0x44 fp=0x848ab528 sp=0x848ab500 pc=0x4b53c
runtime.(*scavChunkData).free(0x0?, 0x2, 0x421?)
	runtime/mgcscavenge.go:1341 +0xd8 fp=0x848ab548 sp=0x848ab528 pc=0x38398
runtime.(*scavengeIndex).free(0x1089000, 0x19e, 0x125, 0x2)
	runtime/mgcscavenge.go:1154 +0xa4 fp=0x848ab580 sp=0x848ab548 pc=0x37fc4
...
runtime.gcMarkTermination()
	runtime/mgc.go:1101 +0x518 fp=0x4c398ef8 sp=0x4c398d18 pc=0x2eba0
runtime.gcMarkDone()
	runtime/mgc.go:941 +0x25c fp=0x4c398f48 sp=0x4c398ef8 pc=0x2e554
runtime.gcBgMarkWorker()
	runtime/mgc.go:1445 +0x36c fp=0x4c398fd8 sp=0x4c398f48 pc=0x2f8bc
runtime.goexit({})
	runtime/asm_riscv64.s:521 +0x4 fp=0x4c398fd8 sp=0x4c398fd8 pc=0x7ec3c
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1237 +0x2c

2023-10-06 20:50 openbsd-riscv64-jsing sys@8ccaaf02 go@6e8caefc runtime [build] (log)

runtime: inUse=0 npages=4
fatal error: allocated pages below zero?

runtime stack:
runtime.throw({0x94b88b?, 0x440009c0?})
	runtime/panic.go:1018 +0x44 fp=0x8c7bd7f0 sp=0x8c7bd7c8 pc=0x53cac
runtime.(*scavChunkData).free(0x751e0?, 0x4, 0x8c7bd858?)
	runtime/mgcscavenge.go:1341 +0xd8 fp=0x8c7bd810 sp=0x8c7bd7f0 pc=0x3dbe0
runtime.(*scavengeIndex).free(...)
	runtime/mgcscavenge.go:1154
...
runtime.newstack()
	runtime/stack.go:1070 +0x2b0 fp=0x8c7bdda8 sp=0x8c7bdbc0 pc=0x6f350
traceback: unexpected SPWRITE function runtime.morestack
runtime.morestack()
	runtime/asm_riscv64.s:204 +0x58 fp=0x8c7bddb0 sp=0x8c7bdda8 pc=0x87e48

goroutine 93 [running]:
	goroutine running on other thread; stack unavailable
created by runtime.gcBgMarkStartWorkers in goroutine 1
	runtime/mgc.go:1237 +0x2c

— watchflakes