x/build/cmd/gitmirror: build repo is broken due to updateDashboard failing to postCommit on a commit with 2 parents

[dmitshur]

From https://farmer.golang.org/#health right now:

This has happened a few times in the last few days. I've fixed previous instances by restarting gitmirror, but since it keeps recurring, this will need to be investigated and fixed more.

/cc @toothrot @cagedmantis

[gopherbot]

Change https://golang.org/cl/208663 mentions this issue: cmd/gitmirror: log error details when Repo.Loop operations fail

[bradfitz]

If it helps:

$ for IP in $(kubectl get pods  -l app=gitmirror -o=jsonpath={.items[*].status.podIP}); do curl http://$IP:8585/debug/goroutines; done

.. to look at the goroutine stacks of the active pods.

[bradfitz]

If it turns out this is hanging due to updating the dashboard, note that all the code's gone as of pending https://go-review.googlesource.com/c/build/+/208697

[dmitshur]

It is. It happened again, and here are the logs:

[...]
2019/11/25 22:18:37 build: sending commit to dashboard: 174fe5663144a72ea7dad1b480912906854505b6[master]("devapp/owners: add owners for mod repository")
2019/11/25 22:18:37 build: updateDashboard failed in repo loop: postCommit: error: parent commit not found
2019/11/25 22:18:49 build: sending commit to dashboard: 174fe5663144a72ea7dad1b480912906854505b6[master]("devapp/owners: add owners for mod repository")
2019/11/25 22:18:49 build: updateDashboard failed in repo loop: postCommit: error: parent commit not found
2019/11/25 22:19:01 build: sending commit to dashboard: 174fe5663144a72ea7dad1b480912906854505b6[master]("devapp/owners: add owners for mod repository")
2019/11/25 22:19:01 build: updateDashboard failed in repo loop: postCommit: error: parent commit not found
[...]

The "parent commit not found" error is likely related to commit golang/build@63986c1 having 2 parents instead of the usual 1.

It sounds like we can let this wait for CL 208697 before trying to make progress here, does that sound good to you @bradfitz?

[bradfitz]

Yup. Let's wait.

That code is all scary and it's not worth debugging. Easier to delete it all.

[gopherbot]

Change https://golang.org/cl/208697 mentions this issue: app, gitmirror, maintner: use maintner for dashboard, not datastore