Skip to content

Commit ee7917e

Browse files
dmitshurgopherbot
authored andcommitted
DashboardBuilders: remove steps to add builders to legacy system
The migration to LUCI is far along now that we aren't adding any more new builders to coordinator. Remove the instructions for doing that, and also move the LUCI section to be on top. This will unfortunately require some links to be updated, but there's no way to avoid that. Keep the security note section since its content generally applies to the new system too. For golang/go#67761. Change-Id: I338b5622b1f9020539d1ab309f39127a4f3e0c93 Reviewed-on: https://go-review.googlesource.com/c/wiki/+/592755 Reviewed-by: Carlos Amedee <[email protected]> Auto-Submit: Dmitri Shuralyov <[email protected]>
1 parent eba3719 commit ee7917e

File tree

1 file changed

+23
-42
lines changed

1 file changed

+23
-42
lines changed

DashboardBuilders.md

+23-42
Original file line numberDiff line numberDiff line change
@@ -2,50 +2,9 @@
22
title: DashboardBuilders
33
---
44

5-
## Coordinator Builders
6-
7-
Build configs (at the top) and host configs (bottom) are listed here:
8-
9-
https://farmer.golang.org/builders
10-
11-
A builder runs on a certain host type. (e.g. `linux-386-387` is a build type. It runs on `host-linux-kubestd`, a Kubernetes-based linux/amd64 host)
12-
13-
They come from the file https://github.com/golang/build/blob/master/dashboard/builders.go
14-
15-
For design details about the coordinator, see https://go.dev/s/builderplan
16-
17-
Information about builder machines, how many are running and their status can be found at https://farmer.golang.org/
18-
19-
### How to set up a builder
20-
21-
1. Talk to golang-dev@ to get a builder host type & hash (they can get one from using the `golang.org/x/build/cmd/genbuilderkey` tool), and put that in ` ~/.gobuildkey` or `~/.gobuildkey-host-foo-bar` or the file pointed to by env var `$GO_BUILD_KEY_PATH`.
22-
1. Define your new builder in https://github.com/golang/build/blob/master/dashboard/builders.go with a new HostConfig and BuildConfig.
23-
1. Have golang-dev deploy the build coordinator rebuilt with the dashboard/builders.go change
24-
1. Have golang-dev modify golang.org/x/build/cmd/buildlet/Makefile to add your port and to uploads its buildlet binary to Google Cloud Storage (you can do this step out of order if your compiler changes aren't yet upstream)
25-
1. Verify you can see the new host & build configs at https://farmer.golang.org/builders
26-
1. (Interm/testing step) Test that your builder key works and you can register:
27-
1. `go get -u golang.org/x/build/cmd/buildlet`
28-
1. `buildlet -coordinator=farmer.golang.org -reverse-type=host-foo-bar -reboot=false`
29-
1. verify it shows up at https://farmer.golang.org/#pools in "Reverse pool summary" and "Reverse pool machine detail"
30-
1. Modify the golang.org/x/build/cmd/buildlet/stage0 binary if/as needed to pass the right flags to the buildlet binary.
31-
1. Put your stage0 binary on your builder, run in a loop under your operating system's process supervisor (systemd, etc). The stage0 binary is responsible for conditionally re-downloading the buildlet binary from Google Cloud Storage for each build. (This lets us evolve the build system without involving each machine owner)
32-
33-
For WIP ports, the steps above can be done out of order as needed. But as a port matures, be sure each step above is done. In particular, make sure that you're not just running a fixed copy of the buildlet binary in a loop forever. We need to be able to update it over time without your involvement. You should be running the stage0 binary (or equivalent shell script or similar for your platform) in a loop instead.
34-
35-
### Builder Requirements
36-
* Internet connection (at least be able to access Google and https://farmer.golang.org)
37-
* Preferably two or more (V)CPUs
38-
* At least 512MiB of memory (1GB or more highly recommended. 512MB might need a small `GOGC` setting to avoid thrashing.)
39-
40-
### Security notes
41-
42-
Generally, community-run builders only run code that's already been reviewed & submitted. We only enable pre-submit testing for builders run by the Go team that have a lot of hardware available. However, the [Gomote tool](https://go.dev/wiki/Gomote) is available for a number of people on the Go team and in the Go community that lets them have arbitrary access to the builders for development & debugging.
43-
44-
For paranoia reasons, you might want to run your builder in an isolated network that can't access any of your internal resources.
45-
465
## LUCI Builders
476

48-
The Go team is migrating the testing pipeline from a custom solution, the coordinator, to [LUCI](https://chromium.googlesource.com/chromium/src/+/master/docs/tour_of_luci_ui.md). [LUCI](https://chromium.googlesource.com/chromium/src/+/master/docs/tour_of_luci_ui.md) is an open source continuous integration system created by the Chrome open source team at Google. The Go team has adopted the use of LUCI in order to leverage a continuous integration solution which is used and supported by a larger group of developers. This should enable the team to provide a more featureful solution to the community.
7+
The Go team has migrated the testing pipeline from a custom solution, the coordinator, to [LUCI](https://chromium.googlesource.com/chromium/src/+/master/docs/tour_of_luci_ui.md). [LUCI](https://chromium.googlesource.com/chromium/src/+/master/docs/tour_of_luci_ui.md) is an open source continuous integration system created by the Chrome open source team at Google. The Go team has adopted the use of LUCI in order to leverage a continuous integration solution which is used and supported by a larger group of developers. This should enable the team to provide a more featureful solution to the community.
498

509
The LUCI system requires builders to run two applications which authenticate to LUCI and receive and process builds. LUCI token daemon generates a token needed to authenticate. The swarming bot uses the token to connect to LUCI and process builds.
5110

@@ -90,3 +49,25 @@ The LUCI system requires builders to run two applications which authenticate to
9049
1. `bootstrapswarm -hostname <hostname>`
9150

9251
1. Verify the bot starts up without any errors in the logs.
52+
53+
### Security notes
54+
55+
Generally, low-capacity builders only run code that's already been reviewed & submitted (post-submit testing). We only enable pre-submit testing for builders run by the Go team that have a lot of hardware available. However, the [Gomote tool](https://go.dev/wiki/Gomote) is available for a number of people on the Go team and in the Go community that lets them have arbitrary access to the builders for development & debugging.
56+
57+
For paranoia reasons, you might want to run your builder in an isolated network that can't access any of your internal resources.
58+
59+
## Coordinator Builders (legacy)
60+
61+
This section describes the custom testing solution used previously by the Go project, prior to the migration to LUCI.
62+
63+
Build configs (at the top) and host configs (bottom) are listed here:
64+
65+
https://farmer.golang.org/builders
66+
67+
A builder runs on a certain host type. (e.g. `linux-386-387` is a build type. It runs on `host-linux-kubestd`, a Kubernetes-based linux/amd64 host)
68+
69+
They come from the file https://cs.opensource.google/go/x/build/+/master:dashboard/builders.go
70+
71+
For design details about the coordinator, see https://go.dev/s/builderplan
72+
73+
Information about builder machines, how many are running and their status can be found at https://farmer.golang.org/

0 commit comments

Comments
 (0)