Skip to content

Commit b5eec30

Browse files
toothrottoothrot
committed
env/windows-arm64: add Ubuntu/Windows ARM64 VM configuration
This change introduces a Windows ARM64 builder image for Go, running on Ubuntu on an a1.metal instance on AWS. The AMI image successfully boots, and automatically starts a Windows ARM64 VM, exposing port 443. The buildlet still needs to be updated for the appropriate mingw path. Additionally, the qemu script does not yet automatically halt the instance on termination, which makes debugging easier. - Ubuntu was chosen over Debian, as the official Debian AMI would not boot on a1.metal instances. - a1.metal is the most affordable AWS instance to support KVM on ARM64. - A prepared qemu image exists in the Go AWS account, under s3://go-builder-data. - Only 4 processors are currently supported in order to avoid a CLOCK_WATCHDOG_TIMEOUT. We're working on investigating how to increase this. - The EC2 metadata service is proxied to the guest OS via qemu. For golang/go#42604 Change-Id: I2a505a8218c2a818f0e52d179c02e6d264d2e422 Reviewed-on: https://go-review.googlesource.com/c/build/+/321959 Trust: Alexander Rakoczy <[email protected]> Run-TryBot: Alexander Rakoczy <[email protected]> TryBot-Result: Go Bot <[email protected]> Reviewed-by: Carlos Amedee <[email protected]>
1 parent d0819ed commit b5eec30

File tree

8 files changed

+426
-0
lines changed

8 files changed

+426
-0
lines changed

‎env/windows-arm64/README.md

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# Windows buildlet images
2+
3+
Windows images are built by creating and configuring VMs hosted on AWS
4+
a1.metal instances then saving the image manually.
5+
6+
## Build and test the Windows builder image
7+
8+
- Prepare the linux QEMU host image by following the instructions in
9+
`env/windows-arm64/aws`.
10+
- Create an a1.metal instance (or other instance that supports KVM)
11+
in AWS.
12+
- Download a Windows 10 ARM64 image.
13+
- Convert vhdx images to qcow2 via the following command:
14+
15+
```shell
16+
qemu-image convert -O qcow2 win.vhdx win.qcow2
17+
```
18+
19+
- SSH to your instance tunneling port 5901, and run `win10-arm64.sh`
20+
script to boot the Windows VM.
21+
- You may need to stop the current VM: `sudo systemctl stop qemu`
22+
- VNC to the tunneled port 5901.
23+
- Open the device control panel, and use the "Search for Drivers"
24+
button to search the virtio drive `D:` for drivers.
25+
- Matching drivers will be automatically installed.
26+
- This is necessary for networking to work on Windows in qemu.
27+
- Download the `startup.ps1` script to the Windows instance, and run
28+
in PowerShell. Check thoroughly for errors.
29+
- Alternatively, you can modify `win10-arm64.sh` to forward ssh
30+
access to the VM, and run PowerShell in the CLI, which is a bit
31+
easier than through VNC.
32+
- Once the image is complete, download the image to your workstation
33+
and upload to `s3://go-builder-data`.
34+
- You can find the appropriate the S3 path referenced in
35+
`env/windows-arm64/aws/prepare_image.sh`.
36+
- Re-run packer to build an AMI with your updated Windows image.
37+
38+
### Notes
39+
40+
- `QEMU_EFI.fd` is from the `qemu-efi-aarch64` Debian package, found
41+
at `/usr/share/qemu-efi-aarch64/QEMU_EFI.fd`. It can be regenerated
42+
with the following command:
43+
44+
```shell
45+
dd if=/dev/zero of=QEMU_EFI.fd bs=1M count=64
46+
dd if=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd of=QEMU_EFI.fd bs=1M count=64 conv=notrunc
47+
```
48+
49+
- `QEMU_VARS.fd` stores saved EFI state when booting a VM. It's
50+
generated via the following command:
51+
52+
```shell
53+
dd if=/dev/zero of=QEMU_VARS.fd bs=1M count=64
54+
```
55+
56+
- The latest virtio driver image can be fetched from:
57+
https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/latest-virtio/virtio-win.iso
58+
59+
- `win10-arm64.sh` is hard-coded to run with 4 processors instead of
60+
the 16 available on an a1.metal instance. Higher numbers of
61+
processors are causing a fatal CLOCK_WATCHDOG_TIMEOUT error from
62+
interrupt requests not arriving in time. qemu-system-x86_64 has a
63+
workaround for this. We're still investigating how to increase this
64+
on aarch64.

‎env/windows-arm64/aws/Makefile

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# Copyright 2021 The Go Authors. All rights reserved.
2+
# Use of this source code is governed by a BSD-style
3+
# license that can be found in the LICENSE file.
4+
5+
env-var-check:
6+
ifndef AWS_ACCESS_KEY_ID
7+
$(error AWS_ACCESS_KEY_ID env var is not set)
8+
endif
9+
10+
ifndef AWS_SECRET_ACCESS_KEY
11+
$(error AWS_SECRET_ACCESS_KEY env var is not set)
12+
endif
13+
14+
create-aws-image: env-var-check
15+
export AWS_MAX_ATTEMPTS=600
16+
export AWS_POLL_DELAY_SECONDS=10
17+
export PACKER_LOG=1
18+
packer build -timestamp-ui packer_image_aws_arm64.json

‎env/windows-arm64/aws/README.md

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# AWS Windows ARM64 Builders
2+
3+
## Machines
4+
5+
The AWS builders use the a1 instance types which are arm64 machines.
6+
The base type used will be a1.metal, which are the cheapest which
7+
expose KVM support.
8+
9+
## Machine image
10+
11+
Machine images are stored on AWS EBS service as a snapshot. New VMs
12+
can use the snapshot as an image by providing the AMI ID as the base
13+
image when a new VM is created.
14+
15+
### Creating a new Ubuntu host image
16+
17+
Requirements:
18+
19+
Two environment variables are required to be set before initiating
20+
the command: `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` should be
21+
set with the appropriate values.
22+
23+
The [packer](https://www.packer.io) binary should be in `PATH`.
24+
25+
Command:
26+
27+
`make create-aws-image`
28+
29+
or
30+
31+
`AWS_ACCESS_KEY_ID=<id> AWS_SECRET_ACCESS_KEY=<secret> make create-aws-image`

‎env/windows-arm64/aws/packer_image_aws_arm64.json

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
{
2+
"variables": {
3+
"aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}",
4+
"aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",
5+
"region": "us-east-2"
6+
},
7+
"builders": [
8+
{
9+
"type": "amazon-ebs",
10+
"iam_instance_profile": "buildetS3ReadOnly",
11+
"access_key": "{{user `aws_access_key`}}",
12+
"ami_name": "go-linux-arm64-host-{{timestamp}}",
13+
"ami_description": "Image for linux-arm64 Go builder hosting windows-arm64",
14+
"instance_type": "a1.metal",
15+
"region": "{{user `region`}}",
16+
"secret_key": "{{user `aws_secret_key`}}",
17+
"source_ami": "ami-0b0c8ae527978b689",
18+
"decode_authorization_messages": true,
19+
"ssh_username": "ubuntu",
20+
"tags": {
21+
"Name": "Ubuntu",
22+
"Created": "{{isotime \"2006-01-02\"}}",
23+
"OS": "Ubuntu 20.04 Focal (ARM64)",
24+
"Release": "Latest",
25+
"Base_AMI_Name": "{{ .SourceAMIName }}",
26+
"Extra": "{{ .SourceAMITags.TagName }}",
27+
"Description": "{{user `description`}}"
28+
},
29+
"launch_block_device_mappings": [
30+
{
31+
"device_name": "/dev/sda1",
32+
"volume_size": 50,
33+
"volume_type": "gp2",
34+
"delete_on_termination": true
35+
}
36+
]
37+
}
38+
],
39+
"provisioners": [
40+
{
41+
"type": "file",
42+
"source": "./win10-arm64.sh",
43+
"destination": "/home/ubuntu/win10-arm64.sh"
44+
},
45+
{
46+
"type": "file",
47+
"source": "./qemu.service",
48+
"destination": "/tmp/qemu.service"
49+
},
50+
{
51+
"type": "shell",
52+
"script": "./prepare_image.sh"
53+
}
54+
]
55+
}

‎env/windows-arm64/aws/prepare_image.sh

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
#!/usr/bin/env bash
2+
3+
# Copyright 2021 The Go Authors. All rights reserved.
4+
# Use of this source code is governed by a BSD-style
5+
# license that can be found in the LICENSE file.
6+
7+
#
8+
# Installs all dependencies for an Ubuntu Linux ARM64 qemu host.
9+
#
10+
11+
set -euxo pipefail
12+
13+
TMP_DIR="$(mktemp -d)"
14+
15+
# Retry apt commands until we succeed.
16+
#
17+
# Metal instances are still being provisioned when we are first able
18+
# to connect over SSH. Some parts of apt are locked or not yet
19+
# present. Retrying a few times seems to do the trick.
20+
for i in $(seq 1 10); do
21+
# Give it a chance to finish before our first try,
22+
# and take a break between loops.
23+
sleep 1
24+
sudo apt-add-repository universe || continue
25+
26+
sudo apt-get update || continue
27+
sudo apt-get upgrade -y || continue
28+
29+
sudo apt-get install -y apt-transport-https ca-certificates curl gnupg-agent gnupg jq software-properties-common \
30+
build-essential ninja-build && break
31+
done
32+
33+
# QEMU Dependencies
34+
sudo apt-get install -y git libglib2.0-dev libfdt-dev libpixman-1-dev zlib1g-dev
35+
36+
# QEMU Extras
37+
sudo apt-get install -y git-email libaio-dev libbluetooth-dev libbrlapi-dev libbz2-dev libcap-dev libcap-ng-dev \
38+
libcurl4-gnutls-dev libgtk-3-dev libibverbs-dev libjpeg8-dev libncurses5-dev libnuma-dev librbd-dev librdmacm-dev \
39+
libsasl2-dev libsdl1.2-dev libseccomp-dev libsnappy-dev libssh2-1-dev libvde-dev libvdeplug-dev libvte-2.91-dev \
40+
libxen-dev liblzo2-dev valgrind xfslibs-dev libnfs-dev libiscsi-dev
41+
42+
# QEMU download & build
43+
wget https://download.qemu.org/qemu-6.0.0.tar.xz
44+
tar xJf qemu-6.0.0.tar.xz
45+
cd qemu-6.0.0
46+
./configure --target-list=arm-softmmu,aarch64-softmmu
47+
make -j16
48+
cd "$HOME"
49+
50+
# S3 CLI
51+
sudo apt-get install -y aws-shell
52+
53+
# Copy pre-prepared Windows 10 image
54+
aws s3 sync s3://go-builder-data/win10 "$HOME"/win10
55+
56+
chmod u+x "$HOME/win10-arm64.sh"
57+
sudo cp /tmp/qemu.service /etc/systemd/user/qemu.service
58+
sudo systemctl enable /etc/systemd/user/qemu.service
59+
sudo systemctl start qemu

‎env/windows-arm64/aws/qemu.service

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
[Unit]
2+
Description=QEMU
3+
4+
[Service]
5+
ExecStart=/home/ubuntu/win10-arm64.sh
6+
7+
[Install]
8+
WantedBy=multi-user.target

‎env/windows-arm64/aws/win10-arm64.sh

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
#!/bin/bash
2+
3+
# Copyright 2021 The Go Authors. All rights reserved.
4+
# Use of this source code is governed by a BSD-style
5+
# license that can be found in the LICENSE file.
6+
7+
/home/ubuntu/qemu-6.0.0/build/aarch64-softmmu/qemu-system-aarch64 \
8+
-name "Windows 10 ARM64" \
9+
-machine virt \
10+
-cpu host \
11+
--accel kvm \
12+
-smp 4 \
13+
-m 8G \
14+
-drive file=/home/ubuntu/win10/QEMU_EFI.fd,format=raw,if=pflash,readonly=on \
15+
-drive file=/home/ubuntu/win10/QEMU_VARS.fd,format=raw,if=pflash \
16+
-device nec-usb-xhci \
17+
-device usb-kbd,id=kbd0 \
18+
-device usb-mouse,id=tab0 \
19+
-device virtio-net,disable-legacy=on,netdev=net0,mac=54:91:05:C5:73:29,addr=08 \
20+
-netdev 'user,id=net0,hostfwd=tcp::443-:443,guestfwd=tcp:10.0.2.100:8173-cmd:netcat 169.254.169.254 80' \
21+
-device nvme,drive=hdd0,serial=hdd0 \
22+
-vnc :3 \
23+
-drive file=/home/ubuntu/win10/win10.qcow2,if=none,id=hdd0,cache=writethrough \
24+
-drive file=/home/ubuntu/win10/virtio.iso,media=cdrom,if=none,id=drivers,readonly=on \
25+
-device usb-storage,drive=drivers \
26+
-chardev file,path=/var/log/qemu-serial.log,id=char0 \
27+
-serial chardev:char0 \
28+
-device ramfb

0 commit comments

Comments
 (0)