Run perf on Windows

[rylev]

It would be useful to also have the ability to run the perf suite on Windows. One open question would be whether this would run as often as the Linux tests for or if we would kick off Windows tests at a different cadence or on-demand.

This might also help with #181

---

There's a few different things we'll need to do to fully complete this issue:

• Just get the code building on Windows Add support for building rustc-perf and running the collector on Windows #865
• Make the collector work in its most basic mode (probably --self-profile) Add support for building rustc-perf and running the collector on Windows #865
• Use xperf + ETW to capture hardware performance counter metrics like instructions retired. Support collecting hardware performance counters on Windows #885
• Coordinate running a Windows box for perf.rlo + set it up
• Add some benchmarks for the Windows box to collect (win-rs or a subset of it might be a good candidate).

Nice to haves:

• [Windows] Implement support for the cpu-clock perf pseudo-counter #897

[Mark-Simulacrum]

I don't personally have familiarity with Windows, but a key piece we'd need is to determine what tooling to use in place of perf on Linux.

[wesleywiser]

I've started working on this. Based on these blog posts, it looks like we can capture a variety of hardware performance counters on Windows. I've also confirmed that this technique works and I can see instructions retired for rustc.

Right now, I'm trying to get the collector building and running on Windows. I'm a bit curious @Mark-Simulacrum if you have any thoughts about how to organize the platform dependent code.

[Mark-Simulacrum]

Hm, so I think there are two big, related, questions here:

• Do we want to collect statistics on perf.rust-lang.org for Windows?
  • Do we want to "double up" statistics, or perhaps only benchmark crates like winapi or windows-rs on Windows?

I'm going to operate on the assumption that the answer to the first is potentially yes, but the answer to the second is no - we don't actually want to be constantly comparing instruction counts on linux vs windows, for example. That means that likely no or very little work needs to be done on the frontend and database schema, and we can limit to the perf-stat invocation supporting Windows equivalents.

I think my recommendation is that we don't try to add another perf collector, but rather substitute the backend in https://github.com/rust-lang/rustc-perf/blob/master/collector/src/rustc-fake.rs#L44 and try to mangle the relevant outputs in that file such that the majority of rustc-perf code doesn't even need to know it's running with "windows perf" vs. "linux perf". Does that sound plausible?

Of course, if there are more bits that are linux-specific I'm not thinking of right now, we should talk about those.

[rylev]

Do we want to "double up" statistics

I'd like to see this land before we get too deep into discussing strategies of when to run the Windows tests. I agree that it's likely overkill to run them as often as we run Linux tests, but I do however want us to run them in some sort of automated way, and I do think it's interesting to run benchmarks that only make sense on Windows like winapi/windows-rs as often as we run the existing benchmarks.

Of course, this all leads to interesting questions of which hardware we'll use to run these tests.

[wesleywiser]

Do we want to collect statistics on perf.rust-lang.org for Windows?

I think that would be great but we'll need to explore how to make that actually happen of course.

Do we want to "double up" statistics, or perhaps only benchmark crates like winapi or windows-rs on Windows?

I think there is some value in "doubling up" some of the crates, not to compare Linux vs Windows but just to make sure we aren't introducing regressions that affect Windows disproportionately for some reason. For now though, I would be very happy just to get windows-rs being benchmarked.

That means that likely no or very little work needs to be done on the frontend and database schema, and we can limit to the perf-stat invocation supporting Windows equivalents.

Yeah, I agree. I'm focusing on just being able to invoke the collector on Windows.

Of course, if there are more bits that are linux-specific I'm not thinking of right now, we should talk about those.

I was more referring to stuff like this:

rustc-perf/collector/src/execute.rs

Lines 26 to 68 in c52ee62

	fn rename<P: AsRef<Path>, Q: AsRef<Path>>(from: P, to: Q) -> anyhow::Result<()> {
	let (from, to) = (from.as_ref(), to.as_ref());
	if fs::rename(from, to).is_err() {
	// This is necessary if from and to are on different
	// mount points (e.g., if /tmp is in tmpfs instead of on
	// the same disk). We don't want to implement a full recursive solution
	// to copying directories, so just shell out to `mv`.
	let ctx = format!("mv {:?} {:?}", from, to);
	let status = Command::new("mv")
	.arg(from)
	.arg(to)
	.status()
	.with_context(|| ctx.clone())?;
	if !status.success() {
	anyhow::bail!("mv {:?} {:?}: {:?}", from, to, status);
	}
	}
	Ok(())
	}
	fn touch(root: &Path, path: &Path) -> anyhow::Result<()> {
	let mut cmd = Command::new("touch");
	cmd.current_dir(root).arg(path);
	command_output(&mut cmd).with_context(|| format!("touching {:?} in {:?}", path, root))?;
	Ok(())
	}
	fn touch_all(path: &Path) -> anyhow::Result<()> {
	let mut cmd = Command::new("bash");
	// Don't touch files in `target/`, since they're likely generated by build scripts and might be from a dependency.
	// Don't touch build scripts, which confuses the wrapped rustc.
	cmd.current_dir(path)
	.args(&["-c", "find . -path ./target -prune -false -o -name '*.rs' | grep -v '^./build.rs$' | xargs touch"]);
	command_output(&mut cmd).with_context(|| format!("touching all .rs in {:?}", path))?;
	// We also delete the cmake caches to avoid errors when moving directories around.
	// This might be a bit slower but at least things build
	let mut cmd = Command::new("bash");
	cmd.current_dir(path)
	.args(&["-c", "find . -name 'CMakeCache.txt' -delete"]);
	command_output(&mut cmd).with_context(|| format!("deleting cmake caches in {:?}", path))?;
	Ok(())
	}

where we shell out to Linux utilities. If it's feasible, I'm planning to convert these to Rust code or otherwise use standard Windows tools. I would rather not require userland Linux tools to be available to run the collector.

[Mark-Simulacrum]

Yes, those should all likely be converted to Rust code - if something proves difficult to convert, we can continue shelling out to a platform-specific utility but hopefully that's not necessary for the most part.

I agree that getting support landed for running the collector manually is the first priority, just wanted to communicate that if we wanted to extend the official collection - particularly to double up testing - we'd likely need to do some database/frontend work as well, depending on the strategy used.