Skip to content

Run perf on Windows #834

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
3 of 6 tasks
rylev opened this issue Jan 29, 2021 · 6 comments
Open
3 of 6 tasks

Run perf on Windows #834

rylev opened this issue Jan 29, 2021 · 6 comments
Labels
O-windows An issue specific to the Windows OS

Comments

@rylev
Copy link
Member

rylev commented Jan 29, 2021

It would be useful to also have the ability to run the perf suite on Windows. One open question would be whether this would run as often as the Linux tests for or if we would kick off Windows tests at a different cadence or on-demand.

This might also help with #181


There's a few different things we'll need to do to fully complete this issue:

Nice to haves:

@Mark-Simulacrum
Copy link
Member

I don't personally have familiarity with Windows, but a key piece we'd need is to determine what tooling to use in place of perf on Linux.

@wesleywiser
Copy link
Member

I've started working on this. Based on these blog posts, it looks like we can capture a variety of hardware performance counters on Windows. I've also confirmed that this technique works and I can see instructions retired for rustc.

Right now, I'm trying to get the collector building and running on Windows. I'm a bit curious @Mark-Simulacrum if you have any thoughts about how to organize the platform dependent code.

@Mark-Simulacrum
Copy link
Member

Hm, so I think there are two big, related, questions here:

  • Do we want to collect statistics on perf.rust-lang.org for Windows?
    • Do we want to "double up" statistics, or perhaps only benchmark crates like winapi or windows-rs on Windows?

I'm going to operate on the assumption that the answer to the first is potentially yes, but the answer to the second is no - we don't actually want to be constantly comparing instruction counts on linux vs windows, for example. That means that likely no or very little work needs to be done on the frontend and database schema, and we can limit to the perf-stat invocation supporting Windows equivalents.

I think my recommendation is that we don't try to add another perf collector, but rather substitute the backend in https://github.com/rust-lang/rustc-perf/blob/master/collector/src/rustc-fake.rs#L44 and try to mangle the relevant outputs in that file such that the majority of rustc-perf code doesn't even need to know it's running with "windows perf" vs. "linux perf". Does that sound plausible?

Of course, if there are more bits that are linux-specific I'm not thinking of right now, we should talk about those.

@rylev
Copy link
Member Author

rylev commented Mar 16, 2021

Do we want to "double up" statistics

I'd like to see this land before we get too deep into discussing strategies of when to run the Windows tests. I agree that it's likely overkill to run them as often as we run Linux tests, but I do however want us to run them in some sort of automated way, and I do think it's interesting to run benchmarks that only make sense on Windows like winapi/windows-rs as often as we run the existing benchmarks.

Of course, this all leads to interesting questions of which hardware we'll use to run these tests.

@wesleywiser
Copy link
Member

Do we want to collect statistics on perf.rust-lang.org for Windows?

I think that would be great but we'll need to explore how to make that actually happen of course.

Do we want to "double up" statistics, or perhaps only benchmark crates like winapi or windows-rs on Windows?

I think there is some value in "doubling up" some of the crates, not to compare Linux vs Windows but just to make sure we aren't introducing regressions that affect Windows disproportionately for some reason. For now though, I would be very happy just to get windows-rs being benchmarked.

That means that likely no or very little work needs to be done on the frontend and database schema, and we can limit to the perf-stat invocation supporting Windows equivalents.

Yeah, I agree. I'm focusing on just being able to invoke the collector on Windows.

Of course, if there are more bits that are linux-specific I'm not thinking of right now, we should talk about those.

I was more referring to stuff like this:

fn rename<P: AsRef<Path>, Q: AsRef<Path>>(from: P, to: Q) -> anyhow::Result<()> {
let (from, to) = (from.as_ref(), to.as_ref());
if fs::rename(from, to).is_err() {
// This is necessary if from and to are on different
// mount points (e.g., if /tmp is in tmpfs instead of on
// the same disk). We don't want to implement a full recursive solution
// to copying directories, so just shell out to `mv`.
let ctx = format!("mv {:?} {:?}", from, to);
let status = Command::new("mv")
.arg(from)
.arg(to)
.status()
.with_context(|| ctx.clone())?;
if !status.success() {
anyhow::bail!("mv {:?} {:?}: {:?}", from, to, status);
}
}
Ok(())
}
fn touch(root: &Path, path: &Path) -> anyhow::Result<()> {
let mut cmd = Command::new("touch");
cmd.current_dir(root).arg(path);
command_output(&mut cmd).with_context(|| format!("touching {:?} in {:?}", path, root))?;
Ok(())
}
fn touch_all(path: &Path) -> anyhow::Result<()> {
let mut cmd = Command::new("bash");
// Don't touch files in `target/`, since they're likely generated by build scripts and might be from a dependency.
// Don't touch build scripts, which confuses the wrapped rustc.
cmd.current_dir(path)
.args(&["-c", "find . -path ./target -prune -false -o -name '*.rs' | grep -v '^./build.rs$' | xargs touch"]);
command_output(&mut cmd).with_context(|| format!("touching all .rs in {:?}", path))?;
// We also delete the cmake caches to avoid errors when moving directories around.
// This might be a bit slower but at least things build
let mut cmd = Command::new("bash");
cmd.current_dir(path)
.args(&["-c", "find . -name 'CMakeCache.txt' -delete"]);
command_output(&mut cmd).with_context(|| format!("deleting cmake caches in {:?}", path))?;
Ok(())
}

where we shell out to Linux utilities. If it's feasible, I'm planning to convert these to Rust code or otherwise use standard Windows tools. I would rather not require userland Linux tools to be available to run the collector.

@Mark-Simulacrum
Copy link
Member

Yes, those should all likely be converted to Rust code - if something proves difficult to convert, we can continue shelling out to a platform-specific utility but hopefully that's not necessary for the most part.

I agree that getting support landed for running the collector manually is the first priority, just wanted to communicate that if we wanted to extend the official collection - particularly to double up testing - we'd likely need to do some database/frontend work as well, depending on the strategy used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
O-windows An issue specific to the Windows OS
Projects
None yet
Development

No branches or pull requests

3 participants