MacOS host paths being written into binaries

[roman-kashitsyn]

While experimenting with rules_rust build reproducibility, I found that the process_wrapper binary build is not deterministic.

How I reproduced the problem:
I created 2 identical directories on my macOS machine with the following content:

$ cat panic.rs 
fn main() {
    panic!("boom!")
}

$ cat WORKSPACE.bazel 
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
    name = "rules_rust",
    sha256 = "6bfe75125e74155955d8a9854a8811365e6c0f3d33ed700bc17f39e32522c822",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/rules_rust/releases/download/0.9.0/rules_rust-v0.9.0.tar.gz",
        "https://github.com/bazelbuild/rules_rust/releases/download/0.9.0/rules_rust-v0.9.0.tar.gz",
    ],
)

load("@rules_rust//rust:repositories.bzl", "rules_rust_dependencies", "rust_register_toolchains")

rules_rust_dependencies()

rust_register_toolchains(version = "1.60.0")

$ cat BUILD.bazel 
load("@rules_rust//rust:defs.bzl", "rust_binary")

rust_binary(
    name = "panic",
    srcs = ["panic.rs"],
    edition = "2018",
)

$ cat .bazelrc 
build --@rules_rust//:source_path_prefix=/source/
build --execution_log_json_file=bazel-build-log.json

Running bazel build :panic produced slightly different binaries.
An inspection of bazel-build-log.json file showed that process_wrapper had different hashes in the two workspaces:

***************
*** 509,513 ****
      "path": "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/rules_rust_tinyjson/libtinyjson-4031717389.rlib",
      "digest": {
!       "hash": "8532cb231d123c1a6f64e3129d11461c4062c1ee499ae37654445582d8dd3e15",
        "sizeBytes": "630696",
        "hashFunctionName": "SHA-256"
--- 509,513 ----
      "path": "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/rules_rust_tinyjson/libtinyjson-4031717389.rlib",
      "digest": {
!       "hash": "2b96b6712a77e791792f9be2d7de9ac89ffda7648724ef3b121071607201eb69",
        "sizeBytes": "630696",
        "hashFunctionName": "SHA-256"
***************
*** 915,919 ****
      "path": "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/rules_rust/util/process_wrapper/process_wrapper",
      "digest": {
!       "hash": "20cbe0042ca135fad3f79c7dc37176b2d9b274d29e2abf4c7707a5eda31fab88",
        "sizeBytes": "760720",
        "hashFunctionName": "SHA-256"
--- 915,919 ----
      "path": "bazel-out/darwin-opt-exec-2B5CBBC6/bin/external/rules_rust/util/process_wrapper/process_wrapper",
      "digest": {
!       "hash": "e32aa301edcfd8ff12584c886befc2cc2a6409a399679dcd91f46e3fc58dc0a9",
        "sizeBytes": "760720",
        "hashFunctionName": "SHA-256"
***************

The source of non-determinism is likely the libtinyjson.rlib file that has different file paths embedded into it:

The first workspace had path "/tmp/sandbox/darwin-sandbox/1/execroot/main/...", the second workspace had path "/private/var/tmp/_bazel_lifted/dba7...561c/sandbox/darwin-sandbox/1/execroot/main/...".

Comment from Rosica Dejanovska:

Looks like we need to temporarily bring back a tiny part of the old C++ process wrapper that does what --subst pwd=${pwd} does. Long term rust-lang/rust#89434 should save us.

[UebelAndre]

I wonder if #1563 would help with this.

[goffrie]

I wonder if #1563 would help with this.

I don't believe so - the issue is that --remap-path-prefix depends on process_wrapper to work, but when building process_wrapper we don't have access to itself so the remapping just doesn't do anything.

However, I believe including tinyjson directly into the process_wrapper crate would work around this issue.

[bazaglia]

I've run into something that might be related. When I run a build on my Ubuntu, but still do it remotely on RBE, it succeeds. On my pipeline machine, I'd expect to see it reusing actions pre-built just like it happens in all the other programming languages rules for Bazel I use. But what I'm seeing instead is my target trying to build again even when running against the same RBE. It complains I don't have clang in the RBE worker which surprised me to see it was able to build on my Ubuntu providing the same --remote_executor I provide in the pipeline and it succeeds then.

Adding clang to my worker fixed the error but I still don't fully understand why RBE isn't working as expected in rules_rust, causing my target to rebuild when it shouldn't. It is trying to rebuild process_wrapper so when searching related issues I found this issue mentioning it isn't deterministic.

Did someone find a workaround for that?

[227 / 815] [Prepa] Compiling Rust (without process_wrapper) bin process_wrapper (6 files) [for tool]
ERROR: /home/actions/.cache/bazel/_bazel_actions/1bab7e98308a5a2a9bf4653db430f9fe/external/rules_rust/util/process_wrapper/BUILD.bazel:6:36: Compiling Rust (without process_wrapper) bin process_wrapper (6 files) [for tool] failed: (Exit 1): rustc failed: error executing command (from target @rules_rust//util/process_wrapper:process_wrapper) 
  (cd /home/actions/.cache/bazel/_bazel_actions/1bab7e98308a5a2a9bf4653db430f9fe/execroot/main && \
  exec env - \
    CARGO_CFG_TARGET_ARCH=x86_64 \
    CARGO_CFG_TARGET_OS=linux \
    CARGO_CRATE_NAME=process_wrapper \
    CARGO_MANIFEST_DIR='${pwd}/external/rules_rust/util/process_wrapper' \
    CARGO_PKG_AUTHORS='' \
    CARGO_PKG_DESCRIPTION='' \
    CARGO_PKG_HOMEPAGE='' \
    CARGO_PKG_NAME=process_wrapper \
    CARGO_PKG_VERSION=0.0.0 \
    CARGO_PKG_VERSION_MAJOR=0 \
    CARGO_PKG_VERSION_MINOR=0 \
    CARGO_PKG_VERSION_PATCH=0 \
    CARGO_PKG_VERSION_PRE='' \
    PATH=/bin:/usr/bin:/usr/local/bin \
    SYSROOT=bazel-out/k8-opt-exec-41DC9EB7/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu_tools/rust_toolchain \
  bazel-out/k8-opt-exec-41DC9EB7/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu_tools/rust_toolchain/bin/rustc external/rules_rust/util/process_wrapper/main.rs '--crate-name=process_wrapper' '--crate-type=bin' '--error-format=human' '--out-dir=bazel-out/k8-opt-exec-41DC9EB7/bin/external/rules_rust/util/process_wrapper' '--codegen=opt-level=3' '--codegen=debuginfo=0' '--remap-path-prefix=${pwd}=' '--emit=dep-info,link' '--color=always' '--target=x86_64-unknown-linux-gnu' -L bazel-out/k8-opt-exec-41DC9EB7/bin/external/rust_linux_x86_64__x86_64-unknown-linux-gnu_tools/rust_toolchain/lib/rustlib/x86_64-unknown-linux-gnu/lib '--edition=2018' '--codegen=linker=/usr/lib/llvm-11/bin/clang' --codegen 'link-args=-fuse-ld=/usr/bin/ld.gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/lib/llvm-11/bin -Wl,--gc-sections -ldl -lpthread -lstdc++ -lm' '--extern=tinyjson=bazel-out/k8-opt-exec-41DC9EB7/bin/external/rules_rust_tinyjson/libtinyjson-4031717389.rlib' '-Ldependency=bazel-out/k8-opt-exec-41DC9EB7/bin/external/rules_rust_tinyjson')
# Execution platform: @aspect_rules_js//platforms:x86_64_linux_remote
error: linker `/usr/lib/llvm-11/bin/clang` not found
  |
  = note: No such file or directory (os error 2)

[illicitonion]

@bazaglia I successfully have rules_rust working with an Ubuntu remote execution environment without clang pre-installed. My guess is you're missing some toolchain for your exec platform. Some things I'd look at for debugging:

1. The output of running with --toolchain_resolution_debug=.*rust.* - it's hard to read, but should tell you what toolchain you're ending up with on your exec platform and why.
2. Make sure you have --incompatible_enable_cc_toolchain_resolution enabled, have a relevant --extra_execution_platforms registered, and have an appropriate --extra_toolchains registered for your exec platform

[jfirebaugh]

However, I believe including tinyjson directly into the process_wrapper crate would work around this issue.

I attempted this solution. First I tried retaining the @rules_rust_tinyjson repository but using copy_file from skylib to copy individual source files at build time. This fails when building process_wrapper in an external repository:

ERROR: /private/var/tmp/_bazel_john/c71b6d936ea520e6edb21b88752a38eb/external/rules_rust/util/process_wrapper/BUILD.bazel:25:36: in rust_binary_without_process_wrapper rule @rules_rust//util/process_wrapper:process_wrapper: 
Traceback (most recent call last):
        File "/private/var/tmp/_bazel_john/c71b6d936ea520e6edb21b88752a38eb/external/rules_rust/rust/private/rust.bzl", line 332, column 42, in _rust_binary_impl
                srcs, crate_root = _transform_sources(ctx, ctx.files.srcs, getattr(ctx.file, "crate_root", None))
        File "/private/var/tmp/_bazel_john/c71b6d936ea520e6edb21b88752a38eb/external/rules_rust/rust/private/rust.bzl", line 172, column 68, in _transform_sources
                src_symlink = ctx.actions.declare_file(paths.relativize(src.short_path, package_root))
        File "/private/var/tmp/_bazel_john/c71b6d936ea520e6edb21b88752a38eb/external/bazel_skylib/lib/paths.bzl", line 186, column 17, in _relativize
                fail("Path '%s' is not beneath '%s'" % (path, start))
Error in fail: Path '../rules_rust/util/process_wrapper/flags.rs' is not beneath 'util/process_wrapper'

An alternative could be to vendor tinyjson sources (i.e. check them into the repository) rather than trying to obtain them via a @rules_rust_tinyjson repository. Is that what you had in mind?

[jfirebaugh]

By the way, does anything prevent process_wrapper itself from being susceptible to the same issue?

[goffrie]

Yes, that's what I was thinking of anyway.

…

On Fri, Dec 23, 2022, 15:20 John Firebaugh ***@***.***> wrote: However, I believe including tinyjson directly into the process_wrapper crate would work around this issue. I attempted this solution. First I tried retaining the @rules_rust_tinyjson repository but using copy_file from skylib to copy individual source files at build time. This fails when building process_wrapper in an external repository: ERROR: /private/var/tmp/_bazel_john/c71b6d936ea520e6edb21b88752a38eb/external/rules_rust/util/process_wrapper/BUILD.bazel:25:36: in rust_binary_without_process_wrapper rule @rules_rust//util/process_wrapper:process_wrapper: Traceback (most recent call last): File "/private/var/tmp/_bazel_john/c71b6d936ea520e6edb21b88752a38eb/external/rules_rust/rust/private/rust.bzl", line 332, column 42, in _rust_binary_impl srcs, crate_root = _transform_sources(ctx, ctx.files.srcs, getattr(ctx.file, "crate_root", None)) File "/private/var/tmp/_bazel_john/c71b6d936ea520e6edb21b88752a38eb/external/rules_rust/rust/private/rust.bzl", line 172, column 68, in _transform_sources src_symlink = ctx.actions.declare_file(paths.relativize(src.short_path, package_root)) File "/private/var/tmp/_bazel_john/c71b6d936ea520e6edb21b88752a38eb/external/bazel_skylib/lib/paths.bzl", line 186, column 17, in _relativize fail("Path '%s' is not beneath '%s'" % (path, start)) Error in fail: Path '../rules_rust/util/process_wrapper/flags.rs' is not beneath 'util/process_wrapper' An alternative could be to vendor tinyjson sources (i.e. check them into the repository) rather than trying to obtain them via a @rules_rust_tinyjson repository. Is that what you had in mind? — Reply to this email directly, view it on GitHub <#1530 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJO2TXUESS65DQFCD6ZBETWOYCSTANCNFSM57XZ5Y6Q> . You are receiving this because you commented.Message ID: ***@***.***>

[goffrie]

By the way, does anything prevent process_wrapper itself from being susceptible to the same issue?

AFAIK rustc only inserts full file paths when instantiating cross-crate dependencies. It otherwise uses the exact paths provided on the command line, which are workspace-relative paths in our case. This could change in the future though.

[scentini]

I have #1728 which should address the issue. It's hard to write a test for it, but does someone volunteer to do two bazel builds (with a bazel clean inbetween) and diff the process wrapper binary and the tinyjson rlib on windows and osx?