Should unit tests spawn new processes instead of threads?

[vlovich]

Imagine a (contrived) piece of code like this:

lazy_static!(Q: Mutex<Vec<u32>> = Mutex::new(Vec::new()));

#[test]
fn test_a() {
   {
     Q.lock().unwrap().push(5);
   }
   assert_eq!(Q.lock().unwrap().len(), 1);
}

fn test_b() {
   {
     Q.lock().unwrap().clear();
   }
   assert_eq!(Q.lock().unwrap().len(), 0);

   {
     let q = Q.lock().unwrap();
     q.push(5);
     q.push(6);
   }
   assert_eq!(Q.lock().unwrap().len(), 2);
}

This is broken by default because cargo test will by default spawn multiple threads for the tests so the global state of Q. The user would have to know to pass --test-threads=1 or the test author would have to create a global mutex to synchronize each test on. This seems like a lot of unnecessary boilerplate, not to mention an unnecessary pain point for new developers to discover when they can't figure out why tests are producing weird results.

There also doesn't seem to be a solution at the moment that would enable you to achieve the benefits of running tests in parallel at all for shared state between tests. The global mutex approach effectively reduces the test code to implicitly running with --test-threads=1. Using separate processes by default (or at least providing a decorator to do it easily) would solve the vast majority of such problems without requiring authors to significantly restructure their code.

FWIW, other test frameworks go with the process sharding approach instead of threads.

[karbfg10k]

I'd really appreciate this at some point. I'm using Rust to functionally verify my companies C code base. Our code is automotive grade(runs on ECU's) so it uses lots of global's and massive functions with significant internal state (the code is auto generated from a model). Being able to spawn tests as processes would allow tests to run in parallel which would be a huge win.

[jorickert]

rusty-fork can be used to work around this issue

[Mark-Simulacrum]

cc #32512

[jrose-signal]

Poking this old issue because having an option to force this, even if it isn't the default, would be very beneficial for running a test suite under miri, which tracks all memory allocated over the lifetime of a program.

[jsenkpiel-godaddy]

It would be nice to be able to denote unit tests to have this as their behavior.

[vlovich]

I looked at rusty-fork just now and it seems incompatible with rstest which provides functionality to parameterize tests more easily.

[zopsicle]

Running tests in separate processes would also allow for #[should_abort] analogous to #[should_panic] which can be useful for testing code that should abort. (Or some generalized way to assert a certain termination status.)

[tianyicui-tsy]

it seems rusty-fork isn't compatible with tests that use tokio (i.e. #[tokio-test] instead of #[test]). Would be great if this is done.

[Cydhra]

An even more obvious use-case are libraries like MPI, since even --num-threads won't help if initializing and de-initializing it multiple times doesn't work.

[mickvangelderen]

I am tasked with writing tests against a code-base that uses global static variables with iterior mutability. I would like to ensure that one test can not influence another. While I would normally push for refactoring the code base for testability, this is not currently an option and a long term effort that requires planning which is out of my control.

Are there any known work-arounds that let me running one test in one process reliably? I had a look at rusty-fork but it seems like that could cause problems of its own. I would like to be able to debug my tests as well. Would it be possible to for someone to write an alternative test running framework that natively supports spawning a new process for each test?

[chenyuanrun]

My program is based on dpdk, which is heavily depend on global variables, when I run multiple test case and each test initialize and deinitialize the dpdk runtime, cargo test failed:

    Finished test [unoptimized + debuginfo] target(s) in 0.03s
     Running unittests src/lib.rs (/xxx/target/debug/deps/dpdk_rs-3b7f90651221b4f1)

running 2 tests
test misc::eal::test::test_handle ... EAL: Detected CPU lcores: 32
EAL: Detected NUMA nodes: 2
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: VFIO support initialized
EAL: DPDK is running on a NUMA system, but is compiled without NUMA support.
EAL: This will have adverse consequences for performance and usability.
EAL: Please use --legacy-mem option, or recompile with NUMA support.
TELEMETRY: No legacy callbacks, legacy socket not created
ok
test multicore::lcore::test::test_lcore_iter ... EAL: FATAL: already called initialization.
EAL: already called initialization.
FAILED

failures:

---- multicore::lcore::test::test_lcore_iter stdout ----
init dpdk with args: ["/xxx/target/debug/deps/dpdk_rs-3b7f90651221b4f1"]
thread 'multicore::lcore::test::test_lcore_iter' panicked at 'called `Result::unwrap()` on an `Err` value: Os { code: 114, kind: Uncategorized, message: "Operation already in progress" }', dpdk-rs/src/multicore/lcore.rs:207:46
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    multicore::lcore::test::test_lcore_iter

test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s

error: test failed, to rerun pass `--lib`

[Imberflur]

Since it isn't mentioned here, another workaround is using nextest which runs each test in a separate process.

[schickst]

We have the same issue, use a lazy_static or singleton for connection strings and other global options.
A flag like '--use-process' would be ideal.