Run build.zig logic in a WebAssembly sandbox

[andrewrk]

Extracted from #14265.

build.zig is handy for adding build logic, but it's not handy to trust a lot of different people's code running natively on the host. A deep dependency tree has the problem that many different build.zig scripts are all running, and the chance of insufficient auditing grows quickly the more dependencies there are.

This issue is to make the zig build system compile build.zig to wasm32-wasi instead of natively, and merely output its build graph, based on the user-provided build options, to stdout. At this point a separate build_runner can execute the requested build steps (or not, depending on the permissions granted).

Perhaps a middle ground, here, would be to run the make() steps directly in the WASI code. But eventually the idea would be that make() steps happen by a separate build runner, not by the wasm guest.

I can foresee a potential escape hatch for opting into running build.zig directly on the host. As an example, the Android SDK package wants to access the Windows registry in some cases. However, I would like to avoid even having this escape hatch if possible.

This issue will require Zig to gain a WASI interpreter, written in Zig. Whereever will we find one?

[ikskuh]

I really like the idea of running the build in a sandboxed environment, but right now i have some use cases that would not be possible.

The first one is that i have a custom build step that compiles resources into a desired target format:

pub const CompileDrawingStep = struct {
     const Self = @This(); 

     sdk: *const Sdk,
     step: Step,
     input_file: FileSource,
     output_file: GeneratedFile,

     pub fn getOutputFile(self: *Self) FileSource {
         return FileSource{ .generated = &self.output_file };
     }

     fn make(step: *Step) anyerror!void {
         const self = @fieldParentPtr(Self, "step", step);

         const src_path = self.input_file.getPath(self.sdk.builder);

         // Just run the input file through the TVG parser once
         {
             const file = std.fs.cwd().openFile(src_path, .{}) catch |err| {
                 std.debug.panic("Failed to open {s}: {s}", .{
                     src_path,
                     @errorName(err),
                 });
             };
             defer file.close();

             validate(self.sdk.builder.allocator, file.reader()) catch |err| {
                 std.debug.panic("{s} is not a valid TinyVG file: {s}", .{
                     src_path,
                     @errorName(err),
                 });
             };
         }

         // No need to cache it, we just verbatim pass the file through 
         self.output_file.path = src_path;
     }
  
     fn validate(allocator: std.mem.Allocator, reader: std.fs.File.Reader) !void {
         var parser = try tvg.parse(allocator, reader);
         defer parser.deinit();
  
         while (try parser.next()) |cmd| {
             _ = cmd;
         }
     }
}; 

I have several of similar build steps (one invokes a format compiler, one converts images via zigimg, ...) that will finally be bundled into a zig package:

https://github.com/Dunstwolke/core/blob/047d36ce01be3fd7ac160c2360f0139b9c0a5856/Sdk.zig#L393-L601

This stuff will be way harder with a purely declarative/sandboxed build runner.

Another example is the Zig Text Template:
https://github.com/MasterQ32/ZTT/blob/master/src/TemplateStep.zig#L43-L278

Imho the ability to do such things without having to put all of that logic into separate executables (in case of Dunstwolke compiler that isn't really possible anyways) is quite a significant feature of the Zig build system.

If we can get such things working (preopen the project directory to read/write file system access might already be enough), it would be awesome to have these things sandboxed

[kuon]

I do not know how difficult would be a zig WASM runner, but I can share my experience about writing sandboxed application.

I'll try to make the "story" short, while preserving context.

When covid vaccine arrived, authorities had to triage patient to vaccinate them in the "best" order possible. Criteria where: age, gender, medical history, location and so on.

They built a system with a "naive" approach and it quickly started to break. The problem was simple: rules where changing daily (especially people age, as they could have a birthday any day) and the database did grow to million of entries. Long story short, I was hired to find a solution.

What I came up with was to build a visual editor, a bit like excel formula editor to create rules, then those rules when compiled to a single elixir function.

And here the sandboxing problem did arise. What I did is use the editor to generate elixir AST, then I traverse this AST and whitelist functions and arguments, once done, this AST is compiled to regular elixir bytecode and executed like normal code. This did yield very good performances.

We could do something similar for build.zig and build.zon/package.zig...

1. The normal zig parser transform the source file into AST.
2. The AST is passed through a whitelisting function. This function traverse the AST and check every statement with a whitelist of function and arguments (for example, we can easily emulate a chroot with file function)
3. The code is compiled and run normally like it is now.

I've been maintaining our elixir engine and it works very well. Of course, if you allow flow control statement you are not protected against things like while(true) {} but those "attack" should be very easy to detect and stop.

[deflock]

Some projects have a concept of Trust, e.g. Visual Studio Code, direnv: on first use of something you are prompted if you trust it to run. Does it make sense to implement something like this first if I don't need sandboxing or will it annoy users?

[kuon]

Some projects have a concept of Trust, e.g. Visual Studio Code, direnv: on first use of something you are prompted if you trust it to run. Does it make sense to implement something like this first if I don't need sandboxing or will it annoy users?

My opinion is that installing a zig package should be safe in all conditions. I think we can make a list of acceptable operations for a package and create a whitelist. A whitelist also add the ability to ask the user. For example, http.get in a build.zig would prompt the user with something like package xxx is trying to download https://... allow? (yes/No), or instead we can deny it by default and add the package user to add an url whitelist to the build.zon.

I feel pretty strongly about package installation being safe, with all the CI we have and the "power" our dev machines often have, many developers in small companies can shutdown their company IT with the correct SSH command. I think we have the opportunity to do it right and we should.

[ikskuh]

@kuon most of that is actually solved by using WASI, as our runner can have a pretty tight integration into what APIs are allowed and on what directories/namespaces/... they are okay

As Zig already ships a really basic WASI runner, i guess it shouldn't be that hard to actually make a wasm interpreter that ships with Zig that can run our build scripts.

Considering the progress of the selfhosted wasm backend, that would also give us basically instant-reponse build time.

[mattnite]

As I see it, the purpose of this ticket is to protect the dev machine or CI server from a compromised package. The result of the build could be infected, but preventing that is not what we're doing here, we're protecting the build procedure itself.

@MasterQ32 has a point in that sandboxing build.zig and outputting a dependency tree eliminates a number of useful and elegant solutions that build.zig affords us. He also makes the comment that these build steps would have to be done in their own executable in that case.

If a dependency's build.zig can spawn a process or cause another program spawn a process, and it runs on the host (not sandboxed), then an attacker needs only infect the code for that executable to sidestep sandboxing.

We could sandbox make() like you mentioned @andrewrk and I think there are some interesting lines of thought down this path:

• a sandboxed build.zig can only spawn sandboxed processes
• we control WASI, the program's view of the outside world. We don't need to use system specific security features like chroot, we can fail a build if a dependency tries to touch a file outside its project directory.
  • we could sandbox C code using WASI libc too.
  • by hooking into WASI syscalls we could trivially create a build.zig specific strace (useful for auditing and debugging)
  • use an allowlist/capability system (like what @kuon mentioned) to allow specific build runners to do specific actions, such as special file or network access.

Policy

Creating policy for capability based systems like this is hard. We have a pretty low-level API for the boundary of the sandbox, and this is the layer at which policy is enforced. So how do we help protect developers while letting them get on with their day?

One thing to keep in mind is that many developers will have pure Zig dependencies. Many C projects will be able to set up their artifacts without reaching into the system. In these cases we wouldn't bump into the sandbox boundary.

By default we could allow a build.zig to access its project directory and network access would be disallowed. This might also cover a large number of projects meaning that many developers will be far safer with zero impact on UX.

From there a developer could modify the root project's allowlist if they did legitimately need a network request. We could even limit where that build script could make connections to. We could fail a build if a connect attempt is made, or provide prompts as @kuon outlined.

Capabilities could be limited to a single dependency so that different build.zig have different "permissions".

The policy could also be part of an application. So the act of managing what dependency capabilities could be kept to when someone updates packages. A project would declare what dependency can do what, and for someone cloning a fresh repo, they'd be able to build the project without having to understand the security requirements of the dependencies of a project. This goes for the CI server scenario as well.

Profiling

Instead of a long line of prompts, another solution could be to run a build in "permissive" mode, which lets it do whatever it wants, and then the compiler takes note of all the activity generated by the child processes, and sets the capabilities automatically. This is a form of trust-on-first-use.