Evil Test Payload

[crepererum]

What

Create some evil payload that tests that our sandbox is tight.

Why

Better test things ourselves than rely on attackers.

How

Create a payload that tries to do at least the following evil things:

• panic/unwind: feat: infrastructure for "evil" test payloads #162
• write loads of data to stdout/stderr: feat: infrastructure for "evil" test payloads #162
• create an unlimited stack via recursion: feat: infrastructure for "evil" test payloads #162
• allocate a lot of in-VM memory: feat: limit memory consumption #185
• pass a lot of data to the host
• create invalid arrow structures (for Array and DataType) and pass that to the host
• fill the in-VM root file system with loads of garbage: test: extend evil test suite #171 & feat: limit memory consumption #185
• try to reach localhost and local network IPs via network
• loop forever and burn CPU time: feat: epoch-based interruption #173
• dereference invalid pointer (like nullptr): feat: infrastructure for "evil" test payloads #162
• emit large values
  • UDF name
  • many input fields
  • deeply nested types

[jdstrand]

This is a nice list!

try to reach localhost and local network IPs via network

You probably already have this in mind, but if we are going to allow networking out (which according to the spec which discusses HTTP requests, recall what we had to do for flux in C2: https://github.com/influxdata/flux/blob/76ef9d5641ec997b891d2df1e33a49ddeab6ebdd/dependencies/url/validator.go#L64 (ie, be sure to also consider the special IPv6 addresses too).

[jdstrand]

fill the in-VM root file system with loads of garbage

It is also work adding an explicit test that files on the host are inaccessible (eg, /etc/passwd, but also named sockets, etc) as well as virtual filesystems like /proc (eg, /proc/<pid>/environ, etc).

Abstract sockets are also something to look out for.

[jdstrand]

Does the wasm sandbox provide a allow list of syscalls? It would be good to ensure that python can't use syscall()/etc to talk to the kernel to do stuff like chmod, execve (I assume we won't allow this), etc but also things like ioctl, mq_*, etc.

[jdstrand]

Compression bombs would be another thing to consider to add to the list.

[crepererum]

fill the in-VM root file system with loads of garbage

It is also work adding an explicit test that files on the host are inaccessible (eg, /etc/passwd, but also named sockets, etc) as well as virtual filesystems like /proc (eg, /proc/<pid>/environ, etc).

As of #126, the guest no longer has ANY access to the host FS. We should however at least add a test that proofs that.

You probably already have this in mind, but if we are going to allow networking out (which according to the spec which discusses HTTP requests, recall what we had to do for flux in C2: https://github.com/influxdata/flux/blob/76ef9d5641ec997b891d2df1e33a49ddeab6ebdd/dependencies/url/validator.go#L64 (ie, be sure to also consider the special IPv6 addresses too).

We're currently implementing "allow by hostname-port-method" and I think that's a bit better than filtering IP addresses TBH.

Abstract sockets are also something to look out for.

There's NO actual socket support for guests. Only the special HTTP WASI APIs are allowed and filtered as described above. A guest cannot even implement their own HTTP client based on raw TCP/UDP.

Does the wasm sandbox provide a allow list of syscalls? It would be good to ensure that python can't use syscall()/etc to talk to the kernel to do stuff like chmod, execve (I assume we won't allow this), etc but also things like ioctl, mq_*, etc.

There is no syscall interface in WASM. By design, you cannot perform any (there's no WASM bytecode sequence that would construct/issue a syscall).

Compression bombs would be another thing to consider to add to the list.

👍

[jdstrand]

We're currently implementing "allow by hostname-port-method" and I think that's a bit better than filtering IP addresses TBH.

How will this work in a general purpose manner? Perhaps I'm misunderstanding, but customers will likely need to be able to specify what hosts are allowed. Will we provide a mechanism for them to do that?

[crepererum]

We're currently implementing "allow by hostname-port-method" and I think that's a bit better than filtering IP addresses TBH.

How will this work in a general purpose manner? Perhaps I'm misunderstanding, but customers will likely need to be able to specify what hosts are allowed. Will we provide a mechanism for them to do that?

We'll likely expose a config option that the customer can set. I think there might be cases where they want to approve specific internal hosts too (esp. for on-prem).

[jdstrand]

@crepererum - thanks again for this issue. In reading more about this (and as you are obviously aware), it is on us to configure the sandbox with the desired security properties within the influxdb code and so having a mechanism to confirm that we have the resource controls, network, filesystem, memory isolation, syscalls, etc, etc, etc in place is paramount for the MVP, but also going forward to ensure we don't regress accidentally turn one of these off/etc. Were you envisioning a unit test? Functional? Something that runs during the build? Something separate in CI?

[crepererum]

There's a multi-layer approach here:

• unit tests: Use the "evil payload" described in this ticket as some kind of smoke test. This mostly ensures what the expected behavior of a potential violation is. It also acts as some kind of demonstration. To some extend, it also ensures that we set the configuration toggles for wasmtime correctly (see next point).
• wasmtime / the WebAssembly construct: We rely on wasmtime to do the actual heavy lifting, similar as infrastructure teams rely on crun, podman and Co to implement containers properly. Of course there's always a risk that said library will have a bug and allow attackers to escape.

I'm not sure if that fully answers your question, but I we may also have a call in case that helps.

[jdstrand]

I'm not sure if that fully answers your question, but I we may also have a call in case that helps.

It does. To be clear, I also don't think that we need to specially test wasmtime/the WebAssembly construct since this is an established upstream, we'll get RUSTSEC advisories, etc.

I do want us to ensure that we are correctly configuring it for the security properties we desire and having the evil payload tests in unit tests is an excellent start since people will immediately see if they broke something, some dependency change breaks something, etc.

The only thing with unit tests is that we aren't testing the end result that users see (ie, the build artifact) and so it is conceptually possible to have the unit tests pass but the resulting builds somehow leave the sandbox open. I'm not familiar enough with the code to say how likely this is, but the impact of getting it wrong suggests to me also having an additional test that feeds the evil payload into a binary build (eg, via CI) is worthwhile. Ideally we'd have this at the MVP, but if we have the unit tests in place, it could be a future improvement IMO.

[crepererum]

The only thing with unit tests is that we aren't testing the end result that users see (ie, the build artifact) and so it is conceptually possible to have the unit tests pass but the resulting builds somehow leave the sandbox open. I'm not familiar enough with the code to say how likely this is, but the impact of getting it wrong suggests to me also having an additional test that feeds the evil payload into a binary build (eg, via CI) is worthwhile. Ideally we'd have this at the MVP, but if we have the unit tests in place, it could be a future improvement IMO.

I'm not following you here 😅 . The evil test payload will be used with the same host code that we use in production. I cannot see how someone would manage to get a different production host with a different configuration out of this. To be specific: the configuration toggles aren't exposed to the library user, we deliberately hide them and set them to "safe defaults". Maybe let's talk about this again once the PR for this ticket is up, because I think it makes things clearer.

[jdstrand]

We're currently implementing "allow by hostname-port-method" and I think that's a bit better than filtering IP addresses TBH.

How will this work in a general purpose manner? Perhaps I'm misunderstanding, but customers will likely need to be able to specify what hosts are allowed. Will we provide a mechanism for them to do that?

We'll likely expose a config option that the customer can set. I think there might be cases where they want to approve specific internal hosts too (esp. for on-prem).

This sounds fine and documentation can cover security considerations when declaring the hosts. Since you want a declarative list, it sounds like you are taking a 'default deny' approach where the default blocks all networking and the user is required to say what they want to allow? (This is great from a security POV)

[jdstrand]

Another thing to think through (perhaps you've already done it): DoS surface. Couple of things OTOH:

• we'll be configuring the sandbox for a certain filesystem size (and that filesystem lives in memory). AIUI UDF invocations will be isolated from each other; does each UDF get its own filesystem? Is there a concept of 'base filesystem' that is shared between UDFs with writes as an overlay that is specific to each UDF? Thinking through someone sending a bunch of queries that fills the (in memory) virtual fs and the potential impact on the host (when all those queries' fs sizes are added together)
• we'll be configuring the sandbox for an upper memory limit. What happens if someone sends in a bunch of queries that each uses up all the memory they can and the effect on the host (when all those queries' memory usage are added together)
• we'll be configuring the sandbox for an upper CPU limit. What happens if someone sends in a bunch of queries that each uses up all the CPU they can and the effect on the host

Conceptually, the above isn't different than sending a bunch of non-UDF queries which make DF use a lot of memory or CPU, but UDFs seem to magnify this a bit (perhaps I'm thinking about it wrong?). Will there be service protections in place for this sort of thing? Rate-limiting? Something else?

[jdstrand]

I'm not following you here 😅 . The evil test payload will be used with the same host code that we use in production. I cannot see how someone would manage to get a different production host with a different configuration out of this. To be specific: the configuration toggles aren't exposed to the library user, we deliberately hide them and set them to "safe defaults". Maybe let's talk about this again once the PR for this ticket is up, because I think it makes things clearer.

Sorry. What I was thinking through was when I think of unit tests, I think of discrete pieces of code that are tested in isolation, which is great. However, something higher in the stack must use those pieces and if a code change affects the higher level code, the lower level unit tests might all pass but the higher level code skips using the code that was tested (perhaps via a compile time flag, perhaps a bug, etc). As a naive example, say you have a function configure_strict_sandbox() and you have unit tests for configure_strict_sandbox() that verify it is bulletproof. Somewhere else higher in the code on startup something calls configure_strict_sandbox() and all is well with the world. Then next year, a refactor changes things and configure_strict_sandbox() isn't called on startup (or similar). The unit tests will still pass, but the resulting binary will not have called configure_strict_sandbox(). Some sort of test with the resulting binaries would catch this.

Of course, there are probably compiler checks (dead code) and unit tests higher in the stack that would test for this so maybe it isn't a big risk, but hopefully you see the type of thing I'm thinking through. This CI thing I'm talking about is just a backstop in case we got something wrong in the unit tests. Since the impact of a misconfigured sandbox is high, I'm thinking that an added CI test could be worthwhile even if the risk of it catching something is low.

Anyway, we don't have to discuss this here and like you said, we can talk about it in the PR.