pass fuzz.F to fuzz functions

[josharian]

The current Fuzz function signature is

func FuzzSomething(b []byte) int

I think we should migrate it to something more like:

import fuzz "github.com/dvyukov/go-fuzz"

func FuzzSomething(fz fuzz.F)

(That import path will obviously have to change if go-fuzz moves into the standard toolchain. Or if we migrate to github.com/go-fuzz/go-fuzz, or the like.)

I imagine starting fuzz.F (fuzzing.F?) with:

type F interface {
  // Bytes returns a byte slice to be used as input.
  Bytes() []byte

  // Skip tells go-fuzz that this input should not added to the corpus, and stops execution.
  // (Similar to return -1 right now.)
  // msg explains why. It is currently just a form of documentation for the user,
  // but you can imagine later gathering stats about whence all the skips.
  Skip(msg string)

  // Interesting tells go-fuzz that this input is interesting and should be given added priority.
  // Equivalent to return 1 right now, except that it does not stop execution.
  Interesting()

  // Fail reports a failure of an invariant and stops execution.
  // Equivalent to a custom panic right now.
  Fail(msg string)
  Failf(msg string, ...interface{})

  // ExitOnCompletion requests that go-fuzz exit the binary after the Fuzz function completes.
  // This dramatically impacts performance and the effectiveness of go-fuzz;
  // it should be used only when it is infeasible to write a fuzz function that can be safely
  // called multiple times.
  ExitOnCompletion()
}

There's plenty more to add, e.g. key-value-based requests for bools/ints/etc instead of having to parse them out of a byte slice. But this would be a good first start.

In order to avoid people having to change their fuzz functions, I'd automatically detect the old style of signature and have go-fuzz-build insert a shim.

Discuss. :)

(P.S. I think you had a similar proposal, Dmitry. I know that I need to go look at it. Apologies.)

[josharian]

Dmitry's original proposal is here: https://docs.google.com/document/u/1/d/1zXR-TFL3BfnceEAWytV8bnzB2Tfp6EPFinWVJ5V4QC8/pub

Relevant passage:

The support is added to the testing package.
User fuzz functions are added to _test.go files and start with Fuzz akin to tests and benchmarks. Fuzz function accepts two arguments: (*testing.F, data []byte).
The data []byte argument is the random input that the function is supposed to use in some way.
The new type testing.F merely implements testing.TB interface:
Log functions accumulate output for the duration of a single invocation; output is printed for failing invocations.
Error/Fatal/Fail functions denote a found bug.
Skip functions terminate current invocation without failing and mark the input as uninteresting.
testing.F type can later be extended with other functions if necessary.
The fuzz function signature can later be allowed to accept multiple randomly-generated arguments of different types. This is useful for fuzz tests that need multiple inputs, for example:
func FuzzRegexp(f *testing.F, re string, data []byte, posix bool) {

Looks like we're more or less on the same page here, except that I propose to keep the signature simple and uniform, and use methods to get fuzz data.

Unless there are objections, I will plan to implement my proposal and see how it looks in practice.

[dvyukov]

Looks like we're more or less on the same page here, except that I propose to keep the signature simple and uniform, and use methods to get fuzz data.

The critical part here is this:

The fuzz function signature can later be allowed to accept multiple randomly-generated arguments of different types. This is useful for fuzz tests that need multiple inputs, for example:
func FuzzRegexp(f *testing.F, re string, data []byte, posix bool) {

I believe this is the future of fuzzing and Go can pioneer here with simplicity and easy of use (as always!). Almost all current fuzzers (notably AFL, LibFuzzer) come from security background, and there you naturally operate on a blob because that's the only thing that can come from external world (network, file, etc). But currently there is a shift towards using fuzzing during normal dev process (testing, correctness, quality, CI, etc). And in this context you have a function that accepts stuff and you want to test it, so you need that stuff to come from the fuzzer. The current solutions are both complex, inflexible and inefficient.
So I would like to keep the possibility of adding more arguments open.
And if for single []byte passing it as argument of as F method is more of a matter of taste, I don't see how we can conveniently pass more arguments via F (and capture their types, statically or via reflect).

[dvyukov]

But overall I agree with idea of adding a context argument as it's much more flexible and extensible.

I always feel a bit nervous designing public stable APIs...

Skip/Fail/Failf look reasonable.

For Interesting maybe we want to pass an integer priority (how much it is interesting). But for most cases it will indeed be binary. Or maybe we want to not give this knob to user and instead rely on fuzzer becoming smarter over time?

Re ExitOnCompletion, do you have any examples? Or other name alternatives to choose from? :)
Should it accept number of times the process can be reused?... 1 will give this behavior, but can also be set to, say, 10. Perhaps this is over-designing.
On a related note, if a test says if it can be reused or not, should it also say what timeout it needs? We currently have it as a flag to go-fuzz, but it's kind of a test property.

[dvyukov]

For context, Ian's proposal of using F.Useful/Discard:
golang/go#19109 (comment)

Skip looks better than Discard, because Skip is testing.T method.

For Interesting/Useful we could also consider Priority name, esp if it accepts an int.

[dvyukov]

golang/go#19109 (comment)
also mentions:

Could there be some default convention say a _fuzz/xxx directory (where xxx corresponds with FuzzXxx) and a method on the *testing.F object to load a different corpus from the _fuzz/ directory if necessary? It seems like it should just know where the corpus is.

I am not sure it's Fuzz function responsibility to know about layout of files on disk. It's probably more of an infra responsibility (i.e. should stay as tool flag).

[dvyukov]

This is an interesting one about reusing Fuzz function in unit tests:
golang/go#19109 (comment)

A first idea is providing a FromT function that creates a stub F from testing.T to use in unit-tests.

[dvyukov]

Looking at libfuzzer flags in case there are others like -timeout which are actually test property:
https://llvm.org/docs/LibFuzzer.html#options
Potentially relevant may be:

-max_len
Maximum length of a test input. If 0 (the default), libFuzzer tries to guess a good value based on the corpus (and reports it).
-timeout
Timeout in seconds, default 1200. If an input takes longer than this timeout, the process is treated as a failure case.
-rss_limit_mb
Memory usage limit in Mb, default 2048. Use 0 to disable the limit. If an input requires more than this amount of RSS memory to execute, the process is treated as a failure case. The limit is checked in a separate thread every second. If running w/o ASAN/MSAN, you may use ‘ulimit -v’ instead.
-malloc_limit_mb
If non-zero, the fuzzer will exit if the target tries to allocate this number of Mb with one malloc call. If zero (default) same limit as rss_limit_mb is applied.
-only_ascii
If 1, generate only ASCII (isprint``+``isspace) inputs. Defaults to 0.

[josharian]

I've been thinking about your proposal:

func FuzzRegexp(f *testing.F, re string, data []byte, posix bool) {
  // use re, data, posix
}

vs what I had in mind:

func FuzzRegexp(fz fuzz.F) {
  re := fz.String("re")
  data := fz.Bytes("data")
  posix := fz.Bool("posix")
  // use re, data, posix
}

And I have come around to your suggestion. I want to record some of my thinking here, though, for future reference.

One challenge when you have anything other than just a []byte as the input is the corpus. What happens if you change your set of inputs? Does that invalidate your existing corpus? How does go-fuzz know, and what is the behavior? Is there a migration script of some kind? If you want to manually seed the corpus, how do you generate the entries?

This interacts with the Fuzz function signature: If you want some amount of resilience in your corpus (e.g. no changes required if you merely add a new input), then you need your inputs to be keyed somehow. I had had in mind providing a key when you request them (as in the code snippet below), but as you note, someone might request strings in a loop. This makes it impossible to know statically what the shape of the corpus entries is.

The downside to having this in the Fuzz function signature is that it makes the variable names significant. If you rename re to r, what does this mean for the existing corpus? I'm guessing that go-fuzz would die, saying something along the lines of "the corpus contains (re, data, posix), but you want (r, data, posix), you need to migrate (or use a different/new corpus)". This is a bit unexpected; we normally think of variable names as unimportant, whereas arguments to a function that take a key are more obviously important. It also means that there are restrictions on the Fuzz function signature (no interfaces, I'm guessing), which again is a bit weird.

The ability to statically detect the desired inputs is, to my mind, decisive.

And there are ways to work around a change in function signature without having to migrate the corpus. E.g. you could make a wrapper fuzz function that modifies the inputs and calls the old (presumably renamed) fuzz function.

So for the moment, I'm going to implement only:

func FuzzSomething(fz fuzz.F, data []byte)

with the plan to later support adding other arguments as well. (Which will requiring figuring out about the corpus.)

Incidentally, I am not sure yet whether fuzz.F should be an interface or a concrete type. I had convinced myself a while ago it had to be an interface, but I no longer remember why. When I implement it I will discover (or not, and use a struct!).

[josharian]

For Interesting maybe we want to pass an integer priority (how much it is interesting). But for most cases it will indeed be binary. Or maybe we want to not give this knob to user and instead rely on fuzzer becoming smarter over time?

I like not having knobs. And they're easier to add than drop. So I'll start without it, particularly since we aren't sure exactly what the knob should look like.

It'd be nice to run some experiments to see how much the hint helps right now, so we know what we're leaving on the table. Perhaps I will do that at some point.

[josharian]

Re ExitOnCompletion, do you have any examples? Or other name alternatives to choose from? :)

Example: the Go compiler. Refactoring it to be re-usable is a gigantic task, but it is not too hard to set up a Fuzz function entry point for it that can be called only once.

Name alternatives: I dunno. An alternative API is to a way to ask go-fuzz to exit the process without considering it a crash. Maybe f.Exit() (like os.Exit but without a rc)? Or f.ExitProcess()? Then in the compiler fuzz function you could defer f.Exit().

Should it accept number of times the process can be reused?... 1 will give this behavior, but can also be set to, say, 10. Perhaps this is over-designing.

I think so. In the general case go-fuzz should do the right thing. But in the compiler case it is hard to do without the control. (I have been using a modified version of go-fuzz.)

On a related note, if a test says if it can be reused or not, should it also say what timeout it needs? We currently have it as a flag to go-fuzz, but it's kind of a test property.

Yes, I think so. SetDeadline, perhaps? It could even be called multiple times.

Concrete use case for doing this from the Fuzz function: When fuzzing starlark, I start with the starlark interpreter but then call Python to compare results. I'd like an aggressive timeout for the first part, but then a very lax timeout for exec'ing Python, which is slow. :) I could adjust my timeout based on how long the work I'm about to do should actually take.

[josharian]

I am not sure it's Fuzz function responsibility to know about layout of files on disk.

I agree, although I do think there should be a reasonable default, and have that be based on the package and fuzz function name. (And also maybe its signature?)

Consider the case in which you have 5 fuzz functions in a package. It'd be nice to be able to call go-fuzz and have it work on all 5 without having to tell it where the corpus is for each of the 5. But we at least agree that the Fuzz function shouldn't know, so I can at least proceed on the topic at hand here. :)

[josharian]

A first idea is providing a FromT function that creates a stub F from testing.T to use in unit-tests.

Interesting, but not germane (I think) to the new fuzz function API. I'd want to think about that a bit more.

[josharian]

libfuzzer flags

Interesting. None strike me as obvious must-haves for v0. Let's do this incrementally.

[josharian]

Based on all the conversation above, I feel reasonably confident in taking a basic step: minimal fuzz.F, accept a single byte slice in the signature. I will work on that and we can add in extra little pieces as we go.