Ability to disable response body logging

[tamasflamich]

Proposal

Having an additional cli switch that gives the ability to disable response body logging to stdout.

vegeta attack -skip-body 

Background

We use vegeta for load testing our large scale server solutions with a distributed attacker setup. It is important to simulate real world scenarios therefore the response of the load test requests needs to contain around 2K of data. The down side of this that logs (stdout) produced by the vegeta process can potentially become very large (tens of gigabytes of data). As a result these logs become tedious to move around.

We created a custom build of vegeta that allows to simply disable body logging.
https://github.com/Skyscanner/vegeta/commit/4bded67b7e173bd1803e47c6b8a6b280ada50a10

We thought this could be useful for the community as well, therefore, we wonder whether this (or a similar) change could be merged back to the main source.

Workarounds

[tsenart]

Related: #295

[tsenart]

After a productive off-line discussion, @xla and I have come to a design that we feel hits a sweet spot.

We propose to extend the dump command with Gob encoding capabilities and then rename it to encode, with roughly the following vegeta encode -h output.

Usage: vegeta encode INPUT_FILE_1 INPUT_FILE_2 ... INPUT_FILE_N
  -from string
    	Input encoding [gob, json, csv] (default "gob")
  -to string
        Output encoding [gob, json, csv] (default "json")
  -output string
    	Output file (default "stdout")

With this command in place, we can defer to the rich structured text based ecosystem of tools to provide the filtering functionality. Examples:

echo "GET http://:6060" | vegeta attack -rate=1000 | vegeta encode | jq 'del(.body)' | vegeta encode -from json -to gob | vegeta report

echo "GET http://:6060" | vegeta attack -rate=1000 | vegeta encode -to csv | cut -d , -f 1-6,8- | vegeta encode -from csv -to gob | vegeta report

Aspects to validate: Marshaling round-trip from any codec to any other codec doesn't lose information. Particularly, JSON is well known for having an upper bound on the numbers it can represent. See OAI/OpenAPI-Specification#704 for a good explanation of the topic. We must ensure we don't hit this cases in one way or another.

EDIT: For the caveat mentioned above, this seems like the way to handle it in Go: https://golang.org/pkg/encoding/json/#Number

[tamasflamich]

Hey @tsenart - Thank you for giving us an update. I really like this idea as - by following the unix philosophy - it gives everyone a great deal of flexibility, while keeping the attack command simple. Hopefully the performance overhead is going to be neglectable.

Looking forward to see the implementation finished.

[tsenart]

Released: https://github.com/tsenart/vegeta/releases/tag/cli/v10.0.0

[tsenart]

With the latest release you can use -max-body to define the maximum number of bytes to be read from responses.

[smartyjohn]

This doesn't really solve the scaling issue. Since vegeta still waits to write the response data (plus latency from the stdout processing), it will use gigs of memory if it cannot write the data fast enough. While -max-body skips reading of network data, you are not doing an accurate simulation of load and latency as the network portion is effectively dropped. A more effective process is what @tamasflamich outlined at the beginning; that network data is fully read, but not saved.

As an aside, the way mentioned above for csv trimming seems broken even omitting the removed -from option (I assume it's a column count mismatch in csv detection). An alternative that works:

vegeta attack ... | encode -to=csv | awk -F , -v OFS=, '{print $1, $2, $3, $4, $5, $6, "", $8, $9}' | ...

Unfortunately, all that string processing is way too heavy for high scalability.