feat: explicit optional compression in dropgz #3648
Conversation
Signed-off-by: Evan Baker <[email protected]>
rbtr
requested review from
camrynl,
paulyufan2,
ashvindeodhar and
thatmattlong
as code owners
There was a problem hiding this comment.
Pull Request Overview
This PR introduces an optional decompression mechanism for embedded blobs in dropgz by adding a new compression flag. Key changes include:
- Introducing a new Compression type with supported constants (None and Gzip) in payload.go.
- Updating the Extract, deploy, and Deploy functions to accept and propagate the compression parameter.
- Adding a new CLI flag for compression in the payload command and removing outdated local-run instructions from the README.
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| dropgz/pkg/embed/payload.go | Added Compression type and integrated conditional decompression. |
| dropgz/cmd/payload.go | Updated command definitions to pass the compression parameter. |
| dropgz/README.md | Removed outdated instructions that enforced gzip compression. |
Comments suppressed due to low confidence (1)
dropgz/README.md:1
- [nitpick] The removal of local run instructions may leave users without guidance on how to use the new '--compression' flag; please update or provide alternative documentation reflecting the new behavior.
-### Running the dropgz locally
|
/azp run Azure Container Networking PR |
|
Azure Pipelines successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
If dropgz's default behavior now assumes uncompressed files, and previously we placed compressed files into the dropgz image (like inside azure-cni), do we also need to update these other dockerfiles (to place uncompressed files), or make it so that dropgz runs with gzip compression (your new flag) in pipelines etc.? Or does this compression on pertain to how the files are transferred out?
There was a problem hiding this comment.
Would it be beneficial to add a new README with updated instructions for running dropgz?
| @@ -32,7 +38,7 @@ func checksum(srcs, dests []string) error { | |||
| if len(srcs) != len(dests) { | |||
| return errors.Wrapf(embed.ErrArgsMismatched, "%d and %d", len(srcs), len(dests)) | |||
| } | |||
| rc, err := embed.Extract("sum.txt") | |||
| rc, err := embed.Extract("sum.txt", compression) | |||
There was a problem hiding this comment.
Were there issues with compression previously? Why are we adding an option to disable compression (since previously my understanding is that it was always enabled?)?
There was a problem hiding this comment.
Is it possible to test that checksum etc. is working when we don't have compression? Has it been tested manually once maybe?
There was a problem hiding this comment.
I have verified that everything still works as expected.
As for why - when I created dropgz it was to ship a bunch of blobs in a single executable that knew how to self-extract those blobs. I made it expect and require that the blobs be pre-gzipped because my use case was to put that dropgz executable in a container and use it to get CNI bins on to Nodes, and I expected that making the executable as small as possible would mean making the image as small as possible and would result in the minimal image pull time and fastest CNI ready.
This may not actually be right...
Containers are stored and transmitted compressed (as gzip by default) and the double compression was not actually reducing the final transmitted image size. Instead, it was adding compute time to decompress twice.
There was a problem hiding this comment.
Also, we try to pre-cache images in the Node VHD, so transmit time on our happy path is 0. It's a waste of time to have additional decompression when the image is already on-disk.
Yes, but dropgz is a separate go mod and is used as a library in those dockerfiles. It must be released before they can be updated, and if we make no change to them they just keep using the older version with compression. |
So let's say this PR gets merged and then we build a new cni/cns image right after-- will this image be broken because we place compressed images inside when we build dropgz, and then try to extract with the default compression as none? |
|
nothing will break or change until we release a dropgz 0.0.13, and update the CNS/CNI dockerfiles to use it |
Previously, dropgz has expected and required that embedded blobs be gzip compressed.
This change makes decompressing the stream optional and defaults to no decompression, by adding a
--compressionflag and simply streaming the embed out if compression is unset.dropgz deploynow defaults to assuming uncompressed embeds. Only gzip compression is supported and may be set.