Serialization format

[MichaelMure]

Bug's data are stored using git Commit, Tree and Blob. Inside a Blob is serialized an OperationPack, that is an array of edit operation on the bug's state.

This OperationPack is currently serialized using golang's gob, which is neat because it just works. However, it might not be the best option for interoperability with other tools in the future.

How should that be serialized ? Json ? In any case, git will compress the data using zlib so a text format might not be that terrible.

Feel free to argue a case here.

[daurnimator]

I just found the project via hackernews. I'd love to give this sort of thing a try, and integrate it into other tools. However using go serialisation rules out all my languages of choice.

I'd say use something JSON based, or if that's not enough, CBOR.

[MichaelMure]

For the record, it would be an easy change to use something else, it's only a few lines of code: https://github.com/MichaelMure/git-bug/blob/d959acc29dcbc467790ae87389f9569bb830c8c6/bug/operation_pack.go#L25-L52

[lukechampine]

My two cents: gob is convenient and efficient, but not a great choice if you want interop with other languages. Unfortunately there just aren't many binary formats that are widely supported, except perhaps protobufs.

JSON is probably your best bet. As you noted, it will be compressed anyway, and if performance is an issue you can always switch to a faster JSON encoder. The only big downside to JSON that I'm aware of is poor support for encoding binary blobs (encoding/json encodes []byte as a base-64 string). But if OperationPack is almost entirely textual data anyway, there's little reason to worry about that.

[avar]

Git does its own delta-compression on top of zlib. You should decide this using a combination of whatever format needs you have (can you add more fields, is it extensible etc.), and how git manages to compress this using both delta compression and zlib, which you can figure out using a large enough set of realistic test data.

[MichaelMure]

To give more details about the requirement, OperationPack hold currently very simple data (string, int, array..), and it's likely to stay the same even when adding new operations. For instance, embedded files are stored in git blobs and then linked in the git tree.

The only tricky part is that an OperationPack is an mixed array of Operation so the decoder need to support that and match the correct go struct for each operation.

[MichaelMure]

With all these format that could fit the bill, the best way to choose would be a benchmark for both performance and blob size for several format (at least JSON and CBOR). Who knows how the git compression behave on something that is already binary.

Maybe the git people could do an educated guess.

[j-f1]

MessagePack is another option, but I feel like MessagePack and CBOR are both designed for getting the smallest possible representation of data, whereas JSON is designed to be human-readable, ASCII-compatible, and simple to parse. Compare JSON’s spec (the sidebar) with the CBOR and MessagePack specs.

[MichaelMure]

I wrote some throwaway code to test the resulting blob size for various format. Here is one run:

Creating repo: /tmp/512275589

GOB
raw: 5210, git: 2216, ratio: 42.53359%
raw: 5536, git: 2320, ratio: 41.907513%
raw: 3987, git: 1768, ratio: 44.34412%
raw: 4407, git: 1893, ratio: 42.95439%
raw: 6368, git: 2593, ratio: 40.71922%
raw: 4905, git: 2143, ratio: 43.690113%
raw: 6524, git: 2660, ratio: 40.772533%
raw: 3315, git: 1549, ratio: 46.726997%
raw: 4116, git: 1780, ratio: 43.24587%
raw: 3928, git: 1751, ratio: 44.577393%
total: 20673

JSON
raw: 4862, git: 1966, ratio: 40.436035%
raw: 5188, git: 2072, ratio: 39.93832%
raw: 3633, git: 1528, ratio: 42.058907%
raw: 4055, git: 1657, ratio: 40.863132%
raw: 6026, git: 2339, ratio: 38.815136%
raw: 4555, git: 1903, ratio: 41.778267%
raw: 6184, git: 2411, ratio: 38.987713%
raw: 2965, git: 1314, ratio: 44.31703%
raw: 3764, git: 1551, ratio: 41.20616%
raw: 3568, git: 1515, ratio: 42.460762%
total: 18256

CBOR
raw: 4746, git: 1961, ratio: 41.319008%
raw: 5071, git: 2065, ratio: 40.72175%
raw: 3524, git: 1527, ratio: 43.33144%
raw: 3944, git: 1656, ratio: 41.98783%
raw: 5902, git: 2337, ratio: 39.59675%
raw: 4440, git: 1899, ratio: 42.77027%
raw: 6062, git: 2410, ratio: 39.755856%
raw: 2852, git: 1308, ratio: 45.862553%
raw: 3652, git: 1543, ratio: 42.25082%
raw: 3463, git: 1507, ratio: 43.51718%
total: 18213

MsgPack
raw: 4746, git: 1980, ratio: 41.71934%
raw: 5072, git: 2087, ratio: 41.147476%
raw: 3521, git: 1541, ratio: 43.765976%
raw: 3941, git: 1665, ratio: 42.24816%
raw: 5902, git: 2357, ratio: 39.935616%
raw: 4439, git: 1914, ratio: 43.117817%
raw: 6060, git: 2425, ratio: 40.016502%
raw: 2853, git: 1323, ratio: 46.372242%
raw: 3654, git: 1558, ratio: 42.638203%
raw: 3464, git: 1526, ratio: 44.053116%
total: 18376

As expected, there is not that much differences after encoding + compression. CBOR consistently win the size contest though.

[MichaelMure]

Note: each serialization format is tested on the same set of randomly generated OperationPack with one Create and 4 AddComment ops.

[avar]

@MichaelMure This test case really isn't meaningful. You're just testing how a given payload compresses with zlib when creating loose objects, since when you add a new object it's compressed, a header is added to it, and it's added to the object store.

Instead, you should after every addition do git add && git commit && git gc. Then measure the total size of the now-packed .git/objects directory, not individual objects.

At that point, these objects will be delta-compressed, so you can see how the size of the repo grows as they're added.

The size of individual objects is pretty much irrelevant. You can have 10 objects that are all 1GB, but delta-compress down to 1GB + 1MB or whatever, or 10GB if they don't delta-compress at all.

[avar]

@MichaelMure Also in reply to:

Who knows how the git compression behave on something that is already binary.

I'm sure there's some obscure edge case where the compression is tweaked for textual content in some way that'll prove me wrong, but in general this doesn't matter at all.

Git's just as good at delta-compressing binary data and non-binary data. What it's not good at compressing (and this goes for any compression), is data that's wildly different from one object to the next.

It just so happens that generally binary data is less delta-compressible, think say two *.mp3s with different songs v.s. a *.txt change to its lyrics.

But for these sort of pack formats I wouldn't expect them to delta-compress any worse than say JSON. It's going to be other things that matter, e.g. let's say you use a JSON encoder where the keys of the payload aren't sorted, and thus are different every time, that'll compress worse than if they're sorted, same for doing the same in some binary key-value format.

I do think that for UI purposes it makes sense to pick a widely implemented & used text format like JSON for introspection purposes and the availability of tooling (e.g. jq), if the compression numbers for it aren't much worse that is.

[MichaelMure]

That's a good point, I'll check the repo size as well, before and after a git gc.