encoding/json: omitempty for arrays is confusing

[skriptble]

What version of Go are you using (go version)?

$ go version
go version go1.11.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOOS="darwin"

What did you do?

https://play.golang.org/p/NlRBFTZHFYd

func main() {
	type Foo struct {
		ID [16]byte `json:",omitempty"`
	}

	var f Foo
	j, _ := json.Marshal(f)
	fmt.Println(string(j))
}

What did you expect to see?

{}

What did you see instead?

{"ID":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]}

This is caused by isEmptyValue inside encoding/json checking to see if the length of the array is 0. For people actually using an array (and not a slice), it is unlikely that they would ever use an array of zero length. It is likely, however, that they might want to omit the result if all of the values in the array are empty. For example, if a user is using [16]byte to represent a UUID, they might want to omit encoding it if it is all zeroes. This was brought up in #11939, specifically this comment. It would be nice if we could do this generically for arrays with values we can determine are empty.

[crhntr]

An array that has a bunch of 'zero values' is not empty. The list of zeros means something. I would expect the array to be printed as is shown in the "what did you see instead" (for example a zero matrix). I played around with this a little and think this is a real issue.

My first attempt would be to implement the json.Marshaler interface for some type encapsulating [16]byte.

type UUID [16]int

func (uuid UUID) MarshalJSON() ([]byte, error) {
	for _, num := range uuid {
		if num != 0 {
			return json.Marshal([10]int(uuid))
		}
	}
	return json.Marshal(/* some zero value */) // <--- this is the real problem in my opinion.
}

Since it seams MarshalJSON is called after the value is established to not be zero, no matter what you put return from MarshalJSON, the value returned is not considered to be an empty value.

I think a json.ErrValueIsZero would be useful in this situation. So you can override the zero-ness of a type.

The only other solutions that I can think of would be to

• implement MarshalJSON for all types that embed the [16]byte
• or make all of those fields pointers to structs *[16]byte

I don't really love either of these options too much.

Notes (my playground): https://play.golang.org/p/GV8rX6f30ba

[agnivade]

IMO, it doesn't seem confusing to me. The array indeed contains 16 zeros (which are totally valid values). So why should it be {} ? It makes sense if it is a slice or a map (which is #27589), but printing zeros sounds like the right behavior to me.

If you want this behavior to be there, this should be a proposal instead of a bug report.

[skriptble]

This is a bug in the semantics of omitempty, which is why I didn't file it as a proposal. The documentation even points out specifically that there is such a thing as an empty array:

The "omitempty" option specifies that the field should be omitted from the encoding if the field has an empty value, defined as false, 0, a nil pointer, a nil interface value, and any empty array, slice, map, or string.

There aren't any caveats added there. For all the other items in the list, any type of those things is considered empty, but with arrays only specific types of arrays can be considered empty and those are arrays that it is doubtful someone would use as a struct field.

What most users who add omitempty as a struct tag to an array field want is for the zero version of an array, e.g. var id [16]byte, to be evaluated as empty.

The confusing part is the definition of an empty array. It doesn't make much sense to provide a struct tag for something that is quite useless in practice and call it out in the documentation. A struct field with [0]byte is always empty and will only ever be empty. There could be a use case for using something like [0]byte as a struct field, but it's likely extremely rare.

[mccolljr]

It's really not a semantics bug, though. Since an array is not a pointer type,[16]byte is transported around as what is functionally a single value that is 16 bytes in length. "Empty" in the context of that text you cited means len(val) == 0, in which case [16]byte is not an empty array (because it has 16 elements, in which 0 could be considered a valid element value).

What most users who add omitempty as a struct tag to an array field want is for the zero version of an array, e.g. var id [16]byte, to be evaluated as empty.

You may prefer this, but many (for example me) do not. As far as Javascript/JSON is concerned, an array with 16 0-valued elements is not an empty array, so considering it as such would be incorrect, wouldn't it? Why not just use *[16]byte or marshal the value to null if you need to treat the zero value in a special way?

The confusing part is the definition of an empty array. It doesn't make much sense to provide a struct tag for something that is quite useless in practice and call it out in the documentation. A struct field with [0]byte is always empty and will only ever be empty. There could be a use case for using something like [0]byte as a struct field, but it's likely extremely rare.

I don't disagree that you'd probably never have [0]byte in as a struct field... but what about as an interface value? It could be a sentinel value that you ascribe special meaning to. It's not common, but it certainly isn't "useless".

I do agree with @crhntr that a special error to return from json.Marshaler implementations noting that a value should be omitted would be useful.

[bcmills]

CC @rsc @dsnet @bradfitz @mvdan for encoding/json

[mvdan]

Reminds me of #28391 - in that issue, text/template's definition of whether a value is true was a bit too stiff. Seems a bit similar to how users have different expectations of what "empty" means here.

I don't personally have an opinion, but I think the current behavior of omitempty is consistent and well documented. You can argue that omitting an empty array is almost never useful, but making omitempty suddenly mean something different like "omit zero values" would likely be a breaking change.

For example, I presume that a slice with zero length and non-zero capacity is empty, but non-zero. That distinction could break dozens of programs in the wild if we suddenly change the meaning of omitempty.

[adrianbrad]

@skriptble I also run into this issue using UUID's. Did you find a workaround?
Interesting issue tho, definitely worth looking into.