proto: remove or improve registration

[awalterschulze]

We started a discussion here
cf10ca0#commitcomment-20160444

[awalterschulze]

I think Registering is breaking people's setups and I am wondering why this is needed in the first place

[awalterschulze]

@bcmills says

proto.MessageType and proto.MessageName require it. So does extension support, and ptypes.DynamicAny, and there may be other cases I'm not thinking of at the moment.

[awalterschulze]

Also to be concrete here one place it is breaking people's setups
#178 (comment)

[awalterschulze]

Register fixes #179 only because ExtensionsMap was removed, which was another controversial move.

[awalterschulze]

proto.MessageType and proto.MessageName are nice features, but not really necessary.
Extension support: ok yes maybe, but sacrificing some safety can also be used to get rid of Register.
How is register needed for DynamicAny?

[bcmills]

Register fixes #179 only because ExtensionsMap was removed,

Register has ~nothing to do with #179. #179 requires having a copy of the generated Descriptor proto package; the only impact that registration has on that is to require the generated Descriptor package to be the same one that other uses of Descriptor depend on. (But that's true in general anyway: within a given binary, there should be one and only one copy of each package needed. To do otherwise leads to bloated binaries.)

[bcmills]

proto.MessageType and proto.MessageName are nice features, but not really necessary.

This is a Chesterton's Fence. If you want to remove these methods, you should start by understanding what purpose they serve (or served). "Not really necessary" merits further explanation.

Extension support: ok yes maybe, but sacrificing some safety can also be used to get rid of Register.

Extensions are already error-prone; sacrificing safety is unlikely to help most users. But can you be more specific about the tradeoff you're proposing? What kinds of errors would it fail to detect?

How is register needed for DynamicAny?

DynamicAny requires the ability to map from a fully-qualified message name to a Go type. That is, it requires proto.MessageType.

[awalterschulze]

How does having one copy of the generated Descriptor work with go vendoring?

My statement about not really necessary was a little strong, sorry. Thank you for explaining DynamicAny's usage of of proto.MessageType.

I have not really thought it through, but don't you think there could be a way to do Extensions without Register, since the user is already passing in the E_generatedextensionstruct

Ok I didn't know about the implementation of DynamicAny using proto.MessageType. Another way might be doing something similar to how extensions work. Ask the user to pass in the correct type to unmarshal Any into. But that will probably be a big change.

[bcmills]

How does having one copy of the generated Descriptor work with go vendoring?

I honestly don't know. The proto package was designed well before vendoring support and in the context of predominantly "monorepo" usage, and I personally haven't had much exposure to Go vendoring tools.

I have not really thought it through, but don't you think there could be a way to do Extensions without Register, since the user is already passing in the E_generatedextensionstruct

Yeah, it does seem possible to do extensions without Register. Thinking about it some more it's likely more of an issue for the (Google-internal) "weak" field feature (the one corresponding to FieldOptions.weak in the descriptor), which is a precursor to extensions. So it might be possible to remove registration from the open-source version of extensions.

---

That still leaves the problem of DynamicAny, though. Given AnyMessageName and UnmarshalAny, it's not obvious to me whether DynamicAny is actually all that useful, but removing it would definitely not be backward-compatible.

[rogeralsing]

I'm not sure I get the context here, but using Any types are not supported in all languages, so having the ability to simulate the same behavior using an envelope message with typename and messagedata, you still need to peek into the registered types if doing cross platform integration.

[awalterschulze]

Thanks for your really cool response @bcmills
I really appreciate it :)

I wonder whether:

• MarshalAny could have taken the messageName as input.
• UnmarshalAny could have taken in a proto.Message not of type DynamicAny and not do any name checking.

Then it doesn't seem like Any needs to call "home" to the proto package

[bcmills]

I wonder whether:

• MarshalAny could have taken the messageName as input.

Possible but error-prone. The message name is proto-path-qualified, so it isn't derivable from the Go name for the type; without registration there's no easy way to detect typo'd names.

However, MarshalAny would be easy to implement (with its current signature) without registration. Generated message types already include a method to access the file and message descriptors, which contain the path and name information. So MarshalAny is (fortunately) not a problem.

• UnmarshalAny could have taken in a proto.Message not of type DynamicAny and not do any name checking.

It already supports that usage, and the name checking could be implemented without registration using the same technique as for MarshalAny.

---

The major use-case that DynamicAny addresses is for programs that use type-switches. With DynamicAny, one can write this code:

var ext ptypes.DynamicAny
if err := ptypes.UnmarshalAny(a, &ext); err := nil {
  …
}
switch m := ext.Message.(type) {
case *somepb.SomeProtoExtension:
  …
case *otherpb.SomeOtherExtension:
  …
default:
  …
}

and the compilation process will ensure that the extensions are valid types and the (implict) names are correct. Without DynamicAny, the function becomes:

switch ptypes.AnyMessageName(a) {
case "some.proto.path.SomeProtoExtension":
  m := new(somepb.SomeProtoExtension)
  if err := ptypes.UnmarshalAny(a, m); err != nil {
    …
  }
  …

case "other.proto.path.SomeOtherExtension":
  m := new(otherpb.SomeOtherExtension)
  if err := ptypes.UnmarshalAny(a, m); err != nil {
    …
  }
  …

default:
  …
}

and now we've lost the compile-time checking on the paths: we'll have to rely on test coverage to avoid typos in the strings. (We've also introduced a bunch of redundancy in the user code: the message types already know their paths, so why should the user repeat them?)

Or, we could avoid DynamicAny by trying each possibility in sequence:

var (
  someExt somepb.SomeProtoExtension
  otherExt otherpb.SomeOtherExtension
)
switch {
case ptypes.UnmarshalAny(a, &someExt) == nil:
  …

case ptypes.UnmarshalAny(a, &otherExt) == nil:
  …

default:
  …
}

but that's O(N) for what ought to be an O(1) switch and also requires fairly sophisticated compiler optimizations to avoid wasting memory. (In my experience, one should not expect such optimizations from the gc compiler.)

There is an equivalent (and somewhat more subtle) use-case for programs which use reflection to process protocol messages using a sort of plugin architecture: currently, a package can use DynamicAny even for types that package doesn't depend on or know about. Such a package could use reflection to implement some generic operation on common fields or nested messages.

[awalterschulze]

I assume the name is the same name that is generated as a parameter to proto.RegisterType. This means that this could also be generated as a method on each type.
So then we chould have

switch ptypes.AnyMessageName(a) {
case (somepb.SomeProtoExtension{}).Name():
  m := new(somepb.SomeProtoExtension)
  if err := ptypes.UnmarshalAny(a, m); err != nil {
    …
  }
  …

case (otherpb.SomeOtherExtension{}).Name():
  m := new(otherpb.SomeOtherExtension)
  if err := ptypes.UnmarshalAny(a, m); err != nil {
    …
  }
  …

default:
  …
}

This is certainly not as simple as the type switch