Use and effects of @JsonIdentityInfo for recursive json schema definitions

[scottslewis]

I'm a little new to the Jackson API, although very experienced in Java.

I'm working on a proposal for adding Groups to the MCP protocol. The mcp-java-sdk uses Jackson (2.19.something I think).

In the Groups proposal I've suggested introducing a nested Group structure, by adding a 'parent' Group attribute. This provides for arbitrary hierarchy of groups.

With normal serialization/deserialization of json with this schema the Group.parent recursive/nested structure could produce a lot of duplication of groups (for example if the same nested hierarchy of Groups was assigned to multiple Tools).

I've heard that the @JsonIdentityInfo annotation can be used for this situation, to replace all but the first given Group instance with a json reference, so that on deserialization the Group tree structure is reconstituted by clients.

I've read the docs for @JsonIdentityInfo and it says this:

...In practice this is done by serializing the first instance as full object and object identity, and other references to the object as reference values....

My question is: Is adding @JsonIdentityInfo to the Group type...e.g. here all that's necessary to get this 'in practice' behavior? (i.e. serialize to json the first reference to a given Group, and send a json reference for all other references?). Or are there other options required for getting this in practice behavior (and/or changing/customizing that behavior)?

[cowtowncoder]

Without knowing full details/intents I could be wrong, but I think use of @JsonIdentityInfo with proper settings should be sufficient, yes.
(I couldn't see what "here" is meant to link to fwtw)

Also quick note on "in practice": that's clumsy way of saying "this is how Jackson implements it" (but implying other implementations could produce output that is not necessarily ordered this way).

[cowtowncoder]

Sounds great! I'll reply in longer form in a bit after lunch.

[cowtowncoder]

If anyone had time and interest to document their learnings, that would be a very valuable source: I could add a markdown doc to this repo (f.ex) or one of other Jackson repos.

I volunteer to do this. I'

Excellent! Based on your experience, work, you are in a very good position and I really appreciate your help.
And want to help best I can.

Before encountering the @JsonIdentityInfo I was thinking that it would have to be a json reference, but perhaps I
don't understand the json spec well enough.

For better or worse basic json spec really has nothing to say about types (beyond json primitives) or object identity (unlike say YAML).
So Jackson had to kind of figure out how to support emerging conventions (wrt @JsonTypeInfo and @JsonIdentityInfo, respectively).

I'm willing and able to document the existing mechanisms and/or propose enhancements to use specified
approaches, if you are willing to help me navigate the codebase and documentation. FWIW I've been using Java
for 30 years now (ugh), and am the project lead for the reference impl of OSGi Remote Services/RSA
specifications (Eclipse Communication Framework), so I'm familiar with impl, docs, and spec work.

Great!

Please let me know if this works for you.. I'll get databind and dependent source code repos.
If you can/will point me to the right classes (in addition to the ones you've pointed me at above) then I'll get started. Thx.

That does work. I think all you need should be jackson-annotations and jackson-databind (note: you probably want 3.x branch, which is default -- 2.x and 3.x versions have no functional differences in this area but 3.x will be more actively worked on):

• https://github.com/FasterXML/jackson-annotations/ for @JsonIdentityInfo and types it refers (values for properties)
  • usually only annotation types included here, but due to need to specify handlers, handler (id generator, resolver) interfaces included
• https://github.com/FasterXML/jackson-databind/
  • JacksonAnnotationIntrospector:
    • public ObjectIdInfo findObjectIdInfo(MapperConfig<?> config, Annotated ann) is used to extract annotations, package as ObjectIdInfo
  • Searching for ObjectIdInfo, it gets used in many places unfortunately: originally collected by POJOPropertiesCollector to be eventually used by:
    • BeanDeserializerBase (base for BeanDeserializer)
    • BeanSerializerBase (base for BeanSeserializer)
  • ObjectIdReader, ObjectIdWriter are specific handler for actual Object Ids included in POJO serializations
  • Tests under src/test/java/tools/jackson/databind/objectid/ (like src/test/java/tools/jackson/databind/objectid/TestObjectIdWithPolymorphic.java) may be useful

I hope these help as starting points. I am happy to answer follow-up questions as necessary.

[scottslewis]

> For better or worse basic json spec really has nothing to say about types (beyond json primitives) or object identity (unlike say YAML). So Jackson had to kind of figure out how to support emerging conventions (wrt `@JsonTypeInfo` and `@JsonIdentityInfo`, respectively).

I was referring to the spec json reference (and associated json pointer: https://json-spec.readthedocs.io/reference.html

> That does work. I think all you need should be `jackson-annotations` and `jackson-databind` (note: you > probably want `3.x` branch, which is default -- 2.x and 3.x versions have no functional differences in this > area but 3.x will be more actively worked on): > > * https://github.com/FasterXML/jackson-annotations/ for `@JsonIdentityInfo` and types it refers (values for properties) > > * usually only annotation types included here, but due to need to specify handlers, handler (id generator, resolver) interfaces included > > * https://github.com/FasterXML/jackson-databind/ > * `JacksonAnnotationIntrospector`: > * `public ObjectIdInfo findObjectIdInfo(MapperConfig config, Annotated ann)` is used to extract annotations, package as `ObjectIdInfo` > * Searching for `ObjectIdInfo`, it gets used in many places unfortunately: originally collected by `POJOPropertiesCollector` to be eventually used by: > > * `BeanDeserializerBase` (base for `BeanDeserializer`) > * `BeanSerializerBase` (base for `BeanSeserializer`) > * `ObjectIdReader`, `ObjectIdWriter` are specific handler for actual Object Ids included in POJO serializations > * Tests under `src/test/java/tools/jackson/databind/objectid/` (like `src/test/java/tools/jackson/databind/objectid/TestObjectIdWithPolymorphic.java`) may be useful > > I hope these help as starting points. I am happy to answer follow-up questions as necessary.

Thanks. I've forked these two repos and will start going through the code pointers you've provided above.

[scottslewis]

I hope these help as starting points. I am happy to answer follow-up questions as necessary.

Thanks! I've debugged through the test and code references you provided above with some of the simple test cases, and have at least the beginnings of understanding of what's happening with the object reference handling. A couple of questions

The object id writer and reader (using default IntGenerator/resolver) obviously serialize/deserialize the integer object @id value.

Are there generators/resolver pairs to use the json reference/json linking specifications? i.e. https://json-spec.readthedocs.io/reference.html

If not, is that because of a technical or architectural problem (i.e. won't work as needed) using the json reference/linking spec? Or some other reason? Jackson has been around for quite a while (again my compliments) so maybe it was just pre-spec wrt supporting serializing/deserializing object graphs?

I'm thinking of cross-language/parser/generator compatibility and wondering: what do you think is the best approach to getting to that? I can easily think of

a) using your existing (default) approach...i.e. int generators/resolvers and writing extensions to every generator/parser for each lang, or

b) Adding support for json reference/linking as new parsers/generators for Jackson, and doing whatever is needed to support that spec in other languages/parser implementations.

wdyt?

Another question: Where do you think would be the best place to add documentation for @JsonIdentityInfo (and generator/resolver effects/defaults? Somewhere in the javadocs? (like @JsonIdentityInfo)...or an existing/new md document(s)?...and/or sample applications or inline docs in existing or new tests? Or a tutorial that combines prose docs with examples, etc?

[cowtowncoder]

The object id writer and reader (using default IntGenerator/resolver) obviously serialize/deserialize the integer object @id value.

Yes, that is one possibility -- but other scalars (most notably JSON Strings representing things like UUIDs) are used as well.

Are there generators/resolver pairs to use the json reference/json linking specifications? i.e. https://json-spec.readthedocs.io/reference.html

I don't think so: the problem with reference-based approached (as opposed to simple non-semantic Ids) is that they infer/require existence of materialized Document object model -- and Jackson does not build one (caller can of course get one as JsonNode, or POJO based, but core Jackson doesn't) as handling is fully streaming (incremental): state is kept but not structure (except for parent element info, stack).
So arbitrary references cannot be necessarily generated or (more problematically) resolved.
I may be wrong about difficulty of implementation tho: if no traversal/de-referencing is needed, existing context stack might suffice for generating accurate absolute path refs. Still, resolving would cause challenges if id/refs came not in document order so it still seems challenging.

So these schemes are partly supported since they were not necessarily around, but more importantly because they would be rather difficult (and with additional book keeping, costly, cpu/memory-wise) to implement.

I'm thinking of cross-language/parser/generator compatibility and wondering: what do you think is the best approach to getting to that? I can easily think of

(a) [~ do what Jackson does, everywhere]
(b) Adding support for json reference/linking a [...]

I would probably suggest selecting one of configurations (or small number) Jackson does (from (a)), f.ex maybe just:

• At least two Id types: (a) positive int ids (preferably incremental but fundamentally just unique id), (b) UUIDs
  • ... oh, but how do you specify which one to use? Possibly just one, positive 64-bit integers?
• Global (per JSON doc) scope

But I am definitely biased