RAML Overlay equivalent?

[darrelmiller]

In a separate document.
Has a separate lifecycle.

How does the overlay reference the source document? - By document structure
Is OperationId a valid way of referencing core documents?

[jstoiko]

Following our discussions during past TSC meetings, here's a draft proposal for an "OAS Overlay" mechanism. The current proposal is equivalent to the "RAML Overlay" mechanism.

Summary of proposed changes:

• add a field "extends" to the root/OpenAPI object
• the value of "extends" is a URL pointing to a valid OpenAPI document or to another overlay document
• if the field "extends" is present,
  • the current document does not need to be valid on its own but the resulting merged document must be a valid OpenAPI document
  • the "OAS Overlay" merging rules defined below are applied
• "OAS Overlay" merging rules:
  • if the property is an Array, all members of the target property are replaced
  • if the property is an Object, non-existing members are appended, existing members are replaced
  • if the target object contains conflicting properties as a result of the merge -- that is, properties that would render the merge object invalid -- such properties are removed

Example:

openapi: 3.1.0
extends: https://raw.githubusercontent.com/OAI/OpenAPI-Specification/master/examples/v3.0/petstore.yaml

servers:
  - url: http://localhost/v1

components:
  schemas:
    Pet:
      required:
        - id
        - name
        - size
      properties:
        id:
          type: string
        size:
          type: string

In the example above,

• the servers single url defined in the source document -- i.e. http://petstore.swagger.io/v1 -- is replaced by the single url http://localhost/v1
• the Pet schema is altered as follows:
  • the property id is changed to a string type, and as a result the target format conflicting property is removed
  • the property size of string type is added, and also added to the list of required fields

[earth2marsh]

Could the URL for extends be relative? That way I could work on a local copy.

Can multiple extends be chained together so several documents that extend each other?

I can see a couple places where overlays would be useful to me:

1. as a way to keep any implementation details for a spec separate from the consumer-oriented doc, such as API management concerns from the API provider's perspective, for example attaching a policy to an operation.
2. to allow a writer to create better descriptions/summaries/titles for attributes like a parameter or an operation without having to edit the spec itself. But since parameters are an array, in order to edit a single description, I would have to reproduce the entire array, no? Is there a way to more precisely target?

[jstoiko]

Those are both good use cases, 2. particularly for the localization of API docs .

Could the URL for extends be relative?

Yes

Can multiple extends be chained together so several documents that extend each other?

Yes and that's an important point. I updated my comment accordingly.

I would have to reproduce the entire array, no? Is there a way to more precisely target?

That's right. Arrays are hard to work with when it comes to merging. Instead of replacing arrays, we could decide to concat or union the overlaying array with the target array but this would introduce another (worse) problem by making it impossible to rewrite arrays from scratch. We could also decide to have a more imperative syntax which would include some sort of merge rule to apply to the target document, like "add" or "replace", but this would be to the detriment of readability/usability (IMHO).

[MikeRalphson]

But since parameters are an array, in order to edit a single description, I would have to reproduce the entire array, no? Is there a way to more precisely target?

In the case of parameters etc, the issue with targeting one element of the array can be worked around by using $refs to named parameters under the components object in the base OAS document.

[earth2marsh]

Thanks for clarifying, @jstoiko. What are some other use cases for merging that are more along the lines of signature changes? For example, a proposed change to an API design?

The crux of this for me comes down to identifying the key use cases. IMO, it is critical to make it easy to invite collaboration (e.g., the second case above, including your localization point), and it would help to keep the syntax simple.

As a thought experiment, say I wanted to update the limit parameter's description in your sample, the easiest I could imagine is:

openapi: 3.1.0
extends: https://raw.githubusercontent.com/OAI/OpenAPI-Specification/master/examples/v3.0/petstore.yaml

operations.listPets.parameters.limit.description: How many items to return per request (max 100)

This is the simplest approach I can imagine, and it uses the otherwise optional operationId as a shortcut to target the specific operation and treat the name property of parameter elements as a pseudo-map. To support reuse, I could use YAML's indenting, so if I also wanted to add a default I could use:

openapi: 3.1.0
extends: https://raw.githubusercontent.com/OAI/OpenAPI-Specification/master/examples/v3.0/petstore.yaml

operations.listPets.parameters.limit:
  description: How many items to return per request (max 100)
  default: 10

[handrews]

Note that we discussed merging of various sorts a lot in the JSON Schema project, and ultimately decided that merging of schemas violates a number of important design principles for schemas and should be avoided. We addressed the use cases with several different proposals for draft-08. This was a discussion across roughly five+ years (from original proposal around the time draft-04 was published in January 2013), two sets of spec editors (the ones that did draft-04, but split on the questions around merging and ultimately abandoned the project, and the set of editors including myself who revived the project about two years later), and 500+ github comments across 10 or so issue exploring different variations, or consolidating and re-starting discussions.

While you may find good merging options for other aspects of OAS, schema merging is a very, very hard problem for a variety of reasons. Which I should probably write up some time but to be honest after the past year's worth of work on it, the topic just exhausts me. It mostly has to do with JSON Schema validation's nature as a constraint system, in which constraints can be added but not removed. Merging violates the "cannot be removed" principle, which changes the processing model in very difficult ways.

[earth2marsh]

Thinking about that thought experiment more, the dot as a delimiter could complicate things since it could be used in parameter names, meaning we'd have to escape it or pick another delimiter. Might also support removing a field by passing in a null value, but again I wonder how often that happens in practice?

[MikeRalphson]

Also, a parameter name is not a unique identifier, the in value is also required.

[earth2marsh]

Good point. I'd be ok with being lazy and picking first match unless you specify, but I get that that might impart a queasy feeling.

[jstoiko]

Might also support removing a field by passing in a null value, but again I wonder how often that happens in practice?

Based on my experience and other people's experience witnessed with RAML, the lack of ability to remove things using overlays has not been an issue. We've also been promoting "adding" over "removing" as a general best practice -- to keep things "safe" -- but I am not sure how this fits in the OAS way of doing.

[earth2marsh]

Two other questions:

1. Can you extend the extends? That way you can re-use an overlay on another spec.
2. Does it make sense to outlaw circular references? You shouldn't be able to refer to an overlay document from the document it overlays.

[earth2marsh]

Two other use cases brought up by @darrelmiller on the 09/14 TSC call:

1. To identify an SLA. You might have a valid spec that doesn't have that information, but you could ask for the SLA variant for that spec.
2. Codegen is another, as we're polluting specs with metadata for codegen. This gives us a way to separate that out.

@whitlockjc (and others) talked about moving the reference into the core document, an extendedBy approach that might even be typed.

@darrelmiller and @webron observed differences of overlays as mutate vs separating out information like codegen-required annotations.

[whitlockjc]

I personally see this features as $ref with mutation (add/change) capability. If I am right, I wonder if we're implementing this properly. My concerns are as follows:

1. This approach only applies to the document as a whole and this could make overlays for deep paths harder
2. What happens if you fat finger something in the overlay (a path mismatch), wouldn't you end up with extraneous paths/properties where the error occurred?
3. This approach appears to be a one-off instead of following an existing technology, like JSON merge, or by extending an existing technology, like JSON References

I personally would like to see a $merge or something similar to $ref.

[jcomputer]

I think a valid point was raised in the call about if there is value in this being supported by the spec. Such value comes when cross-tooling support is beneficial.

To compare, here's two different flows:

• No overlay support:
  1. Edit spec and "overlays" in config editor ("overlays" here means some JSON standard merge format)
  2. Save specs and "overlays"
  3. Run JSON merge tool and save resulting spec(s)
  4. Run validation tools on resulting specs
  5. Run documentation generators on all relevant specs
  6. Run code generators on all relevant specs
  7. Run [whatever other tools] on resulting specs
  8. Maybe share some resulting specs with API callers
• OpenAPI with overlays
  1. Edit spec and overlays in config editor
  2. Save specs and overlays
  3. Run validation tools, which internally generate all the specs, on spec and overlays
  4. Run documentation generators, which internally generate all the specs, on spec and overlays
  5. Run code generators, which internally generate all the specs, on spec and overlays
  6. Run [whatever other tools], which internally generate all the specs, on spec and overlays
  7. Using tooling to generate resulting specs, which you may share some of with API callers

I'm not seeing a meaningful feature gap in one flow or the other (someone please point out if I've missed some tooling type which is more relevant). Overlays sound like they are just for the benefit of the spec writer, not the spec readers (tooling or human), so I don't know that they make sense to be used beyond the spec writing stage. If this is true, I don't think they make sense in the specification.

Related, someone called out the parallels between overlays and $ref. I'd say there are two parts to this: (a) When $ref refer within a file, I'd say they're an optimization and organization of that spec, allowing reduction of size and to relate similar structures. (b) When $ref refer outside the current file, I'd say they're more like overlays and I'd personally argue that is less valuable for the same reasons I gave about overlays. (Some of the tooling I work on can't even support 'b' because of loss of context; the only flow that makes sense in some cases is the "no external file" approach after the editing phase.)

[handrews]

@whitlockjc in the context of JSON Schema, $merge was highly problematic. After five years, 10+ GitHub issues across two repositories, and well over 500 comments, we decisively rejected the proposal.

Instead, we looked at the use cases that people claimed $merge solved and came up with several alternative approaches that produce a more consistent and manageable system.

Given a single schema object in JSON Schema, you can, in most cases, easily reason about the behavior based on just that object. With some of the features introduced to solve some of the $merge use cases, there is a well-defined way to reason about behavior based on that object and its subschemas (including subschemas referenced with $ref). The overall behavior is a predictable combination of the behaviors of the individual schema objects involved.

The problem with $merge is that you cannot express the result of evaluating the merged schema in terms of the results of evaluating the two schemas being merged. This makes writing tools much more difficult, particularly when you get into complicated lazy evaluation situations with $ref.

I can go into this more, although honestly I'd rather not. We've spent tremendous amounts of time and energy on this. If you want to allow merging OpenAPI documents in some way, sure, go ahead. There are places where that would probably make a lot of sense. But JSON Schema will never implement $merge. That topic has been more heavily debated by more people across more years than any other concept in JSON Schema, and it is not going to happen.

Instead, we reworked how we think about $ref (in the context of JSON Schema- outside of JSON Schema feel free to do something else), so that it is defined in terms of the results of the referenced schema, rather than some sort of inclusion / replacement. This makes it behave like all other JSON Schema keywords- it produces results, which are combined in the normal way with the other keywords in the same object. And yes, in this approach, other keywords can be adjacent to $ref.