WIP: Glossary and guide to referencing

best-practices.md

@@ -1,7 +1,7 @@
 ---
 layout: default
 title: Best Practices
-nav_order: 4
+nav_order: 5
 ---
 
 # Best Practices

definitions.md

@@ -0,0 +1,29 @@
+---
+layout: default
+title: Definitions
+nav_order: 3
+---
+
+# Definitions
+
+- **document**: A local file, network resource, or other distinct entity in a particular format such as JSON or YAML
+- **entry document**: The document in an OAD where processing begins, starting with an OpenAPI Object ([3.0](https://spec.openapis.org/oas/v3.0.3#openapi-object), [3.1](https://spec.openapis.org/oas/v3.1.0#openapi-object))
+- **OpenAPI Description** (**OAD**): One or more documents written according to a specific version of the OpenAPI Specification, that together describe an API
+- **OpenAPI Initiative** (**OAI**): The organization responsible for the development of the OpenAPI Specification _(not to be confused with the unrelated and more recent "OpenAI")_
+- **OpenAPI Specification** (**OAS**): The formal requirements for the OpenAPI format, which exists in several versions (e.g. 3.0.3, 3.1.0)
+- **reference**: A connection from one location in an OAD to another in which the reference target is identified by a URI; the `"$ref"` and `"operationRef"` keywords implement references in the OAS
+- **reference removal**: The process of replacing references with their targets; not all references can be removed
+- **reference resolution**: The process of associating all reference sources with their targets, typically in-memory, without needing to remove references; all references in a well-formed OAD are resolvable
+- **Uniform Resource Identifier** (**URI**): An identifier for a resource of any kind, which may or may not indicate its location
+- **Uniform Resource Locator** (**URL**): A URI intended to be used as a locator for retrieving or otherwise interacting with a resource
+- **URI-reference resolution**: The process of resolving a [URI-reference](https://www.rfc-editor.org/rfc/rfc3986.html#section-4.1) against a [base URI](https://www.rfc-editor.org/rfc/rfc3986.html#section-5.1) to produce a full URI (which must include a [scheme](https://www.rfc-editor.org/rfc/rfc3986.html#section-3.1)); this must be done with the values of `"$ref"` and similar keywords before as the first step of reference resolution (or reference removal)
+
+## A note on the history of "document", "definition", and "description"
+
+Up through OAS 3.0.3 and 3.1.0, the OAS contained a section titled "OpenAPI Document", with differing definintions depending the OAS version.  The terms "OpenAPI Definition", "OpenAPI Description", and other variations and combinations were also used within the specification and/or the [learn.openapips.org](learn.openapis.org) site.
+
+In September 2023, the OAI Technical Steering Committee (TSC) agreed to standardize on "OpenAPI Description" for the complete logical entity that describes an API, and to reserve the term "document" (lower-cased) for its common meaning.  Resolving the debate between "OpenAPI Description" and "OpenAPI Definition" hinged on the observation that while "definition" is appropriate when the API code is generated from the OAD, it is not accurate when an OAD is written for an existing API.  The term "description" is accurate in both cases.
+
+## Documents and OpenAPI Descriptions
+
+An OpenAPI Description, or OAD, can be structured as one or more documents.  Those documents are typically stored as local files with `.json` or `.yaml` (or `.yml`) extensions, or as HTTP-accessible network resources of type `application/openapi+json` or `application/openapi+yaml`.  The documents are connected by references, as described in the [Using References In OpenAPI Descriptions](references) guide.

referencing/faq.md

@@ -0,0 +1,74 @@
+---
+layout: default
+title: Referencing FAQ
+nav_order: 1
+parent: Using References
+---
+
+# Referencing FAQ
+
+## Do I need to read the entire referencing guide?
+
+If your OpenAPI Description (OAD):
+
+1. consists only of a single JSON or YAML document
+1. contains only references that are fragments (beginning with a `"#"`, e.g. `{"$ref": "#/components/responses/oops"}`)
+1. does not have any other keywords in the same object as any `"$ref"`
+1. (OAS 3.1 only) does *not* use `"$id"`if you are using OAS 3.1, you are  *not* using the `"$id"` keyword in Schema Objects
+1. (OAS 3.1 only) does *not* use `"$dynamicRef"` or `"$dynamicAnchor"`
+
+Then your use of references is simple enough that your references will do what you expect with any tool that supports them at all.  You probably do not need to read the rest of this page.
+
+## Can an OAD be *completely* "de-referenced" by "removing" all references?
+
+In general, no.  But in many common cases, yes, and tools exist that will attempt this.
+
+There are two types of reasons why de-referencing might not be possible:
+
+* Technical reasons:
+   * Cyclic references are allowed in Schema Objects, and are used to describe recursive tree structures.  JSON Schema itself is a recursive tree structure, so the JSON Schema metaschemas cannot be fully de-referenced (see draft-07 or earlier for how this works with `"$ref"`; draft 2020-12 uses a different technique)
+   * Cyclic references can also occur when describing complex payloads such as `multipart/form-data` where media type descriptions can be nested
+   * References in Link Objects (`"operationRef"`) and Discriminator Objects (`"mapping"`) cannot be replaced with literal values, although there are alternatives to using references in these cases (`"operationId"` and mapping by schema component name, respectively, althouogh there can be challenges with arranging the documents properly for those techniques as well)
+   * Dynamic references in OAS 3.1 (which are not covered in detail by this guide) have runtime behavior which cannot always be predicted ahead of time
+* Social reasons:
+   * In large-scale API ecosystems, different parts of an API may be owned by different entities and connected by references
+   * Independent deployment of different parts of the API and different documents within the OpenAPI Description may make de-referencing impossible, or at least very fragile
+   * Security or other policies may forbid the sort of copying necessary to perform de-referencing; this is most likely to happen in high-security environments such as government or military
+   * Standards compliance policies might require referencing the OAD or schema published by a standards body, rather than copying it
+
+## Can an OAD at least be de-referenced to a single document?
+
+Technically, yes, although in some cases this is more complex than it may seem:
+
+The following sorts of references cannot be removed and replaced with their (possibly modified, depending on the OAS version an source object) targets.  In some cases an alternate technique can be used, while in others the reference value must be updated to reflect the structure of the single consolidated document:
+
+* cyclic references
+* `"operationRef"` in Link Objects (although if `"operationId"`s are managed carefully, `"operationRef"` can be replaced by `"operationId"`)
+* `"mapping"` in Discriminator Objects (although URI-reference values can be replaced with names from the Components Object if handled correctly)
+
+Dynamic references, which are outside the scope of this guide, may need even more complex handling to be preserved in a single-document structure.
+
+For OAS 3.1, note that if the `"$id"` keyword is used, tools must be aware of it to successfully perform de-referencing, so check your tool's documentation carefully.  For an example of using `"$id"` in a technique known as "bundling", see the OpenAPI Blog post [JSON Schema bundling finally formalised](https://www.openapis.org/blog/2021/08/23/json-schema-bundling-finally-formalised).
+
+However, keep in mind that there are other, non-technical reasons why de-referencing may not be feasible.
+
+*De-referencing is not a general solution in large-scale API ecosystems.*
+
+## How is reference "resolution" different from "de-referencing"?
+
+A reference is said to be *resolved* within a tool if:
+
+* Its target has been identified
+* Any modifications to the target required by the OAS have been performed
+* The resulting value has been associated with the reference source in some way that the tool can easily use when needed
+
+While plain JSON documents form a [tree](https://en.wikipedia.org/wiki/Tree_%28data_structure%29) structure, an OpenAPI Description with resolved references is not a tree, but a [graph](https://en.wikipedia.org/wiki/Graph_%28abstract_data_type%29).
+
+## Do I need to understand dynamic references?
+
+This guide does **not** cover OAS 3.1 / JSON Schema draft 2020-12's dynamic references (`"$dynamicRef"` and `"$dynamicAnchor"`).  Dynamic references serve a different purpose than  discussed here.  The following JSON Schema Blog posts explain some uses of dynamic references and anchors:
+
+* [Using Dynamic References to Support Generic Types](https://json-schema.org/blog/posts/dynamicref-and-generics#using-dynamic-references-to-support-generic-types)
+* [Validating OpenAPI and JSON Schema](https://json-schema.org/blog/posts/validating-openapi-and-json-schema)
+
+Note that in some cases, `"$dynamicRef"` cannot be removed or de-referenced, as it will evaluate differently depending on how it is reached at runtime.

referencing/index.md

@@ -0,0 +1,15 @@
+---
+layout: default
+title: Using References
+nav_order: 6
+has_children: true
+has_toc: false
+---
+
+# Using References in OpenAPI Descriptions
+
+Confusion over how and why to use references in OpenAPI Descriptions (OADs) is common, as is confusion over how tools can or should be configured to handle them.  This guide will explain how both simple and complex referencing scenarios work, and when and why you should consider more complex referencing scenarios.
+
+- [Referencing FAQ](faq)
+- [What are References?](what)
+- [Resolving References](resolving)

referencing/resolving.md

@@ -0,0 +1,61 @@
+---
+layout: default
+title: Resolving References
+nav_order: 3
+parent: Using References
+---
+
+# Resolving References
+
+The only difference between URIs and URLs is their intended usage:  if a URI is intended to be used as a locator exactly as it is written, then it can be called a URL.  API endpoints, for example, are URLs.  If a URI is intended to be a stable identifier regardless of where the identified document actually lives, it is a URI, and *not* a URL.
+
+## Resolving internal references
+
+References that consist only of URI fragments, or that have URIs whose non-fragment part resolves to the current document, are easily resolved.  For JSON Pointer fragments (the only kind allowed in 3.0, and the only kind allowed outside of Schema Objects in 3.1), all that is needed is to evaluate the pointer against the document.
+
+### OAS 3.1 Schema Objects and internal references
+
+### Static references to plain-name URI fragments
+
+With OAS 3.1 Schema Objects, it is also possible to use plain-name URI fragments in both static (`"$ref"`) and dynamic (`"$dynamicRef"`) references.  Furthermore, it is possible to use `"$id"` to embed a Schema Object with a different [absolute-URI](https://www.rfc-editor.org/rfc/rfc3986#section-4.3) from the overall document inside that document.  While these references are still internal to the document, they require that the whole document has been parsed, and the keywords that declare the absolute-URIs (`"$id"`) or plain-name URI fragments (`"$anchor"` and `"$dynamicAnchor"`) have been recognized.
+
+As we will see, these scenarios add complexity to resolving external references, as the correct document must be identified and parsed in full before the reference can be resolved.
+
+### Dynamic references
+
+In the case of `"$dynamicAnchor"` and `"$dynamicRef"`, all schema documents containing Schema Objects with `"$dynamicAnchor"` along the evaluation path (following `"$ref"` and `"$dynamicRef"` rather than the literal file structure) of any given instance validation must be parsed and recognized in order to properly resolve the `"$dynamicRef"` for that instance validation.  Dynamic references cannot be fully resolved except in the context of instance evaluation/validation.
+
+This is true for both internal and external dynamic references.  In particular, a `"$dynamicRef"` will always be statically resolvable (treating it like `"$ref"`) to a `"$dynamicAnchor"` in the same document, making all `"$dynamicRef"`s appear to be internal references.  However, if the `"$dynamicRef"` is reached through any external documents on a particular evaluation path, then it will at least potentially function as an external reference.
+
+## Resolving external references
+
+External references, which are references with a target URI pointing to a different document than the one containing the reference, can be handled in several different ways.  The OpenAPI Specification is not clear on the exact requirements, although the JSON Schema Specification makes it clear that references are done through _identifiers_ (URIs) rather than _locators_ (URLs) ([draft-wright-00](https://datatracker.ietf.org/doc/html/draft-wright-json-schema-00#autoid-19) for OAS 3.0; [draft-bhutton-00](https://datatracker.ietf.org/doc/html/draft-bhutton-json-schema-00#section-8.2.3) for OAS 3.1).
+
+Not all tools support external references at all, and some that do only support handling them as URLs.  This guide will examine different approaches, but check your tool documentation for what is supported in your environment.
+
+### Treating reference URIs as URLs
+
+The simplest way to resolve external references is to just treat the URIs as URLs.  This works if either your OAD documents are always located in the same place, or if you can use relative URI-references to handle the difference between (for example) local check-outs of your files vs various development, testing, and production deployments.
+
+However, some use cases make treating reference identifiers as locators challenging:
+
+* Maintaining both JSON and YAML formats, when using file extensions to differentiate them
+* OADs that reference 3rd-party schemas from disparate sources
+
+### Use cases for separating identity (URIs) and location (URLs)
+
+There are many reasons to separate reference identity and location.  This section will briefly list a few, while the next section will use an analogy to make the benefits more concrete for those who have only worked with relatively simple OpenAPI Descriptions (OADs).
+
+Note that the versions of JSON Schema referenced by OAS [3.0](https://datatracker.ietf.org/doc/html/draft-wright-json-schema#section-8) and [3.1](https://datatracker.ietf.org/doc/html/draft-bhutton-json-schema-00#section-8.2.3) explicitly state that reference use URIs as identifiers, not necessarily locators.
+
+Tools that support separating reference identity and location provide various ways to map reference URIs to actual URLs to resolve the references.  These interfaces vary widely, so check your tool's documentation to determine what reference resolution configuration it supports.
+
+Use cases for mapping references to locations other than that given by their URI include:
+
+* Abstracting JSON vs YAML representaitons by omitting any file extension (otherwise you need different reference values in each format, or you will change formats as you follow references which is allowed but may be surprising to users)
+* Using an internal mirror of a 3rd-party schema repository in a high-security environment
+* Managing references that are too complex to handle with relative URI-references alone, such as when you are mixing shared production schemas or other components with local development or tests schemas or components
+
+### An analogy with programming language package managers
+
+TBD; see [OAI/moonwalk#49 (comment)](https://github.com/OAI/moonwalk/discussions/49#discussioncomment-6621150) for a first pass at this example.