Document How to Report Conformance Test Results #1329
Description
What would you like to be added:
A standard process for reporting conformance results, similar to what already exists for upstream Kubernetes https://github.com/cncf/k8s-conformance. After chatting with @hh about how this works in upstream, there are a few details that may need some tweaking for our use cases (please correct me if I misunderstood something).
- Upstream conformance is all or nothing, they don't have the same concept of conformance profiles
- Upstream conformance is per minor version, at least initially we may want to track conformance per patch version
As a general note, each upstream conformance report includes:
- A product.yaml file that describes the implementation, and version of the implementation that was used (example)
- A readme that provides a step by step process describing how conformance tests were run to get the passing result (example)
- A test log that shows conformance tests passing (example)
In addition to the above, we'll likely want some ability to describe partial conformance, or at a minimum, to indicate the extended features that conformance tests pass for with the specified implementation.
This may eventually transition to CNCF control, but it likely makes sense to start this structure in this repo.
Why this is needed:
As we get an increasing number of implementations of this API, it is more important than ever to ensure that each provides a consistent experience. In the future we'll likely want to translate the reports here into an easy to read format on https://gateway-api.sigs.k8s.io/implementations/.