Add a new catalogs
property to workflow resources
#1021
Labels
area: spec
Changes in the Specification
change: documentation
Improvements or additions to documentation. It won't impact a version change.
change: feature
New feature or request. Impacts in a minor version change
Milestone
What would you like to be added:
Add a new
catalogs
property to workflow resources.Why is this needed:
This addition allows authors to define and configure catalogs within a workflow. The catalog's name can serve as shorthand for importing and calling custom functions, using the format
{function}:{version}@{catalog}
.This format enables SDKs and runtimes to differentiate between imported functions and those defined at the top level, ensuring validation for tasks that call non-imported functions. It also guarantees that references to functions are validated at runtime. Additionally, this avoids the need for runtimes to search through all catalogs for a function definition, as the catalog is explicitly identified. It also supports defining functions with the same name and version across different catalogs.
We should define a
default
catalog, which may or may not be provided by the runtimes. This catalog can take any form, entirely at the runtime's discretion, and is automatically available to all workflows without the need for explicit declaration. This approach allows runtimes to offer mechanisms for users to create and share "private" functions specific to their runtime environment.Proposal:
Calling a custom function, imported from a defined catalog:
Calling a custom function, imported from the default catalog:
Additional notes:
Some concepts have been partially discussed in #962.
The text was updated successfully, but these errors were encountered: