Skip to content

Add a new catalogs property to workflow resources #1021

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cdavernas opened this issue Oct 21, 2024 · 0 comments · Fixed by #1022
Closed

Add a new catalogs property to workflow resources #1021

cdavernas opened this issue Oct 21, 2024 · 0 comments · Fixed by #1022
Assignees
Labels
area: spec Changes in the Specification change: documentation Improvements or additions to documentation. It won't impact a version change. change: feature New feature or request. Impacts in a minor version change
Milestone

Comments

@cdavernas
Copy link
Member

What would you like to be added:

Add a new catalogs property to workflow resources.

Why is this needed:

This addition allows authors to define and configure catalogs within a workflow. The catalog's name can serve as shorthand for importing and calling custom functions, using the format {function}:{version}@{catalog}.

This format enables SDKs and runtimes to differentiate between imported functions and those defined at the top level, ensuring validation for tasks that call non-imported functions. It also guarantees that references to functions are validated at runtime. Additionally, this avoids the need for runtimes to search through all catalogs for a function definition, as the catalog is explicitly identified. It also supports defining functions with the same name and version across different catalogs.

We should define a default catalog, which may or may not be provided by the runtimes. This catalog can take any form, entirely at the runtime's discretion, and is automatically available to all workflows without the need for explicit declaration. This approach allows runtimes to offer mechanisms for users to create and share "private" functions specific to their runtime environment.

Proposal:

Calling a custom function, imported from a defined catalog:

document:
  dsl: 1.0.0-alpha4
  namespace: default
  name: lds-session-create
  version: '0.1.0'
use:
  catalogs:
    catalogA:
      endpoint:
        uri: https://custom-functions-catalog.com #this is the base uri, to which the catalog path structure should be appended (/functions/foo/1.0.0/...)
        authentication: #this allows for private and/or secured catalogs
          basic:
            username: admin
            password: 12345
do:
  - callCustomFunction:
      call: greet:1.5.2@catalogA
      with:
        greetings: Hello, World!

Calling a custom function, imported from the default catalog:

document:
  dsl: 1.0.0-alpha4
  namespace: default
  name: lds-session-create
  version: '0.1.0'
do:
  - callCustomFunction:
      call: greet:1.5.2@default
      with:
        greetings: Hello, World!

Additional notes:

Some concepts have been partially discussed in #962.

@cdavernas cdavernas added change: documentation Improvements or additions to documentation. It won't impact a version change. change: feature New feature or request. Impacts in a minor version change area: spec Changes in the Specification labels Oct 21, 2024
@cdavernas cdavernas added this to the v1.0.0 milestone Oct 21, 2024
@cdavernas cdavernas self-assigned this Oct 21, 2024
@cdavernas cdavernas linked a pull request Oct 22, 2024 that will close this issue
8 tasks
@github-project-automation github-project-automation bot moved this from Backlog to Done in Progress Tracker Oct 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: spec Changes in the Specification change: documentation Improvements or additions to documentation. It won't impact a version change. change: feature New feature or request. Impacts in a minor version change
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

1 participant