Convert voluptuous schema to msgspec #752

abhishekmadan30 · 2025-08-26T15:16:44Z

No description provided.

…aph into schema-conversion

src/taskgraph/util/schema.py

docs/concepts/transforms.rst

src/taskgraph/config.py

src/taskgraph/transforms/matrix.py

Type refactor

ahal

Thanks, this is shaping up nicely!

I don't think I have any other major concerns, it's mostly all nits and a minor changes. So I think you can go ahead and start converting Gecko without fear of a major new request (from me at least :p)

docs/concepts/transforms.rst

src/taskgraph/config.py

src/taskgraph/parameters.py

src/taskgraph/transforms/fetch.py

src/taskgraph/transforms/run/__init__.py

src/taskgraph/util/dependencies.py

src/taskgraph/util/schema.py

...utter.project_name}}/taskcluster/{{cookiecutter.project_slug}}_taskgraph/transforms/hello.py

ahal

Nice, thanks for the updates! I don't see any fundamental problems or anything, though I'll reserve the right to request future changes as we test this out in Gecko ;)

I'll avoid approving this for now because I want to hold off landing until we have a working patch for Gecko, but I think you can go ahead and get started on that! Hopefully claude or some clever macros can help with the conversion, because there's going to be a lot of schemas :)

ahal

I didn't get a chance to review the whole thing, but I'll leave the comments I had so far for now and resume tomorrow. Figured you might want to get a head start

ahal · 2025-09-17T20:20:05Z

docs/tutorials/creating-a-task-graph.rst

   from taskgraph.util.schema import Schema
+   from taskgraph.transforms.base import TransformSequence
+
+   # Define the schema using Schema base class for better type checking and performance.


nit: I'd cut out for better type checking and performance here. The user reading this won't know what you're comparing against

ahal · 2025-09-17T20:23:48Z

src/taskgraph/config.py

+
+    # Optional fields
+    project_regex: Optional[str] = None  # Maps from "project-regex"
+    ssh_secret_name: Optional[str] = None  # Maps from "ssh-secret-name"


I guess this maps from works because in the base class we pass in the "rename" arg. Out of curiosity, what does it look like if we wanted to use underscores rather than dashes?

ahal · 2025-09-17T20:29:04Z

src/taskgraph/config.py

+    repositories: Dict[str, Repository]
+
+    # Optional fields
+    register: Optional[str] = None


I feel bad for not commenting this earlier, but do you think we could preserve the old descriptions as comments? The schema isn't really documented other than these descriptions, so I feel it's important that we preserve them somehow. Even if it's just a comment

I think Claude should be able to do this if you get it to examine the diff and teach it about the mapping from dash to underscore.

ahal · 2025-09-17T20:31:13Z

src/taskgraph/config.py

+class RunConfig(Schema):
+    """Run transforms configuration."""
+
+    use_caches: Optional[Union[bool, List[str]]] = None  # Maps from "use-caches"


This shouldn't just be List[str], there are only specific values allowed here (similar to TaskPriority)

ahal

A few more comments, still not all the way through though.

ahal · 2025-09-18T17:48:09Z

src/taskgraph/parameters.py

+                            k: v for k, v in kebab_params.items() if k in ext_fields
+                        }
+                        if ext_params:
+                            msgspec.convert(ext_params, ext_schema)


It looks like this for loop is duplicated in both branches of this if statement. So I think you can consolidate it a bit.

ahal · 2025-09-18T17:55:16Z

src/taskgraph/transforms/docker_image.py

+    # List of package tasks this docker image depends on.
+    packages: Optional[List[str]] = None
+    # Information for indexing this build so its artifacts can be discovered.
+    index: Optional[Any] = None


You should import the schema from gecko_taskgraph.transforms.task and reference the index key there when we come across these "cross references".

ahal · 2025-09-18T18:52:31Z

src/taskgraph/transforms/fetch.py

+    # Description of the task.
+    description: str
+    # Fetch configuration with type and additional fields.
+    fetch: Dict[str, Any]  # Must have 'type' key, other keys depend on type


This should use a sub schema rather than __post_init__.

ahal · 2025-09-18T18:56:19Z

src/taskgraph/transforms/fetch.py

+class StaticUrlFetchSchema(Schema):
+    """Configuration for static-url fetch type."""
+
+    type: str


I think this should be type: Literal["static-url"] rather than str

ahal · 2025-09-18T18:57:56Z

src/taskgraph/transforms/fetch.py

+class GitFetchSchema(Schema):
+    """Configuration for git fetch type."""
+
+    type: str


type: Literal["git"]

ahal · 2025-09-18T19:09:49Z

src/taskgraph/transforms/from_deps.py

+    "partner-repack-ids",
+    "component",
+    "build-type",
+]


I guess there's no getting around this duplication on Python 3.10 and below.. unfortunate, but not the end of the world.

ahal · 2025-09-18T19:12:54Z

src/taskgraph/transforms/from_deps.py

+    # dependency. Attributes of the upstream task may be used as
+    # substitution values in the `artifact` or `dest` values of the
+    # `fetches` entry.
+    fetches: Optional[Dict[str, List[Union[str, Dict[str, str]]]]] = None


Please reference the fetches schema directly if possible.

abhishekmadan30 added 9 commits August 19, 2025 16:45

feat: convert graph config schema to msgspec

192efb8

feat: convert more schemas to msgspec

d34493f

feat: updated fetch schema

8971de1

feat: update 2 more schemas

08dae5f

feat: added more schemas

656e375

feat: update schema

4804bb8

feat: update schema from voluptuous to msgspec

7e7206a

Merge branch 'schema-conversion' of github.com:abhishekmadan30/taskgr…

de18fdf

…aph into schema-conversion

feat: updated some bugs

d6f3589

ahal reviewed Aug 27, 2025

View reviewed changes

src/taskgraph/util/schema.py Outdated Show resolved Hide resolved

docs/concepts/transforms.rst Outdated Show resolved Hide resolved

src/taskgraph/config.py Show resolved Hide resolved

abhishekmadan30 and others added 4 commits August 29, 2025 10:15

feat: used schema wrapper for msgspec

60318b4

feat: cleaned up schema wrapper

7fddbd9

feat: added support for optional keying

1d23c98

Merge branch 'main' into schema-conversion

1a1e79c

abhishekmadan30 marked this pull request as ready for review August 29, 2025 20:35

abhishekmadan30 requested a review from a team as a code owner August 29, 2025 20:35

abhishekmadan30 requested a review from ahal August 29, 2025 20:35

abhishekmadan30 and others added 2 commits September 2, 2025 10:10

fix: updated errors

565a827

Merge branch 'main' into schema-conversion

a679015

abhishekmadan30 changed the title ~~WIP: Convert voluptuous schema to taskgraph~~ Convert voluptuous schema to taskgraph Sep 2, 2025

abhishekmadan30 changed the title ~~Convert voluptuous schema to taskgraph~~ Convert voluptuous schema to msgspec Sep 2, 2025

ahal reviewed Sep 2, 2025

View reviewed changes

src/taskgraph/transforms/matrix.py Outdated Show resolved Hide resolved

abhishekmadan30 added the BREAKING CHANGE Backwards incompatible request that will require major version bump label Sep 2, 2025

ahal and others added 6 commits September 2, 2025 15:13

DO NOT LAND: Use by-level in priority key

9b76e78

Experimental refactor

7d203c6

fix: changed how extra fields work

3baf4eb

fix: updated optionally keyed by fucntionality

ab07a7e

Type refactor

fix: updated tests for optionally keyed by

6849b11

fix: remove extra code

006a6e3

ahal requested changes Sep 3, 2025

View reviewed changes

fix: updated based on review

1b37f95

ahal reviewed Sep 5, 2025

View reviewed changes

ahal self-requested a review September 17, 2025 17:32

fix: compatibility with gecko

750511c

abhishekmadan30 force-pushed the schema-conversion branch from 87e0de9 to 750511c Compare September 17, 2025 19:01

ahal requested changes Sep 17, 2025

View reviewed changes

ahal requested changes Sep 18, 2025

View reviewed changes

ahal reviewed Sep 18, 2025

View reviewed changes

Convert voluptuous schema to msgspec #752

Are you sure you want to change the base?

Convert voluptuous schema to msgspec #752

Uh oh!

Conversation

abhishekmadan30 commented Aug 26, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ahal left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ahal left a comment

Choose a reason for hiding this comment

Uh oh!

ahal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahal left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!