feat: introduce dagster sqlmesh cache #48
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This introduces a cache when translating sqlmesh context to assets
1 task
Member
Author
|
The latest change actually removes most of the caching from here as it's something that is mostly for opensource-observer specifically. Instead, we now make it easier to cache parts of this by adding new methods to the translator and breaking apart the dagster asset factory. |
Member
Author
|
We should review this post-hoc, sadly I need this before we can review another issue on |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a feature that the OSO team has discovered is something we very much need. We might change where this caching takes place in the future if no external party wants to use this feature.
For context, we have a very large SQLMesh project that actually has some models generated through "model factories". The project ends up generating something like 800 models and loading that on a fairly quick machine (AMD Ryzen 9 class + 64GB) ends up taking about 40 seconds. We also need to do some work on our side to likely cache the model generation (to speed up sqlmesh load times in general) but this ended up being an easier touch point. This caching enabled the load time to go from 40 seconds to 5 milliseconds.