Skip to content

table_diff to run multiple data diffs at the same time based on the differences between environments. #4198

@sungchun12

Description

@sungchun12

Whenever a user makes changes, table_diff is a great feature to reach for to validate model differences. However, it is cumbersome to manually write out the specific models to diff each time. There should be a flag to automatically diff all models impacted directly/indirectly based on latest state diff, so it doesn't have to be so tedious.

sqlmesh table_diff prod:dev --impacted
sqlmesh table_diff prod:dev --impacted --show-sample
sqlmesh table_diff prod:dev -i shortcut flag
sqlmesh plan dev --diff-impacted same thing as above but in the scope of the plan changes after this specific plan is applied.
sqlmesh plan dev -di shortcut flag

Considerations:

  • Enforces grain as required for all models to diff. This is reasonable to expect and something I enforced when data-diff was still alive.
  • Run all table diffs concurrently
  • No need to add a flag to diff all models, so we don't tempt users to waste time diffing where not needed

Video example for look and feel

Metadata

Metadata

Labels

FeatureAdds new functionality

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions