Verbosity of summary is too much for large warehouses & github actions

[devon-seitz-cirrusmd]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
There is a text limit to the comment that get published to Github actions. When running Piperider on more than 100+ tables, that comment/comparison markdown is too large for Github to accept.

Describe the solution you'd like
A clear and concise description of what you want to happen.
I think the best way to handle would be to optionally reduce the verbosity of a summary report to only highlight the things/tables that have changed instead of all tables. This way the comment can be drastically shorter then the normal comparison summary.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
I have tried using the piperider github action for comparison, and writing my own action steps using the github comment upload api, but the markdown for the summary is simply too large.

Additional context
Add any other context or screenshots about the feature request here.

[elisawang09]

I’d like to upvote for this problem and expand it a bit.
My team use the piperider compare command for comparing the changes between a feature branch and the main branch. The generated reports contain all models (100+) in the DBT project. It’s quite annoying because we only want to focus on the modified models and the tagged downstream models (approx. 10 models). Therefore, a shorter and more concise comparison report would be significantly enhance our workflow.

Alternatives I've explored

I came across this video on your official YouTube channel that suggests this can be achieved by combining dbt state and the piperider compare-reports --last --tables-from target-only command. However, according to your documentation about how compare works, it appears that there is currently no way to include the argument --tables-from target-only when executing the "piperider compare" command.

Proposed solution

It would be incredibly beneficial if the summary generated by piperider compare could exclusively display the modified models. Introducing support for the '--table-from' argument when invoking the 'piperider compare' command could potentially accomplish this goal."

[even-wei]

Hi folks,

Thanks for your feedback. 🙏
Let me briefly summarize the issues and explain the improvement on the way.

---

1. comment/comparison markdown is too large for Github to accept
We are working on new summary markdown, which highlights altered models, affected downstream, and metrics. It will be released on Thursday (6/15) and you can checkout this gist.

However, it still could be too large to fit in GitHub comments. We are working on replacing some parts in the summary with links to the reports on PipeRider Cloud. The top priority is to highlight changes to our users.

If anything important to you but is not shown in the summary, please let me know. So we can help you more and benefit the community in general.

---

2. piperider compare could exclusively display the modified models
Currently, we are implementing some features that improve piperider compare.

1. by default, piperider compare will profile models based on the tags in the target branch. So you don't need to add tags in the main branch in order to profile those models. This will be release on Thursday (6/15).
2. PipeRider will support dbt selector, so that you can use piperider compare --select state:modified+ --state /tmp/base to run PipeRider on models with code change and affected downsteam. And this should be released on 6/29.

By the way, PipeRider does not profile models without piperider tag by default after version 0.25. If all models are profiled in your environment, you can upgrade PipeRider, remove .piperider/config.yml, and run piperider compare directly. Check our post https://medium.com/inthepipeline/zero-config-code-review-and-data-profiling-tool-for-dbt-projects-8b6de40964b4

[DaveFlynn]

Hi folks, I've got some good news for you today:

@devon-seitz-cirrusmd
The comparison summary for pull requests has been updated and now shows the top 50 resources from each category that have been impacted by the latest code changes.

We'd love if you have a chance to try this out and let us know if it solves your issue, and any feedback you might have.

Please check this PR comment for an example of the current comparison summary.

@elisawang09
From PipeRider 0.29 we have updated the --select option that allows you more control over the resources that you profile.

Please check the Specify resources to profile documentation for more information.

There's also a clip from the latest PipeRider Community Office Hours in which I talk about the --select option and show some examples.

Please try it out and let me know if you have any issues or feedback.

[devon-seitz-cirrusmd]

@DaveFlynn This is great! Do you know if the github action will be updated to allow the --modify

[DaveFlynn]

@devon-seitz-cirrusmd I have contacted the development team and this will be updated very soon. I'll let you know when it is ready. Thanks!

[DaveFlynn]

Hi @devon-seitz-cirrusmd

Support for --modified and --select has been added to the PipeRider GitHub Action.
Please let me know if you try it out.

Thanks,

Dave