Mirror ./examples/community folder on HF

[Wauplin]

Related to internal slack thread.

HF mirror example: https://huggingface.co/datasets/Wauplin/diffusers-community-pipelines-mirror. Each tag/branch is a subfolder with all the pipelines in it.

This PR adds a new CI workflow to host the community pipelines folder (./examples/community) as an HF dataset. The HF dataset will only be a mirror of the content on Github and should be automatically updated by this CI. Action is triggered on commits on new tags (i.e. on new releases) and on commits on main updating the ./examples/community folder.

Once this will be done, we will be able to download community pipeline modules directly from the HF Hub instead of relying on Github files. This should simplify the logic as we will be able to use hf_hub_download which handles cache_dir, force_download, local_files_only, etc... natively. Downloading from Github currently requires the deprecated cached_download that will soon be removed from huggingface_hub.

---

TODO before merging: (by a diffusers maintainer)

• Create a dataset repo in diffusers/community-pipelines-mirror in the diffusers organization on the Hub.
• Create a fine-grained token for https://huggingface.co/diffusers-bot that has write access only to this dataset repo
• Set this newly created token as HF_TOKEN_MIRROR_COMMUNITY_PIPELINES in Github secret

TODO once merged:

• upload all previous tags to the dataset mirror
• get rid of cached_download in a follow-up PR. See draft: Move away from cached_download #8419

[sayakpaul]

LGTM, thanks for this!

I have taken care of the todos (before merging one), too.

[Wauplin]

Thanks for the review and already handling the TODOs @sayakpaul! ❤️
I've addressed your comments so we're good to merge I think :)

[sayakpaul]

No worries. Will merge once the CI is green (although it should not matter since the CI is not supposed to be triggered during PRs).

From the todos after merging:

upload all previous tags to the dataset mirror

What exactly needs to be done here?

[Wauplin]

What exactly needs to be done here?

I actually just pushed a new commit (72783bb) that will allow us to trigger this workflow manually. What we need to do is trigger a deploy for every existing tag + for the main branch to populate the dataset repo. The CI will then take care of updating it when needed. I can take care of that part I think and will let you know once it's done.

[Wauplin]

(sorry, just realized the yml is incorrect. Converted to draft to avoid it being merged)

[Wauplin]

Ok, sorry for the mess. I created and merged #8425, #8426 and #8427 to fix a few things. Files are getting correctly hosted on the Hub: https://huggingface.co/datasets/diffusers/community-pipelines-mirror/tree/main 🎉

Workflow can be manually triggered on https://github.com/huggingface/diffusers/actions/workflows/mirror_community_pipeline.yml.
Next time you do a new release, would be good to check that everything is correctly uploaded as well.

[sayakpaul]

Cool, would you mind updating our Slack channel about this? This is very nice move, indeed.

Next time you do a new release, would be good to check that everything is correctly uploaded as well.

Perhaps we could set up automated reporting for this?

[Wauplin]

would you mind updating our Slack channel about this?

Will do!

Perhaps we could set up automated reporting for this?

A CI checking if the CI failed?

[sayakpaul]

A CI checking if the CI failed?

If the CI isn't successful, we will report the error to a Slack channel.

[Wauplin]

Ah yes, makes sense. Do you have an example of how to do that? Happy to help but if you know how to do it/configure it, can I leave it to you? 🙏

[sayakpaul]

This is how automated reporting is setup for a release: https://github.com/huggingface/diffusers/blob/main/.github/workflows/notify_slack_about_release.yml. I plan to update the benchmarking workflow and this workflow for automated reporting. But could be some days given the upcoming releases.

[Wauplin]

Not a priority indeed! Good luck for the upcoming releases!