Setup "ecosystem CI" to avoid regressions for existing users

[charliermarsh]

I'd like to setup a CI job to run Ruff over some open-source projects that use it already (and are known to have "passing" Ruff configurations). For example, we could have a job that checks out a known-to-be-passing commit of Pandas, and runs latest Ruff over the codebase.

This would be helpful to spot regressions beyond Ruff's own test suite. Similar to https://github.com/vitejs/vite-ecosystem-ci.

Lots of details to work out. E.g, sometimes we'll see new lint failures due to improvements in Ruff (new rules, but also, bug fixes that eliminate false negatives), so this may need to be advisory rather than a blocker to release. We'll also need to pin commits for whichever projects we choose to include.

If anyone is interested in owning this, I'd love help.

[sciyoshi]

Do you imagine this running on every PR? Or simply as a nightly scheduled build?

[charliermarsh]

Probably nightly to start.

(I already do some of this manually before releasing, but it's very one-off -- if we have a new rule that I feel is sufficiently complex, I'll run it over Zulip, Bokeh, or a few other projects that I have checked-out locally.)

[charliermarsh]

As recent examples of the kinds of things I'd want to catch earlier:

• We broke some stuff in cibuildwheel and build (Removes quotes from annotations removes quotes from literal values #2613)
• Not sure what repo it's in but we regressed on N815 (N815: Regression of classmethod-decorators in v0.0.244 #2673)

[Jackenmen]

Few examples from different Python projects running a primer on pull requests and commenting the results:

• [dataclass_transform] support subclass/metaclass-based transforms python/mypy#14657 (comment)
• Replace no-method-argument with no-self-argument when variadic args present pylint-dev/pylint#7518 (comment)
• Let string splitters respect East_Asian_Width property psf/black#3445 (comment)

This is done as a diff between main branch and the pull request so if one finds reported changes acceptable and merges them, they will no longer be reported.

[sbrugman]

(I'd be interested to set this up)