Running pre-commit in parallel

[lsierant]

Summary

Running pre-commit commands in parallel to speed it up.

To get the quickest run, execute it locally with skipping license check. Consider putting export MDB_UPDATE_LICENSES=true into your private-context to always enable generating licenses in pre-commit

Changes

• make precommit is no longer using EVERGREEN_MODE=true, and is skipping generating licenses by default
• added make precommit-with-licenses for easier precommit with generating license step

Proof of Work

Benchmark Comparison

Scenario	Master Run	PR Run	Speedup
No Cache, With Licenses	47.519s	31.897s	1.49x faster
With Cache, With Licenses	24.382s	17.291s	1.41x faster
No Cache, Without Licenses	24.388s	21.783s	1.12x faster
With Cache, Without Licenses	8.229s	4.463s	1.84x faster

The table above is just a summary for the runs below.

No Go Cache, With Licenses, Master Run

git co master
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
$ go clean -cache -modcache
$ time make precommit-with-licenses  
125.07s user 38.88s system 345% cpu 47.519 total

With Go Cache, With Licenses, Master Run

$ time make precommit-with-licenses
32.90s user 12.69s system 187% cpu 24.382 total

No Go Cache, Without Licenses, Master Run

# manually commented update_licenses script
$ go clean -cache -modcache
$ time make precommit
92.78s user 31.70s system 510% cpu 24.388 total

With Go Cache, Without Licenses, Master Run

# manually commented update_licenses script
$ time make precommit
9.46s user 9.83s system 234% cpu 8.229 total

No Go Cache, With Licenses, PR Run

$ go clean -cache -modcache
$ time make precommit-with-licenses  
125.88s user 39.88s system 519% cpu 31.897 total

With Go Cache, With Licenses, PR Run

$ time make precommit-with-licenses
32.95s user 11.44s system 256% cpu 17.291 total

No Go Cache, Without Licenses, PR Run

$ go clean -cache -modcache
$ time make precommit
92.61s user 32.05s system 572% cpu 21.783 total

With Go Cache, Without Licenses, PR Run

$ time make precommit
9.44s user 11.18s system 462% cpu 4.463 total

EVG actually running licenses as part of check_precommit:

[2025/07/02 11:26:08.685] update_licenses: update_licenses: Processing licenses for module: /data/mci/41a0c17df8b491653520d6aef7261120/src/github.com/mongodb/mongodb-kubernetes/public/tools/multicluster
[2025/07/02 11:26:08.689] update_licenses: update_licenses: Processing licenses for module: /data/mci/41a0c17df8b491653520d6aef7261120/src/github.com/mongodb/mongodb-kubernetes
[2025/07/02 11:27:09.129] update_licenses: update_licenses: License processing complete for all modules.

Checklist

• Have you linked a jira ticket and/or is the ticket in the title?
• Have you checked whether your jira ticket required DOCSP changes?
• Have you checked for release_note changes?

Reminder (Please remove this when merging)

• Please try to Approve or Reject Changes the PR, keep PRs in review as short as possible
• Our Short Guide for PRs: Link
• Remember the following Communication Standards - use comment prefixes for clarity:
  • blocking: Must be addressed before approval.
  • follow-up: Can be addressed in a later PR or ticket.
  • q: Clarifying question.
  • nit: Non-blocking suggestions.
  • note: Side-note, non-actionable. Example: Praise
  • --> no prefix is considered a question

[m1kola]

LGTM, but I would love us to move license updates to something like make generate or a similar commands and have a CI which does:

1. Runs make generate
2. Checks if there are any uncommitted changes.
   • Fails, if there are changes
   • Passes, if there are no changes

And in general, split pre-commit into smaller pieces run separately.

[lsierant]

LGTM, but I would love us to move license updates to something like make generate or a similar commands and have a CI which does:

1. Runs make generate

2. Checks if there are any uncommitted changes.

   • Fails, if there are changes
   • Passes, if there are no changes

And in general, split pre-commit into smaller pieces run separately.

@m1kola Totally agree on this!

[nammn]

lgtm; ptal at my comments

[nammn]

LGTM! let's check that the ci linter runs licenses 🤞

[lsierant]

LGTM! let's check that the ci linter runs licenses 🤞

@nammn
yup, it works:

[2025/07/02 11:26:08.685] update_licenses: update_licenses: Processing licenses for module: /data/mci/41a0c17df8b491653520d6aef7261120/src/github.com/mongodb/mongodb-kubernetes/public/tools/multicluster
[2025/07/02 11:26:08.689] update_licenses: update_licenses: Processing licenses for module: /data/mci/41a0c17df8b491653520d6aef7261120/src/github.com/mongodb/mongodb-kubernetes
[2025/07/02 11:27:09.129] update_licenses: update_licenses: License processing complete for all modules.