[GitHub][workflows] Add buildbot information comment to first merged PR from a new contributor #78292
Conversation
…PR from a new contributor This change adds a comment to the first PR from a new contributor that is merged, which tells them what to expect post merge from the build bots. How they will be notified, where to ask questions, that you're more likely to be reverted than in other projects, etc. To do this, I have added a hidden HTML comment to the new contributor greeting comment. This workflow will look for that to tell if the author of the PR was a new contributor at the time they opened the merge. It has to be done this way because as soon as the PR is merged, they are by GitHub's definition no longer a new contributor and I suspect that their author assocication will be "contributor" instead. I cannot 100% confirm that without a whole lot of effort and probably breaking GitHub's terms of service, but it's fairly cheap to work around anyway. It seems rare / almost impossible to reopen a PR in llvm at least, but in case it does happen the buildbot info comment has its own hidden HTML comment. If we find this we will not post another copy of the same information. An example PR can be found here: #84 (exact text content subject to change)
|
@llvm/pr-subscribers-github-workflow Author: David Spickett (DavidSpickett) ChangesThis change adds a comment to the first PR from a new contributor that is merged, which tells them what to expect post merge from the build bots. How they will be notified, where to ask questions, that you're more likely to be reverted than in other projects, etc. To do this, I have added a hidden HTML comment to the new contributor greeting comment. This workflow will look for that to tell if the author of the PR was a new contributor at the time they opened the merge. It has to be done this way because as soon as the PR is merged, they are by GitHub's definition no longer a new contributor and I suspect that their author assocication will be "contributor" instead. I cannot 100% confirm that without a whole lot of effort and probably breaking GitHub's terms of service, but it's fairly cheap to work around anyway. It seems rare / almost impossible to reopen a PR in llvm at least, but in case it does happen the buildbot info comment has its own hidden HTML comment. If we find this we will not post another copy of the same information. An example PR can be found here: DavidSpickett#84 (exact text content subject to change) Full diff: https://github.com/llvm/llvm-project/pull/78292.diff 2 Files Affected:
diff --git a/.github/workflows/merged-prs.yml b/.github/workflows/merged-prs.yml
new file mode 100644
index 00000000000000..1b1503610dac13
--- /dev/null
+++ b/.github/workflows/merged-prs.yml
@@ -0,0 +1,36 @@
+name: "Add buildbot information to first PRs from new contributors"
+
+permissions:
+ contents: read
+
+on:
+ # It's safe to use pull_request_target here, because we aren't checking out
+ # code from the pull request branch.
+ # See https://securitylab.github.com/research/github-actions-preventing-pwn-requests/
+ pull_request_target:
+ types:
+ - closed
+
+jobs:
+ buildbot_comment:
+ runs-on: ubuntu-latest
+ permissions:
+ pull-requests: write
+ if: >-
+ (github.repository == 'llvm/llvm-project') &&
+ (github.event.pull_request.merged == true)
+ steps:
+ - name: Setup Automation Script
+ run: |
+ curl -O -L --fail https://raw.githubusercontent.com/"$GITHUB_REPOSITORY"/main/llvm/utils/git/github-automation.py
+ curl -O -L --fail https://raw.githubusercontent.com/"$GITHUB_REPOSITORY"/main/llvm/utils/git/requirements.txt
+ chmod a+x github-automation.py
+ pip install -r requirements.txt
+
+ - name: Add Buildbot information comment
+ run: |
+ ./github-automation.py \
+ --token '${{ secrets.GITHUB_TOKEN }}' \
+ pr-buildbot-information \
+ --issue-number "${{ github.event.pull_request.number }}" \
+ --author "${{ github.event.pull_request.user.login }}"
diff --git a/llvm/utils/git/github-automation.py b/llvm/utils/git/github-automation.py
index f78d91059ecd36..55659679536f48 100755
--- a/llvm/utils/git/github-automation.py
+++ b/llvm/utils/git/github-automation.py
@@ -208,6 +208,8 @@ def _get_curent_team(self) -> Optional[github.Team.Team]:
class PRGreeter:
+ COMMENT_TAG = "<!--LLVM NEW CONTRIBUTOR COMMENT-->\n"
+
def __init__(self, token: str, repo: str, pr_number: int):
repo = github.Github(token).get_repo(repo)
self.pr = repo.get_issue(pr_number).as_pull_request()
@@ -217,7 +219,9 @@ def run(self) -> bool:
# by a user new to LLVM and/or GitHub itself.
# This text is using Markdown formatting.
+
comment = f"""\
+{PRGreeter.COMMENT_TAG}
Thank you for submitting a Pull Request (PR) to the LLVM Project!
This PR will be automatically labeled and the relevant teams will be
@@ -240,6 +244,64 @@ def run(self) -> bool:
return True
+class PRBuildbotInformation:
+ COMMENT_TAG = "<!--LLVM BUILDBOT INFORMATION COMMENT-->\n"
+
+ def __init__(self, token: str, repo: str, pr_number: int, author: str):
+ repo = github.Github(token).get_repo(repo)
+ self.pr = repo.get_issue(pr_number).as_pull_request()
+ self.author = author
+
+ def should_comment(self) -> bool:
+ # As soon as a new contributor has a PR merged, they are no longer a new contributor.
+ # We can tell that they were a new contributor previously because we would have
+ # added a new contributor greeting comment when they opened the PR.
+ found_greeting = False
+ for comment in self.pr.as_issue().get_comments():
+ if PRGreeter.COMMENT_TAG in comment.body:
+ found_greeting = True
+ elif PRBuildbotInformation.COMMENT_TAG in comment.body:
+ # When an issue is reopened, then closed as merged again, we should not
+ # add a second comment. This event will be rare in practice as it seems
+ # like it's only possible when the main branch is still at the exact
+ # revision that the PR was merged on to, beyond that it's closed forever.
+ return False
+ return found_greeting
+
+ def run(self) -> bool:
+ if not self.should_comment():
+ return
+
+ # This text is using Markdown formatting.
+ comment = f"""\
+{PRBuildbotInformation.COMMENT_TAG}
+@{self.author} Congratulations on having your first Pull Request (PR) merged into the LLVM Project!
+
+Your changes will be combined with recent changes from other authors, then tested
+by our [build bots](https://lab.llvm.org/buildbot/#/console).
+
+If there is a problem with the build, all the change authors will receive an email
+describing the problem. Please check whether the problem has been caused by your
+change, as the change set may include many authors. If the problem affects many
+configurations, you may get many emails for the same problem.
+
+If you are using a GitHub `noreply` email address, you will not receive these emails.
+Instead, someone will comment on this PR to inform you of the issue.
+
+If you do not receive any reports of problems, no action is required from you.
+Your changes are working as expected, well done!
+
+If your change causes an ongoing issue, it may be reverted. This is a [normal part of LLVM development](https://llvm.org/docs/DeveloperPolicy.html#patch-reversion-policy)
+and is not a comment on yourself as an author.The revert commit (or a comment on this PR)
+should explain why it was reverted, how to fix your changes and merge them again.
+
+If you are unsure how to fix a problem, you can send questions in a reply
+to the notification email, add a comment to this PR, or ask on [Discord](https://discord.com/invite/xS7Z362).
+"""
+ self.pr.as_issue().create_comment(comment)
+ return True
+
+
def setup_llvmbot_git(git_dir="."):
"""
Configure the git repo in `git_dir` with the llvmbot account so
@@ -698,6 +760,10 @@ def execute_command(self) -> bool:
pr_greeter_parser = subparsers.add_parser("pr-greeter")
pr_greeter_parser.add_argument("--issue-number", type=int, required=True)
+pr_buildbot_information_parser = subparsers.add_parser("pr-buildbot-information")
+pr_buildbot_information_parser.add_argument("--issue-number", type=int, required=True)
+pr_buildbot_information_parser.add_argument("--author", type=str, required=True)
+
release_workflow_parser = subparsers.add_parser("release-workflow")
release_workflow_parser.add_argument(
"--llvm-project-dir",
@@ -751,6 +817,11 @@ def execute_command(self) -> bool:
elif args.command == "pr-greeter":
pr_greeter = PRGreeter(args.token, args.repo, args.issue_number)
pr_greeter.run()
+elif args.command == "pr-buildbot-information":
+ pr_buildbot_information = PRBuildbotInformation(
+ args.token, args.repo, args.issue_number, args.author
+ )
+ pr_buildbot_information.run()
elif args.command == "release-workflow":
release_workflow = ReleaseWorkflow(
args.token,
|
|
Important to note that this comment will notify all users, there's no way to avoid that. I hope that by |
|
https://discourse.llvm.org/t/rfc-buildbot-information-comment-on-new-contributors-first-prs/76303 to get more diverse feedback. |
* new PR needed for reland
|
Thanks for reviewing! I'm going to wait a few days for any feedback from the RFC before landing. |
|
This is a great idea - thanks! A few thoughts:
|
Given we have so many "but this", "sometimes that" I'm wondering if it's not better for me to update one of the documentation pages, then link to that. Then we have more room to include them. Not sure which yet though. |
Yeah, not mentioning the email reply is probably better. The reviewers of the PR are more likely to have context on why it failed (I would hope), and they can sort it out from there. |
So far, I think I got buildbot failures after 100% of my PRs that landed here (which is 5). My PRs were all changing the LangRef and nothing else, so none of them was a real failure caused by my PR. So I wouldn't say this is a rare occurrence, it seems to be the common case. My conclusion is that I should just entirely ignore these emails... that's probably not right either though, but I am not sure how I could even figure out whether I need to take any action. |
|
Thanks for your feedback, this is exactly what I was looking for!
Yeah my bias comes in here seeing builds go by all the time that do work properly, I forget the important bit is the builds the specific committer sees. So your take away isn't wrong at all. Unfortunate, but not a failure on the part of committers, instead a sign that the system should be improved.
We're asking a lot of folks to understand a whole other system, and explaining how to diagnose problems generally is too much to include here and something I wouldn't want folks to think they've got to spend too much time on. Your feedback certainly backs up Alex's suggestions so I will add those now. |
this may not happen. * Remove replying to emails because again, there's some situations where that won't be helpful.
|
I've updated the text with the feedback so far. I think it could go on https://llvm.org/docs/GitHub.html instead as "what happens after merge" so going to look at that next. |
|
@asb I've updated the text to link to https://llvm.org/docs/MyFirstTypoFix.html#myfirsttypofix-issues-after-landing-your-pr, but there's still a considerable overlap as you suggested. My intent is that the text in the comment warns you about the initial messages you might get, then tells you, if you need to follow up on them, here's the link to a description of all that. |
|
New text LGTM, but it looks like some of the newly added lines aren't line-wrapped while the existing lines are. |
|
Yeah so that's an annoying thing I've had to work around. Unlike RST the lines don't flow automatically regardless of the newlines. So I've tried to lay it out so that the paragraphs aren't too weird looking, but the text is still readable in an editor. Also when you have a |
|
Added a comment to explain that oddity. |
This change adds a comment to the first PR from a new contributor that is merged, which tells them what to expect post merge from the build bots.
How they will be notified, where to ask questions, that you're more likely to be reverted than in other projects, etc. The information overlaps with, and links to https://llvm.org/docs/MyFirstTypoFix.html#myfirsttypofix-issues-after-landing-your-pr. So that users who simply read the email are still aware, and know where to follow up if they do get reports.
To do this, I have added a hidden HTML comment to the new contributor greeting comment. This workflow will look for that to tell if the author of the PR was a new contributor at the time they opened the merge. It has to be done this way because as soon as the PR is merged, they are by GitHub's definition no longer a new contributor and I suspect that their author association will be "contributor" instead.
I cannot 100% confirm that without a whole lot of effort and probably breaking GitHub's terms of service, but it's fairly cheap to work around anyway. It seems rare / almost impossible to reopen a PR in llvm at least, but in case it does happen the buildbot info comment has its own hidden HTML comment. If we find this we will not post another copy of the same information.