PR failure reporter has stopped working

[mhsmith]

If I understand the code correctly, it's supposed to post a comment on a PR if a stable builder fails, and there has been exactly one commit to the branch since the previous success.

The last such comment was around the end of September. Here are a couple of more recent failures which should have caused a comment but didn't:

• October 24: https://buildbot.python.org/#/builders/1380/builds/1693
• October 25: https://buildbot.python.org/#/builders/1594/builds/390

[vstinner]

I didn't find anything relevant in Twisted logs. For example, grep pr_reporter master/twistd.log* returns nothing, and Traceback in the latest 3 log files are not related to pr_reporter.py.

[vstinner]

cc @pablogsal @zware

[pablogsal]

Yeah these two PRs should have triggered and I just confirmed that those builders are in this list:

buildmaster-config/master/master.cfg

Line 288 in d5e7bc5

github_status_builders.append(buildername)

So not sure what is going on :(

[pablogsal]

I am adding more logging for now #549

[pablogsal]

We need to wait until it fails again to investigate why is not happening I am afaid

[pablogsal]

CC: @ambv @encukou If you see a buildbot failure with just one change that was not reported ping here or check the logs to see what's going on

[pablogsal]

I looked today into why there was no comment for https://buildbot.python.org/all/#/builders/1594/builds/600.

Seems the build id for this is: 1701133. You can check it using the API:

https://buildbot.python.org/api/v2/builds/1701133

Checking the logs:

$grep 1701133 twistd.log.3
2024-11-18 14:55:56+0000 [-] Considering reporting build :1701133
2024-11-18 14:55:59+0000 [-] Attempting to issue a PR comment for failed build for build 1701133
...
[HTTP11ClientProtocol (BufferingTLSTransport),client] Issued a dicord comment for python/cpython at b0fcc2c47a34a69c35c1a8031cd0589d3747c1af, issue 126502.

but there is no other trace

[pablogsal]

So looks like

buildmaster-config/master/custom/discord_reporter.py

Line 122 in f2bc68a

"Issued a dicord comment for {repoOwner}/{repoName} "

was hitted but not

buildmaster-config/master/custom/pr_reporter.py

Line 166 in f2bc68a

"Issued a Pull Request comment for {repoOwner}/{repoName} "

[pablogsal]

There is a lot of this:

2024-11-18 15:08:31+0000 [Broker,217389,8.43.85.223] invalid login from unknown user 'cstratak-RHEL7-x86_64'
2024-11-18 15:08:31+0000 [Broker,217389,8.43.85.223] Peer will receive following PB traceback:
2024-11-18 15:08:31+0000 [Broker,217389,8.43.85.223] Unhandled Error
        Traceback (most recent call last):
          File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/cred/portal.py", line 144, in login
            return maybeDeferred(
          File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 212, in maybeDeferred
            result = f(*args, **kwargs)
          File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 2287, in unwindGenerator
            return _cancellableInlineCallbacks(gen)
          File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 2197, in _cancellableInlineCallbacks
            _inlineCallbacks(None, gen, status, _copy_context())
        --- <exception caught here> ---
          File "/srv/buildbot/venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 2014, in _inlineCallbacks
            result = context.run(gen.send, result)
          File "/srv/buildbot/venv/lib/python3.9/site-packages/buildbot/worker/protocols/manager/pb.py", line 83, in requestAvatarId
            raise error.UnauthorizedLogin()
        twisted.cred.error.UnauthorizedLogin:

but that happens all the time and looks unrelated. An looks like is for fetching an avatar??? (requestAvatarId)?

[pablogsal]

Ah I found the problem. Turns out upstream changed how auth is passed down to the service in commit buildbot/buildbot@6b75ddf

I will create a fix

[pablogsal]

I think this is fixed will reopen if we keep having the problem

[vstinner]

It works again: python/cpython#127086 (comment) :-) (the failure is unrelated to the change, but the reporter works as expected.)