Networking is Flaky on GitHub Hosted Runners

[iamrecursion]

Describe the bug
Situations that require networking, such as downloading a file using curl or running tests that utilise localhost, are proving to be very flaky, with spurious timeouts occurring often.

Area for Triage:
Servers

Question, Bug, or Feature?:
Bug

Virtual environments affected

• macOS 10.15
• Ubuntu 16.04 LTS
• Ubuntu 18.04 LTS
• Ubuntu 20.04 LTS
• Windows Server 2016 R2
• Windows Server 2019

Expected behavior
Networking behaviour should be consistent, and not cause spurious test failures due to timeouts that are impossible to reproduce on a non-CI machine.

Actual behavior
As the issue is flakiness, it is difficult to provide a consistent reproduction. The issues primarily occur with the Engine CI in the Enso repository, and manifest as spurious test failures, or failures to download things using curl, and the like.

1. Execute the workflow.
2. Wait for a failure to happen.

Virtually all of the failures (e.g. this one) are spurious and seem to occur due to networking timeouts.

[dsame]

We tried to reproduce the problem from few data centres of Azure and Digital Ocean in different regions and none of the confirmed the network issues by agent itself, but the destination piccolo.link might have the bandwidth limited.

To diagnose the exact problem please add tracepath piccolo.link (on ubuntu) to a shell task and provide the output of it to your Internet/hosting provider.

[iamrecursion]

We only very rarely see a failure due to that link. Primarily we are seeing failures for connections using localhost where they time out.

[dsame]

Can you please provide exact URLs which time out?
And still can you add a task with tracepath localhost and provide a log? Looks like a DNS problem that leads to incorrect name resolving. tracepath can help to detect this situation in your build

[dsame]

We do not have the response for a long time and closing the issue.

@iamrecursion please open new issue or reopen this one in case if you still have the problem

[iamrecursion]

Sorry, I never saw the reply 9 days ago.

• I can't provide URLs as they're randomly generated on localhost. The tests are talking to a local server.
• Logs as follows for Ubuntu. tracepath doesn't seem to exist on the macOS and Windows runners where the problems tend to occur.

Ubuntu Runner

 1?: [LOCALHOST]                        0.006ms pmtu 65536
 1:  localhost                                             0.159ms reached
 1:  localhost                                             0.032ms reached
     Resume: pmtu 65536 hops 1 back 1 

[iamrecursion]

I can't seem to actively re-open this.

[dsame]

In order to make sure the problem is with networking and not with the local service, please add

time ping -o localhost on ubuntu and osx before running the test and send the build logs to us.

Actually having the same issue on all 3 runners shows the problem with the local server itself rather than with network insrastructure.

[iamrecursion]

Quite possibly, yes, but it occurs far more often during periods that I could expect the actions machines to be under heavier load.

Do you still want the time stats?