Race condition at usage component startup

[andrew-farries]

There is a race condition at usage component startup between the billing gRPC service being available and the billing client trying to connect to it:

{"config":{"controllerSchedule":"10s","creditsPerMinuteByWorkspaceClass":{"default":0.1666666667,"gitpodio-internal-xl":0.3333333333},"server":{"services":{"grpc":{"address":":9001"}}}},"level":"info","message":"Starting usage component.","serviceContext":{"service":"usage","version":""},"severity":"INFO","time":"2022-07-29T08:43:30Z"}
{"level":"info","message":"Starting usage controller.","serviceContext":{"service":"usage","version":""},"severity":"INFO","time":"2022-07-29T08:43:30Z"}
{"component":"grpc","level":"warning","message":"2022/07/29 08:43:30 WARNING: [core] [Channel #2 SubChannel #3] grpc: addrConn.createTransport failed to connect to {","serviceContext":{"service":"usage","version":""},"severity":"WARNING","time":"2022-07-29T08:43:30Z"}
{"component":"grpc","level":"warning","message":"  \"Addr\": \":9001\",","serviceContext":{"service":"usage","version":""},"severity":"WARNING","time":"2022-07-29T08:43:30Z"}
{"component":"grpc","level":"warning","message":"  \"ServerName\": \"localhost:9001\",","serviceContext":{"service":"usage","version":""},"severity":"WARNING","time":"2022-07-29T08:43:30Z"}
{"component":"grpc","level":"warning","message":"  \"Attributes\": null,","serviceContext":{"service":"usage","version":""},"severity":"WARNING","time":"2022-07-29T08:43:30Z"}
{"component":"grpc","level":"warning","message":"  \"BalancerAttributes\": null,","serviceContext":{"service":"usage","version":""},"severity":"WARNING","time":"2022-07-29T08:43:30Z"}
{"component":"grpc","level":"warning","message":"  \"Type\": 0,","serviceContext":{"service":"usage","version":""},"severity":"WARNING","time":"2022-07-29T08:43:30Z"}
{"component":"grpc","level":"warning","message":"  \"Metadata\": null","serviceContext":{"service":"usage","version":""},"severity":"WARNING","time":"2022-07-29T08:43:30Z"}
{"component":"grpc","level":"warning","message":"}. Err: connection error: desc = \"transport: Error while dialing dial tcp :9001: connect: connection refused\"","serviceContext":{"service":"usage","version":""},"severity":"WARNING","time":"2022-07-29T08:43:30Z"}

This is easy to trigger when running the component locally. In a preview environment it also happens regularly when the component starts.

This is a warning rather than an error because gRPC dialling includes logic for backoff and automatic retry.

See this comment on #1165.