Avoiding HTTP 504 errors for long-running upstream requests, with Gateway API

[jdoylei]

How can we configure timeouts in Contour - based on Gateway API objects - to avoid HTTP 504 "upstream request timeout" responses from Envoy when our upstream service is slow (response in > 15 seconds)?

Background:

We have Contour's Gateway provisioner installed, currently v1.23.0, so our apps can use Gateway API with dynamic provisioning.

We hit a road-block with a slow upstream service that exceeds 15 seconds and causes Envoy to respond with HTTP 504 "upstream request timeout". I think this means that the upstream service is exceeding Envoy's default 15-second route-level timeout.

I gleaned from the Gateway API guide that I could use ContourDeployment to set some configuration, including the runtimeSettings.envoy.timeouts block with e.g. timeouts.requestTimeout. And I verified using /config_dump that Envoy's config reflects these timeouts settings, e.g. request_timeout. But this requestTimeout/request_timeout is different from the route-level timeout in Envoy's config, and changing it doesn't seem to affect this 15-second limit. It doesn't seem that any of ContourDeployment impacts the route-level timeout.

Outside of Gateway API, it looks like there are different ways of setting the route-level timeout:

• HTTPProxy timeoutPolicy.response
• Ingress annotation projectcontour.io/response-timeout

But I don't see a way using Gateway API and associated objects (GatewayClass, ContourDeployment, Gateway, HTTPRoute, Service) to set this.

How can we use Gateway API and still support a slow upstream service that takes > 15 seconds?

If there's not an immediate solution in the current Contour version, here are some thoughts:

• Can ContourDeployment runtimeSettings.envoy.timeouts get a new field that gives a global value to apply as the route-level timeout for all Envoy routes?
• Can Contour process the annotation projectcontour.io/response-timeout on Gateway, similar to what it does on Ingress? Or even on Service (since there are already Service Contour annotations that seem like they map to Envoy route config)?
• Can Contour implement a policy CRD for settings like the route-level timeout along the lines of the Gateway policy attachment spec? E.g. we would create a ContourTimeoutPolicy with a setting response: '30s' and targetRef pointing to an HTTPRoute?

Thanks for your time!

Environment:

• Contour version: v1.23.0
• Kubernetes version: (use kubectl version): v1.23.8

[github-actions]

Hey @jdoylei! Thanks for opening your first issue. We appreciate your contribution and welcome you to our community! We are glad to have you here and to have your input on Contour. You can also join us on our mailing list and in our channel in the Kubernetes Slack Workspace

[liangyuanpeng]

based on Gateway API objects - to avoid HTTP 504 "upstream request timeout"

Some ref for doc and seems like the gateway API have not deine for timeout today. It's in progress.

• GEP: Timeouts kubernetes-sigs/gateway-api#1742

[jdoylei]

Hi @liangyuanpeng, thank you for the information. I'm glad to see there is progress on this question, especially the detailed proposal documentation already on the Gateway site at Kubernetes Gateway API
GEP-1742: Timeouts. It sounds like the Envoy route-level timeout that I'm running into is reflected in "R Timeout" in the Envoy Timeouts diagram, right?

I see the Gateway API milestone for the "GEP: Timeouts" issue is v1.0.0. Contour has to wait for that Gateway API v1.0.0 release to get the new timeouts API object, and then work on supporting it in the Gateway provisioner, right? Does that mean that the timeframe for being able to configure this timeout with Contour Gateway provisioner is pretty far out? Is it likely to be this year? Thanks in advance for any info you have.

[dlip]

This is a blocker for my team, I've tried setting the following in the ContourDeployment, but it doesn't seem to help.

kind: ContourDeployment
apiVersion: projectcontour.io/v1alpha1
metadata:
  namespace: projectcontour
  name: contour-with-envoy-deployment-params
spec:
  runtimeSettings:
    policy:
      requestHeaders:
        set:
          x-envoy-upstream-rq-timeout-ms: "60000"
    envoy:
      timeouts:
        requestTimeout: 60s

Is there any possible workaround?

[github-actions]

The Contour project currently lacks enough contributors to adequately respond to all Issues.

This bot triages Issues according to the following rules:

• After 60d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, the Issue is closed

You can:

• Mark this Issue as fresh by commenting
• Close this Issue
• Offer to help out with triage

Please send feedback to the #contour channel in the Kubernetes Slack

[dlip]

This is still an issue for me

[skriss]

The Gateway API GEP (kubernetes-sigs/gateway-api#1997) has merged, so when this is available in the API we will look to implement.

ref. kubernetes-sigs/gateway-api#2013