test_create_and_delete_metric_descriptor gives 500 failure

[dhermes]

Done while creating a descriptor

POST https://monitoring.googleapis.com/v3/projects/precise-truck-742/metricDescriptors/

• https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/154552669
• https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/154561565
• https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/154598135

/cc @supriyagarg

[rimey]

Thank you! I believe you spotted a problem in an ongoing rollout. We're taking action to fix it.

You just happened to be hitting our canary server instances.

[rimey]

This should now be resolved.

Occasional, rare 500s are always possible, but if you see something that appears not to be transient, please let us know.

[dhermes]

I mostly agree with you but want to discuss a little before hastily closing. Is there an SLA for the service?

The whole point of the flaky label is that we don't want to have to have 1/4 of our merges to master fail. Brittle system tests are no fun and the more services we support, the worse it gets.

rare 500s are always possible

I somewhat agree (depending on the SLA). However, we got two runs of 500s in a row (4 requests in total) spaced out over several minutes.

The real question is, should we wrap the API calls in a retry for a 500?

[rimey]

Whether to retry API calls in the system tests on 500s is completely up to you. I don't have further advice for you about that.

[dhermes]

@tseaver WDYT about this?

[tseaver]

If it isn't an "expected" error from the point of view of the back-end team, I'd say leave out the retries.

[dhermes]

👍