Elasticsearch pilot not syncing on schedule

[munnerz]

Looking at the logs below, it appears the elasticsearch pilot sometimes does not sync on its schedule (every 10s).

This could either be due to the sync loop blocking due to lack of timeouts somewhere, or the genericpilot or scheduledWorkQueue having some kind of issue.

Needs investigation.

https://storage.googleapis.com/jetstack-logs/pr-logs/pull/jetstack_navigator/162/navigator-e2e-v1-9/90/artifacts/test-elasticsearchcluster-1516211071-11552/dump.txt

/kind bug

[munnerz]

I think this is due to rate limiting in our work queue - when the Pilot is starting, the sync fails a lot due to the ES client not being available thus causing the back off to balloon.