Not only batch writes as we currently do. In particular, retry these requests: http://support.scrapinghub.com/topic/369068-network-timeouts-to-storagescrapinghubcom/ Probably using request HTTPAdapter retries: http://www.python-requests.org/en/latest/api/#requests.adapters.HTTPAdapter