FronteraScheduler._request_is_redirected looks suspicious

[kmike]

See

frontera/frontera/contrib/scrapy/schedulers/frontier.py

Line 169 in d91e056

def _request_is_redirected(self, request):

:

    def _request_is_redirected(self, request):
        return request.meta.get(b'redirect_times', 0) > 0

^^ in Python 2 this is the same as checking for 'redirect_times' key, but in Python 3 b'redirect_times' != 'redirect_times', so this condition won't work with Scrapy's redirect middlewares in Python 3.

Tests are not catching that because in scheduler tests scrapy.Request objects are created with bytes keys in meta:

frontera/tests/test_frontera_scheduler.py

Line 20 in d91e056

rr1 = Request('http://www.example.com', meta={b'redirect_times': 1})

. Scrapy don't create such responses: it always use native strings (str, i.e. bytes in Python 2 and uncode in Python 3) as meta keys.

[voith]

also look at #211

[kmike]

@voith yeah, that's what I was looking at right now :)

[sibiryakov]

@kmike I'm not sure the code using that method is executed

frontera/frontera/contrib/scrapy/schedulers/frontier.py

Line 92 in d91e056

if not self._request_is_redirected(request):

and honestly I don't see much sense in all this. As I can see this check is executed only if

1. there is consumption of start_requests iterable in scrapy spider,
2. some downloader middleware throws Request when processing output.

and then there is a check for redirection and depending on it bypassing of all manager/backend machinery. Looks like an artefact left from early alpha versions.

[rmax]

I got bitten by this bug. When redirect is enabled, the symptom is that the frontera gets the first seed request, receives the redirect, then adds the redirected URL as seed again and nothing happens afterwards.

I haven't verified but I think the scrapy manager handles scrapy's Request objects, and as scrapy sets the redirect_times, the key it's a native string. As @kmike pointed out.

I changed the line to request.meta.get('redirect_times') > 0 and it's working fine.