frontera converts keys in scrapy's response meta to bytes

[voith]

while running crawler in https://github.com/sibiryakov/frontera-google I get the following error.

2016-10-04 23:38:19 [scrapy] ERROR: Spider error processing <GET http://zpravy.idnes.cz/madarsko-zmena-ustavy-zakaz-kvot-migrace-f1c-/zahranicni.aspx?c=A161004_175512_zahranicni_mlb> (referer: None)
Traceback (most recent call last):
  File "/home/voith/Projects/frontera-google/env-3.5/lib/python3.5/site-packages/twisted/internet/defer.py", line 587, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/voith/Projects/frontera-google/env-3.5/lib/python3.5/site-packages/scrapy/spiders/crawl.py", line 68, in _response_downloaded
    rule = self._rules[response.meta['rule']]
KeyError: 'rule'

I ran the crawler without using frontera(only using scrapy) and it ran fine. That gave me a hint that frontera must be causing the error.
from the little debugging that I did, I figured out the following:

> /home/voith/Projects/frontera-google/env-3.5/lib/python3.5/site-packages/scrapy/spiders/crawl.py(68)_response_downloaded()
-> rule = self._rules[response.meta['rule']]
(Pdb) response.meta
{'download_timeout': 180.0, b'frontier_request': <Request at 0x7f1b39a9cc50 https://www.alza.cz/nejobchod meta={b'scrapy_callback': b'_response_downloaded', b'score': 0.16666666666666666, b'scrapy_errback': None, b'scrapy_meta': {b'rule': 0, b'frontier_request': <Request at 0x7f1b39a9cc50 https://www.alza.cz/nejobchod meta={...} body=... cookies={}, headers={}>, b'link_text': b'\r\n\r\n'}, b'domain': {b'netloc': b'www.alza.cz', b'tld': b'', b'subdomain': b'', b'fingerprint': b'4273593b6eda0469d3db862aa0fda848d7a89974', b'name': b'www.alza.cz', b'scheme': b'https', b'sld': b''}, b'fingerprint': b'7ee99dd119df89a7a103e1001b97f1255ea6e466', b'state': 1, b'origin_is_frontier': True, b'jid': 0} body=... cookies={}, headers={}>, b'rule': 0, 'download_latency': 0.33022451400756836, b'link_text': b'\r\n\r\n', 'download_slot': 'www.alza.cz'}

(Pdb) response.meta['rule']
*** KeyError: 'rule'

(Pdb) response.meta[b'rule']
0

Since the keys have been converted into bytes, they cannot be accessed by just using the key name.

I have yet to figure which part of the code is exactly causing this. If I figure it out, I'll definitely submit a fix for this. But I'm logging this for now so that it can be tracked.

[sibiryakov]

This is a design decision http://frontera.readthedocs.io/en/latest/topics/frontier-objects.html#frontera.core.models.Request.meta, since Frontera needs to serialize these objects, and loose all the string type information we wanted to keep it simple.

[voith]

So I went ahead and debugged this a little further

In [1]: from frontera.contrib.backends.remote.codecs.msgpack import Decoder, Encoder

In [2]: from frontera.core.models import Request

In [3]: r = Request('http://example.com', meta={b'frontera_key': b'frontera_value', b'scrapy_meta': {'rule': 0, 'link': 'some_link'}})

In [4]: encoder = Encoder(Request)

In [5]: class DumbResponse:
   ...:     pass

In [7]: decoder = Decoder(Request, DumbResponse)

In [8]: decoder.decode_request(encoder.encode_request(r)).meta
Out[8]: 
{b'frontera_key': b'frontera_value',
 b'scrapy_meta': {b'link': b'some_link', b'rule': 0}}

from the the above output Its clear to me that the whole encoding-decoding scheme is what causes the issue. I think the right place to fix this issue would be in the decoder. May be here: https://github.com/scrapinghub/frontera/blob/master/frontera/contrib/backends/remote/codecs/msgpack.py#L82-L87
@sibiryakov @Preetwinder what do you guys think

[voith]

I think an encoder-decoder should retrieve the original form of a message regardless of it going through the entire encoding decoding scheme. So I think the right place to solve this is at the decoder.

[kmike]

That's fine to have bytes as meta keys in frontera Request objects, but bytes keys in scrapy.Request meta don't look correct: they don't compare equal to strings in Python 3, so this may break scrapy middlewares or user code in Python 3: dont_merge_cookies, dont_obey_robotstxt, dont_cache, dont_redirect, etc. meta keys could stop working.

[kmike]

That's fine to have bytes as meta keys in frontera Request objects

Well, the common recommendation for pain-free unicode in Python is to encode/decode data at the "borders", i.e. when you're interacting with the outside world, and use unicode internally for text data. It is a bit moot in Python 2 because "native strings" are commonly used for constants (and that's fine), but even in Python 2 this approach works well.