Improve IRC Gateway

[roberthoenig]

See zulip/zulip#249

Moving the issue here, let's have a discussion on what parts of the original one are still relevant.

[timabbott]

I think the big thing we need to do here is have step-by-step instructions for how to run it, documenting its limitations. We can probably just put that in the /integrations/ docs. And then we can open issues for fixing the major limitations revealed in that documentation.

[timabbott]

The discussion in zulip/zulip#249 should cover the special Zulip things one needs to know to run it, specifically the need to use manage.py knight to create an "API super user" bot.

[rht]

I tried running the mirror a while ago, found the USAGE content to be sufficient
to get me running. I modified the code so that 1. being an api-super-user is no
longer required, 2. it redirects the IRC messages to a hardcoded topic in a
hardcoded stream (this can be quickly patched as options to the current code
if necessary). Basically it is a one-liner of:
./irc-mirror.py --irc-server=irc.freenode.net --channel='#python-mypy' --nick-prefix='irc_mirror' --site="https://chat.zulip.org" --user=irc-bot@chat.zulip.org --api-key=$API_KEY.
This works.

However, a problem occurred when I tried to enable the bi-directional mode
(forwarding messages in Zulip to IRC) by uncommenting the line

python-zulip-api/zulip/integrations/irc/irc_mirror_backend.py

Line 45 in 57f7088

# self.zulip_client.call_on_each_message(forward_to_irc)

.
This line is a blocking synchronous call, and furthermore it prevents the
forwarding from IRC to Zulip from happening, i.e. this line makes everything
non-functional. The jaraco/irc library also has a problem in that it can't
sustain a connection for longer than 30 min, at which point a ping timeout will
happen.

I tried another library, pydle and reconstructed the public subset of the
mirroring script, but reached a dead end in the bidirectional area because
pydle has only one thread that captures the events happening in the IRC
channel (on_connect, on_channel_message, etc). It can't be hooked to an external
event listener such as events in a Zulip server (to have messages being
forwarded from Zulip to the IRC channel). One plus point for pydle is that it
can sustain a more stable connection than jaraco/irc.

The only stable bidirectional solution that has worked so far is via Matrix
(note: they had the entirety of freenode mirrored). So currently I have #git
bidirectional-mirrored to
https://chat.zulip.org/#narrow/stream/IRC/topic/.23git.
Another possible path is to use nodejs (if there is a better IRC library in the
ecosystem) instead of Python.

[timabbott]

The bidirectional model that we have in mind with this script historically is just running 2 copies (after presumably patching it to make that commented out section an option). If you look at zephyr_mirror.py, it runs 2 copies of zephyr_mirror_backend.py with different options, one for each direction; that system is quite reliable.

The 30m timeout issue with the current library is annoying, though.

[rht]

The bidirectional model that we have in mind with this script historically is just running 2 copies (after presumably patching it to make that commented out section an option).

Yeah, I forgot to tell that I did also attempt to run two specialized scripts -- one for forwarding to IRC (zulip-irc-receiver), another for forwarding from IRC (zulip-irc-sender). This works for up to 30 min only though, after which they have to be restarted again.

[rheaparekh]

The current IRC gateway manages to run with two copies (for both side message forwarding) running. However all the Issues I identified were:

• Ping timeout for zulip-> IRC gateway (issue of the lib jaraco/irc, however they are planning to move to asyncio Add support for asyncio to client jaraco/irc#135, with an open PR on it!)
• Message formatting isn't correct, the users sending the messages in Zulip are not being shown in IRC.
• Messages are replicated (When we forward a message from Zulip->IRC, IRC forward the same message back to Zulip)

[rht]

Ping timeout for zulip-> IRC gateway (issue of the lib jaraco/irc, however they are planning to move to asyncio jaraco/irc#135, with an open PR on it!)

@rheaparekh the PR was merged on May 21 2018. Did this resolve the timeout issue?

[rht]

Confirmed that irc.client_aio.AioReactor fixes the timeout. I ran ./irc-mirror.py --irc-server=irc.freenode.net --channel='#python-mypy' --nick-prefix='irc_mirror' --site="https://chat.zulip.org" --user=irc-bot@chat.zulip.org --api-key=$API_KEY again and had observed the 15-30 min disconnect become stable (1 hour and counting).

Then the remaining problem of running 2 scripts for each direction can be solved with threading once for all.

The patch: rht@7c08c98

[iweiss]

@rht I've been trying the latest on #469 and Zulip->IRC communication doesn't happen. Apparently it's not even calling forward_to_irc(). IRC->Zulip works fine.

[rht]

@iweiss you must make sure the bot is subscribed to the stream.

[iweiss]

@rht Perfect! Many thanks.

Any plans for IRC+SSL?

[rht]

I haven't tested this, but you could try editing irc_mirror_backend.py.

1. Add import ssl and import irc.client.
2. At the line containing self.connection.connect(*args, **kwargs) add an argument so that it becomes self.connection.connect(*args, **kwargs, connect_factory=ssl_factory), where you have to add this line to the line before self.reactor.loop.run_until_complete(: ssl_factory = irc.connection.Factory(wrapper=ssl.wrap_socket).

If this works, I will update the PR to have the flag to enable ssl.

[iweiss]

Sorry but that didn't work. self.connection.connect() doesn't recognise connect_factory.

Traceback (most recent call last):
  File "./irc-mirror.py", line 59, in <module>
    bot.start()
  File "/home/matrix/x/python-zulip-api/zulip-api-py3-venv/lib/python3.6/site-packages/irc/bot.py", line 333, in start
    self._connect()
  File "/home/matrix/x/python-zulip-api/zulip-api-py3-venv/lib/python3.6/site-packages/irc/bot.py", line 182, in _connect
    **self.__connect_params)
  File "/home/matrix/x/python-zulip-api/zulip/integrations/bridge_with_irc/irc_mirror_backend.py", line 37, in connect
    self.connection.connect(*args, **kwargs, connect_factory=ssl_factory)
TypeError: connect() got an unexpected keyword argument 'connect_factory'