slow / nonexistent scraping

[itdaniher]

Hey!

Awesome project, huge fan. Worked on similar things at various points in time, but haven't attacked DHT yet.

I'm trying to run magneticod on a slow machine. Seeing a significant number of WARNING Belated TICK! and with default settings, was unable to scrape any torrents after 6-8hrs of looking for nodes and peers.

Adjusting the tick interval upwards seems to have made a difference (found 6 torrents in an hour or two) but I'm hard-pressed to see any more significant results.

I spent some time investigating your source code - the state machine implementation proved to be a little hard to follow. Have you considered an asyncio / async reimplementation? It looks an awful lot like you've built your own event loop in pure Python, and I'm not sure that'll be ultimately productive.

Anyways, wondering what causes of scraping only very few torrents via DHT might be, given a working NAT config (port forwarding) etc. Keen to get this working

Thanks!

[boramalper]

Hey @itdaniher,

I'm trying to run magneticod on a slow machine. Seeing a significant number of WARNING Belated TICK! and with default settings, was unable to scrape any torrents after 6-8hrs of looking for nodes and peers.

It's true that magneticod has huge performance issues at this point, which limits the network utilization maximisation, as well as sometimes brings the machine on its knees.

Adjusting the tick interval upwards seems to have made a difference (found 6 torrents in an hour or two) but I'm hard-pressed to see any more significant results.

I think it's really too less. What about your Internet connection (speed, presence of a firewall, NAT forwarding...). You mentioned "given a working NAT config (port forwarding)" but currently magneticod randomly chooses a port assigned by the OS at start-up, so I guess NATs can be really problematic at this stage.

I spent some time investigating your source code - the state machine implementation proved to be a little hard to follow. Have you considered an asyncio / async reimplementation? It looks an awful lot like you've built your own event loop in pure Python, and I'm not sure that'll be ultimately productive.

I tried, but couldn't really grasped it. First of all, it seems a bit like a black magic that I don't feel comfortable working with as I felt I am not in the control, but I think it's a matter of familiarity that I can get used to. Other, and more important problem is, asyncio seem to abstract underlying sockets so much that I cannot catch the PermissionError raised by the sendto operation on the underlying UDP socket. The comment from the source code:

# This exception (EPERM errno: 1) is kernel's way of saying that "you are far too fast, chill".
# It is also likely that we have received a ICMP source quench packet (meaning, that we really need to
# slow down.

(Read more here: http://www.archivum.info/comp.protocols.tcp-ip/2009-05/00088/UDP-socket-amp-amp-sendto-amp-amp-EPERM.html)

It's crucial that I catch this error as magneticod depends on this for a very primitive congestion control mechanism.

---

To make a plan:

1. Work on the networking code of magneticod. I feel like a slow connection can lead to snowball effect, worsening congestion, or making magneticod behave unpredictably. The problem is hard to identify, but I'll try to do my best.
2. Allow users to choose which address and port to bind UDP & TCP sockets on, to enable users behind NAT.
3. Make magneticod faster. Currently, the hotspot seems to be the bencode module which encodes & decodes every single message sent & received. Maybe it can be re-written in Cython, but I would like to make sure that this still does not require root permissions to install.

[itdaniher]

diff --git a/magneticod/magneticod/__main__.py b/magneticod/magneticod/__main__.py
index 8344f9e..ebcc802 100644
--- a/magneticod/magneticod/__main__.py
+++ b/magneticod/magneticod/__main__.py
@@ -30,7 +30,7 @@ from . import dht
 from . import persistence


-TICK_INTERVAL = 1  # in seconds (soft constraint)
+TICK_INTERVAL = 5  # in seconds (soft constraint)
 # maximum (inclusive) number of active (disposable) peers to fetch the metadata per info hash at the same time:
 MAX_ACTIVE_PEERS_PER_INFO_HASH = 5

diff --git a/magneticod/magneticod/dht.py b/magneticod/magneticod/dht.py
index cc7d508..a4ee8fa 100644
--- a/magneticod/magneticod/dht.py
+++ b/magneticod/magneticod/dht.py
@@ -42,7 +42,7 @@ class SybilNode:
         self.__true_id = self.__random_bytes(20)

         self.__socket = socket.socket(type=socket.SOCK_DGRAM)
-        self.__socket.bind(("0.0.0.0", 0))
+        self.__socket.bind(("0.0.0.0", 10001))
         self.__socket.setblocking(False)

         self.__incoming_buffer = array.array("B", (0 for _ in range(65536)))

[itdaniher]

It's 100Mbps from the device to speedtest.net. Port 10001 UDP forwarded.

[boramalper]

@itdaniher magneticod also uses TCP to download metadata from the peers discovered. See

magnetico/magneticod/magneticod/bittorrent.py

Line 32 in 257912a

self.__socket = socket.socket()

Do you think it might be a problem in your case?

[itdaniher]

Quite possibly. I'll adjust my router configs accordingly. Surprisingly, metadata was successfully fetched for at least a few doz torrents...

https://docs.python.org/3/library/asyncio-protocol.html#flow-control-callbacks

If I get some free time, I'll investigate if this method of maintaining a UDP buffer actually works on a modern Linux system.

[itdaniher]

Quite possibly. I'll adjust my router configs accordingly.

Investigating L32, looks like this socket is just used for opening TCP connections, not listening?

edit:

read more source, and it looks like there's no reason / need to adjust TCP configs to account for NAT, as it's just used to connect() to a peer.

[boramalper]

@itdaniher Right. I don't even think that it can be the source of the problem, but just in case.

edit: I am really in the dark in terms of network programming, ironically. :)

[itdaniher]

@boramalper well, I'm happy to help as I can allocate the time.

I dedicated a VM ($5/mo) to running magnetico. Works just fine, uses 100% of the CPU though. :)

I ran some cprofile'in and it looks like select and associated socket operations are also pretty heavy. I hypothesise that asyncio might be of use there, especially given that if you buy into the event loop dogma, you get libuv for free (https://github.com/MagicStack/uvloop) which would almost certainly help things....

[itdaniher]

re bencoding: https://pypi.python.org/pypi/bencoder.pyx

[itdaniher]

replaced your bencode.py with:

import typing
import bencoder

Types = typing.Union[int, bytes, list, "KRPCDict"]
KRPCDict = typing.Dict[bytes, Types]


def dumps(obj) -> bytes:
    return bencoder.bencode(obj)

def loads(bytes_object: bytes) -> Types:
    return bencoder.bdecode2(bytes_object)[0]

def loads2(bytes_object: bytes) -> typing.Tuple[Types, int]:
    return bencoder.bdecode2(bytes_object)

BencodeDecodingError = bencoder.BTFailure

improves scraping speed, doesn't require root or gcc to install (afaik)

[itdaniher]

required patching bencoder.pyx as follows:

diff --git a/bencoder.pyx b/bencoder.pyx
index ce67968..f2bddcb 100644
--- a/bencoder.pyx
+++ b/bencoder.pyx
@@ -97,6 +97,13 @@ def bdecode(bytes x):
         raise BTFailure("invalid bencoded value (data after valid prefix)")
     return r

+def bdecode2(bytes x):
+    try:
+        r, l = decode_func[x[0]](x, 0)
+    except (IndexError, KeyError, ValueError):
+        raise BTFailure("not a valid bencoded string")
+    return r, l
+

 def encode(v, r):
     tp = type(v)

[itdaniher]

Only other difference is that bencoder.pyx returns an OrderedDict, not a dict... something has to give, I patched bencoder.pyx but that probably doesn't work for them.