Adds Wireshark pipes support, cleanup OSX tcpdump handling. (#1959)

* Adds Wireshark pipes support, cleanup OSX tcpdump handling.

* SniffSource now only opens a socket on calling `start`.
* WrpcapSink only opens a PcapWriter on calling `start`.
* Adds WiresharkSink for streaming packets to Wireshark through a pipe.
* Adds `fd` parameter to `utils.get_temp_file`, which allows a temporary
  file to be used without closing and re-opening it.
* Adds `utils.get_temp_dir` (used for tests)
* Adds `use_tempfile` parameter to `utils.tcpdump`, which causes it to
  use a temporary file to store packets.
* `utils.tcpdump` now only uses a temporary file by default for calling
  tcpdump on OSX (to work around Apple's broken version of tcpdump).
  stdin is now used with other tools which are not impacted by this bug
  (eg: tshark).
* Adds `read_stdin_opts` parameter to `utils.tcpdump`, which allows
  callers to control the options used for reading packets over stdin
  (previously hard-coded to `-r -`).
* `utils.wireshark` now uses a pipe rather than a temporary file.
  Wireshark itself has options to save this file to disk if desired.

Also cleans up the tests a little:

* Removes `os.remove("test.png")` (which seems to be unused).
* Uses a temporary directory for some pipetool tests.

Includes fixup:

* spelling

* Documentation for WiresharkSink, Pipetools sinks and wireshark():

* Enable `manpages_url`, so we can use manpage links.

* Pipetools: Revise Sinks section to use `py:class` style documentation.
  This makes cross-references to specific sinks much easier.

* Add documentation for `WiresharkSink`.

* Pipetools: Revise Link object section with some word-smithing.

* Usage/wireshark:

  * improve wordsmithing
  * remove hard coded references to `google.com` (which trigger DNS
    lookups), and use an RFC 5737 netmask instead.
  * set a source IP address (otherwise, this gets the host's IP address).
  * remove `Ether` layer, as Wireshark supports `DLT_RAW` (which triggers
    getting the host's MAC address).
  * elaborate on mixed `linktype` issues.

* Pipetool: fixups:

* `QueueSink.recv`: add `block` and `timeout` parameters
* `WrpcapSink`, `WiresharkSink`: add `linktype` parameter
* `WiresharkSink`: add `args` parameter
* tests: sleep longer to work-around race conditions on Windows

* utils: many fixups

* `{Raw,}PcapWriter._write_packet`:

  * Remove unused support for `packet` as tuple, as `write` will always
    unroll iterators for us (and do it better).
  * Always set the `usec` parameter if `sec` was unset.
  * Set `usec=0` if `sec` was set, but `usec` was unset (instead of using
    the current time's usec value
  * PEP-8 fixup, add docstring.
  * Only write the header if there is a packet

* `PcapWriter._write_packet`: support packet as bytes

* `RawPcapWriter.close`: write the file header here if not already written

* `tcpdump()`:

  * Add `linktype` parameter, like `wrpcap(linktype=...)`
  * Add `wait` parameter, which controls whether a program should be run in
    the background. Defaults to `True` (run in foreground).
  * Throw an error if `prog` is not a string.
  * Copy `read_stdin_opts` (for thread safety).

* `tdecode()`: Add `args` parameter (defaults to `-V`, as before), pass
  other `tcpdump()` kwargs.

* `wireshark()`: Run in the background by default, pass other `tcpdump()`
  kwargs.

* Add tests that hit `wireshark`, `tdecode`, `tcpdump` with new parameters,
  and try to pass packets as bytes.

Author: micolous

doc/scapy/advanced_usage.rst

@@ -790,6 +790,7 @@ Two methods are hooks to be overloaded:
 
 * The ``master_filter()`` method is called each time a packet is sniffed and decides if it is interesting for the automaton. When working on a specific protocol, this is where you will ensure the packet belongs to the connection you are being part of, so that you do not need to make all the sanity checks in each transition.
 
+.. _pipetools:
 
 PipeTools
 =========
@@ -882,7 +883,10 @@ class ConsoleSink(Sink)
 Sources
 ^^^^^^^
 
-A Source is a class that generates some data. They are several source types integrated with Scapy, usable as-is, but you may also create yours.
+A Source is a class that generates some data.
+
+There are several source types integrated with Scapy, usable as-is, but you may
+also create yours.
 
 Default Source classes
 ~~~~~~~~~~~~~~~~~~~~~~
@@ -951,25 +955,165 @@ For instance, here is how TransformDrain is implemented::
 Sinks
 ^^^^^
 
+Sinks are destinations for messages.
+
+A :py:class:`Sink` receives data like a :py:class:`Drain`, but doesn't send any
+messages after it.
+
+Messages on the low entry come from :py:meth:`~Sink.push`, and messages on the
+high entry come from :py:meth:`~Sink.high_push`.
+
 Default Sink classes
 ~~~~~~~~~~~~~~~~~~~~
 
-- Sink : does not do anything. This must be extended to create custom sinks
-- ConsoleSink : Print messages on low and high entries
-- RawConsoleSink : Print messages on low and high entries, using os.write
-- TermSink : Print messages on low and high entries on a separate terminal
-- QueueSink: Collect messages from high and low entries and queue them. Messages are unqueued with the .recv() method.
+.. py:class:: Sink
+
+    Does nothing; interface to extend for custom sinks.
+
+    All sinks have the following constructor parameters:
+
+    :param name: a human-readable name for the element
+    :type name: str
+
+    All sinks should implement at least one of these methods:
+
+    .. py:method:: push
+
+        Called by :py:class:`PipeEngine` when there is a new message for the
+        low entry.
+
+        :param msg: The message data
+        :returns: None
+        :rtype: None
+
+    .. py:method:: high_push
+
+        Called by :py:class:`PipeEngine` when there is a new message for the
+        high entry.
+
+        :param msg: The message data
+        :returns: None
+        :rtype: None
+
+.. py:class:: ConsoleSink
+
+    Prints messages on the low and high entries to ``stdout``.
+
+.. py:class:: RawConsoleSink
+
+    Prints messages on the low and high entries, using :py:func:`os.write`.
+
+    :param newlines: Include a new-line character after printing each packet.
+                     Defaults to True.
+    :type newlines: bool
+
+.. py:class:: TermSink
+
+    Prints messages on the low and high entries, on a separate terminal (xterm
+    or cmd).
+
+    :param keepterm: Leaves the terminal window open after :py:meth:`~Pipe.stop`
+                     is called. Defaults to True.
+    :type keepterm: bool
+    :param newlines: Include a new-line character after printing each packet.
+                     Defaults to True.
+    :type newlines: bool
+    :param openearly: Automatically starts the terminal when the constructor is
+                      called, rather than waiting for :py:meth:`~Pipe.start`.
+                      Defaults to True.
+    :type openearly: bool
+
+.. py:class:: QueueSink
+
+    Collects messages on the low and high entries into a :py:class:`Queue`.
+
+    Messages are dequeued with :py:meth:`recv`.
+
+    Both high and low entries share the same :py:class:`Queue`.
+
+    .. py:method:: recv
+
+        Reads the next message from the queue.
+
+        If no message is available in the queue, returns None.
+
+        :param block: Blocks execution until a packet is available in the queue.
+                      Defaults to True.
+        :type block: bool
+        :param timeout: Controls how long to wait if ``block=True``. If None
+                        (the default), this method will wait forever. If a
+                        non-negative number, this is a number of seconds to
+                        wait before giving up (and returning None).
+        :type timeout: None, int or float
+
+.. py:class:: WiresharkSink
+
+    Streams :py:class:`Packet` from the low entry to Wireshark.
+
+    Packets are written into a ``pcap`` stream (like :py:class:`WrpcapSink`),
+    and streamed to a new Wireshark process on its ``stdin``.
+
+    Wireshark is run with the ``-ki -`` arguments, which cause it to treat
+    ``stdin`` as a capture device.  Arguments in :py:attr:`args` will be
+    appended after this.
+
+    Extends :py:mod:`WrpcapSink`.
+
+    :param linktype: See :py:attr:`WrpcapSink.linktype`.
+    :type linktype: None or int
+    :param args: See :py:attr:`args`.
+    :type args: None or list[str]
+
+    .. py:attribute:: args
+
+        Additional arguments for the Wireshark process.
+
+        This must be either ``None`` (the default), or a ``list`` of ``str``.
+
+        This attribute has no effect after calling :py:meth:`PipeEngine.start`.
+
+        See :manpage:`wireshark(1)` for more details.
+
+.. py:class:: WrpcapSink
+
+    Writes :py:class:`Packet` on the low entry to a ``pcap`` file.
+
+    Ignores all messages on the high entry.
+
+    .. note::
+
+        Due to limitations of the ``pcap`` format, all packets **must** be of
+        the same link type. This class will not mutate packets to conform with
+        the expected link type.
+
+    :param fname: Filename to write packets to.
+    :type fname: str
+    :param linktype: See :py:attr:`linktype`.
+    :type linktype: None or int
+
+    .. py:attribute:: linktype
+
+        Set an explicit link-type (``DLT_``) for packets.  This must be an
+        ``int`` or ``None``.
+
+        This is the same as the :py:func:`wrpcap` ``linktype`` parameter.
+
+        If ``None`` (the default), the linktype will be auto-detected on the
+        first packet. This field will *not* be updated with the result of this
+        auto-detection.
+
+        This attribute has no effect after calling :py:meth:`PipeEngine.start`.
+
 
 Create a custom Sink
 ~~~~~~~~~~~~~~~~~~~~
 
-To create a custom sink, one must extend the ``Sink`` class.
+To create a custom sink, one must extend :py:class:`Sink` and implement
+:py:meth:`~Sink.push` and/or :py:meth:`~Sink.high_push`.
 
-A ``Sink`` class receives data like a ``Drain``, from the lower canal in its ``push`` method, and from the higher canal from its ``high_push`` method.
+This is a simplified version of :py:class:`ConsoleSink`:
 
-A ``Sink`` is the dead end of data, it won't be sent anywhere after it.
-
-For instance, here is how ConsoleSink is implemented::
+.. code-block:: python3
 
     class ConsoleSink(Sink):
         def push(self, msg):
@@ -980,32 +1124,44 @@ For instance, here is how ConsoleSink is implemented::
 Link objects
 ------------
 
-As shown in the example, most sources can be linked to any drain, on both lower and higher canals.
+As shown in the example, most sources can be linked to any drain, on both low
+and high entry.
 
-The use of ``>`` indicates a link on the low canal, and ``>>`` on the higher one.
+The use of ``>`` indicates a link on the low entry, and ``>>`` on the high
+entry.
 
-For instance
+For example, to link ``a``, ``b`` and ``c`` on the low entries:
 
->>> a = CLIFeeder()
->>> b = Drain()
->>> c = ConsoleSink()
->>> a > b > c
->>> p = PipeEngine()
->>> p.add(a)
+.. code-block:: pycon
+
+    >>> a = CLIFeeder()
+    >>> b = Drain()
+    >>> c = ConsoleSink()
+    >>> a > b > c
+    >>> p = PipeEngine()
+    >>> p.add(a)
+
+This wouldn't link the high entries, so something like this would do nothing:
+
+.. code-block:: pycon
+
+    >>> a2 = CLIHighFeeder()
+    >>> a2 >> b
+    >>> a2.send("hello")
 
-This links a, b, and c on the lower canal. If you tried to send anything on the higher canal, for instance by adding
+Because ``b`` (:py:class:`Drain`) and ``c`` (:py:class:`ConsoleSink`) are not
+linked on the high entry.
 
->>> a2 = CLIHighFeeder()
->>> a2 >> b
->>> a2.send("hello")
+However, using a :py:class:`DownDrain` would bring the high messages from
+:py:class:`CLIHighFeeder` to the lower channel:
 
-It would not do anything as the Drain is not linked to the Sink on the upper canal. However, one could do
+.. code-block:: pycon
 
->>> a2 = CLIHighFeeder()
->>> b2 = DownDrain()
->>> a2 >> b2
->>> b2 > b
->>> a2.send("hello")
+    >>> a2 = CLIHighFeeder()
+    >>> b2 = DownDrain()
+    >>> a2 >> b2
+    >>> b2 > b
+    >>> a2.send("hello")
 
 The PipeEngine class
 --------------------

doc/scapy/conf.py

@@ -106,6 +106,8 @@
     ]
 }
 
+# Make :manpage directive work on HTML output.
+manpages_url = 'https://manpages.debian.org/{path}'
 
 # -- Options for HTMLHelp output ------------------------------------------
 

doc/scapy/usage.rst

@@ -1533,28 +1533,76 @@ Viewing packets with Wireshark
 Problem
 ^^^^^^^
 
-You have generated or sniffed some packets with Scapy and want to view them with `Wireshark <http://www.wireshark.org>`_, because of its advanced packet dissection abilities.
+You have generated or sniffed some packets with Scapy.
+
+Now you want to view them with `Wireshark <https://www.wireshark.org>`_, because
+of its advanced packet dissection capabilities.
 
 Solution
 ^^^^^^^^
 
-That's what the ``wireshark()`` function is for:
+That's what :py:func:`wireshark` is for!
+
+.. code-block:: python3
 
-    >>> packets = Ether()/IP(dst=Net("google.com/30"))/ICMP()     # first generate some packets
-    >>> wireshark(packets)                                        # show them with Wireshark
+    # First, generate some packets...
+    packets = IP(src="192.0.2.9", dst=Net("192.0.2.10/30"))/ICMP()
 
-Wireshark will start in the background and show your packets.
+    # Show them with Wireshark
+    wireshark(packets)
+
+Wireshark will start in the background, and show your packets.
 
 Discussion
 ^^^^^^^^^^
 
-The ``wireshark()`` function generates a temporary pcap-file containing your packets, starts Wireshark in the background and makes it read the file on startup.   
+.. py:function:: wireshark(pktlist, ...)
+
+    With a :py:class:`Packet` or :py:class:`PacketList`, serialises your
+    packets, and streams this into Wireshark via ``stdin`` as if it were a
+    capture device.
+
+    Because this uses ``pcap`` format to serialise the packets, there are some
+    limitations:
+
+    * Packets must be all of the same ``linktype``.
+
+      For example, you can't mix :py:class:`Ether` and :py:class:`IP` at the
+      top layer.
+
+    * Packets must have an assigned (and supported) ``DLT_*`` constant for the
+      ``linktype``.  An unsupported ``linktype`` is replaced with ``DLT_EN10MB``
+      (Ethernet), and will display incorrectly in Wireshark.
+
+      For example, can't pass a bare :py:class:`ICMP` packet, but you can send
+      it as a payload of an :py:class:`IP` or :py:class:`IPv6` packet.
+
+    With a filename (passed as a string), this loads the given file in
+    Wireshark. This needs to be in a format that Wireshark supports.
+
+    You can tell Scapy where to find the Wireshark executable by changing the
+    ``conf.prog.wireshark`` configuration setting.
+
+    This accepts the same extra parameters as :py:func:`tcpdump`.
+
+.. seealso::
+
+    :py:class:`WiresharkSink`
+        A :ref:`PipeTools sink <pipetools>` for live-streaming packets.
 
-Please remember that Wireshark works with Layer 2 packets (usually called "frames"). So we had to add an ``Ether()`` header to our ICMP packets. Passing just IP packets (layer 3) to Wireshark will give strange results.
+    :manpage:`wireshark(1)`
+        Additional description of Wireshark's functionality, and its
+        command-line arguments.
 
-You can tell Scapy where to find the Wireshark executable by changing the ``conf.prog.wireshark`` configuration setting.
+    `Wireshark's website`__
+        For up-to-date releases of Wireshark.
 
+    `Wireshark Protocol Reference`__
+        Contains detailed information about Wireshark's protocol dissectors, and
+        reference documentation for various network protocols.
 
+__ https://www.wireshark.org
+__ https://wiki.wireshark.org/ProtocolReference
 
 OS Fingerprinting
 -----------------

scapy/pipetool.py

@@ -652,12 +652,11 @@ def push(self, msg):
     def high_push(self, msg):
         self.q.put(msg)
 
-    def recv(self):
-        while True:
-            try:
-                return self.q.get(True, timeout=0.1)
-            except six.moves.queue.Empty:
-                pass
+    def recv(self, block=True, timeout=None):
+        try:
+            return self.q.get(block=block, timeout=timeout)
+        except six.moves.queue.Empty:
+            pass
 
 
 class TransformDrain(Drain):