Adds Wireshark pipes support, cleanup OSX tcpdump handling. #1959

micolous · 2019-03-30T03:06:41Z

New:

WiresharkSink, a PipeTools Sink for streaming packets to Wireshark
utils.get_temp_dir: creates a temporary directory

Extensions:

tcpdump(): Adds use_tempfile, to explicitly control use of a tempfile for packets.
tcpdump(): Adds read_stdin_opts parameter, which allows callers to control the options used for reading packets over stdin (previously hard-coded to -r -). This auto-detects wireshark to use -ki - instead.
tcpdump(), WrpcapSink: Adds linktype parameter.
get_temp_file(): Adds fd parameter, which allows a temporary file to be used without closing and re-opening it.
Expands wireshark(), Source and Pipe documentation.
tdecode(): Add args parameter (defaults to -V, as before), pass kwargs to tcpdump().
QueueSink.recv: add block and timeout parameters.

Fixes:

tcpdump() now only uses a temporary file by default for calling tcpdump on OSX (to work around Apple's broken version of tcpdump). stdin is now used with other tools which are not impacted by this bug (eg: tshark).
wireshark() now uses a pipe rather than a temporary file. Wireshark itself has options to save this file to disk if desired.
SniffSource now only opens a socket on calling start.
WrpcapSink only opens a PcapWriter on calling start.
RawPcapWriter no longer writes a header on write calls with no packets specified.
RawPcapWriter writes a header on close if no header has been written.
QueueSink.recv() no longer busy-loops.
PcapWriter, wrpcap(): support packets as bytes
PEP-8 fixups and docstring revisions to touched methods.

Documentation fixes:

Documentation wordsmithing for wireshark() and Pipes
Use py:function / py:class for Sinks and wireshark() to allow easy cross-referencing, and expand this documentation.
wireshark(): Remove references to google.com, and use RFC 5737 IP addresses instead
wireshark(): Set a source IP address (to avoid looking up host's IP address).
wireshark(): Remove Ether layer, as Wireshark works fine with DLT_RAW for IP packets, which removes the need to lookup MAC addresses
Add manpages_url to configuration, to support :manpage: directives. This is pointing at Debian's server, which should have a good spread of versions of tools, and Debian generally adds manpages for tools that don't have them.

Test fixes:

Removes os.remove("test.png") (which seems to be unused).
Uses a temporary directory for some pipetool tests.

Other changes:

{Raw,}PcapWriter._write_packet: Remove unused support for packet as tuple, as write will always unroll iterators for us (and do it better).
{Raw,}PcapWriter._write_packet: Always set the usec parameter if sec was unset.
{Raw,}PcapWriter._write_packet: Set usec=0 if sec was set, but usec was unset (instead of using the current time's usec value

gpotter2 · 2019-03-31T13:06:52Z

Really like this PR !

If you could rebase, it will fix the AppVeyor failure

scapy/utils.py

codecov · 2019-04-08T06:38:16Z

Codecov Report

Merging #1959 into master will increase coverage by 0.04%.
The diff coverage is 95.91%.

@@            Coverage Diff             @@
##           master    #1959      +/-   ##
==========================================
+ Coverage   85.92%   85.96%   +0.04%     
==========================================
  Files         187      187              
  Lines       42937    42976      +39     
==========================================
+ Hits        36892    36946      +54     
+ Misses       6045     6030      -15

Impacted Files	Coverage Δ
scapy/pipetool.py	`89.91% <100%> (+0.39%)`	⬆️
scapy/scapypipes.py	`86.71% <100%> (+1.37%)`	⬆️
scapy/utils.py	`79.15% <93.33%> (+3.05%)`	⬆️
scapy/supersocket.py	`72.09% <0%> (-2.33%)`	⬇️
scapy/layers/tls/handshake_sslv2.py	`91.32% <0%> (-2.27%)`	⬇️
scapy/layers/tls/record.py	`91.22% <0%> (-1.17%)`	⬇️
scapy/asn1/ber.py	`82.81% <0%> (-0.29%)`	⬇️
scapy/layers/ntp.py	`91.51% <0%> (-0.27%)`	⬇️
scapy/sendrecv.py	`85.1% <0%> (-0.17%)`	⬇️
scapy/packet.py	`79.07% <0%> (-0.17%)`	⬇️
... and 3 more

gpotter2

Great PR, all good to me apart:

https://ci.appveyor.com/project/secdev/scapy/builds/23671224/job/cpf5sdranaeooj9k seem to be a random failure of the WiresharkSink test. Probably some racing problems :/ it should be stabilized
(optional): add a tiny reference in the pipetool documentation (advanced usage) to the new Pipes

micolous · 2019-04-09T04:57:33Z

Found another issue, tcpdump, wireshark and tshark doesn't like integer linktypes... :(

https://github.com/the-tcpdump-group/tcpdump/blob/c36abfe8a86fb2e3d8256fa53d6569a4aa65b39c/tcpdump.c#L1823-L1829

https://github.com/the-tcpdump-group/libpcap/blob/08dc0344b20dfd44958168d75c3a94ca960eb029/pcap.c#L3109-L3119

micolous · 2019-04-10T05:06:22Z

In case it isn't clear already, please don't merge this yet...

Yesterday, I was trying to debug that Windows test flake. I ran the test about 1300 times (automated, of course) in Python 3.7 on Linux, and couldn't trigger a single failure.

Interestingly only the winpcap=true version fails.

micolous · 2019-04-10T06:28:35Z

I'm going to amend the docs a little more, squash and rebase this before it's ready.

* SniffSource now only opens a socket on calling `start`. * WrpcapSink only opens a PcapWriter on calling `start`. * Adds WiresharkSink for streaming packets to Wireshark through a pipe. * Adds `fd` parameter to `utils.get_temp_file`, which allows a temporary file to be used without closing and re-opening it. * Adds `utils.get_temp_dir` (used for tests) * Adds `use_tempfile` parameter to `utils.tcpdump`, which causes it to use a temporary file to store packets. * `utils.tcpdump` now only uses a temporary file by default for calling tcpdump on OSX (to work around Apple's broken version of tcpdump). stdin is now used with other tools which are not impacted by this bug (eg: tshark). * Adds `read_stdin_opts` parameter to `utils.tcpdump`, which allows callers to control the options used for reading packets over stdin (previously hard-coded to `-r -`). * `utils.wireshark` now uses a pipe rather than a temporary file. Wireshark itself has options to save this file to disk if desired. Also cleans up the tests a little: * Removes `os.remove("test.png")` (which seems to be unused). * Uses a temporary directory for some pipetool tests. Includes fixup: * spelling

* Enable `manpages_url`, so we can use manpage links. * Pipetools: Revise Sinks section to use `py:class` style documentation. This makes cross-references to specific sinks much easier. * Add documentation for `WiresharkSink`. * Pipetools: Revise Link object section with some word-smithing. * Usage/wireshark: * improve wordsmithing * remove hard coded references to `google.com` (which trigger DNS lookups), and use an RFC 5737 netmask instead. * set a source IP address (otherwise, this gets the host's IP address). * remove `Ether` layer, as Wireshark supports `DLT_RAW` (which triggers getting the host's MAC address). * elaborate on mixed `linktype` issues.

* `QueueSink.recv`: add `block` and `timeout` parameters * `WrpcapSink`, `WiresharkSink`: add `linktype` parameter * `WiresharkSink`: add `args` parameter * tests: sleep longer to work-around race conditions on Windows

* `{Raw,}PcapWriter._write_packet`: * Remove unused support for `packet` as tuple, as `write` will always unroll iterators for us (and do it better). * Always set the `usec` parameter if `sec` was unset. * Set `usec=0` if `sec` was set, but `usec` was unset (instead of using the current time's usec value * PEP-8 fixup, add docstring. * Only write the header if there is a packet * `PcapWriter._write_packet`: support packet as bytes * `RawPcapWriter.close`: write the file header here if not already written * `tcpdump()`: * Add `linktype` parameter, like `wrpcap(linktype=...)` * Add `wait` parameter, which controls whether a program should be run in the background. Defaults to `True` (run in foreground). * Throw an error if `prog` is not a string. * Copy `read_stdin_opts` (for thread safety). * `tdecode()`: Add `args` parameter (defaults to `-V`, as before), pass other `tcpdump()` kwargs. * `wireshark()`: Run in the background by default, pass other `tcpdump()` kwargs. * Add tests that hit `wireshark`, `tdecode`, `tcpdump` with new parameters, and try to pass packets as bytes.

micolous · 2019-04-11T02:00:56Z

@gpotter2: I squashed fixups into commits that make a bit more sense, and fixed the issues you mentioned.

I've updated the original change description with the extra changes since approval, please take a look!

gpotter2 · 2019-04-11T07:46:40Z

the winpcap=True version already fails on some specific tests. It's using a very old libpcap version

About linktypes, we could hack our way through with something similar to

>>> def hacky_linktype_name(value): 
...:     return next(k for k, v in six.iteritems(scapy.data.__dict__) if k[:3] == "DLT" and v == value)
>>>
>>> hacky_linktype_name(0)                              'DLT_NULL'

as a fallback, though that's a bit messy, it could be useful... what do you think ?

Otherwise, the doc improvement is great and PR looks good overall

scapy/utils.py

micolous marked this pull request as ready for review March 30, 2019 03:07

micolous force-pushed the wireshark-pipe branch from 9089554 to a0c4d29 Compare March 30, 2019 03:09

gpotter2 added the enhancement label Mar 30, 2019

gpotter2 reviewed Mar 31, 2019

View reviewed changes

scapy/utils.py Outdated Show resolved Hide resolved

micolous force-pushed the wireshark-pipe branch from a0c4d29 to 6634a22 Compare April 8, 2019 06:38

gpotter2 approved these changes Apr 8, 2019

View reviewed changes

micolous force-pushed the wireshark-pipe branch from 6eea710 to 348bf26 Compare April 10, 2019 05:07

micolous added 4 commits April 11, 2019 10:43

Pipetool: fixups:

390d96a

* `QueueSink.recv`: add `block` and `timeout` parameters * `WrpcapSink`, `WiresharkSink`: add `linktype` parameter * `WiresharkSink`: add `args` parameter * tests: sleep longer to work-around race conditions on Windows

micolous force-pushed the wireshark-pipe branch from 847e884 to 33c471c Compare April 11, 2019 01:21

gpotter2 reviewed Apr 11, 2019

View reviewed changes

scapy/utils.py Show resolved Hide resolved

gpotter2 merged commit 6dde6ae into secdev:master Apr 29, 2019

gpotter2 mentioned this pull request Apr 29, 2019

Workaround tcpdump lack of -y <integer> support #2005

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Adds Wireshark pipes support, cleanup OSX tcpdump handling. #1959

Adds Wireshark pipes support, cleanup OSX tcpdump handling. #1959

Uh oh!

micolous commented Mar 30, 2019 •

edited

Loading

Uh oh!

gpotter2 commented Mar 31, 2019

Uh oh!

Uh oh!

codecov bot commented Apr 8, 2019 •

edited

Loading

Uh oh!

gpotter2 left a comment

Uh oh!

micolous commented Apr 9, 2019 •

edited

Loading

Uh oh!

micolous commented Apr 10, 2019 •

edited

Loading

Uh oh!

micolous commented Apr 10, 2019

Uh oh!

micolous commented Apr 11, 2019 •

edited

Loading

Uh oh!

gpotter2 commented Apr 11, 2019 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Adds Wireshark pipes support, cleanup OSX tcpdump handling. #1959

Adds Wireshark pipes support, cleanup OSX tcpdump handling. #1959

Uh oh!

Conversation

micolous commented Mar 30, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gpotter2 commented Mar 31, 2019

Uh oh!

Uh oh!

codecov bot commented Apr 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gpotter2 left a comment

Choose a reason for hiding this comment

Uh oh!

micolous commented Apr 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

micolous commented Apr 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

micolous commented Apr 10, 2019

Uh oh!

micolous commented Apr 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gpotter2 commented Apr 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

micolous commented Mar 30, 2019 •

edited

Loading

codecov bot commented Apr 8, 2019 •

edited

Loading

micolous commented Apr 9, 2019 •

edited

Loading

micolous commented Apr 10, 2019 •

edited

Loading

micolous commented Apr 11, 2019 •

edited

Loading

gpotter2 commented Apr 11, 2019 •

edited

Loading