Use file content heuristics to decide file reader. #1962
Conversation
…sed on the magic number.
…ics detection method.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## dev #1962 +/- ##
==========================================
+ Coverage 83.45% 83.49% +0.04%
==========================================
Files 311 312 +1
Lines 54578 54848 +270
Branches 11808 11881 +73
==========================================
+ Hits 45546 45798 +252
- Misses 7832 7835 +3
- Partials 1200 1215 +15
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Dimi1010
added
the
API deprecation
Tests/Pcap++Test/Tests/FileTests.cpp
Outdated
| PTF_ASSERT_NOT_NULL(dynamic_cast<pcpp::PcapNgFileReaderDevice*>(genericReader)); | ||
| PTF_ASSERT_TRUE(genericReader->open()); | ||
| // ------- IFileReaderDevice::createReader() Factory | ||
| // TODO: Move to a separate unit test. |
There was a problem hiding this comment.
We should add the following to get more coverage:
- Open a snoop file
- Open a file that is not any of the options
- Open pcap files with different magic numbers
- Assuming we add a version check for snoop and pcap file: create temp files with bogus data that has the magic number but wrong versions
There was a problem hiding this comment.
3d713ab adds the following tests:
- Pcap, PcapNG, Zst file with correct content + extension
- Pcap, PcanNG file with correct content + wrong extension
- Bogus content file with correct extension (pcap, pcapng, zst)
- Bogus content file with wrong extension (txt)
Haven't found a snoop file to add. Do we have any?
Open pcap files with different magic numbers
Do you mean Pcap content that has just its magic number changed? Because IMO it is reasonable to consider that invalid format and fail as regular bogus data.
Assuming we add a version check for snoop and pcap file: create temp files with bogus data that has the magic number but wrong versions
Pending on #1962 (comment) .
Move it out if it needs to be reused somewhere.
Libpcap supports reading this format since 0.9.1. The heuristics detection will identify such magic number as pcap and leave final support decision to the pcap backend infrastructure.
|
@Dimi1010 some CI tests fail... |
Pcap++/header/PcapFileDevice.h
Outdated
| enum class CaptureFileFormat | ||
| { | ||
| Unknown, | ||
| Pcap, // regular pcap with microsecond precision | ||
| PcapNano, // regular pcap with nanosecond precision | ||
| PcapNG, // uncompressed pcapng | ||
| PcapNGZstd, // zstd compressed pcapng | ||
| Snoop, // solaris snoop | ||
| }; | ||
|
|
||
| /// @brief Heuristic file format detector that scans the magic number of the file format header. | ||
| class CaptureFileFormatDetector |
There was a problem hiding this comment.
If I'm not mistaken, this used to be in the .cpp file, right? Is the reason we moved it to the .h file is to make it easier to test?
If yes, I think we can test it using createReader() - create a temporary fake file with the data we want to test, and delete it when the test is done
There was a problem hiding this comment.
I tried that suggestion initially, but it would have been an extremely fragile unit test. The "pass" conditions would have been checked indirectly.
Also, createReader has multiple return paths for Nano / Zst file formats, which would have caused complications since the format test would have needed to care about the environment it runs at, which it doesn't have to as a standalone.
Any additional changes to createReader could also break the test, which they really shouldn't. For example, I am thinking of maybe adding additional logic for Zst archive to check if the compressed data is actually a pcapng, and not a random file. This would be a nightmare to make compatible with the "spoofed files" test due to assumptions on the test that createReader doesn't do anything more complicated than check the initial magic number.
So, in the end, you end up with a more compilcated unit test to read through that:
- depends on the environment it runs on.
- can be broken not just by changes to the format detector but also changes to the
createReaderfactory, too. - induces requirements on
createReaderas it uses its behavior to testdetectFormat.
There was a problem hiding this comment.
I understand it's better to test CaptureFileFormatDetector as a standalone class, but it requires exposing it in the .h file which is not great (even though it's in the internal namespace). Testing createReader is a bit more fragile, but I don't think the difference is that big. Of course, if we add logic to detect more file types or update the existing detection logic some tests might break, but we easily fix them as needed.
I usually try to avoid the internal namespace where possible because it's still in the .h file and is exposed to users, and we'd like to keep our API as clean as possible
There was a problem hiding this comment.
Testing createReader is a bit more fragile, but I don't think the difference is that big. Of course, if we add logic to detect more file types or update the existing detection logic some tests might break, but we easily fix them as needed.
It is a big difference and it's not always an easy fix. I plan to add the aforementioned Zst checks in another PR after this one, and that would make zst spoofing in createReader impossible, due to zst format automatically being checked for PcapNg or Unknown contents. Therefor you can't rely on the return of createReader to find out what the return of detectFormat was, because nullptr can be returned from several paths from detectFormat return value (Unknown, Nano + unsupported, Zst + unsupported). We have already had issues with tests being silently broken (#1977 comes to mind), so I would prefer to avoid fragile tests if we can.
I usually try to avoid the internal namespace where possible because it's still in the .h file and is exposed to users, and we'd like to keep our API as clean as possible
Fair, it is exposed, but the that is the entire reason of having the internal namespace. It is a common convention that external users shouldn't really touch it. If you want to keep the primary public header files clean there are a couple options:
- I have seen many libraries have a subfolder
internal/detailin their public include folder, where they keep all their internal code headers that need to be exposed. That keeps the "internal" code separate from the "public" code, if users want to read through the headers. This is a common convention used in Boost libraries. "public" headers that depend on internal headers include them from theinternalsubfolder. - In the current case, we have another option. Since the
CaptureFileFormatDetectoris only needed in thecpppart and not in the header part, we can extract it to a fully internal header, kept with the source files. This would prevent it from being exposed in the public API, but the Test project can be manually set to search for headers from "Pcap++/src" too, to allow it to link in the tests.
There was a problem hiding this comment.
It is a big difference and it's not always an easy fix. I plan to add the aforementioned Zst checks in another PR after this one, and that would make zst spoofing in
createReaderimpossible, due to zst format automatically being checked for PcapNg or Unknown contents. Therefor you can't rely on the return ofcreateReaderto find out what the return ofdetectFormatwas, becausenullptrcan be returned from several paths fromdetectFormatreturn value (Unknown, Nano + unsupported, Zst + unsupported). We have already had issues with tests being silently broken (#1977 comes to mind), so I would prefer to avoid fragile tests if we can.
I'm not sure I understand... if we create fake files we know which type to expect, so all the test needs to do is verify the created file device is of the expected type 🤔
- In the current case, we have another option. Since the
CaptureFileFormatDetectoris only needed in thecpppart and not in the header part, we can extract it to a fully internal header, kept with the source files. This would prevent it from being exposed in the public API, but the Test project can be manually set to search for headers from "Pcap++/src" too, to allow it to link in the tests.
I guess we can do that, but I still don't understand why we can't test it with createReader or tryCreateReader
There was a problem hiding this comment.
Curious why wouldn't you want that?
Extracting the archived file just to verify the format is wasteful and might take a long time, especially if it's called on large files. Also - for these large files it'd mean the file is extracted twice - once in createReader and again when actually reading the file
Yes, but I am unsure if it gives a precise error message of what went wrong or just a generic failure error.
If this is indeed the case, maybe we need to fix the LightPcapNg code?
Which is as simple as calling
open()inside the factory function, no? As you said, the backend already does validation, so why not reuse it for the factory validation?
Not necessarily - as far as I know open() checks mostly the header and doesn't go over the rest of the file, so a user can open a file with a correct header but corrupted data and reading the file will fail
There was a problem hiding this comment.
Extracting the archived file just to verify the format is wasteful and might take a long time, especially if it's called on large files.
There is no need to extract the entire file. ZST compression works on independent frames allowing frame-by-frame (streaming) decompression. We only need to decompress the first frame to read the magic number to validate that the archive contents appear to be PcapNG. How large the total file is is irrelevant.
Incidentally, frame-by-frame is also how LightPcapNG reads the ZST archive. It decompresses a frame, reads the fully decompressed PcapNG records in it, and decompresses the next frame if needed.
If this is indeed the case, maybe we need to fix the LightPcapNg code?
Which is C code, and makes it much harder to output a readable error. Not to mention we need to deal with passing that error up the stack.
Not necessarily - as far as I know open() checks mostly the header and doesn't go over the rest of the file, so a user can open a file with a correct header but corrupted data and reading the file will fail
But it will still have passed open(). My idea isn't that createReader should validate that everything is correct. It is that it should validate just enough to guarantee that the returned device can successfully pass an open() call. The device might even be returned already opened and ready for reading, reducing the user side boilerplate.
There is no reason to return a device that can't even be opened, since the user can't do anything with it. It just adds more boilerplate as the user has to do the error handling twice.
If the records afterwards are corrupted at some point, the read should fail when the corrupted data is reached.
There was a problem hiding this comment.
There is no need to extract the entire file. ZST compression works on independent frames allowing frame-by-frame (streaming) decompression. We only need to decompress the first frame to read the magic number to validate that the archive contents appear to be PcapNG. How large the total file is is irrelevant.
Incidentally, frame-by-frame is also how LightPcapNG reads the ZST archive. It decompresses a frame, reads the fully decompressed PcapNG records in it, and decompresses the next frame if needed.
I asked ChatGPT, and there's indeed a way to read just the header using ZSTD_getFrameHeader - is this what you're referring to?
Which is C code, and makes it much harder to output a readable error. Not to mention we need to deal with passing that error up the stack.
Still, I'd be in favor of fixing it in LightPcapNg. In any case - PcapPlusPlus doesn't call Zstd API directly, everything goes through LightPcapNg so I think the right way would be to fix or add it in LightPcapNg
But it will still have passed
open(). My idea isn't thatcreateReadershould validate that everything is correct. It is that it should validate just enough to guarantee that the returned device can successfully pass anopen()call. The device might even be returned already opened and ready for reading, reducing the user side boilerplate.There is no reason to return a device that can't even be opened, since the user can't do anything with it. It just adds more boilerplate as the user has to do the error handling twice.
If the records afterwards are corrupted at some point, the read should fail when the corrupted data is reached.
This is one way to look at it, another approach could say - createReader should do its best to guess the right reader, and whether it can be opened or not is a question for open(). I'm not sure which approach is better so I would vote for the simplest option 🙂
There was a problem hiding this comment.
I asked ChatGPT, and there's indeed a way to read just the header using ZSTD_getFrameHeader - is this what you're referring to?
Not really, but similar. From what I understand, the ZST archive has the following format:
[ZST Header] [CBlock (Compressed Block) 0] [CBlock 1] [CBlock 2] ... [CBlock N]
Those blocks need to be decompressed sequentially starting from CBlock 0. (I was wrong earlier when I mentioned they were completely independent). The thing is, the magic number of the inner file is at the start of CBlock 0, so we don't need to decompress any of the other blocks. ZSTD provides a streaming API. We can be use that to only decompress CBlock 0 to fetch the magic number of the compressed content and stop.
Still, I'd be in favor of fixing it in LightPcapNg. In any case - PcapPlusPlus doesn't call Zstd API directly, everything goes through LightPcapNg so I think the right way would be to fix or add it in LightPcapNg
It is possible to add the check to light_pcapng. It would need to do a check around the time it tries to read the header block. My question is how to communicate that error condition to the pcpp layer? From what I can see in light_pcapng_open_read, if it fails to open the error message is outputted as fprint(...) assertion messages and the return value is nullptr. I suppose we can do something similar, but then the pcpp layer gets a generic pass / fail, and not really a reason why it failed.
This is one way to look at it, another approach could say - createReader should do its best to guess the right reader, and whether it can be opened or not is a question for open(). I'm not sure which approach is better so I would vote for the simplest option 🙂
Both approaches are valid. I think the main decision point are the two questions:
- In what situation do you want to make a device and not open it? I think that would be a pretty rare occasion.
- What can you do with a device that can't open? I think the answer here is nothing, except to destroy it, which raises the question of why return such a device in the first place?
I updated the createReader procedure in ae9caa8 to include a openDevice parameter. I think that would do for a factory procedure for now? Users would have the option of keeping the old behaviour and manually opening the device later, but for the general case it is handled for them and they can immediately begin reading from it for ease of use.
What do you think?
There was a problem hiding this comment.
It is possible to add the check to light_pcapng. It would need to do a check around the time it tries to read the header block. My question is how to communicate that error condition to the pcpp layer? From what I can see in
light_pcapng_open_read, if it fails to open the error message is outputted asfprint(...)assertion messages and the return value isnullptr. I suppose we can do something similar, but then the pcpp layer gets a generic pass / fail, and not really a reason why it failed.
This is a valid question, but maybe PcapPlusPlus doesn't need to know the exact reason - it only needs to know that it failed - fprint will print the reason like GetLastError used in many libraries
Both approaches are valid. I think the main decision point are the two questions:
* In what situation do you want to make a device and not open it? I think that would be a pretty rare occasion. * What can you do with a device that can't open? I think the answer here is nothing, except to destroy it, which raises the question of why return such a device in the first place?I updated the
createReaderprocedure in ae9caa8 to include aopenDeviceparameter. I think that would do for a factory procedure for now? Users would have the option of keeping the old behaviour and manually opening the device later, but for the general case it is handled for them and they can immediately begin reading from it for ease of use.What do you think?
This is generally true with out API - usually when creating a device the next thing to do is opening it. I originall did it to avoid throwing exceptions or handling validations in the c'tor. I'd consider createReader as a "smart constructor" that decides on the object to return, but the rest of the API should stay consistent -> the next action would be calling open()
| } | ||
| }; | ||
|
|
||
| PTF_TEST_CASE(TestFileFormatDetector) |
There was a problem hiding this comment.
Please see my previous comment. Maybe we can create a temp fake file with the expected data and run createReader()
Tests/Pcap++Test/PcapExamples/file_heuristics/pcapng-with-pcap-ext.pcapng.pcap
Outdated
Show resolved
Hide resolved
…in `createReader` instead of having the Format detector assume that is what is intended.
…t detector from libpcap behaviour.
…AII initialization.
3 participants
The PR adds heuristics based on the file content that is more robust than deciding based on the file extension.
The new decision model scans the start of the file for its magic number signature. It then compares it to the signatures of supported file types [1] and constructs a reader instance based on the result.
A new function
createReaderandtryCreateReaderhas been added due to changes in the public API of the factory.The functions differ in the error handling scheme, as
createReaderthrows andtryCreateReaderreturnsnullptron error.Method behaviour changes during erroneous scenarios:
getReadercreateReadertryCreateReadernullptrPcapFileDeviceReadernullptr