In the Filecoin sealing process, two types of files are generated:
- Unsealed Files: Decrypted data files used for regular retrieval.
- CAR Original Files (Content Addressable aRchive): Original content files generated during the data import phase, containing data blocks and their corresponding CID structures.
Currently, the booster-http retrieval logic mainly targets Unsealed files. During retrieval, these files must be converted into CAR files before being returned to the client. This process results in unnecessary resource and storage overhead.
To improve resource utilization and retrieval efficiency, we have extended support for direct retrieval from CAR original files.
With this enhancement, the system can directly read and return CAR files without needing to retain Unsealed files, achieving the following benefits:
- Reduced resource consumption: Eliminates redundant file conversion and I/O operations.
- Storage optimization: Users only need to retain CAR original files, removing the need to store Unsealed copies.
- Improved retrieval performance: CAR files can be directly read and transferred, reducing intermediate processing delays.
-
New parameter:
car-urls- Specifies the list of remote CAR file server URLs.
- Supports multiple URLs separated by semicolons (
;).
-
Example:
booster-http run --car-urls=http://car-server-1;http://car-server-2Added new logic in the tryReadUnsealedPiece method:
- First, attempt to read from a local Unsealed file.
- If the Unsealed file is not found, attempt to read from a remote CAR file using the
car-urlsparameter.
This logic ensures backward compatibility with existing workflows while enabling fast retrieval from the new CAR file path.
- Added a remote CAR file reading service.
- Supports chunked reading and data streaming to reduce network latency.
- Optimized data transmission reliability and error recovery to ensure stable large file reads.