-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Currently, we can only handle table input on a file basis. A more flexible and extensible abstraction is desirable.
Describe the solution you'd like
pub struct PartitionedFile {
/// Path for the file (e.g. URL, filesystem path, etc)
pub file_path: String,
/// Statistics of the file
pub statistics: Statistics,
// Several feasible extensions
// 1. partitioned table
/// Values of partition columns to be appended to each row
pub partition_value: Option<Vec<ScalarValue>>,
// 2. sub-file processing
/// A range of file to process in the current task
pub start: u64,
pub end: u64,
}Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request