Skip to main content

ingest_from_ordered_paths

rust_data_processing::ingestion::unified

Function ingest_from_ordered_paths 

Source 
pub fn ingest_from_ordered_paths<P: AsRef<Path>>(
    paths: &[P],
    schema: &Schema,
    options: &IngestionOptions,
) -> IngestionResult<(DataSet, OrderedBatchIngestMetadata)>
Expand description

Ingest many files in order, concatenate rows into one DataSet, then apply the watermark filter once (if configured).

Semantics: the high-water filter applies to the combined rows, not per file. For each path, this calls ingest_from_path with watermark options temporarily cleared so observers see full per-file loads; the batch watermark runs after concatenation.