Skip to main content

Module unified

rust_data_processing::ingestion

Module unified

Expand description

Unified ingestion entrypoint.

Most callers should use ingest_from_path, which ingests a file into an in-memory crate::types::DataSet using a provided crate::types::Schema.

If IngestionOptions::format is None, the ingestion format is inferred from the file extension.
If an super::observability::IngestionObserver is provided, success/failure/alerts are reported to it.

Structs§

IngestionOptions: Options controlling unified ingestion behavior.
IngestionRequest: Convenience helper for callers that want an owned request object.
OrderedBatchIngestMetadata: Metadata from ingest_from_ordered_paths: which paths were read, the last path in that sequence, and (when incremental watermark options are set) the maximum value in the watermark column after the batch filter — useful for checkpointing the next run.

Enums§

ExcelSheetSelection: How to choose sheet(s) when ingesting an Excel workbook.
IngestionFormat: Supported ingestion formats.

Functions§

infer_schema_from_path: Infer a Schema for an input file.
ingest_from_ordered_paths: Ingest many files in order, concatenate rows into one DataSet, then apply the watermark filter once (if configured).
ingest_from_path: Unified ingestion entry point for path-based sources.
ingest_from_path_infer: Convenience wrapper: infer schema and then ingest.