Function paths_from_directory_scan 

pub fn paths_from_directory_scan(
    root: impl AsRef<Path>,
    relative_pattern: Option<&str>,
) -> IngestionResult<Vec<PathBuf>>

Recursively list files under root, optionally filtered by a glob on the path relative to root, then sort for deterministic ordering.

Intended for incremental directory batches: pair with ingest_from_ordered_paths (or your own ordering) when you need the same file sequence across machines and runs.

• root must exist and be a directory.
• If relative_pattern is None, every regular file under root is included.
• If Some, it is a glob::Pattern matched against each file path relative to root (use forward slashes in the pattern string for portability, e.g. **/*.csv).