Reduce & aggregate semantics

This document describes behavior shared by processing::reduce, processing::multi, and Polars-backed pipeline::DataFrame helpers (reduce, group_by, feature_wise_mean_std).

Null handling

• Numeric aggregates (sum, min, max, mean, variance, std dev, sum of squares, L2 norm): nulls are ignored; only non-null values participate.
• ReduceOp::Count: counts rows in the dataset (includes nulls in that column).
• CountNotNull / non-null counts: count only non-null cells.
• CountDistinctNonNull: distinct values among non-null cells only (null is not a distinct category).

All-null or empty inputs

• Mean, variance, sample std dev, sum of squares, L2 norm: if there is no participating non-null numeric value, the result is Value::Null (not 0).
• Population variance / population std dev with a single non-null value: variance is 0, std dev is 0.
• Sample variance / sample std dev with fewer than two non-null values: Value::Null (undefined).
• Sum / min / max with no non-null values: Value::Null.
• Polars group_by note: some engines return 0 for sum over an all-null group; mean / std in Polars typically stay null for all-null groups. Prefer mean / std when you need “no data” vs “zero total”.

Float rounding and parity

• In-memory stats use f64 (Welford for variance). Very large Int64 values converted to f64 can differ slightly from Polars at the ULP level; integration tests allow a small relative tolerance in those cases.
• Reports (profiling, validation markdown/json) should format floats deterministically at the presentation layer if you need stable diffs.

Casting (CastMode)

• Pipeline cast / cast_with_mode control strict vs lossy casts into our logical types; feature_wise_mean_std and scalar DataFrame::reduce use strict cast to Polars Float64 for numeric stats, consistent with explicit CastMode::Strict behavior for those expressions.

Group-by (ML-oriented)

pipeline::Agg supports per-group:

• CountRows, CountNotNull, Sum, Min, Max, Mean, Median, Variance, StdDev, SumSquares, L2Norm, CountDistinctNonNull.

Median

• ReduceOp::Median (in-memory) and Agg::Median (group-by): nulls ignored; result is always Value::Float64. Even element count: average of the two middle order-statistics (aligned with typical SQL / Polars median behavior).

Combine multiple Agg variants in one DataFrame::group_by call for feature summaries keyed by categorical columns.

Multi-column helpers

• feature_wise_mean_std: one scan over rows; all listed columns must be Int64 or Float64.
• arg_max_row / arg_min_row: first row index on ties.
• top_k_by_frequency: non-null value counts, sorted by count desc then value key (stable tie-break).

Examples in this repo

Copy-pastable Rust snippets (filter/map/reduce, mean/variance/std, DataFrame::reduce, feature_wise_mean_std, arg max/min, top‑k, group_by with Agg) live in:

• API.md — section Processing pipelines (Epic 1 / Story 1.2)
• README.md — Processing pipelines, Cookbook → group-by, and the ML-oriented subsections under processing