Expand description
Kafka streaming ELT (Extract → Load → Transform).
Kafka is not a file connector. RDP treats it like stream frameworks (Flink, Kafka Streams):
- Extract — poll a bounded window from a topic (
poll_kafka_window) or accept records from your host consumer (elt_load_kafka_records). - Load — land raw/semi-structured rows to storage (Parquet, Postgres
COPY, object store) with offsets preserved — no heavy transform in the hot path. - Transform — run Polars SQL / pipeline JSON on landed data in a separate job or stage.
Poll window (Vec<KafkaStreamRecord>) is backpressure / checkpoint sizing — not batch ETL.
Enable native I/O with --features kafka. See docs/KAFKA_ELT.md.