Cloud authentication — Rust, Python, and Java

This document explains where credentials live when rust-data-processing or rdp_jvm_sys reads and writes s3://, gs://, abfss://, and related URIs. For per-connector URLs and copy-paste examples, see CONNECTORS.md.

Fake values below are placeholders only.

Open this guide as the file docs/CLOUD_AUTH.md — it is a single markdown file, not a folder. If the editor says “is a directory”, you clicked a broken #fragment link. Use dedicated files: AMAZON_S3.md · AZURE_ADLS.md · SNOWFLAKE.md.

Core rule

Rust performs cloud I/O. Python and Java are thin wrappers: they pass URIs and pipeline JSON across FFI; they do not pass access tokens, Azure AD secrets, or AWS keys in that JSON.

Credentials are resolved by the object_store crate inside the process that loaded the native library (rdp_jvm_sys .so / .dylib, Python extension, or Rust binary). Those credentials come from the operating-system environment of that process — not from Java APIs, not from System.getenv configuration inside your application code unless your launcher actually exported vars into the process first.

Java / Python                    Rust (same process)
─────────────                    ─────────────────
pipeline JSON  ──FFI──►  parse URI → object_store::parse_url_opts
  (location only)              │
                               ▼
                         credential chain from env / MSI / IAM / keys
                               │
                               ▼
                         GET/PUT to s3:// / abfss:// / gs://

Implementation entry points:

• src/ingestion/object_store.rs — ingest_from_object_store_uri, export_dataset_to_object_store_uri
• src/ingestion/delta_lake.rs — delta_table_uri, write_dataset_to_delta_table (Parquet under the table path)
• bindings/jvm-sys/src/pipeline_run.rs — sources.object_store_uris, sinks databricks, object_store, snowflake stage URIs

parse_url_opts is called with no credential map from callers — only the URL string.

System environment variables (not Java-specific)

Names like AWS_ACCESS_KEY_ID, AZURE_CLIENT_SECRET, and GOOGLE_APPLICATION_CREDENTIALS are standard OS / process environment variables. They are not a special “Java environment” or JVM system property namespace.

Who must see the variables: the single OS process that loads rdp_jvm_sys (e.g. java …, python …, or a Rust binary). When Java calls native code, Rust runs in the same process as the JVM — so Docker/K8s env injection for the container (or pod) is what matters.

Local shell (development)

export AWS_ACCESS_KEY_ID="..."
export AWS_SECRET_ACCESS_KEY="..."
java -jar your-etl.jar   # JVM inherits the shell’s environment

Docker

Inject at container level — not inside Java source:

# Prefer runtime injection (secrets manager, --env-file) over baking secrets into the image
ENV AWS_REGION=us-east-1
# Do NOT commit real keys in Dockerfile layers

docker run --env-file /secure/rdp.env your-image:tag
# or
docker run \
  -e AWS_ACCESS_KEY_ID=... \
  -e AWS_SECRET_ACCESS_KEY=... \
  -e AZURE_TENANT_ID=... \
  -e AZURE_CLIENT_ID=... \
  -e AZURE_CLIENT_SECRET=... \
  your-image:tag

Use a .env file only on the host or in CI to populate docker run --env-file; keep .env out of git (.gitignore). For production, prefer a secret store that mounts or injects env at deploy time.

Kubernetes

Map secrets to pod environment variables (or use workload identity so fewer static keys are needed):

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: etl
      image: your-registry/rdp-etl:latest
      envFrom:
        - secretRef:
            name: rdp-cloud-credentials   # keys: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, …
      # Optional single vars:
      env:
        - name: AZURE_STORAGE_ACCOUNT_NAME
          value: "storacc01"
        - name: GOOGLE_APPLICATION_CREDENTIALS
          value: "/var/secrets/gcp/sa.json"
      volumeMounts:
        - name: gcp-sa
          mountPath: /var/secrets/gcp
          readOnly: true
  volumes:
    - name: gcp-sa
      secret:
        secretName: gcp-service-account

Azure / AWS on K8s: often you omit static keys and bind a ServiceAccount to IAM (EKS) or use Workload Identity (AKS) so the pod gets credentials without AWS_* / AZURE_CLIENT_SECRET in a Secret — still platform env/metadata, not Java config.

Python and Rust binaries

Same rule: set env on the process (export in shell, docker run -e, K8s env, systemd Environment=, etc.). maturin run / cargo run inherit the parent shell unless you inject vars in the job definition.

Quick reference (implemented today)

Two different “auth” stories (do not mix them up)

Amazon S3

Full guide (AWS env vars, IAM role, Docker, K8s, Java/Rust/Python): AMAZON_S3.md.

Google Cloud Storage

URI: gs://demo-gcs-project/rdp/incoming/part-00000.parquet (validation also accepts gcs://).

Rust / Python / Java

• Rust: export GOOGLE_APPLICATION_CREDENTIALS in the shell, then call ingest_from_object_store_uri / export_dataset_to_object_store_uri.
• Python: same env on the notebook or maturin process.
• Java: only the URI in object_store_uris or sink JSON; set GOOGLE_APPLICATION_CREDENTIALS on the pod/container/process (Docker --env-file, K8s env, etc.) — not via Java-only config files alone.

"object_store_uris": ["gs://demo-gcs-project/rdp/incoming/part-00000.parquet"]

Azure ADLS Gen2

Full guide (env vars, Java/Rust/Python, Databricks abfss:// warehouse): AZURE_ADLS.md.

Databricks pipeline sink (kind: databricks)

Java (and Rust/Python via the same layout) often include:

{
  "kind": "databricks",
  "workspace_url": "https://dbc-a1b2c3d4-e5f6.cloud.databricks.com",
  "catalog_uri": "https://dbc-a1b2c3d4-e5f6.cloud.databricks.com/api/2.1/unity-catalog/iceberg",
  "warehouse": "abfss://datalake@storacc01.dfs.core.windows.net/unity/",
  "namespace": "main.curated",
  "table": "fact_scores"
}

Auth you must configure: for abfss:// warehouse → AZURE_ADLS.md; for s3:// warehouse → AMAZON_S3.md.

A Databricks PAT (dapi…) or workspace OAuth app does not authenticate the in-tree write. Those are for Databricks REST, SQL warehouses, and Spark drivers you run separately.

Full Delta transaction logs (ACID, time travel) are not committed yet; see delta_lake.rs.

Snowflake

Full guide (stage AWS_*, optional SNOWFLAKE_*, Docker/K8s, Java/Rust/Python): SNOWFLAKE.md.

Apache Spark handoff

Rust writes Parquet to handoff_uri (s3://, abfss://, or file://).

See CONNECTORS.md — Apache Spark.

SFTP

Status: Implemented in rust-data-processing and rdp_jvm_sys when built with cloud_connectors (Cargo feature file_transfer).

URL shape: sftp://etl_user:FAKE_SFTP_PASS@sftp.example.com:22/rdp/incoming/data.parquet

Pipeline JSON — declare the URI only (no secrets in JSON):

"file_transfer_uris": ["sftp://etl_user@sftp.example.com:22/rdp/incoming/data.parquet"]

Rust downloads the remote file to a temp path, then uses the same CSV/JSON/Parquet/XML readers as local ingest. Set sources.options.format when the extension is ambiguous.

Fallback: land files on S3/ADLS/GCS/local with your own SFTP client, then use object_store_uris or sources.paths.

FTP / FTPS

Status: ftp:// and ftps:// via the same file_transfer module (cloud_connectors feature).

URL: ftp://etl_user:FAKE_FTP_PASS@ftp.example.com:21/rdp/incoming/data.parquet

"file_transfer_uris": ["ftp://etl_user@ftp.example.com:21/rdp/incoming/data.parquet"]

Fallback: same as SFTP — object store or local paths after external sync.

What is never in pipeline JSON

Mental model (all clouds)

Bottom line: Rust obtains storage tokens without Java or Python handing them over, as long as the native library’s process is configured correctly. Databricks workspace OAuth/PAT is a separate concern until REST/catalog integration is added.

Build features

DB read (sources.db_reads) is separate: db_connectorx on JVM — see CONNECTORS.md.

Related docs

• AMAZON_S3.md — Amazon S3 auth (dedicated file)
• AZURE_ADLS.md — Azure ADLS / Blob auth (dedicated file)
• SNOWFLAKE.md — Snowflake stage + optional COPY (dedicated file)
• CONNECTORS.md — shared URLs and language snippets
• java/EXAMPLES.md — JVM pipeline examples
• adr/006-jvm-orchestration-pipeline-json.md — pipeline envelope and source kinds