Skip to main content

Module delta

Module delta 

Source
Expand description

Delta Lake sink connector implementation.

DeltaLakeSink implements SinkConnector, writing Arrow RecordBatch data to Delta Lake tables with ACID transactions and at-least-once delivery (exactly-once opt-in via delivery.guarantee = 'exactly-once').

§Write Strategies

  • Append mode: Arrow-to-Parquet zero-copy writes for immutable streams
  • Overwrite mode: Replace partition contents for recomputation
  • Upsert mode: CDC MERGE via Z-set changelog integration

Exactly-once semantics use epoch-to-Delta-version mapping: each LaminarDB epoch maps to exactly one Delta Lake transaction via txn application transaction metadata in the Delta log.

§Ring Architecture

  • Ring 0: No sink code. Data arrives via SPSC channel (~5ns push).
  • Ring 1: Batch buffering, Parquet writes, Delta log commits.
  • Ring 2: Schema management, configuration, health checks.

Structs§

DeltaLakeSink
Delta Lake sink connector.