Struct IcebergSink 

pub struct IcebergSink { /* private fields */ }

Apache Iceberg sink connector.

Buffers RecordBatch data during each checkpoint epoch and commits to an Iceberg table atomically when the epoch commits. Each epoch produces at most one Iceberg transaction.

Implementations

impl IcebergSink

pub fn new(config: IcebergSinkConfig, _registry: Option<&Registry>) -> Self

Creates a new Iceberg sink with the given configuration.

Trait Implementations

impl SinkConnector for IcebergSink

fn open<'life0, 'life1, 'async_trait>( &'life0 mut self, config: &'life1 ConnectorConfig, ) -> Pin<Box<dyn Future<Output = Result<(), ConnectorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Open the connection and prepare to accept writes.

fn write_batch<'life0, 'life1, 'async_trait>( &'life0 mut self, batch: &'life1 RecordBatch, ) -> Pin<Box<dyn Future<Output = Result<WriteResult, ConnectorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Must be cancellation-safe: the runtime wraps this in tokio::time::timeout. Don’t split a &mut self mutation across an .await. In-flight transactional state may remain open after cancellation; the caller will rollback_epoch it.

fn schema(&self) -> SchemaRef

Expected Arrow schema of input batches.

fn begin_epoch<'life0, 'async_trait>( &'life0 mut self, epoch: u64, ) -> Pin<Box<dyn Future<Output = Result<(), ConnectorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Default: no-op (at-least-once semantics).

fn pre_commit<'life0, 'async_trait>( &'life0 mut self, _epoch: u64, ) -> Pin<Box<dyn Future<Output = Result<(), ConnectorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Phase 1 of 2PC: flush + prepare, do NOT finalize the txn. The runtime persists the manifest between pre_commit and commit_epoch; on failure it calls rollback_epoch. Default delegates to flush().

fn commit_epoch<'life0, 'async_trait>( &'life0 mut self, epoch: u64, ) -> Pin<Box<dyn Future<Output = Result<(), ConnectorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Phase 2 of 2PC: finalize the txn. Called after the manifest is durable. Default: no-op (at-least-once semantics).

fn rollback_epoch<'life0, 'async_trait>( &'life0 mut self, epoch: u64, ) -> Pin<Box<dyn Future<Output = Result<(), ConnectorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Must be idempotent: the runtime calls this on every exactly-once sink after a pre_commit failure, including sinks that never pre_commited.

fn health_check(&self) -> HealthStatus

Defaults to Unknown; connectors should override.

fn metrics(&self) -> ConnectorMetrics

Current sink metrics snapshot.

fn capabilities(&self) -> SinkConnectorCapabilities

Required (no default) so every implementation declares suggested_write_timeout.

fn close<'life0, 'async_trait>( &'life0 mut self, ) -> Pin<Box<dyn Future<Output = Result<(), ConnectorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Close the sink and release resources.

fn flush<'life0, 'async_trait>( &'life0 mut self, ) -> Pin<Box<dyn Future<Output = Result<(), ConnectorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Must be internally bounded — the sink task’s periodic timer calls this on every tick. Thorough drains belong in pre_commit / commit_epoch / close, not here.

fn as_schema_registry_aware(&self) -> Option<&dyn SchemaRegistryAware>

Return a SchemaRegistryAware view, if the sink speaks a schema registry protocol.