Struct KafkaSink 

pub struct KafkaSink { /* private fields */ }

Kafka sink connector that writes Arrow RecordBatch data to Kafka topics.

Operates in Ring 1 (background) receiving data from Ring 0 via the subscription API.

Lifecycle

1. Create with KafkaSink::new
2. Call open() to create the producer and connect to Kafka
3. write_batch() serializes and produces records; checkpoint flushes provide durable at-least-once delivery
4. Call close() for clean shutdown

This connector deliberately rejects exactly-once admission because it does not expose a coordinated external checkpoint namespace/cursor.

Implementations

impl KafkaSink

pub fn new( schema: SchemaRef, config: KafkaSinkConfig, registry: Option<&Registry>, ) -> Self

Creates a new Kafka sink connector with explicit schema.

Panics

Panics if config.format is not a supported serialization format. Call KafkaSinkConfig::validate first to catch this at config time.

pub fn with_schema_registry( schema: SchemaRef, config: KafkaSinkConfig, sr_client: SchemaRegistryClient, ) -> Self

Creates a new Kafka sink with Schema Registry integration.

Panics

Panics if config.format is not a supported serialization format. Call KafkaSinkConfig::validate first to catch this at config time.

pub fn state(&self) -> ConnectorState

Lifecycle state (Created → Running → Closed).

pub fn has_schema_registry(&self) -> bool

Whether Avro schema registration is available.

Trait Implementations

impl Debug for KafkaSink

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Drop for KafkaSink

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl SinkConnector for KafkaSink

fn terminal_task_tracker(&self) -> Option<ConnectorTaskTracker>

Observe detached tasks whose lifetime may outlast this connector value.

fn contract( &self, config: &ConnectorConfig, ) -> Result<SinkContract, ConnectorError>

Declare durability, placement, and input semantics for this exact configuration without opening files, sockets, clients, or transactions.

fn open<'life0, 'life1, 'async_trait>( &'life0 mut self, config: &'life1 ConnectorConfig, ) -> Pin<Box<dyn Future<Output = Result<(), ConnectorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Open the connection and prepare to accept writes.

fn write_batch<'life0, 'life1, 'async_trait>( &'life0 mut self, batch: &'life1 RecordBatch, ) -> Pin<Box<dyn Future<Output = Result<WriteResult, ConnectorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Implementations using ConnectorCancellationPolicy::CancelSafe must remain valid when this future is dropped at a deadline. For ConnectorCancellationPolicy::RetireConnector, cancellation makes the complete connector instance terminal before later work can be processed.

fn schema(&self) -> SchemaRef

Expected Arrow schema of input batches.

fn suggested_write_timeout(&self) -> Duration

Default per-call write_batch I/O timeout. Users can override this via the sink.write.timeout.ms connector property.

fn flush<'life0, 'async_trait>( &'life0 mut self, ) -> Pin<Box<dyn Future<Output = Result<(), ConnectorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Must be internally bounded — the sink task’s periodic timer calls this on every tick. Thorough drains belong in pre_commit / coordinated commit / close, not here.

fn close<'life0, 'async_trait>( &'life0 mut self, ) -> Pin<Box<dyn Future<Output = Result<(), ConnectorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Close the sink and release resources.

fn cancellation_policy(&self) -> ConnectorCancellationPolicy

Deadline behavior required by the underlying client implementation.

fn begin_epoch<'life0, 'async_trait>( &'life0 mut self, _epoch: u64, ) -> Pin<Box<dyn Future<Output = Result<(), ConnectorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Begin checkpoint-owned staging. Called only for an admitted checkpoint-committable contract; weaker sinks use the no-op default.

fn pre_commit<'life0, 'async_trait>( &'life0 mut self, _epoch: u64, ) -> Pin<Box<dyn Future<Output = Result<Option<Vec<u8>>, ConnectorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Flush + prepare, but do not finalize externally. The runtime persists the checkpoint decision before a designated committer finalizes the collected descriptors; on failure it calls rollback_epoch.

fn rollback_epoch<'life0, 'async_trait>( &'life0 mut self, _epoch: u64, ) -> Pin<Box<dyn Future<Output = Result<(), ConnectorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Must be idempotent. The runtime calls this on every checkpoint-committable sink after proving a pre-decision failure, including sinks that never completed pre_commit.

fn flush_interval(&self) -> Duration

Maximum residence time for a non-empty sink buffer before the runtime invokes flush. Checkpoint-committable sinks ignore the periodic timer and flush only through their checkpoint protocol.

fn as_coordinated_committer(&self) -> Option<&dyn CoordinatedCommitter>

Leader-side committer for a checkpoint-committable contract; None for every weaker contract.