Struct CheckpointBatcher 

pub struct CheckpointBatcher { /* private fields */ }

Accumulates state blobs and flushes them as a single compressed object.

Call add for each partition/operator state blob during a checkpoint cycle, then flush at the end. should_flush returns true when the buffer exceeds the configured threshold.

Implementations

impl CheckpointBatcher

pub fn new( store: Arc<dyn ObjectStore>, prefix: String, flush_threshold: Option<usize>, ) -> Result<Self>

Create a new batcher.

prefix is prepended to all object paths (e.g., "nodes/abc123/"). flush_threshold is the uncompressed buffer size that triggers a flush (default: 8 MB).

Errors

Returns std::io::Error if the internal Tokio runtime cannot be created.

pub fn add(&mut self, key: String, data: Vec<u8>)

Add a state blob to the buffer.

The key identifies the partition/operator (e.g., "partition-0/agg"). Call should_flush after adding to check whether the buffer exceeds the threshold.

pub fn should_flush(&self) -> bool

Returns true if the buffer exceeds the flush threshold.

pub fn len(&self) -> usize

Returns the number of entries currently buffered.

pub fn is_empty(&self) -> bool

Returns true if the buffer is empty.

pub fn buffer_size(&self) -> usize

Returns the current uncompressed buffer size in bytes.

pub fn metrics(&self) -> &Arc<BatchMetrics>

Shared metrics handle.

pub fn flush(&mut self, epoch: u64) -> Result<(), CheckpointStoreError>

Flush buffered entries as a single LZ4-compressed object.

The object is written to {prefix}checkpoints/batch-{epoch:06}.lz4. After a successful flush the buffer is cleared.

Does nothing if the buffer is empty.

Errors

Returns CheckpointStoreError on serialization or object store failure.