Skip to main content

Module watermarks

Module watermarks 

Source
Expand description

Kafka watermark integration.

Integrates laminar-core per-partition watermarks and watermark alignment groups with the Kafka source connector.

§Per-Partition Watermarks

Tracks watermarks at Kafka partition granularity, allowing progress even when some partitions are slow or idle:

use laminar_connectors::kafka::watermarks::KafkaWatermarkTracker;
use std::time::Duration;

let mut tracker = KafkaWatermarkTracker::new(0, Duration::from_secs(30));
tracker.register_partitions(4);

// Update individual partitions
tracker.update_partition(0, 5000);
tracker.update_partition(1, 3000); // slow partition

// Combined watermark is minimum (3000)
assert_eq!(tracker.current_watermark(), Some(3000));

// Mark slow partition as idle
tracker.mark_idle(1);

// Now watermark advances (5000)
assert_eq!(tracker.current_watermark(), Some(5000));

§Watermark Alignment

For multi-topic joins, alignment prevents fast topics from building unbounded state while waiting for slow topics:

use laminar_connectors::kafka::watermarks::{KafkaAlignmentConfig, KafkaAlignmentMode};
use std::time::Duration;

let config = KafkaAlignmentConfig {
    group_id: "orders-payments".to_string(),
    max_drift: Duration::from_secs(300), // 5 minutes
    mode: KafkaAlignmentMode::Pause,
};

Structs§

KafkaAlignmentConfig
Configuration for Kafka source watermark alignment.
KafkaWatermarkTracker
Kafka-specific watermark tracker wrapping PartitionedWatermarkTracker.
WatermarkMetrics
Metrics for watermark tracking.
WatermarkMetricsSnapshot
Snapshot of watermark metrics.

Enums§

AlignmentCheckResult
Result of an alignment check.
KafkaAlignmentMode
Alignment mode for multi-source watermark coordination.