Expand description
Kafka watermark integration.
Integrates laminar-core per-partition watermarks and watermark
alignment groups with the Kafka source connector.
§Per-Partition Watermarks
Tracks watermarks at Kafka partition granularity, allowing progress even when some partitions are slow or idle:
ⓘ
use laminar_connectors::kafka::watermarks::KafkaWatermarkTracker;
use std::time::Duration;
let mut tracker = KafkaWatermarkTracker::new(0, Duration::from_secs(30));
tracker.register_partitions(4);
// Update individual partitions
tracker.update_partition(0, 5000);
tracker.update_partition(1, 3000); // slow partition
// Combined watermark is minimum (3000)
assert_eq!(tracker.current_watermark(), Some(3000));
// Mark slow partition as idle
tracker.mark_idle(1);
// Now watermark advances (5000)
assert_eq!(tracker.current_watermark(), Some(5000));§Watermark Alignment
For multi-topic joins, alignment prevents fast topics from building unbounded state while waiting for slow topics:
ⓘ
use laminar_connectors::kafka::watermarks::{KafkaAlignmentConfig, KafkaAlignmentMode};
use std::time::Duration;
let config = KafkaAlignmentConfig {
group_id: "orders-payments".to_string(),
max_drift: Duration::from_secs(300), // 5 minutes
mode: KafkaAlignmentMode::Pause,
};Structs§
- Kafka
Alignment Config - Configuration for Kafka source watermark alignment.
- Kafka
Watermark Tracker - Kafka-specific watermark tracker wrapping
PartitionedWatermarkTracker. - Watermark
Metrics - Metrics for watermark tracking.
- Watermark
Metrics Snapshot - Snapshot of watermark metrics.
Enums§
- Alignment
Check Result - Result of an alignment check.
- Kafka
Alignment Mode - Alignment mode for multi-source watermark coordination.