Skip to main content

Module streaming_coordinator

Module streaming_coordinator 

Source
Expand description

Simplified pipeline coordinator.

Sources push directly to the coordinator via crossfire::mpsc. The coordinator runs on a dedicated single-threaded tokio runtime (laminar-compute thread), isolating CPU-bound event processing from IO tasks (Kafka poll, S3 checkpoint writes, HTTP) on the main work-stealing runtime. SQL execution is delegated to PipelineCallback.

§Architecture

Source task (main tokio runtime)
  │  connector.poll_batch().await
  │
  └──── MAsyncTx ────► StreamingCoordinator (dedicated compute thread)
                             │  callback.execute_cycle()
                             │  callback.write_to_sinks()
                             ▼
                           Sinks

Structs§

StreamingCoordinator
Simplified pipeline coordinator — single tokio task, no core threads.