Module shuffle 

Cross-instance shuffle: message codec, credit flow, wire protocol. Cross-instance shuffle: one persistent TCP connection per peer pair, length-prefixed frames with a 1-byte tag. Backpressure is carried by the per-partition tokio mpsc on the consuming side.

Re-exports

pub use barrier_tracker::BarrierTracker;

pub use message::ShuffleMessage;

pub use transport::SHUFFLE_ADDR_KEY;

pub use transport::ShufflePeerId;

pub use transport::ShuffleReceiver;

pub use transport::ShuffleSender;

Modules

barrier_tracker

Chandy–Lamport barrier alignment. One BarrierTracker per sharded operator; observe(input, barrier) returns Some once every input has seen the same checkpoint id.

message

Wire format for the shuffle: [u32 length][u8 tag][payload].

transport

TCP shuffle: a per-peer connection pool for senders, an accept loop for receivers. Each frame carries a node id in its handshake so the receiver can attribute incoming traffic. See super::message for the wire format.