Struct ClusterController 

pub struct ClusterController { /* private fields */ }

Facade composing the cluster-control primitives.

Implementations

impl ClusterController

pub fn assignable_instances(&self) -> Vec<NodeId>

Node ids eligible to own vnodes: Active peers, plus self unless draining. Unlike Self::live_instances, non-Active peers are filtered so they never receive vnodes.

pub fn set_self_locality(&self, locality: Locality)

Record this node’s own locality. Call once at startup.

pub fn assignable_with_locality(&self) -> Vec<(NodeId, Locality)>

Self::assignable_instances paired with each node’s Locality (peers’ from members_rx, self’s from Self::set_self_locality).

pub fn members_watch(&self) -> Receiver<Vec<NodeInfo>>

Cloneable membership watch for reacting to join/leave events without polling.

pub async fn announce_drain_ack( &self, transition: &AssignmentDrainTransition, ) -> Result<(), String>

Publish and exactly read back proof that every local source reached one exact external input cut for transition.

Errors

Rejects a nonparticipant, stale boot, superseded process, malformed certificate, or failed durable write/read-back.

pub async fn drain_ack_quorum_reached( &self, transition: &AssignmentDrainTransition, ) -> Result<bool, String>

Whether every process in the exact frozen assignment roster durably acknowledged the same certificate and the transition remains the exact unfinalized durable assignment head.

Records from nodes outside fence are ignored because durable per-node slots can outlive membership. A missing or different-version record never contributes to quorum.

Errors

Fails closed when the certificate, current boot roster, assignment head, terminal decision, durable scan, or an expected participant’s record is malformed or unavailable. A controller without a configured assignment store returns Ok(false) while a drain is active.

pub async fn announce_adopted_assignment( &self, adoption: &CheckpointAssignmentAdoption, ) -> Result<(), String>

Publish and exactly read back this process’s adopted assignment map.

Errors

Fails closed for a malformed/stale process report or unavailable control storage.

pub async fn read_adopted_assignments( &self, ) -> Result<Vec<(NodeId, CheckpointAssignmentAdoption)>, String>

Each visible process’s exact adopted assignment identity.

Errors

Fails closed if any visible report is malformed or claims another publisher.

pub fn publish_checkpoint_assignment_fence( &self, fence: Option<CheckpointAssignmentFence>, )

Publish this node’s version-bound checkpoint fence. Called off the hot path by the snapshot watcher; admission and recovery revalidate it locally against current membership.

pub fn publish_checkpoint_drain_transition( &self, transition: Option<AssignmentDrainTransition>, )

Publish the exact locally audited drain transition, or clear it after commit/abort.

pub fn clear_checkpoint_drain_transition_if_matches( &self, expected: &AssignmentDrainTransition, ) -> bool

Clear the locally audited drain transition only when it still equals expected.

The comparison and clear share the process-authority transition lock with publication and terminal process fencing, so a newer transition cannot be erased by stale reconciliation.

pub fn checkpoint_drain_transition(&self) -> Option<AssignmentDrainTransition>

Current locally audited drain transition.

pub fn checkpoint_assignment_watch( &self, ) -> Receiver<Option<CheckpointAssignmentFence>>

Subscribe to local assignment-fence changes so a blocking durability probe can be interrupted when its checkpoint cut becomes stale.

pub fn checkpoint_assignment_fence( &self, assignment_version: u64, ) -> Option<CheckpointAssignmentFence>

Return the exact locally certified assignment cut while every participant remains checkpoint-capable. Active workers that own no vnode do not expand the checkpoint quorum. The clone is retained by the admitted attempt and propagated to followers.

pub async fn checkpoint_assignment_fence_for_leader( &self, assignment_version: u64, leader: &LeaderProof, ) -> Option<CheckpointAssignmentFence>

Return the locally certified assignment only when the announcing leader proof exactly matches the current durable authority. A predecessor retained during an in-progress drain additionally remains bound to the leader term that opened that drain.

pub async fn start_barrier_server( &self, bind_addr: SocketAddr, advertise_host: Option<String>, ) -> Result<SocketAddr, String>

Start the direct gRPC barrier sync server.

Errors

Propagates BarrierCoordinator::start_server errors.

pub async fn start_leased_barrier_server( &self, bind_addr: SocketAddr, advertise_host: Option<String>, process_lease: &ProcessLease, ) -> Result<SocketAddr, String>

Start a cluster control endpoint whose first advertisement is bound to an acquired stable-node process lease.

Errors

Rejects a lease for another node or boot, a conflicting prior identity, or server start.

pub async fn confirm_remote_leader_proof( &self, proof: &LeaderProof, deadline: Instant, ) -> Result<bool, String>

Confirm that one exact remote process still holds a proof read from durable authority.

The RPC returns only a challenge acknowledgement and never returns authority material.

Errors

Fails closed when the control RPC is unavailable, malformed, or misses deadline.

pub async fn audit_assignment_leader_authority( &self, fence: &CheckpointAssignmentFence, expected_proof: Option<&LeaderProof>, deadline: Instant, ) -> Result<LeaderProof, String>

Audit one exact assignment leader against durable and process-local authority.

This is a single bounded attempt intended for a caller that already holds its assignment and execution locks. expected_proof binds predecessor execution during an active drain; a stable assignment passes None and adopts the current durable grant. The audit never waits for convergence and never mutates assignment, lease, or source-gate state.

Errors

Fails closed when the installed assignment, elected participant, leader authority, live local/remote grant, or durable process term changes or cannot be verified before deadline.

impl ClusterController

pub fn instance_id(&self) -> NodeId

This instance’s ID.

pub fn kv(&self) -> &Arc<dyn ClusterKv>

The cluster gossip KV, for advertising/discovering per-stream state.

pub fn current_leader(&self) -> Option<NodeId>

Current leader: the lowest available active certified participant. A locally certified incumbent may retain candidacy while it drains, but a remote draining process is never nominated. When no certified participant is available, only a controller wired to durable leader fencing may nominate the lowest idle worker so it can repair assignment authority.

pub fn is_gossip_leader(&self) -> bool

True if this node is the gossip-elected candidate (lowest active id), ignoring the lease. The lease manager acquires only while this holds.

pub fn is_leader(&self) -> bool

True if this node may act as leader — the gate all leader-gated work inherits. With a lease wired, also requires this exact process’s live local-monotonic grant.

pub fn has_leader_lease_fencing(&self) -> bool

Whether this controller is wired to a durable leader lease. Gossip-only leadership is adequate for best-effort coordination but cannot certify exactly-once cluster decisions.

pub fn leader_fencing_token(&self) -> Option<u64>

Current live fencing token when this node owns the durable lease.

This is an observation for detecting that an operation crossed leadership terms. Reading it does not fence a later object-store write: a durable mutation is fenced only when the storage operation atomically validates the token. Callers must not stamp this value into an otherwise unconditional write and treat that as a correctness boundary.

pub fn capture_leader_proof(&self) -> Option<LeaderProof>

Capture the exact durable leader term currently authorized by local monotonic leases.

Gossip determines candidacy only. A proof is available solely when this process is the candidate, both process and leader deadlines are live, and the observed durable grant has this process’s exact owner identity.

pub fn proof_is_live(&self, proof: &LeaderProof) -> bool

Whether a captured proof remains the exact current locally authorized leader term.

pub fn capture_catalog_bootstrap_proof(&self) -> Option<LeaderProof>

Capture the durable leader grant while forming a cluster with no active member.

Every joining process may contend in a cold start, but the shared lease store admits only one exact owner. As soon as an active member is visible, a joining process ceases to be a candidate and cannot use its observation to publish catalog state.

pub fn catalog_bootstrap_proof_is_live(&self, proof: &LeaderProof) -> bool

Whether a catalog-bootstrap proof remains owned by this cold-start candidate.

pub fn leader_grant_watch(&self) -> Option<Receiver<Option<LeaderLease>>>

Subscribe to leader-grant changes for evented proof cancellation.

After notification, callers must use Self::proof_is_live. Candidacy changes are available separately through Self::leader_candidacy_watch. The lease manager publishes None when its monotonic deadline expires, so no polling loop is required.

pub fn set_leader_lease_watch( &self, lease: Receiver<Option<LeaderLease>>, owner: LeaderLeaseOwner, deadline: Arc<LeaseDeadline>, ) -> Result<(), String>

Wire the exact owner and one fixed local deadline used to fence leader work.

This compatibility seam is intended for fixed-generation tests and externally managed grants. A running super::super::LeaderLeaseManager must use Self::set_leader_lease_runtime_watches so deadline rotation remains visible.

Errors

Rejects an owner for another node or duplicate installation.

pub fn set_leader_lease_runtime_watches( &self, lease: Receiver<Option<LeaderLease>>, owner: LeaderLeaseOwner, deadline: Receiver<Arc<LeaseDeadline>>, ) -> Result<(), String>

Wire the exact owner and generation-scoped local deadlines used to fence leader work.

Readers hold the lease observation while sampling its deadline generation. The manager fences the old deadline, withdraws the old lease, then publishes the next deadline, so a stale lease can never be paired with a later live generation.

Errors

Rejects an owner for another node or duplicate installation.

pub fn leader_candidacy_watch(self: &Arc<Self>) -> Receiver<LeaderCandidacy>

Evented lease candidacy, including cold cluster formation and normal gossip leadership.

pub fn set_active(&self, active: bool)

Mark this node’s active status.

pub fn fence_process_lease(&self)

Permanently fence controller mutations after stable-node lease loss.

pub fn process_lease_is_live(&self) -> bool

Whether this process still owns its stable node identity.

pub async fn wait_for_process_lease_loss(&self)

Wait until the process-local lease deadline expires or is explicitly fenced.

pub fn set_process_lease_deadline( &self, deadline: Arc<LeaseDeadline>, ) -> Result<(), String>

Install the stable-node lease deadline before the runtime starts.

Reinstalling the same shared deadline is idempotent. A different deadline, or installation after terminal process fencing, is rejected so controller and data-plane gates cannot silently follow different lease clocks.

Errors

Rejects installation after fencing or replacement with a different deadline.

pub fn process_lease_deadline(&self) -> Option<Arc<LeaseDeadline>>

Shared process-local lease deadline, when cluster runtime wiring is complete.

pub fn set_process_lease_authority( &self, authority: Arc<ProcessLeaseAuthority>, ) -> Result<(), String>

Install the shared stable-node fencing authority before recovery starts.

Errors

Rejects replacing an already-installed authority.

pub async fn fence_process_incarnation( &self, participant: CheckpointParticipant, deadline: Instant, ) -> Result<ProcessLeaseFence, String>

Prove that an exact assignment participant was durably superseded within deadline.

Errors

Fails closed when no shared authority is installed or fencing cannot be proven in time.

pub fn process_fencing_deadline( &self, io_budget: Duration, ) -> Result<Instant, String>

Bound process fencing by the authority TTL plus the caller’s existing control-plane I/O budget, avoiding a second configurable timeout dimension.

Errors

Fails when no shared authority is installed or the deadline cannot be represented.

pub async fn verify_current_process_incarnation( &self, participant: CheckpointParticipant, deadline: Instant, ) -> Result<bool, String>

Verify one successor roster identity against the current durable process-lease head.

Errors

Fails when no authority is installed or the bounded durable read is uncertain.

pub async fn verify_process_lease_fence( &self, fence: &ProcessLeaseFence, deadline: Instant, ) -> Result<bool, String>

Re-read the exact durable records behind a process fence within deadline.

Errors

Fails closed when no shared authority is installed or verification cannot finish in time.

pub async fn record_assignment_recovery_decision( &self, proof: &LeaderProof, decision: AssignmentRecoveryDecision, deadline: Instant, ) -> Result<RecordAssignmentRecoveryDecisionResult, String>

Admit a recovery assignment only after re-verifying every durable process revocation.

Keeping this operation on the controller binds the process-lease authority and leader authority installed for this runtime. Callers cannot bypass revocation verification by writing a structurally valid recovery decision directly to the leader store.

Errors

Fails closed when either authority is absent, a process fence is not durable, the deadline expires, leadership changes, the decision conflicts, or storage is unavailable.

impl ClusterController

pub async fn announce_barrier( &self, ann: &BarrierAnnouncement, ) -> Result<(), String>

Leader-side announce.

Errors

Propagates BarrierCoordinator::announce errors.

pub async fn announce_prepare_barrier( &self, ann: &BarrierAnnouncement, quorum_window: Duration, ) -> Result<(), String>

Leader-side assignment-certified Prepare publication with its configured quorum window.

Errors

Propagates process-lease and BarrierCoordinator::announce_prepare errors.

pub async fn announce_prepare_barrier_until( &self, ann: &BarrierAnnouncement, attempt_deadline: Instant, retry_window: Duration, ) -> Result<(), String>

Leader-side Prepare publication whose fan-out is fenced by the exact attempt deadline. The retry window remains independent so a long checkpoint timeout does not create long individual RPC attempts.

Errors

Propagates process-lease and BarrierCoordinator::announce_prepare_until errors.

pub async fn observe_barrier_matching<F>( &self, predicate: F, ) -> Result<Option<BarrierAnnouncement>, String>

where F: FnMut(&BarrierAnnouncement) -> bool,

Observe the merged barrier history, validating durable authority only when predicate selects the announcement. Malformed or conflicting histories fail before filtering.

Errors

Propagates merge, transport, and matching reversible-authority failures.

pub async fn observe_checkpoint_prepare( &self, ) -> Result<Option<CheckpointPrepareObservation>, String>

Observe a clustered Prepare with a bounded compatibility timeout.

Errors

Rejects missing, stale, or conflicting authority and assignment certificates.

pub async fn observe_checkpoint_prepare_until( &self, checkpoint_timeout: Duration, ) -> Result<Option<CheckpointPrepareObservation>, String>

Observe a clustered Prepare, validate its leader with one durable-authority read, and report the local assignment disposition without consulting authority again. Hint I/O is bounded by the caller’s budget. Once an exact Prepare is decoded, authority validation uses the earlier of its same-identity direct receipt or stable first gossip observation, which retries cannot refresh. An unrelated stale direct Prepare never shortens the observation window for a newer gossip-only attempt.

Errors

Rejects a zero or overflowing timeout, observation/authority timeout, or missing, stale, conflicting authority and assignment certificates.

pub fn checkpoint_announcement_watch( &self, ) -> Option<Receiver<Option<BarrierAnnouncement>>>

Subscribe to direct checkpoint announcements. Consumers must retain a bounded KV poll: the watch is a latency path, while the merged gossip history remains the fallback.

pub fn checkpoint_prepare_received_at( &self, prepare: &BarrierAnnouncement, ) -> Option<Instant>

Stable local monotonic receipt or first-observation time for this exact Prepare.

pub async fn ack_barrier(&self, ack: &BarrierAck) -> Result<(), String>

Follower-side ack.

Errors

Propagates BarrierCoordinator::ack errors.

impl ClusterController

pub fn new( instance_id: NodeId, kv: Arc<dyn ClusterKv>, snapshot: Option<Arc<AssignmentSnapshotStore>>, members_rx: Receiver<Vec<NodeInfo>>, ) -> Self

Wrap the given primitives.

pub fn new_with_recovery_kv( instance_id: NodeId, kv: Arc<dyn ClusterKv>, recovery_kv: Arc<dyn ClusterKv>, snapshot: Option<Arc<AssignmentSnapshotStore>>, members_rx: Receiver<Vec<NodeInfo>>, ) -> Self

Wrap the given primitives with a separate durable recovery authority.

The barrier/discovery KV remains the low-latency path. recovery_kv owns recovery incarnations, generations, phase announcements, and acknowledgements.

pub fn new_with_recovery_incarnation( instance_id: NodeId, kv: Arc<dyn ClusterKv>, recovery_kv: Arc<dyn ClusterKv>, snapshot: Option<Arc<AssignmentSnapshotStore>>, members_rx: Receiver<Vec<NodeInfo>>, recovery_incarnation: Uuid, ) -> Self

Wrap the control primitives with a caller-generated boot identity.

pub fn set_leader_lease_store(&self, store: Arc<LeaderLeaseStore>)

Install the durable authority used to validate clustered checkpoint barriers.

pub fn install_local_leader_proof_provider(self: &Arc<Self>)

Serve only leader proofs that remain live on this process’s monotonic lease gate.

pub fn checkpoint_authority( &self, ) -> Result<Arc<LeaderLeaseStore>, ClusterCheckpointAuthorityError>

Exact durable authority installed for this cluster controller.

Cluster checkpoint code must use this handle rather than standalone outcome keys.

Errors

Returns super::super::ClusterCheckpointAuthorityError::NotConfigured when no authority exists.

pub fn cluster_min_watermark(&self) -> Option<i64>

Latest recovery-safe cluster watermark installed from an immutable Commit outcome or a validated committed checkpoint.

pub fn publish_cluster_min_watermark(&self, wm: i64)

Mirror the leader’s computed cluster-min watermark into the atomic so its own operators match followers. Monotonic — never lowers the published value.

pub fn committed_source_watermarks_snapshot( &self, ) -> Arc<FxHashMap<String, i64>>

Pin one internally consistent source-frontier snapshot for a compute cycle.

pub fn publish_committed_channel_progress( &self, channels: &[ChannelProgress], ) -> Result<(), String>

Install both scalar and source-keyed frontiers from one immutable committed channel cut.

Publication is monotonic per source, matching Self::publish_cluster_min_watermark. A source withheld by an active uninitialized channel remains unpublished until a later committed cut initializes it.

Errors

Returns an error when the committed channel cut contains an invalid watermark sentinel.

pub fn publish_committed_checkpoint_progress( &self, channels: &[ChannelProgress], source_watermarks: &BTreeMap<String, i64>, ) -> Result<(), String>

Install scalar and source-keyed frontiers from one committed checkpoint index.

The explicit source map retains decisions for a source whose physical channel inventory is empty in this cut. Publication remains monotonic so an older observer cannot regress a newer installed decision.

Errors

Returns an error when channel progress or an explicit source watermark is invalid or the explicit map disagrees with an initialized channel cut.

pub fn replace_recovered_checkpoint_progress( &self, channels: &[ChannelProgress], source_watermarks: &BTreeMap<String, i64>, ) -> Result<(), String>

Replace the process-local committed frontier snapshot with one recovered exact cut.

Unlike live publication, recovery may deliberately rewind to an older checkpoint or to genesis. The caller must hold the pipeline recovery/intake fence; this method serialises against any in-flight live publication and atomically establishes the new logical baseline before intake is reopened.

Errors

Returns an error when channel progress or an explicit source watermark is invalid or the explicit map disagrees with an initialized channel cut.

impl ClusterController

pub fn live_instances(&self) -> Vec<NodeId>

Live instance IDs: Active peers plus self.

pub fn checkpoint_instances(&self) -> Vec<NodeId>

Nodes that must participate in a checkpoint for the current assignment.

Draining owners remain responsible for their old vnodes through the handoff checkpoint, while joining, suspected, left, and missing nodes cannot contribute to a restorable cut. This is deliberately distinct from Self::assignable_instances, which excludes draining nodes so they never receive new ownership.

pub fn note_unresponsive(&self, peers: &[NodeId])

Record peers that failed to ack a capture quorum in time.

pub fn note_responsive(&self, peers: &[NodeId])

Clear peers that acked (they are demonstrably alive).

pub fn note_recovery_responsive(&self, participants: &[CheckpointParticipant])

Clear capture-quorum quarantine only for exact process boots that acknowledged a validated recovery stop round. Entries without a recorded boot, or for a different boot, remain quarantined. The caller must supply publishers from one complete, exact stopped quorum.

pub fn is_unresponsive(&self, peer: NodeId) -> bool

Whether peer has an unresolved capture-quorum failure.

pub fn admit_successor_process( &self, participant: CheckpointParticipant, ) -> bool

Admit a placement candidate only when it was never quarantined or it is a different boot incarnation. Callers must obtain participant from the lease-validated durable control KV.

pub fn begin_drain(&self) -> bool

Mark this node as draining. Returns whether this exact certified leader must retain its lease long enough to coordinate the predecessor cut. Idempotent.

pub fn is_draining(&self) -> bool

Whether this node is draining.

pub fn set_recovering(&self, recovering: bool)

Set or clear the coordinated-recovery fence.

pub fn is_recovering(&self) -> bool

Whether a coordinated restart is in flight on this node.

impl ClusterController

pub fn next_recovery_fault_request( &self, ) -> Result<RecoveryFaultRequest, String>

Allocate the next process-local recovery-fault request identity.

Shared authority converts this ordinal into a cluster-wide fault sequence. Retries retain the original ordinal; exhaustion fails closed instead of wrapping.

Errors

Returns an error if the process-local sequence is exhausted or becomes noncanonical.

pub fn recovery_fault_request( &self, sequence: u64, ) -> Result<RecoveryFaultRequest, String>

Reconstitute one previously allocated request for a bounded retry.

Errors

Rejects zero and ordinals that this controller process has not allocated.

pub async fn recovery_participant_incarnations( &self, participants: &[u64], ) -> Result<Vec<CheckpointParticipant>, String>

Resolve the exact live boot identity for every canonical participant.

Errors

Fails closed on a missing, malformed, nil, duplicate, or unexpected participant identity.

pub async fn available_recovery_participant_incarnations( &self, participants: &[u64], ) -> Result<Vec<CheckpointParticipant>, String>

Resolve every currently readable boot identity among a canonical candidate set. Missing candidates are omitted so placement can exclude lease-revoked or not-yet-started nodes without weakening exact checkpoint/recovery roster validation.

Errors

Fails closed on malformed input, a durable scan error, or a malformed/duplicate identity.

pub async fn recovery_incarnations_match( &self, round: &RecoveryRound, ) -> Result<bool, String>

Whether every current assignment-owner boot identity still equals the frozen round.

Errors

Returns an error when the current incarnation roster is unavailable or malformed.

pub async fn recovery_stopped_incarnations_match( &self, round: &RecoveryRound, ) -> Result<bool, RecoveryControlError>

Whether every owner and evidence reporter still has the boot identity frozen for stopping.

This check belongs only to the Prepare/stopped-evidence boundary. Evidence reporters do not join restore or release liveness quorums after their stopped reports are durable.

Errors

Returns an error when the current incarnation roster is unavailable or malformed.

pub async fn read_recovery_fault_inventory( &self, ) -> Result<RecoveryFaultInventory, String>

Atomically observed shared-authority recovery-fault inventory.

Errors

Returns an error when the leader authority is unavailable or malformed.

pub async fn read_recovery_admission_snapshot( &self, ) -> Result<RecoveryAdmissionSnapshot, RecoveryControlError>

Coherent committed-release and fault view from shared recovery authority.

Errors

Returns a classified error when the authority or immutable terminal cannot be validated.

pub async fn recovery_admission_is_current( &self, snapshot: &RecoveryAdmissionSnapshot, leader_proof: &LeaderProof, ) -> Result<bool, RecoveryControlError>

Confirm this recovery view still has the same terminal, no active faults, and the exact audited leader term.

Errors

Returns a classified error when the current shared authority cannot be validated.

pub async fn report_fault( &self, request: RecoveryFaultRequest, ) -> Result<RecoveryFaultReportOutcome, String>

Publish this process’s fault request so the leader drives a recovery round.

Errors

Fails when the process lease is stale or the request cannot be ordered in shared authority.

pub async fn report_terminal_fault( &self, request: RecoveryFaultRequest, ) -> Result<RecoveryFaultReportOutcome, String>

Publish a deterministic fault that automatic recovery must never consume.

The durable marker cannot be downgraded by a later process or ordinary fault report. The current operational reset boundary is replacement of the cluster authority namespace.

Errors

Fails when the process lease is stale or the request cannot be ordered in shared authority.

pub async fn begin_recovery_release( &self, terminal: &RecoveryAnnouncement, ) -> Result<Option<RecoveryReleaseGuard<'_>>, RecoveryControlError>

Validate this process’s atomically released fault while retaining the local fault-write fence. The caller must hold the returned guard through its source-gate transition.

Ok(None) means the terminal is no longer current, a newer active fault exists anywhere, or the local active/tombstoned slot does not match it. An exact tombstone is idempotent success only while the shared fault inventory remains settled.

Errors

Returns an error for a nonterminal release, malformed state, or failed durable I/O.

pub async fn read_local_fault_report_control( &self, ) -> Result<Option<u64>, RecoveryControlError>

This stable node’s active durable fault sequence, when present.

Errors

Returns an error when shared authority is unavailable or malformed.

pub async fn read_local_recovery_fault_control( &self, ) -> Result<Option<RecoveryFault>, RecoveryControlError>

This stable node’s complete active durable fault, including its terminal disposition.

Errors

Returns an error when shared authority is unavailable or malformed.

pub async fn read_local_fault_report(&self) -> Result<Option<u64>, String>

This process’s durable nonzero fault report with a display-stable error.

Errors

Returns an error when the point read fails or the local slot is malformed.

pub async fn read_fault_reports(&self) -> Result<Vec<(NodeId, u64)>, String>

Each active stable-node report and its globally monotonic authority sequence.

Errors

Returns an error when shared authority is unavailable or malformed.

pub async fn announce_recovered( &self, start: &RecoveryAnnouncement, ) -> Result<(), String>

Publish that this node restored the exact frozen recovery round.

Errors

Returns an error for an invalid round or when this node is outside its frozen quorum.

pub async fn read_recovered( &self, ) -> Result<Vec<(NodeId, RecoveryAnnouncement)>, String>

Each visible node’s exact restored recovery round.

Errors

Fails closed when any visible acknowledgement is malformed.

pub async fn announce_release_ready( &self, release: &RecoveryAnnouncement, ) -> Result<(), RecoveryControlError>

Publish that this exact process is prepared for the pending release intent.

Readiness is published only after local recovery state, shuffle loss accounting, and assignment transport authority are installed while source intake remains closed.

Errors

Returns an error for a non-Release phase, changed local fault, or stale process.

impl ClusterController

pub async fn max_recovery_generation(&self) -> Result<u64, String>

Highest durable recovery generation replicated by any visible participant.

Errors

Fails closed when the durable scan is unavailable or contains malformed state.

pub async fn adopt_recovery_generation( &self, generation: u64, ) -> Result<(), String>

Monotonically persist this participant’s recovery generation and read it back exactly.

Errors

Rejects zero, regression, unavailable durable storage, or a mismatched read-back.

pub fn recovery_incarnation(&self) -> Uuid

This process’s boot-unique recovery identity.

pub fn try_live_local_process_authority_identity( &self, ) -> Result<LocalProcessAuthorityIdentity, String>

Try to capture this process’s live local identity without waiting or performing I/O.

The process-authority transition lock is acquired with try_lock; checkpoint timing instrumentation can therefore fail closed instead of extending the measured pause. The lease is checked before and after the atomic identity load.

Errors

Returns an error when an authority transition is in progress, the local lease is not live, or the installed process identity is non-canonical.

pub async fn read_local_process_authority_evidence( &self, ) -> Result<LocalProcessAuthorityEvidence, LocalProcessAuthorityEvidenceError>

Read this process’s exact local assignment evidence with one bounded checked-KV operation.

Only the local stable-node slot is read; this method never scans shared assignment or recovery records. The process identity and sampled lease are captured before the read and revalidated immediately afterward. The retained report is then validated against the current boot, and identity is revalidated again around the sampled assignment fence. A canonical report from an older boot makes the view unavailable rather than being attributed to this process. A returned adoption also matches the exact locally audited assignment fence sampled before and after the final identity revalidation.

Errors

Fails closed when the process term is unpublished, the lease is not live, the bounded durable read fails or times out, retained bytes are malformed or non-canonical, a current-boot record contradicts the local identity, or identity changes during the read. Checked-storage errors are unavailable even when a backend internally conflates a malformed outer envelope with I/O. Invalid is reserved for logical values returned successfully to this method that then fail payload bounds, canonicality, current-slot validation, or the sampled same-version audited fence.

pub async fn publish_recovery_incarnation(&self) -> Result<(), String>

Publish and read back this process’s incarnation metadata.

This does not grant recovery-write authority. Cluster startup must use Self::publish_leased_recovery_incarnation to bind the durable process term.

Errors

Returns an error when the control write/read is unavailable or the read-back differs.

pub async fn publish_leased_recovery_incarnation( &self, lease: &ProcessLease, ) -> Result<(), String>

Publish the recovery identity only when it matches an acquired stable-node lease.

Errors

Rejects a lease for another node or boot identity, or a failed durable publication.

impl ClusterController

pub async fn audit_recovery_driver_proof( &self, round: &RecoveryRound, boundary: &str, ) -> Result<(), RecoveryControlError>

Audit that this process still owns the recovery round’s exact local and durable leader proof. A same-node fencing-term rotation is supersession just like a driver transfer.

Errors

Returns RecoveryControlError::Superseded when the exact proof is no longer current, RecoveryControlError::Uncertain for retryable authority I/O, and RecoveryControlError::Conflict for unavailable or invalid authority state.

pub async fn announce_recover_prepare( &self, round: &RecoveryRound, ) -> Result<(), String>

Announce phase 1 with the immutable stopped/evidence roster.

Errors

Returns an error unless this node is the round’s current leader and driver.

pub async fn announce_recover_start( &self, round: &RecoveryRound, epoch: u64, ) -> Result<(), String>

Transition the identical prepared round to Start with a target bound into the announcement. A missing or different Prepare is never upgraded.

Errors

Returns an error on lost leadership, an invalid round, or a mismatched prior phase.

pub async fn announce_recover_release( &self, round: &RecoveryRound, epoch: u64, ) -> Result<(), String>

Transition the identical Start to a pending Release. Source gates remain closed until the leader commits the exact compact readiness roster.

Errors

Returns an error on lost leadership, a changed incarnation roster, or a mismatched Start.

pub async fn try_commit_recover_release( &self, release: &RecoveryAnnouncement, ) -> Result<ReleaseCommitStatus, RecoveryControlError>

Commit a pending release after every frozen owner published its compact readiness record.

A pending attempt audits the frozen fault set before returning incomplete readiness. Once complete, the process roster and fault set are validated under the driver’s phase-transition mutex before admitting the content-addressed terminal into durable leader authority.

Errors

Returns a classified uncertain, conflict, or superseded outcome. Missing readiness remains a normal pending status.

pub async fn observe_recover_control( &self, ) -> Result<Option<RecoveryAnnouncement>, RecoveryControlError>

Active recovery announcement with semantic failures separated from uncertain I/O.

Errors

Classifies malformed state, superseded authority, and retryable durable I/O separately.

pub async fn observe_recover_control_for_durable_proof( &self, proof: &LeaderProof, ) -> Result<Option<RecoveryAnnouncement>, RecoveryControlError>

Observe a nonterminal recovery announcement through an exact durable leader proof.

This is intentionally independent of the local gossip candidacy view. It permits a stopped process to finish publishing an already-audited recovery topology after its driver loses candidacy, while the durable fencing term remains unchanged.

Errors

Classifies malformed state, superseded authority, and retryable durable I/O separately.

pub async fn latest_committed_recover_release( &self, ) -> Result<Option<RecoveryAnnouncement>, RecoveryControlError>

Latest irrevocable recovery release admitted by the append-only leader authority.

Errors

Classifies missing/corrupt terminal state as conflict, takeover as supersession, and durable I/O as uncertainty.

pub async fn observe_committed_recover_release( &self, round: &RecoveryRound, epoch: u64, ) -> Result<Option<RecoveryAnnouncement>, RecoveryControlError>

Resolve the exact committed terminal for one pending release intent across leader takeover. An older terminal is unrelated; a same-generation divergence is corruption and a newer terminal supersedes the caller.

Errors

Returns a classified conflict, supersession, or uncertain durable read.

pub async fn retire_committed_recover_release_hint( &self, round: &RecoveryRound, epoch: u64, ) -> Result<bool, RecoveryControlError>

Best-effort cleanup for this driver’s mutable Release discovery hint after the exact terminal is irrevocably present in leader authority. Cleanup never contributes to commit validity; followers resolve the terminal from authority even if this write is uncertain.

Errors

Classifies malformed or divergent local intent separately from retryable I/O.

pub async fn observe_recover( &self, ) -> Result<Option<RecoveryAnnouncement>, String>

Active nonterminal recovery announcement from the locally elected driver.

Errors

Returns a display-stable classified control error.

pub fn recovery_driver_is_current(&self, round: &RecoveryRound) -> bool

Whether the round’s declared driver is the current elected leader in this local view.

pub fn recovery_round_contains_current_process( &self, round: &RecoveryRound, ) -> bool

Whether the assignment-owner quorum names this exact process, not only its stable node id.

pub fn recovery_round_requires_current_process_stop( &self, round: &RecoveryRound, ) -> bool

Whether this exact owner or evidence-reporter process must stop for the round’s Prepare.

pub async fn announce_stopped( &self, round: &RecoveryRound, ) -> Result<(), String>

Ack phase 1 for the exact frozen round.

Errors

Returns an error for invalid state or when this node is outside the stopped roster.

pub async fn read_stopped( &self, round: &RecoveryRound, participants: &[NodeId], ) -> Result<Vec<RecoveryStoppedReport>, RecoveryControlError>

Point-read only the still-missing members of an exact stopped roster.

Errors

Returns a conflict for a noncanonical or out-of-round subset and preserves the same uncertainty/conflict/supersession classification used by recovery quorum polling.

pub async fn clear_recover(&self, round: &RecoveryRound) -> Result<bool, String>

Clear only this driver’s still-identical recovery announcement. The per-controller lock makes the read/clear conditional with respect to a concurrent local phase transition.

Errors

Returns an error when the visible local announcement is malformed.

pub async fn wait_for_barrier<F>( &self, pred: F, timeout: Duration, ) -> Result<Option<BarrierAnnouncement>, String>

where F: FnMut(&BarrierAnnouncement) -> bool,

Wait until the merged barrier history yields an announcement matching pred, or timeout expires (→ Ok(None)). Observation remains side-effect free; event-time progress must come from immutable checkpoint authority. Push-driven off the gRPC announcement watch when available; gossip-KV-only deployments (and KV-only announcements) are covered by a fallback poll — 250ms with the watch, 25ms without.

Errors

Returns the first observation error instead of converting a known protocol or transport failure into a timeout.

pub async fn wait_for_quorum( &self, prepare: &BarrierAnnouncement, expected: &[NodeId], deadline: Duration, ) -> QuorumOutcome

Leader-side: poll until quorum or deadline.

pub fn snapshot_store(&self) -> Option<&AssignmentSnapshotStore>

Assignment snapshot store, if configured.

Trait Implementations

impl Debug for ClusterController

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.