Sitelet https://github.com/restatedev/restate/pull/5149
Skip to content

[admin] Add partition leader election diagnostics - #5149

Open
pcholakov wants to merge 2 commits into
mainfrom
codex/sisko-election-observability
Open

[admin] Add partition leader election diagnostics#5149
pcholakov wants to merge 2 commits into
mainfrom
codex/sisko-election-observability

Conversation

@pcholakov

@pcholakov pcholakov commented Aug 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Adds diagnostics for partition leader election so unexpected target choices can be root-caused from logs. These diagnostics were used to root-cause the premature reconfiguration completion fixed separately in #5150. Observability only: the selection path is extracted into a free function so tests can drive it, but candidate eligibility, scoring, and tie-breaking are unchanged; the new diagnostic types never feed selection.

Leader election combines two independent inputs: the gossip failure detector decides which replicas are eligible, and the legacy GetNodeState poll supplies partition-processor status for scoring. A mismatch between them—a node alive in gossip whose legacy poll failed—silently degrades a warm replica to the lowest score. Previously, nothing recorded the inputs to a given choice.

This PR adds two log streams:

  • INFO, on every target change: partition, configuration version, previous and new target, and the winner's eligibility, status, and generation as seen by both detectors.
  • DEBUG, under a dedicated target (enable with restate_admin::cluster_controller::leader_election=debug): the full per-candidate input set (both detectors' views, status presence and age, replay and run modes, LSN positions, affinity, and score) plus the legacy cluster-state age. The candidate dump is only built when this target is enabled.

The new test documents a failure-detector mismatch the scheduler currently permits: N2 is eligible per gossip while a failed legacy poll marks it dead and removes its processor status; N1 is Starting. Both score zero, and replica-set order selects N1. Once N2's Active status is visible again, N2 is selected. The test only exists to demonstrate potential behavior.

Log sample

Example output

Representative structured output from a controlled three-node run. The values are sanitized from captured logs and line-wrapped for readability; field names and the INFO/DEBUG split reflect this PR.

INFO restate_admin::cluster_controller::service::scheduler:
  selected partition processor leader
  partition_id=25 configuration_version=v206 previous_target=None new_target=N1
  score=2 gossip_eligible=true gossip_state=Alive gossip_node_id=Some(N1:47)
  legacy_get_node_state=Alive replay_status=Some(Active)
  legacy_node_generation=Some(N1:47)

DEBUG restate_admin::cluster_controller::leader_election:
  partition processor leader election diagnostics
  partition_id=25 configuration_version=v206 previous_target=None new_target=N1
  legacy_cluster_state_age_ms=Some(1091)
  candidates=[
    LeaderCandidate { node_id: N1, gossip_eligible: true, gossip_node_id: Some(N1:47), gossip_state: Alive, legacy_get_node_state: Alive, has_affinity: false, has_processor_status: true, replay_status: Active, status_age_ms: 1577, planned_mode: Follower, effective_mode: Follower, last_applied_log_lsn: Some(Lsn(415997)), durable_lsn: Some(Lsn(415539)), last_archived_log_lsn: Some(Lsn(397678)), target_tail_lsn: Some(Lsn(415998)), last_record_applied_age_ms: Some(2837), score: 2, legacy_node_generation: Some(N1:47) },
    LeaderCandidate { node_id: N2, gossip_eligible: true, gossip_node_id: Some(N2:51), gossip_state: Alive, legacy_get_node_state: Alive, has_affinity: false, has_processor_status: false, score: 0, legacy_node_generation: Some(N2:51) }
  ]

INFO restate_admin::cluster_controller::service::scheduler:
  selected partition processor leader
  partition_id=25 configuration_version=v206 previous_target=Some(N1) new_target=N2
  score=2 gossip_eligible=true gossip_state=Alive gossip_node_id=Some(N2:51)
  legacy_get_node_state=Alive replay_status=Some(Active)
  legacy_node_generation=Some(N2:51)

@github-actions

github-actions Bot commented Aug 7, 2026 •
edited
Loading

Copy link
Copy Markdown

Test Results

  8 files  ±0    8 suites  ±0   5m 10s ⏱️ +2s
 61 tests ±0   61 ✅ ±0  0 💤 ±0  0 ❌ ±0 
268 runs  ±0  268 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit d28b6ac. ± Comparison against base commit 1343a8b.

♻️ This comment has been updated with latest results.

@pcholakov pcholakov changed the title chore(admin): log leader-election inputs [admin] Add partition leader election diagnostics Aug 7, 2026
@pcholakov pcholakov mentioned this pull request Aug 7, 2026
Open
@pcholakov
pcholakov force-pushed the codex/sisko-election-observability branch from 627cbde to 6c9f02a Compare August 7, 2026 11:21
pcholakov
pcholakov commented Aug 7, 2026
View reviewed changes
Comment thread crates/admin/src/cluster_controller/service/scheduler.rs
legacy_cluster_state,
nodes_config,
);
info!(

@pcholakov pcholakov Aug 7, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IDK if this might be too noisy, happy to also drop this to debug! if we want to keep the volume lower.

@pcholakov
pcholakov marked this pull request as ready for review August 7, 2026 12:27
@pcholakov
pcholakov requested a review from tillrohrmann August 7, 2026 12:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tillrohrmann tillrohrmann Awaiting requested review from tillrohrmann

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@pcholakov