Adds the full distributed simulation module with an MPI backend enabling
Shadow to run across multiple physical hosts using MPI collectives.

New module: src/main/core/distributed/
- mod.rs: ShardId, PartitionMap, SerializedPacket (UDP/TCP big-endian
  binary format), RemotePacketEvent (batch encode/decode with versioned
  framing), 22 unit tests covering serialization round-trips, partition
  map construction, and deterministic ordering.
- synchronizer.rs: DistributedSynchronizer trait (wait, global_min_next_event),
  SingleShardSynchronizer (no-op for non-distributed mode),
  UnixSocketSynchronizer (binary control protocol over Unix sockets).
- exchange.rs: RemotePacketExchange trait (send, receive),
  NoopRemotePacketExchange, InProcessRemotePacketExchange,
  UnixSocketRemotePacketExchange, DistributedPacketExchangeContext,
  encode/decode helpers shared across all backends, 12 unit tests.
- mpi_backend.rs: MpiSynchronizer via MPI_Barrier+MPI_Allreduce(MIN),
  MpiRemotePacketExchange via MPI_Alltoall+MPI_Send/MPI_Recv, direct
  FFI to libmpi avoiding mpi crate compatibility issues with OpenMPI
  4.1.x, 2 unit tests.

Existing code integration:
- configuration.rs: hidden --distributed-shard-count/id/ipc-socket-dir/
  partition-file CLI options (clap + serde)
- sim_config.rs: global host ID assignment in sorted hostname order
  before any shard filtering, PartitionMap construction (modulo or
  YAML partition file), HostInfo.id field
- worker.rs: WorkerShared distributed state (partition_map, shard_id,
  outbound_remote_packets staging buffer), send_packet remote routing,
  serialize_packet_for_remote (UDP+RustTCP, legacy-C-TCP rejected),
  send_remote_packets/receive_remote_packets exchange integration
- manager.rs: ManagerConfig distributed fields, exchange/synchronizer
  calls at each scheduling window boundary for global time coordination
- controller.rs: auto-selects MpiSynchronizer+MpiRemotePacketExchange
  when distributed_shard_count > 1 (with distributed_mpi feature),
  falls back to Noop+SingleShard for single-process mode
- event.rs: Event::new_packet_with_meta preserves source host/event
  metadata from the sending shard
- shadow.rs: feature-gated MPI_Init/MPI_Finalize hooks

Build system:
- SHADOW_USE_MPI CMake option (default OFF), finds MPI via
  find_package(MPI) and passes distributed_mpi to Cargo features
- distributed_mpi Cargo feature (default OFF)
- build.rs links libmpi via pkg-config (ompi-c)
- Added thiserror 2.0 dependency

Tests:
- src/test/distributed/: two-host UDP config for MPI, CMakeLists.txt
  with udp-distributed-mpi-shadow (2-rank) and
  udp-distributed-mpi-4-shadow (4-rank) CTests (MPI-conditional)
- All 186 lib tests pass in both default and distributed_mpi builds

Co-Authored-By: Claude <noreply@anthropic.com>

…nd MPI tests

Receive-side packet reconstruction:
- deserialize_packet_from_remote() in worker.rs converts SerializedPacket
  back to PacketRc for both UDP and Rust TCP, preserving full header fields
  including TCP flags, SACK blocks, timestamps, and priority
- WorkerShared::receive_remote_packets() now creates real Event objects
  via Event::new_packet_with_meta() and pushes them to destination host
  event queues with preserved source metadata

Configuration validation:
- SimConfig::new() rejects multi-shard mode (shard_count > 1) when
  experimental.use_new_tcp is false, with a clear error message

MPI backend fixes:
- mpi_backend.rs: Rewrote FFI to use correct pointer types for MPI_Comm,
  MPI_Datatype, MPI_Op (8-byte pointers on 64-bit, not c_int)
- MPI_COMM_WORLD and other constants linked via external symbols
  (ompi_mpi_comm_world, ompi_mpi_int64_t, etc.) instead of hardcoded
  0x44000000 sentinel
- Fixed deadlock: receive() reuses sizes from send()'s MPI_Alltoall
  instead of calling a second MPI_Alltoall with zero sizes
- Fixed send_remote_packets() to always call exchange.send() (even when
  empty) to participate in MPI collectives on all ranks
- Fixed window advancement: distributed path now computes window_end as
  global_min + runahead (was incorrectly setting end=start)

Build system fixes:
- WORKSPACE_FEATURES separate from RUST_FEATURES so distributed_mpi is
  not passed to the shim crate (which doesn't define the feature)
- build.rs uses CARGO_FEATURE_DISTRIBUTED_MPI env var instead of
  #[cfg(feature)] which doesn't work in build scripts
- CMake links MPI_C_LIBRARIES directly into the shadow executable

Runtime fixes:
- MPI rank overrides distributed_shard_id from CLI default
- Data directory rewritten to <base>.shard-N per rank
- Manager builds DNS from all hosts but only executes local hosts

End-to-end MPI tests:
- 2-rank test: udp-distributed-mpi-shadow (PASS, 0.74s)
- 4-rank test: udp-distributed-mpi-4-shadow (PASS, 0.60s)
- Cross-shard UDP packet delivery verified via sendto/recvfrom syscalls
- Both tests run with --oversubscribe for single-machine execution

Co-Authored-By: Claude <noreply@anthropic.com>

- Document MPI backend status with end-to-end test results
- 2-rank (0.74s) and 4-rank (0.60s) MPI CTests passing
- Cross-shard UDP packet delivery verified

Covers building with MPI, configuration, host partitioning, running on single
machine and clusters, MPI CTests, architecture overview (window protocol),
determinism guarantees, limitations, and troubleshooting.

Co-Authored-By: Claude <noreply@anthropic.com>

Worktree snuggly seeking pumpkin

Remove the local-only MPI_Alltoallv skip: it gated a collective over
MPI_COMM_WORLD on this rank's local send/recv totals, which deadlocks the
busy ranks whenever some shards exchange packets while another shard is
locally idle. The globally-idle round is still handled by the existing
1-byte dummy buffers. Drop the now-redundant payload-skipped metric.

Reject use_dynamic_runahead in distributed mode (shard_count > 1): per-shard
runahead diverges and breaks cross-shard event ordering. Default is off, so
normal runs are unaffected.

Validated on the 4-node and 64-node ethlambda cluster scenarios.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NBQ3nhNKoHpPtkCdotuHyZ

Adds the experimental `use_host_pair_runahead` option (default true). When
true, runahead uses the smallest latency between distinct hosts (host-pair,
~12ms here), the existing fast cadence. When false, runahead is computed over
all paths including self-loops (~1ms), matching upstream vanilla Shadow's
window so the fork reproduces vanilla's chain.

- configuration.rs: new experimental option + default Some(true).
- manager.rs: select smallest_path_latency_ns when the flag is false; thread
  the value into WorkerShared.
- worker.rs: the self-loop dynamic-runahead filter follows the flag.

Determinism investigation (16-validator ethlambda devnet, 4 hosts/shard x 4
shards) recorded in where-we-are.md: with native_preemption_enabled=false and
model_unblocked_syscall_latency=false, distributed (4 shards) == single-process
== upstream vanilla Shadow, byte-for-byte across all 16 hosts. The flag value
does not affect distributed==single equality (both modes compute runahead
consistently); it only selects whether the fork matches vanilla's 1ms window or
runs the fast 12ms cadence. Run-to-run nondeterminism seen earlier was caused
by native_preemption_enabled=true (documented to break determinism), not by
cross-shard delivery, the runahead flag, or the rand-crate version difference.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NBQ3nhNKoHpPtkCdotuHyZ

…f main

Adds the distributed-fork notice to README.md. Removes where-we-are.md and
explainer-distributed-shadow.html from main; these development docs live only
in the dev branch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NBQ3nhNKoHpPtkCdotuHyZ

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NBQ3nhNKoHpPtkCdotuHyZ

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NBQ3nhNKoHpPtkCdotuHyZ

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NBQ3nhNKoHpPtkCdotuHyZ