A checkpoint that survived, and a log that lost its own resumes

Spacecraft computers reboot, so the voter has to come back from a reset having lost nothing. It now saves its state every 100 ticks the careful way: write a new file, fsync it, rename it over the old one, fsync the folder. A power cut leaves either the old checkpoint or the new one. Dead, stuck and hung replicas stay that way after a resume, because that is hardware state, and a checkpoint from another run is refused.

The first reset test, on an x86 server, killed the voter with kill -9 three times in a 200,000 tick run. Each time it was back within five seconds, and the final counts matched a run with no resets exactly: 149,835 correct, 163 caught, 2 wrong. It passed.

Then the summary said three resumes and the event log said one. The log held a single line, 55000,resumed,3. The header and every earlier event were gone.

The checkpoint was atomic and the log next to it was not. Events went through a BufWriter, so whatever sat in its buffer died with the process, while the checkpoint that said those ticks were done had already reached the disk. The saved state and the record of how it got there disagreed.

Before each checkpoint the log is now flushed and synced with sync_data, so it holds every event up to the checkpoint. The scenario check that kills the voter mid run also asserts that the header appears once and the kill event appears exactly once.

The same test can reset a real board with a sysrq reboot that skips the sync, or through a smart plug. No board has been reset yet.