← notes

One retry before paging, and one all clear after

Oct 10, 2026

A stale timetable now gets one retry before anyone is paged, and an alert now says when its problem is over.

The watchdog paged: two route weeks in the next eight had a timetable older than 36 hours. One request in the nightly run had come back bad. No job failed, the failure streak reset at once, and so the week simply waited a full day for the next run. Asked again by hand, both weeks answered fine.

The week job now asks once more, ten minutes after a bad answer, and treats a blank week as suspect rather than as an answer. It skips that retry once the failure streak has reached the alert threshold, because by then the endpoint is down and a second request only doubles the load on it. One spec removes the guard and fails.

That left the other half of the earlier breaker work. A warning said something broke, and nothing said when it stopped. A one-off DNS miss on the fares relay and a whole morning of {"outbound":null,"inbound":null} from the airline’s timetable read exactly alike, and telling them apart meant opening a console.

Now an alert that actually sends marks its source as open for 30 days, and the first success at that source afterwards sends one message saying it is working again, then clears the mark. Deleting the mark is the lock, so two workers succeeding at once send one message. It is silent when nothing was open, and the hourly dedup still stops a flapping upstream from sending pairs.

A page now comes with its own ending.