9th July Rollback
A major crash occurred on this day, at 20:09 BST. It appears that to prevent some character files from being lost entirely, the entire MUD was restored from a 10-day old backup. Logs from the talker from the period in which the problem was discovered and after the rollback that followed are here: Ubercrash talkerlog
- 2009 07 09: 20:09 BST - Mud goes down.
- 2009 07 09: ??:?? BST - Mud comes back up, many players' accounts missing.
- 2009 07 09: 21:52 BST - Mud down again.
- 2009 07 09: 23:00 BST - Mud back up, rollback in evidence.
What Actually Happened
The Discworld MUD runs on a VMWare/ESX platform. Normally, snapshots of the virtual drives are written to disk every day. According to Sojan, on the 29th something happened to stop the snapshots from flushing the data, and the drive filled and corrupted itself. As a result, the disk crashed, and the latest backup we have is from the 29th.