Connect Rocket Outage - Post Mortem
» Published on
-
Resolved
All of the data that was unavailable as a result of the restoration of our Database on December 27, 2022 has now been recovered and will remain available for customer purposes. On the off chance a customer requires any of this data, please contact us and we'll provide any necessary reports etc. This issue is now resolved.
» Updated -
Post-Mortem
December 27th, 2022 from 15:16 PST (3:16pm Pacific Standard Time) until 16:28 PST (4:28pm Pacific Standard Time) Connect Rocket suffered a significant outage affecting all areas of our service.
While undertaking preparations for a scheduled database update to our production system, a configuration change caused our primary database to become unavailable. Our team was able to create a new database from our backup, which had been created the previous night - December 26th at 22:31 PST (10:31pm Pacific Standard Time) and restore all aspects of the service.
The result of restoring from a backup is a loss of any data added or modified between December 26th at 22:31 PST (10:31pm Pacific Standard Time) and December 27th at 15:16 PST (3:16pm Pacific Standard Time). This would include updates to Members and Groups, Notifications sent and received, Locate Requests and Member Duty Status changes.
The root cause of this incident was a different configuration between our Staging/QA Database and our Production Database. The incident was exasperated by a reliance on a default configuration which had changed between different versions of the database.
How are we going to prevent this from happening again? Review configurations to explicitly set values rather than relying on defaults; Enhance internal control processes to better manage the review of configuration changes before applying them; Better align configurations between our Production and Staging/QA Environments to reduce differences between the two environments.
» Updated