1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
"waiting": "this means the current day's snapshot doesn't exist yet."
         "this state can transition with `restoring` when the snapshot exists."
         "this state can transition to `failing` if date changes without moving to restoring"

"restoring": "this means the snapshot was found and we have started restoring"
             "to a new database instance."

"testing": "this means the new database is online and we will reset the master password"
           "and try to connect to the endpoint and run a test query."

"cleaning": "this means the new database is being torn down."

"completed": "success ending state"

"failing": "this state happen can cause alarm"

"failed" : "failure ending state"




If state machine is waiting for 2017-08-21 and the date changes to 2017-08-22 signal an alarm that
the snapshot never finished! alarm state

if the snapshot arrives set state to `restoring` and being the restore.

if state is `restoring` and new db is not ready, exit

if state is `restoring` and new db is ready, change state to `testing`
change the master database password, connect to database, and run the test query.
if test query is successful, change state to `cleaning`
otherwise set to state to `alarming` and then transition `cleaning`

if `cleaning` and database is gone, set state to `waiting` for next day's snapshot