Oh, we’re about to do a risky thing! Ok, let’s have humans be VERY CAREFUL about executing the risky operation. Oops! We now have a global outage.
Some good Lessons Learned From Reading Post Mortems: 25% of bugs are from simply ignoring an error; 8% are from catching the wrong exception; 2% are from incomplete TODOs; 98% of critical failures can be reproduced in a 3 node cluster; Configuration bugs are the most common cause of really bad outages; every part of a machine can fail; humans are even more error prone than machines; not having the right sort of alerting; “obvious” things still cause a lot of failures.
Я почему раньше злой был? Потому что у меня велосипеда не было.
Ф.С. Хоффман