Some good Lessons Learned From Reading Post Mortems: 25% of bugs are from simply ignoring an error; 8% are from catching the wrong exception; 2% are from incomplete TODOs; 98% of critical failures can be reproduced in a 3 node cluster; Configuration bugs are the most common cause of really bad outages; every part of a machine can fail; humans are even more error prone than machines; not having the right sort of alerting; “obvious” things still cause a lot of failures.

http://danluu.com/postmortem-lessons/