Site Reliability Engineering at Google
1
- In general, for any software service or system, 100% is not the right reliability target because no user can tell the difference between a system being 100% available and 99.999% available
2
- Google used OpenFlow in 2013
3
- Product development performance is largely evaluated on product velocity, which creates an incentive to push new code as quickly as possible. Meanwhile, SRE performance is (unsurprisingly) evaluated based upon reliability of a service, which implies an incentive to push back against a high rate of change.
4
- don't overachieve - Users build on the reality of what you offer, rather than what you say you'll supply, particularly for infrastructure services. If your service's actual performance is much better than its stated SLO, users will come to rely on its current performance.
5
- If you're too willing to take on toil, your Dev counterparts will have incentives to load you down with even more toil, sometimes shifting operational tasks that should rightfully be performed by Devs to SRE.
6
- ...until a better long-term solution was achieved, but some worried this kind of workaround would delay a real fix. This kind of tension is common within a team, and often reflects an underlying mistrust of the team's self-discipline: while some team members want to implement “hack” to allow time for a proper fix, others worry that a hack will be forgotten or that the proper fix will be deprioritized indefinitely. This concern is credible, as it's easy to build layers of unmaintainable technical debt by patching over problems instead of making real fixes.
7
- automate as much as possible
8
- release engineering... damn, didnt even know this existed
9
- balance between stability and agility
10