This doc is an attempt at an introductory guide to readings in resilience engineering, aimed at software engineers. I've tried to organize a few key papers into themes. These papers should be accessible to casual readers.
Let's start by defining what resilience means. The extremely influential researcher David Woods defines resilient organizations as those that can adapt effectively to surprise.
Note that when we use techniques such as retries, fallbacks, circuit breakers, bulkheads, and adaptive concurrency limits, we're designing our system to gracefully handle known failure modes. Woods refers to this as robustness, not resilience. Robustness captuers how well we handle troubles that we can foresee (known unknowns). Resilience capture how well we can handle troubles that we did not foresee (unknown unknowns).
Four concepts for resilience and the implications for the future of resilience engineering by Woods discusses four different common usages of the term resilience, and describes why he considers robustness to be a separate concept.
Another very readable paper on how Woods defines resilience is Resilience is a verb.
Resilience engineering as a field emerged from the safety science community. That's why you'll often see examples from aviation and medicine, as well as other safety critical areas like maritime, space flight, nuclear power, and rail.
Because of this history, the earlier papers that we associate with resilience engineering are reactions to previous ways of thinking about accidents in particular and safety in general.
Note that traditional approaches to safety often focus on minimizing variance associated with humans doing work, using techniques such as documented procedures and enforcement mechanisms for deviating from them.
For those of us who work on cloud web services, we don't have this legacy of enforced procedures to contend with.
The "new look" or "new view" refers to a change in perspective on how accidents happen, which focuses on understanding how actions taken by actors involved in the incident were rational, given what information those actors had at the time that events were unfolding.
Johan Bergström of Lund University has three excellent short (<10 minute) videos:
- Was it technical failure or human error?
- Three analytical traps in accident investigation
- Two views on human error
Reconstructing human contributions to accidents: the new view on error and performance by Dekker and The eror of counting errors by Robert Wears are two great introductory papers on this topic (alas, both paywalled).
Safety-II is a perspective on the role that humans play in safety-critical systems, proposed by Erik Hollnagel. In the Safety-II perspective, it is the everyday, normal work of the humans in the system that create the safety, as opposed to the errors of humans that erode it.
Hollnagel's From Safety-I to Safety-II: A White Paper is a very readable introduction to Safety-II concepts.
Another good article is Dekker's Why do things go right? on the Safety Differently website.
A recurring theme in resilience engineering is about reasoning holistically about systems, as opposed to breaking things up into components and reasoning about components separately. This perspective is known as systems thinking, which is a school of thought that has been influential in the resilience engineering community.
You'll often hear the phrase socio-technical system. This language emphasizes that systems should be thought of as encompassing both humans and technologies, as opposed to thinking about technological aspects in isolation.
How complex systems fail by Richard I. Cook is a great starting point. It's a short paper and very easy to read.
Drift into failure by Sidney Dekker is a book written for a lay audience, so it is also very readable. Dekker draws heavily from systems thinking to propose a theory about how complex systems can evolve into unsafe states.
The systems we are interested in often involve a collection of people working together in some way to achieve a task. One particularly relevant example involves a collection of engineers working together to troubleshoot and repair a system during an ongoing incident.
Common Ground and Coordination in Joint Activity is an oft-cited paper on what is required for people to effectively coordinate when working on tasks together.
One thing we software folk do have in common with the safety-critical world is the increased adoption of automation. Automation introduces challenges, and the nature of these challenges is a topic of many resilience engineering papers.
You might hear the phrase joint cognitive system in the context of automation. This terms refers to systems that do cognitive work that are made up of a combination of humans and software. There is an entire research discipline that studies joint cognitive systems called cognitive systems engineering, initially developed by David Woods and Erik Hollnagel, both of whom would both later go on to play a significant role in developing the field of resilience engineering.
Because resilience engineering researchers like Woods and Hollnagel have their roots in cognitive systems engineering, and because of the ever-increasing use of software automation in society, this community is very concerned about the potential brittleness associated with poor use of automation.
Ironies of automation by Lisanne Bainbridge is a classic paper on the problems that automation can introduce. The paper was originally written in 1983, and continues to be widely cited.
Ten challenges for making automation a team player by Klein et al. is a more recent paper that outlines the requirements for automation to be genuinely effective in socio-technical systems. This work draws heavily from the theme of coordination discussed earlier.
The late Jens Rasmussen is an enormously influential figure in the resilience engineering community.
One of Rasmussen's most famous papers is Risk management in a dynamic society: a modelling problem, published in 1997, which introduces Rasmussen's dynamic safety model.
In this widely cited paper, Rasmussen advocates for a cross-disciplinary, systems-based approach to thinking about how accidents occur. He argues that accidents occur because the system migrates across a dangerous boundary, and this migration occurs during the course of normal work.
Here is a depiction of the model from that paper:
We've already referenced several papers authored or co-authored by David Woods. Woods is a force of nature in the field of resilience engineering, having played a key role in creating the field itself. Woods is incredibly prolific, and has introduced a wide variety of concepts related to resilience engineering.
Woods is interested in resilience engineering principles that apply across an enormous range of different types of systems: whether we're talking about the organs in a biological organism up to organizations like NASA.
Because he's interested in general principles, many of his papers are written at a very abstract level, where he discusses generic concepts such as units of adaptive behavior or saturation.
David Woods uses the metaphor of a system moving within a boundary in his writings on resilience engineering, but in a slightly different way than Rasmussen.
Woods sees the boundary as a competence envelope. There are two different regimes of system behavior: far from the boundary and near the boundary.
When a system is far from the boundary, the system (and its environment) behave as expected. By contrast, when a system grows near to the boundary, surprises happen. Woods uses the metaphor of dragons to capture the surprises that occur when a system moves near the boundary, and how the system's model of the world is violated when it enters this regime.
It is how units within a system adapt when the system moves near the boundary, how these units deal with the dragons, that is one of the prime concerns of Woods.
Woods's Essentials of Resilience, revisited discusses behavior at the boundary, although it doesn't use the dragon metaphor.
Woods's idea of the adaptive universe is characterized by three properties:
- Resources are finite
- Surprise is fundamental
- Change never stops
I haven't found a good introductory paper for the adaptive universe, as it encompasses an enormous number of topics, including the topic of dragons at the boundaries that we discussed earlier.
I recommend watching Woods's Resilience Engineering short course, which covers this topic.
I've written my own notes on the short course, which you might find useful. In particular, you might be interested in my summary notes.
Woods introduced the theory of graceful extensibility to capture how successful systems adapt effectively to surprise. The most relevant paper here is The theory of graceful extensibility: basic rules that govern adaptive systems.