This an introductory guide to readings in resilience engineering, aimed at software engineers. Key papers are organized into themes:
- What is resilience?
- Changing perspectives on accidents and safety
- Complex systems
- Coordination
- Automation
- Boundary as a model (Rasmussen)
- David Woods
The papers linked here should all be accessible to casual readers.
When you're ready for more, check out resilience engineering notes.
A resilient organization adapts effectively to surprise.
Here I'm using the definition proposed by David Woods. Before going into more detail about resilience, it's important to distinguish it from a different concept that Woods calls robustness.
When we talk about designing highly available systems, we usually cover techniques such as redundancy, retries, fallbacks, and failovers. We think about what might go wrong (e.g., server failure, network partition), and design our system to gracefully handle these situations.
Woods uses the term robustness to refer to systems that are designed to effectively handle known failure modes.
Resilience, on the other hand, describes how well the system can handle troubles that were not foreseeable by the designer. You can think of robustness as being able to deal well with known unknowns, and resilience as being able to deal well with unknown unknowns.
- Four concepts for resilience and the implications for the future of resilience engineering by Woods discusses four different common usages of the term resilience. In particular, he describes why he considers robustness to be a different concept.
- Resilience is a verb is another very readable paper on how Woods defines resilience.
Resilience engineering as a field emerged from the safety science community. That's why you'll often see examples from aviation and medicine, as well as other safety critical areas like maritime, space flight, nuclear power, and rail.
Because of this history, the earlier papers that we associate with resilience engineering are reactions to previous ways of thinking about accidents in particular and safety in general.
Note that traditional approaches to safety often focus on minimizing variance associated with humans doing work, using techniques such as documented procedures and enforcement mechanisms for deviating from them.
The "new look" or "new view" refers to a change in perspective on how accidents happen, which focuses on understanding how actions taken by actors involved in the incident were rational, given what information those actors had at the time that events were unfolding.
Johan Bergström of Lund University has three excellent short (<10 minute) videos:
- Was it technical failure or human error?
- Three analytical traps in accident investigation
- Two views on human error
Two great introductory papers (alas, both paywalled) are:
- Reconstructing human contributions to accidents: the new view on error and performance by Dekker
- The error of counting errors by Robert Wears
A great book on putting these ideas into practice in incident investigations is:
Safety-II is a perspective on the role that humans play in safety-critical systems, proposed by Erik Hollnagel. In the Safety-II perspective, it is the everyday, normal work of the humans in the system that create the safety, as opposed to the errors of humans that erode it.
- From Safety-I to Safety-II: A White Paper by Hollnagel is a very readable introduction to Safety-II concepts.
- Why do things go right? by Dekker on the Safety Differently website is another good article.
Ever wonder why resilience engineering advocates natter on about "no root cause?"
A recurring theme in resilience engineering is about reasoning holistically about systems, as opposed to breaking things up into components and reasoning about components separately. This perspective is known as systems thinking, which is a school of thought that has been influential in the resilience engineering community.
When you view the world as a system, the idea of cause becomes meaningless, because there's no way to isolate an individual cause. Instead, the world is a tangled web of influences.
You'll often hear the phrase socio-technical system. This language emphasizes that systems should be thought of as encompassing both humans and technologies, as opposed to thinking about technological aspects in isolation.
- How complex systems fail by Richard I. Cook is a great starting point. It's a short paper and very easy to read.
- Drift into failure by Sidney Dekker is a book written for a lay audience, so it is also very readable. Dekker draws heavily from systems thinking to propose a theory about how complex systems can evolve into unsafe states.
The systems we are interested in often involve a collection of people working together in some way to achieve a task. One particularly relevant example involves a collection of engineers working together to troubleshoot and repair a system during an ongoing incident.
- Common Ground and Coordination in Joint Activity is an oft-cited paper on what is required for people to effectively coordinate when working on tasks together.
One thing we software folk do have in common with the safety-critical world is the increased adoption of automation. Automation introduces challenges, and the nature of these challenges is a topic of many resilience engineering papers.
You might hear the phrase joint cognitive system in the context of automation. This terms refers to systems that do cognitive work that are made up of a combination of humans and software. There is an entire research discipline that studies joint cognitive systems called cognitive systems engineering, initially developed by David Woods and Erik Hollnagel, both of whom would both later go on to play a significant role in developing the field of resilience engineering.
Because resilience engineering researchers like Woods and Hollnagel have their roots in cognitive systems engineering, and because of the ever-increasing use of software automation in society, this community is very concerned about the potential brittleness associated with poor use of automation.
-
Ironies of automation by Lisanne Bainbridge is a classic paper on the problems that automation can introduce. The paper was originally written in 1983, and continues to be widely cited.
-
Ten challenges for making automation a team player by Klein et al. is a more recent paper that outlines the requirements for automation to be genuinely effective in socio-technical systems. This work draws heavily from the theme of coordination discussed earlier.
The late Jens Rasmussen is an enormously influential figure in the resilience engineering community.
- Risk management in a dynamic society: a modelling problem, published in 1997, is one of Rasmussen's most famous papers, which introduces Rasmussen's dynamic safety model.
In this widely cited paper, Rasmussen advocates for a cross-disciplinary, systems-based approach to thinking about how accidents occur. He argues that accidents occur because the system migrates across a dangerous boundary, and this migration occurs during the course of normal work.
Here is a depiction of the model from that paper:
We've already referenced several papers authored or co-authored by David Woods. Woods is a force of nature in the field of resilience engineering, having played a key role in creating the field itself. Woods is incredibly prolific, and has introduced a wide variety of concepts related to resilience engineering.
Woods is interested in resilience engineering principles that apply across an enormous range of different types of systems: whether we're talking about the organs in a biological organism up to organizations like NASA.
Because he's interested in general principles, many of his papers are written at a very abstract level, where he discusses generic concepts such as units of adaptive behavior or saturation.
David Woods uses the metaphor of a system moving within a boundary in his writings on resilience engineering, but in a slightly different way than Rasmussen.
Woods sees the boundary as a competence envelope. There are two different regimes of system behavior: far from the boundary and near the boundary.
When a system is far from the boundary, the system (and its environment) behave as expected. By contrast, when a system grows near to the boundary, surprises happen. Woods uses the metaphor of dragons to capture the surprises that occur when a system moves near the boundary, and how the system's model of the world is violated when it enters this regime.
It is how units within a system adapt when the system moves near the boundary, how these units deal with the dragons, that is one of the prime concerns of Woods.
Woods's Essentials of Resilience, revisited discusses behavior at the boundary, although it doesn't use the dragon metaphor.
Woods's idea of the adaptive universe is characterized by three properties:
- Resources are finite
- Surprise is fundamental
- Change never stops
I haven't found a good introductory paper for the adaptive universe, as it encompasses an enormous number of topics, including the topic of dragons at the boundaries that we discussed earlier.
I recommend watching Woods's Resilience Engineering short course, which covers this topic. I've written my own notes on the short course, which you might find useful. In particular, you might be interested in my summary notes.
Woods introduced the theory of graceful extensibility to capture how successful systems adapt effectively to surprise. The most relevant paper here is: