"Residues" & "The Architect’s Paradox" • Barry O'Reilly & Jacqui Read • GOTO 2025

YouTube Video

Summary

This YouTube transcript is a conversation between Jacqui Read and Barry O’Reilly from the “GOTO Book Club”. The discussion centers around Barry O’Reilly’s books, particularly “Residues: Time, Uncertainty, and Change in Software Architecture” and his new book “The Architect’s Paradox: Uncertainty and the Philosophy of Software Architecture”.

Barry starts by sharing his background as a mathematician turned software developer and architect, including his time at Microsoft where he was responsible for architect education. He recounts his realization that he couldn’t articulate how he actually did architecture, leading him to explore the common traits of successful architects. He identified the ability to cope with uncertainty as the key differentiator. Architects are needed when requirements are vague and the future is unclear. He defines architecture as “decision-making in the face of ignorance”. He emphasizes that software architecture is inherently uncertain because business contexts are constantly changing, unlike programming which is more about correctness against a specification.

Barry discusses the historical influence of traditional engineering on software engineering, stemming from the 1968 NATO conference. Software engineering, in its early days, attempted to emulate the certainty and predictability of other engineering disciplines, aiming to “kill uncertainty”. This approach, rooted in philosophical ideas seeking certainty, led to tools and methodologies that treated uncertainty as an enemy. He argues that the Agile movement was a rebellion against this process-oriented approach, but it didn’t fully address the underlying conceptual framework still borrowed from engineering. His books aim to challenge this conceptual thinking and advocate for a new perspective.

The conversation then dives into “Residuality”. Barry explains it as a concept centered around embracing uncertainty rather than fighting it. He summarizes the core idea: “a random simulation of stress is more effective than gathering requirements.” The process involves:

1. Starting with a naive understanding of the business and a basic architecture.
2. Applying random stressors – hypothetical changes in the business environment (competitor actions, economic shifts, societal changes, even absurd scenarios).
3. Observing how these stressors cause the conceptual model and architecture to collapse.
4. Identifying the residue – what remains functional and what breaks.
5. Developing deltas – small architectural changes to make the collapse more acceptable.
6. Treating these residues as building blocks to optimize for survival under unforeseen circumstances, rather than optimizing for a fixed specification.

He emphasizes that Residuality shifts the focus from correctness (like in programming) to criticality – the ability of the architecture to survive unknown future events. This approach draws inspiration from complexity sciences and philosophical schools of thought different from those traditionally influencing computer science. Barry argues that Residuality helps mitigate common architectural failures related to performance, change, security, and scale by proactively building in resilience.

To illustrate stressors, Barry shares humorous and thought-provoking examples: bacon in electric car chargers, ice cream leading to a diabetes surge in the NHS, and garden sheds as medical facilities. He recounts an anecdote about his professor initially being skeptical but then realizing the power of even seemingly absurd stressors to reveal architectural weaknesses. He highlights that the process, while imaginative, is underpinned by mathematics and provides a structured way to deal with uncertainty.

Finally, they discuss “The Architect’s Paradox”. Barry explains that the paradox is that software engineering relies on logical thinking, which is essential for computation, but this same logical thinking is often inadequate for dealing with the messy, uncertain nature of business contexts. He argues that software engineering inherits a default philosophy, traceable back to figures like Plato, Wittgenstein, and Turing, which emphasizes logic and certainty. His book encourages architects to examine these inherited philosophical assumptions and consider alternative perspectives to better understand and address uncertainty in their work. He suggests that by questioning fundamental beliefs (like the necessity of interfaces everywhere), architects can unlock new ways of thinking and improve their approach to building resilient systems. He emphasizes that understanding the philosophical underpinnings of software engineering can help explain why architectures often fail and how to move beyond these limitations.

Barry concludes by mentioning his active presence on LinkedIn and BlueSky, conferences, and Leanpub where his books and academic articles are available.

Accuracy

The information presented in the transcript is generally accurate in the context of current discussions and evolving thinking within the software architecture and software engineering fields. Here’s a breakdown:

• Uncertainty as a Core Challenge: The emphasis on uncertainty as a central challenge in software architecture is widely accepted and a key driver behind agile methodologies, DevOps, and resilience engineering. This is not a novel concept, but Barry effectively articulates its importance and its often underestimated role.
• Critique of Traditional Engineering Paradigm: The critique of software engineering’s historical attempt to emulate traditional engineering and impose certainty is a valid and recurring viewpoint. The limitations of plan-driven, waterfall-style approaches in the face of changing requirements are well-documented. The comparison to the 1968 NATO conference and its influence is historically accurate and a common reference point in discussions about the evolution of software engineering.
• Agile and Conceptual Thinking: The point that Agile, while addressing process, may not have fundamentally shifted conceptual thinking is a more nuanced argument but has merit. While Agile embraces change, the underlying mental models architects use might still be influenced by deterministic and reductionist thinking patterns.
• Residuality as a Novel Approach: Residuality is presented as Barry O’Reilly’s specific research and contribution. While the term and specific methodology might be novel, the underlying principles of stress testing, scenario planning, and building resilience are established practices in system design and risk management, albeit often applied in different ways. The claim of mathematical proof for its effectiveness would require further investigation into his academic publications, but the concept of formalizing resilience considerations is a valuable direction.
• Philosophy and Software Architecture: The connection between philosophy and software architecture is less mainstream but gaining traction. The idea that our thinking is shaped by underlying philosophical assumptions is a valid point, especially in fields like computer science which have strong ties to logic and mathematics. Exploring philosophical perspectives to challenge conventional software engineering wisdom is a legitimate and intellectually stimulating endeavor.

Points to Consider for Nuance/Further Scrutiny:

• Oversimplification of “Engineering Paradigm”: While the critique of a purely deterministic, certainty-seeking “engineering paradigm” in software is valid, it’s important not to oversimplify or caricature traditional engineering. Many engineering disciplines also deal with uncertainty and probabilistic models, especially in complex systems.
• Novelty of Residuality: While presented as a new approach, Residuality shares similarities with scenario planning, fault injection, chaos engineering, and resilience engineering practices. It’s important to understand the specific differentiators and contributions of Residuality compared to these existing approaches.
• Mathematical Proof: The claim of mathematical proof of Residuality’s effectiveness needs further examination. The nature and scope of this proof and its practical applicability would be important to understand.
• Generalizability of Stressors: The effectiveness of “random” stressors depends on the skill and experience of those generating them. While the examples are illustrative, in practice, a more structured approach to stressor generation might be needed to ensure comprehensive coverage of potential risks.

Overall, the transcript presents a thought-provoking and largely accurate perspective on the challenges of software architecture in the face of uncertainty and proposes an interesting approach (Residuality) and a valuable direction for rethinking software engineering through a philosophical lens.

Resources

Here are 5 resources that would be helpful to learn more about the subjects presented in the transcript:

1. “Residues: Time, Uncertainty, and Change in Software Architecture” by Barry O’Reilly (Leanpub): This is the most direct resource to understand the “Residuality” concept in detail. The book provides a more in-depth explanation of the methodology, its theoretical underpinnings, and practical applications. As mentioned in the transcript, it’s a more accessible version of his academic work.

   • Why it’s relevant: Directly addresses the core concept of Residuality discussed in the transcript.

2. “The Architect’s Paradox: Uncertainty and the Philosophy of Software Architecture” by Barry O’Reilly (Leanpub): This book delves into the philosophical foundations of software architecture and the challenges of dealing with uncertainty. It explores the historical and philosophical roots of our thinking in software engineering and proposes new perspectives.

   • Why it’s relevant: Explores the philosophical aspects of software architecture and uncertainty, as discussed in the latter part of the interview.

3. “Thinking in Systems: A Primer” by Donella H. Meadows: This book provides a foundational understanding of systems thinking and complexity, which are relevant to Barry O’Reilly’s approach. It introduces concepts like feedback loops, system dynamics, and resilience, which are crucial for understanding and managing uncertainty in complex environments like software systems and businesses.

   • Why it’s relevant: Provides a broader context for understanding complex systems and uncertainty, concepts central to Residuality and “The Architect’s Paradox”.

4. “Anti-Fragile: Things That Gain from Disorder” by Nassim Nicholas Taleb: Barry O’Reilly mentions Nassim Taleb’s work as an inspiration for Residuality. “Anti-Fragile” explores the concept of systems that not only withstand stress but actually benefit from it. Understanding anti-fragility can provide a deeper understanding of the goals of Residuality – building systems that thrive in uncertain and changing environments.

   • Why it’s relevant: Provides the philosophical and theoretical background from Nassim Taleb’s work that influenced Barry O’Reilly’s thinking, particularly the concepts of stress and resilience.

5. “Building Evolutionary Architectures: Tradeoffs in Distributed Systems” by Neal Ford, Rebecca Parsons, and Patrick Kua: This book, while not directly about philosophy, provides practical guidance on building software architectures that can evolve and adapt to change and uncertainty. It explores architectural patterns and practices that promote adaptability, resilience, and evolutionary design, aligning with the practical goals of Residuality.

   • Why it’s relevant: Offers a more practical, engineering-focused perspective on building adaptable and resilient architectures, complementing the theoretical and philosophical aspects discussed by Barry O’Reilly and providing actionable strategies.