Are Software Developers Engineers Or Craftsmen?

YouTube Video

Summary

This transcript captures a conversation discussing the application of engineering principles to software development. The speaker argues that software development should be treated as engineering, not just craft, to improve its effectiveness and reliability.

The conversation begins by referencing a prior discussion where the speaker expressed being “on the fence” about whether software development is engineering. He clarifies his position: it’s not inherently engineering, but adopting an engineering mindset significantly improves outcomes. He uses the analogy of Mars missions, highlighting their high failure rate due to the extreme difficulty and need for perfection. This serves to illustrate that even in seemingly impossible domains, a rigorous engineering approach focused on anticipating and mitigating potential failures is crucial. He then references Margaret Hamilton and her work on the Apollo program’s flight control systems, emphasizing the “man-rated” concept and the critical importance of system reliability when lives are at stake.

A key argument is that software development has an advantage over physical engineering: complete control and the ability to design for testability and determinism. The speaker points out the ease of experimentation, iteration, and even deletion in code compared to physical materials. This leads to a discussion on rapid feedback loops, exemplified by test-driven development (TDD) and the ability to quickly discard and rewrite code that isn’t working. This iterative approach is contrasted with a linear, waterfall-style development.

The conversation touches upon “evolutionary architecture” and the idea that each line of code either expands or restricts future options. This ties into the importance of small, incremental changes and avoiding premature commitment to paths that limit future flexibility.

The discussion then moves to practical engineering practices in software, specifically highlighting Continuous Integration and Continuous Delivery (CI/CD) pipelines. The speakers acknowledge the foundational work done in this area, emphasizing the benefits of automated build pipelines, artifact traceability, and comprehensive testing at various stages. Modern deployment techniques like Canary releases and Blue/Green deployments are mentioned as further examples of engineering approaches that minimize risk and enable controlled releases.

The ease of infrastructure changes in the cloud era is contrasted with the complexities of physical infrastructure. The example of moving a science publishing site’s compute location from the UK to the US with minimal code changes in Terraform showcases this advantage.

The conversation then explores “Fitness Functions” from evolutionary architecture, advocating for codifying constraints and architectural rules within the build pipeline itself, automating checks that are often handled by slow, human-driven processes like change request boards and architecture review boards. This aims to reduce human bottlenecks and enforce architectural consistency programmatically.

The importance of testing is reiterated, acknowledging that while tests don’t guarantee defect-free software, they significantly reduce defects. The speaker then humorously mentions his foray into game design research for a simulator project, learning about vectorization and GPU optimization, and discovering unexpected simplicity in the core game loop. He also discusses Behavior Trees as a superior alternative to Finite State Machines for modeling complex behavior, particularly highlighting their testability.

Finally, the conversation concludes by expressing interest in further discussions on related topics like directed graphs versus meshes in interconnected systems, and thanking the audience for their time and engagement. The overall tone is enthusiastic and promotes a more rigorous, engineering-driven approach to software development, leveraging the unique advantages of the digital domain.

Accuracy

The information provided in the transcript is generally accurate and aligns with established knowledge in software engineering and related fields. Let’s break down some key points:

• Mars Mission Failure Rate: While stating a “majority” of Mars missions failed might be a slight exaggeration depending on the specific timeframe and definition of “mission” (orbital vs. landing, etc.), it’s true that Mars exploration has historically been very challenging with a high failure rate, especially for landing attempts. A quick search confirms that historically, a significant portion of Mars missions have indeed failed, particularly during entry, descent, and landing. The speaker’s point about the difficulty and need for precision is accurate.
• Margaret Hamilton and Apollo Program: The reference to Margaret Hamilton and her crucial role in developing the Apollo program’s flight control software is accurate. She is a highly respected figure in software engineering, and her emphasis on “man-rated” systems and rigorous testing is well-documented and a cornerstone of safety-critical software development.
• CI/CD, Evolutionary Architecture, Fitness Functions: These are all well-established concepts and practices within modern software engineering and DevOps. The description of CI/CD pipelines, their benefits, and deployment techniques like Canary and Blue/Green deployments are accurate and reflect industry best practices. Evolutionary architecture and fitness functions are also recognized architectural patterns for building adaptable and resilient systems.
• Test-Driven Development (TDD) and Testing in General: The benefits of testing and the principles of TDD are accurately presented. While tests cannot eliminate all defects, they are widely accepted as a crucial tool for improving software quality and reducing bugs.
• Behavior Trees and Finite State Machines: In the context of game development and AI, the comparison between Behavior Trees and Finite State Machines is accurate. Behavior Trees are indeed often considered more modular, scalable, and testable for complex AI behaviors compared to traditional Finite State Machines, which can become unwieldy and difficult to manage as complexity grows.
• Cloud Infrastructure and Terraform: The example of using Terraform to quickly redeploy infrastructure across regions with minimal code changes is a realistic depiction of the power and flexibility of Infrastructure as Code (IaC) and cloud platforms.

Overall Accuracy Assessment: The transcript presents a generally accurate and insightful overview of software engineering principles and practices. Any minor generalizations (like the “majority” of Mars missions) serve to illustrate a point effectively and do not fundamentally detract from the accuracy of the core message. The discussion aligns well with current industry best practices and established knowledge in software development.

Resources

Here are 5 relevant resources to learn more about the subjects discussed in the transcript:

1. “Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation” by Jez Humble and David Farley: This book is a seminal work on Continuous Delivery and CI/CD pipelines. It provides a comprehensive guide to automating the software release process for faster, more reliable deployments, covering many of the topics mentioned in the transcript related to build pipelines, testing, and deployment strategies.

2. “Building Evolutionary Architectures: Tradeoffs in Practice” by Neal Ford, Rebecca Parsons, and Patrick Kua: This book delves deeply into the concept of evolutionary architecture and fitness functions. It explores how to design systems that can adapt to change over time, covering topics like incremental change, emergent design, and the use of fitness functions to guide architectural evolution, directly addressing the points raised in the transcript about architectural flexibility and automated governance.

3. “Refactoring: Improving the Design of Existing Code” by Martin Fowler: While not explicitly mentioned, the concept of refactoring is crucial for iterative development and maintaining code quality, which underpins many of the engineering practices discussed. This book is a classic guide to refactoring techniques, helping developers improve code structure and design in small, safe steps, aligning with the idea of incremental changes and managing code evolution.

4. “Game Programming Patterns” by Robert Nystrom: For those interested in the game development techniques mentioned, particularly Behavior Trees and Finite State Machines, this online book (available for free) is an excellent resource. It explores various design patterns used in game development, including chapters on State Machines and Behavior Trees, explaining their implementation and use cases in detail, providing practical context to the speaker’s insights. (https://gameprogrammingpatterns.com/)

5. “Site Reliability Engineering: How Google Runs Production Systems” by Betsy Beyer, Chris Jones, Jennifer Petoff, and Niall Richard Murphy: This book, written by Google SREs, provides a comprehensive overview of Site Reliability Engineering principles and practices. It covers many aspects of running reliable and scalable systems, including automation, monitoring, incident response, and performance optimization, all within an engineering-focused framework relevant to the discussion on treating software development as engineering. (https://sre.google/books/)