Continuous Design

Reading notes: Modern Software Engineering - Doing What Really Works to Build Better Software Faster

David Farley is one of the author of the 2010 classic book Continuous Delivery: Reliable Software Releases Through Build, Test and Deployment Automation, but here we are talking about his last work which summarize the fundamentals principles and ideas behind what is currently considered as the best practices.

Better than that, the book also talk about how we could evaluate the practices and technologies to come.

This book puts the engineering back into software engineering.

It presents 5 behaviors and 5 principles to improve our efficiency and the quality of our work.
None of them are related to languages or technology, on the contrary, they are agnostic but not without impact.

I think this book is a perfect summary of all modern practices. It fosters pragmatic data driven choices and fundamental behaviors of our profession. I often refer to it when in doubt, so I hope you’ll learn something too.

Modern Software Engineering is not what we call Software Engineering

Software Development is not Engineering

Naming things is hard. Even our profession is badly named. David Farley starts by talking of our history: the “Software Engineering” term was coined at the end of the 1960s and it was the reason to create Waterfall project management. So this name is now devaluated.

Craft is not enough

That is why the Software Craftsmanship movement was created. It introduced some very valuable ideas: importance of technical skills, creativity during software development, innovation and apprentice schemes to learn.
Craft is not enough because of its lack of scaling: apprenticeship is long and we don’t know how to measure progress.
Craft misses repeatability in process, tools and culture. We need to apply the scientific method to act like other engineering disciplines.

What is Engineering?

David Farley introduces his definition of Engineering:

Engineering is the application of an empirical, scientific approach to finding efficient, economic solutions to practical problems.

He claims that what we often call Software Engineering is not even Engineering.

First, we rarely measure things. Arguments taking account economy, efficiency and all trade-offs are not in our culture. Instead we focus on tools and technological fads instead of design and principles.

Second, we are not creating a material product.
This a fundamental difference: in contrast with other engineering discipline, the model we build and test is exactly our product.
We are not an industry centered on production: building the software is already easy to automate.
Software Engineering is about Design, not production.

Third, code is not our only outcome. The goal is to solve problems.
Software development is all about discovery and learning, in code of course, but also in processes, tools and culture.

Engineering is a method to go faster and create products of better quality.

Reclaiming “Software Engineering”

That is why David Farley repeats:

This book puts the engineering back into software engineering.

In Software Engineering we need to measure things. Quoting the Accelerate book, he explains the 4 measures in software industry:

• Stability
  • Change Failure Rate: the rate at which a change introduces a defect
  • Recovery Failure Time: how long to recover from a failure
• Throughput
  • Lead Time: time between idea and production release (efficiency)
  • Frequency of deployments in production (speed)

These measures helps us to evaluate our process, organization, culture and technology.

It also goes on to dispel a commonly held belief that “you can have either speed or quality but not both.”
This is simply not true.
Speed and quality are clearly correlated in the data from [Accelerate book]. The route to speed is high-quality software, the route to high-quality software is speed of feedback, and the route to both is great engineering.

The Foundations of Software Engineering Discipline

First, we must become experts at learning. We are working on design and need creativity, exploration and discovery.

Second, we face difficulties because of complexity. Information systems struggle with coupling, concurrency, large teams, organizations and communication.
We need to become expert at managing complexity.

5 Behaviors to Optimize for Learning

There are five interlinked behaviors that create a framework to improve Software Engineering and become a real discipline. It’s hard to apply one of them without others, this is really about mindset, discipline and daily practices.

Working Iteratively

Working iteratively means learning, reacting and adapting.

It’s the root of TDD: the red - green - refactor cycle.

It impacts our process, and the organization where we work. The goal is to try ideas, and apply scientific method to reduce uncertainty and gain more confidence.

• working step by step on smaller batches
• improve the code and design
• decrease the cost of change of the software (and maintain it low)
• Instead of long planning, we set goals and, as long as we can measure our progression, we can check that each step brings us closer to them.

So, working iteratively is related to feedback.

Feedback

Feedback allows us to establish a source of evidence for our decisions.

Feedback is what validate our choices and ideas, it is important to adapt our processes and improve our outcomes:

• Code: using TDD (or at least a Test-First) and make smaller steps to produce a better designed solution. Testing is the main feedback for code.
• Continuous Integration: use automated pipelines to test and very regularly (several times a day) merge our code in a main branch. At each moment we need to know the quality of our code and ensure the system is efficiently releasable at any moment, not only when a feature is over.
• Design: iteration and feedback drives our design to be highly modular, with the right abstraction level ad appropriate coupling. These qualities are also (spoiler alert 😉) what helps us to manage complexity (see below ⬇️).
• Architecture: whether we choose monolithic approach or more independent modules (microservices) feedback means continuously check performance, deployability and testability.
• Product Design: we need to use customer feedback to test our ideas, and validate the product we build
• Organization and culture: agile approaches are about feedback on organizations, to improve teams, actions, and improve their situations. Stability and throughput are good indicators to understand team issues

Incrementalism

Working incrementally is complementary of working iterativelly:

Source: https://agilenotion.com/agile-categoriesiterative-incremental-evolutionary/

In a modular system, working incrementally means:

• releasing system step by step and bringing value sooner
• components can be substituted gradually to evolve (and add complexity)
• organizing the work to make it an accretion of our knowledge and understanding of the problem

Complex systems are built this way, upon exploration and experimentation; they are not fully formed in a first draft.

So we need to design our system to be modular (and manage complexity, see below):

• code must be organized to make change easier
• at each step, we must do the minimum (YAGNI) to fulfill requirements while keeping a maximum of options open to make change in the future (and stay incremental)
• change code and our mind as our understanding of the problem deepens (refactoring, improvement of design and code structure…)

Empiricism

In science, empiricism means that hypotheses and theories must be tested against observations and experimentations: we elaborate models that fit our knowledge and try to prove they are not wrong.

In the same way, in software engineering, we should validate that each choice in code and in product has the expected impact on the metrics our solution was designed to improve.

It’s important to always confront our guesses against reality and measurements. So it’s related with experimentation.

Being experimental

Experimentation is the root of all engineering, by using models and simulations.

An experimentation is described by 4 components:

• feedback: how we will collect result of experimentation and use it
• hypothesis: before the experiment, we must be explicit about what will be evaluated
• measurements: what define success and failure of the experimentation, in other words which indicators are measured. Beware of fallacious indicators (like code coverage) which prove nothing
• control the variables: experimentation must eliminate noise and only keep under control what brings information. It must be repeatable and reliable. In Software Engineering, it’s all about infrastructure as code and determinism, used in testing or Continuous Deployment

Experimentation is iterativelly improving software.

5 Tools to Optimize for Managing Complexity

Modularity

One way to decrease complexity is to reduce the problem to small pieces, each one understandable without any context.

A module creates boundaries between outside and inside, exposing only the minimum interface.

Each independent unit is designed to gain flexibility, to be easier to work on, to test, to modify and to deploy.

With tests, we gain more feedback and each piece can be measured by controlling other variables (like engineers building an aircraft: each part is tested separately).

Cohesion

Modularity and cohesion are the fundamentals of software design.

Pull the things that are unrelated further apart, and put the things that are related closer together.

Kent Beck

Cohesion drives a lot of choice in software design and code structure. TDD is a tool helping software engineers to make choice about cohesion with tests giving feedback about what is related or not.

Sometimes writing cohesive software makes us write more code, and it’s OK because:

The primary goal of code is to communicate ideas to humans!

We don’t want to optimize to type less code, but to communicate better.

Separation of concerns

Separation of concerns is the root principle behind modularity and cohesion. Each unit of code (class, method, function, module…) should do one thing.

It means separating:

• different behaviors
• different business concepts
• code at different abstraction level: setting apart essential complexity (complexity related ton the problem we are solving) and accidental complexity (complexity related to computer architecture like display, storage, clustering, security…).

The main tool to isolate code is Dependency Injection which allows us to create abstraction and interfaces between code. For instance, the Ports and Adapter architecture patterns is totally based on it.

Separation of concerns is what keeps our code clean, focused, composible, flexible, efficient, scalable and open to change. It increases determinism and, by that, code testability.

Information Hiding and Abstraction

Each unit of code (module, class, method, function) should hide its implementation details and what data it uses (and how).
Abstraction is the creation of seams in the code to hide these informations.

• Tests should rely on abstraction to make them reliable and avoid to break when implementation details change.
• We need to adjust abstraction to express essential complexity. Domain concepts are natural boundaries between abstractions
• What is not related to our domain should be hidden from domain code. For instance, accidental complexity, or third-party systems and libraries

Managing Coupling

Coupling represent any interdependence between components of a system.
It has a cost, but in the same time it’s something we can’t totally eliminate.

High coupling means any change has a lot of impacts on the codebase. Loose coupling is a quality which improves efficiency, scaling and, as always, testability.

The real reason why attributes of our systems like modularity and cohesion and techniques like abstraction and separation of concerns matter is because they help us to reduce the coupling in our systems.

• Continuous Integration can check that coupled elements still work after a change
• Microservices is an architecture pattern, but also an organization pattern to create decoupled and deployable units to scale better
• Loose coupling is about finding interfaces (contracts) between components. It’s complementary to Separation of Concerns.
• Asynchronous communication drives decoupled designs
• Decoupling may add code to add readability. Again, readability is a fundamental property of good code, so it’s not a problem.

Tools to Support Engineering in Software

When we write software, there are 4 difficulties to overcome:

• solving the right problem
• ensure that the solution works as desired
• create work of high quality
• working efficiently

The Software Engineering Discipline must be agnostic of any technology or mechanism. Its principles are generally applicable even for processes and technologies that doesn’t exist yet.

Whatever methodology we choose, we must be sure it helps us to improve our desired outcomes:

• Testing: to give us feedback and drive design
• Deployability: to be able to deploy our system in some fast, secure, resilient and efficient way
• Speed: which means achieve high stability and throughput
• Control the variables: to gain confidence
• Continuous Delivery: high automation level

But Software Engineering is a human process, organizations we work in are impacting and impacted by software production.
Managing complexity also apply to them.

We need to adopt a model BAPO, which means that first comes the Business Strategy, then we find the System Architecture and determine the Processes to achieve it, and only then we can create the needed Organization of teams, departments, responsibilities and so on.

Conclusion

The author conclude by saying our Discipline is only at its beginning, but by applying these fundamental principles we would already build better software faster.