
In recent years, I've developed distributed systems using domain-driven design, event storming, continuous delivery, and test-first development practices. In my experience, systems designed using these techniques are more likely to provide real value to customers, are easier to maintain, and are more fun to work on! There's no panacea against accidental complexity, though, and it takes experience, skill, and a solid team to create a successful system. In this post, I articulate some of the lessons I've learned and the design concepts I've found to be essential when building evolvable, distributed systems. Upcoming posts in this series will dig deeper into tactical patterns and specific component architecture concepts.

## Don't Build a Distributed System
First-and-foremost, do not distribute across a network if you do not need to! This advice is repeated over-and-over by those that went through failed microservice transformations or startups that attempted to begin with a microservice architecture. As software engineers, one of our primary goals is to reduce unnecessary complexity. Distributing pieces of your system across a network increase complexity, so don't do it without a good reason. One such good reason is: a *tangible* need to scale because specific bottlenecks are or will be preventing the system from providing real value. An example of a bad reason is: we're going to capture so much market share after launch that we need to be able to scale everything right off-the-bat!

Hint: You can still design a well-architected modular system without incurring the costs of distributed messaging up-front.

### MVPs
If you're building an MVP and have yet to prove product-market fit, don't build a distributed platform. Build the MVP well enough to onboard a small batch of initial customers and focus on delighting them. Once you've proven the product is valuable, then evolve your architecture towards the scalability and growth requirements. That said, there may be very good reasons to distribute some parts of your MVP if it is necessary to prove market fit (e.g. scalability, reliability, compliance, geography, isolating 3rd party integrations, etc.), but the goal should be to keep it as simple as possible while still viable. I think of an MVP as an experiment to prove the product solution against the current market forces, and you need to uncover the results as soon as possible to avoid waste. There are a couple approaches to building an MVP that I've seen work well.

One approach is to build the MVP as quickly and cheaply as possible while ignoring architectural quality. This really only works if the app you are building is simple enough because, as we all know, development will inevitably slow down over time with poor architecture. But this can work well if you expect to prove market fit quickly. I've seen this frequently when an entrepreneur wants to try out an idea and outsources development to a contracting firm to build the MVP. The key with this approach is that after you've proven the product's viability, you'll likely need to invest in a major architectural overhaul with an in-house development team that designs it with quality in mind at an appropriate level of scalability. This transition can be extremely difficult because just as the product is gaining traction, you are spending resources on rebuilding instead of providing additional user value.

Another approach is to build the MVP quickly, but with architectural quality in mind from the start using an in-house development team. This may require more initial investment, but provides a more stable product, smoother delivery cadence, and a healthier team in the long run. An experienced product and engineering team will help reduce risk as long as they are laser-focused on early engagement and rapid feedback with initial customers throughout MVP development. The system design concepts I describe here can be applied to an MVP built in this way and will help pave the way for a maintainable, scalable, and evolvable architecture.

## Modularize First, Then Distribute
Whether you're building an MVP or redesigning components of an existing system, if you create modular components with single responsibilities in your system, you do not need to separate those components across the network initially. Each component should have high cohesion within itself and low coupling to other modules with a clean interface boundary. It is cheaper and easier to modify the boundaries between components that are not distributed. Over time these boundaries should stabilize; then, when the need arises an individual component can be extracted into its own service. If designed well, the business logic should not need to change at all and the application logic should only need minor changes. The infrastructure should be the only aspect that needs significant rework. This ensures we're only adding complexity when it is needed.

Identifying the most useful component boundaries is hard and depends on many factors. Expect those boundaries to change over time as the team gains experience through experimentation.
## Design Concepts
Below are some of the most important design concepts I use when creating a modular system. Open and consistent communication with stakeholders is necessary to build a clear understanding of the business domain, which is essential. Markets change over time, so the problems the system addresses should be refined over time. And the solution will change over time as everyone involved learns new things. So we must embrace these changes and expect the system boundaries and models to evolve. Continually practicing techniques like event storming or event modeling throughout a product's lifecycle can help normalize and capture these changes.
### Clean Architecture
Robert Martin (Uncle Bob) coined clean architecture from aspects of hexagonal and onion architecture. It is the idea that our code should be organized in layers with the dependencies pointing in one direction: towards the business logic in the center. This approach builds on the SOLID design principles. If you're unfamiliar, see [Clean Architecture and SOLID Cheat Sheets](https://www.bennyjohns.com/posts/20201101-clean-architecture-and-solid-cheat-sheets)  for a quick overview and visualization.

Our core business logic (i.e. domain logic) at the center describes the valuable purpose of the component, boiled down into clear and concise language with minimal dependencies. The application logic is the next layer out from the center that wires the entire application together and should only directly depend on the core business logic. The outer infrastructure layer includes all additional concerns around state management, persistence, communication, and presentation. This outer layer tends to be where most engineering time is spent, even though the real business value is encoded in the business logic. I use this style of architecture within individual services to clearly separate the business logic from all the necessary complexity to make it run.

### Ubiquitous Language
Ubiquitous language is a fancy domain-driven design (DDD) term that simply means everyone should agree on the language used to describe the business logic, and that language should be used everywhere, including in code. As software engineers, we should name our variables, classes, functions, services, database tables, etc. using the same language that our product team members, customers, and business leaders use. We should also strive to design our data models to reflect the real-world concepts. This significantly reduces misunderstandings, helps avoid over-abstraction, speeds up new team member onboarding, and increases code maintainability. This means we need to communicate with others and decide on the proper terminology and definitions, which is part of our job as software engineers!

### Bounded Contexts
A bounded context is another DDD term that describes where the boundary is drawn around a domain concept. Within a single bounded context, the same ubiquitous language and the same domain models should be used. Separate bounded contexts may (and likely should) use a different ubiquitous language and domain model. Bounded contexts define important boundaries in a system. They provide clarity of the solution and allow separate teams to work in parallel with others with minimal dependencies. For example, you might have separate Storefront, Payment, and Shipping bounded contexts for an online store.

### Aggregates
Another important DDD concept is the aggregate, which is fundamentally a collection of entities that represent part of the domain model. An aggregate can be stored, modified, and passed around according to the business rules. What's most important about an aggregate, however, is the concept that it defines a transactional boundary. Each change to an aggregate should be an atomic transaction. This also means that any business rule that requires consistency must only operate on a single aggregate. This may initially sound trivial, but it is the grounding concept that allows a distributed system to manage state reliably and consistently.

If a new business rule requires data from multiple aggregates to make a decision, there are a few options to discuss with the stakeholders:
1. If the rule is simply nice-to-have, you can make best efforts to follow it, but allow it to be violated occasionally. For example, I've taken this approach when asserting a unique name across a user's list of items. If uniqueness is only a convenience to avoid confusing one item with another and the user can simply rename an item later if there's a collision, you can simply make a best effort to enforce uniqueness when the name is applied. It's okay if a duplicate name sneaks in due to two items being created at the same time, for instance, as long as the UX tradeoff is acceptable.
2. If the rule is an invariant (i.e. requires strict consistency), you can rework your domain model to increase the size of the aggregate to include everything the rule needs. An invariant rule is one that the system must enforce at all times; violations would result in the system being inconsistent.
3. If the rule can be eventually consistent, you can define a compensating process to fix a violation. An eventually consistent rule may be violated temporarily, but will eventually be fixed.

The first "nice-to-have" option is the simplest, but comes with a UX tradeoff. If that's not possible, strive for the second "invariant" option, as long as the new model doesn't make the aggregate too large and the rework effort is not prohibitive. The third "compensating process" option can add significant complexity so should be minimized, but is usually unavoidable in complex domains. If you need to use the third option, these approaches are listed in order of increasing complexity:
1. Define a manual process to compensate for the inconsistency. This requires some sensing mechanism to determine when the rule has been violated and a documented process for fixing it. This can work if the violation only occurs occasionally and may be a good place to start, but adds busy work so should be automated when possible.
2. Write automated compensation logic. Automate the manual process in the previous approach. This works if the compensating logic is fairly simple and can be run on a schedule.
3. Implement the saga pattern in a process manager. This works well if the business rule describes a long-running process with multiple steps involved. This requires each step in the process to have a compensating action that reverses the step, to allow rolling back if any subsequent step fails. If compensating actions continue to fail, there also should be a manual escalation process.

### Commands, Queries, and Events
All messages between modules should be explicitly categorized as a command, query, or an event. This simplifies reasoning about distributed communication patterns, allows for a much clearer understanding of how messages are processed, and enables some basic guarantees around idempotency and state changes.

A command is a message that requests a component to change the state of an aggregate in some way. A query is a message that requests data from a component without changing its state. An event is an asynchronous message to subscribers indicating something happened in the system (e.g. an aggregate's state changed or a failure condition was triggered). Events are powerful tools in evolutionary architectures but there's a lot of nuance around their usage that I will discuss in upcoming posts.

Commands and queries may be synchronous or asynchronous. Events are, by definition, asynchronous. For high throughput and scalability, asynchronous messaging is desirable, but there's additional complexity involved in handling concurrent processing and failures. This complexity can be mitigated with robust messaging libraries and knowledge sharing. Synchronous messaging with request/response style communication is simple and can be a great choice for many situations, but it will eventually result in idle resources and unnecessary bottlenecks so should be avoided if high throughput is necessary.

### Team-First Architecture
If you're building a large system, use the Inverse Conway Maneuver to first organize your teams along the desired architectural boundaries. Team Topologies by Matthew Skelton and Manuel Pais describes how to structure your organization with four team types and their interactions. Most of your cross-functional teams should be focused on delivering user value directly by becoming experts in one or more domains, solving problems within their bounded contexts, and reacting to user feedback. These teams are **stream-aligned**. Three additional team types help stream-aligned teams by: providing a stable **platform** for the other teams to leverage, **enabling** other teams through knowledge transfer, and offloading cognitive load by owning a specific **complicated subsystem** leveraged by other teams.

### Think Experimentally
Over time, I've learned to see the software delivery process as a series of experiments. Every change to a production system can be thought of as a small experiment using the scientific method, for example:
1. **Observe** the current system and market landscape to identify an opportunity.
2. **Hypothesize** that a particular feature will leverage the opportunity.
3. **Predict** that customers will use the new feature.
4. **Experiment** by implementing and releasing it to your users.
5. Capture user feedback and metric **results** as input for further experimentation.

This leans in to modern continuous delivery practices and requires working iteratively (running one small experiment at a time), incrementally (by creating small, cohesive components that can be worked on in isolation from the entire system), and relentlessly seeking feedback (through metrics and user collaboration). Dave Farley's book Modern Software Engineering covers this approach along with other excellent software engineering concepts.

## Reading and Next Steps
The following books were formative to my approach designing systems:
- [Team Topologies](https://teamtopologies.com/)
- [Building Evolutionary Architectures | Thoughtworks](https://www.thoughtworks.com/insights/books/building-evolutionary-architectures)
- [Modern Software Engineering: Doing What Works to Build Better Software Faster: Farley, David](https://www.amazon.co.uk/gp/product/0137314914)
- [Implementing Domain-Driven Design 1, Vaughn, Vernon](https://www.amazon.com/Implementing-Domain-Driven-Design-Vaughn-Vernon-ebook-dp-B00BCLEBN8/dp/B00BCLEBN8)
- [Domain-Driven Design: Tackling Complexity in the Heart of Software: Eric, Evans](https://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software-ebook/dp/B00794TAUG)
- [Clean Architecture: A Craftsman's Guide to Software Structure and Design: Martin, Robert](https://www.amazon.com/Clean-Architecture-Craftsmans-Software-Structure/dp/0134494164)

Stay tuned for upcoming posts in this series where I describe essential tactical patterns and component architecture concepts for building evolvable, distributed systems.


The essential design concepts I use when developing an evolvable, distributed system.

Evolutionary Architecture and Distributed System Concepts


## Acceptance Test-Driven Development
Practicing TDD increases flow and improves design as long as we [test the behavior](/posts/20240124-tdd-and-testing-behavior) of each component instead of its implementation details. Unit testing with TDD focuses on the lower component level, but we can also apply the practice at a higher level by first writing an acceptance test for each user scenario (i.e. acceptance criteria) like this:

[![](/images/posts/ATDD-Double-Loop-TDD.png)](/images/posts/ATDD-Double-Loop-TDD.png)
Acceptance Test-Driven Development (ATDD) is also known as Specification by Example, Double-Loop TDD, Story Test Driven Development, or Behavior Driven Development, though some would argue there are differences between these terms.

## The CI Problem
### Part 1
How do we frequently integrate small, incremental changes within the inner TDD loop? Each change is only a tiny slice of behavior, so the scenario will fail until all slices are complete. The deployability of our software should depend on the success of our acceptance tests, so our pipeline will fail until the scenario is completely functional.

In the past I've done things like comment out the acceptance test I'm currently working on. But that is error-prone, looks messy in version control, and makes it more difficult for others on the team to run the tests. I've also waited until the entire scenario is complete before pushing my changes. But that is not integrating fast enough: I want to be able to safely merge my code after every short TDD cycle.

### Part 2
If we solve this particular issue, then it causes another one: merging rough and unfinished features into trunk increases the cost of accidentally enabling the feature flag and breaking production.

## Proposal
We can solve this by setting up our CI/CD pipeline to integrate aspects of feature flagging with acceptance testing. Ideally we are able to fully control everything within our test specifications in version control without switching to an external system and breaking our focus.

When writing acceptance tests, we tag the unfinished tests as "pending" and our pipeline only gates deployment by running completed tests. We can still run the pending tests locally during development. After a scenario is complete, we remove the "pending" tag and the pipeline now runs that test. Most test frameworks provide tagging functionality.

But now that we're deploying code during development that is not gated behind an acceptance test, we're introducing more risk to production. We can leverage an integration with our feature flagging system to reduce this risk by setting up pipeline rules to:
1. protect features from being enabled in production before they are ready
2. protect unfinished code from being deployed behind an enabled feature flag

Additionally, we could automatically register feature flags when a feature is complete to reduce some toil.

### New Feature Workflow
Here's a walkthrough of the workflow when building a new feature.

#### Developer Experience
1. Break a feature into user scenarios (i.e. acceptance criteria) with product stakeholders
2. Stub out each scenario as an acceptance test, tagged with the feature flag name and "pending" 
3. Write a single acceptance test
4. Use the TDD cycle to deliver incremental changes behind the feature flag and make the acceptance test pass
5. Remove the "pending" tag from the completed acceptance test
6. Demo and capture feedback about the new behavior from stakeholders
7. If the feature is not yet complete, go back to Step 3

Notice that at _any_ step in this process, you can safely merge into trunk without breaking the pipeline. Here's the relevant logic in the pipeline that happens automatically:  

#### Pipeline Logic
1. Parse acceptance test tags
2. If the feature flag doesn't exist in the feature flag system, create it
3. If the test is "pending"
   - And its associated feature flag is enabled for any production users, fail the pipeline
   - Otherwise, disable or hide the flag in the feature flag system, so it cannot be enabled for production users
4. Run acceptance tests except those that are pending

### Changing Existing Features
When changing the scenarios for an existing feature, we also now have some protection against breaking a feature that has already been released. If we need to change the behavior of a released feature, we are enforcing that it is implemented under a _new_ feature flag, so we don't break the user experience during development.

### Reducing Toil
Integrating your feature flag solution with your pipeline in this manner also opens up some additional automation opportunities. For instance, you could remove feature flags from the system when a developer removes them from the scenario tags, but only if the feature has been released for at least 2 weeks and there are no instances of it in the code. Or you could log a warning if a feature has been released for over 4 weeks without being removed from the code yet.

### Further Reading
- [Coding Is Like Cooking » Blog Archive » Outside-In development with Double Loop TDD](https://coding-is-like-cooking.info/2013/04/outside-in-development-with-double-loop-tdd/)
- [Acceptance Testing: What Are Acceptance Test and why They Matter](https://semaphoreci.com/blog/the-benefits-of-acceptance-testing)
- [Acceptance testing your Go CLI. I’m a fan of Acceptance Test Driven… | by William Martin | Medium](https://medium.com/@william.la.martin/acceptance-testing-your-go-cli-d992926bcfe5)
- [Double-Loop TDD](https://www.sammancoaching.org/learning_hours/bdd/double_loop_tdd.html)
- [Acceptance Test Driven Development (ATDD) | Agile Alliance](https://www.agilealliance.org/glossary/atdd/)
- [What Is Acceptance Test Driven Development (ATDD)](https://www.nimblework.com/agile/acceptance-test-driven-development-atdd/)
- [How to Write Automation Tests for Feature Flags with Cypress.io and Selenium | by Todd Seller | Engineers @ Optimizely | Medium](https://medium.com/engineers-optimizely/how-to-write-automation-tests-for-feature-flags-with-cypress-io-and-selenium-98493b5c46bc)


How can we continuously integrate small changes while practicing acceptance test-driven development?

Continuous Delivery with ATDD and Feature Flags


## Red, Green, Refactor
Test-Driven Development (TDD) is the practice of writing a failing unit test before writing the code to make the test pass, then cleaning up your code. This is the "red, green, refactor" cycle.
[![](/images/posts/TDD-Red-Green-Refactor.png)](/images/posts/TDD-Red-Green-Refactor.png)

It helps with clarity of thought, increases flow, and results in modular designs that are maintainable. You can probably tell I'm a fan. It's one of the practices that has helped me become a better software engineer over the years. While there are a lot of resources out there to learn about TDD, most of them gloss over an important aspect that makes it much more effective: it matters _what_ you try to test.

## Test the Behavior
For TDD to be effective it's important to test the _behavior_ of a component instead of its internal implementation details. This allows changing the component's internal logic without changing its tests. If the component's interface to other components in the system does not change and it can be used the same way as before, then the tests should not need to change either. This allows each individual component of the system to be enhanced, fixed, refactored, rebuilt using a different technology, or even completely replaced with minimal changes to the unit tests.

If the tests are tightly coupled to the internal implementation of a component, then it makes it difficult to modify the component without also modifying the tests. By focusing on only testing the behavior of a component, the unit tests should not be touched at all when changing its internal implementation. Instead, the tests become an important tool to verify that the internal change has not broken the component's behavior.

## Component Interfaces
Whenever we build a system, be it large or small, we should break it down into components that each have a unique responsibility. This modularity provides a lot of benefits like easier restructuring, simplicity, maintainability, testability, reducing the cognitive load of software engineers, and more. By identifying the components, we're also defining the boundaries between them and the interfaces they use to communicate with each other.

For each component we should strive to make it easy to change the internal implementation of the component, but we should make it a bit harder to change the interfaces between them. These interfaces are the components' API contracts. The hard part is figuring out the granularity of the components and their responsibility. This comes with experience and depends on many factors. There are many schools of thought on how to break down a system and identify its components (e.g. domain-driven design, event storming, event modeling, clean architecture, etc.), but I generally recommend designing it out visually and running through a few of the most important user scenarios. If you find the current design difficult and complicated, try a different one and keep iterating until you find one that works well.

With some practice, you'll get better at identifying reasonable component boundaries.

## Example Repository Implementation
One component boundary I find particularly useful encapsulates access to persistent data using the repository pattern. When building a feature that requires storing and retrieving data, it's usually beneficial to isolate the details of the database technology from the application logic. A repository's single responsibility is to manage the persistent data for an application (or a portion of it).

Let's build a simple repository in Go that shows the difference between the bad implementation-driven vs the good behavior-driven approaches to writing tests. We'll create a repository that saves and retrieves users. Each user only has a name, and just for simplicity we'll "persist" the data in-memory using a map, but you can imagine a DB engine, file system, or object store instead.

Both approaches are shown below:
- [Implementation-driven approach to testing](#implementation-driven-approach-to-testing) (Don't do it like this!)
- [Behavior-driven approach to testing](#behavior-driven-approach-to-testing) (Do it like this, instead!)

Here's the code from these examples if you want to compare the results side-by-side: [github.com/benjohns1/tdd-and-testing-behavior](https://github.com/benjohns1/tdd-and-testing-behavior)

### Implementation-driven approach to testing
_**Don't do it like this!**_

Here's what a user looks like:

<Code language="go" heading="File: user.go">
{`package app

type User struct {
	Name string
}
`}
</Code>

And here's a stub user repo to start with:
<Code language="go" heading="File: repo/users.go">
{`package repo

type Users struct{}
`}
</Code>

#### Save a User
Let's start with saving a user in the repo. We want a Save() method that accepts a user and returns a possible error. We'll write a test for a repo method Save() that we haven't written yet, so it will fail to compile initially.

##### Test: store a user by name to an empty map
Here's our first failing test, setup with a common table-driven test structure that I find useful:

###### Red Step 1
<Code language="go" heading="File: repo/users_test.go">
{`package repo

import (
	app "github.com/benjohns1/tdd-and-testing-behavior/implementation-driven"
	"reflect"
	"testing"
)

func TestUser_Save(t *testing.T) {
	type args struct {
		user app.User
	}
	tests := []struct {
		name    string
		repo    Users
		args    args
		want    Users
		wantErr error
	}{
		{
			name: "store a user by name to an empty map",
			args: args{
				user: app.User{Name: "Ender"},
			},
			want: Users{
				map[string]app.User{
					"Ender": {Name: "Ender"},
				},
			},
		},
	}
	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			err := tt.repo.Save(tt.args.user)
			if !reflect.DeepEqual(err, tt.wantErr) {
				t.Errorf("Save() error = %v, wantErr %v", err, tt.wantErr)
				return
			}
			if !reflect.DeepEqual(tt.repo, tt.want) {
				t.Errorf("Save() = %+v, want %+v", tt.repo, tt.want)
			}
		})
	}
}
`}
</Code>

This test will fail to compile, so let's write the code to make it pass:
###### Green Step 2
<Code language="go" heading="File: repo/users.go">
{`package repo

import app "github.com/benjohns1/tdd-and-testing-behavior/implementation-driven"

type Users struct {
	users map[string]app.User
}

func (u *Users) Save(user app.User) error {
	u.users = map[string]app.User{
		user.Name: user,
	}
	return nil
}
`}
</Code>

Alright, our test passes!

###### Refactor Step 3
The next step is to refactor our code and clean it up, but the code is so simple at this point let's move on to the next test cycle.

##### Test: store a new user by name to a map that already has a user

###### Red Step 1
<Code language="go" heading="File: repo/users_test.go (partial)">
{`// ...
		{
			name: "store a new user by name to a map that already has a user",
			repo: Users{
				map[string]app.User{
					"Ender": {Name: "Ender"},
				},
			},
			args: args{
				user: app.User{Name: "Valentine"},
			},
			want: Users{
				map[string]app.User{
					"Ender":     {Name: "Ender"},
					"Valentine": {Name: "Valentine"},
				},
			},
		},
// ...
`}
</Code>

This test will fail because of our overly-simplistic implementation from the first test, so let's fix that:
###### Green Step 2
<Code language="go" heading="File: repo/users.go (partial)">
{`func (u *Users) Save(user app.User) error {
	if u.users == nil {
		u.users = map[string]app.User{
			user.Name: user,
		}
	} else {
		u.users[user.Name] = user
	}
	return nil
}
`}
</Code>

This makes our test pass, but now we have a bit of cruft in our logic so let's simplify it:

###### Refactor Step 3
<Code language="go" heading="File: repo/users.go (partial)">
{`func (u *Users) Save(user app.User) error {
	if u.users == nil {
		u.users = make(map[string]app.User, 1)
	}
	u.users[user.Name] = user
	return nil
}
`}
</Code>
And our test is still passing!

##### Test: return an error if the user's name already exists
If a user already exists in our repo with the same name, we want an error to be returned so let's write a test for that:

###### Red Step 1
<Code language="go" heading="File: repo/users_test.go (partial)">
{`// ...
		{
			name: "return an error if the user's name already exists",
			repo: Users{
				map[string]app.User{
					"Peter": {Name: "Peter"},
				},
			},
			args: args{
				user: app.User{Name: "Peter"},
			},
			want: Users{
				map[string]app.User{
					"Peter": {Name: "Peter"},
				},
			},
			wantErr: fmt.Errorf("user name \\\"Peter\\\" already exists"),
		},
// ...
`}
</Code>

###### Green Step 2
<Code language="go" heading="File: repo/users.go (partial)">
{`func (u *Users) Save(user app.User) error {
	if _, exists := u.users[user.Name]; exists {
		return fmt.Errorf("user name \\\"Peter\\\" already exists")
	}
	if u.users == nil {
		u.users = make(map[string]app.User, 1)
	}
	u.users[user.Name] = user
	return nil
}
`}
</Code>

Notice the naive hard-coded error message? It's the simplest code that makes our test pass, but we want to clean that up in the refactor step:

###### Refactor Step 3
<Code language="go" heading="File: repo/users.go (partial)">
{`func (u *Users) Save(user app.User) error {
	if _, exists := u.users[user.Name]; exists {
		return fmt.Errorf("user name %q already exists", user.Name)
	}
	if u.users == nil {
		u.users = make(map[string]app.User, 1)
	}
	u.users[user.Name] = user
	return nil
}
`}
</Code>

Here's [the code for this approach](https://github.com/benjohns1/tdd-and-testing-behavior/tree/main/implementation-driven)

#### Is there a better way?
At this point, we've only written tests and functionality around saving a 
user. We could continue on to build out the functionality to retrieve users taking
 the same approach. It's great that we're using the basic TDD red, green, refactor cycle, but 
first think about what would happen if we needed to change the implementation. Could we add a user ID field index? Or upgrade to a persistent DB instead of in-memory? In either of these scenarios we'd need to _completely_ rewrite all of these tests because they are tightly coupled to the internal implementation of the user repo.

I want to show you a better way.

### Behavior-driven approach to testing
Let's start off fresh with our basic user and repo stub:

<Code language="go" heading="File: user.go">
{`package app

type User struct {
	Name string
}
`}
</Code>

<Code language="go" heading="File: repo/users.go">
{`package repo

type Users struct{}
`}
</Code>

#### Save a User
Just like before, let's start with saving a user in the repo. But instead of thinking in terms of the steps that the repo will need to take to accomplish this, think about how we want the repo to behave when it is used. How does this change our perspective of the Save() method? Well, what behavior do we want from the repo when we save a user? After we save a user, we should then be able to get it back! If we think of it this way, then it doesn't matter _how_ the repo stores the data. It only matters that if we save it, we can then get the same data back.

Let's use TDD to write our first behavioral test.

##### Spec: Should save a user to an empty repo
When writing this first test, we need to spend a bit more time thinking about the interface to get a user back out of the repo, too. In this example we're going to use a GetAll() function that returns a slice of all the users in the repo. We could also retrieve a single user by name, or some other way that aligns with the other use-cases.

###### Red Step 1
Notice the package name is repo_test instead of just repo. We want to test the repo's _interface_ so we only want to access its publicly exported methods. Go allows appending _test to the package name to enforce this.
<Code language="go" heading="File: repo/users_test.go">
{`package repo_test

import (
	app "github.com/benjohns1/tdd-and-testing-behavior/behavior-driven"
	"github.com/benjohns1/tdd-and-testing-behavior/behavior-driven/repo"
	"reflect"
	"testing"
)

func TestUser_Save(t *testing.T) {
	type args struct {
		user app.User
	}
	tests := []struct {
		name      string
		repo      repo.Users
		args      args
		wantErr   error
		wantUsers []app.User
	}{
		{
			name: "should save a user to an empty repo",
			args: args{
				user: app.User{Name: "Ender"},
			},
			wantUsers: []app.User{{Name: "Ender"}},
		},
	}
	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			err := tt.repo.Save(tt.args.user)
			if !reflect.DeepEqual(err, tt.wantErr) {
				t.Errorf("Save() error = %v, wantErr %v", err, tt.wantErr)
				return
			}
			got, err := tt.repo.GetAll()
			if err != nil {
				t.Fatal(err)
			}
			if !reflect.DeepEqual(got, tt.wantUsers) {
				t.Errorf("After Save(), GetAll() = %+v, wantUsers %+v", got, tt.wantUsers)
			}
		})
	}
}
`}
</Code>

###### Green Step 2
To make this test pass, we need both a Save() and GetAll() method. Remember, we're just writing the bare minimum code to make the test pass in this step!

<Code language="go" heading="File: repo/users.go">
{`package repo

import (
	app "github.com/benjohns1/tdd-and-testing-behavior/behavior-driven"
)

type Users struct{}

func (u *Users) Save(user app.User) error {
	return nil
}

func (u *Users) GetAll() ([]app.User, error) {
	return []app.User{{Name: "Ender"}}, nil
}
`}
</Code>

###### Refactor Step 3
Now we refactor our naive code into little better solution. And we have our test to verify it is still correct:

<Code language="go" heading="File: repo/users.go (partial)">
{`type Users struct {
	user app.User
}

func (u *Users) Save(user app.User) error {
	u.user = user
	return nil
}

func (u *Users) GetAll() ([]app.User, error) {
	return []app.User{u.user}, nil
}
`}
</Code>

##### Spec: Should save a user to a repo that already has a user in it

###### Red Step 1
<Code language="go" heading="File: repo/users_test.go (partial)">
{`// ...
		{
			name: "should save a user to a repo that already has a user in it",
			repo: func() repo.Users {
				r := repo.Users{}
				if err := r.Save(app.User{
					Name: "Ender",
				}); err != nil {
					t.Fatal(err)
				}
				return r
			}(),
			args: args{
				user: app.User{Name: "Valentine"},
			},
			wantUsers: []app.User{
				{Name: "Ender"},
				{Name: "Valentine"},
			},
		},
// ...
`}
</Code>

###### Green Step 2
<Code language="go" heading="File: repo/users.go (partial)">
{`type Users struct {
	users []app.User
}

func (u *Users) Save(user app.User) error {
	u.users = append(u.users, user)
	return nil
}

func (u *Users) GetAll() ([]app.User, error) {
	return u.users, nil
}
`}
</Code>

###### Refactor Step 3
Let's make our repo encapsulation a bit better by not allowing mutation of the repo's internal slice:

<Code language="go" heading="File: repo/users.go (partial)">
{`func (u *Users) GetAll() ([]app.User, error) {
	out := make([]app.User, 0, len(u.users))
	for _, user := range u.users {
		out = append(out, user)
	}
	return out, nil
}
`}
</Code>


##### Spec: Should fail if a user's name already exists

###### Red Step 1
<Code language="go" heading="File: repo/users_test.go (partial)">
{`// ...
		{
			name: "should fail if a user's name already exists",
			repo: func() repo.Users {
				r := repo.Users{}
				if err := r.Save(app.User{
					Name: "Peter",
				}); err != nil {
					t.Fatal(err)
				}
				return r
			}(),
			args: args{
				user: app.User{Name: "Peter"},
			},
			wantUsers: []app.User{{Name: "Peter"}},
			wantErr:   fmt.Errorf("user name \\\"Peter\\\" already exists"),
		},
// ...
`}
</Code>

###### Green Step 2
<Code language="go" heading="File: repo/users.go (partial)">
{`func (u *Users) Save(user app.User) error {
	for _, current := range u.users {
		if current.Name == user.Name {
			return fmt.Errorf("user name \\\"Peter\\\" already exists")
		}
	}
	u.users = append(u.users, user)
	return nil
}
`}
</Code>

###### Refactor Step 3
Clean up the naive error message:

<Code language="go" heading="File: repo/users.go (partial)">
{`func (u *Users) Save(user app.User) error {
	for _, current := range u.users {
		if current.Name == user.Name {
			return fmt.Errorf("user name %q already exists", user.Name)
		}
	}
	u.users = append(u.users, user)
	return nil
}
`}
</Code>

Here's [the code for this approach](https://github.com/benjohns1/tdd-and-testing-behavior/tree/main/behavior-driven)

## Conclusion
Take a look at your implementation using the behavior-driven approach. For the same amount of effort, we've also implemented retrieving users from the repo, too. Now let's ask the same questions we did earlier: Could we add a user ID field index? Or upgrade to a persistent DB instead of in-memory? Both of these scenarios could be accomplished without modifying our existing tests (apart from maybe some test setup to implement a real DB backend).

Our behavior tests are much more robust, and they're validating what we really care about: the behavior of the repo. If the interface changes we _want_ the tests to break because that means we've changed the repo's behavior. If we need to refactor the internals of the repo, we can use our tests to be confident that we haven't broken any existing use-cases.

TDD is an incredibly useful practice but only if you are testing behavior!


The importance of testing behavior when using test-driven development

TDD and Testing Behavior


When building a distributed system it's important to consider how to handle business flows that span across multiple aggregates or services. Firstly, we should try to simplify the flow so that a single user action only modifies one aggregate at a time. But sometimes that is not possible so we must decide how to coordinate our components.

_Orchestration_ is when a central controller manages the interactions between services; it explicitly models business logic in one place. In microservice terminology, this is called the _Saga_ pattern.

_Choreography_ is when services communicate by emitting and listening to events; it fully decouples communication.

Both of these approaches are useful in different situations. At the end of this post I boil this "it depends" down to a simple rule of thumb. I use terminology in the examples from domain-driven design and microservice architectures, but this is also relevant for monolithic systems.

For the examples below, imagine we're designing a travel booking system that allows end-users to book a trip across multiple travel services, such as airlines, hotels, rental car agencies, etc.

Cross-Context Example
---------------------

We need to communicate between two subdomains that have different business responsibilities, each managed by separate engineering teams. The subdomains are implemented as bounded contexts:

1.  **Booking** allows an end-user to book services with external providers.
2.  **CRM** allows external providers to manage their end-user customer base and communications.

In Booking, the end-user creates an account with a basic email. CRM eventually needs to know about that user account and create a profile, so that our providers can track their communication history and preferences.

### Orchestration

The Booking context, acting as an orchestrator, sends a direct command to CRM to create a new user profile.

_Pros:_

*   Booking's code explicitly shows each context with which it is communicating.

_Cons:_

*   Booking directly depends on CRM's command structure.
*   If another system wants to know about booking accounts, Booking must also be updated.
*   Booking's responsibility bleeds into needing to directly communicate with other contexts.

### Choreography

Using choreography, on the other hand, Booking emits a User Account Created event, then the CRM listens to the event and creates a new user profile.

_Pros:_

*   Booking has no need to know that CRM is listening.
*   If another system wants to know about booking user accounts, they simply need to listen to the event.
*   Booking's responsibility is solely focused on booking travel, and not communicating with other contexts.

_Cons:_

*   It takes more work to determine who is reacting to the events.
*   CRM needs to keep track of which events it has processed, and ensure it doesn't miss any.

### Winner for Cross-Context Communication?

**Choreography!**

*   Decoupling across bounded contexts is extremely important.
*   Specific business flows shouldn't span across multiple contexts (if they do, then the boundaries should be re-thought).
*   If CRM needs to create user profiles based on other events from other contexts, it is the CRM's responsibility to know about that and listen for them.

In-Context Example
------------------

Within the Booking bounded context, there are multiple services: Reservation, Provider, and Payment.

A user completes a reservation for multiple travel services then enters payment details. When the user confirms, the travel reservation is confirmed for each external service provider, then the payment is processed.

### Orchestration

The Reservation service, acting as an orchestrator, sends a direct command to the Provder service (which communicates to the external travel providers), and if all succeed, sends a direct command to the Payment service.

_Pros:_

*   The Reservation service code clearly shows the order in which the business logic is executed.

_Cons:_

*   The Reservation service directly depends on the Provider and Payment services' command structures.
*   When another provider or service needs to needs to be added to the reservation flow, the Reservation service must be updated.
*   The Reservation service responsibility includes communicating with the Provider and Payment services.
*   The Reservation service must handle compensating actions if a failure occurs anywhere in the flow.

### Choreography

Using choreography:

1.  The Reservation service emits a Reservation Created event.
2.  The Provider service handles this event and communicates with the external service providers.
3.  The Payment service handles this event and processes payment.
4.  Upon failure, compensating events must be emitted and handled where appropriate.

_Pros:_

*   The Reservation service doesn't need to know about the other services.
*   If another system wants to know about reservations, they just need to listen to the events.
*   Reservation's responsibility is solely focused on creating reservations in our system, and not coordinating with other services.

_Cons:_

*   It is hard to follow the basic business logic that a reservation entry results in a provider reservation and a processed payment.
*   To view the current state of the end-user's reservation and payment, one must directly query the Provider and Payment services, or listen to a large number of events, including compensations events if a failure occurred.

### Winner for In-Context Communication?

**Orchestration!**

*   The business logic is very clear and in one place; the Reservation service orchestrates it.
*   Even though there is direct service-to-service communication, it does _not_ need to be synchronous. We can still use decoupled asynchronous communication practices.
*   Minimal coupling within a bounded context is okay, because business logic sometimes must flow across multiple services.
*   Events could _still_ be emitted by each service when something happens, in case external contexts need to know.

Choosing an Approach
--------------------

The entire business logic flow is modeled explicitly in a single place with orchestration, whereas it emerges from the interaction between components with orchestration.

**Rule of Thumb:** _Use orchestration within a single bounded context to model a single business flow; use choreography to communicate across bounded contexts._

If you encounter business flows that span across multiple bounded contexts, making it difficult to use choreography, it might be an indicator to reconsider your domain boundaries.

References
----------

Directly relevant:

*   [Choreography vs Orchestration in the land of serverless by Yan Cui](https://theburningmonk.com/2020/08/choreography-vs-orchestration-in-the-land-of-serverless)
*   [GOTO 2020 - Automating Processes in Modern Architectures by Bernd Ruecker](https://www.youtube.com/watch?v=IBfuKA9Nc3M)

Where to capture orchestration logic:

*   [Domain vs Application Logic by Ben Johns](https://blog.bennyjohns.com/2020/12/domain-vs-application-logic.html)
*   [Services in Domain-Driven Design (DDD) by Lev Gorodinski](http://gorodinski.com/blog/2012/04/14/services-in-domain-driven-design-ddd/)
*   [What is domain logic? by Vladimir Khorikov](https://enterprisecraftsmanship.com/posts/what-is-domain-logic/)


When is it appropriate to use centralized orchestration versus event-driven choreography?

Orchestrate or Choreograph?


Domain vs Application Logic in Domain-Driven Design
===================================================

Problem vs Solution Space
-------------------------

When defining a business problem and planning its solution, keep the two conversations separate so that the problem is understood in as much detail as possible first. Business logic is part of the _Solution Space_ so should be discussed only after thoroughly understanding the problem.

*   The _Problem Space_ identifies the purpose, the "why".
*   The _Solution Space_ is the "how".

Common DDD terminology falls into each category:

#### Problem Space
- Domain
- Problem Domain
- Core Domain
- Subdomain

#### Solution Space
- Business Logic
- Business Rules
- Domain Logic
- Bounded Context


The terms Domain, Problem Domain, Core Domain are synonymous, as are Business Logic, Business Rules, and Domain Logic; these terms are used interchangeably.

After understanding the problem space, we can dig in to the rules that solve the business problem and encode this domain knowledge in explicit logic.

This _Business Logic_ solves the problem at hand but does _not_ address implementation details such as database technology, API definitions, web frameworks, etc. Our business logic should encode what rules are needed to solve the problem with language everyone can understand.

Keep Things Simple
------------------

### Business Logic

Business logic should be created in collaboration with business experts. Our _primary purpose_ as software engineers is to convert that logic into an implementation that can be executed.

So, let's **keep things simple** and separate our domain model from the application details. If we capture the business logic in an isolated area of our code, write the code using terms anyone can understand from the problem domain, and keep all application details out of this area, it is much easier to understand the purpose of the solution and update the business rules when needed.

How do we know what the domain logic is? Ask whether the code makes decisions that have business meaning. Any _business-critical_ decision should be done in the domain logic; everything else is wiring.

### Application Logic

This wiring code is the _Application Logic_. This layer sadly represents the bulk of complexity and effort in many systems, despite the fact that it is simply _necessary_ and has minimal impact on your business's competitive edge: a topic for another day.

The application logic orchestrates executing the business logic and all of the infrastructure details like persistence, UI, external system communication, etc. The application layer hosts an environment for executing domain logic; it collects and transforms external input into forms that the business logic uses to make decisions.

Usually the application layer communicates using primitive data types and converts them into domain-specific data types for clarity.

Application Architecture
------------------------

In order for the application layer to do its job of _orchestration_, it is helpful to further separate implementation details of specific technologies into adapters. Uncle Bob's Clean Architecture is a wonderful approach to this.

### Infrastructure

Infrastructure adapter implementations could include management of the following application concerns, for example:

*   Data persistence (e.g. in-memory, MySQL, file system, DynamoDB)
*   User notifications (e.g. email, SMS)
*   Web API (e.g. REST, GraphQL, web socket)
*   UI frameworks

### Dependency Inversion

Each infrastructure adapter can implement an interface that is defined in the application layer. During startup the implementation can be instantiated and dependency injected into the application layer. This isolates the application layer from infrastructure changes, which has the massive benefit of allowing an effort like swapping a database technology to not require a full application rewrite; only the infrastructure adapter needs to be re-implemented against the same interface.

This leads to a dependency flow where the detailed infrastructure implementations depend on the application layer, and the application layer depends on the domain logic. Our domain logic is kept pristine without any external dependencies.

*   Infrastructure Adapter (e.g. DB, REST API, external service communication) depends on ->
*   Application Logic (orchestrates the flow of requests and executes business logic) depends on ->
*   Business Logic (makes business-critical decisions) depends on nothing

References
----------

*   [Implementing Domain-Driven Design](https://www.amazon.com/Implementing-Domain-Driven-Design-Vaughn-Vernon-ebook-dp-B00BCLEBN8/dp/B00BCLEBN8) Vaughn Vernon
*   [Domain Driven Design Distilled](https://www.amazon.com/Domain-Driven-Design-Distilled-Vaughn-Vernon/dp/0134434420) by Vaughn Vernon
*   [The Clean Architecture](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html) by Robert C. Martin 2012-08-13
*   [What is domain logic?](https://enterprisecraftsmanship.com/posts/what-is-domain-logic/) by Vladimir Khorikov 2016-08-25
*   [Services in Domain-Driven Design](http://gorodinski.com/blog/2012/04/14/services-in-domain-driven-design-ddd/) by Lev Gorodinski 2012-04-14
*   [Application layer vs domain layer?](https://softwareengineering.stackexchange.com/questions/140999/application-layer-vs-domain-layer)
*   [What is the difference between business and application logic?](https://softwareengineering.stackexchange.com/questions/293885/what-is-the-difference-between-business-and-application-logic)


When defining a business problem and planning its solution, keep the two conversations separate...

Domain vs Application Logic


Modern message brokers provide many important benefits to a distributed system:  

1.  Flexible Routing: support for multiple distribution modes (e.g. fan-out, round-robin, priority queues, etc)
2.  High-Throughput: designed for large scalable infrastructures
3.  Durability: resilient against intermittent network connections and provide message buffering during downstream service outages
4.  Service Decoupling: each system component only needs to know about the location of the broker and can be decoupled from other components
5.  Asynchronous Messaging: decoupling requests from responses frees up services so they don't need to wait for downstream replies  
    

On the other hand, message brokers add complexity and a single point of failure that must be managed, so may not be a great fit for systems that are small or have simple communication models. But those kinds of systems are no fun, are they? Larger systems gain a lot of architectural benefits, and the downsides can bit mitigated via clustering and redundancy, which all modern brokers support in some form.

This post digs into asynchronous messaging flows to consider when using a message broker, and how async messaging can be implemented behind a synchronous API.

Handling Large Data Sets
------------------------

Many applications require processing on larger data sets like media file attachments, data migrations, or batch processing. Modern brokers are optimized for large numbers of small messages rather than the reverse, so it's not efficient to send the entire data set in one message. There are two standard approaches for handling this:

### 1\. Reference Message

Store the data externally and send a message over the broker with a reference to the data.

This approach works well if the entire data set needs to be consumed at once, like a file attachment. The data itself can be retrieved using a more appropriate protocol like FTP or HTTP. For example: a producer stores a file in an AWS S3 bucket, sends a message with the object's URI, and the consumer pulls the file back down from S3 when the message is received.

### 2\. Streaming

Split the data into chunks and stream them over multiple messages.

This approach works well if the client can start processing the data without needing the entire set, like video streaming. This leans more heavily on the message broker, and the consumer needs to know how to reassemble the chunks.

Commands and Queries
--------------------

Even without using formal CQRS, it's still helpful to separate the concept of a command vs query message. Requests could come from any agent, be it an automated system or a user of a web app. Requests that modify the state of the system are _commands_; requests that retrieve data without modifying the system are _queries_. In event-driven systems, an _event_ message type may be used to notify subscribers that a command has modified the state of the system. If any of these messages require a large amount of data to be sent over the wire, one of the approaches in the previous section should be used to keep payloads small and streamlined.

Async Flows
-----------

These messaging flows do not describe the various routing mechanisms that are possible with a message broker and only focus on the request and response flows.

To relate the terms in this section to a client-server setup, the _producer_ is the client sending requests, while the _consumer_ is the server receiving requests and responding.

### Queries

Queries are fairly straightforward, and there is really only one async flow, but consideration should be taken if the amount of data being returned is large (as described previously).

1.  Producer listens on a reply-to channel
2.  Producer sends query and includes a reference the reply-to channel
3.  Consumer processes query and sends query data back on the reply channel
4.  Producer receives data from the reply channel

### Commands

Commands should not require synchronous replies, if possible. The success or failure result of the command should be handled asynchronously via one of the variants below. There are pragmatic approaches, however, to allow interoperability with synchronous web standards that return a success or failure result; these are discussed in the next section.

#### No Reply

This fire-and-forget approach only works if the producer does not care whether the command succeeded or failed.

1.  Producer sends command
2.  Consumer processes command, no reply necessary

#### Poll for Result

The producer must poll for a reply via a subsequent query, in this approach.

1.  Producer generates a unique ID for the command
2.  Producer sends command
3.  Consumer processes command, no reply necessary
4.  Producer polls via a query using the unique command ID

#### Reply via Events

Even if you're using an event-driven system, I don't recommend this approach, as it requires emitting an event on command success _and_ on failure, which can clutter up the event streams. But in situations where a failed command truly represents a meaningful domain event, it could be useful.

1.  Producer listens for events
2.  Producer sends command
3.  Consumer processes command and publishes event

#### With Reply-To

This, in my opinion, is the cleanest approach to handling commands if the producer architecture supports listening. It also mirrors the asynchronous query flow.

1.  Producer listens on a reply-to channel
2.  Producer sends command and includes a reference to the reply channel
3.  Consumer processes command and sends reply upon completion
4.  Producer receives reply

Web Frontend Flows
------------------

The async flows described are all fine-and-dandy if the entire stack you are working with is setup for asynchronous messaging. But of course, we all have to make pragmatic concessions for the real world. As with any important architectural decision, the answer to which flow we should adopt is "it depends". For instance: we may have a fully asynchronous, distributed backend system that needs to support a synchronous HTTP 1.1 web API, or perhaps our legacy backend system has a web frontend that makes use of GraphQL subscriptions over web sockets. The following flows build upon the asynchronous flows and describe how they can be adapted to support a variety of common real-world communication patterns.

### HTTP Proxy API Flows

These flows are useful to support a synchronous REST or GraphQL API, where the web clients do not listen for push messages.

#### API with HTTP Proxy Flows

Using an HTTP proxy layer insulates services from HTTP 1.1 so they can communicate solely with async messaging protocols, and it frees the frontend up to communicate via common synchronous web protocols like REST.

##### Synchronous Command or Query

Use a reverse proxy layer that waits for asynchronous replies:

1.  Web client makes API request
2.  Proxy sends an async command or query
3.  Proxy blocks and returns response from async command or query when it is returned via a separate asynchronous channel

##### Asynchronous Command

Use a reverse proxy layer that simply converts to/from the message protocol for commands. The client is then responsible for asynchronously retrieving the command results:

1.  Web client makes API request
2.  Proxy sends async command with no reply expected
3.  Proxy immediately returns a 202 Accepted
4.  Web client polls for result using a Sync Query

### Direct API Flows

One option is to simply not use a message broker at all; the services themselves process a synchronous requests. This approach requires a load balancer for scaling.

1.  Web client makes API request
2.  API sends synchronous command or query directly to service
3.  Service processes command or query and synchronously replies

### Hybrid Flows

This hybrid approach uses both direct and proxy flows, and can work well with CQRS. The main benefit of this approach is to lighten the load on the message broker since it is only used for commands.

*   Command: use one of the HTTP Proxy API flows
*   Query: use the synchronous Direct API flow

### Web Socket Flows

If you're building a realtime web app whose users have highly available network connections and maintaining web socket connections is an option, this approach brings the full advantage of asynchronous messaging all the way to the web client.

*   Backend routes bi-directional messages (via load balancer or routing layer that converts between web socket and internal messaging protocols)
*   Frontend subscribes to queries that the backend will push to the client in realtime


Modern message brokers provide many important benefits to a distributed system...

Commands and Queries with a Message Broker


> Good architectures allow major architectural decisions to be deferred. The job of an architect is not to make decisions, the job of an architect is to defer decisions as long as possible to allow the program to be built in the absence of decisions, so that decisions can be made later with the most possible information.

_— Robert C Martin, [The Principles of Clean Architecture, Norwich UK, 2015](https://www.youtube.com/watch?v=o_TH-Y78tt4)_

Sources from Uncle Bob
----------------------

*   Book: [Clean Architecture: A Craftsman's Guide to Software Structure and Design](https://www.amazon.com/Clean-Architecture-Craftsmans-Software-Structure/dp/0134494164)
*   Blog: [The Clean Architecture, 13 August 2012](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html)
*   Talk: [The Principles of Clean Architecture by Uncle Bob Martin, Norwich UK, 2015](https://www.youtube.com/watch?v=o_TH-Y78tt4)

SOLID Design Principles
-----------------------

#### SRP: Single Responsibility Principle

Does _not_ necessarily mean that a module should do just one thing, but that a module should have only one reason to change.

#### OCP: Open-Closed Principle

A module's behavior can be modified without modifying the core code; class inheritance and composition. For example: to swap out a DB implementation, the business logic should not need to be modified.

#### LSP: Liskov Substitution Principle

Concrete implementations must adhere to the expected interface contracts.

#### ISP: Interface Segregation Principle

Keep interfaces small!

#### DIP: Dependency Inversion Principle

*   Interface abstractions are more stable (change less frequently) than concrete implementations
*   Use the factory pattern to keep concrete implementations out of business logic
*   Use a small number of concrete "main" components, where all DIP violations are gathered

Architecture
------------

> The goal of software architecture is to minimize the human resources required to build and maintain the required system.

_— Robert C Martin, [Clean Architecture: A Craftsman's Guide to Software Structure and Design](https://www.amazon.com/Clean-Architecture-Craftsmans-Software-Structure/dp/0134494164)_

Two words "soft" and "ware" mean an easily changeable product.

The "policy" (i.e. business logic) is essential; the "details" (e.g. I/O, frameworks, protocols, etc) are irrelevant to it.

Defer architectural decisions as long as you can, so you have the most information possible. For example, wait to choose a DB technology and web framework until the core business app is built.

> A good architecture maximizes the number of decisions _not_ made. And you achieve that by using a plugin model.

_— Robert C Martin, [The Principles of Clean Architecture, Norwich UK, 2015](https://www.youtube.com/watch?v=o_TH-Y78tt4)_

### Dependency Inversion

*   UI, frameworks, persistence/database, etc _depend_ on the business logic (not the other way around).
*   Only inject some strategies and factories, keep it limited.
*   Do _not_ inject your business logic directly into the app. Instead, inject into a plugin abstraction, then the plugin distributes objects. This is so business logic and its tests do not need to know about injection framework.
*   Do _not_ couple your app to the framework, keep it at arms length.

![](/images/posts/2019-01-26%20Clean%20Architecture%20Diagram.png)

### Business Logic at a Glance: Source Code Structure

Someone new looking at the code should primarily see the _business logic_; the technical implementation details should be hidden away.

*   Primary focus of structure should be to describe the business domain
*   Implementation details should be low-level and should take a back-seat to domain logic in source code

Printable Cheat Sheets
----------------------

### SOLID Design Principles

<div className="max-w-80">[![](/images/posts/2020-11-01%20SOLID%20Design%20Principles%20Cheat%20Sheet.png)](/images/posts/2020-11-01%20SOLID%20Design%20Principles%20Cheat%20Sheet.png)</div>

### Clean Architecture Black & White

<div className="max-w-80">[![](/images/posts/2019-01-26%20Clean%20Architecture%20Cheatsheet%20B%20and%20W.png)](/images/posts/2019-01-26%20Clean%20Architecture%20Cheatsheet%20B%20and%20W.png)</div>

### Clean Architecture Diagram

<div className="max-w-80 bg-white rounded-lg">[![](/images/posts/2019-01-26%20Clean%20Architecture%20Diagram.png)](/images/posts/2019-01-26%20Clean%20Architecture%20Diagram.png)</div>

### Clean Architecture Diagram with Text

<div className="max-w-96 bg-white rounded-lg">[![](/images/posts/2019-01-26%20Clean%20Architecture%20Diagram%20with%20Text.png)](/images/posts/2019-01-26%20Clean%20Architecture%20Diagram%20with%20Text.png)</div>


Printable cheat sheets to help remember some of Uncle Bob's valuable contributions to the industry

Clean Architecture and SOLID Cheat Sheets


Title photo by [Johann Siemens](https://unsplash.com/photos/EPy0gBJzzZU?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText)

After years of consulting, I find myself continually coming back to three basic principles of system design. I could happily throw away all my books on system architecture and design patterns to instead apply these fundamentals as first principles to software design; or put another way, I believe the entire body of modern software design knowledge stems from these fundamentals. The second and third principles derive directly from the preceding ones. So, when boiled down to its essence, the most fundamental goal of system design is Simplicity.  
  
With every new requirement, complexity inevitably encroaches on the most elegant of designs, but if we hold simplicity as the most important fundamental design goal, we can keep complexity at bay. This is one of the most important roles of a product owner or architect: to be a steward of simplicity. To help accomplish this we can apply the second principle: Loose Coupling between components of the system. This implies that we clearly define boundaries between system components and integrate them in such a way that modifying one component minimally affects other components. The second goal stems from the first, and the third goal stems from the first two. A simple, loosely coupled system promotes Maintainability. Software systems are living, mutable things that interact with living, mutable people and organizations. To be a viable solution, they must be maintainable.  
  
The three fundamental system design principles:  

1.  Simplicity
2.  Loose Coupling
3.  Maintainability

Simplicity
----------

Simple systems promote clarity and rapid iteration. While designing for any requirement, the most important question is, "what is the simplest solution?" Every proposed solution's value should be weighed against its added complexity to the system. Put Occam's razor and the KISS principle into practice. Sometimes this requires us to think about the requirement in a different way or even push back against the requirement itself. This is a hard lesson that I'm still learning: sometimes it is necessary to say "no". However, this should be followed up with a proposed rephrasing of the requirement or alternate approach to accomplishing its business intent, such as moving the requirement elsewhere outside of the system.

  

Scope creep is one of the most insidious enemies of large projects, and it can come from anyone with stake in the project. A couple of innocent, "oh, it would be great if it could also do this" from the client and a few well meaning, "we can do this and it wouldn't be that much more work!" from an enthusiastic developer can turn a simple component into spaghetti. Keep the responsibility of the system and its components as focused and small as possible to accomplish the business need.

  

Simplicity also requires knowing when to use design patterns appropriately to avoid antipatterns. As a young developer whenever I learned a new technique or pattern, I would try to apply it anywhere it would fit, which got me into trouble on numerous occasions and frequently left me with an unmaintainable mess of a solution that was difficult to understand. Design patterns are incredible tools that allow us to stand on the shoulders of giants of our industry, but they should only be used if they reduce the complexity of the system. Abstractions are not innately good and they should be used only if they provide clarity.


A tree is an incredibly complex organism, but abstractions of leaf, branch, trunk, and root clarify and simplify its structure into understandable components.  
  
Software systems must be understood by human beings, so keep them simple.  

Loose Coupling
--------------

Simple systems are loosely coupled systems. Each component should strive for simplicity on its own and the system should strive for simplicity with the interactions between components. Even if you're not using an object-oriented language, you can still use the good old SOLID principles to guide your architecture—just replace the "object" concept with "component":

1.  [Single responsibility principle](https://en.wikipedia.org/wiki/Single_responsibility_principle)
2.  [Open-closed principle](https://en.wikipedia.org/wiki/Open%E2%80%93closed_principle)
3.  [Liskov substitution principle](https://en.wikipedia.org/wiki/Liskov_substitution_principle)
4.  [Interface segregation principle](https://en.wikipedia.org/wiki/Interface_segregation_principle)
5.  [Dependency inversion principle](https://en.wikipedia.org/wiki/Dependency_inversion_principle)

Each of these promote loose coupling between components.

  

Loose coupling allows components to be changed or completely replaced without rewriting other components. By minimizing dependencies between them, we also minimize unwanted side effects when the system changes. And as modern solutions increasingly use cloud services like AWS, Azure, and GCP, loose coupling can mitigate the risk of vendor lock by allowing individual components to be migrated separately.

  

These benefits should be discussed with the business owners early on to help explain why these system design goals are important, and will help provide context when you may need to push back on a requirement or spend some extra time to simplify an approach.

  

Techniques like Domain Driven Design coupled with Event Storming help to collaborate early and clearly define component boundaries that structurally align with the business domain.

Maintainability
---------------

A simple and loosely coupled system lends itself to being maintainable. It should be easy to update, fix, and enhance the business logic, supporting infrastructure, system rules, and code configuration at any point in the system's life cycle. I find it helpful to ask myself, "will a junior developer or business analyst be able to change this, once I am no longer available?" If the answer is "no," then I need to rewrite or redesign the component so its intent is crystal clear.

  

Embrace minimalism in your code and "do more with less". I usually don't consider a task code-complete until I've been able to refactor away any superfluous complexity. I'm not talking about trying to pack as much logic into the smallest amount of code as possible ([code golf](https://en.wikipedia.org/wiki/Code_golf)!), quite the opposite; try to be as verbose as needed to make your logic clear and concise.

  

Unit tests and Test Driven Development techniques help ensure that as a system is updated, previous functionality is not broken. Tests are much more easily written against systems that have small, simple and loosely coupled components with minimal dependencies because there's much less of a need for complicated mocks and stubs.

  

[![](/images/posts/nasa-45075-unsplash.jpg)](/images/posts/nasa-45075-unsplash.jpg)

An astronaut repairing a satellite. Photo by [NASA](https://unsplash.com/photos/5JuLcub3dYg?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText)

Even the most well designed systems require maintenance and future iteration. Plan for it. Usually this means taking extra time once the system is functional to refactor it towards a simpler implementation.

Make It Actionable
------------------

Regardless of methodology, technique, or management approach, the fundamental principles of simplicity, loose coupling, and maintainability guide good system design. Here's how I concretely address these principles in a typical project:

1.  Discuss the benefits of these goals with the business owner or client up front. Set the expectation that time will be needed at the end of the project to optimize for simplicity.
2.  When solutioning for requirements, strive to find the absolute simplest solution, and beware of scope creep.
3.  Take the time to clearly define the boundaries of the system; DDD is an excellent tool for this.
4.  During design and implementation, keep maintainability in mind (i.e. write clear and concise code with unit tests).
5.  After implementation is complete, take extra time to reevaluate solutions and optimize them for simplicity and maintainability.


After years of consulting, I find myself continually coming back to three basic principles of system design...

System Design Principles


[![](/images/posts/2019-02-02%20Catinuous%20Delivery.jpg)](/images/posts/2019-02-02%20Catinuous%20Delivery.jpg)  
Our cat has recently been teaching us about the importance of automation! The pipeline tends to kick-off at about 5 a.m. though. Time to get an automatic feeder. 


Catinuous Delivery


How do we manage the architectural complexity that inevitably arises from using cloud services?  
  
Using Functions as a Service (FaaS) and Backends as a Service (BaaS) (i.e. Serverless) can cause pieces of business logic to be strewn about in many places:  

*   Visual orchestration tools (e.g. Azure's Logic Apps)
*   FaaS logic (e.g. AWS Lambda, Azure Functions, Google Cloud Functions)
*   BaaS setups (e.g. Google Firebase)
*   Cloud configurations

Combined with modern single-page applications that maintain complex logic on the client side and microservices that split an organization's concerns across small modules, it's very easy for business logic to become fragmented and intertwined with infrastructure concerns. Is it possible to build organized and maintainable systems using Domain Driven Design (DDD) techniques and clean architecture patterns?

Forest for the Trees
--------------------

![](/images/posts/2014-08-17_deception_pass_forest.jpg)

Although originally introduced by [Eric Evans](http://domainlanguage.com/about/) in his [seminal book](https://www.amazon.com/dp/0321125215) in 2004, DDD is very well-suited for reigning in the modern chaos. The guiding principle of this technique is that one must strive to understand the business domain and its interactions, first and foremost, and articulate it in a way that is clear to everyone involved, including the business, developers, managers, etc.—everyone. Clear understanding and communication of the business domain is a firm prerequisite.

<div className="bg-white rounded-lg px-8 py-4">![](/images/posts/2019-02-20_bounded_context_example.png "DDD event diagram")</div>

A lower-level DDD event diagram showing specific business logic

  
The result of a DDD approach should be a clear understanding of the business logic (i.e. domain logic) and how that logic is grouped (into domains, subdomains, bounded contexts, and aggregates). Only when the implementation-agnostic groupings (i.e. the domain model) are defined in this way should the discussion turn towards choosing particular technologies. These choices may lead to refining the lower-level specifics of the model (such as reorganizing or splitting an aggregate), but should not modify the higher-level domain architecture. The domain architecture reflects the driving purpose of the business itself and should not be defined by implementation details.  

Into the Weeds
--------------

DDD provides an excellent organizational approach to the system as a whole, but how do we keep the domain logic separate from the implementation details? Within an individual component of the design (e.g. a hosted application, FaaS, external service, SPA), separating the business logic from the infrastructure can be accomplished using various patterns. I prefer [Clean Architecture](http://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html), coined by "Uncle Bob", but the idea has been around for a long time as iterations of layered, hexagonal, and onion architectures. The key concept behind all these patterns is that the business logic should not depend on anything but itself. It does not make direct API calls; it does not query a database; it does not read a configuration file; it does not concern itself with anything infrastructure related at all—all of these things support and depend on the business logic, not the other way around. Anything that the business logic needs is passed into it, and anything it needs to communicate is sent out via an interface or abstraction it defines.  
  

![Clean Architecture: Domain, Application, Presentation, Infrastructure, and Persistence Layers](/images/posts/2019-01-26%20Clean%20Architecture%20Diagram.png)

Clean architecture aligns well with code-based components like applications, microservices, and cloud functions, but how do we apply the same principle to visual or configuration-based components like Azure Logic Apps? There's no easy answer to this because it heavily depends on the particular component; the important thing is that the business logic is separated as much as possible from any external concerns so it can be maintained separately. For instance, a Logic App may retrieve data from an external API in order for the business logic to decide to take an action. The business decision process should be configured in a separate Logic App or Function, rather than embedded within the parent Logic App, and it should be made clear which is which.  

Is Smaller Better?
------------------

Cloud services provide many benefits, but care must be taken to ensure that the domain logic doesn't become entangled with infrastructure concerns. The smaller pieces of business logic that are spread across various services, the more difficult this becomes. Smaller services should only be used if there is a clear benefit. This concept is a balancing factor to the current trend of implementing smaller and smaller microservices and the adoption of FaaS. A clean domain promotes simplicity, maintainability, and flexibility at the highest level.  

People at the Core
------------------

Human discipline is required to maintain organization and cleanliness throughout the life of the domain. Because modern architectures are dependent on myriad implementations, there is no guarantee that each technology will have a technological or automated solution to enforce these practices. This brings us back full circle to the importance of _everyone_ understanding the business domain as articulated by DDD. Policies and practices should encourage people to maintain an organized domain, rather than to take short cuts; they should guide people to "fall into the pit of success" as they say, and this starts with clear communication.


How do we manage the architectural complexity that inevitably arises from using cloud services?

software-architecture