
### Solutions for Philanthropy
[Foundant Technologies](https://www.foundant.com) makes philanthropy easier and more impactful for funders, nonprofits, scholarship providers, and community foundations.

From October 2019 to December 2023, I led and worked on a variety of teams to solve accounting, fundraising, and CRM needs for nonprofits. I was an advocate for iterative, test-driven, and continuous delivery practices with domain-driven design and event storming. I also had the opportunity to mentor juniors and new engineering hires during a period of rapid growth, practice pair programming, and give numerous lunch & learns on a wide variety of topics.

I built, monitored, and maintained dozens of customer-facing back-end services and front-end UIs. To support these services, we created an event-driven microservice platform on AWS with Terraform, Go, Docker, and Vue.js. I also implemented cross-cutting platform utilities for authentication, authorization, REST API generation, CI/CD pipelines, event sourcing, and new service generation.


Foundant Technologies

To hide page elements without IE showing them for a split second when the page loads, add an id to your body tag as soon as the DOM loads. You'll want to do this even before you load any JavaScript libraries like jQuery or Prototype.  
  
This is what your page layout should look like:  

<Code language="html">
{`<!DOCTYPE HTML>  
<html>  
<head>  
    
</head>  
<body>  
    
  <script type="text/javascript">  
document.getElementsByTagName('body')\[0\].id = 'js';  
  </script>  
    
</body>  
</html>
`}
</Code>
  
This will add the 'js' id to your body tag only if the user has Javascript enabled. Adding the script right away makes sure that the elements are hidden as soon as possible, rather than having to wait for jQuery/Prototype to load, etc...  
  
Then, in your CSS file create a selector for your element like so:  

<Code language="css">
{`#element {  
  /\* css for all users \*/  
}  
body#js #element {  
  /\* css for javascript enabled browsers \*/  
  display: none;  
}
`}
</Code>
  
This will hide the element so you can display it later with your JavaScript code, but show it for non-JavaScript users. Ahhh, beautiful degradation.

Instantly Hide Page Elements


Since installing Ubuntu on my home machine about a year ago, I've compiled a list of simple commands that I frequently need, but always have to look them up in the man pages:  

  

scp /path/to/local/file username@host:/path/to/remote/location
- copies a local file to a server via ssh

find . -name filename | xargs -I % rm -rf %
- finds all instances of filename and remove them (CAREFUL!)

ln -s /target/path/to/file /path/to/link
- creates a symbolic link

ls -d \*/
- lists all directories in cwd

du -hs /path/to/directory
- prints the size of directory

du -h --max-depth=1 /path/to/directory
- prints the size of each folder and file in directory


Handy Linux Commands


As I was setting up my home testing server (Ubuntu Server 9.04), I kept running into issues with Deny Hosts while testing my server's authentication methods. It was a real pain to find out how to remove an IP that has been blocked.  
  
First of all, references to WORK\_DIR point to /var/lib/denyhosts/ or something similar on your system.  
The default location for the hosts.deny file is /etc/hosts.deny.  
  
Here's the method I used to unblock an IP:  

1.  Stop DenyHosts
2.  Remove the IP address from hosts.deny
3.  Remove all lines containing the IP address from the following files:  
    *   WORK\_DIR/hosts
    *   WORK\_DIR/hosts-restricted
    *   WORK\_DIR/hosts-root
    *   WORK\_DIR/hosts-valid
    *   WORK\_DIR/users-hosts
4.  Consider adding the IP address to WORK\_DIR/allowed-hosts
5.  Start DenyHosts


Unblocking an IP from Deny Hosts


I decided to try out HostGator's VPS Hosting for a new Drupal project I'm working on. This is the first post in a series that will touch on various aspects of VPS management, Drupal, and video distribution.  
  
I wanted to set up a custom nameserver partially for the learning experience and partially to gain total control over load balancing, if necessary in the future. First of all, I signed up for the Level 1 VPS at HostGator. The low cost and scalability of their plans attracted me, though I've never used them before. After doing some research, I decided to setup a simple nameserver hosted on the VPS that points to my domains on the server. Keep in mind that this is a very simple setup that is not necessarily secure. Check out the resources at the end of this post if you want to dig deeper.  
  
The three files we will need to create or edit are:  

*   /etc/resolv.conf
*   /etc/named.conf
*   /var/named/chroot/var/named/example.com.db

This assumes that you're running BIND in a chroot jail (which is the default setup for HostGator's VPS at the moment).

Step 1
------

Create or edit resolv.conf so that it looks like this:  

File: /etc/resolv.conf
```
nameserver 127.0.0.1
```

You should comment out any other lines if they exist. This tells the host to look at its own nameserver for domain name lookups.  

Step 2
------

Next, create or edit named.conf so that it contains:  

File: /etc/named.conf
```
options {  
  listen-on port 53 { any; };  
};  
zone "example.com" {  
 type master;  
  notify no;  
  allow-query { any; };  
  file "/var/named/chroot/var/named/example.com.db";  
};
```

This tells BIND to listen on port 53 (the default DNS port number) from any IP. The _zone_ section should be repeated for every domain you're serving. The _file_ part should point to each zone file.  
  

Step 3
------

Now we need to create a zone file for the domain:  

File: /var/named/chroot/var/named/example.com.db
```
$TTL 14400  
@ 86400 IN SOA ns1.example.com. admin@example.com. (  
    2008021501 ; serial, todays date+todays  
    86400 ; refresh, seconds  
    7200 ; retry, seconds  
    3600000 ; expire, seconds  
    86400 ) ; minimum, seconds  
example.com. 86400 IN NS ns1.example.com.  
example.com. 86400 IN NS ns2.example.com.  
ns1 IN A 111.222.333.444  
ns2 IN A 111.222.333.445  
example.com. IN A 111.222.333.444  
localhost.example.com. IN A 127.0.0.1  
example.com. IN MX 0 example.com.  
mail IN CNAME example.com.  
www IN CNAME example.com.  
ftp IN A 111.222.333.444
```

Basically, this file sets up each nameserver as ns1.example.com (111.222.333.444) and ns2.example.com (111.222.333.445). This matches the nameservers to each IP.  

Step 4
------

Then, restart BIND:  
```
service named restart
```

Step 5
------

Finally, you need to contact your registrar and register your new nameservers ns1.example.com and ns2.example.com. [Here are some step-by-step instructions](http://www.whsupport.net/register_nameservers.php) for the most common registrars.  
  
And that should do it! Please check out the following resources, as this short example just barely scratches the surface of DNS configuration:  

*   [Quick HOWTO : Ch18 : Configuring DNS](http://www.linuxhomenetworking.com/wiki/index.php/Quick_HOWTO_:_Ch18_:_Configuring_DNS#BIND)
*   [DNS/BIND: Create a basic zone file](http://www.tech-recipes.com/rx/305/dnsbind-create-a-basic-zone-file/)
*   [Chroot-BIND HOWTO](http://www.faqs.org/docs/Linux-HOWTO/Chroot-BIND-HOWTO.html)
*   [Bind: Quick install guide to install and setup Bind (DNS server) in secure (chroot) environment in Linux (CentOS, Redhat Enterprise (RHEL), Fedora).](http://crazytoon.com/2007/06/21/bind-quick-install-guide-to-install-and-setup-bind-dns-server-in-secure-chroot-environment-in-linux-centos-redhat-enterprise-rhel-fedora/)


How to Set Up a Simple BIND Nameserver on CentOS


One quirky thing about RedHat is that they are _very_ conservative as to the packages they include in their repositories. In this case, PHP 5.1.6 was the latest version and I needed at least 5.2 for a Drupal 6 installation.

Step 1
------

[atomicorp.com](http://atomicorp.com/) hosts a repository which we'll add:  

wget -q -O - http://www.atomicorp.com/installers/atomic.sh | sh

Step 2
------

Then we'll install the latest version of PHP using yum:  

```
yum install php
```

Or if an older version of php is already installed:  

```
yum update php
```

Step 3
------

Restart the webserver:  

```
/etc/init.d/httpd restart
```
  
And that's it! Now you should have PHP 5.2 installed.  
  
Resources:  

*   [http://bluhaloit.wordpress.com/2008/03/13/installing-php-52x-on-redhat-es5-centos-5-etc/](http://bluhaloit.wordpress.com/2008/03/13/installing-php-52x-on-redhat-es5-centos-5-etc/)
*   [http://mayurchaudhari.wordpress.com/2009/02/23/upgrade-php-to-5-2-x-using-yum/](http://mayurchaudhari.wordpress.com/2009/02/23/upgrade-php-to-5-2-x-using-yum/)
*   [http://iptrees.wordpress.com/2009/10/11/how-to-upgrade-to-php-5-2-x-on-centosrhelfedoraedora/](http://iptrees.wordpress.com/2009/10/11/how-to-upgrade-to-php-5-2-x-on-centosrhelfedoraedora/)
*   [http://netfirms.com/community/threads/35-Upgrade-to-PHP-5-2-x-on-CentOS-5-Plesk](http://netfirms.com/community/threads/35-Upgrade-to-PHP-5-2-x-on-CentOS-5-Plesk)


Installing PHP 5.2 on CentOS 5

While building a Drupal site for one of my clients, I was having a heck of a time integrating the [Lightbox2 module](http://drupal.org/project/lightbox2) for use with iframe content. For example, I wanted a login page (with [Janrain Engage](http://drupal.org/project/rpx)) and a couple other forms displayed in a lightbox. I wanted these pages to alternately be viewed as regular old nodes if the user didn't have JavaScript enabled, if they linked to the page externally, or if they right-clicked to open in a new tab, etc. The [Lightbox2 documentation](http://drupal.org/node/252260) provides a few different ways of doing this, but—here's the tricky part—none of the techniques address all of these scenarios:  

*   Only the node content should be visible in a lightbox (no sidebars!)
*   The user clicks a link in a lightbox that should target the top window
*   The user clicks a lightbox link that should stay in the lightbox iframe
*   The user clicks a lightbox link that calls a third-party service that redirects the user back to the site via a url token

Basically, I wanted to be able to happily add lightbox links anywhere in the site without worrying about whether the user could possibly come across a page that was formatted to be inside an iframe, or a lightbox that contained all my site's headers and sidebars. Talk about visual clutter!  
  
My solution was simple. For every page, check if it is in an iframe. If so, use CSS to hide the unnecessary page blocks. That way, no matter how you get to the page, it will always be formatted correctly. I'll admit that this does leave something to be desired in that we don't necessarily want to load the _entire_ page if all we want is the content, but short of writing a custom lightbox module, or modifying Lightbox2, it was the best solution I could find.

Step 1
------

Add a wrapper div directly below your body tag with an id of 'no-iframe'. This is our (semantically incorrect, sorry!) 'global variable' that we'll use to hold whether the page is in an iframe or not.  

Step 2
------

Add this script to your theme's page.tpl.php file right before the closing body tag:  

<Code language="javascript">
{
`if (top !== self) {  
  document.getElementById('no-iframe').id = 'in-iframe';  
}
`}
</Code>

This changes our wrapper div's id from 'no-iframe' to 'in-iframe' if, well, it's in an iframe. Ideally, you want to add the rest of your scripts after this snippet in order to apply the CSS rules to it as soon as possible. This cuts down on the time the hidden content is visible to the user.  
  
For example, if your theme is based off the [Zen theme](http://drupal.org/project/zen), the tail end of your page.tpl.php might look like this:  

<Code language="html">
{
`    
</body>  
</html>
`}
</Code>
  
So, now for every page, if our elements are inside #no-iframe, then we display all headers, sidebars like normal. If our elements are inside #in-iframe, then we use CSS to hide our unwanted blocks.  

Step 3
------

Here's an example of my iframe-layout.css:  

<Code language="css">
{
`  /* Layout for pages within an iframe */  
  /* hide unwanted blocks */
  #in-iframe #header,  
  #in-iframe #navbar,  
  #in-iframe #footer,  
  #in-iframe #sidebar-right  
  {  
    display: none;  
  }  
  /* center my main content */  
  #in-iframe #main-inner  
  {  
    width: 660px;  
    margin: 0 auto;  
  }  
  /* fix my content margins to display without the sidebar */  
  #in-iframe #content  
  {  
    width: 660px;  
    margin-left: 0;  
    margin-right: 0;  
  }
`}
</Code>

Obviously, your CSS will differ from mine, depending on your theme and layout, but the key ingredient is the display: none to hide the unwanted blocks.  
  
Now, whenever you open a lightbox, the page that's loaded will automatically know that it's in an iframe, and display only the content you want. And if you link to the node normally, it will show your entire template! We still need one more tweak, though.  

Step 4
------

Create a new JavaScript file for your theme and add some simple jQuery:  

<Code language="javascript">
{
`$('a', '#in-iframe').each(function() {  
  var $this = $(this);  
  if ($this.attr('rel') == 'lightframe') {  
    $this.unbind('click').removeAttr('rel');  
  } else {  
    $this.attr('target', '\_top');  
  }  
});
`}
</Code>

This will find every instance of a link within the lightbox iframe and make sure it targets the parent window, or—for links that should open in a lightbox—will remove the click function that opens a _new_ one, allowing it to open in the current lightbox iframe.  
  
Now, all of our original scenarios are addressed and we've kept it all in the theme layer! Didn't even have to touch template.php :-)

While building a Drupal site for one of my clients, I was having a heck of a time integrating...

Using the Lightbox2 Drupal Module to Display Page Content


While in college, I worked as a freelance web developer providing web, storefront, and UI design solutions for small businesses. I developed a custom MVC PHP framework to make web development easier. During this time, I was also playing in professional bands and teaching private music lessons.


Freelance Work


Here's a slight modification to the handy Google Bookmarks Bookmarklet: 
```
javascript:(function(){var a=window,b=document,c=encodeURIComponent,d=a.open("http://www.google.com/bookmarks/mark?op=edit&bkmk="+c(b.location)+"&title="+c(b.title),"bkmk_popup","left="+((a.screenX||a.screenLeft)-20)+",top="+((a.screenY||a.screenTop)+20)+",height=600px,width=800px,resizable=1,alwaysRaised=1,scrollbars=1,menubar=1,toolbar=1");a.setTimeout(function(){d.focus()},300)})();
```  
A window (or tab depending on your browser settings) pops up with the current page's URL and title loaded into a Google Bookmark for you to add. It allows you to easily browse your existing bookmarks or add a new one, and it increases the window size from the default bookmarklet available on the Google page.


Here's a slight modification to the handy Google Bookmarks Bookmarklet...

Google Bookmarks Bookmarklet


Recently one of my client sites had an issue where the custom theme info was corrupted. The following warnings were being thrown, and the theme wasn't loading:

PHP Error Reporting

```
Warning: Invalid argument supplied for foreach() in /drupal/path/includes/theme.inc on line 497

Warning: Invalid argument supplied for foreach() in /drupal/path/includes/theme.inc on line 502

Warning: Invalid argument supplied for foreach() in /drupal/path/includes/theme.inc on line 497

Warning: Invalid argument supplied for foreach() in /drupal/path/includes/theme.inc on line 502
```

Somehow the theme info database record (which holds a PHP serialized object) was corrupted.

The quick fix for this was to switch to a different theme, and switch back (rebuild the theme info). As this was hard to do without navigation, here are the URLs to hit:

baseurl.com/user
- Login as administrator

baseurl.com/admin/build/themes/select
- Select another theme, and switch back to your original one


Recently one of my client sites had an issue where the custom theme info was corrupted...

Drupal 6 Theme Info Error


### Professional Services
RightNow Technologies was a CRM software provider purchased by Oracle in 2012. As a technical consultant there from August 2010 through December 2012, I worked closely with clients throughout discovery, design, implementation, and support of custom integration solutions. I worked on and eventually led large projects in the financial, health, and public sectors.


Oracle & RightNow Technologies


Game and simulation development for fun and profit!

[pilotwavegames.com](https://pilotwavegames.com)

- Unity
- Unreal Engine
- JavaScript

<Button href="https://pilotwavegames.itch.io">Visit on Itch.io</Button>


Game and simulation development for fun and profit!

Introducing Pilot Wave Games


Genetic algorithm searches for the optimal path to a goal

- Built with p5.js
- Evolutionary algorithm experimentation
- "DNA" encodes the path each agent will attempt during its lifetime
- Mutation rates
- Agent DNA recombination

<Button href="https://pilotwavegames.itch.io/genetic-seeker">Play on Itch.io</Button>
[View Source](https://github.com/benjohns1/genetic-seeker)


Genetic Seeker


Retro screensaver awesomeness

- Built with p5.js
- Interesting patterns emerge from pseudo-random rotation rates

<Button href="https://benjohns1.github.io/p5-random-spinner-viz/index.html">Live Demo</Button>
[View Source](https://github.com/benjohns1/p5-random-spinner-viz)


Spinner Visualization


Simplest way to watch your favorite cryptocurrencies

- Built with Cycle.js, Immutable.js, and Bootstrap 4
- Reactive programming
- Functional programming

[View Source](https://github.com/benjohns1/crypto-watcher)


Static web app to watch cryptocurrency prices

Crypto Watcher


Collect colorful resources and build a base

- Built with Phaser.js

<Button href="https://pilotwavegames.itch.io/bubble-builder">Play on Itch.io</Button>
[View Source](https://github.com/benjohns1/bubble-builder)


Bubble Builder


Race along a lonely and dreamlike path that winds its way through light and mist

- Built in Unreal Engine
- Level tiles, color themes, &amp; item pickups all procedurally generated

<Button href="https://pilotwavegames.itch.io/speed-craft-runner">Download at Itch.io</Button>
[View Source](https://github.com/benjohns1/speedcraftrunner)


Speed Craft Runner


Simulate gas mixing with 3D voxels

- Built in Unity with C#
- Calculates dissipation and mixing of up to 4 gases in each voxel over time

[View Source](https://github.com/benjohns1/gas-voxel-sim)


Gas Voxel Sim


Procedurally generate a spherical planet

- Built in Unity with C#
- Based on and extended from [Sebastian Lague's excellent tutorial series](https://www.youtube.com/playlist?list=PLFt_AvWsXl0cONs3T0By4puYy6GM22ko8)

[View Source](https://github.com/benjohns1/procedural-planet)


Procedural Planet Generation


Procedurally generate 3D terrain

- Built in Unity with C#
- Layered noise functions
- Height and biome based texturing
- Based on and extended from [Sebastian Lague's excellent tutorial series](https://www.youtube.com/playlist?list=PLFt_AvWsXl0eBW2EiBtl_sxmDtSgZBxB3)

[View Source](https://github.com/benjohns1/procedural-landmass-generation)


Procedural Landmass Generation


I've decided to resurrect this old blog to publish some nuggets about software architecture and development, and perhaps, if the mood strikes me, some scientific and philosophical musings. When I first posted here almost nine years ago in 2010, I had recently finished my B.S. in Computer Science and was working as a web developer for small mom and pop companies in my area. Shortly thereafter I was hired to my first "real" job in professional services as a junior consultant for a small software company, learned some industry practices, made some connections, and enthusiastically watched as large tech company acquired it. After the initial excitement wore off, I realized the culture was shifting rapidly away from what I originally enjoyed about the job and that I was working for a truly massive corporation. So a couple of like-minded colleagues and I quit, and we embarked on a journey to start our own consulting company. Many mistakes, lessons, successes, and years later our little firm is going strong, but I've increasingly felt the need to more directly refine my craft of software development. In our line of work (and in others I'm sure), it's easy to get comfortable with a subset of technologies and solutions while losing track of the bigger picture, inevitably getting stuck in the past while the rest of the industry leaps ahead.  
  
My recent interest is in ideas that have been around for a while, some of which have seen a huge resurgence recently and some of which I'm attempting to absorb in a modern context:  

*   Functional Programming
*   Reactive Programming
*   Clean Architecture
*   Domain Driven Design
*   Test and Behavior Driven Development

I come from a web front-end, PHP (I'd be a happy man to never write another line of PHP!), and C# background but recently have been learning about Golang, Elixir, Docker, GraphQL, game development in Unity, various cloud services, machine learning, decentralized ledgers, \[insert additional bright-and-shiny objects here\]. My goal with this blog is to focus and distill these efforts into a form that's personally useful to future-me and potentially others.

Read my older blog posts on [blog.bennyjohns.com](https://blog.bennyjohns.com)


I've decided to resurrect this old blog to publish some nuggets about software architecture and development, and perhaps...

This Old Blog


How do we manage the architectural complexity that inevitably arises from using cloud services?  
  
Using Functions as a Service (FaaS) and Backends as a Service (BaaS) (i.e. Serverless) can cause pieces of business logic to be strewn about in many places:  

*   Visual orchestration tools (e.g. Azure's Logic Apps)
*   FaaS logic (e.g. AWS Lambda, Azure Functions, Google Cloud Functions)
*   BaaS setups (e.g. Google Firebase)
*   Cloud configurations

Combined with modern single-page applications that maintain complex logic on the client side and microservices that split an organization's concerns across small modules, it's very easy for business logic to become fragmented and intertwined with infrastructure concerns. Is it possible to build organized and maintainable systems using Domain Driven Design (DDD) techniques and clean architecture patterns?

Forest for the Trees
--------------------

![](/images/posts/2014-08-17_deception_pass_forest.jpg)

Although originally introduced by [Eric Evans](http://domainlanguage.com/about/) in his [seminal book](https://www.amazon.com/dp/0321125215) in 2004, DDD is very well-suited for reigning in the modern chaos. The guiding principle of this technique is that one must strive to understand the business domain and its interactions, first and foremost, and articulate it in a way that is clear to everyone involved, including the business, developers, managers, etc.—everyone. Clear understanding and communication of the business domain is a firm prerequisite.

<div className="bg-white rounded-lg px-8 py-4">![](/images/posts/2019-02-20_bounded_context_example.png "DDD event diagram")</div>

A lower-level DDD event diagram showing specific business logic

  
The result of a DDD approach should be a clear understanding of the business logic (i.e. domain logic) and how that logic is grouped (into domains, subdomains, bounded contexts, and aggregates). Only when the implementation-agnostic groupings (i.e. the domain model) are defined in this way should the discussion turn towards choosing particular technologies. These choices may lead to refining the lower-level specifics of the model (such as reorganizing or splitting an aggregate), but should not modify the higher-level domain architecture. The domain architecture reflects the driving purpose of the business itself and should not be defined by implementation details.  

Into the Weeds
--------------

DDD provides an excellent organizational approach to the system as a whole, but how do we keep the domain logic separate from the implementation details? Within an individual component of the design (e.g. a hosted application, FaaS, external service, SPA), separating the business logic from the infrastructure can be accomplished using various patterns. I prefer [Clean Architecture](http://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html), coined by "Uncle Bob", but the idea has been around for a long time as iterations of layered, hexagonal, and onion architectures. The key concept behind all these patterns is that the business logic should not depend on anything but itself. It does not make direct API calls; it does not query a database; it does not read a configuration file; it does not concern itself with anything infrastructure related at all—all of these things support and depend on the business logic, not the other way around. Anything that the business logic needs is passed into it, and anything it needs to communicate is sent out via an interface or abstraction it defines.  
  

![Clean Architecture: Domain, Application, Presentation, Infrastructure, and Persistence Layers](/images/posts/2019-01-26%20Clean%20Architecture%20Diagram.png)

Clean architecture aligns well with code-based components like applications, microservices, and cloud functions, but how do we apply the same principle to visual or configuration-based components like Azure Logic Apps? There's no easy answer to this because it heavily depends on the particular component; the important thing is that the business logic is separated as much as possible from any external concerns so it can be maintained separately. For instance, a Logic App may retrieve data from an external API in order for the business logic to decide to take an action. The business decision process should be configured in a separate Logic App or Function, rather than embedded within the parent Logic App, and it should be made clear which is which.  

Is Smaller Better?
------------------

Cloud services provide many benefits, but care must be taken to ensure that the domain logic doesn't become entangled with infrastructure concerns. The smaller pieces of business logic that are spread across various services, the more difficult this becomes. Smaller services should only be used if there is a clear benefit. This concept is a balancing factor to the current trend of implementing smaller and smaller microservices and the adoption of FaaS. A clean domain promotes simplicity, maintainability, and flexibility at the highest level.  

People at the Core
------------------

Human discipline is required to maintain organization and cleanliness throughout the life of the domain. Because modern architectures are dependent on myriad implementations, there is no guarantee that each technology will have a technological or automated solution to enforce these practices. This brings us back full circle to the importance of _everyone_ understanding the business domain as articulated by DDD. Policies and practices should encourage people to maintain an organized domain, rather than to take short cuts; they should guide people to "fall into the pit of success" as they say, and this starts with clear communication.


How do we manage the architectural complexity that inevitably arises from using cloud services?

Cloud Services and Architectural Complexity


[![](/images/posts/2019-02-02%20Catinuous%20Delivery.jpg)](/images/posts/2019-02-02%20Catinuous%20Delivery.jpg)  
Our cat has recently been teaching us about the importance of automation! The pipeline tends to kick-off at about 5 a.m. though. Time to get an automatic feeder. 


Catinuous Delivery


This is the first in a series of simple tutorials explaining the usage of HTTP middleware in Go.  
  
Middleware is useful for injecting logic before and after a request handler, without needing to modify the business logic of the handler itself. For instance, you can use the middleware pattern to log some basic information about every request sent to the server, then apply some standard authentication for those requests that need it, and finally apply the appropriate handler to run the actual business logic.  
  
In this context middleware is a specific case of the [decorator pattern](https://en.wikipedia.org/wiki/Decorator_pattern); it is typically used to cleanly add logic to a web request handled by a server. Middleware can be chained to wrap multiple pieces of functionality around a single request (e.g. logging, authentication, authorization, request/response encoding, error handling). Historically, the term "[middleware](https://en.wikipedia.org/wiki/Middleware)" has been used in SOA and enterprise architectures to cover any type of software that facilitates communication between two other systems (e.g. Service Bus), but is being used more frequently to refer to a logical layer _within_ a single service.  
  
This pattern helps encapsulate logic to adhere to the [Single responsibility principle](https://en.wikipedia.org/wiki/Single_responsibility_principle) for [SOLID](https://en.wikipedia.org/wiki/SOLID) design and [Clean Architecture](http://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html).

Part 1 - Loop Chain Middleware Example
--------------------------------------

[![](/images/posts/mike-alonzo-3347-unsplash.jpg)](/images/posts/mike-alonzo-3347-unsplash.jpg)

Photo by [Mike Alonzo](https://unsplash.com/@mikezo?utm_medium=referral&utm_campaign=photographer-credit&utm_content=creditBadge)

  
The code for this tutorial can be found here:  
[https://github.com/benjohns1/go-middleware-example/blob/master/loopchain/main.go](https://github.com/benjohns1/go-middleware-example/blob/master/loopchain/main.go)  
  
This simple approach uses a range loop to apply each middleware function in order. Technically, this isn't really the "middleware" pattern, but it serves the same purpose and provides a great introduction. We will build upon this simple approach with a more useful implementation in Part 2.  

### Step 1 - A Simple HTTP Server

First let's create a main func to start a simple server:  

main.go:
<Gist src="https://gist.github.com/benjohns1/f1bd0325e47a8879d54b5cb11967f29f.js"/>

We're using the standard http library to handle the "/api/v1" route with our aptly-named businessLogic function. You can run the server with:
```
go run main.go
```
and go to [localhost:8080/api/v1](http://localhost:8080/api/v1) in your browser to see this in action.  
  
One thing to note is that we're wrapping the function that handles the actual http request in a closure (the businessLogic function) so that we can inject some setup values into the handler when the server starts. The first injected value is just a static string, but the second value is a func that is run whenever a request is invoked. Try hitting the endpoint multiple times to see the pseudo-random numbers from runTimeFunc. This isn't important for this particular example, but is being shown now because closures are key to more advanced middleware implementations.  

### Step 2 - Adding Middleware

Now let's create a logging function that will log some generic information about a request:

main.go (partial):
<Gist src="https://gist.github.com/benjohns1/b6aff6e71adefbe127452ea2143d5ef0.js"/>

How do we add this to our route handler in a way that can be reused? This is where the usefulness of middleware comes in! We can write a new function that will chain these handlers together and return a final handler that wraps all of this nicely and neatly for the route:  

main.go (partial):
<Gist src="https://gist.github.com/benjohns1/d9b238f9da534ae24a4c0d94d6c146e1.js"/>

The chain function takes any number of http.HandlerFunc functions and returns a new http.Handler that runs all of them in order. We can create and add as many middleware function handlers as we want to this chain:  
```
http.Handle("/api/v1", chain(logger, moreMiddleware, evenMoreMiddleware, businessLogic("99999", runTimeFunc)))
```
  
These handlers are completely decoupled from each other and can easily be applied to multiple routes in various configurations.  
  
[Part 2](/posts/20190209-go-middleware-part-2) will cover a slightly more advanced implementation approach using recursive functions that allow applying logic before _and_ after the subsequent handler as well as short-circuiting the request chain.  

### Final Server Code

main.go:
<Gist src="https://gist.github.com/benjohns1/9e423f4c5462cb1742af050574632e6f.js"/>

[Go Middleware - Part 2](/posts/20190209-go-middleware-part-2)


This is the first in a series of simple tutorials explaining the usage of HTTP middleware in Go.

Go Middleware - Part 1


In this second part of the Go middleware tutorial series, we'll cover a recursive approach that provides a couple benefits beyond the simple loop chain example from [part 1](/posts/20190206-go-middleware-part-1), including:  

1.  the ability to _wrap_ around the next handler in the chain (i.e. add logic before and after the subsequent handlers are applied)
2.  the ability to short-circuit the request chain in any middleware handler (e.g. if authentication fails)

Part 2 - Recursive Chain Middleware Example
-------------------------------------------

[![](/images/posts/szabo-viktor-1154603-unsplash.jpg)](/images/posts/szabo-viktor-1154603-unsplash.jpg)

Photo by [Szabo Viktor](https://unsplash.com/photos/XDNiTGp3bKM?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText)

The code for this tutorial can be found here:  
[https://github.com/benjohns1/go-middleware-example/blob/master/recursivechain/main.go](https://github.com/benjohns1/go-middleware-example/blob/master/recursivechain/main.go)  

### Step 1 - Simple Server

Just as we did in part 1, let's start with a simple working HTTP server:  

main.go:
<Gist src="https://gist.github.com/benjohns1/deb30888549c60751eb73cb9b60d5544.js"/>

Run the server:
```
go run main.go
```

Test in a browser:
```
localhost:8080/api/v1
```  
Notice the businessLogic function returns a closure around the actual http handler, so we can pass in an all-important config value "12345" when the server is started.  

### Step 2 - Recursive Chain

Rather than use a simple loop that calls handlers one after another like we did in part 1, instead we're going to pass the second handler as a parameter to the first handler, pass the third handler as a parameter to the second handler, and so on:
```
firstHandler(secondHandler(thirdHandler(fourthHandler())))
```  
Each handler will then be responsible for calling (or not!) the next one in the chain. We can do better than a bunch of eye-bleeding nested functions and use recursion:  

main.go (partial):
<Gist src="https://gist.github.com/benjohns1/3d2473a03e122eabac17665f38341f4f.js"/>

The middlewareFunc type describes the new type of middleware function we need to pass into the chain function. It accepts the next handler in the chain and returns the current handler, which wraps the next handler (this should make more sense in a bit).  
The chain function just provides the nice variadic parameter syntax and converts the final handler function into the http.Handler interface that the http router accepts.  
The recurseChain function does the actual work of sending the next handler as a parameter to the current one recursively, up until there aren't any more handlers, and it finally returns an empty function to complete the chain.  
  
To use this chain function, modify the businessLogic to:  

1.  wrap the http handler in a new closure that passes-in the next handler
2.  and call the next handler where appropriate

main.go (partial):
<Gist src="https://gist.github.com/benjohns1/03bf8ef76a2a5acbfcde7c8a206a5076.js"/>

Now our businessLogic function accepts the same config parameter, but it returns the middlewareFunc type function that chain() accepts. This middlewareFunc accepts the next handler in the chain as a parameter and returns the actual handler function that processes the request. The actual handler function has access to the outer config and handler parameters via closures; it can call next(w, r) to continue to the next handler in the chain. In this case next(w, r) just calls the empty function returned from the last iteration of recurseChain.  

### Step 3 - Wrap and Short-Circuit

Okay, thanks a lot, now we've just made things more complicated! Wait, wait, we should [KISS](https://en.wikipedia.org/wiki/KISS_principle). We're adding complexity because it gives us the ability to actually wrap our nested handlers in functionality like this:  

main.go (partial):
<Gist src="https://gist.github.com/benjohns1/7b8ed952520dad541f1dfc52209fa50f.js"/>

and short-circuit the request chain like this:  

main.go (partial):
<Gist src="https://gist.github.com/benjohns1/7673cbf8750ea8d6d107f43f0c91132c.js"/>

The logger middleware prints an entry before _and_ after all subsequent handlers run. The shortCircuitLogic randomly decides not to call any of the subsequent handlers in the chain by returning without calling the next function.  
  
As you can see this provides a lot of flexibility with our middleware, while still keeping each function decoupled from the others.  
  
In Part 3 we'll look at a common variant of this approach that doesn't require the final handler function to call an empty handler.  

### Final Server Code

main.go:
<Gist src="https://gist.github.com/benjohns1/0f6430b1ca1bef824247fea04d00ee4d.js"/>

[Go Middleware - Part 1](/posts/20190206-go-middleware-part-1)

[Go Middleware - Part 3](/posts/20190215-go-middleware-part-3)


In this second part of the Go middleware tutorial series, we'll cover a recursive approach that provides a couple benefits beyond the simple loop chain example from part 1.

Go Middleware - Part 2


In this third part of the Go middleware tutorial series, we'll quickly look at a common variant on the recursive middleware implementation from [part 2](/posts/20190209-go-middleware-part-2).  

Part 3 - Final Handler Middleware Example
-----------------------------------------

[![](/images/posts/samuel-zeller-379406-unsplash.jpg)](/images/posts/samuel-zeller-379406-unsplash.jpg)

Photo by [Samuel Zeller](https://unsplash.com/photos/oBb-Y26PJgg?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText)

This variant separates the idea of middleware from the final handler that processes the request's business logic. This means that each middleware chain can have any number of middleware handler functions, but only a single final handler runs at the end of the chain. This allows the final handler to not need to accept a "next" handler, keeping our business logic a bit cleaner.  
  
The code for this tutorial can be found here:  
[https://github.com/benjohns1/go-middleware-example/blob/master/finalhandler/main.go](https://github.com/benjohns1/go-middleware-example/blob/master/finalhandler/main.go)  

### Step 1 - Modified Chain Functions

An additional parameter can simply be added to the chain functions:  

main.go (partial):
<Gist src="https://gist.github.com/benjohns1/0e74d47c067d9e8845745f1731abadd0.js"/>

We're passing in a new http.HandlerFunc as the first parameter and using it as the final function in the chain (instead of an empty function as we were previously).  

### Step 2 - Simplified Business Logic

Then we pass our handler in as the first parameter in the chain (rather than the last) and remove the extra closure from the businessLogic function so that it just supplies an http handlerFunc and doesn't need to call next():  

main.go (partial):
<Gist src="https://gist.github.com/benjohns1/66e7e099b9a93b887c03bcd9c36b0627.js"/>

This keeps our primary request business logic clean, and with the handler function as the first parameter it's easier to read in a long list of routes.  
  
In Part 4 we'll look at passing custom state up and down the request chain in a type-safe way.

### Final Server Code

main.go:
<Gist src="https://gist.github.com/benjohns1/afd9adbde74aca973fac732c7fd82180.js"/>

[Go Middleware - Part 2](/posts/20190209-go-middleware-part-2)

[Go Middleware - Part 4](/posts/20190224-go-middleware-part-4)


In this third part of the Go middleware tutorial series, we'll quickly look at a common variant on the recursive middleware implementation from part 2.

Go Middleware - Part 3


In this fourth part of the Go middleware tutorial series, we'll discuss passing custom state along the request chain. This provides the flexibility to pass data from one middleware handler to another one somewhere else in the chain.  

Part 4 - Custom State Middleware Example
----------------------------------------

We want to be able to pass custom data from one middleware function to another, but without breaking encapsulation or modularity. We also want to be very explicit and type-safe about the data that is being passed so it is easy to maintain.  
  
The code for this tutorial can be found here: [https://github.com/benjohns1/go-middleware-example/blob/master/customstate/main.go](https://github.com/benjohns1/go-middleware-example/blob/master/customstate/main.go)

### Step 1 - Simple Server

We'll create a simple server that uses the recursive middleware approach from [part 2](/posts/20190209-go-middleware-part-2) with a few modifications:  

main.go:
<Gist src="https://gist.github.com/benjohns1/ec5f243af7c72c6be9e3d9dbc1d5ced1.js"/>

This code should be familiar but for a couple slight differences:  

1.  In main() we're passing in a function literal (i.e. anonymous or lambda function) to the businessLogic function that randomly returns true or false. This is just for brevity and is not all that relevant.
2.  In businessLogic() we're explicitly setting a success or failure HTTP status code, depending on whether the decider function returns true or false.

Wouldn't it be nice to be able to log that status code in our logger middleware? But take a look at the http.ResponseWriter interface; there's no way to retrieve its value after its been set!  

### Step 2 - Implement http.ResponseWriter

Here's an example of where Go's interfaces really shine. We can define a new struct type to contain the data that we need to keep track of, and as long as we implement the http.ResponseWriter interface, our struct can be passed through _any_ middleware handler without issue (even through handlers that don't know or care about our custom data):  

main.go (partial):
<Gist src="https://gist.github.com/benjohns1/67878e5bdf2aed233ee8cf78dea205d3.js"/>


The logResponseWriter type stores the original http.ResponseWriter sent to the logger function and passes through all of the required interface function calls. It extends the standard functionality by storing the status code in an easily-accessible parameter when the WriteHeader interface method is called.  
  
The logger function creates our custom struct and passes it into the next middleware handler in the chain (which is fine because the struct implements the http.ResponseWriter interface). It then accesses the status parameter for logging after the subsequent handlers have run. Notice that we didn't touch a single line of our business logic to implement this, because we leveraged the http.ResponseWriter interface's existing WriteHeader function.  

### Step 3 - Custom Data

You might notice that this only works well for the status code in particular (or for any data that is passed to an existing http.ResponseWriter interface method). So how do we pass additional custom data?  

[![](/images/posts/customize-all-the-things.jpg)](/images/posts/customize-all-the-things.jpg)

In order to do this, both middleware handlers need to be aware of some sort of interface for getting/setting the data. In this example we'll modify our business logic so that it is aware of the logResponseWriter struct, but you can imagine decoupling this further using additional interfaces.  

main.go (partial):
<Gist src="https://gist.github.com/benjohns1/f4cab6ed6428f14feaeb311062ffec01.js"/>

We've added a new field "extra" to store our custom string data. The logger function logs this string after the request is complete.  
  
The businessLogic function does a bit of work to check if the custom interface is implemented. It uses a type assertion to check whether a logResponseWriter was passed in as the http.ResponseWriter, and if so it has type-safe access to the "extra" field.  
  
You can extend this approach to pass any type of custom data up and down the middleware chain. This has a lot of benefit over using, say, a [context](https://golang.org/pkg/context/) to store values on the response writer object, because it clearly defines type-safe fields in an explicit struct, making the code much more readable and less error-prone. However, keep in mind that this approach is still more complicated than simply passing parameters into functions, so only use it when needed.  
  
In the final [part 5](/posts/20190317-go-middleware-part-5) of this series, we'll look at a slightly more realistic and structured example of a JSON API that uses the middleware approaches we've discussed.  

### Final Server Code

main.go:
<Gist src="https://gist.github.com/benjohns1/f329429b730f2abd7f72067036e08353.js"/>

[Go Middleware - Part 3](/posts/20190215-go-middleware-part-3)

[Go Middleware - Part 5](/posts/20190317-go-middleware-part-5)


In this fourth part of the Go middleware tutorial series, we'll discuss passing custom state along the request chain.

Go Middleware - Part 4


In this fifth and final part of the Go middleware tutorial series, we'll use what we've learned to create a more structured API example that includes basic request logging, authorization, and response formatting.  
  
Middleware will help simplify our API code and promote the [single responsibility principle](https://en.wikipedia.org/wiki/Single_responsibility_principle).

Part 5 - API Middleware Example
-------------------------------

The code for this tutorial can be found here: [https://github.com/benjohns1/go-middleware-example/tree/master/apiexample](https://github.com/benjohns1/go-middleware-example/tree/master/apiexample)  

### API Example Code

Rather than building up the code from scratch as in previous tutorials, we're going to start with the completed code and discuss each component. The project is structured as follows:

*   [businessdomain/](https://github.com/benjohns1/go-middleware-example/tree/master/apiexample/businessdomain) - contains all business logic, separately from infrastructure
    *   [businesslogic.go](https://github.com/benjohns1/go-middleware-example/blob/master/apiexample/businessdomain/businesslogic.go) - example logic
*   [middleware/](https://github.com/benjohns1/go-middleware-example/tree/master/apiexample/middleware) - contains all API middleware infrastructure
    *   [auth.go](https://github.com/benjohns1/go-middleware-example/blob/master/apiexample/middleware/auth.go) - middleware to handle authentication
    *   [chain.go](https://github.com/benjohns1/go-middleware-example/blob/master/apiexample/middleware/chain.go) - middleware chain helpers
    *   [logger.go](https://github.com/benjohns1/go-middleware-example/blob/master/apiexample/middleware/logger.go) - middleware to handle API logging
*   [main.go](https://github.com/benjohns1/go-middleware-example/blob/master/apiexample/main.go) - entry point

### Business Logic

This is where the meat and potatoes of the application reside. All the logical rules that the business, client, or user expects from the system should be driven from here without depending on implementation details such as HTTP, DB technology, etc. It's a great idea to write your business logic as decoupled as possible from infrastructure concerns, as has been recommended time and time again with architectural patterns like [The Clean Architecture](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html), Hexagonal Architecture, and Onion Architecture.

businessdomain/businesslogic.go:
<Gist src="https://gist.github.com/benjohns1/71e377bada15692099442b12b661a1a9.js"/>

In this example, we're not actually _doing_ anything except returning a static struct, but notice that this package does not depend on anything related to the API delivery mechanism or database; it could easily be reused for a terminal or GUI application, for example.

### Middleware Chain

Now for the API infrastructure: this is where the recursive chain style of middleware is implemented as described in [Part 2](/posts/20190209-go-middleware-part-2) of this series. This could be replaced by any of the variants we've discussed or by one of the many router/mux packages available.

middleware/chain.go:
<Gist src="https://gist.github.com/benjohns1/a7072db1ee68d2ad064db408eb062e40.js"/>

The Factory type defines a valid function that can be chained as middleware.

### Logging Middleware

Similar to what we've shown in previous examples, this file contains HTTP request logging middleware.

middleware/logger.go:
<Gist src="https://gist.github.com/benjohns1/16f2518c45c4e01cea780e8458b03fb9.js"/>

The Logger function is of type Factory so it can be used directly as a middleware function. We're also implementing a custom WriteHeader function to capture the status code as described in [Part 4](/posts/20190224-go-middleware-part-4).

### Authorization Middleware

This middleware provides two strategies for authorization:

1.  Random Auth - randomly reject or accept a request (example purposes only!)
2.  Public Auth - always authorized

middleware/auth.go:
<Gist src="https://gist.github.com/benjohns1/be823fcd5737890cd8e29a45c87136e6.js"/>

The Auth function accepts an auth strategy function as a parameter to determine whether the user is authorized, and returns a Factory function that can be chained as middleware.

### Putting It All Together

Each of the previous files contain a specific and unique responsibility, and our business logic has no dependency on the API delivery mechanism or infrastructure. Let's put it all together into one informative (but... probably useless) API!

main.go:
<Gist src="https://gist.github.com/benjohns1/035fa81fd9768c05e5ec925f837ee5ed.js"/>

Our main function creates two routes, each using a different auth strategy:

1.  [localhost:8080/api/v1/random/](localhost:8080/api/v1/random/)
2.  [localhost:8080/api/v1/public/](localhost:8080/api/v1/public/)

The requestHandler function calls the business logic and transforms the return value into an appropriate HTTP response. Notice again that the business logic knows nothing about the transport mechanism; the final data format and transport type is handled here. This could be extended to allow additional data formats and more complex transformation steps using the middleware techniques we've just learned.  
  
This concludes the Go Middleware series. Now go forth a conqueror and win great API victories!  
  

[Go Middleware - Part 4](/posts/20190224-go-middleware-part-4)


In this fifth and final part of the Go middleware tutorial series, we'll use what we've learned to create a more structured API example...

Go Middleware - Part 5


Title photo by [Johann Siemens](https://unsplash.com/photos/EPy0gBJzzZU?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText)

After years of consulting, I find myself continually coming back to three basic principles of system design. I could happily throw away all my books on system architecture and design patterns to instead apply these fundamentals as first principles to software design; or put another way, I believe the entire body of modern software design knowledge stems from these fundamentals. The second and third principles derive directly from the preceding ones. So, when boiled down to its essence, the most fundamental goal of system design is Simplicity.  
  
With every new requirement, complexity inevitably encroaches on the most elegant of designs, but if we hold simplicity as the most important fundamental design goal, we can keep complexity at bay. This is one of the most important roles of a product owner or architect: to be a steward of simplicity. To help accomplish this we can apply the second principle: Loose Coupling between components of the system. This implies that we clearly define boundaries between system components and integrate them in such a way that modifying one component minimally affects other components. The second goal stems from the first, and the third goal stems from the first two. A simple, loosely coupled system promotes Maintainability. Software systems are living, mutable things that interact with living, mutable people and organizations. To be a viable solution, they must be maintainable.  
  
The three fundamental system design principles:  

1.  Simplicity
2.  Loose Coupling
3.  Maintainability

Simplicity
----------

Simple systems promote clarity and rapid iteration. While designing for any requirement, the most important question is, "what is the simplest solution?" Every proposed solution's value should be weighed against its added complexity to the system. Put Occam's razor and the KISS principle into practice. Sometimes this requires us to think about the requirement in a different way or even push back against the requirement itself. This is a hard lesson that I'm still learning: sometimes it is necessary to say "no". However, this should be followed up with a proposed rephrasing of the requirement or alternate approach to accomplishing its business intent, such as moving the requirement elsewhere outside of the system.

  

Scope creep is one of the most insidious enemies of large projects, and it can come from anyone with stake in the project. A couple of innocent, "oh, it would be great if it could also do this" from the client and a few well meaning, "we can do this and it wouldn't be that much more work!" from an enthusiastic developer can turn a simple component into spaghetti. Keep the responsibility of the system and its components as focused and small as possible to accomplish the business need.

  

Simplicity also requires knowing when to use design patterns appropriately to avoid antipatterns. As a young developer whenever I learned a new technique or pattern, I would try to apply it anywhere it would fit, which got me into trouble on numerous occasions and frequently left me with an unmaintainable mess of a solution that was difficult to understand. Design patterns are incredible tools that allow us to stand on the shoulders of giants of our industry, but they should only be used if they reduce the complexity of the system. Abstractions are not innately good and they should be used only if they provide clarity.


A tree is an incredibly complex organism, but abstractions of leaf, branch, trunk, and root clarify and simplify its structure into understandable components.  
  
Software systems must be understood by human beings, so keep them simple.  

Loose Coupling
--------------

Simple systems are loosely coupled systems. Each component should strive for simplicity on its own and the system should strive for simplicity with the interactions between components. Even if you're not using an object-oriented language, you can still use the good old SOLID principles to guide your architecture—just replace the "object" concept with "component":

1.  [Single responsibility principle](https://en.wikipedia.org/wiki/Single_responsibility_principle)
2.  [Open-closed principle](https://en.wikipedia.org/wiki/Open%E2%80%93closed_principle)
3.  [Liskov substitution principle](https://en.wikipedia.org/wiki/Liskov_substitution_principle)
4.  [Interface segregation principle](https://en.wikipedia.org/wiki/Interface_segregation_principle)
5.  [Dependency inversion principle](https://en.wikipedia.org/wiki/Dependency_inversion_principle)

Each of these promote loose coupling between components.

  

Loose coupling allows components to be changed or completely replaced without rewriting other components. By minimizing dependencies between them, we also minimize unwanted side effects when the system changes. And as modern solutions increasingly use cloud services like AWS, Azure, and GCP, loose coupling can mitigate the risk of vendor lock by allowing individual components to be migrated separately.

  

These benefits should be discussed with the business owners early on to help explain why these system design goals are important, and will help provide context when you may need to push back on a requirement or spend some extra time to simplify an approach.

  

Techniques like Domain Driven Design coupled with Event Storming help to collaborate early and clearly define component boundaries that structurally align with the business domain.

Maintainability
---------------

A simple and loosely coupled system lends itself to being maintainable. It should be easy to update, fix, and enhance the business logic, supporting infrastructure, system rules, and code configuration at any point in the system's life cycle. I find it helpful to ask myself, "will a junior developer or business analyst be able to change this, once I am no longer available?" If the answer is "no," then I need to rewrite or redesign the component so its intent is crystal clear.

  

Embrace minimalism in your code and "do more with less". I usually don't consider a task code-complete until I've been able to refactor away any superfluous complexity. I'm not talking about trying to pack as much logic into the smallest amount of code as possible ([code golf](https://en.wikipedia.org/wiki/Code_golf)!), quite the opposite; try to be as verbose as needed to make your logic clear and concise.

  

Unit tests and Test Driven Development techniques help ensure that as a system is updated, previous functionality is not broken. Tests are much more easily written against systems that have small, simple and loosely coupled components with minimal dependencies because there's much less of a need for complicated mocks and stubs.

  

[![](/images/posts/nasa-45075-unsplash.jpg)](/images/posts/nasa-45075-unsplash.jpg)

An astronaut repairing a satellite. Photo by [NASA](https://unsplash.com/photos/5JuLcub3dYg?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText)

Even the most well designed systems require maintenance and future iteration. Plan for it. Usually this means taking extra time once the system is functional to refactor it towards a simpler implementation.

Make It Actionable
------------------

Regardless of methodology, technique, or management approach, the fundamental principles of simplicity, loose coupling, and maintainability guide good system design. Here's how I concretely address these principles in a typical project:

1.  Discuss the benefits of these goals with the business owner or client up front. Set the expectation that time will be needed at the end of the project to optimize for simplicity.
2.  When solutioning for requirements, strive to find the absolute simplest solution, and beware of scope creep.
3.  Take the time to clearly define the boundaries of the system; DDD is an excellent tool for this.
4.  During design and implementation, keep maintainability in mind (i.e. write clear and concise code with unit tests).
5.  After implementation is complete, take extra time to reevaluate solutions and optimize them for simplicity and maintainability.


After years of consulting, I find myself continually coming back to three basic principles of system design...

System Design Principles


Survive and build a factory on a hostile world

- Built in Unreal Engine with blueprints visual scripting
- Item crafting tree
- Structure building system with rotation and plugin enhancements
- Connectable conveyors and factory inputs/outputs
- Inventory system with item filtering
- Tutorial missions

<Button href="https://pilotwavegames.itch.io/factored">Download at Itch.io</Button>
[View Source](https://github.com/benjohns1/factored)


Factored


A few weeks ago, I decided to try [Svelte](https://svelte.dev/)'s [Sapper framework](https://sapper.svelte.dev/) to handle the front-end of a simple app. I'm using this small project to try out some new technologies and to practice TDD/BDD. I had already built a back-end API in Go so just needed to build and wire-up a UI. I don't have a lot of experience in modern front-end tech or end-to-end testing, so I was excited to see that Sapper comes ready-to-use with the [Cypress test framework](https://www.cypress.io/). I was able to write some simple BDD(ish) tests to define the functionality I wanted and then implement in Svelte fairly seamlessly. Overall this was a natural and fluid flow, after overcoming a small gotcha.

### Cypress Tests Stall After Visiting New Pages

Most of my tests started with a cy.visit(url) to ensure a fresh page reload, but I quickly found that any click or DOM action within my Svelte components timed-out and failed sporadically, but frequently:  
CypressError: Timed out retrying: Expected to find \[...\]  
  
I believe this was due to the fact that Sapper does an excellent job of preloading and chunking out the app code and data asynchronously, so Cypress identified that the page was completely loaded and continued with the test script before all of the Sapper app chunks had actually been fully loaded.  
  
Initially, I solved this with a hacky cy.visit(url).wait(1000) that would wait for 1 second after the page loaded. Using wait() for something like this is pretty horrible practice, but it allowed me to continue development. But anyone who has worked with UI or E2E testing knows that one of the biggest issues is the time it takes to run them. Even with my little app only partially completed, testing time quickly ballooned out to over a minute to run all the tests due to the full second of wait time.  

#### Solution

My final solution uses Svelte's lifecycle functions to set a DOM attribute when all components are available. Use the onMount() function in the top-level routes/\_layout.svelte component, wait an additional tick() to ensure all child components had been loaded, then set an attribute in the DOM that Cypress can easily check:

<Code language="svelte">
{`
// File: src/routes/\_layout.svelte

<script>
	import Nav from '../components/Nav.svelte';
	import { onMount, tick } from 'svelte';

	export let segment;

	let testID = 'loading';
	onMount(() => {
		tick().then(() => {
			testID = 'loaded';
		});
	});
</script>

<style>
  ...
</style>

<Nav {segment}/>

<main data-test={testID}>
	<slot></slot>
</main>
`}
</Code>

In Cypress create a simple command called visitWait or similar that waits for the correct attribute value:  

<Code language="javascript">
{`
// File: cypress/support/commands.js

Cypress.Commands.add("visitWait", (url, options) => {
  cy.visit(url, options);
  cy.get('[data-test=loaded]');
});
`}
</Code>

In your tests, simply use cy.visitWait(url) as you would normally use cy.visit(url) and Cypress will wait just long enough for all the Svelte components to load before continuing with your test script.


A few weeks ago, I decided to try Svelte's Sapper framework to handle the front-end of a simple app...

Setting Up Cypress for Svelte's Sapper


Automated hierarchical task network planner

- Written in TypeScript
- Encode the problem domain as a set of simple and compound tasks
- The network determines how best to decompose a goal task into an optimal plan

[View Source](https://github.com/benjohns1/htn-planner)


Javascript package to automate hierarchical task network planning

HTN Planner


I was looking for a quick and easy way to put together a personal static site and _almost_ went with Svelte Sapper because I've been enjoying it so much using it for another side project. But in the end I decided to use Next.js instead because it has been around for a while (Sapper is modeled after it), it is so incredibly easy to deploy using Zeit's Now for free, and I wanted to use TypeScript without the hassle of trying to get it to work with Svelte (which apparently _is_ possible). I also wanted to brush up on React since it had been a while.  
  
Here's my target front-end stack:  

*   [Next.js](https://github.com/zeit/next.js)  
    
*   [TypeScript](https://www.typescriptlang.org/)  
    
*   [Material UI](https://material-ui.com/)  
    
*   [Jest](https://jestjs.io/) with [Enzyme](https://airbnb.io/enzyme/)  
    

After working with the Next.js and Material UI examples to get a good starting template, here's the final result, which I plan on starting with for static sites moving forward:

[github.com/benjohns1/next-ts-mui-jest-template](https://github.com/benjohns1/next-ts-mui-jest-template) [![](/images/posts/template-screenshot.png)](https://github.com/benjohns1/next-ts-mui-jest-template)  
  
Grab it and try it out yourself!  
```
npx degit https://github.com/benjohns1/next-ts-mui-jest-template my-project
cd my-project
npm i
npm run dev

```


I was looking for a quick and easy way to put together a personal static site and...

Next.js with TypeScript, Material UI, and Jest


A good ol' task app... but with a scheduler!

- Designed with DDD
- Built with TDD/BDD
- Svelte Sapper front-end
- Front-end testing with Cypress
- Go API and scheduler back-end
- Infrastructure with Terraform and Docker on AWS

[View Source](https://github.com/benjohns1/scheduled-tasks)


Scheduled Tasks


Founded by myself and two business partners in 2013, we cultivated long-lasting partnerships with public and private sector clients across multiple industries. We solved business problems using a range of cloud, in-house, and custom integrated solutions, primary around the Oracle RightNow CRM solution and general software system development. 45 North eventually was sold to [CPI.Solutions](https://www.cpi.solutions/) in 2019.

### CXDev Community
We regularly wrote content for the CXDeveloper blog and moderated discussions. It became a valuable resource to the community of Oracle RightNow developers, consultants, and clients.


45 North Solutions


Terraform leads the way in the infrastructure-as-code world; I've been using it exclusively for my personal projects and wherever I can at work. It provides peace of mind during deployments, stabilizes build pipelines, and helps speed up my adoption of cloud infrastructure. It's no silver bullet, but it's an excellent tool that solves an increasingly complex problem.

### History of the Problem

In traditional applications and old-school organizations, there are extreme difficulties managing on-premise infrastructure:  

1.  Costly hardware installations, operations, and training
2.  Slow reaction to scaling needs
3.  Wasted extra capacity
4.  Manual configuration
5.  Fixed ops team resources
6.  Tickets submitted in a queue means long wait times to make changes

The first 3 are problems that cloud infrastructure and IaaS has been solving for years now. Most modern companies are leveraging the power of cloud providers to fulfill much of their infrastructure needs nowadays. But cloud services don't solve issues #4, 5, or 6 so companies still struggle with:  

*   Difficulty scheduling infrastructure changes alongside application changes
*   Sluggish reaction time when things go wrong
*   Overhead managing ops teams

Additionally, increasing adoption of microservices and serverless introduces new complexities:  

*   Many small, incremental deployments
*   Polyglot data
*   Highly abstracted infrastructure
*   Cloud vendor lock-in

#### Enter Infrastructure as Code

Infrastructure as code tools help organizations address these issues head-on by encoding all infrastructure changes as human-readable code in a consistent way. This provides the benefits of:

*   Efficient and predictable deployments
*   Change tracking
*   Automation
*   Robust disaster recovery
*   Ease of creating and destroying infrastructure
*   Enabling self-service

[Hashicorp's Terraform](https://www.terraform.io/) has quickly become the industry standard solution. It not only provides the previous benefits, but also helps fight against vendor lock-in by supporting a huge number of providers with the ability to write your own.  

### Example

[https://github.com/benjohns1/terraform-example](https://github.com/benjohns1/terraform-example)

  

In the example above, there's a tiny serverless application written in Go that runs on AWS Lambda and API Gateway. It can be built locally from the app/build directory. For deployment, Terraform configurations in the app/deploy define the application zip file to upload to AWS Lambda, and some basic settings for the API Gateway.  
  
The Terraform module in module/aws-serverless-app generically describes what infrastructure is needed in AWS to expose a lambda function behind an API. The app uses this module and passes the specific parameters for itself.  
  
This modular architecture allows us to encapsulate and reuse portions of our infrastructure code. You can imagine multiple apps that provide different business functionality, but all run on lambda behind an API gateway; the aws-serverless-app module could be reused to deploy all of them.  
  
This example was derived directly from Yevgeniy Brikman's excellent talk on [Automated Testing for Terraform, Docker, Packer, Kubernetes, and More](https://www.youtube.com/watch?v=xhHOW0EF5u8)  

*   [https://github.com/gruntwork-io/infrastructure-as-code-testing-talk](https://github.com/gruntwork-io/infrastructure-as-code-testing-talk)


Terraform leads the way in the infrastructure-as-code world...

Why Terraform?


> Good architectures allow major architectural decisions to be deferred. The job of an architect is not to make decisions, the job of an architect is to defer decisions as long as possible to allow the program to be built in the absence of decisions, so that decisions can be made later with the most possible information.

_— Robert C Martin, [The Principles of Clean Architecture, Norwich UK, 2015](https://www.youtube.com/watch?v=o_TH-Y78tt4)_

Sources from Uncle Bob
----------------------

*   Book: [Clean Architecture: A Craftsman's Guide to Software Structure and Design](https://www.amazon.com/Clean-Architecture-Craftsmans-Software-Structure/dp/0134494164)
*   Blog: [The Clean Architecture, 13 August 2012](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html)
*   Talk: [The Principles of Clean Architecture by Uncle Bob Martin, Norwich UK, 2015](https://www.youtube.com/watch?v=o_TH-Y78tt4)

SOLID Design Principles
-----------------------

#### SRP: Single Responsibility Principle

Does _not_ necessarily mean that a module should do just one thing, but that a module should have only one reason to change.

#### OCP: Open-Closed Principle

A module's behavior can be modified without modifying the core code; class inheritance and composition. For example: to swap out a DB implementation, the business logic should not need to be modified.

#### LSP: Liskov Substitution Principle

Concrete implementations must adhere to the expected interface contracts.

#### ISP: Interface Segregation Principle

Keep interfaces small!

#### DIP: Dependency Inversion Principle

*   Interface abstractions are more stable (change less frequently) than concrete implementations
*   Use the factory pattern to keep concrete implementations out of business logic
*   Use a small number of concrete "main" components, where all DIP violations are gathered

Architecture
------------

> The goal of software architecture is to minimize the human resources required to build and maintain the required system.

_— Robert C Martin, [Clean Architecture: A Craftsman's Guide to Software Structure and Design](https://www.amazon.com/Clean-Architecture-Craftsmans-Software-Structure/dp/0134494164)_

Two words "soft" and "ware" mean an easily changeable product.

The "policy" (i.e. business logic) is essential; the "details" (e.g. I/O, frameworks, protocols, etc) are irrelevant to it.

Defer architectural decisions as long as you can, so you have the most information possible. For example, wait to choose a DB technology and web framework until the core business app is built.

> A good architecture maximizes the number of decisions _not_ made. And you achieve that by using a plugin model.

_— Robert C Martin, [The Principles of Clean Architecture, Norwich UK, 2015](https://www.youtube.com/watch?v=o_TH-Y78tt4)_

### Dependency Inversion

*   UI, frameworks, persistence/database, etc _depend_ on the business logic (not the other way around).
*   Only inject some strategies and factories, keep it limited.
*   Do _not_ inject your business logic directly into the app. Instead, inject into a plugin abstraction, then the plugin distributes objects. This is so business logic and its tests do not need to know about injection framework.
*   Do _not_ couple your app to the framework, keep it at arms length.

![](/images/posts/2019-01-26%20Clean%20Architecture%20Diagram.png)

### Business Logic at a Glance: Source Code Structure

Someone new looking at the code should primarily see the _business logic_; the technical implementation details should be hidden away.

*   Primary focus of structure should be to describe the business domain
*   Implementation details should be low-level and should take a back-seat to domain logic in source code

Printable Cheat Sheets
----------------------

### SOLID Design Principles

<div className="max-w-80">[![](/images/posts/2020-11-01%20SOLID%20Design%20Principles%20Cheat%20Sheet.png)](/images/posts/2020-11-01%20SOLID%20Design%20Principles%20Cheat%20Sheet.png)</div>

### Clean Architecture Black & White

<div className="max-w-80">[![](/images/posts/2019-01-26%20Clean%20Architecture%20Cheatsheet%20B%20and%20W.png)](/images/posts/2019-01-26%20Clean%20Architecture%20Cheatsheet%20B%20and%20W.png)</div>

### Clean Architecture Diagram

<div className="max-w-80 bg-white rounded-lg">[![](/images/posts/2019-01-26%20Clean%20Architecture%20Diagram.png)](/images/posts/2019-01-26%20Clean%20Architecture%20Diagram.png)</div>

### Clean Architecture Diagram with Text

<div className="max-w-96 bg-white rounded-lg">[![](/images/posts/2019-01-26%20Clean%20Architecture%20Diagram%20with%20Text.png)](/images/posts/2019-01-26%20Clean%20Architecture%20Diagram%20with%20Text.png)</div>


Printable cheat sheets to help remember some of Uncle Bob's valuable contributions to the industry

Clean Architecture and SOLID Cheat Sheets


Modern message brokers provide many important benefits to a distributed system:  

1.  Flexible Routing: support for multiple distribution modes (e.g. fan-out, round-robin, priority queues, etc)
2.  High-Throughput: designed for large scalable infrastructures
3.  Durability: resilient against intermittent network connections and provide message buffering during downstream service outages
4.  Service Decoupling: each system component only needs to know about the location of the broker and can be decoupled from other components
5.  Asynchronous Messaging: decoupling requests from responses frees up services so they don't need to wait for downstream replies  
    

On the other hand, message brokers add complexity and a single point of failure that must be managed, so may not be a great fit for systems that are small or have simple communication models. But those kinds of systems are no fun, are they? Larger systems gain a lot of architectural benefits, and the downsides can bit mitigated via clustering and redundancy, which all modern brokers support in some form.

This post digs into asynchronous messaging flows to consider when using a message broker, and how async messaging can be implemented behind a synchronous API.

Handling Large Data Sets
------------------------

Many applications require processing on larger data sets like media file attachments, data migrations, or batch processing. Modern brokers are optimized for large numbers of small messages rather than the reverse, so it's not efficient to send the entire data set in one message. There are two standard approaches for handling this:

### 1\. Reference Message

Store the data externally and send a message over the broker with a reference to the data.

This approach works well if the entire data set needs to be consumed at once, like a file attachment. The data itself can be retrieved using a more appropriate protocol like FTP or HTTP. For example: a producer stores a file in an AWS S3 bucket, sends a message with the object's URI, and the consumer pulls the file back down from S3 when the message is received.

### 2\. Streaming

Split the data into chunks and stream them over multiple messages.

This approach works well if the client can start processing the data without needing the entire set, like video streaming. This leans more heavily on the message broker, and the consumer needs to know how to reassemble the chunks.

Commands and Queries
--------------------

Even without using formal CQRS, it's still helpful to separate the concept of a command vs query message. Requests could come from any agent, be it an automated system or a user of a web app. Requests that modify the state of the system are _commands_; requests that retrieve data without modifying the system are _queries_. In event-driven systems, an _event_ message type may be used to notify subscribers that a command has modified the state of the system. If any of these messages require a large amount of data to be sent over the wire, one of the approaches in the previous section should be used to keep payloads small and streamlined.

Async Flows
-----------

These messaging flows do not describe the various routing mechanisms that are possible with a message broker and only focus on the request and response flows.

To relate the terms in this section to a client-server setup, the _producer_ is the client sending requests, while the _consumer_ is the server receiving requests and responding.

### Queries

Queries are fairly straightforward, and there is really only one async flow, but consideration should be taken if the amount of data being returned is large (as described previously).

1.  Producer listens on a reply-to channel
2.  Producer sends query and includes a reference the reply-to channel
3.  Consumer processes query and sends query data back on the reply channel
4.  Producer receives data from the reply channel

### Commands

Commands should not require synchronous replies, if possible. The success or failure result of the command should be handled asynchronously via one of the variants below. There are pragmatic approaches, however, to allow interoperability with synchronous web standards that return a success or failure result; these are discussed in the next section.

#### No Reply

This fire-and-forget approach only works if the producer does not care whether the command succeeded or failed.

1.  Producer sends command
2.  Consumer processes command, no reply necessary

#### Poll for Result

The producer must poll for a reply via a subsequent query, in this approach.

1.  Producer generates a unique ID for the command
2.  Producer sends command
3.  Consumer processes command, no reply necessary
4.  Producer polls via a query using the unique command ID

#### Reply via Events

Even if you're using an event-driven system, I don't recommend this approach, as it requires emitting an event on command success _and_ on failure, which can clutter up the event streams. But in situations where a failed command truly represents a meaningful domain event, it could be useful.

1.  Producer listens for events
2.  Producer sends command
3.  Consumer processes command and publishes event

#### With Reply-To

This, in my opinion, is the cleanest approach to handling commands if the producer architecture supports listening. It also mirrors the asynchronous query flow.

1.  Producer listens on a reply-to channel
2.  Producer sends command and includes a reference to the reply channel
3.  Consumer processes command and sends reply upon completion
4.  Producer receives reply

Web Frontend Flows
------------------

The async flows described are all fine-and-dandy if the entire stack you are working with is setup for asynchronous messaging. But of course, we all have to make pragmatic concessions for the real world. As with any important architectural decision, the answer to which flow we should adopt is "it depends". For instance: we may have a fully asynchronous, distributed backend system that needs to support a synchronous HTTP 1.1 web API, or perhaps our legacy backend system has a web frontend that makes use of GraphQL subscriptions over web sockets. The following flows build upon the asynchronous flows and describe how they can be adapted to support a variety of common real-world communication patterns.

### HTTP Proxy API Flows

These flows are useful to support a synchronous REST or GraphQL API, where the web clients do not listen for push messages.

#### API with HTTP Proxy Flows

Using an HTTP proxy layer insulates services from HTTP 1.1 so they can communicate solely with async messaging protocols, and it frees the frontend up to communicate via common synchronous web protocols like REST.

##### Synchronous Command or Query

Use a reverse proxy layer that waits for asynchronous replies:

1.  Web client makes API request
2.  Proxy sends an async command or query
3.  Proxy blocks and returns response from async command or query when it is returned via a separate asynchronous channel

##### Asynchronous Command

Use a reverse proxy layer that simply converts to/from the message protocol for commands. The client is then responsible for asynchronously retrieving the command results:

1.  Web client makes API request
2.  Proxy sends async command with no reply expected
3.  Proxy immediately returns a 202 Accepted
4.  Web client polls for result using a Sync Query

### Direct API Flows

One option is to simply not use a message broker at all; the services themselves process a synchronous requests. This approach requires a load balancer for scaling.

1.  Web client makes API request
2.  API sends synchronous command or query directly to service
3.  Service processes command or query and synchronously replies

### Hybrid Flows

This hybrid approach uses both direct and proxy flows, and can work well with CQRS. The main benefit of this approach is to lighten the load on the message broker since it is only used for commands.

*   Command: use one of the HTTP Proxy API flows
*   Query: use the synchronous Direct API flow

### Web Socket Flows

If you're building a realtime web app whose users have highly available network connections and maintaining web socket connections is an option, this approach brings the full advantage of asynchronous messaging all the way to the web client.

*   Backend routes bi-directional messages (via load balancer or routing layer that converts between web socket and internal messaging protocols)
*   Frontend subscribes to queries that the backend will push to the client in realtime


Modern message brokers provide many important benefits to a distributed system...

Commands and Queries with a Message Broker


Domain vs Application Logic in Domain-Driven Design
===================================================

Problem vs Solution Space
-------------------------

When defining a business problem and planning its solution, keep the two conversations separate so that the problem is understood in as much detail as possible first. Business logic is part of the _Solution Space_ so should be discussed only after thoroughly understanding the problem.

*   The _Problem Space_ identifies the purpose, the "why".
*   The _Solution Space_ is the "how".

Common DDD terminology falls into each category:

#### Problem Space
- Domain
- Problem Domain
- Core Domain
- Subdomain

#### Solution Space
- Business Logic
- Business Rules
- Domain Logic
- Bounded Context


The terms Domain, Problem Domain, Core Domain are synonymous, as are Business Logic, Business Rules, and Domain Logic; these terms are used interchangeably.

After understanding the problem space, we can dig in to the rules that solve the business problem and encode this domain knowledge in explicit logic.

This _Business Logic_ solves the problem at hand but does _not_ address implementation details such as database technology, API definitions, web frameworks, etc. Our business logic should encode what rules are needed to solve the problem with language everyone can understand.

Keep Things Simple
------------------

### Business Logic

Business logic should be created in collaboration with business experts. Our _primary purpose_ as software engineers is to convert that logic into an implementation that can be executed.

So, let's **keep things simple** and separate our domain model from the application details. If we capture the business logic in an isolated area of our code, write the code using terms anyone can understand from the problem domain, and keep all application details out of this area, it is much easier to understand the purpose of the solution and update the business rules when needed.

How do we know what the domain logic is? Ask whether the code makes decisions that have business meaning. Any _business-critical_ decision should be done in the domain logic; everything else is wiring.

### Application Logic

This wiring code is the _Application Logic_. This layer sadly represents the bulk of complexity and effort in many systems, despite the fact that it is simply _necessary_ and has minimal impact on your business's competitive edge: a topic for another day.

The application logic orchestrates executing the business logic and all of the infrastructure details like persistence, UI, external system communication, etc. The application layer hosts an environment for executing domain logic; it collects and transforms external input into forms that the business logic uses to make decisions.

Usually the application layer communicates using primitive data types and converts them into domain-specific data types for clarity.

Application Architecture
------------------------

In order for the application layer to do its job of _orchestration_, it is helpful to further separate implementation details of specific technologies into adapters. Uncle Bob's Clean Architecture is a wonderful approach to this.

### Infrastructure

Infrastructure adapter implementations could include management of the following application concerns, for example:

*   Data persistence (e.g. in-memory, MySQL, file system, DynamoDB)
*   User notifications (e.g. email, SMS)
*   Web API (e.g. REST, GraphQL, web socket)
*   UI frameworks

### Dependency Inversion

Each infrastructure adapter can implement an interface that is defined in the application layer. During startup the implementation can be instantiated and dependency injected into the application layer. This isolates the application layer from infrastructure changes, which has the massive benefit of allowing an effort like swapping a database technology to not require a full application rewrite; only the infrastructure adapter needs to be re-implemented against the same interface.

This leads to a dependency flow where the detailed infrastructure implementations depend on the application layer, and the application layer depends on the domain logic. Our domain logic is kept pristine without any external dependencies.

*   Infrastructure Adapter (e.g. DB, REST API, external service communication) depends on ->
*   Application Logic (orchestrates the flow of requests and executes business logic) depends on ->
*   Business Logic (makes business-critical decisions) depends on nothing

References
----------

*   [Implementing Domain-Driven Design](https://www.amazon.com/Implementing-Domain-Driven-Design-Vaughn-Vernon-ebook-dp-B00BCLEBN8/dp/B00BCLEBN8) Vaughn Vernon
*   [Domain Driven Design Distilled](https://www.amazon.com/Domain-Driven-Design-Distilled-Vaughn-Vernon/dp/0134434420) by Vaughn Vernon
*   [The Clean Architecture](https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html) by Robert C. Martin 2012-08-13
*   [What is domain logic?](https://enterprisecraftsmanship.com/posts/what-is-domain-logic/) by Vladimir Khorikov 2016-08-25
*   [Services in Domain-Driven Design](http://gorodinski.com/blog/2012/04/14/services-in-domain-driven-design-ddd/) by Lev Gorodinski 2012-04-14
*   [Application layer vs domain layer?](https://softwareengineering.stackexchange.com/questions/140999/application-layer-vs-domain-layer)
*   [What is the difference between business and application logic?](https://softwareengineering.stackexchange.com/questions/293885/what-is-the-difference-between-business-and-application-logic)


When defining a business problem and planning its solution, keep the two conversations separate...

Domain vs Application Logic


When building a distributed system it's important to consider how to handle business flows that span across multiple aggregates or services. Firstly, we should try to simplify the flow so that a single user action only modifies one aggregate at a time. But sometimes that is not possible so we must decide how to coordinate our components.

_Orchestration_ is when a central controller manages the interactions between services; it explicitly models business logic in one place. In microservice terminology, this is called the _Saga_ pattern.

_Choreography_ is when services communicate by emitting and listening to events; it fully decouples communication.

Both of these approaches are useful in different situations. At the end of this post I boil this "it depends" down to a simple rule of thumb. I use terminology in the examples from domain-driven design and microservice architectures, but this is also relevant for monolithic systems.

For the examples below, imagine we're designing a travel booking system that allows end-users to book a trip across multiple travel services, such as airlines, hotels, rental car agencies, etc.

Cross-Context Example
---------------------

We need to communicate between two subdomains that have different business responsibilities, each managed by separate engineering teams. The subdomains are implemented as bounded contexts:

1.  **Booking** allows an end-user to book services with external providers.
2.  **CRM** allows external providers to manage their end-user customer base and communications.

In Booking, the end-user creates an account with a basic email. CRM eventually needs to know about that user account and create a profile, so that our providers can track their communication history and preferences.

### Orchestration

The Booking context, acting as an orchestrator, sends a direct command to CRM to create a new user profile.

_Pros:_

*   Booking's code explicitly shows each context with which it is communicating.

_Cons:_

*   Booking directly depends on CRM's command structure.
*   If another system wants to know about booking accounts, Booking must also be updated.
*   Booking's responsibility bleeds into needing to directly communicate with other contexts.

### Choreography

Using choreography, on the other hand, Booking emits a User Account Created event, then the CRM listens to the event and creates a new user profile.

_Pros:_

*   Booking has no need to know that CRM is listening.
*   If another system wants to know about booking user accounts, they simply need to listen to the event.
*   Booking's responsibility is solely focused on booking travel, and not communicating with other contexts.

_Cons:_

*   It takes more work to determine who is reacting to the events.
*   CRM needs to keep track of which events it has processed, and ensure it doesn't miss any.

### Winner for Cross-Context Communication?

**Choreography!**

*   Decoupling across bounded contexts is extremely important.
*   Specific business flows shouldn't span across multiple contexts (if they do, then the boundaries should be re-thought).
*   If CRM needs to create user profiles based on other events from other contexts, it is the CRM's responsibility to know about that and listen for them.

In-Context Example
------------------

Within the Booking bounded context, there are multiple services: Reservation, Provider, and Payment.

A user completes a reservation for multiple travel services then enters payment details. When the user confirms, the travel reservation is confirmed for each external service provider, then the payment is processed.

### Orchestration

The Reservation service, acting as an orchestrator, sends a direct command to the Provder service (which communicates to the external travel providers), and if all succeed, sends a direct command to the Payment service.

_Pros:_

*   The Reservation service code clearly shows the order in which the business logic is executed.

_Cons:_

*   The Reservation service directly depends on the Provider and Payment services' command structures.
*   When another provider or service needs to needs to be added to the reservation flow, the Reservation service must be updated.
*   The Reservation service responsibility includes communicating with the Provider and Payment services.
*   The Reservation service must handle compensating actions if a failure occurs anywhere in the flow.

### Choreography

Using choreography:

1.  The Reservation service emits a Reservation Created event.
2.  The Provider service handles this event and communicates with the external service providers.
3.  The Payment service handles this event and processes payment.
4.  Upon failure, compensating events must be emitted and handled where appropriate.

_Pros:_

*   The Reservation service doesn't need to know about the other services.
*   If another system wants to know about reservations, they just need to listen to the events.
*   Reservation's responsibility is solely focused on creating reservations in our system, and not coordinating with other services.

_Cons:_

*   It is hard to follow the basic business logic that a reservation entry results in a provider reservation and a processed payment.
*   To view the current state of the end-user's reservation and payment, one must directly query the Provider and Payment services, or listen to a large number of events, including compensations events if a failure occurred.

### Winner for In-Context Communication?

**Orchestration!**

*   The business logic is very clear and in one place; the Reservation service orchestrates it.
*   Even though there is direct service-to-service communication, it does _not_ need to be synchronous. We can still use decoupled asynchronous communication practices.
*   Minimal coupling within a bounded context is okay, because business logic sometimes must flow across multiple services.
*   Events could _still_ be emitted by each service when something happens, in case external contexts need to know.

Choosing an Approach
--------------------

The entire business logic flow is modeled explicitly in a single place with orchestration, whereas it emerges from the interaction between components with orchestration.

**Rule of Thumb:** _Use orchestration within a single bounded context to model a single business flow; use choreography to communicate across bounded contexts._

If you encounter business flows that span across multiple bounded contexts, making it difficult to use choreography, it might be an indicator to reconsider your domain boundaries.

References
----------

Directly relevant:

*   [Choreography vs Orchestration in the land of serverless by Yan Cui](https://theburningmonk.com/2020/08/choreography-vs-orchestration-in-the-land-of-serverless)
*   [GOTO 2020 - Automating Processes in Modern Architectures by Bernd Ruecker](https://www.youtube.com/watch?v=IBfuKA9Nc3M)

Where to capture orchestration logic:

*   [Domain vs Application Logic by Ben Johns](https://blog.bennyjohns.com/2020/12/domain-vs-application-logic.html)
*   [Services in Domain-Driven Design (DDD) by Lev Gorodinski](http://gorodinski.com/blog/2012/04/14/services-in-domain-driven-design-ddd/)
*   [What is domain logic? by Vladimir Khorikov](https://enterprisecraftsmanship.com/posts/what-is-domain-logic/)


When is it appropriate to use centralized orchestration versus event-driven choreography?

Orchestrate or Choreograph?


After trying out many of the other open-source options available, I decided to write a simple app to send files quickly and securely.

- Containerized web app written in Go & vanilla Javascript
- Upload a file and generate sharable links
- Single admin user authentication
- File expiration time can be set by duration or date
- Password-protect file access

[View Source](https://github.com/benjohns1/blinkfile)


BlinkFile


## Red, Green, Refactor
Test-Driven Development (TDD) is the practice of writing a failing unit test before writing the code to make the test pass, then cleaning up your code. This is the "red, green, refactor" cycle.
[![](/images/posts/TDD-Red-Green-Refactor.png)](/images/posts/TDD-Red-Green-Refactor.png)

It helps with clarity of thought, increases flow, and results in modular designs that are maintainable. You can probably tell I'm a fan. It's one of the practices that has helped me become a better software engineer over the years. While there are a lot of resources out there to learn about TDD, most of them gloss over an important aspect that makes it much more effective: it matters _what_ you try to test.

## Test the Behavior
For TDD to be effective it's important to test the _behavior_ of a component instead of its internal implementation details. This allows changing the component's internal logic without changing its tests. If the component's interface to other components in the system does not change and it can be used the same way as before, then the tests should not need to change either. This allows each individual component of the system to be enhanced, fixed, refactored, rebuilt using a different technology, or even completely replaced with minimal changes to the unit tests.

If the tests are tightly coupled to the internal implementation of a component, then it makes it difficult to modify the component without also modifying the tests. By focusing on only testing the behavior of a component, the unit tests should not be touched at all when changing its internal implementation. Instead, the tests become an important tool to verify that the internal change has not broken the component's behavior.

## Component Interfaces
Whenever we build a system, be it large or small, we should break it down into components that each have a unique responsibility. This modularity provides a lot of benefits like easier restructuring, simplicity, maintainability, testability, reducing the cognitive load of software engineers, and more. By identifying the components, we're also defining the boundaries between them and the interfaces they use to communicate with each other.

For each component we should strive to make it easy to change the internal implementation of the component, but we should make it a bit harder to change the interfaces between them. These interfaces are the components' API contracts. The hard part is figuring out the granularity of the components and their responsibility. This comes with experience and depends on many factors. There are many schools of thought on how to break down a system and identify its components (e.g. domain-driven design, event storming, event modeling, clean architecture, etc.), but I generally recommend designing it out visually and running through a few of the most important user scenarios. If you find the current design difficult and complicated, try a different one and keep iterating until you find one that works well.

With some practice, you'll get better at identifying reasonable component boundaries.

## Example Repository Implementation
One component boundary I find particularly useful encapsulates access to persistent data using the repository pattern. When building a feature that requires storing and retrieving data, it's usually beneficial to isolate the details of the database technology from the application logic. A repository's single responsibility is to manage the persistent data for an application (or a portion of it).

Let's build a simple repository in Go that shows the difference between the bad implementation-driven vs the good behavior-driven approaches to writing tests. We'll create a repository that saves and retrieves users. Each user only has a name, and just for simplicity we'll "persist" the data in-memory using a map, but you can imagine a DB engine, file system, or object store instead.

Both approaches are shown below:
- [Implementation-driven approach to testing](#implementation-driven-approach-to-testing) (Don't do it like this!)
- [Behavior-driven approach to testing](#behavior-driven-approach-to-testing) (Do it like this, instead!)

Here's the code from these examples if you want to compare the results side-by-side: [github.com/benjohns1/tdd-and-testing-behavior](https://github.com/benjohns1/tdd-and-testing-behavior)

### Implementation-driven approach to testing
_**Don't do it like this!**_

Here's what a user looks like:

<Code language="go" heading="File: user.go">
{`package app

type User struct {
	Name string
}
`}
</Code>

And here's a stub user repo to start with:
<Code language="go" heading="File: repo/users.go">
{`package repo

type Users struct{}
`}
</Code>

#### Save a User
Let's start with saving a user in the repo. We want a Save() method that accepts a user and returns a possible error. We'll write a test for a repo method Save() that we haven't written yet, so it will fail to compile initially.

##### Test: store a user by name to an empty map
Here's our first failing test, setup with a common table-driven test structure that I find useful:

###### Red Step 1
<Code language="go" heading="File: repo/users_test.go">
{`package repo

import (
	app "github.com/benjohns1/tdd-and-testing-behavior/implementation-driven"
	"reflect"
	"testing"
)

func TestUser_Save(t *testing.T) {
	type args struct {
		user app.User
	}
	tests := []struct {
		name    string
		repo    Users
		args    args
		want    Users
		wantErr error
	}{
		{
			name: "store a user by name to an empty map",
			args: args{
				user: app.User{Name: "Ender"},
			},
			want: Users{
				map[string]app.User{
					"Ender": {Name: "Ender"},
				},
			},
		},
	}
	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			err := tt.repo.Save(tt.args.user)
			if !reflect.DeepEqual(err, tt.wantErr) {
				t.Errorf("Save() error = %v, wantErr %v", err, tt.wantErr)
				return
			}
			if !reflect.DeepEqual(tt.repo, tt.want) {
				t.Errorf("Save() = %+v, want %+v", tt.repo, tt.want)
			}
		})
	}
}
`}
</Code>

This test will fail to compile, so let's write the code to make it pass:
###### Green Step 2
<Code language="go" heading="File: repo/users.go">
{`package repo

import app "github.com/benjohns1/tdd-and-testing-behavior/implementation-driven"

type Users struct {
	users map[string]app.User
}

func (u *Users) Save(user app.User) error {
	u.users = map[string]app.User{
		user.Name: user,
	}
	return nil
}
`}
</Code>

Alright, our test passes!

###### Refactor Step 3
The next step is to refactor our code and clean it up, but the code is so simple at this point let's move on to the next test cycle.

##### Test: store a new user by name to a map that already has a user

###### Red Step 1
<Code language="go" heading="File: repo/users_test.go (partial)">
{`// ...
		{
			name: "store a new user by name to a map that already has a user",
			repo: Users{
				map[string]app.User{
					"Ender": {Name: "Ender"},
				},
			},
			args: args{
				user: app.User{Name: "Valentine"},
			},
			want: Users{
				map[string]app.User{
					"Ender":     {Name: "Ender"},
					"Valentine": {Name: "Valentine"},
				},
			},
		},
// ...
`}
</Code>

This test will fail because of our overly-simplistic implementation from the first test, so let's fix that:
###### Green Step 2
<Code language="go" heading="File: repo/users.go (partial)">
{`func (u *Users) Save(user app.User) error {
	if u.users == nil {
		u.users = map[string]app.User{
			user.Name: user,
		}
	} else {
		u.users[user.Name] = user
	}
	return nil
}
`}
</Code>

This makes our test pass, but now we have a bit of cruft in our logic so let's simplify it:

###### Refactor Step 3
<Code language="go" heading="File: repo/users.go (partial)">
{`func (u *Users) Save(user app.User) error {
	if u.users == nil {
		u.users = make(map[string]app.User, 1)
	}
	u.users[user.Name] = user
	return nil
}
`}
</Code>
And our test is still passing!

##### Test: return an error if the user's name already exists
If a user already exists in our repo with the same name, we want an error to be returned so let's write a test for that:

###### Red Step 1
<Code language="go" heading="File: repo/users_test.go (partial)">
{`// ...
		{
			name: "return an error if the user's name already exists",
			repo: Users{
				map[string]app.User{
					"Peter": {Name: "Peter"},
				},
			},
			args: args{
				user: app.User{Name: "Peter"},
			},
			want: Users{
				map[string]app.User{
					"Peter": {Name: "Peter"},
				},
			},
			wantErr: fmt.Errorf("user name \\\"Peter\\\" already exists"),
		},
// ...
`}
</Code>

###### Green Step 2
<Code language="go" heading="File: repo/users.go (partial)">
{`func (u *Users) Save(user app.User) error {
	if _, exists := u.users[user.Name]; exists {
		return fmt.Errorf("user name \\\"Peter\\\" already exists")
	}
	if u.users == nil {
		u.users = make(map[string]app.User, 1)
	}
	u.users[user.Name] = user
	return nil
}
`}
</Code>

Notice the naive hard-coded error message? It's the simplest code that makes our test pass, but we want to clean that up in the refactor step:

###### Refactor Step 3
<Code language="go" heading="File: repo/users.go (partial)">
{`func (u *Users) Save(user app.User) error {
	if _, exists := u.users[user.Name]; exists {
		return fmt.Errorf("user name %q already exists", user.Name)
	}
	if u.users == nil {
		u.users = make(map[string]app.User, 1)
	}
	u.users[user.Name] = user
	return nil
}
`}
</Code>

Here's [the code for this approach](https://github.com/benjohns1/tdd-and-testing-behavior/tree/main/implementation-driven)

#### Is there a better way?
At this point, we've only written tests and functionality around saving a 
user. We could continue on to build out the functionality to retrieve users taking
 the same approach. It's great that we're using the basic TDD red, green, refactor cycle, but 
first think about what would happen if we needed to change the implementation. Could we add a user ID field index? Or upgrade to a persistent DB instead of in-memory? In either of these scenarios we'd need to _completely_ rewrite all of these tests because they are tightly coupled to the internal implementation of the user repo.

I want to show you a better way.

### Behavior-driven approach to testing
Let's start off fresh with our basic user and repo stub:

<Code language="go" heading="File: user.go">
{`package app

type User struct {
	Name string
}
`}
</Code>

<Code language="go" heading="File: repo/users.go">
{`package repo

type Users struct{}
`}
</Code>

#### Save a User
Just like before, let's start with saving a user in the repo. But instead of thinking in terms of the steps that the repo will need to take to accomplish this, think about how we want the repo to behave when it is used. How does this change our perspective of the Save() method? Well, what behavior do we want from the repo when we save a user? After we save a user, we should then be able to get it back! If we think of it this way, then it doesn't matter _how_ the repo stores the data. It only matters that if we save it, we can then get the same data back.

Let's use TDD to write our first behavioral test.

##### Spec: Should save a user to an empty repo
When writing this first test, we need to spend a bit more time thinking about the interface to get a user back out of the repo, too. In this example we're going to use a GetAll() function that returns a slice of all the users in the repo. We could also retrieve a single user by name, or some other way that aligns with the other use-cases.

###### Red Step 1
Notice the package name is repo_test instead of just repo. We want to test the repo's _interface_ so we only want to access its publicly exported methods. Go allows appending _test to the package name to enforce this.
<Code language="go" heading="File: repo/users_test.go">
{`package repo_test

import (
	app "github.com/benjohns1/tdd-and-testing-behavior/behavior-driven"
	"github.com/benjohns1/tdd-and-testing-behavior/behavior-driven/repo"
	"reflect"
	"testing"
)

func TestUser_Save(t *testing.T) {
	type args struct {
		user app.User
	}
	tests := []struct {
		name      string
		repo      repo.Users
		args      args
		wantErr   error
		wantUsers []app.User
	}{
		{
			name: "should save a user to an empty repo",
			args: args{
				user: app.User{Name: "Ender"},
			},
			wantUsers: []app.User{{Name: "Ender"}},
		},
	}
	for _, tt := range tests {
		t.Run(tt.name, func(t *testing.T) {
			err := tt.repo.Save(tt.args.user)
			if !reflect.DeepEqual(err, tt.wantErr) {
				t.Errorf("Save() error = %v, wantErr %v", err, tt.wantErr)
				return
			}
			got, err := tt.repo.GetAll()
			if err != nil {
				t.Fatal(err)
			}
			if !reflect.DeepEqual(got, tt.wantUsers) {
				t.Errorf("After Save(), GetAll() = %+v, wantUsers %+v", got, tt.wantUsers)
			}
		})
	}
}
`}
</Code>

###### Green Step 2
To make this test pass, we need both a Save() and GetAll() method. Remember, we're just writing the bare minimum code to make the test pass in this step!

<Code language="go" heading="File: repo/users.go">
{`package repo

import (
	app "github.com/benjohns1/tdd-and-testing-behavior/behavior-driven"
)

type Users struct{}

func (u *Users) Save(user app.User) error {
	return nil
}

func (u *Users) GetAll() ([]app.User, error) {
	return []app.User{{Name: "Ender"}}, nil
}
`}
</Code>

###### Refactor Step 3
Now we refactor our naive code into little better solution. And we have our test to verify it is still correct:

<Code language="go" heading="File: repo/users.go (partial)">
{`type Users struct {
	user app.User
}

func (u *Users) Save(user app.User) error {
	u.user = user
	return nil
}

func (u *Users) GetAll() ([]app.User, error) {
	return []app.User{u.user}, nil
}
`}
</Code>

##### Spec: Should save a user to a repo that already has a user in it

###### Red Step 1
<Code language="go" heading="File: repo/users_test.go (partial)">
{`// ...
		{
			name: "should save a user to a repo that already has a user in it",
			repo: func() repo.Users {
				r := repo.Users{}
				if err := r.Save(app.User{
					Name: "Ender",
				}); err != nil {
					t.Fatal(err)
				}
				return r
			}(),
			args: args{
				user: app.User{Name: "Valentine"},
			},
			wantUsers: []app.User{
				{Name: "Ender"},
				{Name: "Valentine"},
			},
		},
// ...
`}
</Code>

###### Green Step 2
<Code language="go" heading="File: repo/users.go (partial)">
{`type Users struct {
	users []app.User
}

func (u *Users) Save(user app.User) error {
	u.users = append(u.users, user)
	return nil
}

func (u *Users) GetAll() ([]app.User, error) {
	return u.users, nil
}
`}
</Code>

###### Refactor Step 3
Let's make our repo encapsulation a bit better by not allowing mutation of the repo's internal slice:

<Code language="go" heading="File: repo/users.go (partial)">
{`func (u *Users) GetAll() ([]app.User, error) {
	out := make([]app.User, 0, len(u.users))
	for _, user := range u.users {
		out = append(out, user)
	}
	return out, nil
}
`}
</Code>


##### Spec: Should fail if a user's name already exists

###### Red Step 1
<Code language="go" heading="File: repo/users_test.go (partial)">
{`// ...
		{
			name: "should fail if a user's name already exists",
			repo: func() repo.Users {
				r := repo.Users{}
				if err := r.Save(app.User{
					Name: "Peter",
				}); err != nil {
					t.Fatal(err)
				}
				return r
			}(),
			args: args{
				user: app.User{Name: "Peter"},
			},
			wantUsers: []app.User{{Name: "Peter"}},
			wantErr:   fmt.Errorf("user name \\\"Peter\\\" already exists"),
		},
// ...
`}
</Code>

###### Green Step 2
<Code language="go" heading="File: repo/users.go (partial)">
{`func (u *Users) Save(user app.User) error {
	for _, current := range u.users {
		if current.Name == user.Name {
			return fmt.Errorf("user name \\\"Peter\\\" already exists")
		}
	}
	u.users = append(u.users, user)
	return nil
}
`}
</Code>

###### Refactor Step 3
Clean up the naive error message:

<Code language="go" heading="File: repo/users.go (partial)">
{`func (u *Users) Save(user app.User) error {
	for _, current := range u.users {
		if current.Name == user.Name {
			return fmt.Errorf("user name %q already exists", user.Name)
		}
	}
	u.users = append(u.users, user)
	return nil
}
`}
</Code>

Here's [the code for this approach](https://github.com/benjohns1/tdd-and-testing-behavior/tree/main/behavior-driven)

## Conclusion
Take a look at your implementation using the behavior-driven approach. For the same amount of effort, we've also implemented retrieving users from the repo, too. Now let's ask the same questions we did earlier: Could we add a user ID field index? Or upgrade to a persistent DB instead of in-memory? Both of these scenarios could be accomplished without modifying our existing tests (apart from maybe some test setup to implement a real DB backend).

Our behavior tests are much more robust, and they're validating what we really care about: the behavior of the repo. If the interface changes we _want_ the tests to break because that means we've changed the repo's behavior. If we need to refactor the internals of the repo, we can use our tests to be confident that we haven't broken any existing use-cases.

TDD is an incredibly useful practice but only if you are testing behavior!


The importance of testing behavior when using test-driven development

TDD and Testing Behavior


## Acceptance Test-Driven Development
Practicing TDD increases flow and improves design as long as we [test the behavior](/posts/20240124-tdd-and-testing-behavior) of each component instead of its implementation details. Unit testing with TDD focuses on the lower component level, but we can also apply the practice at a higher level by first writing an acceptance test for each user scenario (i.e. acceptance criteria) like this:

[![](/images/posts/ATDD-Double-Loop-TDD.png)](/images/posts/ATDD-Double-Loop-TDD.png)
Acceptance Test-Driven Development (ATDD) is also known as Specification by Example, Double-Loop TDD, Story Test Driven Development, or Behavior Driven Development, though some would argue there are differences between these terms.

## The CI Problem
### Part 1
How do we frequently integrate small, incremental changes within the inner TDD loop? Each change is only a tiny slice of behavior, so the scenario will fail until all slices are complete. The deployability of our software should depend on the success of our acceptance tests, so our pipeline will fail until the scenario is completely functional.

In the past I've done things like comment out the acceptance test I'm currently working on. But that is error-prone, looks messy in version control, and makes it more difficult for others on the team to run the tests. I've also waited until the entire scenario is complete before pushing my changes. But that is not integrating fast enough: I want to be able to safely merge my code after every short TDD cycle.

### Part 2
If we solve this particular issue, then it causes another one: merging rough and unfinished features into trunk increases the cost of accidentally enabling the feature flag and breaking production.

## Proposal
We can solve this by setting up our CI/CD pipeline to integrate aspects of feature flagging with acceptance testing. Ideally we are able to fully control everything within our test specifications in version control without switching to an external system and breaking our focus.

When writing acceptance tests, we tag the unfinished tests as "pending" and our pipeline only gates deployment by running completed tests. We can still run the pending tests locally during development. After a scenario is complete, we remove the "pending" tag and the pipeline now runs that test. Most test frameworks provide tagging functionality.

But now that we're deploying code during development that is not gated behind an acceptance test, we're introducing more risk to production. We can leverage an integration with our feature flagging system to reduce this risk by setting up pipeline rules to:
1. protect features from being enabled in production before they are ready
2. protect unfinished code from being deployed behind an enabled feature flag

Additionally, we could automatically register feature flags when a feature is complete to reduce some toil.

### New Feature Workflow
Here's a walkthrough of the workflow when building a new feature.

#### Developer Experience
1. Break a feature into user scenarios (i.e. acceptance criteria) with product stakeholders
2. Stub out each scenario as an acceptance test, tagged with the feature flag name and "pending" 
3. Write a single acceptance test
4. Use the TDD cycle to deliver incremental changes behind the feature flag and make the acceptance test pass
5. Remove the "pending" tag from the completed acceptance test
6. Demo and capture feedback about the new behavior from stakeholders
7. If the feature is not yet complete, go back to Step 3

Notice that at _any_ step in this process, you can safely merge into trunk without breaking the pipeline. Here's the relevant logic in the pipeline that happens automatically:  

#### Pipeline Logic
1. Parse acceptance test tags
2. If the feature flag doesn't exist in the feature flag system, create it
3. If the test is "pending"
   - And its associated feature flag is enabled for any production users, fail the pipeline
   - Otherwise, disable or hide the flag in the feature flag system, so it cannot be enabled for production users
4. Run acceptance tests except those that are pending

### Changing Existing Features
When changing the scenarios for an existing feature, we also now have some protection against breaking a feature that has already been released. If we need to change the behavior of a released feature, we are enforcing that it is implemented under a _new_ feature flag, so we don't break the user experience during development.

### Reducing Toil
Integrating your feature flag solution with your pipeline in this manner also opens up some additional automation opportunities. For instance, you could remove feature flags from the system when a developer removes them from the scenario tags, but only if the feature has been released for at least 2 weeks and there are no instances of it in the code. Or you could log a warning if a feature has been released for over 4 weeks without being removed from the code yet.

### Further Reading
- [Coding Is Like Cooking » Blog Archive » Outside-In development with Double Loop TDD](https://coding-is-like-cooking.info/2013/04/outside-in-development-with-double-loop-tdd/)
- [Acceptance Testing: What Are Acceptance Test and why They Matter](https://semaphoreci.com/blog/the-benefits-of-acceptance-testing)
- [Acceptance testing your Go CLI. I’m a fan of Acceptance Test Driven… | by William Martin | Medium](https://medium.com/@william.la.martin/acceptance-testing-your-go-cli-d992926bcfe5)
- [Double-Loop TDD](https://www.sammancoaching.org/learning_hours/bdd/double_loop_tdd.html)
- [Acceptance Test Driven Development (ATDD) | Agile Alliance](https://www.agilealliance.org/glossary/atdd/)
- [What Is Acceptance Test Driven Development (ATDD)](https://www.nimblework.com/agile/acceptance-test-driven-development-atdd/)
- [How to Write Automation Tests for Feature Flags with Cypress.io and Selenium | by Todd Seller | Engineers @ Optimizely | Medium](https://medium.com/engineers-optimizely/how-to-write-automation-tests-for-feature-flags-with-cypress-io-and-selenium-98493b5c46bc)


How can we continuously integrate small changes while practicing acceptance test-driven development?

Continuous Delivery with ATDD and Feature Flags


In recent years, I've developed distributed systems using domain-driven design, event storming, continuous delivery, and test-first development practices. In my experience, systems designed using these techniques are more likely to provide real value to customers, are easier to maintain, and are more fun to work on! There's no panacea against accidental complexity, though, and it takes experience, skill, and a solid team to create a successful system. In this post, I articulate some of the lessons I've learned and the design concepts I've found to be essential when building evolvable, distributed systems. Upcoming posts in this series will dig deeper into tactical patterns and specific component architecture concepts.

## Don't Build a Distributed System
First-and-foremost, do not distribute across a network if you do not need to! This advice is repeated over-and-over by those that went through failed microservice transformations or startups that attempted to begin with a microservice architecture. As software engineers, one of our primary goals is to reduce unnecessary complexity. Distributing pieces of your system across a network increase complexity, so don't do it without a good reason. One such good reason is: a *tangible* need to scale because specific bottlenecks are or will be preventing the system from providing real value. An example of a bad reason is: we're going to capture so much market share after launch that we need to be able to scale everything right off-the-bat!

Hint: You can still design a well-architected modular system without incurring the costs of distributed messaging up-front.

### MVPs
If you're building an MVP and have yet to prove product-market fit, don't build a distributed platform. Build the MVP well enough to onboard a small batch of initial customers and focus on delighting them. Once you've proven the product is valuable, then evolve your architecture towards the scalability and growth requirements. That said, there may be very good reasons to distribute some parts of your MVP if it is necessary to prove market fit (e.g. scalability, reliability, compliance, geography, isolating 3rd party integrations, etc.), but the goal should be to keep it as simple as possible while still viable. I think of an MVP as an experiment to prove the product solution against the current market forces, and you need to uncover the results as soon as possible to avoid waste. There are a couple approaches to building an MVP that I've seen work well.

One approach is to build the MVP as quickly and cheaply as possible while ignoring architectural quality. This really only works if the app you are building is simple enough because, as we all know, development will inevitably slow down over time with poor architecture. But this can work well if you expect to prove market fit quickly. I've seen this frequently when an entrepreneur wants to try out an idea and outsources development to a contracting firm to build the MVP. The key with this approach is that after you've proven the product's viability, you'll likely need to invest in a major architectural overhaul with an in-house development team that designs it with quality in mind at an appropriate level of scalability. This transition can be extremely difficult because just as the product is gaining traction, you are spending resources on rebuilding instead of providing additional user value.

Another approach is to build the MVP quickly, but with architectural quality in mind from the start using an in-house development team. This may require more initial investment, but provides a more stable product, smoother delivery cadence, and a healthier team in the long run. An experienced product and engineering team will help reduce risk as long as they are laser-focused on early engagement and rapid feedback with initial customers throughout MVP development. The system design concepts I describe here can be applied to an MVP built in this way and will help pave the way for a maintainable, scalable, and evolvable architecture.

## Modularize First, Then Distribute
Whether you're building an MVP or redesigning components of an existing system, if you create modular components with single responsibilities in your system, you do not need to separate those components across the network initially. Each component should have high cohesion within itself and low coupling to other modules with a clean interface boundary. It is cheaper and easier to modify the boundaries between components that are not distributed. Over time these boundaries should stabilize; then, when the need arises an individual component can be extracted into its own service. If designed well, the business logic should not need to change at all and the application logic should only need minor changes. The infrastructure should be the only aspect that needs significant rework. This ensures we're only adding complexity when it is needed.

Identifying the most useful component boundaries is hard and depends on many factors. Expect those boundaries to change over time as the team gains experience through experimentation.
## Design Concepts
Below are some of the most important design concepts I use when creating a modular system. Open and consistent communication with stakeholders is necessary to build a clear understanding of the business domain, which is essential. Markets change over time, so the problems the system addresses should be refined over time. And the solution will change over time as everyone involved learns new things. So we must embrace these changes and expect the system boundaries and models to evolve. Continually practicing techniques like event storming or event modeling throughout a product's lifecycle can help normalize and capture these changes.
### Clean Architecture
Robert Martin (Uncle Bob) coined clean architecture from aspects of hexagonal and onion architecture. It is the idea that our code should be organized in layers with the dependencies pointing in one direction: towards the business logic in the center. This approach builds on the SOLID design principles. If you're unfamiliar, see [Clean Architecture and SOLID Cheat Sheets](https://www.bennyjohns.com/posts/20201101-clean-architecture-and-solid-cheat-sheets)  for a quick overview and visualization.

Our core business logic (i.e. domain logic) at the center describes the valuable purpose of the component, boiled down into clear and concise language with minimal dependencies. The application logic is the next layer out from the center that wires the entire application together and should only directly depend on the core business logic. The outer infrastructure layer includes all additional concerns around state management, persistence, communication, and presentation. This outer layer tends to be where most engineering time is spent, even though the real business value is encoded in the business logic. I use this style of architecture within individual services to clearly separate the business logic from all the necessary complexity to make it run.

### Ubiquitous Language
Ubiquitous language is a fancy domain-driven design (DDD) term that simply means everyone should agree on the language used to describe the business logic, and that language should be used everywhere, including in code. As software engineers, we should name our variables, classes, functions, services, database tables, etc. using the same language that our product team members, customers, and business leaders use. We should also strive to design our data models to reflect the real-world concepts. This significantly reduces misunderstandings, helps avoid over-abstraction, speeds up new team member onboarding, and increases code maintainability. This means we need to communicate with others and decide on the proper terminology and definitions, which is part of our job as software engineers!

### Bounded Contexts
A bounded context is another DDD term that describes where the boundary is drawn around a domain concept. Within a single bounded context, the same ubiquitous language and the same domain models should be used. Separate bounded contexts may (and likely should) use a different ubiquitous language and domain model. Bounded contexts define important boundaries in a system. They provide clarity of the solution and allow separate teams to work in parallel with others with minimal dependencies. For example, you might have separate Storefront, Payment, and Shipping bounded contexts for an online store.

### Aggregates
Another important DDD concept is the aggregate, which is fundamentally a collection of entities that represent part of the domain model. An aggregate can be stored, modified, and passed around according to the business rules. What's most important about an aggregate, however, is the concept that it defines a transactional boundary. Each change to an aggregate should be an atomic transaction. This also means that any business rule that requires consistency must only operate on a single aggregate. This may initially sound trivial, but it is the grounding concept that allows a distributed system to manage state reliably and consistently.

If a new business rule requires data from multiple aggregates to make a decision, there are a few options to discuss with the stakeholders:
1. If the rule is simply nice-to-have, you can make best efforts to follow it, but allow it to be violated occasionally. For example, I've taken this approach when asserting a unique name across a user's list of items. If uniqueness is only a convenience to avoid confusing one item with another and the user can simply rename an item later if there's a collision, you can simply make a best effort to enforce uniqueness when the name is applied. It's okay if a duplicate name sneaks in due to two items being created at the same time, for instance, as long as the UX tradeoff is acceptable.
2. If the rule is an invariant (i.e. requires strict consistency), you can rework your domain model to increase the size of the aggregate to include everything the rule needs. An invariant rule is one that the system must enforce at all times; violations would result in the system being inconsistent.
3. If the rule can be eventually consistent, you can define a compensating process to fix a violation. An eventually consistent rule may be violated temporarily, but will eventually be fixed.

The first "nice-to-have" option is the simplest, but comes with a UX tradeoff. If that's not possible, strive for the second "invariant" option, as long as the new model doesn't make the aggregate too large and the rework effort is not prohibitive. The third "compensating process" option can add significant complexity so should be minimized, but is usually unavoidable in complex domains. If you need to use the third option, these approaches are listed in order of increasing complexity:
1. Define a manual process to compensate for the inconsistency. This requires some sensing mechanism to determine when the rule has been violated and a documented process for fixing it. This can work if the violation only occurs occasionally and may be a good place to start, but adds busy work so should be automated when possible.
2. Write automated compensation logic. Automate the manual process in the previous approach. This works if the compensating logic is fairly simple and can be run on a schedule.
3. Implement the saga pattern in a process manager. This works well if the business rule describes a long-running process with multiple steps involved. This requires each step in the process to have a compensating action that reverses the step, to allow rolling back if any subsequent step fails. If compensating actions continue to fail, there also should be a manual escalation process.

### Commands, Queries, and Events
All messages between modules should be explicitly categorized as a command, query, or an event. This simplifies reasoning about distributed communication patterns, allows for a much clearer understanding of how messages are processed, and enables some basic guarantees around idempotency and state changes.

A command is a message that requests a component to change the state of an aggregate in some way. A query is a message that requests data from a component without changing its state. An event is an asynchronous message to subscribers indicating something happened in the system (e.g. an aggregate's state changed or a failure condition was triggered). Events are powerful tools in evolutionary architectures but there's a lot of nuance around their usage that I will discuss in upcoming posts.

Commands and queries may be synchronous or asynchronous. Events are, by definition, asynchronous. For high throughput and scalability, asynchronous messaging is desirable, but there's additional complexity involved in handling concurrent processing and failures. This complexity can be mitigated with robust messaging libraries and knowledge sharing. Synchronous messaging with request/response style communication is simple and can be a great choice for many situations, but it will eventually result in idle resources and unnecessary bottlenecks so should be avoided if high throughput is necessary.

### Team-First Architecture
If you're building a large system, use the Inverse Conway Maneuver to first organize your teams along the desired architectural boundaries. Team Topologies by Matthew Skelton and Manuel Pais describes how to structure your organization with four team types and their interactions. Most of your cross-functional teams should be focused on delivering user value directly by becoming experts in one or more domains, solving problems within their bounded contexts, and reacting to user feedback. These teams are **stream-aligned**. Three additional team types help stream-aligned teams by: providing a stable **platform** for the other teams to leverage, **enabling** other teams through knowledge transfer, and offloading cognitive load by owning a specific **complicated subsystem** leveraged by other teams.

### Think Experimentally
Over time, I've learned to see the software delivery process as a series of experiments. Every change to a production system can be thought of as a small experiment using the scientific method, for example:
1. **Observe** the current system and market landscape to identify an opportunity.
2. **Hypothesize** that a particular feature will leverage the opportunity.
3. **Predict** that customers will use the new feature.
4. **Experiment** by implementing and releasing it to your users.
5. Capture user feedback and metric **results** as input for further experimentation.

This leans in to modern continuous delivery practices and requires working iteratively (running one small experiment at a time), incrementally (by creating small, cohesive components that can be worked on in isolation from the entire system), and relentlessly seeking feedback (through metrics and user collaboration). Dave Farley's book Modern Software Engineering covers this approach along with other excellent software engineering concepts.

## Reading and Next Steps
The following books were formative to my approach designing systems:
- [Team Topologies](https://teamtopologies.com/)
- [Building Evolutionary Architectures | Thoughtworks](https://www.thoughtworks.com/insights/books/building-evolutionary-architectures)
- [Modern Software Engineering: Doing What Works to Build Better Software Faster: Farley, David](https://www.amazon.co.uk/gp/product/0137314914)
- [Implementing Domain-Driven Design 1, Vaughn, Vernon](https://www.amazon.com/Implementing-Domain-Driven-Design-Vaughn-Vernon-ebook-dp-B00BCLEBN8/dp/B00BCLEBN8)
- [Domain-Driven Design: Tackling Complexity in the Heart of Software: Eric, Evans](https://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software-ebook/dp/B00794TAUG)
- [Clean Architecture: A Craftsman's Guide to Software Structure and Design: Martin, Robert](https://www.amazon.com/Clean-Architecture-Craftsmans-Software-Structure/dp/0134494164)

Stay tuned for upcoming posts in this series where I describe essential tactical patterns and component architecture concepts for building evolvable, distributed systems.


The essential design concepts I use when developing an evolvable, distributed system.

Foundant Technologies

Solutions for Philanthropy

Related Posts