Accidental Complexity

You Just Didn’t See it Coming

You had plans for just one of something. Soon you have two of something. Now you have three or four, and that something that was easy and manageable becomes a real struggle.

I once joked that if my wife and I were planning to have two children, we should just have them both at the same time - twins! Put all the effort in at once. It would be easier, right? Not so fast!

The more of something you accumulate, the more complexity increases, sometimes exponentially.

• Getting a babysitter for one child may be easier and safer than getting a babysitter to watch two babies.
• Pushing a baby around in a stroller in the park is a breeze, but that double-wide stroller is much more of a struggle.
• You get one baby fed and back to bed and then the other one wakes. When are you going to sleep?

This same thing happens in software development, but it doesn’t take 9 months to arrive. In most cases, a haphazard decision during development - a decision taking less than 9 seconds - can set you on a course for a major rewrite one day when you find yourself buried in technical debt.

Some of this complexity is inherent. Other complexity sneaks in and can be identified as accidental complexity. A seemingly innocuous decision that seemed like a good idea turns into a costly liability. You just didn’t see it coming!

In simple terms, if you take the complexity of an application, and subtract the inherent complexity - that which is required by the problem domain - you are left with accidental complexity.

Let’s look more deeply into inherent and accidental complexity and why accidental complexity can be costly, yet avoidable. Let’s discuss a typical path to complexity and then consider an example.

The Path to Accidental Complexity

How accidental complexity comes into being can itself be complex.

Consider this flow, from idea to building out the idea:

1. Start with the idea
2. As you describe it by defining it on paper, you uncover gaps and resolve the gaps with complexity
3. As you describe it to someone else, you uncover additional gaps and resolve the gaps with complexity
4. As you interact with a “paper prototype”, you uncover gaps and resolve the gaps with complexity

In some cases, the resulting complexity may just simply be “inherent complexity.” Inherent complexity is the complexity that must exist because a problem is just that hard. Uncovering gaps leads to revealing the entire scope of the complexity of a problem.

Later, I’ll look at how to avoid accidental complexity. First, let’s see how accidental complexity creeps in.

Creating Charts Example

Consider a graphical interface that allows a user to create charts.

The depiction of the above flow might look like this:

1. (idea): Let users drag different kinds of charts to a canvas and define properties for each chart …
2. (describe it on paper): We allow 5 different chart types and here’s a list of each of the properties for each chart type …
3. (describe it to someone): Using these 5 chart types, this one allows only 1 data series. This other one allows stacked bar charting, etc…
4. (paper prototype): One specific chart type has properties in a pop-up dialog and another chart has properties flying out from the side margin …

You see that as you went along, you developed a greater and greater understanding of the problem and how to solve it within software. That’s perfect! You want that kind of result.

Something else happened along the way.

As the idea evolved, the designer added unnecessary complexity. The designer decided that certain charts would allow the entry of properties one way, but other charts another way.

In interacting with the prototype, it might have seemed that such a design made the most sense. Testing should reveal unnecessary complexity.

Testing the Design

There’s another step that is absolutely crucial and often missed. A fifth step should be testing the design. Without testing the prototype, the main cost avoidance benefit of the prototype is missed. So many designers skip this step. Not to overstate it, but it is kind of a tragedy.

The paper prototype allows you to see the design in action, but at some point, you need to step back and look at the problem with fresh eyes. It is best to do this after some period of time has passed, like a few days, when possible.

As you consider the problem, you might recognize that the user will understand the application better by standardizing the entry of properties into one way for all charts. How does this help?

• Reduced support costs due to standardized usage, greater satisfaction in the use of the application, and likely less customer churn because the paying user is more satisfied with the product (if similar steps are taken for the whole application)
• Reduced development costs. Having one way of entering properties likely allows for the creation of this one entry spot that can be shared among all the charts. Even if there are different properties, there’s a likelihood of shared structures. It also sets the stage for another simplifying approach I’ll discuss later.

To be sure, this was a simple example. I kept this simple in order to avoid paragraphs of setting the stage. You might think, “Well, it’s obvious. Of course, you use a standard widget for the entry of properties.” Yet, it’s amazing how often blatantly simple things like this show up in poor designs. That’s fine if it happens before testing, but it should never be allowed to exist after testing the design.

Side note: This same thing should happen at the technical layer. Program code can itself have accidental complexity. Probably, that happens more often then it does at the application level. I’ll discuss this more later.

How to Avoid - Business Design Stage

How do you avoid this accidental complexity when designing? There are multiple ways in which to knock out accidental complexity, and I’ll look at each in turn.

• Looking for commonalities and ways in which to create a standard approach to similar issues
• Invent a completely different way to solve the problem
• Consider if there might be a way to eliminate some inherent complexity
• Decide if there’s an “offline” or manual approach to certain elements

Commonalities

As in the example, where there were multiple ways of entering properties, what aspects if any, can be combined into a single “widget” of functionality. This widget likely becomes a chunk of code when you get to the technical design, but it also becomes a standard block of functionality that the user understands.

The user is more easily able to find what the user is looking for. This becomes a really nice side benefit. The user experience is enhanced and this is something you should always look to improve. By doing this, you’ve also reduced development costs, or time to market, another great benefit.

Accidental complexity expands quickly when these commonalities are duplicated in many places. This duplication represents complexity that was unnecessary and not part of the inherent complexity.

Invent a Different Way

It’s important to iteratively flesh out your understanding of a problem as depicted earlier. Start with an idea and brainstorm a solution, write it down, discuss it with others and so on.

Once this problem is understood, it is time to get really creative! At this point, go wild in your thinking about alternatives. Think of what seems as the most ridiculous things and consider all kinds of what-if scenarios.

Maybe you find out that charts are not as helpful as a different solution? What if you really just need to depict a trend rather than provide 5 different kinds of charts so the user can determine the trend manually?

Sometimes you get locked into a legacy way of doing things, and you don’t at first realize that there’s a modern data science or machine learning approach you could apply, for example.

Of course, the opposite is also just as likely. You want to use the latest “cool thing” and you make the application more complex for the sake of the prestige of using a specific technology.

If that prestige is necessary to sell the product, then there may be an engineering compromise that helps create product viability? In such a case, you might argue that the added complexity this creates is inherent because it is needed to sell the product? That enters the realm of philosophy, so let’s skip that detour and leave that discussion for another time.

Back to our chart example. It may turn out that you have an easier problem to solve by changing the approach. The user doesn’t have to look at a bunch of charts and infer something. You can simply tell the user, skipping a big step of manual labor for the user. The easier problem is less expensive to implement. Everyone wins in such a case!

Don’t limit yourself to reinventing the entire problem. Maybe there’s just a single step that can be reinvented. This can also represent a nice win, even if it isn’t a home run.

Check this out. If there’s an easier way to do something, then the supposed inherent complexity is actually accidental complexity! The very act of removing complexity reveals accidental complexity.

You can be absolutely sure you’re dealing with inherent complexity, but as you wring out more drops of complexity, each drop flows out as accidental complexity!

Eliminate Inherent Complexity

In the previous point, you invented a different way of doing things, which eliminated the complexity inherent in the problem, as defined. That’s a huge win, but often you can’t invent an entirely new way of doing something.

The way you do something often involves multiple steps. As in any process analysis, you should examine if there might be steps that can be eliminated or somehow combined and simplified.

Determine if there’s some way of reducing the scale of the problem that simplifies the user experience, and the effort of developing the solution in software.

Sometimes, just examining if a step can be eliminated causes you to realize something not previously considered possible that becomes a winning feature in the application.

As with the previously suggested way of removing accidental complexity, if a step in the inherent complexity can be removed, then that step represents accidental complexity.

Is There An Offline or Manual Solution?

Just because you’re building a software application, it doesn’t mean that everything needs to be done within the application. It depends, of course, upon the goals of the software, your users and their expectations, and other considerations.

This step of reduction, like all the others, should be judicious and carefully considered. It may be prudent to consult with your user base, if that’s possible, and to test user satisfaction when taking this approach.

In some cases, such as in the case of providing a complex interface for analyzing data, it may be better to just provide a means for downloading data to a spreadsheet. This would be particularly the case for users who are accountants, data scientists, statisticians and some engineering disciplines. There is a high likelihood they would all be more satisfied with that.

Imagine creating something akin to a spreadsheet in a web application. That would take a very long time to create and may not help the users in any way. In doing so, you’ve accidentally created a bunch of complexity.

How to Avoid - Technical Design Stage

There’s definitely some overlap in both the business and the technical design stage. There’s also lots of deep technical topics I could go into, but won’t do that here. This list is a high-level list of items you can knock out.

• Leverage commonalities in the business design
• Avoid copying and pasting of code
• Use frameworks and previously written code
• Document complexity within the code

Leverage Commonalities

The one-two punch of design is looking for commonalities in the business design and then making use of that in implementation. Make routines that can be used in multiple locations. Those routines are:

• Written once
• Tested once
• Debugged once
• Regression tested once

Of course, these steps may occur more than once and this is overly simplified, but the point is that creating redundancy requires you to pay for the redundancy for as many places as that redundancy occurs. A bug found in one place likely means that same bug occurs in the other places it is used. That also means, that fixing it in the one place, likely fixes it in all the other places.

As with the business design, creating more than one of something, when one would suffice, is accidental complexity.

Avoid Copying and Pasting of Code

Copying code from one place to another may feel like leveraging commonalities, but it is a horrible practice that creates accidental complexity. Initially, the code is exactly the same. Early on, when one finds bugs in the copied code, those bugs are fixed when copied to all the redundant areas. So far so good, but it goes down-hill quickly.

As changes are made, over time, the redundantly copied code may change in some areas, but not in others. If it were a common routine, that routine could be enhanced to accommodate these variations. Bugs appearing in the copied code are often not corrected in all places.

In fact, each bit of copied code must be visited and fixed in all places when a bug is discovered in one of the copies or the original. That changed code now looks different. The programmer needs to read, understand, change, and test each bit of copied code. It becomes a quite laborious task. Let’s hope the programmer finds all occurrences in often hundreds of lines of code.

Copying and pasting code is an awful practice that most professional developers should never accept, yet it does happen quite often. When it does, the practice opens the doors for accidental complexity.

Use Frameworks

Frameworks are previously written bits of code that perform some task. Stability can vary, but generally speaking, frameworks are typically well tested by people around the world, and used in various use cases. When bugs appear, they are eventually addressed. When yet not addressed, an issues list is available and those trouble areas can be avoided or workarounds developed in wait for a better fix.

By using frameworks, you are able to avoid writing a lot of code that would then need to be tested and debugged as well as maintained over time. Many frameworks are available at no cost through open source development.

How this relates to accidental complexity, is that these frameworks are often scrutinized by a number of professionals who have determined good ways, if not the best ways, to write and structure technical activities. So by using these frameworks, you are able to more easily avoid accidental complexity.

When rushed to develop some functionality for a customer or employer, a developer might not spend the amount of time that would be required to meaningfully design some functionality that is quite well designed in a framework. A sloppy design will almost always produce some accidental complexity.

Document Complexity

Some concerns of accidental complexity are future concerns. You might design some functionality and only include inherent complexity, but the inherent complexity may in and of itself be complex.

In such a case, it is quite important to document the functionality that was created. This can both be in the form of business and technical documentation.

By having such documentation, it is much easier for somebody to come along later and recognize what was created. When future developers are unable to determine what was done previously, that often creates opportunities for functionality to be created around the existing functionality, and duplication or other accidental complexity is created.

In her article 6 Reasons Why Reading Code Is More Important Than Writing, Nazlican Kurt rightly points out that, “Reading code is like reading another programmer’s mind and logic.” This is no easy task. She also points out that, “All people think differently.” How I choose to tackle a problem using programming, will likely be different from the way another programmer tackles the same problem. Simple problems are often very similar and easy to decipher, but the more difficult the problem to solve, often the more difficult the code is to read due to this very issue.

When documentation is not available and the program is very difficult to understand, it becomes almost impossible to determine what functionality should be removed, in the case where functionality is being changed in a substantial way which necessitates that some code be removed.

When code is not understood, it is common to leave in place structures that are uncertain as to their function. This is considered the safe approach so that functionality doesn’t end up missing. As it relates to accidental complexity, this isn’t safe. In fact, it creates a technical debt that stays with the product sometimes for the life of the product.

There are plenty of areas where documentation should be considered. Some to consider might be:

• Interfaces between two application, as it is a common area of complexity
• Calculations, such as in an algorithm, can sometimes be difficult to follow when not using mathematical symbology
• Any areas that have some non-typical development pattern

In general, if you believe that an average developer might not be able to quickly see what is going on, document it.

Wrap Up

Accidental complexity can sneak up and bite you, but if you remain diligent, it is relatively easy to avoid it. In the process of avoiding it, you can sometimes realize new and novel ways of performing some task.

Being diligent in the design phase can often reap significant rewards. Taking time to re-examine designs is a very important step. In doing these things, you can of avoid accidental complexity, and significantly reduce current and future costs.