Why is Software Design Difficult?

Software design requires a depth and breadth of experience that takes years to develop.  While better training in software design might shorten the time required to master design by better codifying knowledge and experience for others to follow, attaining knowledge is easier said than done. Jonathan Mugan in his book The Curiosity Cycle explains:

Knowledge is more complicated than simply putting available pieces of information together because much of the world is hidden from view. [...] What we know determines what we see, therefore learning new things changes how we experience the world.

If it were a matter of obtaining some bit of information, then anyone could be a great designer after reading a book.  Knowledge is an important part of design since it changes how you see the world, but perhaps even more important are the mental models we create to explain the world.  These models are born through experience and seen through the lens of our current knowledge.  Thinking about the psychology around how we build models is a fantastic way for us to understand software design - both generally and in specific instances.  As in, "Why is design so hard?" but also "How do you 'do' design?"

The Curiosity Cycle

Design is all about learning.  Learning about the problem, the solution, and where those two ideas meet.  As children we learned to learn through the vehicle of curiosity.

The Curiosity Cycle from Mugan's book by the same name.

The cycle has three main building blocks.

• Concepts - pieces (observations about elements and ideas) we pull from sensory experience. Elements are people, places, and things while an idea is something we think about and can put a boundary around.
• Models - specify how concepts are put together to form useful information.
• Tests - experiences that tell us how models work.

As an example, consider what you know about a door.  A concept you might observe is that doors have doorknobs.  A model you might build is "turning doorknobs opens doors." To test this model, you ask a question (form a hypothesis) and try it out.  Maybe you find a doorknob that doesn't turn. Now you'll need to identify the concept of a locked door, update your model, and the cycle starts over again.

Individuating concepts is about pulling cohesive pieces out of the continuous whole experience. This is often an automatic event, but not always. Model building requires reflection on what you've observed and some abstraction. You need to come up with some way to describe what you've observed.

Tests are extremely important in this cycle. Tests can verify your model but also show (perhaps more importantly that your model is incomplete. An incomplete model is partially formed and does not reliably predict the environment. Incomplete models are extremely important because they help us to form new concepts.  New concepts, in turn, help us to improve the models... and the cycle continues.

Curiosity-Driven Software Design

So why is software design so difficult?  Consider this question from the context of the curiosity cycle.

• Code is easy to see and most of us have direct experience writing, testing, running, and reading code. Therefore concepts from code are easy to discern through basic observation.  Write enough code and think about it long enough and you'll discover many useful and interesting concepts.

• Many design concepts are not easily observable in the artifacts.  Early architectural design is pure thought stuff.  Code, the most observable artifact, can easily obfuscate and often betrays intended architectural meaning. In addition, many models are dynamic or the combination of several structures (e.g. people + business + infrastructure + running system) and cannot be easily seen.

• Many developers lack the necessary concepts to describe a model even when that model is a good one. For example, try to describe a car to someone who's never seen a car before.  Is it a horseless carriage?  Perhaps the Hyperloop is a modern example.  "You know, it's one of those pneumatic tubes you'd find at the bank but for people!"

And yet, developing the models is still really important even though it's difficult.  What makes this model building so difficult?  I don't have a full answer for this, but here are some thoughts.

• Inventing, observing, and describing concepts relevant to your system that can sometimes only exist as ideas is a challenge.  This is required for design. 
• Simultaneously developing models of both the software structures and the domain in which the system lives is a challenge. Learning new ideas from either of these things changes the world view..
• Testing models is difficult and also essential to successfully individuating concepts and forming a good enough model of the world to build something.
• Asking good questions seems to be as much art as science.  Some people are better at asking the right questions at the right times than others.  This is unfortunate because asking questions is the fuel that keeps curiosity cycle turning.

The first few chapters of the Curiosity Cycle are available as a preview on Amazon Kindle.  The primary purpose of the book is to help arm parents with a new perspective on how kids learn so they can help their kids develop a life long love of learning.  While that's nice, I quickly lost interest (even as a new parent).  All that said, the first few chapters are absolutely fascinating and the writing is extremely accessible with generally good references.  Definitely check out the Kindle preview.