Thursday, July 15, 2010

Improvisational Architecture

If you were to go to two improvisational jazz performances, even two concerts by the same band, you'd hear different music at each show. The thrill of experiencing something unique, as it is created, without any kind of real plan or rehearsal, is both exciting and entertaining. Sometimes the music is so spectacular you feel like you're witnessing a miracle. Other times the group can't seem to get it together and the show is one long, painful indulgence in artistic expression. No matter the outcome, what you hear during that performance will never be experienced again.

Music has always served as a great metaphor for thinking about software development. At XP2010 during one of the most original key notes I've ever seen, Bjørn Halterhaug and John Pål Inderberg, professors of musicology and improvisation respectively, from the University of Technology and Science in Trondheim, Norway and both veteran jazz musicians, discussed, through music and words how they approach improvisational jazz, the mechanics for making it work, and the general implications on collaboration among artists. It's tempting to romanticize improvisational jazz as a truly spontaneous creation but this isn't exactly how it works in reality. For any improvisational jazz band to be successful its members must exhibit seven key characteristics.
  1. Provocative Competence Interrupting Habit Patterns
  2. Embracing Errors as a Source of Learning
  3. Minimum Structures that allow Maximum Flexibility
  4. Distributed Task: Continual Negotiation toward Dynamic Synchronisation
  5. Reliance of Retrospective Sense Making as Form
  6. Hanging out: Membership in Communities of Practice
  7. Alternating between Soloing and Supporting
I propose that these seven characteristics apply directly to how teams who espouse emergent design should approach software architecture.

Music and Design

Experienced musicians have an attenuated ear which enables them to hear music in ways that novices can't. Drawing on a well-developed playbook of clichés, an expert in the standup bass for example can not only follow the minimally, well-defined structure of a song but also weave pleasing bass licks into the musical ramblings of his band mates, adjusting his actions based on the paths his band mates choose with appropriate responsiveness as the song develops. Part of it is musical maturity, a taste or style developed over years of playing, part of it is also trust both in himself as a musician but also in the rest of the group, that they too will not overreact to mistakes and are also able to follow a musical flow as it develops.

Well-defined but minimalist structure. Contrast this with the music played by a symphony which is also well-defined but has explicit and detailed structure.

When I hear this I immediately think of architecture as seen through the lens of agile software processes which advocate strongly for emergent design. Agile software development processes and agile culture in general strongly encourages teams to become improvisational architects. Create a minimalist, well-defined design of the system and then allow the developers to evolve the system, creating the design as they develop, playing off the code written by their fellow teammates as the system emerges. The romantic view of this is appealing to some: "Let's go have a code jammin' session, you dig?" But the results of an emergent design can be just as varied as improvisational jazz bands; some designs are elegant and spectacular successes, some designs are flaming piles of disaster, most turn out somewhere in between - usually good enough for customers who aren't paying too much and who expected a somewhat unpredictable but functionally adequate outcome.

So if you enjoy the thrill of experiencing once-in-a-lifetime moments of creation, the outcome of which can never be predicted or known ahead of time, then emergent design is probably right up your alley. As demonstrated by the near exclusive emphasis on code and shipping, many agile developers fancy themselves as code jammin', improvisational architects.

The exact opposite of emergent design, of course, is the easy to hate (and rightly so) enemy of creating things that actually work, Big Design Up Front, where a heavy cost is levied against a project in the beginning without producing any usable code whatsoever. Sticking with the music metaphor, the well-defined, explicit, and detailed structures of a symphony might take a composer months or even years to create. But in exchange for this planning you get a piece of music that is guaranteed to be executed (for all intents and purposes) in a nearly identical way in every performance in which it is played. I am not saying you should use a Big Design Up Front approach, in reality, even Mozart tried out music and made incremental deliveries to his sponsors as he wrote.

While certainly a degree of improvisation happens when we develop software, customers paying for software usually expect more predictability in the outcome. So, while it might be ok to spend 30 bucks for a fun first date with a girl at a jazz club - if only for the experience of it - when I pay 100 bucks a ticket to see the symphony play Mozarts's Requiem, I would be more than a little disappointed, angry, and confused if the first chair violinist "felt a groove" in the middle of the second movement and began jamming. The greater the cost of a project, the more highly valued predictability in it will be.

Who can be an Improvisational Architect?

Mozart was a true genius, a master of composition. To play Mozart, you need to be well practiced but you don't need to have his level of genius. On the other hand, most members of a great improvisational jazz group must be experienced musicians - experienced in composition as well as technical playing ability. Even then, a great group will usually avoid disaster but occasionally it will still happen because of the nature of improvisation.

In my experience, teams that plan to allow their designs to emerge do an ok job of providing the minimal structure that is essential for evolving a design. But only a few developers have the necessary experience and background to truly improvise a design. The implication? Most teams are terrible at improvising architecture because they have neither a solid enough understanding of the core fundamentals for software architecture nor the experience with using architectural design to reason about a system.

I believe that most of the seven characteristics of improvisation are deeply ingrained in agile culture, but agile teams often fail to fully embrace all seven characteristics of improvisation when designing a software system. A team without the experience, who doesn't understand design cliché's (architectural styles and patterns) will have an unpredictable outcome when allowing the design of a system to emerge as the system is developed. This is something the agile community must work to resolve.


Cory Foy was thinking ahead and had the foresight to record video of the performance/keynote.  It comes in 6 parts.
Ingvald Skaug was also inspired by this talk and wrote an interesting essay which explains kanban as agile jazz. Check it out for a process perspective on the idea of "minimal structure for maximum flexibility." If you believe Conway's Law, then it isn't coincidence that we would apply the characteristics of improvisation in jazz to both process and design.