The Agile versus Waterfall argument divides software developers. There are those that believe iterative development to be superior way of developing software, and those that believe up-front planning to be a superior way of developing software. A couple of weeks ago I had a short but passionate argument with one of my co-workers. They are super super smart, and we each stand on opposite sides of the up-front planning divide. My colleague’s argument is that you need to plan in order to execute fast and efficiently at scale. That fast execution at scale requires considerable up-front planning.
This puzzled me greatly. Real Options shows that up-front planning is inferior as an approach because you commit early and destroys lots of options. Lean tells us that planning is a form of Muda or waste. This is particularly interesting because if people smarter than me think this, then real options could be a flawed concept and I break the model… and learn as a result. After much discussion and thought, I think I may understand the problem. The insight comes from thinking about the problem using the Cynefin model for social complexity developed by Dave Snowden.
The Cynefin model consists of four quadrants (snigger – In joke) of “Simple”, “Complicated”, “Complex”, and “Choatic” and a fifth quadrant “Unordered” for when you do not know which of the other quadrants you are in. A “Simple” system is over constrained and has cause/effect behaviour. “Complicated” is the realm of the expert. “Chaos” means there are not enough constraints. “Complex” means there are just enough constraints to create an emergent behaviour which can be nudged and tweaked into a system with desirable behviour. “Complicated” means an expert can reduce the problem to component parts with predictable behaviour. “Complex” means there are feedback loops, agents and limits that create an emergent behaviour that cannot necessarily be predicted. Essentially, the difference between the two can be two can be distilled to “A complex system interacts with its context and evolves as a result, and a complicated system does not interact with its context or evolve.”
If you know in advance what is needed, you are in a complicated system. You can specify and plan in detail to create an optimal delivery at scale (A long lead time and short duration). Hence the space shuttle. Complicated systems struggle to react to changes in context which can cause analysis paralysis. In product development, complicated systems are managed and dominated by experts. The axioms of this paradigm are untested assumptions. This is the realm of early commitment and the slaying of valuable options. As a cautionary note, these systems have the potential to suffer a catastrophic failure if they choose to ignore the context changes.
If you are in a situation where the context is dynamic and an expert cannot predict in advance, you are in a complex system. You cannot specify and plan in detail because the context changes before you can deliver. Development in a complex system consists of several safe to fail experiments followed by fast developments to enhance beneficial developments and dampen non-beneficial developments. In product development, decisions are made by studying data and the impacts of safe to fail experiments and small iterative investments. Products and their context co-evolve. The axioms of this paradigm are hypotheses which are to be challenged and tested. This is the realm where real options are an optimal strategy (Check out Dave’s paper from 1857).
So the answer is quite simple. If your product is in a complicated context, use experts and up-front planning. If you are in a complex context, create options and learn to respond faster than your competition.
A final note. If you think you are complicated context, and you are operating in a market with competition… I’m sorry, you are wrong. You are actually in a complex context.
Despite my best efforts, I now find the Cynefin framework to be a useful tool to understand* and operate** the context within which I work. If you want to find out more about Cynefin, sign up for one of Dave’s course at www.cognitive-edge.com. Oh, and refer to the quadrants as quadrants. He love’s that.