“In theory there is no difference between practice and theory. In practice there is.” This week I saw one of those examples.
Nice.
ROI $$ (agile) > ROI $ (waterfall)
A great compelling argument for those promoting agile over waterfall. The only problem is that it is naive and WRONG. The argument can only be had in retrospect. The organisation runs two projects, one agile and one waterfall, and compares the ROI on the two projects. I am not aware that anyone has run such an experiment. It is a religious belief without proof.
In reality, the comparison of the two approaches is done before the project starts. The comparison is made on the expectation of the ROI. Those performing the evaluation compare what they expect the ROI to be. The fact that they are considering waterfall indicates the project has a knowable outcome and is in the complicated domain of the Cynefin framework. The functionality is known. If at this point, it is known that the functionality could be reduced, then it would for both. However, when we look at the costs, waterfall only has one test and production release and agile has lots. Given the information at the time, Agile will cost more compared to the waterfall approach. The fact that they are considering waterfall also would tend to indicate a risk averse culture at play which means they wont consider risks or changes.
In summary
Expectation ( ROI (agile) ) < Expectation ( ROI (waterfall) )
Which means that if you encounter someone competent from Finance or a Finance background and start talking about ROI, you are going to sound like an idiot. Someone will use your lack of understanding to undermine your argument to their advantage.
Now lets consider Risk Adjusted ROI. Risk Adjusted ROI is exactly as it sounds. It adjusts the ROI for risk. There are many types of risk associated with an IT investment. Steve Freeman and I wrote an article in The Agile Times (published by the Agile Alliance in 2005) where we list the three categories of risk.
- Delivery risk – Failure to deliver the investment within the agreed constraints
- Business Case risk – Failure to deliver the identified value
- Damage to the Existing Business Model – The investment damages the exisitng business model
Agile can be used to manage all three categories of risk better, but only if it is applied properly. A Scrum or Kanban project that only releases once a year is not managing risk (and is not really a Scrum or Kanban project).
Now lets compare the expectation of an agile and a waterfall project. All of a sudden, all of the real options that exist in an agile project and make it less risky, can be considered in the valuation. i.e.
Expectation ( Risk Adjusted ROI (agile) ) > Expectation ( Risk Adjusted ROI (waterfall) )
And this is where theory and practice come in. In theory, the ROI argument is good enough. You read a few books and write down that ROI is the way to compare agile and waterfall. The practitioner tries this out and it does not work. They then have to dig deeper into the theory to come up with a something that does work. “As simple as possible but not simpler”. “A simple as possible” is the battle cry of the theorist. “but not simpler” is the retort of the practitioner. Finding “As simple as possible but not simpler” for your context can only be determined by praxis, the co-evolution of theory and practice.
For me, a humbling example of this on a personal level was when I was trying to come up with a definition of business value in 2003. I read “The Goal” by Eliyahu Goldratt and thought “A project creates value when it increases revenue or reduces costs”. I asked the business manager on the trading floor and he said. “Nope! Eighty percent of our investments are to protect revenue, either retain customers or build risk management systems.” – To quote Jabe… “Praxis makes perfect.”
The IT Risk Manager 
April 12th, 2015 at 10:41 am
One adjustment. Even if we were only releasing yearly, I think some kind of inner Agile still helps with delivery risk because it gives more realistic tracking.
April 12th, 2015 at 2:32 pm
Hi Steve
I agree, as long as the code is released to a small sub-set of users in a production environment.
Regards
Chris
April 12th, 2015 at 11:00 am
Nice post Chris, thanks. I’d love to read the article you mention from Agile Times. I’ve failed to find it via Google.
Couple of things occurred to me while reading, sorry if these are obvious.
I’d argue that the principle financial benefits of agile over waterfall are reduced time to value, and reduced sunk cost before value.
As we know the power of discovery and feedback means we’re unlikely to deliver exactly what we set out to build. I appreciate that if something is large and complicated that may be the case, but I’ve personally never seen it.
April 12th, 2015 at 2:40 pm
Hi Stuart
Glad you like the post.
We are in agreement. Reduced time to value/reduced sunk cost before value (or duration as they are referred to in bond maths) are risk measures. There is also the benefit from a time value of money perspective but that is insignificant (and ignorable) compared to risk impact.
Discovery and feedback are also risk management tools that do not impact the expected ROI. In fact they are ignored in ROI calculations unless you are explicitly calculating using real options. 😉
Chris
April 12th, 2015 at 11:13 am
A couple of assumptions that are worth challenging in your
Expectation ( ROI (agile) ) < Expectation ( ROI (waterfall) )
assessment
1) It assumes that batch transaction costs in testing and release are significant, invoking 'economies of scale' arguments
2) It assumes that value is only realised once the entire solution is complete, neglecting CoD
Combine them both, and your minimum economic batch size (which is a U-shaped optimisation curve) is quite high.
Reduce either (or both) and the minimum economic batch sizes shrinks very rapidly, changing the RoI picture quite dramatically. And that's in an environment with low risk of inflight change.
http://everydaylean.info/2013/03/smed-is-a-gamechanger/
April 12th, 2015 at 2:50 pm
Hi Martin
1) The expectation of the ROI would assume that the transaction costs would be same for agile and waterfall, and hence the fact you are doing more transactions would indicate a lower ROI.
2) The revenue would be assumed to be the same at the time of making the decision. Are you invoking the insignificant time value of money argument that any finance person would ignore.
To be clear. I am arguing for smaller batch sizes. My argument is that ROI is the wrong way to argue it. Instead, you should argue based on risk adjusted return on investment.
April 12th, 2015 at 2:58 pm
Hi Chris
1) the effort that the Agile community has put into reducing the transaction costs of batching to enable reduction in minimum economic batch size is what convinced me that Agile has Lean bones. Making the transport batch in particular virtually pain free (and risk free, to support your argument) hugely reduces that cost argument.
2) it’s not an NPV type argument (although that is non zero), but a benefit per time period starting earlier one. And a de risking (again, to support you!) of meeting truly fixed deadlines
April 12th, 2015 at 3:15 pm
Hi Martin
1) Could not agree more. As the costs reduce, they reduce the cost argument even more.
2) And again, I agree.
Single malt?
April 12th, 2015 at 7:36 pm
Multiple single malts, I expect.
April 12th, 2015 at 9:57 pm
Chris: Thanks for the invitation to comment. I shall do my best …
No doubt considering risk is of value. The note you’ve written here, however, is not convincing me regarding how or why to do it.
In “refuting” the standard ROI(a) > ROI(w) argument, you entirely ignore earlier receipt of value. You also ignore the opportunity to stop the project early either due to sufficient value having been provided, or due to discovering infeasibility.
You assume without support that “only one test” is cheaper than the many tests of an Agile approach. The waterfall’s single test comes very late in the game. Smaller but important tests early may well be better. And the cost of testing and release does not grow linearly with number of each. In fact, a waterfall release can be incredibly expensive because of the long and unpredictable debugging period, followed by the inevitable hasty release with many defects still in the system.
In short, it is far from clear that multiple tests and multiple releases actually cost more than one.
Moving on, I believe the new assumptions you choose are at least as faulty, if not more so, than the ones you begin by denying.
You assume that choice of waterfall implies known requirements and complicated domain. This is IME invariably false on both counts. Requirements are never fully known. Expected requirements are rarely if ever accomplished. And no software project, ever, lives in the complicated domain: all are complex owing to the presence of humans in the mix.
Now, perhaps this is not intended to be a true analysis or argument, but instead to characterize the beliefs of “management” that their requirements are known, their domain is only complicated, and so on, and then to provide a more glib but no more (or less?) true argument that has more power to convince, if not more validity.
If that’s the case, I’d advise saying so. As it stands, while I can see using risk as a conversation point in addressing whether to go Agile, the logic does not bear weight here, if I’m correct.
There is no guarantee of that, of course. 🙂
In addition, I feel sure as I read this and think about it, that you and I (and any third party) would each be operating from different assumptions about the world that underlies this kind of analysis. In a conversation, we might uncover such things, and might be able to settle on one model to discuss at a time, and might even get to a better understanding of what different world states suggest in terms of different approaches to the project.
Along those lines, here’s a thing that I think would be interesting. “We” produce a model of a situation, a product, some set of endogenous characteristics, and evaluate various styles of development against that model. “We” might then experiment with various exogenous events and examine resilience.
One such exogenous event type would be to assume some sort of arrivals (Poisson, one might imagine) of information impacting the assumptions behind the product idea, so that one could see how well various approaches might respond to that information.
Anyway, I could be entirely off base, as I frequently am. But I don’t think your refutation holds together, and I find your assumptions in your example to be unlike the real world.
April 13th, 2015 at 10:44 am
Hi Ron
Thank you for your comments. It is clear I have failed to get my message across clearly so I am particularly grateful for them.
Before I start I want to make it clear that I believe that Agile delivery is always superior to Waterfall. The issue I’m raising is that the expectation of the ROI of an Agile approach is worse than the expectation of the ROI of a Waterfall approach. Using ROI arguments with experts in Finance is not a convincing argument. As such, we should use Risk Adjusted ROI (RAROI) as it correctly states that the expectation of the RAROI of Agile is better than the expectation of the RAROI of Waterfall.
In “refuting” the standard ROI(a) > ROI(w) argument, you entirely ignore earlier receipt of value. – I’m arguing the opposite. An ROI approach only incorporates the time value of money (which is not that significant) aspect of the early value of money. It does not consider the risk aspect which is more significant. Hence, we need to use Risk Adjusted ROI to incorporate these options.
You also ignore the opportunity to stop the project early either due to sufficient value having been provided, or due to discovering infeasibility. I’m arguing the opposite. ROI does not incorporate these options unless they are explicitly valued.
You assume without support that “only one test” is cheaper than the many tests of an Agile approach. The waterfall’s single test comes very late in the game. Smaller but important tests early may well be better. And the cost of testing and release does not grow linearly with number of each. In fact, a waterfall release can be incredibly expensive because of the long and unpredictable debugging period, followed by the inevitable hasty release with many defects still in the system. Once again we are in agreement. My argument is that ROI does not include these tests (options), whereas RAROI does.
In short, it is far from clear that multiple tests and multiple releases actually cost more than one. Multiple tests do cost more with an ROI analysis, but they deliver significantly more value in an RAROI analysis.
You assume that choice of waterfall implies known requirements and complicated domain. This is IME invariably false on both counts. Requirements are never fully known. Expected requirements are rarely if ever accomplished. And no software project, ever, lives in the complicated domain: all are complex owing to the presence of humans in the mix. Could not agree more. Once again, in an ROI analysis the assumption is that the requirements are known, and that they exist in the complicated domain. A Risk Adjusted ROI would consider the following risks:
1. Risk that the requirements are not fully known, or accurate.
2. Risk that key individuals leave the project (and impact the skill liquidiity which causes a constraint).
3. Risk that unforeseen technical problems occur.
Now, perhaps this is not intended to be a true analysis or argument, but instead to characterize the beliefs of “management” that their requirements are known, their domain is only complicated, and so on, and then to provide a more glib but no more (or less?) true argument that has more power to convince, if not more validity. Correct. I do not believe the requirements are complicated. Management, especially in a risk averse culture, believe that that they are, or rather hope that they are and make decisions accordingly. I should have made that point more clearly.
I suspect that we agree violently on this subject and the language I am using is causing confusion. In an attempt to clarify, let me try to create an ROI and RAROI
The ROI for a Waterfall Project:
equals
Discounted returns of expected future cash flows.
minus
Project Costs
The ROI for an Agile Project:
Discounted returns of expected future cash flows (Higher due to getting cash flows earlier)
minus
Project Costs (Higher due to additional testing & release costs)
The RAROI for a Waterfall Project:
equals
Discounted returns of expected future cash flows (The cash flows should be risk adjusted as well as simply discounted due to interest rates, resulting in a much smaller value).
minus
Project Costs
The RAROI for an Agile Project:
equals
Discounted returns of expected future cash flows (Higher due to getting cash flows earlier, especially given risk adjustment)
plus Option to increase investment
plus Option to change requirements
plus Value realised by reducing uncertainty through the tests
minus
Project Costs (Higher due to additional testing & release costs)
less Option to terminate early without losing investment to date
less Option to release without having to manually test
Using RAROI results in a much higher value for the Discounted Cash Flows in Agile than Waterfall.
The Options in the RAROI create a significantly more robust argument for using Agile than Waterfall.
Thank you for your thought provoking comments. I need to better refine my message.
Regards
Chris
April 13th, 2015 at 12:06 pm
[…] by Ron’s Comments on the Risk Adjust ROI of Agile and Waterfall post, I wrote a simple example to illustrate the point I was trying to […]