This is one of those posts where I’m looking for feedback. Ideally practitioners who’ve made this work but also thought leaders who’ve put serious thought into the matter.
For the past year or so, I’ve been thinking about metrics. When measuring the effectiveness of your development investment process, I’ve come to the conclusion that you need three outcome metrics… A “quality metric, a “value” metric and a “time to value” metric. You need the three metrics because any two can be gamed. For example, you can maintain high quality and a low time to value by releasing software that delivers no value.
Quality metrics are fairly straight forward… number of bugs released is a fairly standard metric. Value metrics are fairly easy to define but sometimes harder to measure… e.g. profit, revenue, number of customers. The tricky metric is the “time to value”.
For years I’ve been chanting the “Cycle Time” mantra from the Lean world. When it comes to looking at metrics in the real world of software development, “Cycle Time” is a great inspiring concept but it ain’t much use helping a company performing an Agile Transformation. The problem is knowing when to call the start on the investment. When you trawl through the historical data of real investments, it’s difficult to compare them. The “time to value” metric is really about risk management. A development system with a short “Time to Value” will be less risky than one with a longer one. Initial thoughts are to consider the start of any work on an investment to the time it is released. This penalises investments where a small amount is invested up-front to surface and address risk (the value of business analysis). As a result, the approach is often to ignore the analysis in the “time to value” metric and focus on the point that the “team” commits to delivering value. i.e. The point it is pulled into a sprint. Ignoring the analysis means that over-investment can occur to reduce risk. In effect, “time to value” metrics are messy in the real world. To illustrate this, consider the following three investments:
The three investments have the same start date and release date, however (1) is clearly better than (2) which is clearly better than (3). In the real world, it is even harder to compare investments as the investments are spread across several teams with different rates of investment. This is a solved problem in finance where investors need to compare different bonds and match assets to liabilities of differing size and frequency. Duration was one of the simplist and earliest ways of comparing the risk of bonds. The approach converts a number of cash flows into a single cash flow at certain point in time. This is quite simple. Imagine a seesaw in a children’s playground. All of the cash flows in the investment are placed at the time they are made on one side of the seesaw, and all of them at a single point on the other side. They are placed such that the seesaw balances.
Duration ( tD ) = Sum of ( Cash Flow * Time to release ) / Sum of Cash Flows.
( Note: In bond maths, the Cash Flow would be the present value of the cash flow.).
Using the examples above, gives the following results…
1. (28*1 + 1*2 + 1*3) / 30 = 33 / 30 = 1.1
2. (10*1 * 10*2 + 10*3) / 30 = 60 / 30 = 2
3. (1*1 + 1*2 + 28*3) / 30 = 87 / 30 = 2.9
The lower the value for the duration of the investment, the lower the risk. The thing I like about this metric is that it drives the behaviour we want. Smart investments where analysis is used to reduce investment risk before a short time from commitment to investment.
The time for the investment should be the point that the team makes a commitment to the investment. In Scrum, this would be the start of the sprint.
So I’d like to hear others thoughts on this.
Discount Factors and the time value of money.
Whenever you discuss investments, thought leaders cannot help trying to add complexity to the situation. My advice is ignore the time value of money as its effect is not that significant. Adding time value of money makes the calculation much less intuitive to people using the numbers.
If time value of money is used, the inverse ( exp[ + rt ] ) of the discount factor ( exp – rt] ) should be used. This is because you are calculating the value of the investment at the time of release in the future whereas in finance you are calculating the current value of a future cash flow.
The IT Risk Manager 
December 8th, 2013 at 9:10 am
Interesting article Chris.
For people who don’t know duration, it might make sense to mention that it’s a bit like finding the centre of gravity of the investment.
As far as far as knowing which is better, that’s easiest to explain if we reduce the smaller spends to zero, so we’re comparing immediate and delayed results of a single investment. Then the duration/cog can be understood as a way to measure that week the pattern of spends is less simple.
As for us, we are very naive on metrics at the moment – but this is some great food for thought!
December 8th, 2013 at 9:22 am
Also meant to say – the gold dust for me here is being honest about the entire investment, and not measuring from when it is sprint-ready.
December 8th, 2013 at 9:41 am
I accidentally deleted Kent McDonald’s comment as I’m learning how to use WordPress on the iPad (Have they never heard of the UX term “forgiveness” ).
By Kent “metrics to use in order to gauge the effectiveness of an effort.
A question about your example, why is Investment example 1 clearly better than 2, which is clearly better than 3? I think I might know why, but would like more clarification on that one.”
Kent
Great question, as always.
Check out Pete’s comment. Also, The return on a software investment comes after the release. We there want to minimise the time between our investment and the return. As the time increases, the investment becomes riskier. There are a number reasons for this.
1. The longer the time, the more chance that the market can move. (E.g. A competitor releases something similar or a customer buys something else).
2. The longer the time, the more the developers will forget and as a result it will take them longer to fix/change the product.
3. The longer it takes, the more decisions that get made before you get feedback from the market. In other words, the more decisions made based on hypotheses rather than the results of the experiment.
4. This approach reduces the investment (work) in progress.
December 8th, 2013 at 10:20 am
Hi Chris, like the Time To Value approach over the Cycle Time or Lead Time approach very much.
Your post also reminds me of Software By Numbers from Mark Denne and Jane Cleland-Huang.
Would like to have a simple and elegant way to build this into daily operations.
December 8th, 2013 at 11:04 am
Hi Martien
Thank you.
This is similar to “Software by Numbers” but there is one significant difference. This works on historic known cash flows. The flaw with the SBN approach is that it works with uncertain future predictions of cash flows and then adds accuracy by discounting them. (Cost of delay is a much better approach for incorporating the impact of ordering of investments.)
As for operationalising the metric, there are two parts. Part one is to calculate the value of duration. Part two is to get the organisation to use the metric to improve. The first part is pretty straight forward. If you are interested I will do a blog post to explain how to do it. The second part is the real challenge and I have not managed to do that part yet. Looking for ideas. 🙂
December 8th, 2013 at 11:28 am
Thanks for the quick reply. Yes, I am interested in both the calculation of the value duration and how to get the organization to self-improve based on this metric.
My current experience is that adopting these kind of metrics and using them during the monthly operations review is a slow process, often because of political forces, unless there is an enlightened CxO. However, once Feature Lead Time and Feature Throughput (or #Features per Release) are installed, things start flowing.
I am collecting a number of pearls and pearl language (subtle nuance on patterns and pattern language) on metrics.
For example, see http://pearllanguage.org/Jumpstart,
http://pearllanguage.org/Metric_drives_behavior, http://pearllanguage.org/Kanban, http://pearllanguage.org/Start_what_you_finish_and_finish_what_you_start.
December 9th, 2013 at 8:02 am
Am I just missing something or do the numbers on the three investments not match the calculation and the message of the article?
December 10th, 2013 at 6:41 pm
Nice post Chris
As a practitioner (and sometimes thinker) I’m also actively involved in working out what to measure and have been taking the Jim Highsmith and Don Reinertsen approach, so seeking to measure business value and quality over the constraints of schedule, scope and cost. Cost of Delay (CoD) is a very useful proxy for business value delivered by (as I know you know) seeking to put a value on the cost of not delivering the business impact at a certain time. I have been using visual modeling techniques for this, so a linear payback function for a predictable revenue return or manual operations cost avoidance versus say a step-function for a time-sensitive regulatory fine. However, as you mention for the investment side, getting a present value on those situations to have one number is too complicated for day to day acceptance.
Using the duration approach for assessing the riskiness on the investment side is a very nice method to employ and makes perfect sense and is easily applicable across different situations. All I would say, is that to get a holistic view on whether to make the investment at all in the first place would be to combine this with a CoD calculation on the payback side (as not all paybacks are instantaneous on delivery date).
I do appreciate that regardless of payback function though, the duration on the investment is a very meaningful ‘tell’ on riskiness of the delivery approach!
I’ve also only just discovered this site Chris, so no doubt in a few minutes time will find a complimentary post about CoD as well!
December 11th, 2013 at 9:38 am
Hi Jonathan
I agree with the use of Cost of Delay. Troy Magennis and I were discussing how to incorporate it into the duration calculation yesterday. Its a remarkably simple thing to do. You simply add the cost of delay to the investment amount when you make the commitment. e.g. Building business case (feature) A means we delay business case (features) B & C which have a combined cost of delay of $40K a month. Our sprints are one week long, so we have a cost of delay of $10K which we can add to our investment amount. We do the same at each commitment (investment) point. I need to think further on whether adding cost of delay to duration will drive the behaviour we want. Incorporating COD to duration at a later point in an investment will make it look better than it is.
Duration has a couple of really nice properties. First, it scales easily. You can use duration to meaure a single point investment, an MVP across teams, for a team, or for a division or enterprise. Second, all you need to do is convert the investments into a common unit (e.g. Dollars, Man days, Team Weeks). The units cancel out when you calculate the duration.
The Challenge with COD is that you need to convert all outcome values into common units. Not all investments result in a dollar value. This is a decision for the executive to provide the exchange rate between different metrics (e.g. New Customers in Asia versus Dollar Revenue in USA). That is hard but achievable given the right level of engagement. The hardest part of COD is to get a decent estimate of the value delivered. Whilst cost actuals versus estimates follow a log-normal (Weibel fn – hat tip to Troy) with a mean of 2, I have seen return estimates that are out by three or four orders of magnitude. Getting good estimates of return is the real challenge with COD. If the estimates of return are no good, the COD process falls apart.
Look forward to your post on COD.
December 11th, 2013 at 3:56 pm
Hi Chris
I liked the post and having experience with the concept of duration, I like the approach.
In the closing paragraphs you introduce the concept of TVM and simultaneously dismiss the need to consider TVM as it affect is not significant.
Maybe it might help the readers get more comfortable with that statement if we were to illustrate the relative merits of each investment used in your duration example, applying TVM. I would expect that due the extreme difference in investment profile, that the order when applying TVM will remain the same. I am just interested whether the relative benefit of one investment over another changes at all.
I would love to dust off my calculator and submit my findings to you but I am not confident that I would be right, it’s been too long.
Interest to hear your thoughts.
Thanks
Ben
December 12th, 2013 at 2:38 am
HI Chris,
I just put down some more thoughts in response to this post on my blog here: http://www.beyondrequirements.com/outcomebasedmetrics/
September 29th, 2016 at 7:27 pm
[…] Matts recently wrote a post about Time to Value as an Outcome based Process Metric . I have some thoughts about the topic and decided to write the up here. I could have recorded a […]
January 4th, 2017 at 2:10 pm
[…] Outcome based Process Metrics – A focus on Time to Value. […]
October 4th, 2018 at 11:50 am
[…] Matts recently wrote a post about Time to Value as an Outcome based Process Metric . I have some thoughts about the topic and decided to write them up here. I could have recorded a […]