Outcome based Process Metrics – A focus on Time to Value.

This is one of those posts where I’m looking for feedback. Ideally practitioners who’ve made this work but also thought leaders who’ve put serious thought into the matter.

For the past year or so, I’ve been thinking about metrics. When measuring the effectiveness of your development investment process, I’ve come to the conclusion that you need three outcome metrics… A “quality metric, a “value” metric and a “time to value” metric. You need the three metrics because any two can be gamed. For example, you can maintain high quality and a low time to value by releasing software that delivers no value.

Quality metrics are fairly straight forward… number of bugs released is a fairly standard metric. Value metrics are fairly easy to define but sometimes harder to measure… e.g. profit, revenue, number of customers. The tricky metric is the “time to value”.

For years I’ve been chanting the “Cycle Time” mantra from the Lean world. When it comes to looking at metrics in the real world of software development, “Cycle Time” is a great inspiring concept but it ain’t much use helping a company performing an Agile Transformation. The problem is knowing when to call the start on the investment. When you trawl through the historical data of real investments, it’s difficult to compare them. The “time to value” metric is really about risk management. A development system with a short “Time to Value” will be less risky than one with a longer one. Initial thoughts are to consider the start of any work on an investment to the time it is released. This penalises investments where a small amount is invested up-front to surface and address risk (the value of business analysis). As a result, the approach is often to ignore the analysis in the “time to value” metric and focus on the point that the “team” commits to delivering value. i.e. The point it is pulled into a sprint. Ignoring the analysis means that over-investment can occur to reduce risk. In effect, “time to value” metrics are messy in the real world. To illustrate this, consider the following three investments:

20131207-193858.jpg

The three investments have the same start date and release date, however (1) is clearly better than (2) which is clearly better than (3). In the real world, it is even harder to compare investments as the investments are spread across several teams with different rates of investment. This is a solved problem in finance where investors need to compare different bonds and match assets to liabilities of differing size and frequency. Duration was one of the simplist and earliest ways of comparing the risk of bonds. The approach converts a number of cash flows into a single cash flow at certain point in time. This is quite simple. Imagine a seesaw in a children’s playground. All of the cash flows in the investment are placed at the time they are made on one side of the seesaw, and all of them at a single point on the other side. They are placed such that the seesaw balances.

20131207-195718.jpg

Duration ( tD ) = Sum of ( Cash Flow * Time to release ) / Sum of Cash Flows.

( Note: In bond maths, the Cash Flow would be the present value of the cash flow.).

Using the examples above, gives the following results…

1. (28*1 + 1*2 + 1*3) / 30 = 33 / 30 = 1.1
2. (10*1 * 10*2 + 10*3) / 30 = 60 / 30 = 2
3. (1*1 + 1*2 + 28*3) / 30 = 87 / 30 = 2.9

The lower the value for the duration of the investment, the lower the risk. The thing I like about this metric is that it drives the behaviour we want. Smart investments where analysis is used to reduce investment risk before a short time from commitment to investment.

The time for the investment should be the point that the team makes a commitment to the investment. In Scrum, this would be the start of the sprint.

So I’d like to hear others thoughts on this.

Discount Factors and the time value of money.

Whenever you discuss investments, thought leaders cannot help trying to add complexity to the situation. My advice is ignore the time value of money as its effect is not that significant. Adding time value of money makes the calculation much less intuitive to people using the numbers.

If time value of money is used, the inverse ( exp[ + rt ] ) of the discount factor ( exp – rt] ) should be used. This is because you are calculating the value of the investment at the time of release in the future whereas in finance you are calculating the current value of a future cash flow.

About these ads

About theitriskmanager

A IT programme manager specialising in delivering trading and risk management systems in Investment Banks. I achieve this by focusing on risk rather than cost. A focus on costs can lead to increased costs. View all posts by theitriskmanager

11 responses to “Outcome based Process Metrics – A focus on Time to Value.

  • Pete (@PeteBuckney)

    Interesting article Chris.

    For people who don’t know duration, it might make sense to mention that it’s a bit like finding the centre of gravity of the investment.

    As far as far as knowing which is better, that’s easiest to explain if we reduce the smaller spends to zero, so we’re comparing immediate and delayed results of a single investment. Then the duration/cog can be understood as a way to measure that week the pattern of spends is less simple.

    As for us, we are very naive on metrics at the moment – but this is some great food for thought!

  • Pete (@PeteBuckney)

    Also meant to say – the gold dust for me here is being honest about the entire investment, and not measuring from when it is sprint-ready.

  • theitriskmanager

    I accidentally deleted Kent McDonald’s comment as I’m learning how to use WordPress on the iPad (Have they never heard of the UX term “forgiveness” ).

    By Kent “metrics to use in order to gauge the effectiveness of an effort.

    A question about your example, why is Investment example 1 clearly better than 2, which is clearly better than 3? I think I might know why, but would like more clarification on that one.”

    Kent

    Great question, as always.

    Check out Pete’s comment. Also, The return on a software investment comes after the release. We there want to minimise the time between our investment and the return. As the time increases, the investment becomes riskier. There are a number reasons for this.

    1. The longer the time, the more chance that the market can move. (E.g. A competitor releases something similar or a customer buys something else).
    2. The longer the time, the more the developers will forget and as a result it will take them longer to fix/change the product.
    3. The longer it takes, the more decisions that get made before you get feedback from the market. In other words, the more decisions made based on hypotheses rather than the results of the experiment.
    4. This approach reduces the investment (work) in progress.

  • Martien van Steenbergen

    Hi Chris, like the Time To Value approach over the Cycle Time or Lead Time approach very much.

    Your post also reminds me of Software By Numbers from Mark Denne and Jane Cleland-Huang.

    Would like to have a simple and elegant way to build this into daily operations.

    • theitriskmanager

      Hi Martien

      Thank you.

      This is similar to “Software by Numbers” but there is one significant difference. This works on historic known cash flows. The flaw with the SBN approach is that it works with uncertain future predictions of cash flows and then adds accuracy by discounting them. (Cost of delay is a much better approach for incorporating the impact of ordering of investments.)

      As for operationalising the metric, there are two parts. Part one is to calculate the value of duration. Part two is to get the organisation to use the metric to improve. The first part is pretty straight forward. If you are interested I will do a blog post to explain how to do it. The second part is the real challenge and I have not managed to do that part yet. Looking for ideas. :-)

  • Sven Andrä

    Am I just missing something or do the numbers on the three investments not match the calculation and the message of the article?

  • jonathanwhitaker

    Nice post Chris

    As a practitioner (and sometimes thinker) I’m also actively involved in working out what to measure and have been taking the Jim Highsmith and Don Reinertsen approach, so seeking to measure business value and quality over the constraints of schedule, scope and cost. Cost of Delay (CoD) is a very useful proxy for business value delivered by (as I know you know) seeking to put a value on the cost of not delivering the business impact at a certain time. I have been using visual modeling techniques for this, so a linear payback function for a predictable revenue return or manual operations cost avoidance versus say a step-function for a time-sensitive regulatory fine. However, as you mention for the investment side, getting a present value on those situations to have one number is too complicated for day to day acceptance.

    Using the duration approach for assessing the riskiness on the investment side is a very nice method to employ and makes perfect sense and is easily applicable across different situations. All I would say, is that to get a holistic view on whether to make the investment at all in the first place would be to combine this with a CoD calculation on the payback side (as not all paybacks are instantaneous on delivery date).

    I do appreciate that regardless of payback function though, the duration on the investment is a very meaningful ‘tell’ on riskiness of the delivery approach!

    I’ve also only just discovered this site Chris, so no doubt in a few minutes time will find a complimentary post about CoD as well!

    • theitriskmanager

      Hi Jonathan

      I agree with the use of Cost of Delay. Troy Magennis and I were discussing how to incorporate it into the duration calculation yesterday. Its a remarkably simple thing to do. You simply add the cost of delay to the investment amount when you make the commitment. e.g. Building business case (feature) A means we delay business case (features) B & C which have a combined cost of delay of $40K a month. Our sprints are one week long, so we have a cost of delay of $10K which we can add to our investment amount. We do the same at each commitment (investment) point. I need to think further on whether adding cost of delay to duration will drive the behaviour we want. Incorporating COD to duration at a later point in an investment will make it look better than it is.

      Duration has a couple of really nice properties. First, it scales easily. You can use duration to meaure a single point investment, an MVP across teams, for a team, or for a division or enterprise. Second, all you need to do is convert the investments into a common unit (e.g. Dollars, Man days, Team Weeks). The units cancel out when you calculate the duration.

      The Challenge with COD is that you need to convert all outcome values into common units. Not all investments result in a dollar value. This is a decision for the executive to provide the exchange rate between different metrics (e.g. New Customers in Asia versus Dollar Revenue in USA). That is hard but achievable given the right level of engagement. The hardest part of COD is to get a decent estimate of the value delivered. Whilst cost actuals versus estimates follow a log-normal (Weibel fn – hat tip to Troy) with a mean of 2, I have seen return estimates that are out by three or four orders of magnitude. Getting good estimates of return is the real challenge with COD. If the estimates of return are no good, the COD process falls apart.

      Look forward to your post on COD.

  • Ben Williams

    Hi Chris

    I liked the post and having experience with the concept of duration, I like the approach.

    In the closing paragraphs you introduce the concept of TVM and simultaneously dismiss the need to consider TVM as it affect is not significant.

    Maybe it might help the readers get more comfortable with that statement if we were to illustrate the relative merits of each investment used in your duration example, applying TVM. I would expect that due the extreme difference in investment profile, that the order when applying TVM will remain the same. I am just interested whether the relative benefit of one investment over another changes at all.

    I would love to dust off my calculator and submit my findings to you but I am not confident that I would be right, it’s been too long.

    Interest to hear your thoughts.

    Thanks

    Ben

  • Kent McDonald (@BeyondReqs)

    HI Chris,
    I just put down some more thoughts in response to this post on my blog here: http://www.beyondrequirements.com/outcomebasedmetrics/

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 43 other followers

%d bloggers like this: