Value and Metrics

A significant risk in any business is that you do not know where you are. Value is the thing that we aim to deliver, but value data or metrics are how we measure our success at achieving our goal. The only way we know for certain that we have achieved our goal is when the value data shows us success. For example, we aim to deliver $2million from a feature. The only way we know we have achieved that is when we have the $2million in our account.

Simple right?

Nope.

• How do we know that it is the new feature that has delivered the value?
• Do we have to wait until the money rolls in until we know how we are doing?

If we have a single feature that people are buying it may be fairly simple, but that is rarely the case as most applications have many features. We have to put more thought into how we look at our value data. First, we need value data on how the user is using the new feature.

• Do the users use the feature? Did the
• How often to do they use the feature?
• For how long do they use the feature?

Our value data impacts how we might introduce a feature to our users. Do we offer a couple of free goes before they have to pay or do they pay before they use? Offering a couple of free goes allows us to gather value data (or metrics) on how people use the feature. We want to know how long the user spends on each step of using the feature and whether they stop using the feature before they complete all the steps. This will help us understand whether the user values the output enough to put the effort in to provide all the inputs. The value data will tell us which step result in people dropping out of the process.

Context is everything. Context drives how you use value data. A web site is different to an installed application. Retail is different to enterprise. The value of each customer impacts how much we are prepared to test on them. The golden rule is to get feedback on our business case.

If we have an application that the user can download for free and have a couple of free goes before they pay, we have to build a business model ( or belief system ) around how we will make money.

For example, we assume 1,000,000 download the application. 50% use the first free go with the feature, of which 20% use the second free go, of which 10% then pay for the feature. The feature costs $10.00. This would result in revenue of 1,000,000 * 50% * 20% * 10% * $10.00 = $100,000. We could release an earlier version of the feature which tests to see how many people download and use the free versions. We can look at when people drop out of using the feature.

We aim to deliver value but the value data (or metrics) let us know whether things are going to plan.

Value is not enough, we need to consider the return on investment.

Return on Investment breaks Feature Injection.

For many years I have focused on value as the driver of IT development. “Break the model” is the part of Feature Injection that tells us to look for examples that do not fit within the “Olaf” (Model). For some time I’ve been aware that there are examples of development investment that do not fit nicely within the Feature Injection process. These examples have been at the edge of my peripheral vision. I’ve been aware of them but almost subconsciously ignored them because they do not fit in the model. This is exactly the bias that “Break the Model” attempts to address.

What are they? Improvements to the inputs and processes (expecially non functional requirements). Obvious they are valuable but they do not deliver value to the user as we know all of the value in a system comes from consuming the output. i.e. The outcome.

According to the definition of Feature Injection, all value comes from the output which is true, however improving inputs and processes also brings benefit to the user which is also true.

I have been considering a number of successful companies which do not have an obvious source of income. For example, twitter and instagram. For these companies, they value increasing the number of people who have installed their software (network size) and the amount that people use their network (network activity).

Users of an application will probably download it because it is valuable to them.

The amount of usage will depend on the return on investment of the application. This is a function of the value of the application (which is in it’s output) and the investment they need to make in order to get that value.

The investment from the users perspective is determined by the effectiveness of the inputs and processes. From the users perspective, the investment takes many forms, some of which are:

• Money ( to buy the software / service )
• Time ( How long you must work to get the value you seek )
• Mental investment ( How much brain power is required to derive the value)
• Delay ( How long you must wait for the value )
• Frustration ( How easy and forgiving the application is to use. Whether the app play “snakes and ladders” with the user. )
• Training ( How much time and effort must be invested learning to operate the application )
• Transferability ( Whether the investment is transferrable to another context or application )
• Specialisation.

This leads to an extension of the rules for generating business value.

Development generates business value when it reduces the investment that a user needs to make in order for them to acheive the value they desire.
This is an addition to the existing definition rather than a rewrite. It also means that Feature Injection will need an additional step so that it becomes.

1. Hunt the value (i.e. Increase value)
2. Inject the features
3. Break the model
4. Reduce the user’s investment.

Example – Cameras*.

Cameras have no independent value in the process of image making. The value of a camera is to capture images that can be shared with others across time and location. The value is in the images that are viewed at some later date.

The value in the process has never changed. The value is in the viewing of the images. If we cannot view the image, then the camera could easily be replaced with a stone with loss of value to value stream.

In the early days of cameras a low quality photo image would be created by a very expensive process that involved learning optics, chemistry and building a dark room. This meant that the process was limited to a very small number of people who only captured very high value images (The Queen’s Wedding and Coronation).

The photo production process was automated but still specialist. This meant that people could use a camera to capture images but there was a delay of days to recover the images. The cost of investment was reduced so we could capture images of lower value (Normal people’s wedding’s, birthdays and holidays).

One of the problem with the process was a lack of forgiveness. You never knew whether your images had been captured. This changed with the introduction of the Polaroid Camera which provided instant feedback. You could now take photos until you got it right.

The electronic camera wiped out much of the production process and thus cost of the images.  (Now every day events could be captured).

The internet and social media sites wiped out the remaining cost of getting the images in front of your intended viewers. (Even trivial events can be shared at very little cost).

This has lead to an exponential increase in the number of captured images.

Feature Injection was broken. It failed to properly incorporate non functional requirements related to inputs and processes in the investment decision process. We can fix this now and start to look for more examples that do not fit.

*Deliberately chosen to get a response from @Cyetain

Its not a model, its an Olaf.

At ScanAgile in Helsinki (great conference), Olaf Lewitz pointed out that the “model” in the “break the model” part of Feature Injection is not a model as we would normally describe it.

My understanding of the term model is “A simplification of reality”. This was my understanding when I started using the term.

Olaf pointed out that what we had in “Break the model” was “A summary of examples”. The important part is that the process used to create it is to incrementally add examples and adjust it as we add examples that are different in some way. We do not create our model by simplifying the reality of the situation. We do not have a good word for this and so we are calling it an Olaf until a better name is discovered. In effect, we now have “Break the Olaf” in  Feature Injection which seems a bit harsh on Olaf, so I think we will stick with “Break the Model” until we find the right name.

So what is an Olaf? An Olaf is a set of examples that represent our reality. We use it as a filter to identify new examples that do not fit in the set.

e.g. We have a white square and a black square in our set. We then spot a red square. We know that this does not exist in our current set so we add it.

In its simplest form, the Olaf is a the set of examples itself. However, it becomes hard to spot new examples by comparing them to the existing set of examples. To make it easier to spot new examples, we abstract properties and behaviours.

“Break the model” means we have found an example that does not fit into our set of examples.

e.g. We start with a black square.

Our Olaf is a black square.

We spot a white square which breaks the Olaf.

Our Olaf is now a black or white square.

We spot a red square which breaks the Olaf. We abstract black, white and red into “coloured”.

Our Olaf is now a “coloured” square.

We spot a triangle which breaks the Olaf. We abstract triangle and square into straight sided shapes.

Our Olaf is now a coloured “straight sided shape.

We spot a circle……….

The key thing is that the Olaf describes our existing examples so that we can more easily spot new examples. Once we spot an example, we simply add it to the list of examples. To make things easier, we may create an abstraction of the examples.

Let me know if you think of a better name for an Olaf.

QCon New York

Just finished the initial draft of slides for my presentation at QCon New York. Will be tweaking, adding and pruning over the next few weeks as I remove content and hope to add humour.

Amr Elssamadisy asked me to put together a track on the “Agile Individual”. I did what any sane person would do and invited the people I want to hear speak. The only constraint was they had to talk about Agile Individuals.

Mike Hill is a legend from the early days of XP. He will be talking about how individuals make amazing teams. If he does not offend, then he will have to try harder. Mike Roberts is just super super smart. He will be bustin’ and bruisin’ some Agile processes in the hope that the focus will be on individuals instead. Matt Wynne did a hilarious talk at Cukeup last year. The additional constraint I placed on him was that his talk had to be just as funny. If not, I’ll banish him to a cave in Scotland. I’ll be talking about Learning… and how to do it the lazy way. Sue McKinney will be on Late ( a private joke ) talking about the Executive view of the individual in Agile.

All will be excellent presentations except mine which currently has four hours of material that I hope to squeeze into 45 minutes.

Risk – Its all about time!

I was talking to a colleague about risk. They mentioned that they list the risks and assign a probability and impact  to each risk. This appears to be the standard approach taught on many management courses.

“I take a different approach” I said “I only consider impact”.

I am interested in two measures.

• How long it will take to fix the problem.
• How long the business can survive with the problem.

If the time to fix the problem is less than the time the business can survive, we do not have a problem.

If the time to fix is greater than the time available, then we need some more options.

Olav and I mention a number of examples in our InfoQ risk article

Can you think of other examples?

Is TDD always the best solution? The third way?

Recently I have been involved in an “Agile” transformation. I have been forced to look at a number of teams and assess which tools from the Agile toolkit are relevant.

Rather than simply foist Agile goodness on the teams, I’ve been engaged in a discussion about the relevance of the tools… especially from a risk perspective (what a surprise!).

To help people understand Agile I have been casting it in a historical perspective against the (perverted version of) Royce’s Waterfall approach with sign-offs and change control mechanisms. Adopting an iterative development approach mitigates the need for those sign-offs and change control. In addition, the iterations help business investors mitigate a whole bunch of other risks.

Multiple iterations cause some additional issues. There are transaction costs associated with an iteration such as regression testing and releases. Without careful management, some of these transaction costs (regression testing) will increase with each iteration which will start to have a negative impact on the teams ability to deliver new functionality.

The traditional approach is to perform extensive manual regression testing. This increases with each iteration and as a result the economics of the situation lead to larger iterations which breaks the risk management goodness of small iterations.

The agile approach to this problem is to automate testing. To ensure comprehensive and effective coverage, it is necessary to consider testing as a primary concern. Thus test driven development comes to the fore.

The team I work on has an interesting context. The system is not seen outisde of the organisation. It is not used to make payments to external entities. The users do not use the information solely to make decisions. As a result the users are tolerant to bugs… provided they can be fixed quickly. The team can turn around bug fixes within a few minutes to an hour which is acceptable to the business. So this presents a “third way” to address the regression testing overhead. Minimal testing with the acceptable risk of introducing bugs that are fixed quickly (including by roll back which has not yet happened).

It will cost the business investor a lot of time and money to retro-fit effective automated tests. It will take time and effort for the team to develop the skills to learn TDD and build automated tests around the legacy code base. Time and effort that the business investor could spend on other things.

Currently the team are able to produce several releases a week. The business investors are happy with the performance of the team.

Would TDD be the best approach or would the “Third Way” be more appropriate to this context? Would the introduction of TDD break the existing development process from the business investor’s perspective?

Should TDD always be foisted on a team? Or not? Would a team be Agile if they did not do TDD?

Thoughts?

P.S. I am aware that TDD brings many other benefits beyond regression testing.

Forgiveness

I met my first User Experience designer in the late nineties. My boss hired a guy with a PhD in the subject. I asked him “What is the most important thing in interface design?”. His answer “Forgiveness”. He explained that Forgiveness was a UX concept that allowed the user to gracefully recover from failure. Without me knowing it, he was one of my earliest mentors in the field of real options.

I now consider Forgiveness to be an indication of a high quality system. When your interaction with the system fails, does it do so gracefully, or does it require you to jump through a number of painful hoops until you get back to where you were originally? Of course, the only way to see how a system handles failure is to experience failure. Until you experience failure, you do not know whether the interface will perform as described. Many systems claim to handle failure gracefully but not all system do. If you really want a deep understandinf of a system, you need to see how it fails.

When I think of systems, I think of People and Relationships and Groups and Organisations and Communites and Societies, oh and Computer Systems. For me, the most interesting systems are those that contain people ( Complex Adaptive Systems ).

Sometimes, failure in one of these systems means expulsion or a loss of status. These systems discourage people from experimenting, or playing to close to the edge of chaos. By doing so, these systems discourage deep learning at the edge. They discourage people from seeking out and challenging the axiom ( or beliefs ) of the system. The Catholic Church’s original treatment of Gallileo is a classic example of this. Eventually the Church forgave Gallileo but it might have been five hundred years to late for him to care.

Just over two weeks ago I attended the CALMalpha event. This event hoped to bring together Cynefin, Agile and Lean communities to explore whether there were any ways that they could help each other learn. For me it was a bust. I did not really learn anything* as I was already familiar with most of the material. I had fun and I enjoyed meeting old friends and new. Meeting new friends and old justifies the attendance. Although I did not learn much I am grateful to the faculty for making the event happen. In particular Joseph Pelrine who prepared a lot of new material to describe his journey with Cynfein.

Several tweetable soundbites (you can check them on twitter I think ) spring to mind from the event about the Cynefin version of social complexity…

• Seek out the cynics as they really care. The rest are just corporate survivors.
• Experiment in a “safe to fail” environment.
• The only way to understand a complex environment is to act within it.
• Ritual Dissent uses ritual to protect people from negative feedback.
• Complex situations are understood in retrospect.

As I journeyed home I compared Ritual Dissent to Feature Injection’s “Break the Model”. One advantage of Ritual Dissent is that it provides a mechnism for those with negative intent to feedback on ideas whereas otherwise they may not. I had not realised how important the ritual was to the process.

A significant proportion of the activity in the Agile and Lean Communities takes place on line in e:mail groups, linkedin groups, blogs, Skype and twitter. Conferences are an opportunity to re-energise and build new collaborations. Conferences are not the conversation, they are a high bandwidth blip in the conversation.

The Cynefin community seems to be much more close knit and more based on face to face contact with the central leadership. Joseph’s social network analysis seemed to indicate as much. This is possibly because most of the intereaction is through courses to date rather than through conferences and meetups.

Here are my retrospective thoughts.

Prompted by Willem’s blog, I wrote up my experiences of the CALMalpha event. I did it in the style of Ritual Dissent. In effect, I acted within the system to understand it.

I failed. And I apologised. I failed to apply Ritual Dissent correctly. Did I do it intentially? No. Did I pay attention properly when it was explained? No, I was hungover and too busy avoiding having to take my turn.

I was the cynic. I was the one to be sought out because I cared. If I did not care, it would have been much easier to keep my comments private.

There are a number of questions in my mind.

• Did I fail in a safe environment?
• Does the Cynefin community really value cynics?

and of course…

• Does the Cynefin community exhibit forgiveness?

Last week I went to #ScanAgile where Dave Snowden and Jospeh Pelrine were also speakers. I was more than a little uncertain how  things would play out as my failure had been pretty spectacular. I had a wonderful chat with Joseph Perine about the whole episode at breakfast. I was then fortunate enough to have a walk around Helsinki with Dave Snowden, Mika Latokartano (who acted as our guide) and Olaf Lewitz. Dave treat Mika and I to a wonderful luinch before Mika drove Dave and I to the airport.

Does the Cynefin community exhibit forgiveness? That’ll be a bug yes.

I learned a lot since the Cynefin event. I look forward to helping the CALMbeta event (from a distance as I limit my travel to the USA).

* not quite true. I did learn a lot from Jabe Bloom about photography. His ideas have already started to affect the way I take pictures.