Measurement can be a tool or a weapon. It is a well-known fact that most people tend to focus more on the activities against which they are being measured. This may generate good or bad results, positive or negative attitudes, constructive or destructive behaviours. A typical bad result is seen in the British (and others) education system, in which schools are funded based on their pass-rates (percentage of students who have successful exam results); schools and colleges therefore have focused their efforts on pushing students through the exams rather than educating them (this includes training them to pass the test and selecting easier exams or more generous scoring whenever possible).
However, without measurement, there is no progress or improvement possible - if you don't have measurements, you cannot even decide what improvement means, let alone determining whether progress has been made towards the desired result.
Two problems generally plague the concept of measurement. Firstly, we tend to measure things that are easy to measure (money, hours, widgets, errors, people…) rather than what would be useful. Secondly, possibly as a consequence of the previous, we measure only the cost and rarely the value.
Measuring the value as well as the cost would allow us to identify what are the activities that have good value versus those which are only cost - money and time wasters. However, that determines identifying what we mean by value. In most organizations I visit, when I ask senior management for their definition of quality, they cannot answer. When I ask how they see their organization in 3, 5 or 10 years, they appear to believe that things will continue as they are. When I ask about customer satisfaction, they reply that they seek to satisfy the customers' requirements. They cannot tell me what added value they provide which would differentiate them from a different supplier, offering the same products and services, possibly at a lower cost.
If your organization has no added-value, you will not survive. You need to identify what is unique to your product and services, the thing that will bring your prospects to you rather than to your competitors. That is the foundation for your definition of quality, which, in turn, is the basis for your measurement programme.
Once you have established your vision (and made sure that the vision is believed and shared by the team members) you can then determine what are the things you want to measure and why you want to measure them. Generally, I would say that the measurements should fall into two categories:
1. What are the activities which add value to the product or service?
2. What are the behaviours you want to encourage among the staff?
1. Activities adding value.
The measurements of activities adding value need to be seen from the customer's point of view: identify the activities which will directly impact customer satisfaction: those activities that lead to a more rapid delivery or a higher quality.
The first thing to measure in these activities is - of course - the value being created. Instead of measuring the cost of the activity (how long did it take, what was the budget), we need to start measuring the value of the activity (what is the percentage of defects removed from the product, what is the time saved).
The second thing we need to consider is the activity's flow: how much productive time was included in the activity and how much waiting time. I remember defining with a company the fact that we need to produce a policy. The results were that we needed 1 day to write the policy and three months to get it signed. This is a typical flow problem: the signature itself was a short time (supposing that the senior manager read the policy before signing it, it may have taken up to an hour), but three months were needed to get it to be the "next-item" in the manager's in-tray. This is an easy example of bad flow, but others are just as frequent: how long does it take to get a response to a request for requirements clarification from your customer? How long does it take to find the correct, up-to-date template of the document you need to fill in? How often do you get the training you need either too early (you have forgotten before you get to implement) or too late (you have already done the work without training)?
Understanding the value-flow and the waste related to it is critical when setting up a measurement programme.
2. Staff behaviour.
Most measurement programmes I have seen appear to be focused on ensuring that people have done a number of activities and tasks. The result is that people learn to do the work even if they do not see the value - or clearly see that this is pointless bureaucracy. In fact, the behaviour you want to encourage is one of continuous improvement and identification of issues and bottlenecks. You do not want your teams to follow some practices if they can clearly see there is a more efficient way to achieve the result.
The behaviour you want to encourage is largely one of communication and control. You need to encourage the bearers of bad news and risks - these are the people who are making your organization move forward, while the "yes-men" which are so often rewarded in large multinationals are only trying to satisfy themselves and help their career, even if they know that this is detrimental to the organization as a whole.
Bonuses should be based on the number of suggestions and improvements they have initiated or the implementation of which they have facilitated. They should certainly never be based on compliance to what might be perceived as a heavy, pointless bureaucracy.
One traditional problem that hurts organizations is the relationship between the work performed and the effort estimated. This problem is found in many areas and forms:
- the estimates are systematically reduced in order to make sure that sales gets the contract, then the product is delivered "late" (possibly corresponding to the original estimate);
- if more time than is necessary has been estimated, the person doing the work will work more slowly and use up the time that was allocated.
When the organization tells me that 80% of their projects are delivered late, it is easy to identify the problem. When I hear that they always deliver on time, I get more worried: are they being productive or over-estimating the work and then take the time allocated?
Can we measure the accuracy of the estimating activities? Yes - just like every thing else. You need to accept that estimating is not a precise science but always ends up with a guess, no matter how many tools and techniques you use, no matter how much data you have at your disposal, in the end, an estimate is still only a guess - hopefully an educated guess. Sometimes you over-estimate, sometimes you under-estimate...
What would you expect to see if things were done correctly? That is what you want to measure. And then, you want to check, expect and understand that the measurements you put in place may encourage the wrong behaviour; so, remember to clearly advertise why you are measuring, what you are looking for and that no misuse of measurements for personal reward or punishment will be tolerated. No mystery to this, if you think it through.
A strongly recommended best practice to be considered is what is commonly called GQM (or GQIM):
- G: Identify what is the goal which is prompting you to measure. This is not a definition of measurement but a starting point to determine the need for measurement. For example, reduce the number of defects.
- Q: What questions do I need to answer before I can achieve the goal. For example, "how many defects to we have?", "what is the origin of the defects?", "what are the kind of defects we generate?"
- I: It may be useful at this point to determine an indicator which would show the progress or the results, this may include a graph demonstrating that the amount of effort and cost being applied to correcting the defects, or the cumulative number of defects in a product throughout the lifecycle.
- M: Finally, we can determine and define the measurements that are going to help in this process, this would include clear definitions of what we mean by the word defect, when the measurement should be collected, etc.
Measurements can be a powerful tool or a terrible weapon depending on how well they are defined and used. I would strongly recommend getting help in identifying and communicating measurement methods and principles if you have not had prior success in this area.