Reviving CMMI

Part 1 (19 May 2013)

Recently it seems I am regularly being contacted by (or informed about) people and organizations trying to revive CMMI*. The market appears to be shrinking for the professionals in the process improvement world; at least, the market is not growing as fast as the number of professionals being certified to teach it or use it for official appraisals. Add to that, all the "experts" and other consultants who have set up shop without having official training or recognition and you get into a lose-lose situation. And so, various groups appear to be talking about making CMMI faster, leaner, easier, cheaper, more secure. They are offering different approaches and ideas, some of them very good. My first concern is that the model is easier and cheaper to implement, it will have to be stripped of a lot of its more advanced requirements, and the benefits thereof. But the model is not being used as efficiently - or as frequently as it should.

One of the most difficult choice people need to make regarding a loved one is the decision that maybe it is time to "pull the plug", to stop trying to keep alive a dying parent. As a CMMI instructor, appraiser and consultant, it pains to suggest this, but maybe the time has come to pull the plug on CMMI.

About thirty years ago, the US Department of Defense required that all their software suppliers should be "Maturity Level 3"; but they have now stopped because they noticed no significant improvement in the quality or reliability of the products and services they were purchasing! Companies all over the world have been applying CMMI and nothing has not noticeably improved!

I believe that it is an excellent product and, through its various transformations from maturity questionnaire to software-CMM to CMMI v.1.3, the product has generally got stronger. There are a few things I would have done differently, but overall, the product got stronger and better.

However, it is not delivering the expected results and, notwithstanding the strong focus on measurement, there are still no clear statistical data on the benefits. Yet, it works. I know: I have seen it.

Maybe this is the time to accept that there is a fundamental issue with the model and an in-depth review of what it is supposed to be is necessary. As I see it, there is a big weakness in the tool as it tries to be two different things.

If it is to be a measurement tool, used for appraisals and measuring the aptitude of potential suppliers, then the appraisal methodology needs to be simplified - an ISO audit typically lasts a couple of days; why can't we do it as efficiently? On the other hand, if it is to be used as a process or performance improvement tool, then it should be completed and extended with implementation suggestions.

As long as we don't know what the purpose of the model is, we will be stuck with this hybrid version, neither here nor there, not this or that, trying to be staged and continuous.

Currently, the value of the model for efficiency and continuous quality improvement is undermined by the vast majority of users who are only interested in getting a maturity level. They are doing the minimum required to get the level. As soon as the lead appraiser leaves the organization, they stop doing the tasks that were just recognized. The result is that we see organizations all over the world who have fooled an appraiser into giving them an undeserved rating - thereby demonstrating that apparent high-maturity organizations continue to deliver bad quality, late. The natural conclusion is that the model is a waste of time and money.

At the same time, more consultants and appraisers are continuing to be produced, with little experience in the real world. The pressures of the market, of the people trying to game the results, makes it very difficult for these young consultants to have any sense of solid ethics.

Now, it is time to let go and do a postmortem, identify what went wrong. Then, we can lay the CMMI to rest and create something which works more efficiently - maybe CMMI 2.0?

(*CMMI is the Capability Maturity Model Integration, licensed by the CMMI Institute. It is a process model used to measure the maturity of an organization with regard to their processes and work practices. CMMI is a registered trademark of the Carnegie Mellon University, based in Pittsburgh, PA)

Part 2 (23 May 2013)

It seems that my previous post called "Reviving CMMI" generated quite a lot of reactions. Some people seem to understand that I was suggesting to let the old girl die, some encourage the idea, others were horrified at this attempted matricide. A few reacted supportively or critically without giving enough information for me to know what they thought I had said.

So, I decided to clarify my feelings on the subject. I am a process improvement consultant, I have been living off CMMI for many years and would not recommend cutting off the hand that feeds me without careful consideration.

  1. CMMI cannot be considered as fit for purpose
    This is largely because the owners of the model, the user community and the market do not agree on its purpose. The CMMI Institute (following on in the footsteps of the SEI) places a large emphasis on appraisals and maturity levels, publishing numbers of appraisals, time to reach a level, number of maturity levels per country and per industry - in fact all the measurements and data produced are directly measurements of the appraisal results. But, at the same time, we are presenting CMMI as a tool for process and productivity improvement rather than a certification diploma.
    Without a clear understanding of the purpose, it is not possible to design something fit for purpose.
  2. As a Tool for Improvement
    As a tool for productivity improvement, the model does not contain enough information to facilitate a seriously useful and helpful implementation. There are hidden relationships between process areas and practices which are not easily identified or understood.Personally, I try to use the model as an efficiency and quality improvement tool; I need to spend an excessive amount of time clarifying the cause and effect relationships within the structure of the model. I also need to explain in detail how to understand the purpose and meaning of things within a business context. The standard training does not explain the evolution from maturity level 2 to 3 and beyond. There is a vague statement that it is not recommended to skip levels, but no clear rationale clarifying what are the risks and consequences. The relationship between specific and generic practices is not sufficiently clear in the model or the training. These are vital facts if you want to use the model for your business. If the model is to be focused on improving quality and productivity, it needs to include more information on how to apply it successfully.
  3. As an Appraisal Mode
    An ISO audit takes a couple of days, a CMMI appraisal can take a couple of weeks. Why? A number of "certified lead appraisers" do not appear to understand the purpose of the model. There was a recent case of an organization which was required, according to their appraiser, to have a separate policy document for each CMMI process area, clearly stating the name and structure of the PA - this is not the goal of the model, but people with no experience of the "real" world are being authorized to appraise successful organizations; they are frequently focused on respecting the comma of the law without understanding.
    The current appraisal method spends a lot of time trying to find evidence of practices, but could be a lot more focused on the impact and results of successful implementation of recognized and accepted best practices.

Moving forward

As I stated, I believe it is time to perform an in-depth lessons-learned analysis to find out what went wrong and how to correct the product, making it into something that will have the impact which was promised.

This must start with an understanding of the purpose of the beast. If we are talking about a tool for process improvement, we need an approach to educating of practitioners and users, which focuses a lot more on the practical side of change management and improvement. We need more information regarding the implementation of the practices. Potentially, this may mean that the core model gets completed with a series of "recipe books" for different industries or contexts. I would like to see the model completed with clear business related impact and influence statements, clarifying why things need to be done to save those who are implementing the letter of the law from their own stupidity.

I would like to see CMMI separated and organized so as to distinguish the improvement potential from the appraisal requirements. I am not sure if both can survive with the same name, but trust that the SCAMPI appraisal methodology can be adapted to other models and standards and be recognized in its own right. The appraisal methodology needs to focus a lot more on the business and cultural aspects of the model, stopping lead appraisers seeking to burden businesses with bureaucracy because a sub-practice says that is the way it should be.CMMI should be perceived as a pragmatic approach to assist organizations increasing job satisfaction and customer satisfaction. And we should be able to demonstrate that from the beginning. The appraisal method should focus on measuring the results, not the practices.

Part 3 (03 June 2013)

Third item on this topic, I know. Some people believe that I spend too much time complaining without proposing a solution, so here is my proposal: measurement and analysis should be expected from the start.

Many times, when I have asked for evidence of the implementation of measurement and analysis, I have been provided with evidence of project monitoring and control. Measuring that your project tasks are progressing, that you are respecting delays and budgets is not part of MA, it is part of WMC/PMC. It should not be difficult to include in either the model or the appraisal methodology a better set of examples of how MA should be applied, and place this as a requirement in the appraisal process. Currently, the appraisal method focuses on the practices; during the training, we say that the goals are what is important and are required, the practices are only "expected". However, during the appraisal, we focus on measuring the practices rather than the goals. We assume that if all the practices are in place, then the goal must be satisfied, if one practice is missing, then the goal is not satisfied. And even if we did, I cannot help but notice that the goals (and the purpose) statement all focus on activities, on tasks, on practices, and not on results.

Let's require business measurements and demonstrations of results for each goal. CMMI is (supposedly) there to help achieve business productivity results and not just to do a series of tasks to get a certificate we can hang up behind the reception desk. Why do you do "Requirements Management"? Show me results. Show me that the number of issues related to unidentified change requests has diminished; show me the reduction in unexpected requirements appearing during the V&V stage; show me data that show how the time to find a deviation from customer requirements in your project plans and other work products has gone down, because you have implemented a successful "two way traceability". Show me measurements that demonstrate the impact on the quality of your products and services, on customer satisfaction. If you cannot show me that, you have not implemented a useful approach to requirements management. Of course, people will argue that you cannot measure improvements at maturity level 2, you can only really have useful measurements at maturity level 4. But that is not true. You do not need control charts or five years of history to show a trend. Because if you did, no one would manage to satisfy the expectation for a quality trends in the PPQA process area at maturity level 2 - or maybe you have just skipped over that passage and focused on having a static checklist? If you implement a practice, whether from the CMMI or elsewhere, it is because you expect to see a change one that practice is performed; if you expect to see a change, there is something which can be measured.This is not a complicated addition to the model, it is only a clarification of the real purpose behind each one of the goals. A change in the appraisal process of this magnitude would ensure that people understand that CMMI actually has a benefit, and it would allow us (finally) to have some decent metrics as to the value of the approach.

I don't know if anyone will read this or pay attention, but I am glad I got it off my chest. Hopefully, my next post will be about something else. Maybe I will develop a business based appraisal system of my own, but if I am not supported by the community at large, no one will use it and I will waste my time: it is so much easier to do what the model says without thinking.

and you might get a certificate to hang up, a logo to put on your website...

Leave a comment

Make sure you enter the (*) required information where indicated. HTML code is not allowed.

back to top