Larry Dribin: When I first started working it was with Automatic Electric, a company that manufactured telephone switches. They assigned me to a very large 200 man-year project, in which the original PERC project management technique was utilized. I learned how to use this and as a result of the experience, mistakenly concluded that this was how all projects were managed in the industry. Not surprisingly, we had very high success rates; in particular, one 5 year project that was brought in within a month of schedule and right on budget.

After my Automatic Electric experiences, I left for what I thought were more advanced companies. It turned out, though, that the companies I came across were not that advanced at all. In fact, everywhere I went I became the guy who introduced process. Because I really enjoyed this kind of work, I decided to join a consulting firm that was specifically focused on process improvement.

The company was Digital Equipment Corporation. I started out with them in the area of computer aided manufacturing. This was about the same time that Motorola was developing Six Sigma, winning the Malcolm Baldridge Award and making the move from hardware to software in the manufacture of their radios and telephones. I got very involved in helping Motorola recognize the need for software process improvement. Digital was a leader in that area at the time. Consequently, I wound up moving over from their manufacturing side to their software side.

After Digital, I took a position with Cap Gemini. I joined an organization within Cap Gemini that had been formed from Howard Rubins old company, Rubin Associates. The team was called The IT Excellence Group and they were entirely focused on IT process improvement. That was in 1994. I have been involved with IT process improvement ever since.

Larry Dribin: Never has process improvement been more important than right now when the American computer industry is being seriously threatened by offshore competition. A key reason for this is that other countries, notably India and China, have a greater appreciation of the value of process improvement. We certainly have excellent techniques for process improvement at our disposal here in our own country; however, we don't use them nearly as fully or as effectively as the Indians or the Chinese. They are leveraging process improvement aggressively, to drive a wedge into the American market. In so doing, they are jeopardizing our IT industry and our jobs. It really should be a national concern.

Based on my work at more than thirty U.S. companies, I think our industry has been just terrible at running IT development and IT operations. Almost none of the companies I have worked for have used metrics very effectively and few have ever really understood process improvement. The most common and pervasive understanding of improvement is that you get rid of someone who is not doing a good job. But without a process orientation, just getting rid of someone won't lead to improvement.

Larry Dribin: So many companies don't do good portfolio management or good quality assurance. Half the IT organizations don't check time, the other half dont have QA groups. I think the key, in light of this, is to try to get these companies literally one by one, project by project to start following proven software engineering processes. We all know what they are. Its time to start getting serious about making use of them.

CMMI is growing in popularity year by year; each year they double the number of people involved. Nevertheless, many companies don't really understand the CMM and they wind up implementing it the wrong way for the wrong reasons. Part of the remedy for this is to take a look at the agile development people and to combine a lot of their ideas with the CMM. In fact, my current research focus revolves around the creation of an agile implementation for the CMM.

Larry Dribin: I would contend that the CMM road map is really about basic project management and software engineering skills. While I don't think you necessarily have to follow the CMM to improve these skills, it really is the best framework out there. The problem is that most organizations have so many areas that need improvement. The real challenge winds up becoming where and how to start.

Larry Dribin: First of all, the CMM framework provides a comprehensive definition of process. Unlike other standards, the CMM maps out the creation of a process environment through the assembly of process focused groups. Furthermore, much of the CMM is focused on metrics based project management. It asks that we create project plans, monitor our plans, and track our estimates. Many of the developers and project mangers want to do that, but when push comes to shove, they often don't. Using the CMM as a tool helps them do what they really want to do and, at the same time, to manage their customers better. We all know that we should do reviews and inspections. We all know that we should do unit testing. But so often project managers are under so much pressure from their customers that they don't really know how to say no. Someone who worked at the SEI once said that the CMM could all be summarized in one phrase, learning how to say no to your customer in a professional way.

Larry Dribin: Success seems to require a strong executive sponsor who has been in place for a long time. Many of these process improvement initiatives take years.

I was recently involved in a project where the leaders said that they wanted to implement the CMM, but they lacked a solid understanding of the process. Not surprisingly, when we started to implement it, they began to waver because they found it difficult and time consuming.

This actually epitomizes a major disadvantage that the American IT industry faces. Because most of our own development organizations have been around a long time, we have to do a lot of unlearning and relearning. The Indians and the Chinese, in contrast, are able to just start up greenfield: they define all their processes, hire everybody they need, and mandate that everybody follow the process. By comparison, we now have some people in the U.S. who have been developing mainframe applications the same way for over 25 years. Its not easy to ask them to make changes. You need a lot of strength and a long time to get them to understand the need to retool and modernize.

That being said, the first key to progress is a strong sponsor who will last through the typical reorganization that most IT organizations go through every two years. The second key is to pick a framework. I think the CMM has the best one but it is by no means the only one. You could decide to implement RUP, or maybe you might want to have all of your project managers PMIcertified. Those are just a couple of alternatives.

Larry Dribin: First of all, you shouldn't have a process defined until you have measures for your process. The paradox, of course, is that you usually can't do a very good job measuring without first having a defined process. The two go hand in hand.

I think the real problem in most IT organizations is that there are way too many measures that is, too many measures that have too little value. One organization I am familiar with had 115 measures of which one was actually useful in managing the IT business. The CMMI talks about four core measures, and I believe that these are the measures that organizations need to make use of. One basic metric is how much time a group of individuals spends on a task. Another basic metric is the number of defects found at the end of a given period and how much genuine productivity occurred.

At the macro level, I feel that the balanced score card is a good technique, one that can be coupled with the Goal-Question-Metric approach for defining goals. You can start out by asking what metrics you need for finding out whether your strategies and your more detailed tactics are working. Later on, you can move ahead with the scorecard. Scorecards are dashboards that provide real value to different stakeholders on a project. Such data often tends to be available in organizations. Its just that no one ever bothers to put it into a form that anyone can understand.

Usually stakeholders are very excited when they see a scorecard. It helps drive, and make the case for, process improvement. After all, it's very hard to ask a project manager to stop in order to do a review that you really need. But with the scorecard, youre going to find defects earlier. And one way to do that is to measure how many defects come out of a project that has utilized peer reviews (versus a project that hasn't). Pretty quickly you start to see that there is a tremendous value in peer reviews. Even though it increases time up front, it winds up saving time later.

Larry Dribin: When I design a metric, I create a metric catalog with private measures and public measures. Certain measures are private; for instance, you wouldn't really want to know the productivity of each programmer at the CIO level. Not to mention the fact that very often your best programmers will have the worst productivity because youve given them the hardest problems (and also because they are frequently helping all the people around them). You also have to be very careful not to publish certain measures to some people. But on the other hand, you will want to publish other measures that help reinforce the kind of behavior you desire. More frequently than not, however, the publication of measures is counterproductive.

As far as automation is concerned, its wise to remember what Watts Humphrey once said a long time ago: Don't go into automation too quickly. If you don't know what youre doing with a new power saw you could easily cut your finger off. You have to first learn how to measure and how to figure out the kinds of pieces you need. Too many companies have bought tools IT governance tools, estimating tools, or metric tools and then spent all their time playing with the tools without ever understanding the core uses of their measures. Tools can also wind up generating too many measures that you don't need. My advice, in light of this, is to focus on identifying the core measures and the derived measures first, before jumping into automation.

The easiest way to do this is to start out with some simple spreadsheets that present simple measures and then learn how they use them. Not until you have used them for a while, three or four months, and become proficient at this should you begin to trouble yourself with automating the process. I think this counsel applies to all processes. Figure out the process first and then figure out where automation can help. Its much better to start simply.

Let me give you one little side note that exemplifies American backwardness in metrics as compared to the rest of the world. I have gone to many U.S. companies and asked many senior IT managers what defect density is. Most of them don't have a clue that defect density refers to the number of defects over the number of lines of code that have been produced (over a long period of time). In sharp contrast, I was in the Netherlands recently where I had an opportunity to visit one software companys development organization. I discovered that they had graphs on the walls showing the defect density for every release they had completed over the last seven or eight years. We have a long way to go before we catch up with the rest of the world.

Larry Dribin: A lot of the organizations that measure haven't done a very good job of it. I mentioned earlier a company which used 115 measures, only one of which was useful. Their root problem was that their time codes weren't defined sharply enough. This is a common problem. Consequently, one of the first things that companies can do is to review all of their time codes so that they can make sure they are getting meaningful time data at an appropriate level of granularity. Another common failure is the improper coding of defect data within an organizations help desk or incident management system. When the data doesn't get coded properly, the defects cannot get tracked back to a specific application, release, or project. Companies need to make sure that their defect coding is complete and precise.

I've found that if you put together a very simple view of four or five metrics that project managers find useful, they will eventually want more. This will actually help you insofar as it will get your constituents and stakeholders more receptive to an overhaul of their measurement system. At that point, you can help them re-design in a more agile style. In many cases, they will come up with a dashboard in a month or two that has four or five useful measures and that can be iterated every month. You can then add some additional metrics and at the end of maybe six to nine measures you will have a pretty good dashboard at your disposal that management can use to run the business.

Another benefit of this dashboard technique is that you can start getting all of your raw data, good or bad, into a data warehouse where you can slice it, dice it, and combine it to come up with more useful measures.

Larry Dribin: I think the key to everything we have talked about is that most organizations getting involved with process improvement don't really understand that process improvement is about organizational change. We may want to change the behavior of 50 developers or 200 QA individuals. To do this, however, we really have to get out of the world of software and into the world of organizational change management. Theres been a lot written about this, but too often the software process improvement people have never been exposed to the literature.

We need to ask ourselves questions like: How do I look at the organization culture? What changes do I need? How can I contribute to creating the vision for the future? John Kotter probably has the best book I've seen on this. Its called Leading Change. Its written at a high level, but I think it offers great insights for changing our organizations.

The CMMI was actually designed with this in mind. There were a number of folks on the original CMM design team who were very familiar with change management and industrial psychology.

Larry Dribin: The Balanced Scorecard by Kaplan and Norton is another valuable book on metrics. Certainly many of Watts Humphreys books on software process improvement are helpful. And the SEI and CMM teams original book, Extreme Programming, is very useful, too (I actually feel that the first edition is much better than the second). Finally, there is always Fred Brooks timeless Mythical Man Month. Brooks offers a deep understanding of the mistakes we can make in software development and of the real challenges that we face in getting it right.