Engineering Metrics

To measure is to know.

&

If you cannot measure it, you cannot improve it.

• Lord Kelvin

Metrics and Monitoring

A little context to set the stage: “metrics,” in the sense that I use the term here, refer to measured or calculated values used to assess status and drive decisions. In the world of software this is often confused with “monitoring,” whereby many states and values are tracked and/or measured in order to serve as the basis of early warnings or alerts if a suspicious change is detected or a threshold is crossed.

For example, if you routinely check the average time it takes to load your company’s web pages, you can  use that average time for “monitoring” to warn of a sudden increase. If you use that same average time to guide your engineering team’s work with the goal of decreasing web page load times, then it is being used as a “metric.” 

It should be clear that the same value can be used for both monitoring and as a metric, or it can be switched to one or the other of these uses at different times. For example, you may monitor page load times until they exceed a specified limit (somewhere around two to three seconds is often used), at which time you may decide to make these times a metric you track while the engineering team works to reduce it. Once the average page load time falls back below the chosen value, you may switch back to monitoring.

If the same numbers can be used in either way, why make a distinction? 

You should monitor a wide range of values across many systems. Monitoring is intended to be a tripwire that alerts you to possibly dangerous circumstances. You can err on the side of too much monitoring and reduce the number of places you monitor only if you find yourself receiving false alarms.

Metrics are used  to assess the ongoing state of something for the purpose of driving actions and decisions about it. There is a limit to how many values you can regularly track and act upon – choose too many and you lose focus. Whether particular metrics are used by your engineering team to guide them (as in the average page load time example, above) or presented to management to provide evidence of an engineering issue, the choice of metrics should be made carefully and restricted to a small set.

Software Engineering And Metrics

Software engineers tend to be suspicious of metrics. While many of the processes involved in software engineering can be readily measured, many of the measures traditionally used as metrics to judge the quality of software, people, and processes are quite flawed. 

The archetypal example is Lines of Code (LOC). If used as a rough guide to the overall complexity of a software project, LOC has some relevance. But many companies have used this metric to represent programmer productivity, with the implicit assumption that programmers add value primarily by increasing LOC. This ignores efficient programming, refactoring, time spent mentoring, and all of the other ways in which the most proficient engineers actually increase productivity.

A side note: I regularly look at the word count for the blog articles I create, not to judge whether or not I am being productive, but to get a feel for my succinctness and whether or not I should break a topic up into multiple posts. The word count is at 1007 at the time I write this – still reasonable for a single post on my site. For this limited purpose, a simple metric like word count is useful.

So Why Bother?

If you avoid all metrics because poorly conceived metrics are subject to abuse, you are throwing out the baby with the bath water. 

Metrics can provide quantitative insight into what is going on with your people and processes. Take reasonable care to make appropriate choices and they can illuminate your current situation and help drive action in desired directions. When clarity and agreement are reached regarding chosen metrics, they can provide a common understanding and a way to measure a particular issue.

Used within a team, they can provide a concrete goal the team can work toward and enable them to evaluate the success of their efforts. Instead of asking a team to generally “do better,” you can agree upon specific metrics and measure how well they are doing.

Metrics can be an excellent bridge across teams; for example, between engineering and the rest of the company (also known as the real world). Metrics can illustrate progress, problems, and challenges in an easy-to-understand way that non-engineers can grasp. Company leaders appreciate a concise view of the performance of (and challenges facing) the engineering teams. Metrics can help you get alignment with management on the people, resources, and time you need to address your goals.

So how do you get to the point where you have useful metrics? By iteratively developing them until they meet your needs.

Develop Your Metrics

Ideal metrics don’t usually fall into your lap – you will need to develop them.

First: avoid analysis paralysis. It is easy to not choose anything because nothing is exactly right. Perfect metrics are rare – you should be aiming for useful metrics. Consider various options, then pick a simple starting point. You may need to tweak them a bit to match your needs.

For example, you may be concerned about the ongoing quality of your product. You could choose “new bugs in the past month” (NBPM) as your metric, and track its rise (or, ideally, its fall), recognizing that it is an imperfect but potentially useful stand-in for the quality trend of your product.

It is not unusual to find that the initial version of a metric does not track closely enough with the desired action/behavior/value. Don’t give up. First, try to refine the metric. If that is not possible, you may need to discard it and try another. If you can’t measure exactly what you would like to, find a proxy. If you use an indirect metric, note the ways in which it differs from the ideal metric.

You may find that the NBPM rate tends to rise over time, but fluctuates more dramatically during months in which you release more code. You could refine the metric to represent only the bugs serious enough to be fixed, or you could normalize it based on the number of releases. You could also switch to something like “Bugs reported by customers in a month,” if that is a better metric.

Your goal is to get a useful value that helps you understand your current situation, and whether or not you are having a (hopefully positive) impact on it. Repeat this process until you find a value that works for you.

With metrics, like code, it is useful to get another set of eyes involved. You may believe that you have selected the right values to track. But other folks may be able to point out holes in your reasoning, or limitations in the utility of your proposed metrics, or better alternatives. 

Get buy-in on your metrics! Get the people who will need to use them involved in the process.  Explain the reasoning behind your choices and provoke discussion. If your metrics are intended to provide information that drives behavior (if they are metrics and not just monitoring) you need to achieve the agreement and understanding of the involved stakeholders. Otherwise they will be only “your metrics,” and will be subject to misinterpretation. You want people to think of them as “our metrics” and truly understand them.

When you have chosen metrics that you believe are useful, spend a little time documenting them. Carefully explain the source of the metric, the intent behind it, and any caveats (like whether it is actually a proxy for a difficult-to-measure value). Metrics, like everything else, are subject to interpretation. The context you provide will help others understand and accept the metrics you propose. This is useful both within an engineering organization where people may be leery of metrics, and outside the organization where they may not be familiar with their sources and caveats.

If NBPM is related to only certain types of bugs or normalized by number of releases (or days of the month), make these points clear in your documentation. Keep your documentation with the metrics themselves, so people can refer to it if they have questions. Provide historical context or any useful background. Discuss them with your engineers if you plan to have them work to minimize or eliminate a problem. If you plan to use a value as a proxy for product quality, discuss this with your management team. 

Metrics can lose their utility over time as issues are resolved or your focus switches to more important areas. This is a natural end-of-life for a metric. When a metric is no longer useful, retire it.  This could be a good thing if the issue it was created to address has been resolved! At this point you may need to consider a new metric.

Where to Start

There are oodles of websites that provide examples of metrics relevant to various topics. Browse around to get some ideas. But base your metrics on the specific issues you are trying to address.

Do you believe software quality is a problem for your teams? Google “software quality metrics” and look through the results (I found 130 million results but did not read them all). If you believe your teams are not delivering regularly or need to improve some aspect of your product’s performance, search the web for various ideas. Most of the results you will find provide the context and reasoning behind the metrics they suggest.  Take your time and compare various perspectives. In the end, you don’t want someone else’s ideal metrics – you want metrics that work for you and your teams.

The process of figuring out what to measure and how to measure it can feel overwhelming, particularly when you are analyzing more abstract qualities that resist direct measurement. A reference I found to be both practical and a little bit inspirational is Douglas W. Hubbard’s How to Measure Anything. Mr. Hubbard worked as a consultant for years, finding ways to measure values that most of us mere mortals would think unmeasurable.  The first part of this book is a good read just for the stories about measurement challenges and how they were addressed! The latter parts of the book delve more deeply into the basis of good and accurate measurements and involve math and statistics. Read just what you need to address your current challenges and keep the book around for future reference.

Developing useful metrics is like developing useful code. You get better at it with careful thought and a little time. Become proficient at developing insightful metrics and they will help you and your team shed light on issues and drive toward solutions.

Side note: This blog article is at 1843 words after completing the first pass. A little long, but still reasonable for a single post.