Selecting Metrics for Your Agile Teams
One of our favorite sayings is “you can’t improve that which you do not measure.” When working with clients, we often emphasize the need to select and track performance metrics. It’s quite surprising (disheartening really) to see how many companies are limping along with decision-making based entirely on intuition. Metrics-driven institutions demonstrably outperform those that rely on “gut feel” and are able to quickly refocus efforts on projects that offer the greatest ROI.
Just as your top-level business KPIs govern strategic decision making, your agile teams (and their respective services) need their own “tactical” metrics to focus efforts, guide decision making, and make performance measurable. The purpose of agile development is to deliver high quality value to your customers in an iterative fashion. Agile facilitates rapid deployment, but also allows you to garner feedback from your customers that will shape the product. Absent a set of KPIs, you will never truly understand the effectiveness of your process. Getting it right, however, isn’t an easy task. Poorly chosen metrics won’t reflect the quality of service, will be easily gamed, or difficult to track, and may result in distorted incentives or undesirable outcomes.
In contrast, well-chosen metrics make it simple to track to performance and shape team incentives. For example, a search service could be graded against the speed and accuracy of search results while the shopping cart service is measured on the percentage of abandoned shopping carts. These metrics are simple, easy to track, difficult to game, and directly reflect the quality of service.
Be sure to dedicate the time and the mental energy needed to pick the right metrics. Feedback from team members is essential but the final selection isn’t something you can delegate. After all, If I’m allowed to pick my own performance metrics — I can assure you I’m always going to look awesome.
To keep you on the right track, below is a checklist of considerations to take into account before finalizing the selection of metrics for your agile teams:
- A handful of carefully chosen metrics should be preferred over a large volume of metrics. Ideally, each Agile team (and respective service) should be evaluated/tasked with improving 2-3 metrics (no more than 5). We have witnessed at least one company that proposed a total of 20 different metrics to measure an agile teams performance! Needless to say, being graded on that many metrics is disempowering at best, and likely to illicit either a panic attack or total apathy from your team. In contrast, having only a handful metrics to be graded against is empowering and helps to focus efforts.
- Easy to collect and or calculate. One startup suggested they would track “Engineering Hours Spent Bug-Fixing” as a way to determine code quality. The issue was quickly raised: Who would be doing this tracking? And how much time/effort did they estimate it would take? It became obvious that tracking the exact amount of time spent would add a heavy productivity-tax to an already burdened engineering team. While providing a very granular measure, the cost of collecting this information simply outweighed the benefits. Ultimately we helped them decide that the “Number of Customer Service Tickets per Week” was the right metric. Sometimes a cruder measure is the right choice, especially if it is easier to collect and act upon.
- Directly Controllable by the Team. Choose metrics that your agile team has more or less direct control over. A metric they contribute towards indirectly is less empowering than something they know they own. For example, when measuring a search service the “Speed and Accuracy of Search” is preferable to “Overall Revenue” which the team only indirectly controls.
- Reflect the Quality of Service. Be sure to pick metrics that reflect the quality of that service. For instance, the number of abandoned shopping carts reflects the quality of a shopping cart service, whereas number of shopping cart views is an input metric but doesn’t necessarily reflect service quality.
- Difficult to Game. The innate human tendency to game any system should be held in check by selecting metrics that can’t easily be gamed. Simple velocity measures are easily (read: notoriously) gamed while the number of “Severity 1” incidents your service invoked can’t be so easily massaged.
- Near Real Time Feedback. Metrics than can be collected and presented over short-time intervals are the most actionable. Information is more valuable when fresh — providing availability data weekly (or even daily) will foster better results than a year-end update.
Most importantly, well-chosen metrics tracked regularly should pervade all aspects and all levels of your business. If you want your business to become a lean, performance driven machine, you need to step on the scale every day. It can often be uncomfortable, but it’s necessary to get the returns of which you are capable.