Make Sure Metrics Don’t Kill Your Business

Measuring programming progress by lines of code is
like measuring aircraft building progress by weight.
—Bill Gates

Business metrics—the quantifiable measurements by which a company’s performance is gauged—are part of the broad area of business intelligence (BI) or business analytics (BA).

Measuring programming progress by lines of code is
like measuring aircraft building progress by weight.
—Bill Gates

Business metrics—the quantifiable measurements by which a company’s performance is gauged—are part of the broad area of business intelligence (BI) or business analytics (BA).

In this regard, metrics are the application of a mathematical expression to a set of data to analyze it and obtain a figure. This figure helps to quantify a business process and, consequently, determine its performance status.

Metrics are a core component of obtaining a view of the business in figures. Some key metrics that many organizations consider important are profit, net income, cost of goods, and services sold.

Almost everyone agrees about the importance and benefits of being able to measure performance in order to make it possible for an organization to improve business performance, make more sales, or increase customer satisfaction.

But metrics when defined and managed improperly can not only give poor results but also mislead and create more problems than they solve. Why? In the important 1998 paper Metrics: You Are What You Measure!, it’s explained as follows:

Every metric, whether it is used explicitly to influence behavior, to evaluate future strategies, or simply to take stock, will affect actions and decisions. If a brand manager knows that, in his or her company’s culture, a “good brand is a high share brand,” he or she will make decisions to maximize market share—even if those decisions inadvertently sacrifice long-term profit or adversely affect other brands in the company’s portfolio.

The authors cite several other examples of how metrics can bias performance and outcome measurements. Thus, defining and using inadequate metrics can potentially be quite misleading, if not fatal, for an organization.

The Pitfalls of Designing Metrics

The above-mentioned document describes seven pitfalls as examples of how metrics can give counter-productive results. In brief, they are as follows:

1. Scope in time—delaying rewards. Metrics that are defined with a view to long-term goals may be discounted by users, who tend to focus on shorter term goals, to say the least fall short on expectancies.
    
2. Uncertainty dependence—using risky rewards. Metrics that depend on an uncertain outcome will be risky.
    
3. Impact–scope—making metrics hard to control. A metric that considers factors that are out of the scope of a certain group will be hard to control; the goal may be easily dismissed and thus will represent little or no overall impact for the organization.
    
4. Goal deviation—losing sight of the goal. A metric that takes into consideration the wrong goal or loses the goal along the way may detract from the organization’s desired outcomes.
    
5. Precise irrelevance—choosing metrics that are precisely wrong. Accuracy does not ensure usefulness. A metric that is accurate but not relevant can be misleading.
    
6. Metric tyranny—assuming your managers and employees have no options. Metrics that demand a great amount of effort from managers and employees will soon lead to extra payments and costs.
    
7. Tunnel view—thinking narrowly. Metrics not designed to anticipate the future or to promote innovation may limit a business’s growth.

Designing metrics is a difficult process, and it is not always possible to avoid all these pitfalls. But understanding the potential pitfalls can go a long way toward minimizing their effect.

To be relevant, metrics must evolve. Some metrics created to serve a specific purpose might not be useful over time. I would add an additional pitfall to the list: treating metrics, once created, as written in stone. The most important drivers of metrics are improvement and maximizing gains (profit, performance, customer satisfaction, etc.). Once a metric is identified as no longer effective, it must be reviewed, modified, or even replaced to better reflect actual business conditions and performance.

Guidelines for Good Metrics

Good metrics are hard to identify. There is no one-size-fits-all set of metrics that will suit all organizations. Each company must design its own metrics according to its purpose, its structure, and its business goals. Along with being mindful of the pitfalls, here are some guidelines for implementing successful metrics:

1. Listen to your users. The driver for the metric is the business. Listen to what your stakeholders have to say and understand what they want to achieve. Knowing how the stakeholders work will not only help you define the goal, but also lead you to their expectations. Compare your metric against these expectations.
    
2. Prioritize elements for major effect. Consider for your metrics those components that will have greater impact on your metrics and that will you be able to affect or improve.
    
3. Stick to your objective. Always keep your goal in mind and refer to your stakeholders to keep you on track. Keep informed and solicit their feedback.
    
4. Establish your initial expectations. When defining metrics, try to define your expectations as well. Perhaps a reasonable value is already known, or you can estimate a value based on experience.
    
5. Avoid uncertainty. As much as possible, define metrics based on your available facts. Establish your calculations based on data that is available and under your control.
    
6. Test and improve. Test your metric against your expectations and take your stakeholders opinion into account. Make it possible for your users to contribute to evolving your metrics.

These guidelines will help keep your metrics focused, manageable, and useful. Additionally, from the perspective of managers and IT departments tasked with measuring performance, the chances of formulating good metrics are vastly improved by understanding the business as a whole. Understanding the business case for implementing metrics plays a basic role in formulating the right metrics.

Achieving Balanced Metrics

Accuracy is important, but it is not necessarily the only component of a good metric. We try to design metrics with a high degree of accuracy and precision.

In scientific fields, accuracy and precision represent different measures. Accuracy is the degree of closeness of measurements to a quantity’s true value. Precision is the degree to which repeated measurements under the same conditions produce the same results.

Creating a precision metric does not necessarily mean creating an accurate one. But just because a value is repeatable doesn’t necessarily make it the number, or even the measure, we are looking for.

The key to obtaining a metric that truly measures what we want to measure is striking the right balance between the following elements:

• The goal of the metric (what will be measured)
• The accuracy of measurement (obtaining the real measure)
• The necessary precision to repeat the measurement (automating the process of obtaining it)

Finally, metrics can be the core of an effective program for measurement and improvement within your organization, but can also be misleading and deceptive, generating more losses than profit. To achieve better performance, look to design metrics that truly represent what you need to measure and that are as accurate and precise as possible, as well as aligned to your organization’s strategy.