Tag: metric

 

The ‘observer effect’, a concept having its genesis in physics, notes that the measurements of certain systems cannot be done without affecting the system itself. Interestingly, this physical phenomenon could also be seen to be at play in the social sciences when economic participants interact.

And such interactions could result in both good and bad outcomes, owing to the act of measurement and depending on what is being measured.

A company that scores poorly in customer satisfaction surveys could be expected to push itself to improve its products and services to achieve higher customer satisfaction. The race to achieving industry leadership is premised on topping the market share metric. One of the markers of a start-up having arrived is reflected in its unicorn status. Thus, the appeal around measurement and its ostensible relevance to achieving desirable outcomes, is undeniable. After all, ‘what gets measured, gets managed’, as the adage goes.

Vanity metrics

In contrast, if the choice of the metrics is less rigorous, it would ensure that what gets measured gets gamed. The saga of the Ease of Doing Business rankings, which involved methodology manipulation to favour the rankings of select economies, presents the dark side of questionable measurements. Determining the worth of companies based on vanity metrics such as total registered accounts (as opposed to active accounts) is another approach that could arguably be decried. The thickening of annual reports because of the push towards more disclosures — a noble objective per se — tends to enable companies to mingle valuable information with less consequential information and, hence, obscure more than reveal.

History has enough examples to suggest that when a metric is conflated with the broader outcome that it is aimed at measuring, it results in unintended or sub-optimal consequences. Thus, optimising the efforts towards achieving results around some narrow metrics, could violate the objective itself, as the famed Cobra effect demonstrates.

Therefore, the choice of the metric and what it seeks to measure must be an important design consideration. Overall, while design simplicity is desirable, rigour ought not be sacrificed at the altar of simplicity.

As an example, choosing simple metrices such as ‘averages’ or ‘medians’ for tracking the performance of any target parameter could be alluring, but one would risk losing vital information if the ‘variability’ metric around the average/ median is ignored. This would be analogous to an image projection of a 3D object, say, a cube onto a 2D surface, which would make us see a square, but would reflect a metaphorical loss of information.

Counterproductive move

Likewise, choosing metrics that do not fully embody the essence of what is intended to be measured, could be counterproductive. This just emphasises the point that not everything that matters can be measured, and not everything that can be measured matters. Another relevant aspect is to be clear whether the metric is aimed at measuring the process elements adopted to achieve the outcome or at measuring the outcome itself. There is an argument to be made that objectives only serve the limited purpose of guiding the direction in which progress is to be galvanised.

It is the ‘process’ that needs to do the heavy-lifting and, hence, progress on the process dimensions should be measured, if the process effectiveness and efficiency are to be enhanced. Indeed, this approach works in most cases.

However, it is also to be recognised that the outcome must not become subservient to the process. If companies that are among the largest contributors to greenhouse gas emissions, or companies that produce addictive products are seen to muster healthy Environmental, Social, and Governance (ESG) Ratings — by doing enough that they score well on the ESG rating firms’ scoring criteria — it reflects the triumph of process over outcome.

Conversely, a measuring system focussed mostly on outcomes, according lesser reverence to process/ path, could be useful in certain settings, but would have its own follies. This would be a creed that is a proponent of Milton Friedman, with its focus squarely on a company’s profitability, not necessarily on social good. In effect, knowing well what is intended to be measured and knowing well the limitations of the metrics chosen to do so, could be said to be the cornerstones of tracking performance.

As an example, if one is tracking the state of the economy and using proxies such as automobile sales, airline passenger traffic, hotel occupancies, quick service restaurant sales, retail mall revenues et al, to judge the strength of the recovery, one would need to be conscious that these metrics would only convey the consumption patterns of a small strata of the economy. Even if growth impulses for these metrics were to strengthen, these would not be informative of a broad-based economic recovery.

Standardised metrics

Finally, from a systemic standpoint, there is also a case for having standardised metrics of measurement, which could be uniformly applied to the measurands for achieving consistency and comparability. The accounting standards are a case in point. An illustration of the manifestation of different accounting standards prevailing in different jurisdictions was when the German car manufacturer Daimler reported Deutsche Mark 615 million in net profits under the German accounting rules, but a loss of Deutsche Mark 1.84 billion under the US rules. The implications of such wide reporting variations for the companies’ managements and the investors could be substantial. Closer home, a couple of years ago, capital markets regulator SEBI had introduced standardised probability of default benchmarks across the various rating categories as measures of the performance of the credit rating agencies (CRAs).

With the standards being quite exacting for the top rating categories (NIL default rates permissible in the AAA and the AA categories over select time horizons), if the CRAs were to ‘target’ the performance benchmarks, there could possibly be unwarranted conservatism seeping in while taking rating decisions. The nuance is that the financial system would be better served if the CRAs tighten and uphold the rating standards and assign ratings in a manner that secures the deserved rank ordering of credits, while the performance benchmarks are put to work only ex-post not ex-ante.

Philosophically, this would be the equivalent of stating that success need not be pursued, but be allowed to ensue by not caring about it. With this, the ‘observer effect’ in the above example could be put to rest.

 

(The writer is Head, Credit Policy, ICRA)