Why rankings should not be taken at face value

Atanu Biswas

Professor, Indian Statistical Institute, Kolkata

Global indices and rankings are routinely released throughout the year by several international organisations, mostly private, on some of the most important aspects of society: hunger, inequality, education, democracy, environmental pollution, women’s empowerment, etc. Some of them invariably hit the headlines and create an uproar. They certainly help in drawing our attention to the sorry state of affairs on some important issues that have persisted for decades. But the problem occurs when we deviate too much from the main issues and concentrate on the values of these indices and rankings.

We are delighted if our country’s index is up by 0.1, say, over the previous year (where a higher value indicates a better condition) and we beat Sri Lanka or Namibia, say, on the ranking list. On the contrary, the sky falls when the index value is decreased by 0.2 or when we fall behind Pakistan or Zambia, for example, in the ranking.

But, as a statistician, I wonder if we should pay so much attention to these indices, either by blindly accepting them or by harshly criticising them. I’m not sure, though, how other societies react to these indices. However, the rat race is instilled in our genes from a young age; accepting any type of grading or ranking without questioning it may be our instinct. But it’s worthwhile to scrutinise the underlying methodologies of these indices and rankings to assess how much importance they really deserve.

First, consider the Global Hunger Index (GHI) that created a huge uproar and debate in the country recently. India has slipped six places since 2021 and is ranked 107 out of 121 countries in 2022. Well, the GHI attempts to measure and track hunger globally and is prepared by two European NGOs. Zero is the best possible score (reflecting no hunger), and 100 is the worst. India’s 2022 GHI score of 29.1 is considered ‘serious’. The GHI score is calculated by combining four component indicators, namely ‘undernourishment’, ‘child stunting’, ‘child wasting’, and ‘child mortality’, with equal weights. Certainly, there may be many other useful indicators of hunger. But the reasons why these four and only these four have been chosen and why they are of equal importance are never clear.

Consider the World Economic Forum’s Global Gender Gap Index, which is obtained by calculating the gender gap between women and men in four key areas — health, education, economy and politics — to gauge the state of gender equality in a country. Well, why were only these particular four pillars chosen? And should these pillars deserve equal weight?

The rankings of educational institutions create a lot of noise in society. The Times Higher Education World University Rankings, for example, consider ‘teaching’, ‘research’, ‘citations’, ‘international mix’ and ‘industry income’ with weights of 30 per cent, 30 per cent, 32.5 per cent, 5 per cent, and 2.5 per cent. Overall, 44.5 per cent of the weight is given to the “reputational survey”, 15 per cent to teaching, and 19.5 per cent to research. The QS World University Rankings, on the other hand, have weights of 40 per cent, 20 per cent, 20 per cent, 10 per cent, 5 per cent, and 5 per cent for “academic peer review’ obtained from an internal global academic survey, ‘faculty/student ratio’, ‘citations per faculty’ as a measurement of research impact, “employer reputation’ based on a survey of graduate employers, ‘international student ratio’ and ‘international staff ratio’. Survey-based opinions receive 50 per cent of the total weight.

Certainly, the components and their weights in most of these indices are chosen heuristically, in a subjective way. They can be fixed in an objective way, instead. There may be tens of components capable of contributing significantly to an index. By mathematically optimising one or more carefully defined objective function(s), one can try to trace a few important components (or combinations of components), as well as their weights, if sufficient data is available. However, the same objective function(s) may not be appropriate for every society. The concept of food and nutrition, for example, is certainly not the same in North America and sub-Saharan Africa.

Thus, can any such index be described as ‘insensible’, ‘biased’, ‘inappropriate’ or ‘misinformed’? Concepts regarding any attribute vary from society to society. The idea of democracy in China, Syria and the UK would differ, for example. The yardsticks of excellence for universities such as Oxford and Visva Bharati ought to be different due to their social perspectives and the people they cater to.

Also, data from different corners of the globe is not consistent and is not always equally reliable. Often, the unavailable data is ‘estimated’, based on past data and several auxiliary information sources by making different kinds of assumptions — maybe a bit unrealistic sometimes. And nobody knows how good such estimates are. Quite often, a part of the data is estimated by a survey of a few thousand people.

However, it’s not quite clear how these surveys are designed, how objectively the survey subjects are chosen, how robust their opinions are, or how reliable the conversion of these opinions to scores is. Different people have different ideas about topics like democracy, education and inequality, and their perspectives are frequently skewed. How would ‘errors’ in calculating indices and rankings be quantified? Without a concrete answer, an index value makes no sense. And how much importance do these indices based on such “derived” data deserve?

In fact, I’m quite concerned that these indices, which are sometimes poorly constructed, frequently contested and subject to a number of reasonable objections, are diverting our attention away from the true picture of some key issues in society. Quite often, our discussions turn into political debates. There’s little doubt that we need to improve our social conditions, but not in comparison to Yemen or Bangladesh. These indices might help understand the situation a bit. Not always, though. And that’s all.

They possibly don’t deserve to be taken at face value unless more serious efforts make them sensible, comparable, objective and coherent. That should be true whether our position on an index is five steps up or four steps down from last year’s.

Join Whatsapp Channel of The Tribune for latest updates.