Rankings are akin to Christmas and Thanksgiving, they come back every year. Schools, health practitioners, hospitals… Virtually every profession, brand, city can be found ranked online. One can even find rankings of the poshest villages in the UK or Kellis’ best songs. And even if they took a hit recently, with Harvard and Yale pulling out of the US News rankings of law schools, universities and colleges rankings remain as popular as ever and carry a big weight into students’ decisions of where to study. Companies and brands like rankings because they help grab the attention of customers. And marketers use them to demonstrate the expertise of their brands and products, establish authority.
University rankings concept. Inscription and book, vintage style.
Why Consumers Love Rankings
Consumers love rankings for three main reasons. First, they offer a way to make the decision-making process seem objective and rational. It gives the illusion that products and services can be compared in objective ways. Second, by simplifying reality and giving consumers the most relevant factors to consider, rankings help consumers overcome their limited capacity to process information by saving some previous information processing resources – something consumer psychologists refer to as ‘cognitive economy’. Third, rankings can be used for social identity and self-esteem purposes. For consumers, knowing that their brand is the favourite brand on the market, that their employer is a top employer in its category can be important, and a driver of consumer loyalty or employee retention.
From a decision-making point of view, rankings constitute a form of heuristic – a mental shortcut, which is used by consumer to make faster decisions. They address a trade-off between making the best possible decision – which is practically impossible as it would require too much time and efforts – and a decision that is pragmatic and comes at a fraction of the associated decision-making costs. In sum, heuristics help consumers improve their decision-making process and make it more efficient. Other forms of heuristics include awards (e.g. ‘voted product of the year’) or expert endorsement (e.g. a car recommended by a formula one driver).
The reality portrayed by rankings – that of an objective appraisal relying on only a few variables and scores – is, however, an illusion. Rankings are based on the mostly flawed premise that it is possible to objectively assess something, be it a physician or a museum. Psychologists know how difficult this is. Perhaps one of the most debated and controversial takes on measurement and ranking is that of intelligence. The development of a universal measure of General Intelligence (psychologists call it ‘g’) has occupied psychologists for decades, and led to the rise of IQ tests, but very little certainty that the measure is accurate.
The top 3 most common ranking mistakes
Number 1: Incorrect or biased information
Data in rankings is often based on declarative answers from companies who voluntarily entered the ranking. Knowing you will be ranked on what you declare is not the most objective way to start a ranking process. Second, since some companies may self-nominate to join a ranking, or instead opt out, rankings are based not on a comprehensive list of all eligible competitors, but rather on a self-selected sub-sample.
Number 2: A partial representation of reality
Rankings only capture a fraction of reality. Criteria that are used in rankings are often chosen on the basis that they capture emerging social trends. If your favourite ranking now includes a criterion capturing the ‘environmental impact’, does it really reflect a meaningful change or is it simply a way to capitalise on the heightened climate change impact of most activities? Another aspect that makes it difficult for rankings to accurately capture reality are measurement issues. Whenever you are trying to capture a phenomenon, it is important to ensure the measure used does what it says it does. For instance, food prices are an important aspect of what makes cities liveable but comparing the price of the same basket of items worldwide misses two important things: that food is very cultural and geography-dependent – what we eat depends on where we live – and that comparing prices falls short of comparing quality - your tomatoes may be cheaper in one city vs another, but are they of the same quality?
Number 3: Positions are never accurate in rankings
Rankings often compound ‘scores’, and simply assume that differences between scores, no matter how small or big, accurately reflect different positions. An issue with this approach is that very few rankings make raw data available, so that it is not possible to know if these differences are statistically significant. In other words, is the product ranked number 1 significantly superior to the product ranked number 2? A change in the order within a ranking, especially at the top, is the bread and butter of commercial rankings, which thus have an incentive to create some variations year on year in a ranking, albeit artificially. Rankings, especially in their most simplified form, give an impression of stability and act more like photos, rather than movies - thus being more of a static representation of reality. Yet, whatever is being ranked, positions can move fast.
How To Create Better Rankings
First, rankings should be best thought in terms of leagues, not tables. Rankings should focus on creating pools of similar-level entities, rather than illusionary tables. Within each league, one could indicate movement – e.g. trending up or down - rather than absolute positions. With such an approach, there would not be a number one, unless they are in a league of their own. Second, rankings should focus on criteria objectivity, measurement, and data transparency, making data available for scrutiny and further analysis. Self-reported criteria should be abandoned, as they create a conflict of interest for companies taking part in these.
