The world is in the midst of a statistical revolution. Everything from professional sports to advertising to education to technology performance to climate can be measured and quantified with more precision than ever before. Statisticians and big data analysts are in demand and finally being listened to Yet the world still runs on heuristics and simple stats – numbers that don’t reflect what they purport to show; figures that can be faked for profit and political gain; facts that are simple, intuitive, and completely misleading. You know them in sports with stats like batting average, wins, passing yards; in education with test scores; in technology with prices and analyst rankings. And as we’re finding out in collaboration with the leaders of Bhutan, in the wealth of nations with GDP. RampRate’s purpose in life can be described in many ways – removing corruption and waste in technology; driving social impact; aligning incentives with performance. But one key ingredient through all of this is getting decisions to be made on the right set of numbers rather than the old standbys that can mislead – or be actively used to mislead.
What Can Go Wrong with NumbersBroadly speaking, there are three primary data faults:
- No measurement – going by gut feel and heuristics. Examples abound in everything from sales made based on golfing outings to educators resisting any measurement of performance outside of peer opinion.
- Wrong measurement – the data doesn’t say what you think it does. Old-school sports stats are the most glaring examples, with numbers like batting averages and wins only loosely correlated with actual player performance and the Moneyball era of the last 2 decades fixing century-old misconceptions in a highly visible way.
- Juking the stats – as the Wireput it, when you make robberies into larcenies, majors become colonels and mayors become governors. For every stat that is used to allocate resources or power, there is usually someone with the incentive to fake that stat for personal benefit.
Measurement Failures in Corporate IT SourcingIn our world of IT sourcing, we deal with all three on a regular basis:
- The first error is most common in high growth environments with no internal controls – where the first available or top branded solution is bought as a substitute for market research because there’s no time or resources for market resources. But it’s also omnipresent in risk assessment, which oscillates between ignorance and hyper-avoidance.
- The second error is most common in stabilizing environments where CFOs get control of technology spend and demand the wrong metrics are optimized – like a cost per server or employee rather than for the job that staff member or machine gets done. Focus on cost itself, rather than value is a core one – as is ignoring social impact and reputational risk of associating with providers that pollute the environment, mistreat their employees, or mislead their investors.
- The last one – juking the stats – obviously shows up in unscrupulous sales tactics. But it is also the natural reaction of otherwise conscientious IT staff when faced with unreasonable numbers requests and evaluation standards – when they see management measuring the wrong thing, they fake the results so that they can keep doing what they think is right.