As we note in our previous post, there are myriad benefits to pushing aside gut instinct and relying on data analysis to forge ahead of competitors.
In that post, we cite new research that finds that 70% of the top performing companies have strong executive champions for analytics.
However, overcoming the human tendency to rely on judgment rather than data can be challenging.
In fact, as the amount of data goes up, the importance of human judgment should go down.
That’s the assertion of Andrew McAfee, principal research scientist at the Center for Digital Business in the MIT Sloan School of Management.
McAfee acknowledges that the statement goes against the traditional business and management thinking of many because education management is largely based on teaching and refining judgment.
“And whether or not we’re in b-school, we’re told to trust our guts and instincts, and that (especially after we gain experience) we can make accurate assessments in a blink,” McAfee notes. “This is the most harmful misconception in the business world today (maybe in the world full stop).”
While he notes that human intuition is real, it is, however, very faulty.
For example, he points out that highly trained pathologists don’t do as well as image analysis software at diagnosing breast cancer and that procurement professionals perform worse than algorithms at predicting which suppliers will perform well.
When people apply their judgments to the output of data-driven analysis, they generally are less successful than the algorithms. But when human experts provide input to a data analysis model, the quality typically goes up, McAfee adds.
“So pathologists’ estimates of how advanced a cancer is could be included as an input to the image-analysis software, the forecasts of legal scholars about how the Supremes (court justices) will vote on an upcoming case will improve the model’s predictive ability, and so on,” he says.
Despite this argument, McAfee concedes that for many firms, adopting this paradigm shift will be challenging. Most people making decisions at companies today believe they are very successful and they fear that turning over decision making to algorithms will diminish their power and value.
Before they embrace this notion, companies will need to see many examples of how much worse human judgment is compared to data analysis so that employees begin to care enough about the faulty human decisions to turn to those made by computers.
He points to parole boards as support for his assertions. He notes that 18 states in the past 25 years have replaced parole boards with sentencing guidelines, and those that retain boards increasingly are relying on algorithms to predict the risk of recidivism.
“The consequences of bad parole decisions are hugely consequential to voters, so parole boards where human judgment rules are thankfully on their way out,” McAfee concludes. “In the business world it will be competition, especially from truly data-driven rivals, that brings the consequences to inferior decision makers. I don’t know how quickly it’ll happen, but I’m very confident that data-dominated firms are going to take market share, customers, and profits away from those who are still relying too heavily on their human experts.”
via Data Analytics and the Benefits of Ignoring Gut Instincts (Part 2) | Trends and Outliers.
Data Analytics and the Benefits of Ignoring Gut Instincts (Part 2)
