The battle of Predictive Analytics vs Intuition
Among Human Resources, Organisational Development and Leadership circles there is somewhat of a religious debate between using an intuitive approach to decision making against using a data-driven approach or “predictive analytics”.
Predictive analytics is a broad term for a methodology that includes gathering historical data and with techniques that combine data mining, statistical modeling and machine learning to identify patterns to make predictions about the outcome of a particular decision. Discovering risks and opportunities and modelling how several courses of action might play out, before implementing it in a real situation.
Consider these two facts; 1. Most organisations budget for 40 to 70 percent of their operating expenses to employee related expenses. 2. Important decisions affecting an employee base are commonly made subjectively, without proper validation, with a lack of rigor.
Within HR there are broad sweeping and significant opportunities to make a material difference to the largest slice of an organisation’s cost base and the most often the biggest influence on revenue – people and culture. So consider that a balanced approach that uses data, analysis and systematic reasoning to validate anecdotal, employee survey data or intuition is significantly more effective at reducing risk, improving performance, reducing turnover, workforce planning or sometimes selecting the right candidate.
There is a growing acceptance of the value predictive modelling within HR can play in validating intuitive decisions. A combined approach that balances data-driven decision making to test intuition. However, there are some professionals that believe they are largely infallible decision makers, that they are correct the majority of the time. It’s often these same people that sometimes have a reluctance in accepting a contrary view identified by predictive analytics.
It’s absolutely true that no CPU can replace the human brain. However when making a decision, it’s simply too hard for most mere mortals to identify, weight, assess and model-out how the different variables that effect a decision will lead to an outcome and which of these outcomes will be more favorable.
In Neuroscientist and successful author, Dr David Rock’s book “Your Brain at Work” he describes the limited processing power your brain (more specifically the frontal lobe) has when accessing memory and assessing variables to make a decision. Additionally in the ground-breaking research into cognitive load theory by Miller, Simon and Chase we know that our working memory and processing power is more limited than most people would readily admit.
It’s simple, we need help. That help is more readily available with the advancement in technology, the conscious effort to collect high quality, relevant data and a growing appetite for balancing and validating intuition with data driven decision making.
Research conducted by Ian Ayres outlined in his NY Times best seller “Super Crunchers” details over a 130 studies into the accuracy of judgment of experienced managers. In every single case the power of predictive analytics matched or beat the manager’s intuition. In fact, in 46% of these studies it found the manager’s intuition significantly underperformed an algorithmic prediction.
Research shows that high performing businesses are much more likely to view predictive analytics as a core capability than a low performing competitor. That those environments that embrace predictive modeling within their decision making are yielding big rewards. In the study “Strength in Numbers” by Brynjolfsson, Hitt and Kim in 2010 of 179 large companies, those adopting data-driven decision making were seeing productivity gains of 5 to 6 per cent higher than those of their competitors. Although the percentage gains are commonly higher for small to medium organisations, 5 to 6 per cent is a massive chunk of change for a multi billion dollar fortune 500 company.
There is a long way to go for predictive analytics to be used in the same manner and sophistication that its used in other functions like marketing, sales and finance. How we are going to make decisions is going to change over this next decade and it’s effectively equivalent to a change in mindset moving from “I think” to “I know”.
So whilst you can’t and would never replace intuition, when next making an important decision, consider testing it with predictive analytics.
Why you"re making the wrong decision: predictive analytics vs intuition
