Predictive analytics often sounds a bit like quantum mechanics: fiendishly complex to look at and wildly counter-intuitive. So when someone sells you a tool, how can you verify that the black box is fit for its purpose or even lives up to the vendor’s claims?
Of course, there never are ironclad guarantees around prediction; the future can have more shapes and colors than we ever imagined. But you’ll be more likely to get the best out of your predictive analytics if you ask the following basic questions in evaluating new predictive models:
Q1: Are the data good? Make sure the data used, as well as the processes that generate and organize it, are of the highest possible quality, and that you fully understand them. As a hedge fund manager who applied data-driven trading strategies once said, “You can spend endless time and resources only to find eventually a bug in your data.”
You can get problems even with the best data. In a recent study of the daily close prices of S&P500 stocks, we “discovered” what looked like striking patterns in stock movements. When we looked more closely, we found that the patterns were explained by the fact that what was meant by the “close price” of a stock is very loosely defined. The term had multiple definitions, all of which were applied, including, for instance, the “last price before 4 PM” and the “last price reported in the closing auction.” Once we defined the term more narrowly, the patterns disappeared. We imagine that many other people will fall into a similar trap and see patterns where none exist because they haven’t thought hard enough about what the data they’re looking at actually are.
Q2: Does the model tell a story? Sound models usually tell a clear story. If the predictive analytics you’re using don’t give you one, beware—the models may need to be refined. This is not to say that the story has to be what you expect or that it has to be a simple one-line statement. It is rather that the story has to be understandable to the people basing their decisions on it. It is not about avoiding complex statistical models, which of course are often necessary, but about having thought through, refined, and simplified the models enough to be able to understand them.
Q3: Is the math as simple as it can be? It’s natural to assume that the model that does the most with the most variables will be the best model. But theory, such as Statistical Learning Theory in Machine Learning, teaches us that predictability typically first improves and then deteriorates as model complexity increases, so adding complexity should not be a goal in itself. This is precisely why describing a model using a simple story that makes economic sense is often a good sign of a refined model with the right level of complexity. Of course, there are equal risks to oversimplifying the math, so don’t go too far in that direction either. Einstein is often quoted as saying, “Everything should be as simple as it can be, but not simpler,” a good principle to apply to predictive analytics.
Q4: Is the predictive accuracy stable across different, fresh data? All too often analysts study historical data to develop a model that explains that data and then apply the model to the exact same data to make a prediction about the future. For example, developing a credit scoring model using data of past defaults and then testing the model on that same data is an exercise in circularity: you’re predicting what you’ve explained already. So make sure that your analysts apply the model to fresh data in new contexts. More importantly, check that the predictive accuracy of the models is reasonably close to how well the model succeeded in explaining the data it was developed to explain; prediction accuracy should be similar across multiple environments and data samples.
Q5: Is the model still relevant? Confirmation and other behavioral biases, along with the “sunk cost” fallacy, often encourage people to see predictability where there is none, especially if there was some before. But if the data don’t support your predictions, you should be prepared to jettison your model—possibly multiple times. Even if your model has a track record, you should still test whether it remains relevant to the economic and business context. Using predictive models of demand developed during growth years or for a price sensitive market segment may fail miserably when market conditions change. This makes predictive analytics by nature risky: they are valid as long as the reality they were developed in, the data describing a specific market during a specific time, is also valid.
Developing and applying predictive analytics requires a delicate balance between art and science. A deep contextual understanding with very honest interpretation and story development should be balanced with hard facts and statistical analysis. And always keep in mind that the most predictive analytics models can become fully non-predictive in a fortnight—just imagine how some financial models looked the day after the Lehman collapse.
via How to Tell If You Should Trust Your Statistical Models – Theos Evgeniou – Harvard Business Review.
How to Tell If You Should Trust Your Statistical Models
