Can retailers trust their machine learning models?
As we inch closer to Black Friday and the start of the holiday shopping extravaganza, retailers are putting the finishing touches to the demand forecasts they use to predict the mix of merchandise they’ll carry this winter. There are a lot of variables to juggle, including COVID, the economy, and the weather. This seems like a perfect use case for the increasingly sophisticated machine learning models that are trending in the industry. But can they trust their predictions?
Over the past decade, retailers and other companies in the consumer goods supply chain have begun upgrading their demand forecasting systems in hopes of gaining traction in this super competitive industry. .
Forward-looking retailers, in particular, are replacing the largely deterministic approaches that were favored in the past – which used simple linear regression models based on historical data with relatively static assumptions about the state of the world – with probabilistic approaches that bring more data into the equation and rely on more sophisticated machine learning algorithms, like neural networks and XGBoost, to generate more detailed forecast ranges.
According to supply chain consultant Stefan de Kok, new probabilistic approaches have the potential to provide more accurate forecasts for demand planning than older deterministic approaches.
“The root of the value of the probabilistic approach is that it can correctly distinguish between error and natural variability, and between signal and noise, which is impossible from the deterministic perspective,” de Kok writes. in a 2021 article on the subject on LinkedIn. “Probabilistic approaches provide rich information to identify risks and opportunities at all levels of detail, enabling informed business decisions. They also provide a perfect delineation of the things you can control and improve versus the things you can’t.
The potential for greater accuracy and more nuanced forecasting has convinced retailers to at least add probabilistic forecasts to their kit. It’s worth nothing, however, that deterministic models always have the advantage of generally greater precision, according to de Kok, since they are usually expressed as exact numbers.
Although the probabilistic approach has advantages over the deterministic approach, it also proves to be a little more difficult to continue to perform well in the real world. This is due to a combination of reasons, including the use of more data and the possibility of errors in the data; the black box nature of machine learning models; and the general nature of assumptions about the future.
While the probabilistic approach has benefits, it’s critical that retailers frequently check the predictions made by machine learning models to make sure they don’t go off the rails, says Liran Hason, machine learning expert and founder and CEO of Aporia, a provider of observability tools for machine learning models.
“Using machine learning models offers many potential positive outcomes,” says Liran Hason, machine learning expert and founder and CEO of Aporia. “But it’s very important to watch them very closely, I would say on a weekly basis, especially for large retailers, when they’re dealing with big [number] of stores in different counties.
Aporia works with customers in a number of industries, including retail, automotive and financial services, so its machine learning observability tools aren’t designed specifically for retailers, says Hason. But among the retailers Aporia has worked with, Hason has seen greater adoption of deep learning approaches as well as decision trees and XGBoost.
“They are able to store a lot more data,” he says. “So in a sense, the models we’re getting are much more accurate. But some of that accuracy means they might also be over-equipped for certain situations.
Data drift can occur when underlying assumptions about the world have changed. This can happen in an instant in retail, for example when a hot new product hits the market and disrupts pre-existing shopping behavior. When this happens, model accuracy will suffer.
“While they may work very well during the research phase or for a period of time, they won’t always be accurate,” Hason says. “And then identifying the right time to recycle the model when the data is distorted by the model, or maybe you should just fall back on another mechanism. These kinds of questions become very business-critical.
Companies use Aporia to alert them when their machine learning models behave abnormally. The software works by continuously testing the model at hand with different permutations of inputs and analyzing the response.
“Our product is an observability platform,” Hason says. “So we allow users to visualize and see what decisions are made by these learning machines [models] for their company, their performance and also for different populations. So the model may work fine for a specific state, but [not so well] for another state.
In addition to detecting data drift, it can also help explain how a black box model actually works. This is an important factor for today’s complex neural network-based approaches, which use a multitude of hidden layers to improve prediction accuracy.
“The fact that machine learning models are black boxes also creates the challenge of understanding, well, can I trust this prediction or not,” Hason says. “So part of the offering we’re making is to explain the ability of those decisions.”
Retailers don’t need a magic box to tell them to prepare for a turbulent holiday shopping season in 2022. With inflation flirting with 40-year highs and a looming recession, consumer sentiment n isn’t particularly happy. When you add COVID surges and supply chain disruptions to the mix, you have a potentially volatile situation. Some retailers have already reduced prices due to overstock conditions. Machine learners have their work cut out for them this fall.
Top 10 Ways AI is Driving Price Optimization in Retail
Stay on top of the ML model and data drift
Keep your models on the straight and narrow