In the CrossEngage Customer Prediction Platform (CPP), marketers and data scientists can quickly and easily create predictive models that predict future customer behavior to make better marketing and CRM decisions. In the wake of the Corona crisis, the question is coming up again and again, how this crisis affects the performance of predictive models that the CPP users are creating in the platform.
This question cannot be answered universally, as the Corona crisis has very different effects on different industries and business models. To be able to estimate possible effects nevertheless, various scenarios and their effects on the prediction quality of the models are described below using examples. A distinction is made between effects during the crisis and effects after the crisis. Furthermore, it is described how effects on the forecast quality of the models can be determined quickly and easily with the help of so-called backtests.
All effects that will be described here generally apply to all possibilities of customer evaluation methods. Both for the use of machine learning models and the use of heuristics such as RFM models.
During the Crisis
Scenario 1: Homogeneous Effect on the Business
Business performance, such as sales, profits, conversion rates, sales of goods, or market demand, is largely homogeneously affected by the Corona crisis – positively or negatively.
This means that the purchasing behavior of all customers has changed to the same extent across all products and sales channels. For example, a homogeneous effect on business performance has been experienced by some companies operating exclusively in e-commerce, whose sales across all product groups have increased as a result of the closure of stationary stores in the lockdown, as demand for their products could now be met exclusively online, where they were already represented.
In this case, the absolute level of conversion and revenue predictions will deviate more strongly from the actual values (higher or lower) than is usually the case. However, the models are still able to differentiate between good and bad customers, so that the best customers can still be selected when selecting campaigns.
Scenario 2: Heterogeneous Effect on the Business
The business is affected differently by the Corona crisis depending on specific product groups, customer groups, or distribution channels.
In this case, the absolute level of the predictions will also deviate more than usual from the actual values. Besides, however, the ability of the models to differentiate between good and bad customers may also be impaired. How strong this effect is, depends on how heterogeneous the change in customer consumption behavior takes place.
For example, online mail-order pharmacies and drugstores experienced a significant increase in new customers, especially during the first Corona lockdown in spring 2020. However, these new customers only purchased certain product groups (for example, only disinfectants, masks, etc.).
So in this case, the new customer business growth is linked with specific product categories. These new customers may show no interest in the usual top sellers of these companies.
For predictive modeling, it may therefore be important to exclude these new customers from the model training, as they might influence the pattern recognition too much. However, such conclusions should be tested in any case before implementation by comparing the “normal” models with the adjusted models.
Furthermore, it may make sense to create separate predictive models for such specific customer groups (depending on the availability of sufficient amounts of data) to be able to target this customer group.
After the Crisis
Scenario 3: Return to Previous Consumption Behavior
Customers return to their previous consumer behavior after the turbulent phase of the Corona crisis and behave exactly as they did before the crisis.
The models created before the crisis can be used again as usual. The Corona crisis period should be excluded in the process of creating new models so that no “false” patterns distort the training of the new models.
Scenario 4: Permanent Change in Consumer Behavior
The Corona crisis has led to a permanent change in the consumer behavior of customers.
One example here is the mail-order business with office supplies and office equipment. If the trend of remote working will remain in the long term, the purchasing patterns of the customers would probably change permanently. It could be that more private individuals will buy than before (usually it was B2B) and other products will be top sellers than before the crisis.
In this case, the buying behavior is changing in a long-term manner. Predictive models would recognize different patterns in the data than before the crisis. In such a case, it must be taken into account not to analyze these periods together in one single model as they differ greatly in the buying patterns of the customers.
Depending on the extent of the change in the buying behavior, existing models must be checked individually to determine whether they can still be used.
Testing the Prediction Quality
To test the impact on the prediction quality (accuracy of the predictions and ability to differentiate between good and bad customers), a so-called backtest can be used. A backtest is a predictive model that is applied to historical data. The customer behavior is being predicted for a specific period, which already happened in the past.
Then, these “predictions” can be compared to already observed results. If the results of the predictions and reality are similar, the hypothesis is that the model, which was used to make the predictions, is well defined and ready for “live” use.
With a backtest, one can easily test how the model will probably perform in a future deployment. This provides a risk-free way to determine the effect of the Corona crisis on prediction performance.
If the backtest shows that the model’s predictive performance is compromised, it is possible to split the model into multiple submodels to increase the models’ predictive performance. Here, too, backtests can be used to determine whether the predictions and reality matched in history.
Furthermore, across our customers, we are currently observing the following effects:
- The predictions continue to work reliably for many customers, especially in periods when relaxations of economic life are decided after tougher restrictions (for example, when stationary stores were allowed to reopen after the first lockdown in the spring of 2020).
- Customers who are particularly hard hit by the Corona crisis have managed to leverage business performance potentials through new use cases and specially created models that would have been very difficult to identify without the use of predictive models. In the following, we would like to give an example of such a new use case.
Use Case: Converting Successfully Stationary Customers to Online Buyers
A fashion retailer with stationary stores and an online shop was strongly affected by the Corona-related stationary store closures in spring 2020. Significant parts of the expected sales were lost. Although the online mail-order business grew decently, it was not able to compensate for the sales losses.
To encourage more customers who would normally only buy in stationary stores to buy online during this period, the company created prediction models in the CPP that compute the probability of an online purchase specifically for purely stationary customers – a so-called “online affinity”. Subsequently, customers rated as having a high probability of making an online transaction received a print mailing with a reference to the possibilities and advantages of the online shop.
The results of the campaign were very pleasing. Compared to the zero group, this campaign achieved a lift of 65% on the conversion rate and a lift of 135% on the contact value. Overall, after deducting all costs, this mailing campaign resulted in a profit in the mid-five-digit range.
A renewed evaluation of the campaign after the reopening of the stores showed exciting developments: The stationary-store-only customers showed increased sales both in the stores and in the online shop even after the stores had opened. This is an indication that these customers have been developed into good multichannel customers and that stronger brand loyalty will hopefully also enable higher sales to be realized in the long term by staying more in the consideration set of the customers.
Our conclusion from the observations so far is that in many cases, the predictive models continue to work well. In areas with strong changes in customer behavior, it is often possible to create new models that allow a proper differentiation of “good” and “bad” customers for the respective use case, despite the given situation. Predictive models also can enable new use cases that can help a business to suffer from the uncertainty.
As before the Corona crisis, identifying the right use case is the most important success factor when using machine learning for customer scoring.