It is understandable that investors, especially those new to applications of Alternative Data in investment management, assume that their best ROI models will be the ones with the highest R^2.

 

When modeling a KPI for a public company, one of the goals of the model is to predict the likelihood of a surprise announcement. Getting the direction of the surprise correct is itself important as surprises in top-line Revenues have been shown to precede abnormal returns. But it is equally as important to remember that not ALL surprises lead to a significant market reaction. Some KPIs tend to move the markets more than others. It is also worth noting that management’s forward guidance is often more important than the past quarter results. We talk more about this concept here.

 

At Maiden Century, we quantify the market’s tendency to react to a surprise for every model and every KPI on the IDEA platform as “Relevance“. The metric is normalized to a scale between 0 and 100 with 100 meaning that the share price has a strong relationship to the KPI surprise in the five subsequent days following the announcement. A value of zero means that there is no observed relationship or the magnitude of the move in share price is not significant following a surprise. In short, the model may be a great fit but it’s not monetizable.

 

Let’s take Wayfair (W) as an example. One of your datasets may measure the KPI “US Direct Retail Revenue”. While the model for this KPI has high Accuracy, meaning that you can trust the reliability of the forecast, the relevance is low. This means that even if the model is correct and you predict the direction of a KPI surprise, the market is less likely to react favorably to you in the days following the announcement.

 

Wayfair (Sample datasets, IDEA)

 

This situation is reversed in the case of Lululemon (LULU). Looking at Retail Revenue we see a fairly mediocre accuracy score of 60, but a strong relevance score of 91. This means that you might be less confident in the model, but if you get it right, you are likely to be rewarded with a favorable reaction in share price.

 

Lululemon (Sample datasets, IDEA)

 

 

Now consider Amazon (AMZN), and their KPI for North American Revenue.

 

Amazon (Sample datasets, IDEA)

 

In this case, both the Accuracy and Relevance are exceptionally high, meaning this is likely to be a high ROI model for investors. It’s a model you can generally trust and it tracks a KPI you can monetize.

 

What do you think about the relevance measure? We’d love to hear from you.