Share

A/B Testing

Our current model calculates similarity scores between campaigns and user profile vectors. 

The similarity score is the cosine similarity between the content+title of the page and the aggregated content read by the user, as stored in their user profile. 

In the current setup, based on an experiment on a small number of users, we found p(threshold) = 0.7 to be an appealing value for filtering the relevant notifications.

Future work involves further tuning of this parameter based on experiments on a more significant number of users. 

When we run A/B tests, we track many different parameters of the models and business metrics. 

But we generally test the model based on Click-through Rate(CTR) wrt clicks vs. sent and clicks vs. received. 

The test usually runs on millions of users, exploring variations of base ideas. 

We typically have scores of A/B tests running in parallel daily, but the key advantage is that they allow our decisions to be data-driven. 

According to Client data, comparing other push services against Oriel.io, the Client's CTR improved by approximately 4.19X, and the absolute average growth in CTR is around 6.32%(based on data between 01-06 -2021 and 01-07-21).

We invite you to test multiple variations of push notifications for the same subscriber segment and monitor the improvements as you go along.

Happy Testing!

Experience the benefits of direct

interaction with your website visitors

Try it out for free!

Signup for your free trial in under 2 minutes, sync your hostname and start collection subscribers for free!

Try 14 days for free, cancel at any time