Follow

Understanding RevIQ

Understanding RevIQ

RevIQ is a collection of patented machine learning algorithms that help you maximize app-generated revenue. When you send the most effective message at the optimal time, users are less inclined to uninstall your app. You realize revenue you might have otherwise lost.

RevIQ uses four types of optimization:

SendOptimally Send Time Optimization

SendOptimally Send Time Optimization uses historical data to message users on the device they are most likely to be using at the time they are most likely to be using it. SendOptimally weights times when previous goals have been achieved more heavily than just user usage time. This targets times users might be more likely to achieve goals.

RevIQ Message Optimization

RevIQ Messaging Optimization uses machine learning to distribute the message producing the highest number of engagements. If no goals are selected, message optimization learns the best message for engagement. If a campaign has goals, message optimization selects the best message for primary goal conversion.

Engagement-Based Optimization

For lifecycle and adaptive campaigns without primary goals, RevIQ determines message distribution by monitoring user engagement. As users respond to daily push and email message variants, RevIQ ranks the variants according to an evaluation of their popularity. The most popular message variant is immediately distributed more widely as determined by the RevIQ algorithm. As a lifecycle or adaptive campaign unfolds, RevIQ ensures that the most popular variant is distributed the widest.

Goal-Based Optimization

For campaigns incorporating primary goals such as conversion, lifecycle, or adaptive campaigns, RevIQ determines message distribution by monitoring goal achievement. As users fail to achieve a campaign primary goal, RevIQ distributes message variants uniformly and monitors goal achievement to determine which variant is the most effective. RevIQ ranks the variants according to their popularity. The most popular message variant is immediately distributed more widely as determined by the RevIQ algorithm. As a campaign unfolds, RevIQ ensures a much larger distribution of the variant proven to be the most effective.

How Does Message Optimization Work?

For message optimization, RevIQ uses a randomization process to assign messages to users. Initially, each user has an equal probability of being assigned to any message, but as time goes by, RevIQ adjusts the probability distribution across messages to favor the more successful ones. Nevertheless, the assignment is always probabilistic, even at the beginning. Although you might expect to see equal message distribution, user assignment across messages might not be precisely equal. Kahuna verifies the uniformity of the sample on the first day using a statistical test, but the RevIQ algorithm does not assume uniformity on subsequent days.

For example, consider the following distribution on the first day:

A B C D Control
20 24 32 18 80

The total number of users is 94 + 80 = 174. The control group is somewhat smaller than the treatment group here. At this point in the campaign, the distribution of users in the treatment group is more important. Later in the campaign, the control group typically grows to approximate the size of the entire treatment group.

If the 94 treatment users were uniformly distributed across A-D, the average would be 23.5 users. You could assess the deviance from this standard using a chi-square test, which considers the squared sum of the standardized residuals:

The more the sample diverges from uniformity, the greater the chi-square statistic. If it crosses a reasonable threshold, you can rule out the assumption of it being drawn from a uniform distribution. The p value (probability of observing a sample at least as extreme, or uncharacteristic of a uniform distribution) is shown here:

Users are distributed over four options, leading to three degrees of freedom (the green series above), since their sum is constrained to the size of the treatment group. Accordingly, you find the p value to be around 18%. This is considerably higher than the low single digit threshold you might set to rule out the null hypothesis (of uniformity).

You could now conclude the evidence is not strong enough to rule out the assumption of uniform distribution.

Randomization process on Day 1

On Day 1 of the campaign, when RevIQ does not have knowledge of conversions or user engagement, every user selected for the campaign is randomly assigned a message copy. Before acquiring knowledge on conversion rates, each message has equal probability of being assigned to a user selected for campaign messaging.

Randomization and distribution on Day 2, onwards

As RevIQ aggregates performance data, it begins skewing the distribution of messages towards the variant with more primary goal achievements. If a campaign does not track a primary goal, user engagement with the application is used to determine message distribution. This ensures the message selection process for each user selected for the campaign is still randomized but the probability for receiving the higher performing message is increased by RevIQ. It is important to note that RevIQ continues to test all the variants as long as it is running. It always attempts to optimize based on aggregate, moving data. Distribution is proportional to the conversion or engagement rates in the campaign.

The raw logs files Kahuna provides include details on pushes delivered, users who became more engaged, completed a goal, or uninstalled as a result of the campaign. For example, a log file might include 1883 rows of data. Out of this, there are 1038 pushes delivered, 414 more engaged users, 396 goals completed and 35 uninstalls. This adds up to the total 1883 rows of data generated for that day.

When you look at the raw logs, avoid considering unique tokens as a metric. Some users might have received messages on different devices (with different push tokens) over an extended time period.

Goals in Kahuna are based on user events, completion of these events within defined attribution windows, and by the most recent campaign they received.

After a user is sent a campaign message, Kahuna tracks goal completion for 48 hours (Kahuna can customize this for each user event passed to Kahuna), and attributes it to the campaign from which the user was messaged. If a user receives another campaign message tracking the same goal within the 48-hour goal tracking window, goal completions are attributed to the latest campaign the user received. For example, if a user is sent a campaign on March 25th at 9:00 am and another campaign on March 25th at 1:00 pm, the goals are attributed to the first campaign from 9:00 am until 1:00 pm. From 1:00 pm until 48 hours later, barring any additional campaigns being sent during this time, the goals will be attributed to the second campaign.

On the campaign details page of a given campaign, unique goals completed by a user attributable to that particular campaign are displayed as conversions. In the logs generated, Kahuna displays every goal conversion irrespective of campaign attribution or user uniqueness. You will find users who complete campaign goals multiple times in the attribution window and complete them on different devices. Due to these differences, goal completions from logs and the UI do not match.

SendOptimally Conversion Time Optimization

SendOptimally Conversion Time Optimization discovers the ideal time for a conversion push. With Conversion Time Optimization, 98 percent of the users who organically achieve a campaign goal do not receive a message. This avoids messaging those who would have achieved the goal with no messaging; for example, if a user continues to shop after moving to cart, SendOptimally messages the remaining two percent of users according to the specified interval for the campaign:

Within Minutes

The optimal send time between 30 minutes and 2 hours.

Within Hours

The optimal send time between 1 and 7 hours.

Within Days

The optimal send time between 1 and 7 days.

Intervals with the highest conversion rate receive more messages until a definitive winning time is selected. SendOptimally messages all the users who have not achieved the goal after the 98 percent organic conversion time has passed.

RevIQ Cross-Channel Optimization

If you typically hard code messages for delivery across either a single channel or all available channels, your message campaigns might be unproductive. RevIQ Cross-Channel Optimization discovers the optimal communication channel for each user of your mobile or web application. Cross-Channel Optimization uses machine learning algorithms to test user response over time. After an algorithm-determined number of delivered messages, Cross-Channel Optimization settles on the single communication channel with the best user response rate. Cross-Channel Optimization determines response rate by counting the following:

  • Push notification clicks
  • Email opens
  • In-app message primary goal achievement

The default response rate for each new user is 10% for each communication channel. Cross-Channel Optimization then increments response rates for each channel as they occur and applies its algorithms to determine the optimal channel for each user over time. For legacy users, Cross-Channel Optimization refers to their response rate history determines an optimal channel. When response rate data volume is large for a given user, Cross-Channel Optimization can determine the optimal channel soon after you begin activating it in messages and campaigns. Cross-Channel Optimization continues to test user response, and if it determines a user has begun to respond more reliably to a different channel, messages then go to that channel.

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Submit a request

Comments