Question 1

You have been tasked with managing TikTok Ads account at Preply. Current spend is ca. $200K/month (dataset attached) and we are looking to further increase the NSGMV from this channel at the best possible ROI. For this exercise please provide a clear high-level summary; as well as a detailed performance analysis on current account performance.

Account overview

Preply’s TikTok ad account currently operates with a monthly spend of approximately $197,820, distributed across Web, iOS, and Android. The account is structured into Prospecting, Retargeting, and Awareness layers, with a primary goal of driving New Subscriber GMV (NSGMV) at the most efficient ROI.

  • Total monthly spend: $197,820

  • Total NSGMV (New Subscriber GMV): $71,164

  • Average NSGMV ROI: 46.37%

  • Average CPA: $217.91

  • Average ARPU: $89.89

Awareness campaigns are active at the top of the funnel and are primarily targeting Spain. Prospecting campaigns are present across all three platforms: iOS, Android, and Web. Retargeting campaigns, however, are only running for Android and Web as iOS has attribution limitations caused by platform restrictions since the rollout of iOS 14. iOS campaigns show the widest variety of ad sets. Android campaigns are targeting a global audience and include both prospecting and retargeting. Web campaigns are mainly focused on English-speaking users, and like Android, they include both prospecting and retargeting flows.

Performance analysis

The top-performing prospecting campaigns (by ROAS) are:

  1. Prospecting_Web_English ROAS: 45.58%, Spend: $20,856.13

  2. Prospecting_Web_English (duplicate entry) ROAS: 44.39%, Spend: $17,076.78

  3. Prospecting_Android_Global ROAS: 43.33%, Spend: $11,601.14

Prospecting_Web_English campaign accounts for over $59K, making it the largest spending campaign overall.

The strongest ROAS in retargeting is observed in campaigns focused on users who have viewed a tutor profile:

  1. Retargeting_Web_ViewedTutorProfile_CreativeChallenge ROAS: 92.34%

  2. Retargeting_Android_ViewedTutorProfile_CreativeChallenge ROAS: 80.89%

  3. Retargeting_Web_VisitedCheckout_CreativeChallenge ROAS: 78.91%

Please create a comprehensive action plan to improve account performance; (eg. what actions, changes, etc. would you operate).

Action plan

1. Prospecting_Android_Global_Broad campaign has lowest AOV:

  1. Prospecting_Android_Global_Broad_UGC, AOV $78,49

  2. Prospecting_Android_Global_Broad_Spark, AOV $78,54

  3. Prospecting_Android_Global_Broad_CreativeChallenge, AOV $79,08

  4. Prospecting_Android_Global_Broad_Image AOV, $79,96

While it's natural to have lower AOV for Android compared to iOS and Web, it might be useful to see if we can tweak targeting settings like countries and device settings to attract customers with higher order value. Alternatively, it might be beneficial to revisit user journey and how product connects to the ad copy.

2. Prospecting_Android_Global_Broad_Image has the low CTR and low ROAS (even lower than in awareness campaigns). Perhaps we are not meeting users where they are in the discovery phase. It might be useful to revisit the assets:

  • Prospecting_Android_Global_Broad_Image

  • Prospecting_Web_English_Broad_Image

3. In relation to spend check for ad fatigue:

  • Prospecting_Web_English_Broad_Image

  • Prospecting_Web_English_Broad_Spark

4. Put more money to assets with high ROAS in prospecting and little spend (sub 5K):

  • Prospecting_iOS_ES_Interest_UGC

  • Prospecting_iOS_Tier1_Interest_Spark

  • Prospecting_iOS_ES_Interest_Spark

5. Run awareness campaigns in the US

6. Ongoing A/B testing

Please prepare 2 scenarios for next month planning: 1) 50% monthly budget increase and, 2) 80% monthly budget increase; outlining development for key metrics (including overall NSGMV and NSGMV ROI) as well detailed budget allocation. [for reference, current budget =current monthly spend]

Budget reallocation

Before I scaled the budget, I decided to reallocate funds from low-performing campaigns to the campaigns that I think have potential. I took budget from the following underperforming campaigns. They have quite high CPC, which leads to high CPA and low ROAS.

  • Prospecting_iOS_Tier1_Interest_CreativeChallenge

  • Prospecting_Web_English_Broad_CreativeChallenge

  • Prospecting_Android_Global_Broad_Image

I used the funds to up the budget by 50% for the following campaigns:

  • Prospecting_iOS_ES_Interest_UGC

  • Prospecting_iOS_Tier1_Interest_Spark

  • Prospecting_iOS_ES_Interest_Spark

I've selected three them as they have strong middle-of-the-ballpark performance and all have smaller budgets under $5,000, allowing for room for scaling.

I then reallocated the remaining budget to the Awareness US campaign, mirroring the structure of Awareness_ES to assist with prospecting conversions.

Budget scaling

A +50% or +80% increase in monthly spend without disrupting performance, the strategy is to gradually increase daily budgets by 20% every 48 hours. This approach avoids triggering the learning phase and helps maintain stable CPAs and ROAS. This allows to scale in five and seven days respectively, allowing plenty of room during the months for last minute creative changes if needed.

ROAS impact of two potential budget increases is estimated based on the assumption that ROAS declines proportionally with increased spend. Specifically, I projected a 20% drop in ROAS for the +50% budget scenario and a 30% drop for the +80% scenario. Using the original ROAS figures and spend levels, I recalculated NSGMV for each campaign by applying the reduced ROAS to the new spend values. This enabled me to quantify how much revenue each campaign could generate under scaled budgets while accounting for expected performance drop.

If we want to carry the same scaled budget into the next month, the plan is to scale down by combining budget reductions with creative refreshes, avoiding entering the learning phase twice (once for each change).

Question 2

You have been tasked to come up with a process to improve creative performance on TikTok for Preply. What would be your testing framework? Please give examples of types of tests you would prioritize.

The following steps will lead to a successful A/B test:

  • Determining a hypothesis to test. Before setting up a split test, we need to determine what to test and how that can help inform ad strategy. This will help focus on what variables to test, design and run an effective A/B test.

  • Setting up test groups with large differences in variables. When choosing testing variable, the settings for the two test groups differ significantly. There should be obvious differences to ensure the two ad groups don't produce similar results, so the system can determine a winning ad group.

  • Setting an appropriate test budget. The budget for split test should produce sufficient results to give confidence in identifying the best performing strategy. When creating a split test, the system will provide the Estimated Testing Power based on the budget set.

  • Allocating an adequate testing duration. Testing for a minimum of 7 days to obtain the most reliable results. split tests can run for a maximum of 30 days. Testing for less than 7 days may not be long enough to determine a winner, while long testing may lead to a shortfall in the budget.

  • Setting a large audience. It's recommended to expand audience to avoid insufficient sample size when running a split test.

  • Ensuring a higher estimated power value. Power value is the likelihood of detecting potential differences in ad groups, which helps determine chances of finding a winning result (80%+).

  • Avoiding making changes. Making changes to the ad group after split test started running may impact its results or cause the ad group to go into review again. It is recommended to not change split test settings after it has started running.