Results and Recommendations from Our Privacy Sandbox Testing

As a long-standing participant in the Privacy Sandbox initiative, NextRoll believes that Google’s proposed APIs can be a privacy-preserving alternative to third-party cookies (3PCs). Recently, we concluded our eight-week test of the Privacy Sandbox in order to gauge the viability of its Protected Audience API (PAAPI), Topics API, and Attribution Reporting API (ARA). Our results from this experiment indicate that, while these private APIs are functional, more work is required for Privacy Sandbox to sustain a healthy digital advertising ecosystem moving forward. This is not an unexpected outcome given how nascent these APIs are, and we look forward to continuing our work and partnerships to usher in a significantly more private web for users while maintaining advertising efficacy for our clients and the web’s many publishers. 

Let’s discuss a high-level overview of our testing methodology, findings, and a review of our team’s suggestions for Privacy Sandbox improvement. 

Testing Overview 

NextRoll’s latest Privacy Sandbox experiment, which ran during April and May 2024, tested the viability of Google’s proposed APIs using our entirely new bidding technology. This test compared ad performance of two groups: a control group, which bought impressions using our current 3PC-based system, and a treatment group, which bought impressions using a newly-built cookieless system. 

Our test included more than 20,000 campaigns across nearly all of our advertisers. NextRoll covered the advertising costs of these experiments for our clients. The average campaign in the treatment group showed about 65% as many impressions as the average campaign in the control group.

Our Findings

Overall, we observed 3x higher CPM and 2x higher CPC in the treatment group relative to the control group. A few other observations worth mentioning:

• We have a pacing mechanism that accounts for the amount of supply in the market. Given the low volume of PAAPI impression opportunities, we had to modify this mechanism to allow for much higher bid prices so we could purchase more data to learn from (once again, at NextRoll’s expense). This can simultaneously explain the higher CPMs and CPCs we observed, though more testing is necessary.

• We found the Aggregation Service to be scalable and performant. When appropriately configured, it was able to handle reports quickly at reasonable cost.

• Using the Topics API instead of 3PC-based features in our machine learning models decreased their predictive accuracy. This suggests that the Topics API cannot fully replace 3PCs to infer the likelihood that a user will interact with a product or brand. This is unsurprising as the removal of 3PC features includes the users’ browsing behavior on the advertisers’ sites, which PAAPI is the proposed replacement for.

• Across all of our customers, only 75% of the page visits that supported third-party cookies also supported interest groups. This is effectively a decrease in audience size that likely negatively impacted performance in the treatment group.

• We observed an average bid response time 5x above the upper bound of latency that we aim for in our 3PC-based bidder.

The magnitudes of these decreases in key performance metrics are beyond the level that would be acceptable to the advertising ecosystem today. However, we stress that these are very early days and there is a lot of head room to improve these numbers. Indeed, this has been our position for some time. Software development is always an iterative process, and new technologies take time to come into their own as the understanding around them increases. The good news is that the APIs demonstrate that private advertising is a true possibility. As proponents of a private web, we now have some recommendations on where we can focus our efforts next.

Recommendations for Privacy Sandbox Improvement

We believe the following improvements can lead to a substantial improvement in performance: 

1. Higher publisher adoption rates: More publishers running PAAPI auctions would increase the available ad inventory and improve campaign performance. We believe that low publisher adoption is a key driver of poor performance in our experiment as a simple matter of supply and demand curves. We address latency concerns below.

2. Adjustments to default level of noise added to attribution reporting: Noise is currently adjustable during the testing period, and we found noise to be high for granular reporting requirements by small, independent, and boutique advertisers receiving minimal conversions. The current approach can favor larger advertisers, which receive many conversions in a short period of time, and are thus less impacted by the reporting noise.

3. Increased interest group duration: Compared to the current 30 day duration, expanding to 90 days would enable longer-lived interest groups that increase a campaign's audience size and support longer sales cycles for attribution. Campaigns with larger audience sizes will see better performance. In addition, with NextRoll being a pioneer in remarketing, we know there is significant value in reminding users to revisit a potential purchase.

4. Enhanced fraud prevention efforts: Our research has identified several attack vectors that could lead to more ad fraud than we see today. This would be a further headwind to publisher adoption and hence also a headwind to increased performance. For obvious reasons, we will not disclose what we’ve uncovered, but we have shared our concerns with Google and we are actively working on potential solutions.

5. Committed full deprecation date: As opposed to phased deprecation, such as the method advocated by Criteo, we believe that once the largest gaps are resolved, Google should commit to a date when third-party cookies will be fully deprecated. Different DSPs and publishers may have different requirements for what they need in order to move from 25% to 50% of traffic using PAAPI, for example. Providers should be able to roll out PAAPI-based bidding at their own pace based on the experience and learnings they’ve gained along the way. Phased deprecation also introduces more deadlines, increasing the chances one or more of those deadlines will be missed by an earnest adopter. Allowing ad tech providers to scale up at their own pace up to a certain deadline will increase the chances that the Privacy Sandbox is successfully adopted.

6. Decreased bidding latency: Latency in the bidding process leads to an increase in the time it takes an ad to render on a publisher’s page. This creates a poor user experience and decreases the chance that the user views the ad, resulting in DSPs bidding less on the impression opportunity. By decreasing latency, publisher adoption is likely to increase.

Looking Forward

The results we observed during our eight-week experiment are a starting point for exciting, new technologies. With the suggestions above, we expect ad performance to improve significantly. We fundamentally believe that performance and privacy are not mutually exclusive. We share Google’s goal of creating privacy-preserving advertising that protects consumers and also supports the open web with its myriad publishers and participants. We look forward to continuing our existing partnerships, forging new ones, solving hard, meaningful problems, and ultimately bringing about a better web for everyone.

This article was authored by Andrew Pascoe, NextRoll’s Vice President, Data Science Engineering.