How to Evaluate and Buy Online Adv Effectiveness Research

How to Evaluate and Buy Online Ad Effectiveness Research | Part 4: Cookies

by Marc Ryan January 17, 2012

Often misunderstood, always controversial, the ubiquitous cookie is the main mechanism for tracking ad exposure in ad effectiveness studies. True to its pedigree, the cookie enjoys quite a bit of notoriety, frequently showing up in Wall Street Journal headlines. Some consider it to be the Achilles heel of online ad measurement because it’s so susceptible to deletion. The idea that an individual could be exposed to an ad, delete the cookie associated with the ad, and subsequently be sampled for a control cell seems to be a deal breaker to some buyers of ad effectiveness research.

I understand this perspective, but the issue is certainly not so cut and dry. However, before we get into more detail, let’s look at how the cookie is deployed on the average study. The cookies used in online ad effectiveness studies operate in one of two basic ways: storage or identification.

In the storage model, all of the information about what a viewer of the advertising has been exposed to is stored in a cookie on the viewer’s machine. This means that the record of ad exposures lives in a cookie on the browser, and when that cookie is deleted the information about ad exposure is lost forever.
In the identification model, the cookie maintains a unique identifier and the ad exposure information is stored in a central database and keyed to the identifier in the cookie. However, while the information about the ad exposure is maintained indefinitely in the database, if a viewer deletes the cookie containing the identifier then the linkage between the user and the database is lost. An example of the storage model is the approach used by Safecount, which stores ad exposure information in a cookie on your browser (http://www.safecount.net/yourdata.php). On the other hand, InsightExpress employs an identification model where the cookie only stores a unique identifier.

Knowing who was exposed to the campaign is obviously critically important because it tells us who to sample for our research study. Yet it’s also important to know who wasn’t exposed to our campaign – those that don’t carry the cookie. Obviously, this is where people get nervous. If my friend Peter sees an ad for Bayer Asprin eight times, we can imagine the ad should have an extremely significant impact on Peter. On the other hand, if he deletes that cookie, the next time we see Peter we’ll think he’s never seen the ad. In fact, the next time we see Peter we might ask him to be part of our control cell as a respondent we think has never seen the advertising. Clearly, he has seen the ad but since he deleted his cookies we no longer can associate Peter with exposure to the advertising. Moreover, if we include him in our control cell as a respondent, his opinion is likely going to taint the results since he’s supposed to represent people who are unexposed to the advertising.

While at face value this scenario may seem to destroy the purity of our experimental design, it’s important to understand that in the grand scheme of things cookie deletion only has an impact on the assignment of test and control in some circumstances. This is because cookie deletion only becomes an issue when a campaign has a high degree of reach amongst the measured audience. This concept is easily explained using two examples.

Example 1: Low Reach Amongst Target

Let’s say I’m running a campaign on Yahoo! targeting a general audience. In this example, I’m running 5.5 million impressions on Yahoo! with an average frequency of one. Let’s assume that these impressions are run-of-site, so every one of Yahoo’s 145 million visitors has an equal chance of seeing the ad. This means that 139.5 million visitors to the site won’t see the ad while the remaining 5.5 million will see the ad.

If I survey 10,000 unexposed people for my control cell, the likelihood of a cookie deleter showing up in my sample is extremely low. Why? Well, think about the population I’ll be drawing my control sample from: people on Yahoo! unexposed to my advertising. There are 145 million of those individuals. Because of cookie deletion I’ll think that number is actually higher. Of the 5.5 million people that saw my ad, I can assume that 32% (the average cookie deletion rate) will delete their cookie, and as you will recall, once that cookie is deleted I’ll have no idea if someone has seen my ad. So 32% of 5.5 million is 1.76 million which you can add to the 139.5 million that didn’t see the ad giving you the total size of the audience that I think did not see the ad.

Right away you’ll notice something important: our deleters are a very small portion of our total unexposed audience and, in fact, they represent just 1.3% of the total unexposed audience. If true, then I can expect that of the 10,000 people I sample for my control cell 1.3% – or 130 of them – will have actually seen the ad but subsequently deleted their cookies. Those 130 are the people I need to worry about because since they have seen the ad they’ll likely have a higher opinion of the brand, thus polluting the purity of my control cell.

Here’s the big question: what impact did the deleters have on the results? Let’s assume the ad had a positive impact which resulted in 31% of people exposed to the ad wanting to buy the advertised product. This is in contrast to only 25% of people not exposed to the ad who want to buy the product. So if I split my 10,000 respondents into two groups – the 9,870 that didn’t see the ad and the remaining 130 who did – I can simply take a weighted average of these two numbers to see what influence the deleters had on my results.

In this omniscient example we know the truth to be that 25% of people who didn’t see the ad wanted to purchase the product. Now a weighted average including the people who did see the ad but deleted their cookies results in a purchase intent score for my control group of 25.07%. Barely an impact! The odds of a lot of cookie deleters showing up in my sample are just too low to worry about.

Example 2: High Reach Amongst Target

If I take the same campaign as above and change my target market to Hispanic moms between 25 and 34, there may be only 7,000,000 of those individuals on Yahoo! So my 5.5 million impressions could in fact reach a much higher percentage of the target market. For argument sake, let’s say we reached 60% of the target audience, or 4.2 million target viewers. In this example, only 2.8 million of the total target didn’t see our advertising. But don’t forget that on top of that 2.8 million we need to add in the people we think didn’t see our advertising (the cookie deleters). If 32% of the people that saw the ad delete their cookie there are another 1.3 million viewers of the campaign that I would think didn’t see the advertising. So I’ll think that the total number of people that didn’t see the advertising is 4.1 million (2.8M who didn’t see the ad + 1.3M who deleted their cookies). When I sample from this group I’ll inadvertently include in my survey sample a large number of people who did in fact see the advertising. I know that approximately 32% of my control sample did in fact see the tested advertising. That number stands in stark contrast to the measly 1.3% in the previous example.

By running through the same impact calculations, we learn that instead of the true purchase intent of 25% for this control cell, those cookie deleters will artificially inflate our control up to almost 27%. That’s enough of a difference in my data to adversely impact the results of my research.

The Reach Variable

As you can see, when the reach of an ad campaign is relatively low amongst the research audience the odds of someone who deletes their cookie being included in the sample for control is ridiculously low. And even if we accidentally included them in our analysis, the impact that those people have on the overall results is incidental.

On the other hand, a high reach campaign can be very adversely impacted by cookie deletion. The truth is that the high reach audience is more common than the low reach audience. It’s often the case that the structure of the analysis increases the reach amongst audiences we care about. This happens because we don’t really measure Yahoo! in aggregate with the research we conduct, we often measure defined audiences on small sections of sites (e.g. Yahoo! Autos).

As these examples illustrate, the impact that cookie deletion has on the measurement of your campaign is entirely dependent on your special circumstances. If you’re doing a heavily targeted measurement or if you’re running a heavy reach campaign, you’re bound to encounter problems with cookie deletion polluting the results from your control cell.

Misattribution of Frequency

But let’s say that you’re running the ideal campaign (if one were to exist) where cookie deletion was not an issue. Are you still safe from its effects? Well, when it comes to misattribution of an exposed respondent into the control cell, I’d say yes you’re in the clear…with one big caveat.

No matter how big or small your campaign might be, your understanding of the impact of frequency on ad effectiveness is entirely flawed as a result of cookie deletion. Our numbers show that, for an average campaign, approximately 50% of the cookies that are assigned to a frequency of one have indeed seen the ad multiple times. That’s a massive misattribution of frequency. This point is crucial, especially when trying to understand the impact of frequency on campaign metrics. Cookie deletion within your exposed audience is resulting in data that suggests viewers see the advertising significantly fewer times than they actually did.

When you think about it, this makes absolute sense. Take the example above with my friend Peter. We know he saw the tested advertisement eight times before he deleted his cookies. The next time he sees the advertising we’ll register him as seeing the ad once. Why? Because as soon as he deleted the original cookie he became someone we thought had never seen the ad, so his next exposure has to be at a frequency of one. Taking this example a step further, if Peter saw the ad eight times and deleted the cookie after every time he saw the advertising we’d think he’s eight different people who saw the ad once. Each deletion turns him into a new viewer of the ad with no way of recreating that history.

Mending an Achilles Heel

So is cookie deletion the Achilles Heel of the ad measurement industry? In short yes. But the long answer is that cookie deletion is not necessarily a problem when your campaign has a low target audience reach. However, if you plan to measure the impact of frequency (and all of our clients do), cookie deletion makes that analysis almost irrelevant.

Is there any good news here? Well, of course there is. At InsightExpress we firmly believe in making sure you’re adjusting for cookie deletion,. In fact we take it a step further and adjust not only for deletion but also for the fact that a viewer might be seeing your ads in multiple browsers or on multiple devices. We employ a patent pending approach called a universal ID that doesn’t just apply an aggregate cookie deletion fudge factor as many competitors do, but specifically identifies when people are deleting and reassigns their exposures in our data warehouse. This ensures that when our analysts are looking at data about exposure to an ad campaign they will know that everyone has the correct exposure assignment.

This post is part of Marc’s series on “How to Evaluate and Buy Online Ad Effectiveness Research.” To read other posts in this series, click here.