An Exploration of Advertising Effectiveness Methodologies: Comparing Recall to Opportunity to See

by InsightExpress April 5, 2012

Yesterday Molly Elmore, Vice President of Research here at InsightExpress (not to mention regular InsightfulAnalytics blog contributor) authored an article in MediaPosts’ Metrics Insider newsletter on a comparison of recall to Opportunity to See.  Like MediaPost, we think this is an important topic and hope you find the article of interest.

Below is an extended version of her piece, which includes several supporting charts unavailable in the Metrics Insider article.   If you have any questions about Molly’s analysis or would like to discuss her findings in more detail, she can be reached at melmore@insightexpress.com.

An Exploration of Advertising Effectiveness Methodologies: Comparing Recall to Opportunity to See

The Evolution of Advertising Effectiveness Research

Over the past decade, cross-media research has become increasingly important to advertisers.  Today’s marketers utilize multiple media channels to reach their target audience, and advertising research methodologies have also evolved to compare those channels on their ability to educate and persuade.  InsightExpress recently examined two popular methodologies to determine if and how their results differ.

Dueling Approaches

The first methodology is based on consumer recall of advertising. After a campaign launches, a respondent is asked a series of survey questions where one shows an advertisement that is part of the campaign being measured.  Those who recall seeing the ad are classified as “exposed” and those who do not are considered “unexposed.”  Comparing the two groups leads to a statistical determination of the effectiveness of the campaign at changing awareness and perceptions towards the brand.

The second methodology is based on the “opportunity to see” an ad  (OTS).  Here, respondents are asked about recent media consumption habits including specific TV shows, channels, and magazines.  In the digital world, cookie data is collected on advertisements sent to their browser.  This media consumption data is compared to the campaign’s media plan or a TV post buy report to determine who had the opportunity to see the advertisements.  Respondents who consume the “right” media but not at the specific time that the advertisements ran are assigned to the “unexposed” or “control” group to provide a baseline for comparison.  As with recall, the two groups are statistically compared to determine if any changes in attitudinal measures resulted from advertising exposure.

The Contest: Recall vs. Opportunity To See

To understand the benefits and limitations of these two approaches, we examined a random sampling of studies that contained the ad recall questions and the OTS questions.  They ranged in complexity and included a single site online advertising effectiveness study, a multi-site online advertising effectiveness study, and a multi-channel cross-media study.  Results were calculated using the OTS and recall methodologies, with data averaged across the various studies to determine how the results differed.

The data showed considerable differences between the two methodologies.  As seen below, the OTS aggregation revealed that, on average, the campaigns were moderately effective at boosting two of the exchange of information measures: awareness of advertising for the brand, and association between either messaging in the creative or sponsorship elements with the brand.

The recall-based approach led to very large increases in every measure, including the difficult to increase persuasion measures like favorability and intent.  But this was not surprising, since for Recall the exposed group just includes people who remembered having seen the advertisement.  Clearly, this approach would only be correct if 100% of the people exposed recalled that exposure.

The next step in our comparison was to understand the accuracy of each at assigning people to the correct group, (which can be difficult to do).  However, the digital channel tracking data provides information on which people were served the ads in the campaign allowing us to see if people correctly recall their exposures.

Two digital campaigns were used for this second part of the analysis.  The cookie tracking data was compared to respondents’ recall of the ads shown again within the survey.

Between 16% and 25% of those who were not shown the ads incorrectly recalled exposure to the campaign.  Only 52% to 69% of those who were shown the ads remembered the exposure.  Two campaigns may not provide enough data to definitely conclude what the false positive levels are across channels, but this analysis illustrates that people appear to have inaccurate memories for advertising recall.

While there may never be a perfect methodology when it comes to assigning people to cross-media exposure groups, using recall likely overstates the effect of campaigns considerably.  It may be tempting to use it because the results are likely going to be very positive, however, marketers and researchers looking for accurate findings should be cautious of the recall-based approach.

Filed under: Advertising Effectiveness,Cross Media Research,In the News,Research Insights

Tags: ad effectiveness, best practices, InsightExpress, MediaPost, Metrics Insider, Molly Elmore, opportunity to see, recall