But what really works and actually moves voters? In this polarized world, does persuasion even work, or are we just trying to mobilize voters?
Well, How Experiments Help Campaigns Persuade Voters: Evidence from a Large Archive of Campaigns’ Own Experiments makes a valiant effort, but falls short in a couple of areas.
While this literature does add to our knowledge, it has a significant blind spot. Due to the research design, only ads from Democratic campaigns were studied—but more on that later.
Additionally, we must critically evaluate the findings, noting that it appears the ad testing company, Swayable, has researchers listed as authors on the paper, and the findings seem to be extremely favorable to the company’s revenue goals.
Noting these two major concerns, let’s explore this study to see how political campaigns (remember—Democratic campaigns only) use experiments to figure out which ads work best.
Spoiler alert: It’s more complicated than you think.Title: How Experiments Help Campaigns Persuade Voters: Evidence from a Large Archive of Campaigns’ Own Experiments
Link: Read the study
Peer Review Status: Peer-reviewed
Citation: Hewitt, L., Broockman, D., Coppock, A., Tappin, B. M., Slezak, J., Coffman, V., Lubin, N., & Hamidian, M. 2024. “How Experiments Help Campaigns Persuade Voters: Evidence from a Large Archive of Campaigns’ Own Experiments.” American Political Science Review, 118(4): 2021-2039. doi:10.1017/S0003055423001387.
Methodology
The study analyzed a treasure trove of data from 146 experiments run by the tech platform Swayable, which worked with Democratic and left-leaning campaigns. These experiments tested 617 ads on over 500,000 respondents. Here’s the gist:
- Type of Study: Randomized survey experiments
- Sample Size: Over 500,000 respondents, including diverse demographic and political groups
- Experimental Design: Ads were tested on treatment groups, while control groups viewed neutral videos (e.g., public service announcements).
- Measures: Respondents rated their likelihood to vote for a candidate and their favorability toward that candidate on a scale of 0 to 10. Results were adjusted for variables like gender, age, and partisanship to ensure accuracy.
Each experiment aimed to measure the persuasive impact of specific ad features, such as emotional tone, messenger characteristics, and informational content. Data collection spanned two election cycles, offering a comprehensive look at how ad effectiveness varies by context.
Factors Considered in Ad Effectiveness
Figure 3 of the study provides a detailed analysis of the features evaluated for their impact on ad effectiveness. The following factors were explored:
- Tone of the Ad: Whether the ad was positive, negative, or contrast.
- Message Content: Inclusion of new facts, emotional appeals (anger, enthusiasm, fear), or policy details.
- Messenger Characteristics: The demographic attributes, partisanship, or credibility of the spokesperson.
- Production Quality: The perceived professionalism or “polished” nature of the ad.
- Pushiness: How assertive the ad was in delivering its message or call to action.
The results revealed a lack of consistency across contexts. For instance, ads emphasizing enthusiasm might perform well in one election but have no discernible effect in another.
Similarly, while emotional appeals were hypothesized to boost effectiveness, the actual impact was context-dependent and often minimal. This inconsistency underscores the challenge of predicting ad success without experimentation.
Results and Findings
- Overall Effectiveness: Ads had small but significant persuasive effects. On average:
- Vote choice shifted by 2.3 percentage points in 2018 down-ballot races.
- Effects dropped to 1.2 points in 2020 down-ballot races and
- Effects dropped to 0.8 points in the 2020 presidential race.
- Variation in Effectiveness: Persuasion varied significantly among ads. Some were 50% more effective than the average, while others were 50% less effective.
- Unpredictable Features: Conventional wisdom about what makes ads effective—like emotional appeals or testimonials—had limited and context-dependent predictive power. What worked in 2018 didn’t necessarily work in 2020.
- Implications for Campaigns: Experimentation is invaluable. Simulations showed that campaigns investing in testing ads could dramatically improve their impact, especially in close elections.
Simulations and Their Impact
In what could almost be considered a side note, the researchers conducted simulations to explore the potential value of ad experimentation in campaign strategy. Here’s what they did and what they found:
- Method: They modeled scenarios in which campaigns invested resources in testing ads to find the most persuasive ones. This allowed them to model/estimate how much more effective overall advertising efforts could become with such targeted approaches.
- Findings: The simulations showed that even modest investments in experimentation could yield significant returns, particularly in close elections. Choosing a highly persuasive ad over a less effective one could shift the needle in tight races, demonstrating the importance of data-driven decision-making.
Once again, it’s important to note that this research was published in conjunction with an ad testing platform.
Critiques of the Research
- Generalizability: As discussed, the data came exclusively from Democratic campaigns using the Swayable platform. No conservative ads were included in the analysis. Given the significant personality differences often found between liberals and conservatives, these results may not be generalizable to right-leaning campaigns or voters.
- Generalizability (con’t): Additionally, the analysis did not include ballot iniatives or other type of influence campaigns, which are typically more nonpartisan.
- Attrition and Sampling Issues: Some experiments lacked complete data on respondents who dropped out, potentially skewing results.
- Timing Limitations: Findings are specific to the 2018 and 2020 elections and may not generalize to future elections with different political dynamics, particularly as campaigns continue to learn and improve.
- Industry Sponsorship/Relationship: The research was conducted only with ads tested by the Swayable platform. Some of the findings support the use of ad testing. While I’m not accusing the researchers of anything, it’s important to note the relationship. It’s crucial to be critical when interpreting the results.
I will note the authors do a good job and are transparent in offering these caveats.
Additional Areas of Potential Study
- Bipartisan Analysis: Expanding the dataset to include Republican campaigns could reveal whether persuasion techniques differ by political party / ideology.
- Non-partisan Analysis: Expanding the dataset to include ballot initiatives and issue ads could reveal whether persuasion techniques differ from partisan activities.
- Primary / General Election Analysis: Expanding the ads studied to group intra-party (primary) versus inter-party (general) could reveal some significant differences. This is especially critical given the substantial number of elections decided in the primary, influenced by sorting and gerrymandering.
- Field Experiments: Testing ads in real-world settings—not just surveys—would enhance ecological validity. Would the results be ‘replicated’ in focus groups?
- Ad Context: Future research could explore how competing ads or media coverage influence ad effectiveness.
Conclusion
For me, this is one of those frustrating studies. When you read the title, the title promises answers, but the study raises more questions than it answers.
This research sheds light on how Democratic campaigns use experiments to refine their strategies, emphasizing the need for ad testing in a rapidly changing political environment. However, as Steve Schale, a Democratic operative from Florida, points out, ad-testing can become an obsession that overshadows the larger narrative of the campaign. He writes, “”We (Democrats) were addicted to ad-testing, to the point that it drove decision-making more this cycle than the desire or need to tell a story.”
The study concludes that ad effectiveness is deeply tied to the specific context of each election, meaning strategies that worked in 2018 might not have the same impact in 2020, let along in 2026 or 2028. This highlights the challenge for campaigns: predicting what will work is difficult, and there are no guarantees. In fact, attempting to replicate past success is likely to be a futile endeavor.
A cynic would write, the authors conclude that persusion is extremely context driven, and what may work in one election may not work in another….whomp whomp. Therefore, campaigns should test ads.
Nevertheless, the cumulative impact of even small shifts in ad effectiveness can influence election outcomes, particularly in tight races. The study underscores the growing importance of data-driven decision-making in modern campaigns, but it also leaves many questions unanswered—questions that future research will need to address to refine our understanding of political ad effectiveness.
