Research Stories

Who’s cheating on your survey? What browsing data reveal about “googling” the answers

Online surveys can’t stop people from looking up the answers to knowledge questions. By linking a survey to its respondents’ browsing data, collected with their consent, we caught it happening, question by question. Here is how common it is, who does it, and whether it changes what we learn.

Illustration The question is real (wave 2), and so is the search: one of dozens logged while people were taking the survey.

In the summer of 2017, ahead of Germany’s federal election, an online survey asked: “By whom is the chancellor of the Federal Republic of Germany elected?”

No interviewer was watching. So 12% of the people who answered it opened another tab and asked a search engine. Most typed something like wer wählt den bundeskanzler (“who elects the chancellor”). Some pasted the whole question into the search box.

We know this because their browsers were logging every page they visited, with their consent.

Graphic: Illustration, not data. One survey session as the tracker logged it: survey starts, facebook.com, mail.google.com, a Google search for “wer wählt den bundeskanzler”, a news article about the FDP, amazon.de, survey ends.

Online surveys are cheap and fast, and social scientists rely on them more and more. But without an interviewer in the room, people can “google” the answers to knowledge questions. That can make the public look better informed than it is, and it can distort what we learn about who knows what about politics.

Earlier studies measured this indirectly: by asking people afterward whether they looked anything up, by adding obscure “catch” questions that almost nobody can answer without searching, or by logging when people switch browser windows. Depending on the method and the sample, they found cheating rates anywhere from 3% to 72%.

Graphic: The same illustrated survey session.

We took a more direct route. Our respondents came from YouGov’s Pulse panel in Germany. Its members agree to install software that records the web pages they visit on their laptops, desktops, and phones.

That let us line up two records for each person: when they started and finished the survey, and every page they opened in between. We searched those pages for keywords tied to each question (“Merkel” and “Partei,” say), checked every hit by hand, and then screened all visits to the sites those checks turned up.

Graphic: Illustration, not data. In the same survey session, two keywords are flagged: “bundeskanzler” in the Google search, which a check by hand confirms as a lookup, and “FDP” in a news article, which the check rules out.

Each dot is one of the respondents whose survey sessions we could find in the browsing data.

Over several survey waves, they answered commonly used political knowledge questions of four kinds: factual questions about the rules of the electoral system, visual elite questions (the party of politicians shown in photos), verbal elite questions (the names of the parties’ top candidates, typed into a box), and event questions (which recent news stories had really happened).

Graphic: 685 grey dots in a grid, one per respondent.

23% of them looked up the answer to at least one question.

Graphic: The same grid of 685 dots. 160 of them, 23%, are highlighted: people who looked up at least one answer.

But serial cheaters are rare. Most of those who looked something up did it once or twice: the typical cheater looked up two answers. 92% of them looked up fewer than a quarter of the questions they saw.

Graphic: The highlighted dots regroup into columns by how many answers each person looked up. Most stack up at one or two lookups; a thin tail stretches to the right.

Having another tab open is not the same as cheating. 66% of respondents visited some other website while taking the survey. Only 35% of those were looking up answers. The rest went largely to search engines, video sites, social media, email, and online shops.

That matters for methods that treat switching windows as cheating. They may flag many “innocent” people.

Graphic: Two bars. 66% of respondents visited another website while taking the survey. 23% looked up an answer, about a third of those who browsed.

Now each dot is one of the 68 items, placed by the share of answers to it that were looked up. Most sit close to zero. A few stand far out.

Graphic: The dots rearrange into 68 dots along a horizontal scale from 0% to 13%, one per item, placed by the share of answers that were looked up. Most sit below 3%; a handful lie far to the right.

Sorting them by type shows the pattern. When people had to type a candidate’s name into a box, they looked up 8% of answers on average. Questions about election rules came next (6%), then matching photos of politicians to their parties (3%), which is harder to search for. Recent events were looked up least (1%).

The single most googled item was the one about who elects the chancellor: 12% of answers in the second wave. (It’s the Bundestag.)

Graphic: The same 68 dots split into four rows by question type. Candidate names (average 8%) and election rules (6%) sit furthest right, politician photos (3%) in the middle, and recent events (1%) near zero. The top dot, who elects the chancellor in wave 2, is labeled at 12%.

In a companion survey in the United States, the picture looked very different. Of the 409 respondents whose browsing we could check reliably, only four looked anything up, six times in total, all on questions about recent events. That is fewer than 1% of respondents. Just 35 of them visited other websites during the survey at all.

One difference: right before the knowledge questions, the US survey asked people to promise not to look anything up. “Will you answer the following questions without help from outside sources?”

We can’t say the pledge caused the difference. It wasn’t randomly assigned, and the people and the questions differed too. But the pattern fits earlier experiments that found pledges reduce cheating, although those found smaller effects. A later study that did randomize a pledge, by Matthew Graham, put it plainly: “Deterrence works: a pledge not to cheat reduces search by half.”

Asking people afterward is no substitute. None of the four admitted to looking anything up. Eight others said they had, but we found no searches on the knowledge questions for any of them.

Graphic: A new grid of 409 dots, one per US respondent. Only four are highlighted.

Who looks things up? We modeled every single answer, taking both the person and the question into account.

People with more formal education were more likely to cheat, and so were people who feel well informed about politics. But people who actually know more about politics were less likely to cheat.

Spending lots of time on survey websites made no difference, and neither did how hard a question was.

Graphic: A chart of estimates with intervals. Right of zero means more likely to cheat. High education and feeling well informed sit to the right; actual knowledge sits clearly to the left. Gender and age lean right but their intervals cross zero. Habitual survey-taking and item difficulty sit on zero.

Could researchers spot cheating without tracking data? A common shortcut is response time: someone who looks up an answer should take longer. In our data they did. The typical question took 34 seconds when people answered on their own and 99 seconds when they looked something up.

Yet a model that combines response times with what surveys usually know about respondents and questions still did poorly. It caught only 60% of real lookups, and only 60% of the answers it flagged were real lookups. Because cheating is rare, relying on response times could miss many lookups and flag even more honest answers.

Graphic: Two distributions of response times on a log scale. Answers that were looked up (median 99 seconds) sit clearly to the right of honest answers (median 34 seconds), but the two overlap a lot.

So does cheating change what we learn about political knowledge? We ran 31 versions of a standard model of who knows more about politics, one for every combination of gender, age, education, political interest, and efficacy. We ran each twice, once with the raw knowledge scores and once with lookups counted as wrong, for six different knowledge measures.

Each dot is one coefficient compared across the two versions: 480 in all. A dot outside the band (a z-score beyond ±1.96) would mean that correcting for cheating changed the result by more than chance would explain. None do.

Knowledge also predicted political interest, certainty about one’s vote, efficacy, and the likelihood of voting about equally well among cheaters and non-cheaters. In most cases the differences were minor, and none was statistically significant.

Graphic: 480 dots in a narrow swarm around zero, all inside a shaded band from −1.96 to 1.96. None fall outside it.

Cheating happens, but it doesn’t seem to distort the big picture. Still, most researchers don’t have tracking data to catch it after the fact. So the best defense comes before: ask respondents to commit to not looking things up, and use questions that are hard to google, like matching photos of politicians to their parties.

Online surveys come with trade-offs. Valid knowledge measures don’t have to be one of them.

Graphic: Back to the 68 items in four rows by type. The politician-photo row is emphasized.

How we made this

The German data come from five survey waves fielded by YouGov in 2017, linked to the browsing data of panel members who had agreed to passive tracking. Four waves included knowledge questions. We kept respondents whose survey sessions appear in the browsing data. To flag lookups, we searched the pages visited during each survey session for question-specific keywords, checked every hit by hand, and then screened all visits to the sites those checks turned up. All keywords and every validated lookup are listed in the paper’s online appendix.

There are limits. We can’t see devices that weren’t tracked, or offline sources such as asking someone in the room. We also have to assume that a search during the survey was a lookup, not someone checking an answer they had already given.

Newer study mentioned

See all 68 items in a table

Share of answers that respondents looked up, by item. German panel, waves 2 to 5, 2017.

ItemTypeWaveAnswersLooked upShare (%)
Dobrindt calls for the end of diesel cars by 2030Recent events2545173.1
Gauland sentenced for seditionRecent events4553152.7
Trump appoints a general as chief of staffRecent events2545142.6
Schulz calls for a universal basic incomeRecent events2545132.4
Morales referendum sparks mass protestsRecent events2545122.2
UN Security Council toughens sanctions on North KoreaRecent events2545122.2
US Senate abolishes ObamacareRecent events2545112.0
Knife attack in a Hamburg supermarketRecent events2545112.0
Carmakers commit to free diesel hardware updatesRecent events2545101.8
Green MP Twesten leaves her group in Lower SaxonyRecent events254591.7
Unemployment falls below 4% in AugustRecent events355791.6
Berlin votes against keeping Tegel AirportRecent events551971.3
Greens rule out a Jamaica coalitionRecent events455371.3
Ryanair to take over Air BerlinRecent events355771.3
Etihad bids for Air BerlinRecent events455361.1
Merkel asks carmakers for free hardware updatesRecent events355761.1
Russia and China veto a North Korea resolutionRecent events455350.9
Parties want a five-year Bundestag termRecent events455350.9
Schulz asks Merkel for another TV debateRecent events455350.9
Volkswagen to end combustion cars by 2040Recent events455350.9
Weidel reportedly employed an asylum seeker off the booksRecent events455350.9
AfD deputy Arppe resignsRecent events355750.9
Schulz presents an education reformRecent events355750.9
FDP comes third in the federal electionRecent events551940.8
Russian hackers accessed voting softwareRecent events551940.8
Gauland open to a coalition with the Union and FDPRecent events551940.8
Bomb explodes on the London UndergroundRecent events455340.7
Juncker calls for the euro in every EU countryRecent events455340.7
Hurricane floods southern TexasRecent events355740.7
North Korea announces a hydrogen bomb testRecent events355740.7
Merkel names the Greens as her preferred partnerRecent events355740.7
Bavaria reports a 30% drop in reported rapesRecent events551930.6
Rohingya flee military violence in MyanmarRecent events551930.6
Threatening letters sent to top politiciansRecent events551930.6
Alleged rape near a refugee shelter in Upper BavariaRecent events551930.6
Two Germans arrested in TurkeyRecent events355730.5
Earthquake in Mexico kills more than 200Recent events551920.4
Supermarkets pull eggs over fipronilRecent events254520.4
EU to co-fund Hungary’s border fenceRecent events355720.4
Trump to rethink the Iran nuclear dealRecent events551910.2
Who elects the chancellor?Election rules25456712.3
Who elects the chancellor?Election rules5519407.7
Which vote decides the parties’ strength?Election rules2545193.5
Which vote decides the parties’ strength?Election rules551971.3
Top candidate(s): GreensCandidate names45535610.1
Top candidate(s): Left PartyCandidate names4553529.4
Top candidate(s): AfDCandidate names4553478.5
Top candidate(s): FDPCandidate names4553478.5
Top candidate(s): SPDCandidate names4553315.6
Top candidate(s): CDU/CSUCandidate names4553305.4
Party of Heiko Maas (photo)Politician photos2545183.3
Party of Angela Merkel (photo)Politician photos2545183.3
Party of Peter Altmaier (photo)Politician photos2545173.1
Party of Katrin Göring-Eckardt (photo)Politician photos2545173.1
Party of Peter Altmaier (photo)Politician photos5519163.1
Party of Heiko Maas (photo)Politician photos5519163.1
Party of Alice Weidel (photo)Politician photos2545162.9
Party of Martin Schulz (photo)Politician photos2545152.8
Party of Dietmar Bartsch (photo)Politician photos2545142.6
Party of Anton Hofreiter (photo)Politician photos2545142.6
Party of Christian Lindner (photo)Politician photos2545142.6
Party of Dietmar Bartsch (photo)Politician photos5519112.1
Party of Katrin Göring-Eckardt (photo)Politician photos5519112.1
Party of Anton Hofreiter (photo)Politician photos5519112.1
Party of Christian Lindner (photo)Politician photos5519112.1
Party of Angela Merkel (photo)Politician photos5519101.9
Party of Martin Schulz (photo)Politician photos5519101.9
Party of Alice Weidel (photo)Politician photos5519101.9

The paper

Simon Munzert, Sebastian Ramirez-Ruiz, Pablo Barberá, Andrew M. Guess, and JungHwan Yang (2024). “Who’s cheating on your survey? A detection approach with digital trace data.” Political Science Research and Methods 12(2), 390–398. DOI: 10.1017/psrm.2022.42

← More Research Stories