Can you trust your data in 2024?

Honorariums attract fraud because people will cheat for even small amounts of money, especially if they can do it on a large scale.

Straight lining, speeding and poor-quality open-ended answers were once the telltale traces of fraud. These human-driven behaviors are not so hard to detect if you look carefully for them. However, things have changed drastically since the advent of artificial intelligence (AI) technologies like ChatGPT. Now AI has made it much easier to generate fraudulent respondents that are hard to detect, because these models can be trained to provide more varied and human-like answers.

Fortunately, greed is fraud’s Achillies’ heel. Fraudsters want to qualify for every study and get paid each time. In most surveys, a positive answer is what qualifies you to complete the study. People tend to move forward in surveys if they are aware of a brand, would consider purchasing it, have ever purchased it, or even just agree with a statement.

With AI being flexible, and wanting to avoid detection, bots can be programmed to vary their answers somewhat, so that a bot might agree moderately to one item and agree strongly to another, occasionally disagreeing. That is to throw off the analysis you would do to detect straight liners—people who give the same answer over and over. This makes these bots very difficult to detect, posing a significant concern. Let’s consider a couple of case studies.

Two fraudsters: one nakedly greedy, the other more subtle

In a recent business to business tracking study for an agricultural product, we saw awareness of all the brands soar upwards by 15-20%. That was more than a little suspicious as there were no other indications of such a large change in the market. Users of these products are low incidence and so multiple suppliers were used to obtain a sufficient sample.

When we broke out the sample by supplier, we quickly identified the source of the problem. One of the suppliers, whom we’ll call Supplier Y, provided a sample that was, amazingly, aware of just about every product. This naked greed was so blatant that it was obvious to anyone who looked. But another supplier, that we’ve labeled P, was more subtle and tried to cover their tracks by tempering their greed. Only half of their sample was aware of everything—a result that can blend in more readily but is still detectable if you have other reliable sources to compare to.

A more subtle, and worrisome, fraud       

In a multi-country study of public opinion, we found another more audacious yet subtle fraud afoot. When we first examined the sample, we found a few straight liners, speeders, and respondents with poor-quality open ends, but this was a drop in the bucket compared to a more sophisticated bot infiltration. We used pattern detection algorithms to identify a large group of “respondents” who agreed with just about everything—even when it made no sense.

This segment used the “say yes/agree with everything” strategy, but the bots varied their answers enough to avoid standing out like a sore thumb. These bots alternated between strong and moderate agreement and even occasionally disagreed. This means they evade any straight-line detection and can only be identified through a more flexible pattern recognition algorithm.

Our preliminary analysis identified 4 groups in the data—one of them being the “Oddly Agreeable.” They had a clear tendency to agree with everything, with a mix of strong and moderate agreement.

What confirmed the fraudulent nature of these “respondents” was their tendency to agree with things that were contradictory. For example, 95% agreed with the statement, “If we don’t all act on climate change now, we will ruin the planet for the generations to come.” However, 85% also agreed with, “I think we are fine to keep using energy the way we have been. No change is needed.” Clearly, this does not make sense. The fact that there were roughly 1,600 “respondents” like this—accounting for 40% of the sample in one country—was a very real concern.

Fraud has an impact

These bots distort study results. Consider this simple example: let’s assume you are a non-profit focused on educating consumers about climate change and the importance of reducing the amount of carbon they produce. If we had left the bots in, the study would tell you that 27% of people think climate change is a lie. That’s a dishearteningly large number of people requiring a great deal of education. But when we remove the fake respondents, we see that the real number is just half as large (15%). Clearly, bots can have a huge impact on things like market sizing, demand estimation and concept testing—pretty much any research-driven decision you might make.

Multiple defenses against fraud       

Complex and evolving fraud demands sophisticated detection methods. No one approach will reliably catch fraudsters. We saw from the multi-country study that classic methods of catching fraudsters missed a more subtle, but very large problem of 1600 bots masquerading as people. Clearly, a multi-pronged approach is necessary.

At the Angus Reid Group, we’ve developed Survey Sentinel^TM an AI-driven tool that employs advanced pattern detection algorithms to identify bots, as well as knowledge traps that bots can’t identify. It also includes hidden questions that bots can “see” but humans can’t, time stamps to identify speeders, analysis of open ends that includes AI-driven lookups for duplications and web-based sources, tools to identify duplicate IP addresses, and an ever-evolving set of approaches that allow us to take a multi-dimensional look at whose answers are to be trusted, and whose are to be deleted.

We employ it when people are recruited into our Angus Reid Forum communities, and we deploy it on each survey, in case a bad actor slips past. Our Survey Sentinel^TM never sleeps.