Teen on TikTok disrupted thousands of science studies with just one video
Thousands of scientific studies must have thrown away weeks of data because of a 56-second TikTok video made by a teenager.
The July 23 video is short and simple. It opens with Sarah Frank, a recent high school graduate from Florida and self-proclaimed “teenage author,” sitting in her bedroom and smiling for the camera.
âWelcome to the scrambles I recommend trying – part one,â she says in the video, pointing users to the Prolific.co website. âBasically it’s a bunch of surveys for different amounts of money and different time periods. “
This video was viewed 4.1 million times in the month after posting and sent tens of thousands of new users to the Prolific platform. Prolific, a tool for scientists conducting behavioral research, did not have free screening tools in place to ensure that it provided representative population samples in each study. Suddenly, scientists accustomed to getting a wide range of subjects for their prolific studies saw their surveys inundated with responses from young women Frank’s age.
For researchers relying on representative samples of the American population, this demographic shift was a major problem with no obvious cause and no immediately clear way to solve it.
Doing science on the internet
While not particularly well known, Prolific is one of a small collection of online tools that have transformed the way businesses and scientists study the way people think and act. The first and largest of these research platforms is Amazon-owned Mechanical Turk, which launched in 2005 as a general-purpose platform for crowdsourcing work on repetitive tasks. Soon after its release, behavioral scientists realized its potential value for their research and quickly revolutionized several areas of research.
âBefore Mechanical Turk existed, all social science research had to take place in a laboratory. You would need to bring in sophomores to college and put them through questionnaires, surveys and so on, âsaid Nicholas Hall, director of the Behavioral Lab at the Stanford School of Business.
âIt’s a business that requires a lot of time and manpower. Online research makes it so much easier. You schedule a survey … you put it online, and in one day you have 1,000 responses, âsaid Hall. The edge. âIt changed the face of the social sciences.
Today, Stanford’s Behavioral Lab primarily uses the newer and smaller Prolific platform for online studies, Hall said. While many of Mechanical Turk’s customers are large companies conducting corporate research, Prolific directs its product towards scientists.
The smaller platform offers more transparency, promises to treat survey participants more ethically, and promises better research topics than alternative platforms like Mechanical Turk.
Scientists doing this type of research in the United States generally want a group of subjects whose mother tongue is English, who are not too accustomed to psychological surveys, and who together constitute a demographic that is reasonably representative of the American population.
Prolific, most agree, has done a good job providing high quality content. The sudden change in platform demographics threatened to upend that reputation.
Consequences of TikTok
In the days and weeks after Frank posted his video, researchers struggled to figure out what was happening to their studies.
Analysis N = 300 Prolific study. Very generic title and description. Inquiry questions mainly on social comparisons and money
91% of respondents are women, 7% are men. Don’t think I mis-coded anything, and no unusual survey restrictions.
How is it going ?
– Sebastian Deri (@SebastianDeri) August 6, 2021
Member of the Stanford Behavioral Laboratory job on a Prolific forum, âWe noticed a huge jump in the number of platform participants in the US pool, from 40k to 80k. Which is good, though, now a lot of our studies have a gender bias where maybe 85% of the participants are female. In addition, the average age is around 21 years old.
Wayne State psychologist Hannah Schechter appears to have been the first person to resolve the case.
âIt may be far-fetched,â she tweeted, referring to Frank’s video, âbut given the timing, the virality of the video and the user’s subscriber demographicsâ¦â
Long-time prolific pollsters complained on reddit that Frank had made it difficult to find paid surveys for the overflow platform.
âNow it’s just another bullshit site to spend hours and make money on,â wrote one user, who said he already made $ 30 per week on the platform.
Frank, who “estimated” that she made a total of about $ 80 taking surveys on Prolific before her video, said The edge she also noticed a difference on the platform.
âFewer studies have been available to me and to everyone,â she said. The edge. “I have received some really nasty comments accusing me of ruining the site on my own and being selfish, even though I did not receive any compensation for this video.”
She added that she hoped Prolific would be able to put a system in place to deal with its changing demographics.
âI’m also predicting that a lot of people who signed up after seeing my video will forget about it and the push will subside,â she said.
Prolific co-founder and CTO Phelim Bradley said The edge that a lot of new users seem to be dropping.
âBefore Tiktok, around 50% of responses on our platform were from women,â he wrote in an email. “The push pushed that number up to 75% for a few days, but since then that number has tended to go down, and we’re now back to about 60% of responses coming from women.”
According to Bradley, about 4,600 studies were interrupted by Frank’s TikTok, or about a third of the total that was active on the platform during the surge. Of these, he said, the vast majority should be recoverable.
Prolific reimbursed researchers whose studies were significantly impacted by the increase in the number of women participating in surveys and introduced a new suite of demographic screening tools. The company announced the measures a month after Frank posted his video. The company has now reorganized by putting a team in charge of demographic balancing in order to recognize and respond more quickly to this kind of problem in the future.
âHonestly, we were taken a bit by surprise and we didn’t predict the magnitude of the impact,â Bradley said.
The push isn’t that bad. Refreshing the interview pool likely has long-term benefits, says Vlad Chituk, a Yale psychology graduate student who was conducting several pilot studies on Prolific when the wave hit. When subjects take part in many psychological surveys, they learn the tricks scientists use to collect data, which can impact how they answer future survey questions. The new subjects provide better quality data.
âYoung women who love TikTok are people too,â he said.
As for Frank, she says her side stampede video is now the most popular TikTok she has ever posted.
âIt certainly didn’t occur to me that the video was going to explode. I just posted it for my friends and my followers, not for the reach it ended up getting, âshe said. “I think it blew up because the site is so cool and people love the efficient ways to make money.”
For now, Frank has taken off most of her side activities as she settles into her freshman year at Brown.
Correction 9/27: A source from Prolific initially said the company did not implement demographic balancing in July. He informed us after the post that they had such a screening tool available for a fee at the time.