Elisabeth Bik Quit Her Job to Spot Errors in Research Papers — and Has Become the Public Face of Image Sleuthing

Feature

GABRIELA HASBUN FOR NATURE FOR GABRIELA HASBUN SEEING DOUBLE Elisabeth Bik quit her job to spot errors in research papers — and has become the public face of image sleuthing. By Helen Shen

ARE YOU A SUPER SPOTTER? Elisabeth Bik identified five duplicated areas in these cell microscopy images from separate experiments in two figures in a paper. We have simplified the labels. Can you spot the duplications? (Answers overleaf).

a b c d

2 (2019). , E0214018

3 , E91566 (2014); RETRACTION 14 RETRACTION , E91566 (2014); PLOS ONE 9 PLOS

4 SOURCE: S. GENG ET AL. S. GENG SOURCE:

the Belgian message. Attached are images or years chasing things which turn out to not of western blots — the results of a common be valid,” he says. test to detect proteins in biological samples Bik is not the world’s only image sleuth, but — from a published research paper. The writer she is unique in how publicly she presents her wants to know: does Bik see anything fishy in work. Many image checkers work behind the this paper? Have these pictures been digitally scenes, publishing their findings in research altered? papers and writing privately to journals; a few Bik, a microbiologist from the Netherlands are hired by journals or institutions. Some who who moved to the United States almost two flag up image problems work under pseudo- decades ago, is a widely lauded super-spotter nyms, preferring not to be identified. But Bik of duplicated images in the scientific litera- posts her finds almost every day on Twitter and ture. On a typical day, she’ll scan dozens other online forums, in the process teaching of biomedical papers by eye, looking for others how to spot duplications and pressur- instances in which images are reused and reported as results from different experi- “The things ments, or where parts of images are cloned, flipped, shifted or rotated to create ‘new’ data she calls out are (see ‘Are you a super-spotter?’; answers over- usually real issues.” leaf at ‘Did you spot them?’). Her skill and doggedness have earned her a ebruary the fourteenth starts like most worldwide following. “She has an uncommon ing journals to investigate papers. In so doing, other days for Elisabeth Bik: checking ability to detect even the most complicated she’s generated an “avalanche of reactions” her phone in bed, she scrolls through manipulation,” says Enrico Bucci, co-founder and awareness about the problem, says Bucci. a slew of Twitter notifications and of the research-integrity firm Resis in Samone, Bik estimates that her discoveries have led to at private messages from scientists seek- Italy. Not every issue means a paper is fraud- least 172 retractions and more than 300 errata ing her detective services. Today’s ulent or wrong. But some do, which causes and corrections — but all too often, she says, first request is from a researcher in deep concern for many researchers. “It’s a her warnings seem to be ignored. Belgium: “Hi! I know you have a lot of terrible problem that we can’t rely on some In April 2019, Bik announced that she had Fpeople asking you to use your magic powers aspects of the scientific literature,” says left her paid job at a biomedical start-up firm to analyse figures, blots and others but I just Ferric Fang, a microbiologist at the Univer- and would pursue image integrity work full- wanted to ask your opinion…” sity of Washington, Seattle, who worked on time, free of charge, for at least a year. A year After pouring a cup of coffee, Bik sits down a study with Bik in which she analysed more on, she shows no signs of changing course — at the long, wooden dining table that serves than 20,000 biomedical papers, finding prob- even though she has faced harassment, and at as her workstation at her home in California. lematic duplications in roughly 4% of them times been overwhelmed with requests. She’s She checks her e-mails on a giant 34-inch (E. M. Bik et al. mBio 7, e00809-16; 2016). “You also shared her files with computer scientists curved monitor, and takes a closer look at have postdocs and students wasting months trying to develop software to spot duplicated

Feature DID YOU SPOT THEM? her collection of nearly 2,000 turtle figurines Here are the duplicated sections Bik saw. — gathered from travels and friends — which she keeps in a wall of glass cabinets. Most a b c d prized of all her spreadsheets, however, is a collection of more than 3,300 questionable papers, most of them flagged because of an 1 issue with their images. (Bik sometimes raises other concerns with papers, such as around plagiarism or conflicts of interest.) On a day without interruptions, Bik can peruse 100 papers or so, adding between 1 and 20 hits to her database (see ‘Advanced

2 super-spotter test’; answers overleaf at ‘Did (2019). , E0214018 you spot them?’). A repeated smudge here or there, or a familiar smattering of data points: the visual indicators of duplication leap out at Bik from the screen. The collection is large enough to generate its own leads. It was look- 3 ing at other papers by authors in her mBio data set, for instance, that led Bik last November to 14 RETRACTION , E91566 (2014); a case that generated her widest media cover-

age so far: a cluster of papers co-authored by ONE 9 PLOS Cao Xuetao, a prominent immunologist who has advocated for stronger research integrity 4 in China, and who is the president of Nankai University in Tianjin. (Most of the articles

listed Cao’s other affiliation, at the National ET AL. S. GENG SOURCE: Key Laboratory of Medical Immunology in images across millions of papers, although the lead to an in-depth inquiry of the frequency of Shanghai.) Bik and other pseudonymous com- programs will probably always need human problems in biomedical work. They sampled menters flagged apparent issues in more than verification. “I’m enjoying it so much that I 20,621 papers, with Bik screening each — a task 60 papers at PubPeer. feel I just want to keep on doing this,” she says. Fang says only she could do — before passing China’s ministry of education said it would on her finds to Fang and Casadevall for cor- investigate the articles, and Cao replied at Hooked by a double smudge roboration. “It’s like a magic trick,” says Fang. PubPeer that he would re-examine the man- Bik stumbled into image sleuthing around “When it’s pointed out to you how it works, you uscripts, and that he was confident that the 2013, when, as a staff scientist at Stanford Uni- can start to see it.” The team found 782 papers publications remained valid. Some authors versity in California, she read articles about with what they termed “inappropriate” dupli- replied swiftly on the site to point to honest scientific integrity and plagiarism. Out of cations, and Bik notified the relevant journals. errors. In one case, apparent duplicate images curiosity, she googled quotes from her own The team reported the work in 2016 in mBio, at were in fact supposed to represent the same published papers, and quickly found that which Casadevall is editor-in-chief. experiment but were not clearly labelled other authors had lifted text without giving Bik spent so much of her spare time on as such, an explanation that Bik accepts. In credit. “I was hooked. I was angry,” she says. duplicated images that last year she decided another, authors posted raw data and said the “I immediately got fascinated about it, like to leave her job as director of science at Astarte data seemed similar only after being processed how other people get fascinated by reading Medical in Foster City, California. “I realized I for a paper. In still others, authors said there about crimes.” At one point, while examining a was getting more enthusiastic about image had been accidental mistakes, and by May this PhD thesis containing plagiarized text, some- duplication work than my real job,” she says. year, 13 of the flagged papers had received thing even more compelling caught her eye: a “It’s an impressive decision to make,” says corrections, most stating that scientific con- western-blot image with a distinctive smudge. Jennifer Byrne, a molecular biologist at the clusions weren’t affected. (Cao and China’s The same image appeared in another chap- University of Sydney in Australia and herself education ministry didn’t comment further ter, supposedly for a different experiment. a data-integrity sleuth who hunts for faulty for this article.) The chapters had also appeared as research genetic sequences in published papers. “It was Sometimes, Bik’s finds have pointed to articles, with the same errors, Bik saw. She very brave and, to be honest, pretty selfless.” suspected large-scale operations. This year, e-mailed journal editors in January 2014; in Bik does not get paid for most of her work, she and others have flagged a series of more June, she anonymously reported the papers but does some occasional paid consulting, than 400 papers that, they say, contain so many online at PubPeer, a website where scientists and receives modest sums through a Patreon similarities that they could be the product of a can discuss published papers. These were Bik’s crowdfunding page. After decades of working ‘paper mill’ — a company that produces papers first reports of suspected manipulation in the and saving, she expects her current situation to order. Several image detectives worked to literature. After an investigation by Case West- will be sustainable indefinitely. flag and collate the papers, including pseu- ern Reserve University in Ohio, the articles donymous sleuths @mortenoxe, @TigerBB8 were retracted in 2015 and 2016. The duplication database and @SmutClyde, who posted a list of papers Hunting for and cataloguing these images Bik now operates out of a light-filled dining in January, on a blog run by science journalist became a hobby. Then Bik contacted Fang and room, with floor-to-ceiling windows over- Leonid Schneider. “Finding these fabricated Arturo Casadevall, a microbiologist at Johns looking a garden filled with fruit trees and images should not rely solely on the work of Hopkins University in Baltimore, Maryland. other plants, which she has catalogued in a unpaid volunteers,” Bik wrote in February The trio decided that Bik’s rare talent could spreadsheet. She also has a spreadsheet for on her own blog. Journals say they are now

investigating the papers, many of which are ADVANCED SUPER SPOTTER TEST authored by doctors in Chinese hospitals, and Elisabeth Bik identified eight duplicate areas in these western blot images from separate experiments in some retractions are already being prepared. four figures in a paper. We have simplified the labels. Can you spot the duplications? (Answers overleaf). Bik’s data have revealed some insights into a b c d e f factors that correlate with image duplication. Her mBio paper reported that duplicated 1 images had a slight tendency to occur more

frequently in lower-impact journals. The paper (2019). , E0220600 2 also examined a subset of 348 articles flagged in PLoS ONE: taking into account the frequency i a d of publication in the journal, it seemed that a g d h papers from China and India were more 1 likely to contain problematic images. But Bik 1 doesn’t target one country’s authors, she says.

2 14 RETRACTION (2014); , E102195 “I search for problematic papers, regardless 2 of what country they are from,” she wrote in November. In all, Bik has flagged up dupli- a j d k cations in papers with lead authors from 49 countries. 1

Gamified sleuthing 2

Nearly every day, Bik posts images with ONE 9 ET AL. PLOS TAN Y. SOURCE: suspected problems to Twitter under the hashtag #ImageForensics, challenging her images. Last July, Bik commented on an image: Defects on an old microscope can create dark audience — which has almost tripled in the “For those of you who did not get an NIH R01 spots that seem the same on every image. past year to more than 60,000 followers — to grant around 2005, this is where that money “She has a good track record,” says Bernd spot the matches before she posts her answers. was spent on instead.” Pulverer, chief editor of The EMBO Journal, The puzzles attract numerous guesses within But there is also risk, especially for someone who calls Bik a world leader in manual image minutes, and some eagle-eyed players spot who refers to herself as “blunt and snarky” on screening. “The things she calls out are usually issues that she misses. (She gives out emoji her Twitter biography. “At some point, I am real issues.” medals to top performers.) Bik says she afraid people will sue me,” she says. She tries hears from some followers who have picked to keep her critiques to research papers, Public or private up skills from her and spotted problematic rather than accusing their authors. Bik has not Although many praise Bik for her work, some images while peer-reviewing manuscripts. faced a lawsuit, but has been harassed and has say the concerns shouldn’t be aired in public “I feel I’m changing people’s way of looking sometimes taken time off Twitter. One person before they are flagged privately to journals at these images,” she says. The work is some- e-mailed her former colleagues at Stanford or research institutions. “It’s very problem- times overwhelming for Bik, who calls her- arguing that she had abused her research grant atic,” says Lauran Qualkenbush, president self a “super introvert”. Last November, she funding by pursuing image integrity investi- of the US Association of Research Integrity tweeted: “I am getting so many (anonymous) Officers. She says that, in cases in which foul emails with people who want me to check cer- “We cannot, play is suspected, a public outing might hinder tain authors or papers that I cannot possibly investigative procedures by universities. “If follow up. So many names … And so much hid- unfortunately, someone did conduct research misconduct den pain among honest scientists about these clone Elisabeth.” intentionally, and then they’re alerted to the dishonest coworkers.” concern, it’s a great opportunity for them to Bik also posts detailed reports on what she destroy evidence,” she says. sees to PubPeer, and occasionally comments gations during work hours. (Bik says this was Bik — in common with other image sleuths there to support other tipsters. Many PubPeer untrue.) Another posted personal information — says she’s tried informing journals privately, users post their criticisms under pseudonyms on PubPeer (now removed). “I’ve been called but the case often seems to go nowhere or — as does Bik in some cases, if she feels very a bitch a couple of times,” she says. “It comes take too long to resolve. (She also notes that worried about litigious authors. But she has with the work I do.” researchers have opportunities to destroy posted more than 2,100 comments under Because she posts under her real name, Bik evidence even if investigations occur in pri- her own name at the site since 2014. “What says she errs on the side of caution, sometimes vate.) Between 2014 and 2015, Bik reported distinguishes Elisabeth is her willingness to deciding not to flag cases online, especially all 782 questionable papers from her 2016 identify herself, which is extremely admira- those with blurry or low-resolution images. On mBio study directly to journals through ble. It certainly helps with people taking the her own science-integrity blog, many entries e-mail. Some journals were unprepared for allegations seriously,” says Mike Rossner, a begin with some version of the phrase: ‘This the sheer volume of Bik’s reports. She flagged former managing editor at the Journal of Cell post is not an accusation of misconduct’. Sus- 348 papers of concern to PLoS ONE in a raft of Biology and president of Image Data Integrity, picious images don’t always point to corrupt 30 e-mails, each with 10 or 20 attachments. a consultancy firm in San Francisco, California. actions, she says: researchers might have mis- “That obviously created a backlog because we Being unemployed and independent gives takenly uploaded a file twice when preparing were not equipped to deal with it,” says edi- Bik the freedom to speak her mind, she says. figures, for instance. Then there are technical tor-in-chief Joerg Heber. Eventually, in 2018, “This one looks like nobody gave a fork about artefacts: membrane-thin slices cut sequen- the journal formed a three-person team ded- putting together a good science paper,” she tially from a piece of tissue can stick together icated to investigating image integrity and tweeted in March, with an accompanying fig- along one edge and flip open butterfly-style, other publication ethics cases full-time. “We ure panel that contained multiple duplicated creating an apparent mirrored duplication. published around 100 retractions last year.

Feature DID YOU SPOT THEM? Acuna’s team published on the bioRxiv pre- In the original paper, the figures were far apart. Finding duplicates between papers is even harder. print server preliminary results of an analysis

a b c d e f that extracted 2 million images from 760,000 papers (D. E. Acuna et al. Preprint at bioRxiv 1 http://doi.org/dtp2; 2018). It proved too computationally intensive to compare every

image with every other, but the team looked (2019). , E0220600 2 at image reuse within and across papers by the same authors. After manually examining i a d a g d h a sample of more than 3,700 of the match- ing images that the software flagged, the 1 1 researchers identified 40 cases that they all agreed were probably fraudulent; almost half

2 14 RETRACTION (2014); , E102195 2 of these involved the same image being used to 9 represent different results in different papers. a j d k Current technology is good at detecting outright duplications, and flipped or rotated 1 copies, says Bucci. His company, Resis, uses proprietary software to scan scientific manu- 2 scripts for its clients, which include journals and

research institutions. But complex problems ONE PLOS AL. ET TAN Y. SOURCE: are tougher, such as two images that share a Many of these were among cases that had Many — including Bik — argue that combating small overlapping area, but are otherwise com- been raised by her,” says Heber. The team is image manipulation and duplication requires pletely different. Advances in machine learning still working through Bik’s original tips, as well system-wide changes in science publishing, could be the key to detecting these and other as other cases. Bik gives PLoS ONE credit for its such as greater pre-screening of accepted man- subtle patterns automatically, he says. efforts, and says she receives regular notifica- uscripts. “My preference is not to have to clean But better software will need more data. tions of PLoS ONE retractions and corrections up the published literature, but to do it before- Machine-learning algorithms require training that have stemmed from her leads. But with hand,” says Rossner. He helped to introduce with an abundance of images that are known many of the nearly 800 cases in the mBio data universal image pre-screening of accepted to contain duplications. Bik has shared with set still unresolved, Bik’s patience is wearing manuscripts at the Journal of Cell Biology nearly Acuna images from hundreds of ‘dirty’ and thin. “I can tell you that 60–70% have not been 20 years ago. At the EMBO Press, says Pulverer, ‘clean’ papers from her 2016 study. And at the addressed after five years, so now, yes, I’m journals have pre-screened accepted papers Humboldt University of Berlin, researchers going to take it more publicly,” she says. for faulty images since 2013. But most journals funded by the publisher Elsevier are devel- The due-diligence process to check still do not pre-screen or (as with Nature) spot- oping a searchable database of images from concerns with papers often takes much longer check only a subset of papers before publica- retracted papers. For now, the collection has than people expect, Pulverer says. He and tion. “Image screening is not common right fewer than 500 entries, and most are in the Heber note that waiting for responses and raw now,” says Chris Graf, director of research life sciences and medicine and contributed by data from authors, and sometimes research integrity at the publisher Wiley. Elsevier, so the team wants more publishers to institutions, can be time-consuming. But the tide is slowly turning; Wiley participate. The publisher says that some of its Bik says she realizes that investigations publishes a few journals that screen images, journals are piloting image-checking software, take time. But she argues that journals could and is “preparing to launch a screening ser- and its goal is to provide all its journals with use expressions of concern more quickly vice” with the Journal of Cellular Biochemis- automated systematic checking. and frequently to notify other researchers try and the Journal of Cellular Physiology, Graf Until recently, Bik was unimpressed by the of potential problems, while possibly years- says. The journal Science has editorial coordi- software available. Now, she says, “I have full long investigations are pending. Heber says nators who check accepted manuscripts for confidence that in the next two years, comput- PLoS ONE uses expressions of concern when signs of image manipulation, but they don’t ers will be usable as a mass way of screening it has gathered enough information to be con- have capacity to check for some issues, such manuscripts.” But both Bik and Acuna say that cerned, but might hold off if an investigation as whether figures have been flipped, rotated people will always need to check the results is running smoothly, in favour of reaching a or duplicated, says executive editor Monica of such programs, especially to weed out resolution such as a correction or retraction. Bradford. instances where images can and should look Nature’s editor-in-chief, Magdalena Skipper, similar in certain parts. says that expressions of concern, which alert A job for AI? For now, Bik has plenty of work to do. This readers to “serious concerns” with a paper, are Many researchers say automation is the key to morning’s tip from Belgium looks like it might “a formal and permanent part of the scientific improving image integrity at a large scale. “We be a hit. Some of the western-blot bands — record; as such, we endeavour to use them cannot, unfortunately, clone Elisabeth,” says normally fuzzy and rounded like tiny black judiciously, adding them to papers once we Daniel Acuna, a computer scientist at Syracuse caterpillars — sport unusually sharp, pixellated have evidence that it is appropriate to do so”. University in New York, whose group is one edges, she says; these could be an innocent These days, Bik typically reports her of a handful working on algorithms to detect artefact introduced when a picture is com- discoveries directly on PubPeer. Some journals problematic images. Although Bik excels at pressed to a smaller size, or could suggest the and publishers track activity on the site, so she finding duplicated images in a single paper, application of photo-editing tools. “I’m going can reach journal editors and the public. “It’s computers could help to find more dupli- to ask him for the rest of the paper,” says Bik. more important to flag these papers and not cations between papers by comparing hun- worry about what happens behind the scenes dreds of thousands or millions of papers — an Helen Shen is a science journalist based in with these institutes,” she says. unfeasible task for humans, he says. In 2018, Sunnyvale, California.