FEATURE

London COVID-19 BMJ: first published as 10.1136/bmj.m2469 on 24 June 2020. Downloaded from [email protected] Cite this as: BMJ 2020;369:m2469 http://dx.doi.org/10.1136/bmj.m2469 Why covid-19 antibody tests are not the game changer the UK Published: 24 June 2020 government claims described antibody testing as “game changing” in the pandemic. But experts have grave concerns over how good the tests are—or even what they mean. Stephen Armstrong investigates Stephen Armstrong freelance journalist On 21 May England’s health and social care secretary, companies Abbott and Roche. In May the UK , called reliable SARS-CoV-2 antibody government purchased 10 million test kits from the testing on a large scale “an important milestone” in two companies after validation from Public Health developing “immunity passports.” Such documents England (PHE) that they worked. Media reports at would, said Hancock, differentiate between people the time stated that the Abbott and Roche tests who had recovered from covid-19 and those still boasted 99% and 100% specificity, respectively, but vulnerable to infection, freeing many from social these figures came from the companies themselves distancing and giving people who have the antibodies and were based on those initial PHE studies and their assurances of what they could safely do. own marketing material rather than peer reviewed research. On 10 June, however, the head of the NHS Test and Trace programme, Dido Harding, said not enough Roche’s marketing material claims a sensitivity of was known about the level of protection that 100% 14 days after a patient was confirmed to have coronavirus antibodies provided. “I know we all want covid-19 through a polymerase chain reaction (PCR) it to be true that if we have antibodies it will then test.2 Abbott claims 100% accuracy 17 days after mean we are free to do things others are not,” she symptom onset.3 The PHE studies evaluating the kits told a press briefing. “But at the moment . . . if we use date of symptom onset.4 The discrepancy between have an antibody test what it tells you is you have the times used makes it difficult to compare the two antibodies.” test kits, yet alone standardise them for use by the http://www.bmj.com/ NHS. And on 24 June The BMJ published a letter from a group of senior clinical academics and physicians The accuracy of a test relates to whether it makes publicly questioning the government’s antibody errors: whether there are people with the disease who testing strategy,1 just as the first systematic review wrongly get negative test results, and whether there of studies on covid-19 antibody tests is set to be are people without the disease who wrongly get published. Jon Deeks, professor of biostatistics at the positive test results. “Saying a test is 100% accurate

University of Birmingham and The BMJ’s chief implies to the public that neither of these two types on 23 September 2021 by guest. Protected copyright. statistician, who led the Cochrane review, said the of error occurs,” says Deeks. “The reports from PHE analysis shows that “we don’t have much data [on make it clear that this statement is misleading.” the tests] and we can’t trust any of it.” A spokesperson for Roche Diagnostics UK told The The meaning of accuracy BMJ, “Our test is designed to be used as an aid to identify who has previously been exposed to the virus The Cochrane review, an advance copy of which The and has been through a rigorous regulated process . BMJ has seen, concludes that the data supporting . . This is based on extensive testing and validation, existing antibody tests are so vague that it’s including the measurement of over 5400 samples. impossible to know how accurate the tests are, especially for people with mild or no symptoms or “We are rolling out antibody tests to the NHS as part after symptoms have gone. of the crucial next step in understanding the spread of this virus, and providing greater confidence and Test accuracy, says Matt Keeling, professor of reassurance as we move into the next phase of our populations and disease at the University of Warwick, response to this pandemic.” is measured in sensitivity and specificity. High sensitivity means that if you’ve previously been Testing the testers infected with SARS-CoV-2 the test will correctly However, it’s not just the tests that are uncertain. identify this, while high specificity means that if you Even the studies evaluating the tests have problems, haven’t been infected the test will correctly identify as the Cochrane review found. Standards for reporting that. “This is important, as any [people with] false diagnostic accuracy studies have been around for a positives could incorrectly assume they have had the decade,5 says Deeks, yet none of the 54 studies the virus and therefore are a lower risk,” he explains. Cochrane review saw fit those. “We cannot properly At the time of writing two of the world’s leading tell anything about the research until the researchers suppliers of covid-19 antibody tests are the drug start following those guidelines.” the bmj | BMJ 2020;369:m2469 | doi: 10.1136/bmj.m2469 1 FEATURE

For instance, PHE’s report on Roche’s test has changed since the protection. The measles vaccine offers lifetime immunity, but the original study was posted online. The controversial claim to 100% most effective cholera vaccine, Dukoral, provides antibodies that BMJ: first published as 10.1136/bmj.m2469 on 24 June 2020. Downloaded from accuracy was based on a subgroup of patients who had the longest protect for only five years. time between the start of symptoms and being tested for antibodies In fact, there’s currently no evidence that covid-19 antibodies confer and had originally consisted of just eight patients tested after six immunity to the virus at all—and it’s conceivable they won’t, says to seven weeks. But the report The BMJ accessed online at the time Edwards. Immunity passports are a nice idea but may well never of writing includes patients without symptoms, boosting the number come to fruition. of patients to 10 and reducing the accuracy to a score of nine out of 10.6 A PHE spokesperson told The BMJ that this update was due to In a statement to The BMJ the Department of Health and Social Care two samples being incorrectly labelled as missing interval data and for England said, “We do not currently know how long an antibody that this had been noted in the document control section of the response to the virus lasts, nor whether having antibodies means report. a person cannot transmit it to others.” But it reiterated that antibody testing “will play an increasingly important role as we move into More importantly, the assessments of the tests were based on the next phase of our response to this pandemic.” samples and not patients—and almost certainly some patients will have contributed multiple samples that will make the results look It further said, “Antibody testing is helping us learn about the level more precise than they actually are, says Deeks. and length of immunity following infection and how the virus is spreading across the country,” adding that PHE is currently running “Where will these tests be used? That’s where you do the study,” the SIREN study of 10 000 healthcare workers to establish whether he says. “The samples [used in evaluating these tests] are probably antibodies indicate any kind of immunity to covid-19. from the patients in hospitals, who will be the most severely affected, and their antibody response is not that of people with mild Epidemiological studies such as SIREN could well make use of symptoms or those who are asymptomatic.” antibody tests. Wider testing could measure the spread of the disease across the country, identifying vulnerable population groups and Because the origin and severity of disease in the samples are not geographical regions. The Chinese city of Wuhan, for instance, used known, it is not possible to check whether they are representative antibody tests to check its population of around 10 million as it of the typical patient groups receiving an antibody test in real life. eased out of lockdown. But the authors of the 24 June letter to The And patients without covid-19 but with similar respiratory illnesses BMJ believe that such use is not in the UK government’s plans. were not included, so it’s difficult to tell whether false positives might arise from such people. Moreover, as the studies were Will Irving, professor of virology at the University of Nottingham undertaken in expert PHE laboratories, the performance of the tests and a signatory to the letter, said using antibody tests for when used in practice may not be as good. epidemiological surveys was possible but would need the structure

of a formal epidemiological study, taking into account geography, http://www.bmj.com/ According to Sheila Bird from Edinburgh University’s College of ethnicity, age, and sex. “This is not the same as simply testing a Medicine and Veterinary Medicine there are several problems with random set of volunteers who wish to know their antibody status,” PHE’s evaluation of the Roche and Abbott tests, including quality he said. of samples and the absence of data on age and sex, sample sizes, and use of repeat samples. Wasted money? PHE acknowledged that a small number of samples were repeat So what do we know about the currently available covid-19 antibody samples from different time points in the patient’s disease. It said tests? They are not a good indicator of current infection or for telling that samples reflected the general population. “Our evaluations people to isolate, says Andrew Preston, reader in microbial on 23 September 2021 by guest. Protected copyright. have been completed in record time using the samples and tests pathogenesis at the University of Bath. They are much more accurate that were available to us. We are confident that the volume of if used 14 days after symptoms arise, but currently people with samples and methodology was of a high standard,” its spokesperson symptoms are told to self-isolate for 14 days anyway, making taking said. a test moot. “Any laboratory using these tests is still required to complete their The UK government has already spent £16m (€18m; $20m) buying own evaluation to ensure the tests perform as described—our work antibody tests from China that proved inaccurate, many of which is designed to reduce the amount of local work required. Our now lie in storage. When it ordered 10 million of the Roche and evaluation work is ongoing and, as more tests become available, Abbott tests, financial details of the deal were not disclosed. If the we will continue to refine our approach.” Abbott test is supplied at cost, however, there is evidence to indicate that the NHS will spend £79 per test.7 Purpose unclear The demand for an expensive, fast turnaround on uncertain The big question is: what will these tests actually be used for? tests—with NHS trusts required to offer tests at short notice, building “What people really want to know from these tests is: am I safe from up to testing thousands of samples a day8—is arguably a waste of infection?” says Al Edwards, associate professor in biomedical public money, given that the tests are neither clinically urgent nor technology at Reading University. “These tests, at the moment, a public health priority. can’t answer that.” Irving told The BMJ, “The government has been focusing on arbitrary Neither the Abbott nor the Roche test detects antibodies against and meaningless targets, such as numbers of tests per day, without SARS-CoV-2’s outer “spike protein,” which studies have indicated any regard for the clinical value of those tests, whether or not the are most important for neutralising the virus. Both tests detect tests are actually carried out, or whether the results are returned in antibodies for a different protein termed N. a timely fashion to someone who knows the patient and is able to interpret those results.” Moreover, although antibody tests can measure the concentration of antibody in the blood, we don’t yet know what level is related to

2 the bmj | BMJ 2020;369:m2469 | doi: 10.1136/bmj.m2469 FEATURE

The government’s capacity target for 200 000 tests by the end of May included antibody testing as well as PCR tests. By 31 May it had BMJ: first published as 10.1136/bmj.m2469 on 24 June 2020. Downloaded from carried out 23 000 antibody tests, and as at 2 June testing capacity included 40 000 antibody tests. But, says Irving, “Focusing simply on numbers of tests done, without any consideration of the quality of those tests results, is contrary to the basic principles of pathology testing.” “I think the government bought the tests because they were caught out in not having the reagents or the tests for virus testing early on,” says Preston. “The idea of buying antibody tests to create immunity passports looked attractive, but using them like that is some way off. “If they’re not being used for full epidemiological studies, I think their main function will be in deciding who to vaccinate, assuming the vaccine comes. That will be useful—but I’m not sure it’s a game changer.”

Competing interests: I have read and understood the BMJ policy on declaration of interests and have no relevant interests to declare.

Provenance and peer review: Commissioned; not externally peer reviewed.

1 Andersson M, Low N, French N, etal. Rapid roll out of SARS-CoV-2 antibody testing—a concern. BMJ 2020;369:m2420. 2 Roche. Roche’s covid-19 antibody test receives FDA emergency use authorization and is available in markets accepting the CE mark. 3 May 2020. https://www.roche.com/media/releases/med- cor-2020-05-03.htm. 3 Bryan A, Pepper G, Wener MH, etal. Performance characteristics of the Abbott Architect SARS-CoV-2 IgG assay and seroprevalence in Boise, Idaho. J Clin Microbiol 2020;(May). doi: 10.1128/JCM.00941-20. 4 . Covid-19: laboratory evaluations of serological assays. 19 May 2020. https://www.gov.uk/government/publications/covid-19-laboratory-evaluations-of-serological-

assays. http://www.bmj.com/ 5 Bossuyt PM, Reitsma JB, Bruns DE, etal. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ 2015;351:h5527doi: 10.1136/bmj.h5527. 6 Deeks J. Twitter. 11 Jun 2020. https://twitter.com/deeksj/status/1271167411371364353?cxt=HH- wWgsC-zdKBjKQjAAAA. 7 Bridge-Wilkinson A. Atrumed sister company to supply Abbott’s newly approved Covid-19 antibody test. Parliamentary Review. 18 May 2020. https://www.theparliamentaryre- view.co.uk/news/atrumed-sister-company-to-supply-abbott-s-newly-approved-covid-19-antibody- test. on 23 September 2021 by guest. Protected copyright. 8 NHS England. Antibody testing programme roll out for NHS staff and patients [letter]. 25 May 2020. https://www.england.nhs.uk/coronavirus/wp-content/uploads/sites/52/2020/05/antibody- testing-programme-letter-25-may-2020.pdf.

the bmj | BMJ 2020;369:m2469 | doi: 10.1136/bmj.m2469 3