Evaluation of Microscan Bacterial Identification Panels for Low

diagnostics

Article Evaluation of MicroScan Bacterial Identiﬁcation Panels for Low-Resource Settings

Sien Ombelet 1,2,* , Alessandra Natale 3, Jean-Baptiste Ronat 3,4,5, Olivier Vandenberg 6,7,8 , Liselotte Hardy 1 and Jan Jacobs 1,2

1 Department of Clinical Sciences, Institute of Tropical Medicine, 2000 Antwerp, Belgium; [email protected] (L.H.); [email protected] (J.J.) 2 Immunology & Microbiology Department, KU Leuven, 3000 Leuven, Belgium 3 Médecins Sans Frontières, Operational Center Paris, 75019 Paris, France; [email protected] (A.N.); [email protected] (J.-B.R.) 4 Team ReSIST, INSERM U1184, School of Medicine, University Paris-Saclay, 94807 Villejuif, France 5 Bacteriology-Hygiene Unit, Assistance Publique—Hôpitaux de Paris, Bicêtre Hospital, 94270 Le Kremlin-Bicêtre, France 6 Center for Environmental Health and Occupational Health, School of Public Health, Université Libre de Bruxelles (ULB), 1050 Brussels, Belgium; [email protected] 7 Innovation and Business Development Unit, Laboratoire Hospitalier Universitaire de Bruxelles—Universitair Laboratorium Brussel (LHUB-ULB), Université Libre de Bruxelles (ULB), 1050 Brussels, Belgium 8 Division of Infection and Immunity, Faculty of Medical Sciences, University College London, London WC1E 6BT, UK * Correspondence: [email protected]

Abstract: Bacterial identification is challenging in low-resource settings (LRS). We evaluated the Mi- croScan identification panels (Beckman Coulter, Brea, CA, USA) as part of Médecins Sans Frontières’ Mini-lab Project. The MicroScan Dried Overnight Positive ID Type 3 (PID3) panels for Gram-positive Citation: Ombelet, S.; Natale, A.; organisms and Dried Overnight Negative ID Type 2 (NID2) panels for Gram-negative organisms were Ronat, J.-B.; Vandenberg, O.; Hardy, assessed with 367 clinical isolates from LRS. Robustness was studied by inoculating Gram-negative L.; Jacobs, J. Evaluation of MicroScan species on the Gram-positive panel and vice versa. The ease of use of the panels and readability of the Bacterial Identification Panels for instructions for use (IFU) were evaluated. Of species represented in the MicroScan database, 94.6% Low-Resource Settings. Diagnostics (185/195) of Gram-negative and 85.9% (110/128) of Gram-positive isolates were correctly identified 2021, 11, 349. https://doi.org/ 10.3390/diagnostics11020349 up to species level. Of species not represented in the database (e.g., Streptococcus suis and Bacillus spp.), 53.1% out of 49 isolates were incorrectly identified as non-related bacterial species. Testing Academic Editor: Raul Colodner of Gram-positive isolates on Gram-negative panels and vice versa (n = 144) resulted in incorrect identifications for 38.2% of tested isolates. The readability level of the IFU was considered too high Received: 17 January 2021 for LRS. Inoculation of the panels was favorably evaluated, whereas the visual reading of the panels Accepted: 17 February 2021 was considered error-prone. In conclusion, the accuracy of the MicroScan identification panels was Published: 19 February 2021 excellent for Gram-negative species and good for Gram-positive species. Improvements in stability, robustness, and ease of use have been identified to assure adaptation to LRS constraints. Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in Keywords: clinical bacteriology; low-resource settings; bacterial identification; blood cultures; Mi- published maps and institutional affil- croScan identification system iations.

1. Introduction Copyright: © 2021 by the authors. Bacterial Identification in Low-Resource Settings Licensee MDPI, Basel, Switzerland. Clinical bacteriology laboratories equipped to diagnose bacterial infections and per- This article is an open access article form antibiotic susceptibility testing (AST) are scarce in low-resource settings (LRS) [1,2]. distributed under the terms and conditions of the Creative Commons Correct identification of bacteria in clinical samples is a cornerstone for proper management Attribution (CC BY) license (https:// of bacterial infections because both empirical and directed antibiotic treatments depend on creativecommons.org/licenses/by/ the identified organism [3,4]. 4.0/).

Diagnostics 2021, 11, 349. https://doi.org/10.3390/diagnostics11020349 https://www.mdpi.com/journal/diagnostics Diagnostics 2021, 11, x 2 of 20

Diagnostics 2021, 11, 349 Correct identification of bacteria in clinical samples is a cornerstone for proper manage-2 of 19 ment of bacterial infections because both empirical and directed antibiotic treatments depend on the identified organism [3,4]. CurrentCurrent “gold“gold standard”standard” identificationidentification methodsmethods inin high-resourcehigh-resource settings,settings, suchsuch asas matrix-assistedmatrix-assisted laser laser desorption/ionization desorption/ionization time-of-flight time-of-flight mass mass spectrometry spectrometry (MALDI-TOF (MALDI- MS)TOF and MS) molecular and molecular techniques techniques are not are adapted not adapted to diagnostic to diagnostic laboratories laboratories in LRS in [1 LRS,5]. Most [1,5]. laboratoriesMost laboratories in LRS in still LRS rely still onrely “conventional” on “conventional” phenotypic phenotypic identification identification techniques, techniques, in whichin which isolates isolates are inoculatedare inoculated on different on different culture culture media media containing containing different different carbohydrates carbohy- anddrates enzyme and enzyme substrates, substrates, and interpretation and interpretati of teston results of test is carriedresults outis carried using dichotomousout using di- decisionchotomous trees decision [6–8]. Commercializedtrees [6–8]. Commercialized panels consisting panels of consisting phenotypic of tests,phenotypic such as tests, the Analyticalsuch as the Profile Analytical Index (APIProfile®) panels Index (bioM(API®é)rieux, panels Marcy (bioMérieux, l’Etoile, France) Marcy offer l’Etoile, probability- France) basedoffer probability-based identification; combined identification; test results combined are compared test results to databases are compared containing to the databases profiles ofcontaining thousands the of profiles bacterial of isolates thousands [9]. of bacterial isolates [9]. InIn responseresponse toto thethe globalglobal riserise inin antimicrobialantimicrobial resistanceresistance andand thethe needneed toto implement implement context-tailoredcontext-tailored antibiotic antibiotic stewardship stewardship programs, programs, the non-governmentalthe non-governmental humanitarian humanitarian orga- nizationorganization Médecins Médecins Sans FrontiSans èFrontièresres (MSF) (MSF committed) committed to developing to developing an all-in-one, an all-in-one, affordable, af- andfordable, transportable and transportable clinical bacteriology clinical bacteriolo laboratory,gy laboratory, the Mini-lab the [ 10Mini-lab]. Because [10]. the Because Mini-lab the isMini-lab intended is forintended use in for LRS use by in non-expert LRS by non-ex laboratorypert laboratory technicians, technicians, a system fora system bacterial for identificationbacterial identification is required is that required is easy that to inoculate,is easy to easyinoculate, to read, easy inexpensive, to read, inexpensive, and with long and shelfwith life.long After shelf defining life. After technical defining specifications technical specifications (target product (target profile) product and performing profile) and a marketperforming analysis, a market the Dried analysis Overnight, the Dried MicroScan Overnight ID panels MicroScan by Beckman ID panels Coulter by (Brea, Beckman CA, USA)Coulter were (Brea, chosen CA, becauseUSA) were they chosen have abecause long shelf they life have and a can long be shelf read life without and can the be use read of automatedwithout the instruments. use of automated instruments. ToTo meetmeet Mini-lab Mini-lab specifications, specifications, Beckman Beckman Coulter Coulter developed developed a customized a customized “Research “Re- Usesearch Only” Use ID Only” panel ID containing panel containing all test wells all test for wells the identification for the identification of both Gram-negative of both Gram- andnegative Gram-positive and Gram-positive organisms organisms on one single on one panel, single coded panel, MSFNPID1. coded MSFNPID1. The Gram-negative The Gram- andnegative Gram-positive and Gram-positive test wells were test assembledwells were in assembled separate groups in separate on the groups panel, andon the therefore, panel, knowledgeand therefore, of the knowledge Gram stain of result the Gram is still stain essential result to readis still the essential panel correctly to read (Figure the panel1). The cor- primaryrectly (Figure objective 1). The of this primary study objective was to assess of this the study diagnostic was to accuracy assess the of diagnostic these identification accuracy panelsof these with identification clinical isolates panels originating with clinical from isolates LRS. A originating secondary from objective LRS. was A secondary to assess theob- easejective of usewas of to the assess system. the ease of use of the system.

Figure 1. Example of an inoculated customized Médecins Sans Frontières (MSF) ID panel (MSFN- PID1): Gram-negative test wells on the top side and Gram-positive test wells on the opposite side. Gram-positive and Gram-negative test panels can thus be inoculated simultaneously; for reading, one of the two sides must be picked.

Diagnostics 2021, 11, 349 3 of 19

2. Materials and Methods 2.1. Identiﬁcation Panels We evaluated the Dried Overnight Positive ID Type 3 (PID3) and Dried Overnight Negative ID Type 2 (NID2) panels and the customized version of the MSF Negative Positive ID Type 1 (MSFNPID1) panel. We will further use the terms “Gram-positive panels” and “Gram-negative panels” when referring to the composition of test wells. When dealing with the specific panels, we will use the terms PID3, NID2, and MSF ID panels, respectively. Table1 demonstrates the different tests performed on each panel and reagents to be added. The Beckman Coulter LabPro software version 4.43 and the MicroScan autoSCAN- 4 automated reader were used for this evaluation. Lot numbers of panels are listed in Supplementary Table S1.

Table 1. Test wells on the MicroScan Gram-positive and Gram-negative panels (Dried Overnight Positive ID Type 3 (PID3) and Dried Overnight Negative ID Type 2 (NID2), respectively) or embedded in the customized MSF panel (MSFNPID1) and reagents to be added. Underlined substrates are wells for which coverage with mineral oil before incubation is needed after inoculation. Substrates in bold are tests for which additional reagents need to be added after incubation and before reading.

Substrate. Abbreviation Panel Substrate Abbreviation Panel Glucose GLU Gram-negative Lactose LAC Gram-positive Sucrose SUC Gram-negative Trehalose TRE Gram-positive Inositol INO Gram-negative Mannose MNS Gram-positive Adonitol ADO Gram-negative Ribose RBS Gram-positive Rhamnose RHA Gram-negative Inulin INU Gram-positive Melibiose MEL Gram-negative Mannitol MAN Gram-positive

Penicillin G 4 µg/mL P4 Gram-negative PNP-β-D-Glucuronide PGR Gram-positive

Kanamycin 4 µg/mL K4 Gram-negative PNP-β-D-Galactopyranoside PGT Gram-positive Colistin 4 µg/mL Cl4 Gram-negative Indoxyl Phosphatase IDX Gram-positive Cephalothin 8 µg/mL Cf8 Gram-negative Phosphatase PHO Gram-positive Nitrofurantoin 64 g/mL Fd64 Gram-negative Pyrrolidonyl-B-naphtylamide PYR Gram-positive Tobramycin 4 g/mL To4 Gram-negative Bile-Esculin BE Gram-positive Cetrimide CET Gram-negative Pyruvate PRV Gram-positive Lysine LYS Gram-negative Bacitracin BAC Gram-positive Ornithine ORN Gram-negative Crystal Violet CV Gram-positive Tryptophan Deaminase TDA Gram-negative Bacitracin 0.05 g/mL MS Gram-positive Esculin ESC Gram-negative Novobiocin 1.6 g/mL NOV Gram-positive

o-Nitrophenyl-D-Galactopyranoside ONPG Gram-negative Optochin OPT Gram-positive Citrate CIT Gram-negative 6.5% NaCl NACL Gram-positive Malonate MAL Gram-negative Sorbitol SOR Both panels Acetamide ACE Gram-negative Rafﬁnose RAF Both panels Tartrate TAR Gram-negative Arabinose ARA Both panels Oxidation Base Control OF/B Gram-negative Arginine ARG Both panels Oxidation of glucose OF/G Gram-negative Nitrate NIT Both panels

Hydrogen Sulﬁde H2S Gram-negative Voges-Proskauer VP Both panels Indole IND Gram-negative Urea URE Both panels Decarboxylase Base Control DCB Gram-negative

2.2. Clinical Isolates and Reference Strains A total of 367 isolates were tested (Tables2 and3), including 332 anonymized clinical isolates and 35 reference strains corresponding to the most common bloodstream pathogens or contaminants in LRS. Of the clinical isolates, 66.9% originated from sub-Saharan Africa, Diagnostics 2021, 11, 349 4 of 19

26.8% from Asia, 4.2% from South America, and 2.1% from Europe (See Supplementary Table S2 for details per species). Reference strains were obtained from the Belgian Sciensano (Scientiﬁc Institute of Public Health) (https://www.wiv-isp.be/QML/activities/external_ quality/rapports/_nl/rapports_annee.htm; accessed 18 February 2021) as part of external quality assessment programs. The collection included 323 isolates (195 Gram-negative and 128 Gram-positive) of species represented in the MicroScan database. The remaining 44 isolates, belonging to species that were not present in the MicroScan database, were used to look for anticipated incorrect identiﬁcations.

Table 2. Overall performance of the MicroScan Gram-negative, Gram-positive, and MSF ID panels for species represented in the MicroScan database. N = total number of isolates tested. Details of incorrect identiﬁcations with their probability scores are listed in Table3. * Formerly part of Streptococcus bovis group.

Correct Identification Incorrect Identification Identification Species N High Probability Low Probability High Probability Low Probability Not Possible Score Score Score Score Enterobacterales Escherichia coli 28 24 3 - 1 - Klebsiella pneumoniae 13 13 - - - - Klebsiella oxytoca 3 3 - - - - Enterobacter cloacae complex 15 14 - - 1 - Citrobacter freundii complex 9 9 - - - - Kluyvera ascorbata 1 1 - - - - Salmonella Typhi 10 10 - - - - Salmonella Paratyphi A 10 9 - 1 - - Salmonella Typhimurium 10 10 - - - - Salmonella Choleraesuis 9 8 - - 1 - Shigella species 10 9 - - 1 - Morganella morganii 4 4 - - - - Proteus mirabilis 7 7 - - - - Providencia rettgerii 1 1 - - - - Enterobacterales 130 122 (93.8%) 3 (2.3%) 1 (0.8%) 4 (3.1%) 0 (0%) Non-fermenting Gram-negative organisms and Aeromonas/Vibrio species Pseudomonas aeruginosa 11 11 - - - - Acinetobacter baumannii 9 8 - 1 - - Achromobacter xylosoxidans 10 8 - 1 1 - Burkholderia cepacia 15 13 1 - 1 - Stenotrophomonas maltophilia 15 14 1 - - - Aeromonas species 3 2 - - 1 - Vibrio alginolyticus 2 2 - - - - Non-fermenters 65 58 (89.2%) 2 (3.1%) 2 (3.1%) 3 (4.6%) 0 (0%) Total Gram-negative isolates Total Gram-negative 195 180 (92.3%) 5 (2.6%) 3 (1.5%) 7 (3.6%) - Micrococcaceae Staphylococcus aureus 14 14 - - - - Staphylococcus hominis 3 2 1 - - - Staphylococcus epidermidis 11 10 - 1 - - Staphylococcus lugdunensis 3 3 - - - - Staphylococcus lentus 1 - - 1 - - Staphylococcus haemolyticus 6 3 - 2 1 - Listeria monocytogenes 2 2 - - - - Micrococcus species 4 4 - - - - Total Staphylococcus/Micrococcus 44 38 (86.4%) 1 (2.3%) 4 (9.1%) 1 (2.3%) 0 (0%) Diagnostics 2021, 11, 349 5 of 19

Table 2. Cont.

Correct Identification Incorrect Identification Identification Species N High Probability Low Probability High Probability Low pProbability Not Possible Score Score Score Score Streptococcaceae Enterococcus faecalis 7 7 - - - - Enterococcus faecium 7 4 - 2 - 1 Enterococcus casseliflavus 1 - - 1 - - Streptococcus agalactiae 13 11 1 - 1 - Streptococcus anginosus 8 2 2 3 - 1 Streptococcus pneumoniae 12 9 - 2 1 - Streptococcus gallolyticus * 10 9 1 - - - Streptococcus lutetiensis * 1 - - - 1 - Streptococcus pyogenes 13 13 - - - - Streptococcus dysgalactiae (group C & G) 6 6 - - - - Streptococcus mitis 5 2 3 - - - Streptococcus salivarius 1 1 - - - - Total Streptococcus/Enterococcus species 84 64 (76.2%) 7 (8.3%) 8 (9.5%) 3 (3.6%) 2 (2.4%) Total Gram-positive isolates Total Gram-positive 128 102 (79.7%) 8 (6.3%) 12 (9.4%) 4 (3.9%) 2 (1.6%) Overall total 323 282 (87.3%) 13 (4.0%) 15 (4.6%) 11 (3.4%) 2 (0.6%)

Table 3. Overall performance of the MicroScan Gram-negative and Gram-positive panels for micro-organisms not represented in the MicroScan database. All identifications by the MicroScan system are by definition incorrect because the species are not represented in the database. The expected result for these species is “identification not possible”. N = total number of isolates tested. Overall, * 44 different isolates were tested, five of which on both the Gram-positive as the Gram-negative panel.

Identification Not Low Probability High Probability Species N Possible Identification Identification Gram-positive bacilli (Bacillus & Corynebacterium species) 16 1 8 7 Streptococcus suis 11 - 5 6 Burkholderia species (non-cepacia, non-pseudomallei) 8 2 2 4 Yeast species (on Gram-positive panel) 9 - - 9 Yeast species (on Gram-negative panel) 5 - 5 - Total (species not represented in MicroScan database) 49 * 3 (6.1%) 20 (40.8%) 26 (53.1%)

2.3. Reference Identiﬁcation of Clinical Isolates Identification of clinical isolates was confirmed by MALDI-TOF MS, using Microflex™ device (Bruker Daltonics, Billerica, MA, USA) with MALDI Biotyper® (MBT) 7854 MSP Library. Serotyping of Salmonella was performed with the Salmonella antisera of Pro-lab diagnostics (Richmond Hill, ON, Canada). Optochin disks (Rosco Diagnostica, Taastrup, Denmark) were used to distinguish between Streptococcus pneumoniae and other viridans Streptococcus species, after identification by MALDI-TOF as Streptococcus pneumoniae/mitis/oralis/parapneumoniae.

2.4. Preparation of Suspension Frozen isolates (stored at −80 ◦C) underwent two passages on sheep blood agar. Inoculation from single colonies was conducted using the Prompt™ Inoculation System (Beckman Coulter, Brea, CA, USA), consisting of a wand designed to hold a speciﬁc quantity of bacterial material and a 30 mL Prompt™ Inoculation Water bottle (Beckman Coulter, Brea, CA, USA). As per the manufacturer’s instructions, some slow-growing Streptococcus species were inoculated using the turbidity standard method, in which a 0.5 McFarland solution was prepared in 3 mL of Inoculum Water, of which 100 µL was pipetted into 25 mL of Inoculum Water. Diagnostics 2021, 11, 349 6 of 19

2.5. Inoculation of Panels The bacterial suspension was transferred to the panels using MicroScan Renok (inoculating device by Beckman Coulter, Brea, CA, USA), which delivers 115 µL of broth suspension to each well. Mineral oil was added to the indicated wells (Table1). Panels were incubated in an atmospheric incubator for 16–18 h at 35 ◦C.

2.6. Reading of the Panels Reagents were added and panels were read visually using transmitted light and magnifier with interchangeable white or black backgrounds according to the manufacturer’s instructions for use (IFU). We used a microplate viewer prototype adapted for this purpose by JP Selecta (Barcelona, Spain) (Figure2). The results were recorded on PID3 and NID2 worksheets (supplied by the manufacturer) and transferred to the LabPro software (version 4.43). The LabPro software calculated the “biotype”, a code containing the biochemical profile, and generated the corresponding identification for each biotype with probability scores. Results with high probability scores (≥85%) were considered reliable, while results with low probability scores (<85%) were considered “unconfirmed”, as defined by the IFU. No additional tests for confirmation of low probability identifications were performed in Diagnostics 2021, 11, x 7 of 20 our study. If the biochemical profile did not match any identification in LabPro’s database, the result generated was “very rare biotype”.

Figure 2. Prototype of the microplateFigure 2. Prototype viewer by ofJP theSelecta, microplate used for viewer the visual by JP reading. Selecta, The used background for the visual at the reading. bottom The can back- be changed to white. ground at the bottom can be changed to white.

2.7.2.7. RobustnessRobustness TestingTesting IncorrectIncorrect panel:panel: ToTo anticipateanticipate possiblepossible errorserrors inin GramGram stain,stain, wewe inoculatedinoculated 6464 Gram-Gram- negativenegative isolatesisolates onon Gram-positiveGram-positive panelspanels andand vicevice versaversa (80(80 isolates).isolates). IncorrectIncorrect algorithm:algorithm: ForFor Gram-positiveGram-positive organisms,organisms, thethe LabProLabPro softwaresoftware askedasked toto choosechoose betweenbetween thethe algorithmsalgorithms forfor StreptococcaceaeStreptococcaceae ((StreptococcusStreptococcus speciesspecies andand Enterococ-Enterococ- cuscus speciesspecies (catalase(catalase negative)negative) versusversus MicrococcaceaeMicrococcaceae ((MicrococcusMicrococcusspecies, species,Staphylococcus Staphylococcus species,species, and Listeria monocytogenes monocytogenes (catalase(catalase positive). positive). To To assess assess the the impact impact of errors of errors in mi- in microscopycroscopy or orcatalase catalase results, results, we wecompared compared the theresults results of Gram-positive of Gram-positive bacteria bacteria withwith both boththe Streptococcaceae the Streptococcaceae and Micrococcaceae and Micrococcaceae algorithm. algorithm. Hemolysis:Hemolysis: ForFor thethe Streptococcaceae,Streptococcaceae, visualvisual assessmentassessment ofof beta-hemolysisbeta-hemolysis (complete(complete hemolysis)hemolysis) onon bloodblood agaragar waswas partpart ofof thethe biotype.biotype. InIn manymany LRS,LRS, sheepsheep bloodblood agaragar isis hardhard to come by, hence hemolysis patterns may be less reliable in these settings [11,12]. To assess the impact of hemolysis reading, we compared results for beta-hemolytic streptococci with hemolysis read as positive versus negative. Oxidase: For Gram-negative organisms, the software required the result for oxidase testing. We used OxiSticks™ Oxidase Swabs (Hardy Diagnostics, Santa Maria, CA, USA) for this purpose. For a subset of Stenotrophomonas maltophilia isolates, oxidase testing was repeated with Bactident® oxidase (Merck Millipore, Burlington, MA, USA).

2.8. Comparison between Automated and Visual Reading, Repeatability and Inter-Observer Agreement The panels of 190 isolates were read both by visual and automated reading. To assess the “repeatability” of the panels, 31 isolates were tested in triplicate. Panels were inoculated from a single blood agar plate and processed and read simultaneously and by the same operator. “Inter-observer agreement” of visual reading was assessed for 50 isolates, subsequently read by two operators blinded to each other’s results. For the above comparisons, species agreement was defined as an identical species identification; biotype agreement was defined as an identical biotype, implying identical results for each separate test well.

2.9. Definitions of Correct and Incorrect Identifications

Diagnostics 2021, 11, 349 7 of 19

to come by, hence hemolysis patterns may be less reliable in these settings [11,12]. To assess the impact of hemolysis reading, we compared results for beta-hemolytic streptococci with hemolysis read as positive versus negative. Oxidase: For Gram-negative organisms, the software required the result for oxidase testing. We used OxiSticks™ Oxidase Swabs (Hardy Diagnostics, Santa Maria, CA, USA) for this purpose. For a subset of Stenotrophomonas maltophilia isolates, oxidase testing was repeated with Bactident® oxidase (Merck Millipore, Burlington, MA, USA).

2.8. Comparison between Automated and Visual Reading, Repeatability and Inter-Observer Agreement The panels of 190 isolates were read both by visual and automated reading. To assess the “repeatability” of the panels, 31 isolates were tested in triplicate. Panels were inoculated from a single blood agar plate and processed and read simultaneously and by the same operator. “Inter-observer agreement” of visual reading was assessed for 50 isolates, subsequently read by two operators blinded to each other’s results. For the above comparisons, species agreement was defined as an identical species identification; biotype agreement was defined as an identical biotype, implying identical results for each separate test well.

2.9. Definitions of Correct and Incorrect Identifications Correct identification was defined as concordance between the reference identification method and the identification given by MicroScan software up to the level provided by the MicroScan system (species level for most organisms, serotype level for certain Salmonella serotypes and genus level for Micrococcus species, for instance). We assessed the potential impact of incorrect identifications in the light of diagnostic and therapeutic consequences, as well as epidemiological significance. For species not represented in the database or when panels were incorrectly used (e.g., Gram-positive isolate on Gram-negative panel), the expected “correct” identification by the MicroScan system was “very rare biotype”, indicating the system did not find sufficient agreement with expected biotypes of species available in the database. We will further refer to this result as “no identification possible”.

2.10. Management of Incorrect Identifications For incorrect identifications with high probability scores, MALDI-TOF was repeated. If initial reference identification was confirmed, the MicroScan panel was repeated or included in the repeatability testing panel: if the isolate was correctly identified upon repeat MicroScan testing, we assumed technical error on first testing and accepted the second result. For repeatability testing, if at least two of three repetitions yielded again an incorrect result, the isolate was considered incorrectly identified. For Streptococcus species, all incorrect identifications were repeated using the turbidity standard method instead of the Prompt™ method according to the MicroScan panel IFU.

2.11. Ease of Use Ease of use was assessed by surveying the operators for feedback on each of the components of the system. The readability level of the IFU was assessed using Flesch–Kincaid Grade Levels (https://www.online-utility.org/english/readability_test_and_improve.jsp; accessed 18 February 2021) [13].

3. Results 3.1. Results for Species Included in the MicroScan Database Gram-negative isolates: Gram-negative isolates represented in the MicroScan database were correctly identified with a high probability score in 92.3% of 195 isolates (Table2). An additional 2.6% of isolates were correctly identified but with a low probability score. Enterobacterales were more accurately identified than non-fermenters, with correct identifications in 96.1% and 90.0%, respectively. Table4 shows an overview of the incorrectly Diagnostics 2021, 11, 349 8 of 19

identified isolates. Most errors for Enterobacterales (6/9) were at the species level and had limited clinical relevance. Six out of 15 Stenotrophomonas maltophilia isolates tested oxidase positive with the OxiSticks™; this led to incorrect identifications as other non-fermenters in four isolates. Repeat oxidase testing with Bactident® oxidase tests gave negative results for all isolates; the corrected biotypes all produced correct identifications with high probability scores.

Table 4. Details of incorrectly identified isolates. Potential clinical relevance is also indicated; limited clinical relevance means that the two pathogens are similar in clinical, epidemiological, and infection control aspects. Probability scores are reported by the MicroScan LabPro software with two decimals. A probability score of ≥85.00% is considered as “high probability”. N = total number of isolates tested. * As identified by matrix-assisted laser desorption/ionization time-of-flight mass spectrometry (MALDI-TOF) (and serotyping when necessary).

Nr. of Isolates Incorrect Result Probability Species/Serotype * with Incorrect Clinical Relevance of Error by MicroScan Score Identification/N Gram-negative species Escherichia coli 1/28 Escherichia fergusonii 67.88% Limited Enterobacter cloacae 1/15 Enterobacter amnigenus 58.26% Limited Moderate: pathogens are clinically similar, Salmonella Paratyphi A 1/10 Salmonella Typhi 98.87% but distinction has epidemiological relevance High: pathogens are both clinically Salmonella Choleraesuis 1/9 Salmonella Typhi 57.06% and epidemiologically distinct High: pathogens have different clinical presentations; Shigella species 1/10 Klebsiella ozaenae 72.69% Shigella infection is of high epidemiological importance (outbreak potential) Acinetobacter baumannii 1/9 Acinetobacter lwoffii 92.88% Moderate: pathogens may have different clinical presentation Rhizobactrum radiobacter 49.88% Achromobacter xylosoxidans 2/10 Limited Burkholderia cepacia 94.19% Aeromonas caviae 1/3 Aeromonas hydrophila 61.71% Limited Gram-positive species Staphylococcus epidermidis 1/11 Staphylococcus hyicus 91.14% Limited Staphylococcus lentus 1/1 Staphylococcus sciuri 96.73% Limited Staphylococcus lugdunensis 53.67% Staphylococcus haemolyticus 3/6 Moderate: pathogens may have different clinical presentation Staphylococcus lugdunensis 93.66% Enterococcus raffinosus 95.39% Enterococcus faecium 2/7 Moderate: pathogens have different clinical significance Enterococcus durans 99.68% Limited: both are associated Enterococcus casseliflavus 1/1 Enterococcus gallinarum 59.68% with increased vancomycin resistance Streptococcus mitis/oralis 86.99% High: Streptococcus anginosus is associated with purulent Streptococcus anginosus 3/8 Streptococcus salivarius 88.56% infections; other viridans streptococci are more likely Streptococcus parasanguinis 99.99% contaminants or implicated in infectious endocarditis Rhodococcus equi 99.99% High: Streptococcus pneumoniae is a more common Streptococcus pneumoniae 3/12 Gemella species 51.48% pathogen than Rhodococcus equi or Gemella and Gemella species 95.79% clinical presentation is very different Streptococcus agalactiae 1/13 Streptococcus dysgalactiae 99.99% Limited; except for screening in pregnancy

Gram-positive isolates: Of 128 Gram-positive isolates, 79.7% and 6.3% were correctly identified with high and low probability scores, respectively (Table2). Low probability identifications were more frequent for Streptococcaceae than for Micrococcaceae (Staphy- lococcus, Micrococcus, and Listeria species), i.e., 8.3% of isolates versus 2.6%, respectively. Incorrect identifications were especially common—and considered as highly clinically relevant—for Streptococcus pneumoniae (3/12) and Streptococcus anginosus (4/8) (Figures3 and4). All S. aureus isolates (n = 14) were identified correctly. Diagnostics 2021, 11, 349 9 of 19 Diagnostics 2021, 11, x 10 of 20

Diagnostics 2021, 11, x 10 of 20

FigureFigure 3.3.Streptococcus Streptococcus pneumoniaepneumoniaeclinical clinical isolate isolate incorrectly incorrectly identiﬁed identified as as Gemella Gemella species; species; PNP- PNP-β-βD- - D-Glucuronide test (PGT) was considered negative (any shade of yellow = positive). A positive GlucuronideFigure 3. Streptococcus test (PGT) pneumoniae was considered clinical negative isolate (anyincorrectly shade ofidentified yellow = as positive). Gemella Aspecies; positive PNP- resultβ- result for PGT would have led to correct identification of S. pneumoniae with a probability score of forD-Glucuronide PGT would havetest (PGT) led to was correct considered identiﬁcation negative of S. (any pneumoniae shade ofwith yellow a probability = positive). score A positive of 94.79%. 94.79%. Only indoxyl phosphatase (IDX) test was positive in this panel. Onlyresult indoxyl for PGT phosphatase would have (IDX) led to test correct was identification positive in this of panel. S. pneumoniae with a probability score of 94.79%. Only indoxyl phosphatase (IDX) test was positive in this panel.

Figure 4. Streptococcus anginosus reference isolate incorrectly identified as S. mitis/oralis. Note that both PHO and PGT wells were considered positive when this panel was read; “any shade of Figure 4. Streptococcus anginosus reference isolate incorrectly identified as S. mitis/oralis. Note Figureyellow” 4. leadsStreptococcus to a positive anginosus result according reference isolateto the incorrectlyinstructions identified for use (IFU). as S. mitis/oralis.A negative reading Note that that both PHO and PGT wells were considered positive when this panel was read; “any shade of bothwould PHO have and resulted PGT wells in the were correct considered identification positive of when S. anginosus this panel. Note was also read; that “any Voges–Proskauer shade of yellow” yellow” leads to a positive result according to the instructions for use (IFU). A negative reading leads(VP) totest a was positive judged result negative according because to the “pale instructions pink” is forto be use considered (IFU). A negative negative reading according would to the have would have resulted in the correct identification of S. anginosus. Note also that Voges–Proskauer IFU. A positive result of VP would have led to an identification of S. parasanguinis. resulted(VP) test in was the judged correct negative identification because of S.“pale anginosus pink” .is Note to be also considered that Voges–Proskauer negative according (VP) testto the was judgedIFU. A negativepositive result because of VP “pale would pink” have is to led be to considered an identification negative of accordingS. parasanguinis to the. IFU. A positive result3.2. Isolates of VP would Not Represented have led to in an the identification Database of S. parasanguinis. 3.2. IsolatesTesting Not of Representedbacteria not in represented the Database in the MicroScan database led to incorrect high 3.2. Isolates Not Represented in the Database probability identification in 53.1% of 49 isolates tested, and only for 6.1% of isolates, the Testing of bacteria not represented in the MicroScan database led to incorrect high expectedTesting result of bacteria “identification not represented not possible” in the was MicroScan generated database (Table led4). toThe incorrect yeast species high probability identification in 53.1% of 49 isolates tested, and only for 6.1% of isolates, the probabilitywere consistently identification misidentified in 53.1% by ofthe 49 MicroScan isolates tested, Gram-positive and only forpanel 6.1% as ofeither isolates, Staphylo- the expected result “identification not possible” was generated (Table 4). The yeast species expectedcoccus cohnii result (Micrococcus/ “identification Staphylococcus not possible” algorithm) was generated or Rhodococcus (Table4). The equi yeast (Streptococcus species were al- were consistently misidentified by the MicroScan Gram-positive panel as either Staphylo- consistentlygorithm) (Table misidentified 5). by the MicroScan Gram-positive panel as either Staphylococcus coccus cohnii (Micrococcus/ Staphylococcus algorithm) or Rhodococcus equi (Streptococcus al- cohnii (Micrococcus/Staphylococcus algorithm) or Rhodococcus equi (Streptococcus algorithm) (Tablegorithm)5). (Table 5).

Diagnostics 2021, 11, 349 10 of 19

Table 5. Most frequent incorrect identiﬁcations with high probability score for species not represented in the database or “incorrect” use of the panels (Gram-positive isolates inoculated on Gram-negative panels and vice versa, Streptococcus using Micrococcus/Staphylococcus algorithm and vice versa). N = total number of isolates tested.

Algorithm Used Number of Isolates Species. N Result (for Gram-Positive Panels) with this Result Species not represented in the database Micrococcus 9 Staphylococcus cohnii Yeast species 9 Streptococcus 9 Rhodococcus equi Micrococcus 6 1 Micrococcus species Corynebacterium species Streptococcus 6 1 Rhodococcus equi Bacillus species Micrococcus 10 4 Staphylococcus auricularis Gram-positive species on Gram-negative panel Streptococcus dysgalactiae 2 2 Providencia rustigiannii Streptococcus suis 10 9 Providencia rustigiannii Staphylococcus lentus 1 1 Providencia rustigiannii Streptococcus agalactiae 5 1 Providencia rustigiannii Gram-negative species on Gram-positive panel Escherichia coli/paracoli Streptococcus 10 3 Rhodococcus equi Micrococcus 3 Staphylococcus cohnii Acinetobacter baumannii Micrococcus4 1 Staphylococcus auricularis Streptococcus 1 Rhodococcus equi Stenotrophomonas maltophilia Streptococcus 4 2 Rhodococcus equi Burkholderia cepacia Streptococcus 4 3 Rhodococcus equi Burkholderia thailandensis Streptococcus 4 1 Rhodococcus equi Burkholderia ubonensis Streptococcus 1 1 Rhodococcus equi Gram-positive species using incorrect algorithm on Gram-positive panel Staphylococcus aureus Streptococcus 14 3 Rhodococcus equi Micrococcus species Streptococcus 4 2 Rhodococcus equi Staphylococcus lentus Streptococcus 1 1 Rhodococcus equi Streptococcus gallolyticus Micrococcus 10 9 Listeria monocytogenes 9 Micrococcus species Streptococcus pneumoniae Micrococcus 12 2 Staphylococcus auricularis Streptococcus pyogenes Micrococcus 13 2 Staphylococcus auricularis Streptococcus agalactiae Micrococcus 13 1 Staphylococcus auricularis Streptococcus anginosus Micrococcus 8 1 Staphylococcus auricularis Streptococcus mitis/oralis Micrococcus 5 1 Staphylococcus auricularis

3.3. Robustness Testing Incorrect panel: Inoculating a Gram-positive isolate on the Gram-negative panel or vice versa led to an incorrect identification with a high probability score in 32.6% of 144 isolates tested (Table6). This risk was higher for isolates wrongly inoculated on the Gram- negative panel (43.8%) compared to the Gram-positive panel (18.8%). A total of 38.2% isolates inoculated on the wrong panel generated the expected result “no identification possible”. Rhodococcus equi was a common incorrect identification with the Gram-positive panel Diagnostics 2021, 11, 349 11 of 19

(n = 27); when the Gram-negative panel was used for Gram-positive isolates, identification of Providencia rustigiannii recurred (n = 13) (Table5).

Table 6. Results when isolates were tested on an incorrect panel or using an incorrect algorithm, i.e., inoculation of Gram- negative species on Gram-positive panels or vice versa, or of Gram-positive species incorrectly assigned as “Streptococcus” or “Micrococcus” based on faulty catalase or microscopy results. The expected result for this type of incorrect panel use is “identification not possible”. For Gram-negative species inoculated on Gram-positive panel, we assumed Micrococcus interpretation for the results in this table, as most of them would be catalase positive. Note that all results other than “identification not possible” are wrong; detailed results for each species can be found in Supplementary Table S3. Incorrect identifications with high probability are listed in Table5. N = total number of isolates tested.

Number Number Low Number High Species. N Panel/Algorithm Used “Identification not Probability Probability Possible” (%) Identification (%) Identification (%) Gram-negative species tested on Gram-positive panels and vice versa Enterobacterales 41 27 9 5 Gram-positive Non-fermenters 23 10 6 7 Total Gram-negatives on 64 37 (57.8%) 15 (23.4%) 12 (18.8%) Gram-positive panel Staphylococcus species 16 1 11 4 Streptococcus species 53Gram-negative 15 12 26 Gram-positive bacilli 11 2 4 5 (Bacillus & Corynebacterium species) Total Gram-positive on 80 18 (22.5%) 27 (33.8%) 35 (43.8%) Gram-negative panel Staphylococcus/Micrococcus species using Streptococcus species algorithm and vice versa Staphylococcus/Micrococcus species 60 Streptococcus algorithm 35 11 14 and Gram-positive rods Streptococcus/Enterococcus species 95 Micrococcus algorithm 27 24 44 Total Gram-positive 155 62 (40.0%) 35 (22.6%) 58 (37.4%) with incorrect algorithm Total tested on incorrect 299 117 (39.1%) 77 (25.8%) 105 (35.1%) panel/incorrect algorithm

Incorrect algorithm: When Staphylococcus species were interpreted erroneously using the Streptococcus algorithm, this led to incorrect high probability identification for 23.3% of isolates tested (Table6). Vice versa, interpreting the results of Streptococcus species with the Micrococcus algorithm led to incorrect high probability identifications in 46.3% of isolates tested. Hemolysis: Impact of hemolysis on identification of beta-hemolytic Streptococcus species (S. agalactiae, S. pyogenes, S. dysgalactiae) was limited; beta-hemolysis results led to different identifications in only 2 of 32 isolates (one S. agalactiae isolate and one S. pyogenes isolate).

3.4. Comparison between Automated and Visual Reading, Repeatability and Inter-Observer Agreement Automated versus visual reading: Visual reading was slightly more accurate than automated reading (90.6% correct versus 87.5% correct), except for Streptococcus species. For species included in the MicroScan database, species agreement between visual and automated reading was 91.9%, and biotype agreement 58.1% (Supplementary Table S4). Repeatability was better for automated reading than visual reading (89.7% versus 88.7% for species agreement; 87.5% versus 79.0% for biotype agreement). Especially for Staphylococcus species and Enterobacterales, repeatability was excellent with 100% biotype agreement for the automated reading. The lowest repeatability was seen for Streptococcus species (70.8% biotype agreement for automated reading versus 73.1% for visual reading). Diagnostics 2021, 11, 349 12 of 19

Inter-observer agreement: On 50 isolates read by two operators, species and biotype agreements were 98.0% and 66.0%, respectively. VP (Voges-Proskauer test) was the test most often read differently by different observers (inter-observer and visual-automated) (Supplementary Table S5).

3.5. Ease of Use Instructions for use: The instructions for interpretation (e.g., color) of positive or negative results were often equivocal (Table7). Flesch–Kincaid Grade Levels of the Gram-negative, Gram-positive, and MSF panel IFU were 9, 11, and 10, respectively. Flesch–Kincaid Grade Levels refer to US grade levels (i.e., years of schooling) necessary to understand the text. Inoculation and reading: Both the Prompt™ and the RENOK system (Beckman Coul- ter, Brea, CA, USA) were considered user-friendly and time-efficient, especially compared to other inoculation methods using 0.5 McFarland standards and pipettes. Reading of the panels was considered rather difficult. Visual assessment of positivity of wells was frequently doubtful. A harmonized workflow for reading multiple panels was hindered by the re-incubation requirements; reagents could only be added to the panels when certain combinations of tests were positive—if this was not the case, re-incubation for another 24 h was needed. Furthermore, waiting times between adding reagents and visual reading were variable, ranging from immediate reaction to 20 min. Automated reading was more user- friendly but a visual check of the required positive wells was still necessary. Re-incubation of panels was needed for 15.3% of panels. Obtaining identifications: The worksheets provided with the PID3 and NID2 panels had some tests in different positions compared to the real-life panel or the software picture (Figure5). To obtain identification results from the biotype, the LabPro software and a freely available online Biotype Lookup Tool could be used. Customized panels: Because Gram stain results were still necessary to interpret the results of the inoculated panels, advantages in the usability of the customized MSF ID Diagnostics 2021, 11, x 14 of 20 panels were mainly logistic (simplicity, stock management). On the other hand, care should be taken when reading the panels to avoid reading the wrong side.

Figure 5. WorksheetWorksheet for the NID2 panel. Tests Tests marked in orange are in a different order on the panel than they are on the worksheet (see Figure 1 1).).

Table 7. Ease-of-use assessment with a focus on use in low resource settings of different aspects of the MicroScan identification panels. Abbreviations: IFU = instructions for use; PEP = peptidase; PYR = pyrrolidonyl-b-naphtylamide; VP = Voges–Proskauer test

Favorable Observations Unfavorable Observations Storage of panels and reagents VP2 reagent must be reconstituted with 95% ethanol (not Long shelf life of the panels and of most reagents provided); after reconstitution, vial is stable for only 2 weeks Many reagents (nitrate, VP, ferric chloride, Kovac’s) require Panels and some reagents can be stored at room temperature storage at 2–8 °C Instructions for use, labeling, and packaging Panels are packaged separately with desiccant, (StripPax®, Reconstitution of VP2 reagent not explained in IFU, only on box of Multisorb Technologies, New York, NY, USA) reagent Sturdy outer packaging IFU are not delivered in the box with the products; they are available online but not easy to find Flesch Kincaid Grade level of IFU was 9.14 for the Gram-negative IFU, 11.06 for the Gram-positive IFU and 10.30 for the MSF panel IFU; these levels point to a high level of education needed to IFU were delivered at request by the distributor comprehend the text Instructions in IFU for interpretation of negative/positive wells are often ambiguous. e.g., yellow to orange color for carbohydrate tests must be considered positive, whereas orange to red must be considered negative Use of reagent names is inconsistent between IFU and reagent bottles (e.g., 5% alpha naphtol in IFU, VP2 on bottle) Reagent bottles are labeled with the test name: easy to add right The PYR test reagent has “PEP” on the bottle and not “PYR” reagent to right test well Limited space for writing patient identification and sample number on the panel Inoculation and reading Most panels could be inoculated using the Prompt™ system

Diagnostics 2021, 11, 349 13 of 19

Table 7. Ease-of-use assessment with a focus on use in low resource settings of different aspects of the MicroScan identiﬁcation panels. Abbreviations: IFU = instructions for use; PEP = peptidase; PYR = pyrrolidonyl-b-naphtylamide; VP = Voges–Proskauer test.

Favorable Observations Unfavorable Observations Storage of panels and reagents Long shelf life of the panels and of most reagents VP2 reagent must be reconstituted with 95% ethanol (not provided); after reconstitution, vial is stable for only 2 weeks Panels and some reagents can be stored at room temperature Many reagents (nitrate, VP, ferric chloride, Kovac’s) require storage at 2–8 ◦C Instructions for use, labeling, and packaging Panels are packaged separately with desiccant, (StripPax®, Multisorb Technologies, New York, NY, USA) Reconstitution of VP2 reagent not explained in IFU, only on box of reagent Sturdy outer packaging IFU are not delivered in the box with the products; they are available online but not easy to find IFU were delivered at request by the distributor Flesch Kincaid Grade level of IFU was 9.14 for the Gram-negative IFU, 11.06 for the Gram-positive IFU and 10.30 for the MSF panel IFU; these levels point to a high level of education needed to comprehend the text Instructions in IFU for interpretation of negative/positive wells are often ambiguous.e.g., yellow to orange color for carbohydrate tests must be considered positive, whereas orange to red must be considered negative Use of reagent names is inconsistent between IFU and reagent bottles (e.g., 5% alpha naphtol in IFU, VP2 on bottle) Reagent bottles are labeled with the test name: easy to add right reagent to right test well The PYR test reagent has “PEP” on the bottle and not “PYR” Limited space for writing patient identification and sample number on the panel Inoculation and reading Most panels could be inoculated using the Prompt™ system Prompt™ is not advised for use in slow-growing Streptococcus species; we repeated testing of 14 Streptococcus isolates with turbidity standard method due to incorrect results with Prompt™; 50% of these had a correct result with turbidity standard method Prompt ™ was easy to use and time-efficient Large amounts of plastic waste associated with the RENOK system RENOK system was easy to use and time-efficient Difficulties in disposing of waste without spillage Difficulties in incubation of panels without spillage Favorable observations Unfavorable observations IFU recognizes that interpretation of test results requires trained clinical staff Visual reading of panels was considered difficult as positivity of test wells was often doubtful Automated reading was more user-friendly as it removed subjectivity and autoSCAN-4 does not indicate automatically that panels must be re-incubated eliminated clerical errors autoSCAN-4 reader dimensions are very large: 48.5 cm × 24 cm × 58 cm Prototype microplate viewer (JP Selecta) with magnifier and interchangeable Both white and black background needed for reading of wells white/black background considerably facilitated the reading Reagents can be added only after visual assessment of certain wells indicating the need for re-incubation; panels must be re-incubated if specific combinations of wells are negative—there are 6 of these combinations that consist each of 1 to 3 wells unevenly distributed on the panels Different reading times for different reagents Re-incubation was required for 15.3% of panels in our study; 13.3% (27/203) of Gram-negative isolates (mostly non-fermenters) and 17.7% (29/164) of Gram-positive isolates (mainly Streptococcus isolates). Diagnostics 2021, 11, 349 14 of 19

Table 7. Cont.

Favorable Observations Unfavorable Observations Software and Biotype Lookup Tool LabPro software and reader easy to use Beckman Coulter worksheets not delivered with the panels; available online but hard to find Calibration and QC of reader are alerted by the reader automatically and do not Test position on worksheet different from test position on the panel (for Gram-negative panel) (Figure5). require user input Biotype Lookup Tools freely available online Many of the confirmatory tests proposed by the software are unlikely to be easily available in LRS Examples of recommended confirmatory tests: 1. E. coli: cellobiose fermentation, growth in potassium cyanide and Christensen urea agar Both the online Biotype Lookup Tool and LabPro software propose additional tests 2. Salmonella Typhi: trehalose fermentation, serotyping to confirm low-probability identification 3. S. pneumoniae: leucine aminopeptidase and bile solubility 4. S. anginosus: beta-D-acetylgalactosaminidase, alpha-glucosidase, beta-glucosidase, hyaluronidase, neuraminidase Diagnostics 2021, 11, 349 15 of 19

3.6. Stability The shelf life of the panels was within the limits predefined by the Mini-lab target product profile (minimally 12 months); storage conditions for the panels (2–25 ◦C) were not entirely within the Mini-Lab specifications (2–40 ◦C). Packaging of individual panels was of good quality, providing air-sealed individual aluminum-plastic pouches with a humidity indicator. Other components were provided within sturdy cardboard boxes. The shelf life of the VP2 reagent (Voges Proskauer test) was too short; it could be used for just two weeks after reconstitution of the reagent.

4. Discussion 4.1. Performance of the MicroScan Panels Our results showed excellent performance of the Gram-negative panel, with 94.9% of isolates correctly identiﬁed to the species level, which was in line with several previous studies evaluating the MicroScan panels [14–16]. The performance of MicroScan for Gram- negatives was similar or better than this of API® (95.2%), Phoenix™ system from Becton– Dickinson (Franklin Lakes, NJ, USA) (92.5%), or Vitek® 2 from bioMérieux (Marcy-l’Etoile, France) (88.6%) as reported by other studies [17,18]. The Gram-positive panel had a lower performance with 85.9% of tested isolates correctly identified up to species level. This was again in line with other published studies for Staphylococcus and Enterococcus species. There were no previous publications that reported the accuracy of MicroScan panels for Streptococcus species. Performance of MicroScan for Gram-positive isolates was lower than the reported performance of API® (92.6%), Phoenix™ (93.6% accuracy), or Vitek® 2 (90.2%) [17,19]. All Staphylococcus aureus isolates were correctly identified in our study, but Streptococcus species were identified less accurately, generating clinically relevant misidentifications for Streptococcus pneumoniae and, to a lesser extent, Streptococcus anginosus. Overall, 7.4% of the isolates tested would need additional testing; this was needed more often for Gram-positive than Gram-negative isolates. Compared to conventional phenotypic testing as it is still conducted in most LRS, MicroScan performed considerably better—a study in Nigeria on 145 Enterobacterales isolates, comparing MALDI-TOF to conventional testing, found only 57.2% accuracy on genus level and 33.1% on species level [20].

4.2. Testing Isolates Not Represented in the Database, Robustness Testing When inoculated on the MicroScan panels, isolates not represented in the database were misidentified with high probability scores in half of the cases, and only in a minority of them (6.1%), the user was alerted about the impossibility of identification. Likewise, testing on inappropriate panels (Gram-positive isolates on the Gram-negative panel and vice versa) or using an incorrect algorithm regularly led to incorrect identification with high probabilities instead of the expected “no identification possible” result. By contrast, errors in interpretation of beta-hemolysis did not impact the identification of hemolytic Streptococcus species. The experiences of misidentifications at robustness testing highlight the importance of accurate performance and interpretation of orienting tests such as Gram stain or “spot tests” such as catalase and oxidase [21,22]. This poses challenges in LRS, as laboratory staff is frequently not familiar with clinical bacteriology [5] and the Gram stain is error-prone in inexperienced hands [23–27]. Apart from training, supervision, and guidance by the laboratory information support system, careful selection and quality control of orienting tests is needed, as was shown by the present experience with oxidase brands. Based on the present study (Table5), identification results such as Rhodococcus equi and Providencia rustigiannii (both very rare in clinical bacteriology) should raise alerts to incorrect identifications.

4.3. Comparison between Automated and Visual Reading, Repeatability and Inter-Observer Agreement Visual reading was slightly more accurate than reading by the autoSCAN-4 reader, except for Streptococcus species. The species agreement of 87.5%, was, however, lower than Diagnostics 2021, 11, 349 16 of 19

the 95% reported for the previous version of the reader (autoSCAN-3) [28]. No studies reporting agreement between visual reading and autoSCAN-4 reader have been published, although some studies showed excellent accuracy of autoSCAN-4 for Gram-negative bacteria [29,30]. Agreement between repetitions was higher with automated reading, which was not unexpected given the subjectivity in the reading of certain wells. Repeatability was poor for Streptococcus species.

4.4. Adaptation to LRS: Stability, Ease-of-Use The temperature stability of the MicroScan panels, currently assured up to 25 ◦C, does not fulfill the requirements for tropical settings because cool storage (<30 ◦C) is not always feasible [31]. The shelf life of the reconstituted VP2 is too short, particularly for laboratories processing small amounts of samples. In addition, the 95% ethanol needed for reconstitution is often not easy to find in LRS. As to ease of use, strong points are the MicroScan inoculation system. On the downside, visual reading of panels was considered complicated, poorly amenable to a swift work-flow, and error-prone. In the absence of the automated reader, the dedicated prototype microplate viewer (allowing quick change from white to the black background) considerably facilitated the reading process. The customized panels provided all tests on one panel; however, Gram stain results are still needed for reading and interpretation. Added value was therefore considered limited. The Flesch–Kincaid Grade Level scores of the IFU indicated that a high level of schooling is required to understand the IFU. For patient leaflets in LRS, Flesch–Kincaid levels below 6 are desirable [13,32]. Although the language used in professional documents may be more complex, it should be reminded that the IFU will often not be available in the user’s native language, therefore low complexity is still preferable. To obtain identification results, the user must purchase the LabPro software or use the (freely available) online Biotype Lookup Tool. For results with low probability scores, confirmatory tests are proposed by the software; however, many of these tests are unlikely to be available in LRS (Table7).

4.5. Recommendations for Use and Further Development of the MicroScan System Within the Mini-Lab project, the current limitations of the panels and IFU will be tackled by training, adapted laboratory procedures, bench aids, and the microplate viewer for visual reading. The Mini-lab project will further use the present results to create an expert decision support system that will anticipate potential errors such as those related to Gram stain interpretation. These expert rules will be made accessible outside MSF. In addition, we encourage the manufacturer to mitigate to the best possible extent the risks described above. To build in an extra fail-safe for Gram stain errors, the development of control wells for Gram-positive, Gram-negative, and yeast growth (consisting of vancomycin, colistin, and amphotericin B, respectively) has been proposed to the manufacturer. We further suggest simplifying the IFU and harmonize it with packaging. Bench aids and improved worksheets could be included with the product. Accuracy can further be improved over time by adding more LRS pathogens and contaminants to the database. Lastly, extended shelf-life testing and stability testing in tropical environments are necessary to assure product quality in LRS.

4.6. Strengths and Limitations of the Study To our knowledge, this is the ﬁrst study evaluating the MicroScan system on clinical isolates from LRS and the largest evaluating both Gram-positive and Gram-negative organisms, among which typical LRS pathogens [33,34]. Furthermore, we assessed robustness and ease of use. Robustness testing of in vitro diagnostics is encouraged to anticipate end users’ challenges, particularly in LRS [35,36]. To our knowledge, other commercial phenotypic identiﬁcation systems have not yet been tested for robustness. Because we did not dispose of Biosafety Level-3 (BSL-3) facilities, we could not test pathogens such as B. pseudomallei and Brucella species. Laboratory technicians were not Diagnostics 2021, 11, 349 17 of 19

blinded to the isolate’s identiﬁcation when interpreting results; however, we do not believe this led to bias in the interpretation of test wells because they were unaware of which tests were supposed to be positive for which species. Lastly, this evaluation was performed in a reference laboratory in optimal conditions, by experienced laboratory technicians, and results can therefore not be extrapolated to a ﬁeld setting. Performance and usability should be evaluated prospectively in LRS settings by the intended end user.

5. Conclusions Challenged with clinical isolates from LRS, MicroScan Dried Overnight Identiﬁcation panels had excellent performance for Gram-negative organisms. For important Gram- positive pathogens, accuracy ranged from excellent (S. aureus) to fair (S. pneumoniae and S. anginosus). The study further identiﬁed potential improvements in stability, robustness, and ease of use to assure adaptation of the MicroScan system to the constraints of LRS, particularly outside the MSF Mini-lab setting.

Supplementary Materials: The following are available online at https://www.mdpi.com/2075-4 418/11/2/349/s1, Table S1: Lot numbers of tested panels, Table S2: Geographical origin of the isolates, Table S3: Detailed results for incorrect use of the panels, Table S4: Agreement between visual and automated (autoSCAN-4) reading, Table S5: Results for inter-observer agreement between two independent readers who read the panels visually. Author Contributions: Conceptualization, S.O., A.N., J.-B.R., O.V., L.H., and J.J.; data curation, S.O.; formal analysis, S.O.; funding acquisition, J.-B.R. and J.J.; methodology, S.O., A.N., L.H., and J.J.; project administration, S.O., A.N., and J.-B.R.; supervision, L.H. and J.J.; visualization, S.O.; writing— original draft preparation, S.O., L.H., and J.J.; writing—review and editing, A.N., J.-B.R., and O.V. All authors have read and agreed to the published version of the manuscript. Funding: This study was funded by Médecins Sans Frontières (Mini-lab project). Institutional Review Board Statement: Because the study did not involve research on human or animal subjects, data, or material, ethical clearance was waived. Bacterial isolates were obtained from ITM travel clinic and microbiological surveillance studies (surveillance of antimicrobial resistance among consecutive blood culture isolates in tropical settings, Institutional Review Board number 613/08, Ethical Committee Antwerp University Hospital 8/20/96). Informed Consent Statement: Not applicable (study did not involve human subjects). Data Availability Statement: The database for this manuscript will be made open access. Access requests for ITM research data can be made to ITM’s central point for research data access by means of submitting the completed Data Access Request Form. These requests will be reviewed for approval by ITMs Data Access Committee (https://www.itg.be/E/data-sharing-open-access). Acknowledgments: We thank Kaat Eggermont, Tine Vermoesen, and Marta Oller Sala for the laboratory work involved in this study, and the diagnostic laboratory of the ITM travel clinic, and our partners of the surveillance studies for providing the clinical isolates. Furthermore, we are indebted to Birgit De Smet for her valuable feedback on the use of the panels in the field. Last but not least, we would like to thank the Mini-lab team and the Mini-lab scientific committee for their continued and valued scientific and practical input. Conflicts of Interest: The authors declare no conflict of interest. The manufacturer of the panels had no role in the design of the study, in the collection, analyses, or interpretation of data, in the writing of the manuscript, or in the decision to publish the results.

References 1. Ombelet, S.; Ronat, J.B.; Walsh, T.; Yansouni, C.P.; Cox, J.; Vlieghe, E.; Martiny, D.; Semret, M.; Vandenberg, O.; Jacobs, J.; et al. Clinical bacteriology in low-resource settings: Today’s solutions. Lancet Infect. Dis. 2018, 18, e248–e258. [CrossRef] 2. Petti, C.A.; Polage, C.R.; Quinn, T.C.; Ronald, A.R.; Sande, M.A. Laboratory Medicine in Africa: A Barrier to Effective Health Care. Clin. Infect. Dis. 2006, 42, 377–382. [CrossRef][PubMed] 3. The European Committee on Antimicrobial Susceptibility Testing. Breakpoint Tables for Interpretation of MICs and Zone Diameters. Version 10.0. 2020. Available online: http://www.eucast.org (accessed on 18 February 2021). Diagnostics 2021, 11, 349 18 of 19

4. CLSI. M100: Performance Standards for Antimicrobial Susceptibility Testing, 30th ed.; Clinical and Laboratory Standards Institute: Wayne, NJ, USA, 2020. 5. Jacobs, J.; Hardy, L.; Semret, M.; Lunguya, O.; Phe, T.; Affolabi, D.; Yansouni, C.; Vandenberg, O. Diagnostic Bacteriology in District Hospitals in Sub-Saharan Africa: At the Forefront of the Containment of Antimicrobial Resistance. Front. Med. 2019, 6. [CrossRef][PubMed] 6. Franco-Duarte, R.; Cernˇ áková, L.; Kadam, S.; Kaushik, K.S.; Salehi, B.; Bevilacqua, A.; Corbo, M.R.; Antolak, H.; Dybka-St˛epień, K.; Leszczewicz, M.; et al. Advances in chemical and biological methods to identify microorganisms—From past to present. Microorganisms 2019, 7, 130. [CrossRef] 7. Leber, A.L. Clinical Microbiology Procedures Handbook; ASM Press: Washington, DC, USA, 2016. 8. Jorgensen, J.H. Manual of Clinical Microbiology, 11th ed.; Jorgensen, J.H., Carroll, K.C., Funke, G., Pfaller, M.A., Eds.; ASM Press: Washington, DC, USA, 2015; ISBN 9781555817374. 9. Sandle, T. Microbial identification. In Pharmaceutical Microbiology; Elsevier Inc.: Amsterdam, The Netherlands, 2016. 10. Natale, A.; Ronat, J.-B.; Mazoyer, A.; Rochard, A.; Boillot, B.; Hubert, J.; Baillet, B.; Ducasse, M.; Mantelet, F.; Oueslati, S.; et al. The Mini-Lab: Accessible clinical bacteriology for low-resource settings. Lancet Microbe 2020, 1, 56–58. [CrossRef] 11. Russell, F.M.; Biribo, S.S.N.; Selvaraj, G.; Oppedisano, F.; Warren, S.; Seduadua, A.; Mulholland, E.K.; Carapetis, J.R. As a bacterial culture medium, citrated sheep blood agar is a practical alternative to citrated human blood agar in laboratories of developing countries. J. Clin. Microbiol. 2006, 44, 3346–3351. [CrossRef] 12. Zomorodian, K.; Javad, M.; Safaei, A.; Bazargani, A. Analysis of beta-hemolysis in human blood agars by Streptococcus pyogenes Analysis of beta-hemolysis in human blood agars by Streptococcus pyogenes. J. Microbiol. Methods 2011, 85, 233–234. [CrossRef] [PubMed] 13. WHO. Technical Guidance Series (TGS) for WHO Prequalification—Diagnostic Assessment Guidance on Test Guidance on Test Method Validation for In Vitro Diagnostic Medical Devices TGS–4; Tech. Guid. Ser. WHO Prequalification—Diagnostic Assess; WHO: Geneva, Switzerland, 2017; pp. 1–23. Available online: http://apps.who.int/bookorders.%0Ahttp://apps.who.int/iris/bitstream/10665/ 258971/1/WHO-EMP-RHT-PQT-TGS4-2017.04-eng.pdf?ua=1 (accessed on 18 February 2021). 14. Jin, W.Y.; Jang, S.J.; Lee, M.J.; Park, G.; Kim, M.J.; Kook, J.K.; Kim, D.M.; Moon, D.S.; Park, Y.J. Evaluation of VITEK 2, MicroScan, and Phoenix for identification of clinical isolates and reference strains. Diagn. Microbiol. Infect. Dis. 2011, 70, 442–447. [CrossRef] 15. Rhoads, S.; Marinelli, L.; Imperatrice, C.A.; Nachamkin, I. Comparison of MicroScan WalkAway system and Vitek system for identification of gram-negative bacteria. J. Clin. Microbiol. 1995, 33, 3044–3046. [CrossRef] 16. Snyder, J.W.; Munier, G.K.; Johnson, C.L. Direct comparison of the BD phoenix system with the MicroScan WalkAway system for identification and antimicrobial susceptibility testing of Enterobacteriaceae and nonfermentative gram-negative organisms. J. Clin. Microbiol. 2008, 46, 2327–2333. [CrossRef][PubMed] 17. Chatzigeorgiou, K.S.; Sergentanis, T.N.; Tsiodras, S.; Hamodrakas, S.J.; Bagos, P.G. Phoenix 100 versus Vitek 2 in the identification of gram-positive and gram-negative bacteria: A comprehensive meta-analysis. J. Clin. Microbiol. 2011, 49, 3284–3291. [CrossRef] 18. O’Hara, C.M. Manual and Automated Instrumentation for Identication of Enterobacteriaceae and Other Aerobic Gram-Negative Bacilli. Clin Microbiol Rev 2005, 18, 147–162. [CrossRef][PubMed] 19. Von Baum, H.; Klemme, F.R.; Geiss, H.K.; Sonntag, H.G. Comparative evaluation of a commercial system for identification of gram-positive cocci. Eur. J. Clin. Microbiol. Infect. Dis. 1998, 17, 849–852. [CrossRef][PubMed] 20. Jesumirhewe, C.; Ogunlowo, P.O.; Olley, M.; Springer, B.; Allerberger, F.; Ruppitsch, W. Accuracy of conventional identification methods used for Enterobacteriaceae isolates in three Nigerian hospitals. PeerJ 2016, 2016, 1–12. [CrossRef][PubMed] 21. Baron, E.J. Rapid Identification of Bacteria and Yeast: Summary of a National Committee for Clinical Laboratory Standards Proposed Guideline. Clin. Infect. Dis. 2001, 33, 220–225. [CrossRef] 22. Baron, E.J.; York, M.K.; Ferraro, M.J.; Rex, J.H.; Body, B.A.; Forbes, B.A.; Poole, F.M.; Sahm, D.F.; Tenover, F.C.; Turnidge, J.D.; et al. M35-A2 Abbreviated Identification of Bacteria and, 2nd ed.; CLSI: Wayne, NJ, USA, 2008; ISBN 1562386816. 23. Rand, K.H.; Tillan, M. Errors in interpretation of gram stains from positive blood cultures. Am. J. Clin. Pathol. 2006, 126, 686–690. [CrossRef] 24. Samuel, L.P.; Balada-Llasat, J.M.; Harrington, A.; Cavagnolo, R. Multicenter assessment of gram stain error rates. J. Clin. Microbiol. 2016, 54, 1442–1447. [CrossRef] 25. Munson, E.; Block, T.; Basile, J.; Hryciuk, J.E.; Schell, R.F. Mechanisms to assess Gram stain interpretation proficiency of technologists at satellite laboratories. J. Clin. Microbiol. 2007, 45, 3754–3758. [CrossRef] 26. Chandler, L. Challenges in Clinical Microbiology Testing, 1st ed.; Elsevier Inc.: Amsterdam, The Netherlands, 2013; ISBN 9780124157835. 27. Yuan, S.; Astion, M.L.; Schapiro, J.; Limaye, A.P. Clinical impact associated with corrected results in clinical microbiology testing. J. Clin. Microbiol. 2005, 43, 2188–2193. [CrossRef] 28. Ellner, P.D.; Myers, D.A. Preliminary evaluation of the autoSCAN-3, an instrument for automated reading and interpretation of microdilution trays: Identification of aerobic gram-negative bacilli. J. Clin. Microbiol. 1981, 14, 326–328. [CrossRef] 29. Rhoden, D.L.; Smith, P.B.; Baker, C.N.; Schable, B. autoSCAN-4 system for identification of gram-negative bacilli. J. Clin. Microbiol. 1985, 22, 915–918. [CrossRef] 30. Gavini, F.; Husson, M.O.; Izard, D.; Bernigaud, A.; Quiviger, B. Evaluation of Autoscan-4 for identification of members of the family Enterobacteriaceae. J. Clin. Microbiol. 1988, 26, 1586–1588. [CrossRef][PubMed] Diagnostics 2021, 11, 349 19 of 19

31. World Health Organization—R&D Blueprint. COVID-19 Target Product Profiles for Priority Diagnostics to Support Response to the COVID-19 Pandemic v.0.1; WHO: Geneva, Switzerland, 2020. 32. Gillet, P.; Maltha, J.; Hermans, V.; Ravinetto, R.; Bruggeman, C.; Jacobs, J. Malaria rapid diagnostic kits: Quality of packaging, design and labelling of boxes and components and readability and accuracy of information inserts. Malar. J. 2011, 10, 39. [CrossRef] [PubMed] 33. Reddy, E.A.; Shaw, A.V.; Crump, J.A. Community-acquired bloodstream infections in Africa: A systematic review and meta- analysis. Lancet Infect. Dis. 2010, 10, 417–432. [CrossRef] 34. Deen, J.; von Seidlein, L.; Andersen, F.; Elle, N.; White, N.J.; Lubell, Y. Community-acquired bacterial bloodstream infections in developing countries in south and southeast Asia: A systematic review. Lancet Infect. Dis. 2012, 12, 480–487. [CrossRef] 35. WHO. Technical Specification Series. 2020. Available online: https://www.who.int/diagnostics_laboratory/guidance/technical- specifications-series/en/ (accessed on 18 February 2021). 36. Gerth-Guyette, E.; Malacad, C.C.; Demonteverde, M.P.; Faulx, D.; Lochhead, M.J.; Lupisan, S.P.; Leader, B.T.; Tallo, V.L. Understanding user requirements to improve adoption of influenza diagnostics in clinical care within Metro Manila. Health Sci. Rep. 2018, 1, 1–8. [CrossRef][PubMed]