4 Factors That Affect Research Reproducibility Improve Reproducibility in Your Lab by Focusing on These Key Sources of Variability WHITE PAPER Getting on with Discovery The fiery debate over reproducibility in science 4 Factors That Affect has burned strong over the past several years, and the flames don’t show any signs of dying Research Reproducibility 1. Reagents down just yet. No matter how scientists view 2. Equipment reproducibility in their respective fields — from 3. Personnel a full-blown crisis to a minor issue that doesn’t 4. Methods impact the credibility of published findings — the significance of reproducible experimental design is undeniable. Factor #1: Reagents All fields of science, immunology included, were built on a foundation of replicating Antibodies and Cells experiments and studying breakthroughs Verifying the quality of reagents used in research for expanding the knowledge base, inspiring can prevent wasted time, wasted resources and more experimentation, and ideally, leading unverifiable results. Unvalidated reagents can to additional breakthroughs. Without snowball down the research chain, leading to reproducibility, the flywheel of experimentation retracted publications and failures in pre-clinical and discovery can’t gain momentum. and clinical trials. Front-end validation, while time-consuming, is an important step that many According to a Nature survey of 1,500 scientists, researchers choose to skip without thinking more than 70% have tried and failed to through the potential consequences. reproduce the results of another scientist’s experiment, and perhaps more shocking and Antibody quality control starts with procurement. concerning, over 50% have failed to reproduce Look for a vendor that offers detailed product their own experiments.1 documentation, has a substantial list of research publications using their products and is An Answer to the Problem? comfortable providing customer references. Widespread reproducibility mandates have been debated and undoubtedly would The vendor should be able to provide test results be costly, time-consuming and difficult to to show that the antibody binds to its target and implement and enforce across the board. not to related proteins. This negative control is While we await an industry-wide standard, if as important as a positive control since cross- one ever comes, there are many things we can reactivity is possible even with monoclonal take responsibility for in our labs and research antibodies. practices to improve reproducibility. Check every product’s Certificate of Analysis We’ve outlined four of the primary sources of before making a purchase to confirm the purity variation in experimental results and provided and viability of the cells and ensure the product tips and research examples of how we’re and donor specifications match your experimental improving reproducibility in our immunology requirements. A good Certificate of Analysis will and inflammation research lab. be lot-specific to reflect testing of that particular lot, not just a general product description. 2 Look for details like: Culture Media D Number of live cells per vial Culture media is an easily overlooked variable in D Percentage of viable cells experimental design as researchers typically use D Expression of cell surface antigens their favorite media or the one they deem most (types of cells in the vial) appropriate for the experiment at hand. However, D Cell purity differences in culture media environment can D Donor age have a major impact on cell behavior and thus D Donor gender cause reproducibility issues. D Donor race D Donor height The data in figures one and two show the D Donor weight effect of culture medium on monocytes. D Donor blood type Human monocytes were cultured in either D Donor HLA type DME/F12 or IMDM for five days. Both media were supplemented with 10% human serum and Regardless of the product information given 10 ng/mL recombinant human M-CSF. They were by a vendor, always do your due diligence then stimulated by the addition of 100 ng/mL LPS by testing your antibodies and cell products and culture medium was collected after 48 hours. before starting your experiments to account for The monocytes cultured in DME/F12 made more discrepancies in the analysis and changes during IFNγ, IL-10, IL-6 and TNF-α, but the production of shipping, handling or storage. Test your products IL-13 was equivalent in the two media. using the technology you plan to use in your experiment. If you will be using antibodies for immunohistochemistry (IHC), test in IHC. Figure 1 Eect of Culture Medium on Production of Cytokines by Monocytes IFNγ, IL-10, IL-13 and IL-1β production 600 measured after culture of monocytes in 500 DME/F12 and IMDM. 400 300 200 pg/mL Cytokine 100 0 IFN-γ IL-10 IL-13 IL-1β IMDM DME/F12 3 Figure 2 Eect of Culture Medium on IL-6 and TNF-α Production IL-6 and TNF-α production measured 10,000 after culture of 9,000 monocytes in DME/F12 8,000 and IMDM. 7,000 6,000 5,000 4,000 3,000 2,000 pg/mL Cytokine 1,000 0 IL-6 TNF-α IMDM DME/F12 One way to reduce this variability is to use the enough for a single researcher, and nearly same culture media as described in the original impossible for a scientist in another lab to acquire. experiment when attempting to replicate a research finding. However, not all culture media So what can you do? While fetal calf serum (FCS) are created equal. is a popular medium supplement, it does fall victim to extreme variability. There are several When using serum-containing media, there serum-free media on the market, which can are natural variations in each animal’s genetics, be great fits for your immunology experiments blood, environment and diet that create and have far less variability. If FCS or other extremely high lot-to-lot variability. Acquiring serum-containing media are required, focus on enough serum from the same lot is challenging providing better documentation of your serum selection and lot characteristics. Figure 3 Proliferation of Tetanus Toxoid-Specic T Cells A medium designed to be used without serum was supplemented with 300,000 human serum or with fetal calf serum and used to support a T cell 200,000 proliferation assay. The medium supplemented with FCS supported greater proliferation 100,000 at the highest peptide concentration. The medium that was not supplemented allowed 0 Relative Luminescence Units Luminescence Relative greater proliferation 0.001 0.01 0.1 1 10 with low peptide concentrations. µg/mL Peptide Serum-Free Medium Medium + Human Serum Medium + Fetal Calf Serum 4 Factor #2: Equipment Calibration Lab equipment and instruments lose their experimentation is making sure all equipment is calibration over time, which can lead to calibrated and capable of measuring its intended unreliable and imprecise results. An often results with accuracy, precision and safety. overlooked component of responsible Precision vs Accuracy Precision Precision Precision Precision Accuracy Accuracy Accuracy Accuracy Following a proper calibration schedule be calibrated consistently to ensure proper will make your results more reliable and measurement. Reports should include the reproducible. But how do you know if your calibration at the outset of measurements as equipment is calibrated? And how often should well as after adjustment. This habit will help you check your equipment for accuracy? inform your calibration frequency and let you know if your instruments are off. Follow these Everything from your largest, most expensive suggestions for some popular lab equipment. equipment to your smallest instruments should Equipment Calibration Frequency Suggestions Biological Safety Cabinets Manufacturer-recommended Hire a metrologist to do a Incubators calibration intervals or before professional, NIST-traceable Centrifuges and after major experiments calibration Biannual or before major Test the full range of volume the Pipettes experiments pipette can dispense Refer to the original manual for pH Meters Before every use exact calibration standards 5 Variation Two different pieces of the same equipment are microplate readers, a Packard Fusion™ and a going to differ — whether they are the same BioTek Synergy™ HT. The proliferation values as model and year, or two models from different measured by the Packard are lower but more decades. All equipment wears differently, even if consistent within triplicates as shown by the used for the same purposes over the same time smaller error bars based on standard deviation. period in the same lab conditions. The BioTek plate reader returned values that were higher but also had a greater standard deviation Figures four and five show the difference and coefficient of variation. in measuring luminescence using two Figure 4 BC3 T Cell Proliferation BC3 T cell proliferation using a Packard Fusion™ 500,000 microplate reader to 450,000 measure luminescence. 400,000 350,000 300,000 250,000 200,000 150,000 100,000 50,000 Relative Luminescence Units Luminescence Relative 0 0 0.1 0.3 1 3 10 µg/mL Peptide Figure 5 BC3 T Cell Proliferation BC3 T cell proliferation using a BioTek Synergy™ 500,000 microplate reader to 450,000 measure luminescence. 400,000 350,000 300,000 250,000 200,000 150,000 100,000 50,000 Relative Luminescence Units Luminescence Relative 0 0 0.3 1 3 µg/mL Peptide 6 The first step in controlling machine-to-machine D How long has the equipment been in use variation is to follow a consistent calibration and in what capacity? schedule. Beyond calibration, there are a few D Does the machine have a history of things you can do to reduce the impacts of damage or refurbishment? equipment variability. D What are the terms of the warranty? Even used equipment should have a warranty. 1. Contain your experiments to one set of equipment.