4 Factors That Affect Research Reproducibility Improve Reproducibility in Your Lab by Focusing on These Key Sources of Variability

WHITE PAPER

Getting on with Discovery The fiery debate over reproducibility in science 4 Factors That Affect has burned strong over the past several years, and the flames don’t show any signs of dying Research Reproducibility 1. Reagents down just yet. No matter how scientists view 2. Equipment reproducibility in their respective fields — from 3. Personnel a full-blown crisis to a minor issue that doesn’t 4. Methods impact the credibility of published findings — the significance of reproducible experimental design is undeniable. Factor #1: Reagents All fields of science, immunology included, were built on a foundation of replicating Antibodies and Cells experiments and studying breakthroughs Verifying the quality of reagents used in research for expanding the knowledge base, inspiring can prevent wasted time, wasted resources and more experimentation, and ideally, leading unverifiable results. Unvalidated reagents can to additional breakthroughs. Without snowball down the research chain, leading to reproducibility, the flywheel of experimentation retracted publications and failures in pre-clinical and discovery can’t gain momentum. and clinical trials. Front-end validation, while time-consuming, is an important step that many According to a Nature survey of 1,500 scientists, researchers choose to skip without thinking more than 70% have tried and failed to through the potential consequences. reproduce the results of another scientist’s experiment, and perhaps more shocking and Antibody quality control starts with procurement. concerning, over 50% have failed to reproduce Look for a vendor that offers detailed product their own experiments.1 documentation, has a substantial list of research publications using their products and is An Answer to the Problem? comfortable providing customer references. Widespread reproducibility mandates have been debated and undoubtedly would The vendor should be able to provide test results be costly, time-consuming and difficult to to show that the antibody binds to its target and implement and enforce across the board. not to related proteins. This negative control is While we await an industry-wide standard, if as important as a positive control since cross- one ever comes, there are many things we can reactivity is possible even with monoclonal take responsibility for in our labs and research antibodies. practices to improve reproducibility. Check every product’s Certificate of Analysis We’ve outlined four of the primary sources of before making a purchase to confirm the purity variation in experimental results and provided and viability of the cells and ensure the product tips and research examples of how we’re and donor specifications match your experimental improving reproducibility in our immunology requirements. A good Certificate of Analysis will and inflammation research lab. be lot-specific to reflect testing of that particular lot, not just a general product description.

2 Look for details like: Culture Media DDNumber of live cells per vial Culture media is an easily overlooked variable in DDPercentage of viable cells experimental design as researchers typically use DDExpression of cell surface antigens their favorite media or the one they deem most (types of cells in the vial) appropriate for the experiment at hand. However, DDCell purity differences in culture media environment can DDDonor age have a major impact on cell behavior and thus DDDonor gender cause reproducibility issues. DDDonor race DDDonor height The data in figures one and two show the DDDonor weight effect of culture medium on monocytes. DDDonor blood type Human monocytes were cultured in either DDDonor HLA type DME/F12 or IMDM for five days. Both media were supplemented with 10% human serum and Regardless of the product information given 10 ng/mL recombinant human M-CSF. They were by a vendor, always do your due diligence then stimulated by the addition of 100 ng/mL LPS by testing your antibodies and cell products and culture medium was collected after 48 hours. before starting your experiments to account for The monocytes cultured in DME/F12 made more discrepancies in the analysis and changes during IFNγ, IL-10, IL-6 and TNF-α, but the production of shipping, handling or storage. Test your products IL-13 was equivalent in the two media. using the technology you plan to use in your experiment. If you will be using antibodies for immunohistochemistry (IHC), test in IHC.

Figure 1 E ect of Culture Medium on Production of Cytokines by Monocytes IFNγ, IL-10, IL-13 and IL-1β production 600 measured after culture of monocytes in 500 DME/F12 and IMDM. 400 300 200

pg/mL Cytokine 100 0 IFN-γ IL-10 IL-13 IL-1β IMDM DME/F12

3 Figure 2 E ect of Culture Medium on IL-6 and TNF-α Production IL-6 and TNF-α production measured 10,000 after culture of 9,000 monocytes in DME/F12 8,000 and IMDM. 7,000 6,000 5,000 4,000 3,000 2,000 pg/mL Cytokine 1,000 0 IL-6 TNF-α IMDM DME/F12

One way to reduce this variability is to use the enough for a single researcher, and nearly same culture media as described in the original impossible for a scientist in another lab to acquire. experiment when attempting to replicate a research finding. However, not all culture media So what can you do? While fetal calf serum (FCS) are created equal. is a popular medium supplement, it does fall victim to extreme variability. There are several When using serum-containing media, there serum-free media on the market, which can are natural variations in each animal’s genetics, be great fits for your immunology experiments blood, environment and diet that create and have far less variability. If FCS or other extremely high lot-to-lot variability. Acquiring serum-containing media are required, focus on enough serum from the same lot is challenging providing better documentation of your serum selection and lot characteristics.

Figure 3 Proliferation of Tetanus Toxoid-Speci c T Cells A medium designed to be used without serum was supplemented with 300,000 human serum or with fetal calf serum and used to support a T cell 200,000 proliferation assay. The medium supplemented with FCS supported greater proliferation 100,000 at the highest peptide concentration. The medium that was not supplemented allowed 0

Relative Luminescence Units Luminescence Relative greater proliferation 0.001 0.01 0.1 1 10 with low peptide concentrations. µg/mL Peptide Serum-Free Medium Medium + Human Serum Medium + Fetal Calf Serum

4 Factor #2: Equipment

Calibration Lab equipment and instruments lose their experimentation is making sure all equipment is calibration over time, which can lead to calibrated and capable of measuring its intended unreliable and imprecise results. An often results with accuracy, precision and safety. overlooked component of responsible

Precision vs Accuracy

Precision Precision Precision Precision Accuracy Accuracy Accuracy Accuracy

Following a proper calibration schedule be calibrated consistently to ensure proper will make your results more reliable and measurement. Reports should include the reproducible. But how do you know if your calibration at the outset of measurements as equipment is calibrated? And how often should well as after adjustment. This habit will help you check your equipment for accuracy? inform your calibration frequency and let you know if your instruments are off. Follow these Everything from your largest, most expensive suggestions for some popular lab equipment. equipment to your smallest instruments should

Equipment Calibration Frequency Suggestions

Biological Safety Cabinets Manufacturer-recommended Hire a metrologist to do a Incubators calibration intervals or before professional, NIST-traceable Centrifuges and after major experiments calibration

Biannual or before major Test the full range of volume the Pipettes experiments pipette can dispense

Refer to the original manual for pH Meters Before every use exact calibration standards

5 Variation Two different pieces of the same equipment are microplate readers, a Packard Fusion™ and a going to differ — whether they are the same BioTek Synergy™ HT. The proliferation values as model and year, or two models from different measured by the Packard are lower but more decades. All equipment wears differently, even if consistent within triplicates as shown by the used for the same purposes over the same time smaller error bars based on standard deviation. period in the same lab conditions. The BioTek plate reader returned values that were higher but also had a greater standard deviation Figures four and five show the difference and coefficient of variation. in measuring luminescence using two

Figure 4 BC3 T Cell Proliferation BC3 T cell proliferation using a Packard Fusion™ 500,000 microplate reader to 450,000 measure luminescence. 400,000 350,000 300,000 250,000 200,000 150,000 100,000 50,000

Relative Luminescence Units Luminescence Relative 0 0 0.1 0.3 1 3 10 µg/mL Peptide

Figure 5 BC3 T Cell Proliferation BC3 T cell proliferation using a BioTek Synergy™ 500,000 microplate reader to 450,000 measure luminescence. 400,000 350,000 300,000 250,000 200,000 150,000 100,000 50,000

Relative Luminescence Units Luminescence Relative 0 0 0.3 1 3 µg/mL Peptide

6 The first step in controlling machine-to-machine DDHow long has the equipment been in use variation is to follow a consistent calibration and in what capacity? schedule. Beyond calibration, there are a few DDDoes the machine have a history of things you can do to reduce the impacts of damage or refurbishment? equipment variability. DDWhat are the terms of the warranty? Even used equipment should have a warranty. 1. Contain your experiments to one set of equipment. If your lab has two incubators, for example, keep all of your samples for Factor #3: Personnel one experiment in the same incubator. This will eliminate any variability from Training and Mentorship jumping back and forth between machines Ensure every researcher under your management throughout the study. We even see variation is adequately trained in experimental design. in evaporation from the front to the back of Provide training and establish design standards the incubator, so keep positioning in mind. to which everyone must adhere. 2. Provide thorough documentation. Always note which equipment you used in Every scientist has an individual style when your methods, from the brand, model and executing even the simplest procedures, like serial numbers to the date of last calibration pipetting liquids, mixing solutions or counting and any other pertinent information. Since cells. Draw attention to those differences by equipment is a likely source of variation comparing results and discussing the reasons between labs, it should be noted. for any discrepancies. This hands-on approach, 3. Understand the idiosyncrasies of each coupled with basic lab-wide standardization machine. Realize that each machine can guidelines, will help your scientists appreciate give slightly different results. For example, the proper procedures. a used microplate reader for luminescence may give lower values than a new machine. Beyond training, put in place an expectation and Understand these differences and know foundation of strong mentorship from senior how to account for them within and researchers. This is usually built into the hierarchy between experiments. in larger labs. In smaller research labs, mentorship needs to come from the top. Owners, managers Also be selective when making new equipment or experienced scientists should make a purchases to be sure you’re getting the best concerted, defined effort to understand available product for your budget from a reputable the professional goals of each researcher, vendor that offers a reasonable warranty. identify their strengths and opportunities for improvement and set aside time to work with Buying used equipment can be a major cost each individual to advance their knowledge, savings if you know what to look for. When skills and confidence. evaluating previously owned lab equipment, do your research and ask the right questions. Researchers who feel valued and are given the tools they need to succeed will be more fruitful and loyal in the long term.

7 Statistical Analysis Cell-based assays have a much higher potential Researchers too often fall into the trap of for variation than a chemical assay due to the sensationalizing conclusions with the goal nature of the immune system. Understanding of publishing their works in high-visibility, the degree of variability in a cell-based assay prestigious journals. This can lead to cherry- and the sources of it will help researchers picking results that, while promising or account for inherent variations in the assay to interesting, may not be reproducible. avoid improper conclusions.

If you have a sound hypothesis, your experiments Here are three common factors that should be are properly designed and executed and you carefully controlled to reduce variability in any stick to the original intent of your research, your cell-based assay. results should speak for themselves and garner the attention they deserve. DDSelecting Cells Cell type will affect the conditions and In Nature’s survey, nearly 90% of respondents outcomes of the assay. There are major endorsed “better understanding of statistics,” differences between primary cells and cell “better monitoring/supervision” and “more lines that make each suitable for different robust design” as the top approaches to boost types of assays. Follow the kit instructions reproducibility in science. or protocol carefully and procure the ideal cell type to ensure proper measurement. “Being at the cutting edge of science means that sometimes results will not DDPipetting Accuracy Pipetting errors are very common and be robust. We want to be discovering can lead to false data. It’s important to new things but not generating too calibrate your pipette instruments and many false leads.” follow proper techniques to ensure even distribution in all wells. Marcus Munafo, a biological psychologist at the University of Bristol, UK DDControlling Edge Effects Edge effects are errors resulting from uneven distribution of cells in a well plate, where cells in inner wells are Factor #4: Methods exposed to different conditions than those in outer wells. The causes are many, Variation in Cell-Based Assays including evaporation during incubation, Cell-based assays, when developed and temperature control and plate stacking. executed correctly, are an important tool in the Researchers have concluded that simple drug discovery process that can confirm how a changes, such as pre-incubation of newly drug candidate interacts with and responds to seeded plates at room temperature, can an organism. It’s important to follow the assay reduce edge effects.2 instructions closely and use the materials and methods exactly as described, otherwise risk errors or unreliable outputs.

8 Inconsistent Methods However, increased cell metabolism can also Methods and measurement should be carefully increase ATP per cell, which may complicate considered and tailored to each experiment. Any interpretation. changes in methods or measurement techniques can yield inconsistent results, whether in your lab If measuring proliferation with the uptake of or in others. 3H-thymidine or bromodeoxyuridine, which label cells replicating their DNA, only the cells One common example of inconsistent methods in S phase will be labeled. These two methods impacting results is measuring cell proliferation. can produce vastly different results, especially When using CellTiter-Glo®, a reagent based on with immune cell populations in which a small ATP content, increased cell metabolism and subset may proliferate with the majority simply higher ATP per cell is interpreted as proliferation. maintaining their metabolism.

Figure 6 Proliferation Measured by ATP PBMC proliferation measured by ATP 40,000 content using 35,000 CellTiter-Glo®. 30,000 25,000 20,000 15,000 10,000 5,000

Relative Luminescence Units Luminescence Relative 0 Control Tetanus PHA LPS CMV 1406JN12 1559JY12

Figure 7 Proliferation Measured by BrdU Uptake PBMC proliferation measured by uptake of 1.600 bromodeoxyuridine. 1.400 1.200 1.000 0.800 0.600 0.400

Absorbance at 450 nm at Absorbance 0.200 0.000 Control Tetanus PHA LPS CMV 1406JN12 1559JY12

9 We also see differences in the amount of Conclusion cytokines measured using ELISA versus the Meso Scale method, as a second example. This can Depending on the size of your lab and type of be traced, in some cases, to the nature of the research you perform, sweeping overhauls in the cytokine used to prepare the standard curve and way you run your lab and design experiments the accuracy of its concentration. may or may not be necessary. Only you can be the judge of your needs. Another variable that should be carefully selected is the analysis software. While several However, you should be conscious of your tools may be designed with the same basic responsibility to publish reproducible findings principles and analytic concepts, each could or deliver valid data to your customers. Even return different results when analyzing the same implementing some of the suggestions dataset. Flow cytometry hardware combined mentioned above could give your research a with software can increase or decrease sensitivity significant boost in reproducibility. of detection for rare cell types. Consider using more than one type of software to get a better picture of your full dataset.

Beefing up your design standards, method selection and measurement techniques will not only increase your intra-lab reproducibility but should help outside scientists replicate your findings as well.

References 1 Baker M. 1,500 Scientists Lift the Lid on Reproducibility. Nature. 2016. 2 Lundholt B, Pagliaro L, Scudder K. A Sample Technique for Reducing Edge Effect in Cell-Based Assays. J. Biomol. Screen. Getting on with Discovery 2003; 8: 566-70. 866-513-1894 | www.astartebio.com © Astarte Biologics, Inc.