Experimental Design

Jargon to be defined:

Observational Study vs Experiment

Controlled or Comparative Experiment

Control group

Subjects, Factors, Levels, Treatments

Confounding, Randomization

Blinding, Double Blind

91 Interested in assessing the effects of one or more factors on a response variable.

Observe response of subjects (or experimen- tal unit) to combination of levels of each fac- tor.

Combination called a treatment

Example: plant several different types of wheat in different fields. Use different amounts of fer- tilizer on different fields.

The individual fields are the “subjects”.

Two factors: wheat type and amount of fertil- izer.

Levels of the factors: levels of seed types will be names like say “Durum” or “Semolina”. Levels of “amount of fertilizer” will be things like “100 kg per hectare”.

Treatments are things like: Semolina wheat with 200 kg per hectare of fertilizer. 92 In an experiment the experimenter controls the assignment of (at least some) factors to subject.

Some examples to illustrate strengths and weak- nesses of observational studies and experiments.

Example: Two trials of Salk polio vaccine in 1954.

Study 1: (NFIP) : Parents of grade 2 children asked to consent to treatment with vaccine. Those consenting given vaccine. Compared to Grade 1 and 3 students in same school dis- tricts.

Study 2: (Randomized controlled double-blind):

Parents of children asked to consent to treat- ment with vaccine or placebo.

Among consentors: half assigned at random to treatment (vaccine), half to control (placebo)

93 Results of trial (cases per 100,000 children)

NFIP Design (1,075,000 children)

# Children Rate Gr 2 vaccinated 225,000 25 Gr 1 & 3 725,000 54 Gr 2 no consent 125,000 44

Randomized trial (750,000 children)

# Children Rate Vaccinated 200,000 28 Control 200,000 71 No consent 350,000 46

94 Features to note:

1) No consent group does better than controls.

2) NFIP control group mixture of potential no consent and potential consent groups – they weren’t asked.

3) The vaccinated group did better but vaccine not perfect.

4) NFIP design biased against the vaccine.

5) Consent rate higher in NFIP design.

Another important feature: randomized trial was double blind.

Important since diagnosis difficult – might be influenced by knowledge of treatment status.

95 Another example: portacaval shunt.

Meta-analysis. Study of 51 studies of porta- caval shunt as treatment for hemorrhage in cirrhosis of the liver. Comment: study from 1966.

Enthusiasm

Marked Moderate None No Controls 24 7 1

Controls, not 10 3 2 randomized Randomized, 0 1 3 controlled

96 Commentary:

1) Poor designs lead to much more enthusiasm for treatment.

2) Sometimes use “historical controls” – records or success rates from hospital over past few years. Problem: patients in study not like gen- eral patient.

3) No randomization leads to assignment of patients to treatment arm (jargon) based on prognosis in doctors’ minds.

4) Portacaval shunt operation appears still to be used. But perhaps for other conditions?

5) Three year survival rate for treated patients: about 60% in all trials.

6) Same for controls in randomized trials.

7) Only 45% for controls in not randomized trials. Treatment is confounded with condi- tion of patient at start of trial; sick patients excluded from surgery – makes surgery look good. 97 Other topics in experimental design:

1) Consider assessing effect of drug on blood pressure:

Design 1: completely randomized design. Get 200 patients. Split into 2 groups at random. Compare two groups.

Design 2: blocked design: Get 100 male and 100 female patients. Split each group of 100 into 2 groups of 50 at random. Make com- parison within each sex (and examine average too).

Design 3: matched pairs. Get 200 patients. Make 100 pairs. In each pair put two patients with similar health conditions. Assign 1 mem- ber of each pair at random to treatment.

Design 3 can be more efficient at detecting small differences.

Design 3, like 2, is blocked. 98 2) Principal defect of observational studies: cannot rule out presence of confounding vari- able.

3) Principal defect of controlled experiment: not always easy to generalize to experience to be expected in real world.

In experiments all patient groups usually do better than in historical records — more at- tention paid to patients.

Similar effects seen in many other experiments – conditions not a match for real world.

99 Look back at Salk vaccine: not perfect — re- duced rate from 71 to 28 per 100,000.

142 of 200,000 controls developed polio

56 of 200,000 vaccinated developed polio.

Q1) could picking people to give treatment to at random accidentally produce such a lop- sided comparison?

Q2) if we had had same rates in say 25,000 children:

7 vaccinated cases and about 18 unvaccinated would that have been persuasive?

What about 3 cases in 10,000 vaccinated and 8 cases in 10,000 controls?

Next: assess strength of evidence using prob- ability calculations. 100