The Scientific Method

The Scientiﬁc Method

DA2205

September 22, 2014 Recap of previous lecture

• Do computer science research as scientiﬁc research because 1. Computer science can be viewed as a science: the transformation of artiﬁcial and natural information processes and/or

2. It forces you to have good research methods

• Research is hard because of 1. intrinsic diﬃculty of discovering the unknown,

2. personal failings - self-deception,

3. bias and limitations imposed by the community.

• But the scientiﬁc method has allowed for rapid and not too many misguided declarations of progress! The computer scientiﬁc method

Obsolete Scientiﬁc Method Computer Scientiﬁc Method • Hypothesis • Hunch

• Sequence of experiments. • 1 experiment & change all parameters • Change 1 parameter per experiment. • Discard if it doesn’t support hunch • Prove/Disprove Hypothesis. • Why waste time? • Document for others to repro- We know this. duce results.

Source: “How to have a bad career in research/academia” by David Patterson, Feb 2002. Good research methods help you

• Identify interesting new questions you hadn’t thought of. (Exploratory data analysis) • Ask questions that are scientifically meaningful. (Falsifiable hypotheses) • Avoid fixating on one favored hypothesis. (Multiple working hypotheses) • Devise evaluations that maximize what you can learn. (Experimental design) • Avoid seeing patterns in randomness. (Hypothesis tests) • Understand when you have collected sufficient evidence. (Statistical power)

• Formulate broader theories. (Modeling) It discourages this scenario The Significance of Research

9-Oct-10 2 Today’s lecture

• Review of the scientific method - Definition of Scientific Theory, Hypothesis and Law.

- Review of reasoning in research ? Deduction, Induction, Abduction

- Expanding Scientiﬁc knowledge and falsiﬁcation.

- The Hypothetic–deductive scheme.

• Scientiﬁc method and computer science

• The variety of research endeavour Review of the scientiﬁc method Scientiﬁc method

• The scientiﬁc method helps us do this

Observation

Theory World

Validation Important functions of the scientiﬁc method

• Validation ? Can a general statement be judged as true, false or probable?

• Analysis of hierarchical structure ? Reductionism: Understand complex system by reducing them to the interactions of their parts. ? Emergence: Produce complex, interesting high-level function by combining simple low-level mechanisms in simple ways.

• Causality ? Deﬁne and explore the connection between actions. Encoding of scientiﬁc knowledge or premises

• Scientiﬁc Hypothesis: - Proposed explanation for an observable phenomena.

• Scientiﬁc Model: - A physical, conceptual, or computer-based representation of a system.

• Scientiﬁc Theory: - A comprehensive set of ideas that explains a phenomenon.

• Scientiﬁc Law: - Describes a phenomena, often mathematically. Theory building should be the goal because

• Theories lie at the heart of what it means to do science. ? Production of generalizable knowledge ? Serve to explain and predict ? Concepts, relationships, causal inferences

• Theory provides orientation for data collection ? Theoretical perspective allows eﬃcient observation of the world

• Theories allow us to compare similar work ? Theories include precise deﬁnition for the key terms ? Theories provide a rationale for which phenomena to measure

• Theories support analytical generalization ? Provide a deeper understanding of our empirical results ? ...and hence how they apply more generally ? Much more powerful than statistical generalization Reasoning within the scientiﬁc method Deduction explained

• Deduction is Arguing from the general to the particular

• Example

All Frenchmen like red wine. Pierre is a Frenchman. =⇒ Pierre likes red wine.

• Terminology of the example:

All Frenchmen like red wine. ← premise Pierre is a Frenchman. =⇒ Pierre likes red wine. ↑ ↑ premise conclusion Virtues of deduction

• A deduced conclusion is deﬁnitely true if ? premises (axioms) are true and ? reasoning carried out correctly.

• An example is Pythagoras’ Theorem can be deduced from Euclid’s axioms.

• For deductive logic: discovery process ≡ justiﬁcation process

• However, most mathematical theorems are not discovered solely by the exercise of deductive reasoning.

• Use deductive reasoning to justify or falsify an inspiration or an intuitive belief. Limitations of deduction

• Deductive reasoning uncovers what is implicit in our premises.

• Unfortunately, deductive reasoning cannot bring us knowledge of the world beyond our premises.

• It can however bring us awareness about our premises.

• But how can we expand our knowledge??? Limitations of deduction

• Deductive reasoning uncovers what is implicit in our premises.

• Unfortunately, deductive reasoning cannot bring us knowledge of the world beyond our premises.

• It can however bring us awareness about our premises.

• But how can we expand our knowledge??? Reasoning to potential new knowledge: Induction Induction explained

• Induction is Arguing from the particular to the general

• Example of inductive inference:

The ﬁrst ﬁve eggs in the box are rotten. All six eggs have the same best-before date stamped on them. Infer the sixth egg will be rotten too.

• Inductive inference does not guarantee the conclusion is true. An inductive argument

• Start with observations or experimental results.

• On the basis of these, start framing general principles that take the observations into account.

• Illustration of its limitation( enumerative induction) ? Someone from Europe, having seen many swans, all of them white, comes to the conclusion “All swans are white”.

? He anticipates the next swan to appear will also be white.

? The generalization is conﬁrmed with every new swan that is seen.

? Then, visiting Australia, the person comes across a black swan and has to think again. An inductive argument

• Start with observations or experimental results.

• On the basis of these, start framing general principles that take the observations into account.

• Illustration of its limitation( enumerative induction) ? Someone from Europe, having seen many swans, all of them white, comes to the conclusion “All swans are white”.

? He anticipates the next swan to appear will also be white.

? The generalization is conﬁrmed with every new swan that is seen.

? Then, visiting Australia, the person comes across a black swan and has to think again. The inductive scientiﬁc method

• Generation of a possible hypothesis ? Gather evidence – if possible eliminate irrelevant factors. (somewhat contradictory as need a hypothesis to define relevancy.) ? Conclusions inferred from the evidence leads to a hypothesis. • Refinement of the hypothesis ? Experiments are devised to test out the hypothesis, based on its predictions. ? If necessary the hypothesis is modified to take into account the results of the later experiments. ? A general theory is framed from the hypothesis and its related experimental data. • Verify or falsify hypothesis ? Use this theory to make predictions. On the basis of these can confirm or disprove the theory. Reasoning to potential new knowledge: Abduction Abduction

• Infer a as an explanation of b.

• Abduction allows the precondition a to be inferred from the consequence b.

• Deduction and abduction diﬀer in the direction in which a rule like “a entails b” is used for inference.

• Truth of the assumptions do not guarantee the truth of the conclusion. What is abduction?

“...a method of reasoning in which one chooses the hypothesis that would, if true, best explain the rele- vant evidence. Abductive reasoning starts from a set of accepted facts and infers their most likely, or best, explanations.” Example of abductive reasoning

• You notice “The lawn is wet”.

• If “it rained last night”, it would be unsurprising that “the lawn is wet.”

• By abductive reasoning, the possibility that it rained last night is reasonable. Example of abductive reasoning

• You notice “The lawn is wet”.

• If “it rained last night”, it would be unsurprising that “the lawn is wet.”

• By abductive reasoning, the possibility that it rained last night is reasonable. Note • Abducing rain last night from the observation of the wet lawn can lead to a false conclusion.

• Maybe dew, lawn sprinklers, or some other process could have resulted in the wet lawn. Abductive reasoning in science

• Abduction selects, among the hypotheses that are being considered, the one that best accounts for the evidence.

• Abductive reasoning is closely related to the statistical method of maximum likelihood estimation.

• There exist several obvious threats to validity ? Small hypotheses spaces.

? Small amounts of evidence to explain. Both are key challenges to the scientiﬁc practice. Challenges of abduction

• Creating hypothesis spaces likely to contain the “true” hypothesis ? Approach – Create large hypothesis spaces.

• Knowing when more valid hypotheses are missing from your hypothesis space ? Approach – Constantly evaluate your hypothesis space, and expand the hypotheses space whenever the data become extremely unlikely, given any current hypothesis.

• Creating good sets of evidence to explain ? Approach – Seek diverse and independent evidence with which to evaluate hypotheses. Reasoning to potential new knowledge: The hypothetic–deductive reasoning scheme Scientiﬁc reasoning: Hypothetic–deductive scheme

• Science in its discovery mode is not propelled by logic.

• Scientiﬁc reasoning is a dialogue between the possible and the actual.

• There are two thought process which alternate and interact the imaginative ←→ the critical

• Imaginative process: Formation of a hypothesis ? Form an opinion, take a view, make an informed guess, which may explain the phenomena under investigation

• Critical process: Expose the hypothesis to criticism ? Empirical testing of logical consequences of the hypothesis. ? If predictions accurate =⇒ gain conﬁdence in the hypothesis. ? If predictions not accurate =⇒ hypothesis wrong. Scientiﬁc reasoning: Hypothetic–deductive scheme

• Science in its discovery mode is not propelled by logic.

• Scientiﬁc reasoning is a dialogue between the possible and the actual.

• There are two thought process which alternate and interact the imaginative ←→ the critical

• Imaginative process: Formation of a hypothesis ? Form an opinion, take a view, make an informed guess, which may explain the phenomena under investigation

• A supposition or proposed explanation made on the basis of limited evidence as a starting point for further investigation.

• Also – theory, theorem, thesis, conjecture, supposition, postulation, postulate, proposition, premise, assumption

• Origin – late 16th century: via late Latin from Greek ? hupothesis ≈ “foundation”, ? from hupo (≈ “upon”) + thesis (≈ “placing”). Types of hypotheses

• Existential An entity or phenomenon exists. Atoms contain uncharged subatomic particle (neutrons)

• Compositional An entity or phenomenon consists of a number of components (perhaps with a speciﬁed frequency). Atoms consists of protons, neutrons and electrons

• Correlational Two measurable quantities have a speciﬁed association. An element’s atomic weight and its properties are correlated

• Causal A given behaviour has a speciﬁed causal mechanism. The low reactivity of the noble gases is caused by their full outer shell of valence elements. Good property of a hypothesis: Falsiﬁability Karl Popper’s “conjectures and refutations”

• Conﬁrmations of a theory are typically easy to ﬁnd when sought.

• True conﬁrmations should be surprising. ? If we didn’t know the theory, we should have expected an event which was incompatible with it.

• Good scientiﬁc theories are prohibitions. ? They forbid certain things from happening. The more a theory forbids, the better it is.

• A theory which is not refutable by any conceivable event is non-scientiﬁc. Irrefutability is not a virtue but a vice. “No amount of experimentation can ever prove me right; a single experiment can prove me wrong” – Albert Einstein

“A theory which cannot be mortally endangered cannot be said to be alive.” – W.A.H. Rushton Falsiﬁability

In The Logic of Scientiﬁc Discovery (1934) Popper argued: • Falsiﬁability is the logical possibility that an assertion can be shown to be false by evidence.

• “Falsiﬁable” does not imply “false”. Instead, if a falsiﬁable proposition is false, then its falsehood can be shown by experiment, theorem or simulation.

• There are degrees of falsiﬁability – some theories are more falsiﬁable than others.

• Falsiﬁes theories can be rescued by introducing some ad hoc change, but only by lowering their apparent validity. Examples of non-falsiﬁable statements

• An alien spaceship crashed in Roswell New Mexico.

• A giant white gorilla lives in the Himalayan mountains.

• Loch Ness contains a giant reptile.

In each case, if the statement happens to be wrong, all you will ever ﬁnd is an absence of evidence. Examples of falsiﬁable statements

• No alien spaceships have ever landed in Roswell New Mexico. - Find just one spaceship and the statement is disproven.

- Do not need an exhaustive elimination of all possibilities.

- Just one spaceship will do it!

• All humans live forever. - Find just one dead person and the statement is disproven.

- Do not need an exhaustive elimination of all people. What makes hypotheses unfalsiﬁable?

• Vagueness The theory that does not predict any particular experimental outcome.

• Complexity A theory that can “explain” any experimental result.

• Special pleading Traditional experimental methods are claimed not to apply. Degree of falsiﬁability

• Number of experiments.

• Probability that a given experimental result would be produced given that the theory is false ? How many other outcomes were possible?

? In the absence of the theory, what is the probability distribution over those outcomes?

• Diversity of attempts. ? Potential unknown causal factors

? Potential experimental ﬂaws “Science is what we have learned about how to keep from fooling ourselves” – Richard Feynman Scientiﬁc Enquiry

Prior Knowledge Initial Hypothesis

Observe what is wrong with the current theory?

Experiment Theorize manipulate reﬁne/create a the variables better theory

Design Design empirical tests of the theory Reasoning to potential new knowledge: Multiple Working Hypotheses Why use multiple working hypotheses?

• Personal investment – Helps to separate you from your hypotheses; Shifts your personal investment from the hypothesis to the hypothesis test.

• Focus – Reinforces a focus on falsiﬁcation rather than conﬁrmation.

• Eﬃciency – Allows experiments and proofs to be designed to distinguish among several competing hypotheses rather than evaluating a single one.

• Harmony – Limits the potential for professional conﬂict and rejection because all hypotheses are considered rather than only one.

Alternatives

• Method of the ruling theory – Data are sought to conﬁrm the one theory held by a researcher.

• Method of the working hypothesis – Data are sought to falsify the current hypothesis of a researcher.

• Method of multiple working hypotheses – Data are sought and multiple hypotheses are evaluated based on those data, disconfirmed hypotheses are abandoned, and a new (potentially modified) set of hypotheses are evaluated in the next round. “The story of a theory’s failure often strikes readers as sad and unsatisfying. Since science thrives on self-correction, we who practice this most challenging of human arts do not share such a feeling. We may be unhappy if a favoured hypothesis loses or chagrined if theories that we proposed prove inadequate. But refutation almost always contains positive lessons that overwhelm disappointment, even when... no new and comprehensive theory has yet filled the void.” – Stephen Jay Gould Some myths about science

• “Scientists follow the scientiﬁc method” ? There is no one method ? Many methods available... ? ...and all of them have known ﬂaws ? Scientists use imagination, creativity, prior knowledge, perseverance.

• “Scientiﬁc knowledge is general and absolute” ? Empirical Induction used to build evidence ? Scientists often get it wrong... ? ...but Science (as a process) is self-correcting

Source: Steve Easterbrook. ASE’ 07, Tutorial T2: Empirical Research Methods for SE, 2007 Relevance for Computer Science Research Scientiﬁc method and the computer science researcher

Some questions we should consider • When does the CS researcher need the scientiﬁc method?

• How does the CS researcher use the scientiﬁc method?

First let’s think about what a CS researcher does? Scientiﬁc method and the computer science researcher

Some questions we should consider • When does the CS researcher need the scientiﬁc method?

• How does the CS researcher use the scientiﬁc method?

First let’s think about what a CS researcher does? Core technologies of computing

Core Technologies Algorithms Artificial Intelligence Compilers Computational Science Computer Architecture Data Mining Data Security Data Structures Databases Decision Support Systems Distributed Computation e-Commerce Graphics Human-Computer Interaction Information Retrieval Management Information Systems Natural-Language Processing Networks Operating Systems Parallel Computation Programming Languages Real-Time Systems Robots Scientific Computation Software Engineering Supercomputers Virtual Reality Vision Visualization Workflow

Have we missed anything? The Great Principles of Computing

At a high-level within all these technologies dealing with

• Principles of Mechanics

• Principles of Design Computing: Principles of Mechanics

The fundamental laws and recurrences of computation 1. Computation What can be computed and what are the limits of computing?

2. Communication How can the transmission of messages occur from one point to another?

3. Coordination How can multiple entities cooperate to achieve a single goal?

4. Recollection How can information be stored and retrieved?

5. Automation What cognitive tasks, ordinarily associated with human intelligence, can be automated? Computing: Principles of Design

Architecture: How to arrange the components of a system? Process: How we arrange our work load and thoughts? 1. Simplicity How can we avoid complexity?

2. Performance How can we predict performance of complex systems?

3. Reliability How do we keep devices and information available when needed?

4. Evolvability Can our design be adaptable to changes of function or scale?

5. Security How do we allow or limit access as needed? Addressing these challenges requires the Scientiﬁc Method

1. Information systems are often complex =⇒ only understand, validate and improve them by ? performing experiments, ? collecting data, and ? interpreting the data.

2. This requires Scientiﬁc Method in conjunction with mathematical models, theory and statistics ? formulate hypotheses, design experiments, ? test hypotheses, validate system and predict performance Addressing these challenges requires the Scientiﬁc Method

1. Information systems are often complex =⇒ only understand, validate and improve them by ? performing experiments, ? collecting data, and ? interpreting the data.

“Unlike other scientists, who study chemical reactions, processes in cells, bridges under stress, animals in mazes, and so on, we study computer programs that perform tasks in environments.” – Paul Cohen, 1995

Whether your subject is a rat or a computer program, the task of science, to provide theories to answer 3 basic research questions: How will a change in the agent’s • system affect its behavior given a task and an environment? • task affect its behavior in a particular environment? • environment affect its behaviour on a particular task?

Source: Paul Cohen, Empirical Methods for Artiﬁcial Intelligence, MIT Press, 1995. A framework for computer science research problems

System Task

Behaviour Environment Explanation of elements

• System – Aspects inﬂuenced by a system designer ? Speciﬁc algorithm used, system architecture, data structures, parameter settings, etc.

• Task – Aspects inﬂuenced by a prospective user ? Speciﬁc queries, requests, input data, etc.

• Environment – Aspects inﬂuenced neither by designer nor user ? Network environment, available cycles or memory, etc.

• Behaviour – Performance of the task by the system within the environment. Framework allows generation of many hypotheses

• Compositional An entity or phenomenon

System Task consists of a number of components (perhaps with a speciﬁed frequency)

• Correlational Two measurable quantities

Behaviour have a speciﬁed association Environment • Causal A given behaviour has a speciﬁed causal mechanism The Varieties of Research Endeavour Scientists cannot be pigeon-holed

“Scientists are people of very dissimilar temperaments doing different things in very different ways. Among scientists are collectors, classifiers and compulsive tidiers-up; many are detectives by temperament and many are explorers; some are artists and others artisans. There are poet-scientists and philosopher-scientists and even a few mystics. ...and most people who are in fact scientists could easily have been something else instead.” – Peter Medawar (1915-1987)

Source: Peter Medawar, ’Hypothesis and Imagination’ in the The Art of the Soluble, 1967 What do researchers spend most of their time doing?

Construct a general theory

Identify an Find ﬂaws important exception in a previous to a theory experiment

Use a theory to explain Devise a research an observation question Establish a Gather data relationship Design an Design an between Compare algorithm experiment results from variables Run an theory and Identify an experiment experiment existence proof Devise a new test of a hypothesis Make a conjecture

Construct a Devise a new measurement theoretical proof or technique

Unify disparate theories Not everything

• These are a researcher’s core activities.

• Not everything a researcher typically does. Time also spent ? Reading the literature

? Getting resources

? Building infrastructure (software, labs, etc.)

? Teaching students

? Building professional relationships

? Reviewing papers

• These auxiliary tasks enhance and is scaﬀolding for the core activities. Construct an algorithm or system

• Much “doing computer science” is about building the infrastructure to do computer science

• Examples include building new... ? Compilers

? Garbage collectors

? Networking protocols

? Machine learning algorithms

• But, of course, this isn’t everything we do (or should do)... Construct an algorithm or system

• Much “doing computer science” is about building the infrastructure to do computer science

• Examples include building new... ? Compilers

? Garbage collectors

? Networking protocols

? Machine learning algorithms

• But, of course, this isn’t everything we do (or should do)... Identify a research question

• Identify a research question about which a hypothesis can be formulated

• These hypotheses are typically about ? Algorithms

? Tasks, or

? Environments

• Questions about ? Individual elements (e.g., existence proofs)

? How changes in one element aﬀect another

? Comparisons of two more more elements holding others constant

• Often iterative and done by multiple researchers Make a conjecture

• Formulate a hypothesis or predict an experimental result

• Types ? Existential (e.g., phase transitions)

? Compositional (e.g., behavior regions of CSPs)

? Correlational (e.g., Training set size/tree size)

? Causal (e.g., bias propagation)

• Want multiple hypotheses that explain existing observations, rather than just a single one. Make a conjecture

• What makes a good conjecture? ? Falsiﬁable (Popper)

? Important, if true (or false)

? Consistent with existing observations

? Not obviously true

• Examples ? Structure of DNA

? Mendeleev’s periodic table Gather data

• Naturalistic observations (e.g., router performance in actual networks)

• Carefully selected observations (i.e., quasi-experimental design)

• Experiments

• Sets of previously published experiments (i.e., Meta-analysis) Identify a relationship between variables

• The basis for both correlational and causal hypotheses

• Many variations ? Most common is two continuous variables (e.g., scatterplot), but also...

? Non-linear relationships

? More than two variables

? Discrete variables

• Emphasizes the importance of exploratory data analysis Unify previously separate theories

• Synthesize multiple theories into one

• Why is this useful? ? Simplicity – Why have two theories when you can have one?

? Expanded evidence and methods – Evidence and methods from both areas can be used to examine the theory

? Faster error correction– If one theory is falsiﬁed or modiﬁed, well know that both should be Identify an important exception

• Results in ? Assignment of exception to a trivial cause (e.g., experimental error), or

? Recognition of a misclassiﬁcation of previously observed phenomena, or

? Recognition of new subclass of situation that requires a diﬀerent theory, or

? A wholescale revision in the theory. Exciting when the unexpected happens

“The most exciting phrase to hear in science, the one that heralds new discoveries, is not ’Eureka!’ (I found it!) but ’That’s funny ...” – Isaac Asimov Discover an existence proof

• Only slightly diﬀerent from identifying an important exception, but often about phenomena about which no theory exists

• Goals ? Describe new example accurately and as completely as possible

? Find other examples that share characteristics

? Attempt to generalize to a description of the class Determine a new test of a hypothesis

• Goal – Maximize probability of falsiﬁcation (assuming hypothesis is false)

• Valuable when... ? Independent of prior attempts • Diﬀerent experimental or theoretical methods • Diﬀerent data

? Similar to, but better than, prior attempts • Higher accuracy • Rule out alternative explanations (better control) Design an experiment

• Specify algorithm, task, and environment

• Identify key dependent variables and appropriate measurement techniques

• Use experimental methods ? Control or randomize key independent variables (potential sources of variation)

? Use replication, blocking, and modeling to maximize value of data collection Formulate a measurement or technique

• Surprisingly common key to important scientiﬁc discoveries

• Examples ? Watts & Strogatz Clustering coeﬃcient & path length

? Kirkpatrick & Selman Measures of problem diﬃculty

? Friedman (and others) Bias-variance analysis Other Varieties (theory and experiment)

• Devise a theoretical proof

• Run an experiment

• Compare results from theory & experiment

• Find ﬂaws in existing experiments/proofs

• Use a theory to explain an observation Other Varieties (large theories)

• Formulate a general theoretical framework

• Examples ? Quantum mechanics

? Plate tectonics

? Evolution by natural selection

? Germ theory of disease

? Gravitation