Experimental Research

Experimental Research in Language Learning

By Mohammad Adnan Latief [email protected]

University of Pittsburgh State University of Malang 2009

1 2009-2010 University of Pittsburgh-State University of Malang EXPERIMENTAL RESEARCH IN LANGUAGE LEARNING By Mohammad Adnan Latief

Abstract: Experimental research manipulates and controls the cause variable and proceeds to observe the change in the effect variable. There are several types of experimental research; some of them are True experimental, Quasi-experimental, and Pre-experimental. To the experimental findings, there are threats to its validity; some of them are History, Maturation, Testing, Instrumentation, Statistical Regression, Experimental Mortality or Attrition, Selection-Maturation Interaction, Experimental Treatment Diffusion, Compensatory Rivalry by the Control Group, and Resentful Demoralization of the Control Group. The causal research that does not manipulate the independent variable is called Ex-Post Facto Research.

Experimental research measures the effect of one manipulated and controlled (independent) variable to another (dependent) variable, like the effect of different methods of teaching to the students’ achievement, the effect of an English training method to participants’ English skills. Experimental research is a powerful research method to establish cause-and- effect relationship (Borg, W.R., Gall, M.D.1989:639) involving two or more variables, the variable that becomes the cause (independent) and the variable that becomes the effect. The cause variable is the condition or event that exists or occurs prior to the effect. The effect variable is a condition or event subsequent to the causal condition. The cause variable produces a change in the effect variable (Charles, C.M. 1995: 247). In Experimental research, the researcher deliberately controls and manipulates the conditions or makes a change in the value of the independent variable and observes the effect of that change in the dependent variable (Cohen, L., Manion, L.1994:164). Educational research method has been very fruitful in the production of the theoretical knowledge in education as well as in the improvement of the quality of educational curriculum materials and practices. Many experiments carried by educational researchers are concerned with testing the effect of new educational materials and practices on students’ learning. Thus, the results of educational experiments may have a direct impact on the adoption of new curriculum materials and teaching methods in the schools (Borg, W.R., Gall, M.D.1989:640)

2 2009-2010 University of Pittsburgh-State University of Malang Types of Experimental Research The key problem in experimentation is establishing suitable control for interfering variables so that any change in the dependent variable can be confidently attributed only to the experimental treatment manipulated by the researcher (Borg, W.R., Gall, M.D.1989:641). One way to control the presence of the interfering variables is random selection of the samples involved in the research. The researcher’s access to sample selection determines the types of the experimental research; true experimental, quasi experimental, or Pre-Experimental.

True Experimental Research In a true experimental research, the researcher has access to random selection of the samples and random assignment of the samples into experimental and control groups to ensure the equivalence of groups and to control for many interfering variables that might otherwise contaminate the results of the investigation (Charles, C.M. 1995:247). This true- experimental research design can be used with Pre-test-Post-test Control-Group Design or Post-test-only Control-Group Design (Borg, W.R., Gall, M.D.1989:674) The example procedure for Pre-test-Post-test Control-Group Design involves the following steps 1. Define the accessible population of the research (e.g. all the registered 1st year students of State Senior high School One in Malang) 2. Randomly select a number of the members of the population as the sample (e.g. 100 students out of all the 1st year Senior High School One students in Malang) 3. Randomly select one half of the members of the sample and assign them into group A and assign the rest into group B (e.g. 50 students of group A and 50 students of group B) 4. Give a pre-test to assess the level of knowledge or skills of group A and group B. Analyze the results of the pre-test to ensure that the average scores of the two groups are not significantly different. 5. Randomly assign one of the groups into the Experimental group and the other one into the Control group. 6. Give the experimental treatment to the experimental group and the control treatment to the Control group.

3 2009-2010 University of Pittsburgh-State University of Malang 7. Give a post-test to assess the results of the treatments of both groups 8. Compute the difference between the average score of the Experimental group and the Control group.

Quasi-Experimental Research In educational settings, very often it is not possible to select the sample randomly out of all the population students. When the researcher can only assign randomly different treatments to two different classes, the researcher uses quasi-experimental research design (Charles, C.M. 1995: 247). The procedure in quasi-experimental research involves fewer steps than the true experimental research as follows. 1. Define the accessible population of the research (e.g. all the registered 1st year students of State Senior high School One in Malang) 2. Select two classes out of all the existing classes which are of equivalent level (e.g. Class 1-C and class 1-F of State Senior high School One in Malang) 3. Randomly select one of the classes into Experimental group and the other one into the Control group (e.g. Class 1-C as the Experimental group and Class 1-F as the Control group) 4. Give the experimental treatment to the experimental group and the control treatment to the Control group. 5. Assess the results of the treatments of both groups 6. Compute the difference between the average score of the Experimental group and the Control group.

Pre-Experimental Research Sometimes an experimental research is conducted without a control group. In this research, the group is given a pre-test before the experimental treatment. After the treatment is finished, the post test is administered to see the achievement. The effectiveness of the instructional treatment is measured by comparing the average score of the pre-test and the post test. When it turns out that the post test average score is significantly higher than the average score of the pre-test, then it is concluded that the instructional treatment is effective. Another

4 2009-2010 University of Pittsburgh-State University of Malang name for this pre-Experimental Design is One-Group Pre-test-Post-test Design ((Borg, W.R.,Gall, M.D.1989:670) This pre-Experimental research can be done to investigate, for example, the effectiveness of the One-Semester Intensive English Course for Freshmen of English Department, State University of Malang, the effectiveness of bilingual classes in Science classes in SMAN 3 Malang, the effectiveness of a Master Degree training for English teachers teaching in Islamic high Schools. Cohen, L. & Manion, L (1994:165) gives an example of Pre-Experimental research as follows A group of subjects are given a test on the attitudes towards minority groups. Since the result of the pre-test shows that the subjects’ tolerance toward minority groups is still very low, a ten-week curriculum project designed to increase tolerance of ethnic minorities is introduced as an experimental manipulation. Following the experimental treatment, the researcher has again measured group attitudes and proceeded to account for differences between pre-test and post-test scores by reference to the effects of the treatment.

The weakness of Pre-Experimental research is the lack of a comparative group to control for interfering variables that might contaminate the results of the research.

Threats to validity in experimental Research The key problem in experimentation is establishing suitable control for the threats to experimental validity which come from unexpected interfering variables in the process of experiments. Some of the interfering variables that may threaten the validity of the experimental research findings are history, maturation, testing, instrumentation, statistical regression, experimental mortality, selection-maturation interaction, experimental treatment diffusion, compensatory rivalry by the control group, compensatory equalization of treatments, and resentful demoralization of the control group.

History When the experiment lasts a long time, it is possible that other events may occur and interfere the progress of subjects’ learning. When the experimental treatment is developing more motivation to the students in the experimental group, then the motivation may develop positive attitude of the students toward learning, if the achievement of the experimental group

5 2009-2010 University of Pittsburgh-State University of Malang is better than that of the control group, then the difference in achievement may not only be caused by the different treatment but also by the different motivation and attitude of the students. Since the different motivation and attitude are resulted from the process of the implementation of the experimental treatment, this interference is called history (See Borg, W.R.,Gall, M.D.1989:644).

Maturation Maturation is related to biological and psychological development of the students while the experiment is in progress. Maturation interferes the results of educational experiments if it happens that the biological as well psychological development of the students in the experimental group, which may affect students’ cognitive ability in learning, is faster than that in the control group. The difference in biological as well as in psychological development may happen because, for example, the parents of the students in the experimental group have better socioeconomic levels. These socioeconomic background characteristics of the students in the two groups are usually not assessed by the researcher.(See Borg, W.R.,Gall, M.D.1989:644).

Testing In the experiment using pre-test and post-test, if the post-test is the same instrument with the pre-test, the students might learn from the pre-test. If it happens that the students in the experimental group remember the questions of the pre-test and have found out the correct answers before the post-test is administered, their answers to the post-test do not show the result of the experiment. This pre-test effect gives interfering variables which is usually very difficult to control. This effect can be controlled by using equivalent tests for the post-tests (See Borg, W.R.,Gall, M.D.1989:644).

Instrumentation In an experimental research involving observational judgement to rate the performance of the subjects between pre treatment and post treatment, the instrumentation effect may interfere in the process of measuring the results of the experiments. When the researcher observes the same students’ performance the second time after the treatment, he or she might not be able to give as objective rating to the quality of the students’ performance as the rating

6 2009-2010 University of Pittsburgh-State University of Malang given in the first observation simply because the researcher may expect (subconsciously) a more favourable change in the students’ performance.. If this happens, then instrumentation effect has occurred and it is difficult to control this effect (See Borg, W.R.,Gall, M.D.1989:644).

Statistical Regression Statistical regression may also contribute to the interfering variable that lowers the validity of the results of educational experiment. Statistical regression happens because usually students with extremely high scores or extremely low scores tend to have a change in their post-test scores toward the mean. The extremely high scorers will tend to have lower scores in the post-test, while the extremely low scorers tend to have higher scores in the post- test not as affected by the experimental treatment. This effect may reduce the researchers’ confidence to conclude that the effect is only caused by the treatment. (See Borg, W.R.,Gall, M.D.1989:644).

Experimental Mortality (Attrition) This threat happens because some members of the sample withdraw from the experimental program for whatever reasons, because they are sick so they cannot attend the post test, they do not like the treatment so they quit the class, they move to another school, etc. so that the number of the members of the two groups is not equal. If all the smart students from the control group leave and all the poor students from the experimental group leave the experiment, then the two groups lose their equivalence. This threat is called Experimental mortality or attrition (See Borg, W.R.,Gall, M.D.1989:646).

Selection-Maturation Interaction If the experimental treatment and the control treatment are conducted at diferent schools which have different admission policy so that the members of the experimental group are on the average six months older than the average age of the control group, for example, then the results of the experiment is not only caused by the different treatments but also affected by the age of the students (See Borg, W.R.,Gall, M.D.1989:646)..

7 2009-2010 University of Pittsburgh-State University of Malang Experimental Treatment Diffusion If some members of the control group borrow and use the instructional materials which are supposed to be used only by the members of the experimental group, or some of the members of the control group do the assignments which are supposed to be done only the members of the experimental group, or the teacher of the control group also uses instructional materials which are supposed to be used only in the experimental group, then the experimental treatment diffusion occurs. If this happens, the results of the experiment is clouded (See Borg, W.R., Gall, M.D.1989:647).

Compensatory Rivalry by the Control Group When the members of the control group perceive that they are in competition with the members of the experimental group, they may get highly motivated to win the competition. If this happens and the research finds no difference between the average post-test scores of the experimental group and the average post-test scores of the control group, the lack of difference can be attributed to the control group’s unusual motivation rather than to treatment effects. (See Borg, W.R., Gall, M.D.1989:647). This is also one the interfering variables that lowers the validity of the findings of the experiment.

Resentful Demoralization of the Control Group If the members of the control group perceive that the members of the experimental group are receiving more desirable treatments, they may demoralize themselves so that in the post-test the control group may perform unusually lower than normal. If it turns out that there is a difference in the post-test average scores between the experimental group and the control group, then the difference is not merely caused by the different treatments (See Borg, W.R., Gall, M.D.1989:648).

Ex-Post Facto Research Ex-Post facto research measures the cause-and effect relationship without manipulating the independent variables. While the experimental research starts from manipulating and controlling the independent variables and proceeds to observing the effect on the dependent

8 2009-2010 University of Pittsburgh-State University of Malang variables, the Ex-Post Facto Research starts from identifying the existing conditions and proceeds to finding the cause of the conditions. If we observe that the students in a language department are dominated by female students, then we may conduct an Ex-post Facto research to find out whether sex influences the choice of students into language department. If we observe that most of the freshmen of the English Department graduating from State Senior high School achieve English skills better than those graduating from Private Senior High Schools, then we may want to conduct an Ex-Post facto Research to find out whether different High School background of the students affect students’ achievement in learning English skills.

Research Problems and Objectives in Experimental Research Following are examples of research problems and research objectives in experimental research.

Research Objective: This study tries to investigate the effectiveness of the use of language lab in increasing the students’ listening skill. Research problem: Do Senior High School students learning listening skill in language lab achieve better than those students learning listening skill in conventional classroom?

Research Objective: This study tries to measure the effectiveness of the One-Semester Intensive English Course for freshmen of English Department, State University of Malang. Research Problem: Do the freshmen of English Department State University of Malang have higher English skills after taking the One-Semester Intensive Course?

Research Objective: This study tries to measure the effectiveness of bilingual classes in Science classes in SMAN 3 Malang.

9 2009-2010 University of Pittsburgh-State University of Malang Research Problem: Do students of Science classes in SMAN 3 Malang achieve better after taking bilingual classes?

Research Objective: The objective of this research is to measure the effectiveness of a Master Degree training for English teachers teaching in Islamic High Schools. Research Problem: Do the English teachers teaching in Islamic High Schools have better professional performance after getting Master Degree from State University of Malang?

Concluding Remark Experimental Research is very powerful in measuring cause-effect relationship of variables. It can be used to measure the effectiveness of instructional materials and strategies by comparing with other instructional materials and strategies. The problem with the experimental research is in controlling interfering variables that may lower the validity of the research findings.

Reference Borg, W.R. Gall, M.D. 1989 Educational Research An Introduction Fifth Edition White Plains: Longman Inc.

Charles, C.M. 1995 Introduction to Educational Research Second Edition White Plains: Longman, Ltd.

Cohen, L., Manion, L. 1994 Research Methods in Education Fourth Edition London and New York: Routledge

10 2009-2010 University of Pittsburgh-State University of Malang