The Evaluation of Bilingual Education Programs for Language-Minority, Limited-English-Proficient Students: a Status Report with Recommendations for Future Development
Total Page:16
File Type:pdf, Size:1020Kb
DOCUMENT RESUME ED 321 557 FL 018 677 AUTHOR Tallmadge, G. Kasten; And Others TITLE The Evaluation of Bilingual Education Programs for Language-Minority, Limited-English-Proficient Students: A Status Report with Recommendations for Future Development. INSTITUTION RMC Research Corp., Mountain View, Calif. SPONS AGENCY Department of Education, Washington, DC. PUB DATE Sep 87 CONTRACT 300-85-0140 NOTE 232p.; For related documents, see ED 291 650 and FL 018 675. PUB TYPE Guides - Non-Classroom Use (055) EDRS PRICE MF01/PC10 Plus Postage. DESCRIPTORS *Bilingual Education Programs; Elementary Secondary Education; English (Second Language); *Evaluation Methods; *Limited English Speaking; *Measurement Techniques; *Program Evaluation; Research Design; *Research Methodology; Statistical Analysis; Test Validity ABSTRACT A discussion of the evaluation of bilingual education programs focuses on building a comprehensive framework for local efforts at evaluation. The discussion begins with an introduction to the legislative history of bilingual education programs and the evolution of their evaluation. This is followed by a review of the literature on current practices and problems in program evaluation, looking at the kinds of inferences that caa be drawn from program evaluations and the threats to the validity of those inferences. Strategies for redcing threats to validity ar..: examined. A chapter is devoted to treatment, student, and setting variables that have been identified as potentially interactive on the basis of either theoretical formulations or empirical findings. Lists of the variables and methods for obtaining and documenting relevant information are presented. Four sources of systematic error associated with simple measurements of growth are discussed, and types of tests and other measures for assessing bilingual education program impact are examined. Eight evaluation designs reported in the literature are described, and problems associated with them are reviewed. Several approaches to the measurement of outcomes on a common scale are evaluated. A bibliography of over 400 items is included. (MSE) *********************************************************************** Reproductions supplied by EDRS are the best that can be made from the original document. *********************************::************************************* Phase I Report RESEARCH CORPORATION The Evaluation of Bilingual Education Programs for Language-Minority, Limited-English-Proficient Students: A Status Report with Recommendations for Future Development U.S. DEPARTMENT OF Office of Educahonal EDUCATION Research and Improvement EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) )1Trusdocument has been G. K. Tallmadge received kern the personreproduced as originating it or oroanozabOn T. C. M. Lam O Minor changes have been made to improve N. N. Gamel reproduchon Quality Pants of mew or opuuonsstated In this docu- ment do nor neceSsantyrepresent othcol OERI posmon or palmy September 1987 RMC Resesrch Corporation 2570 West El Camino Real, Mountain View, CA 94040 Prepared for the U.S. Department of Education N Washington, D.C. N 00 BEST COPY AVAILABLE 1.L 9 THE EVALUATION OF BILINGUAL EDUCATIOr T PROGRAMS FOR LANGUAGE-MINORITY, LIMITED-ENGLISH-PROFICIENT STUDENTS: A STATUS REPORT WITH RECOMMENDATIONS FOR FUTURE DEVELOPMENT G. K. Tallmadge Tony C. M. Jam Nona N. Gamel September 1987 Prepared for: U.S. Department of Education The research reported herein was performed pursuant to Contract No. 300-85-0140 with the U.S. Department of Education. Contractors undertaking such projects un- der government sponsorship are encouraged to express freely r professional judgment in the conduct of the project. Points of view or opinit..,tated do not, therefore, necessarily represent official U.S. Department of Education positionor policy. 3 TABLE OF CONTENTS Page List of Tables and Figures Acknowledgments vii SUMMARY AND RECOMMENDATIONS 1 The Need for Improved Evaluation Practices 2 A. Validity-Based Framework for Evaluation 2 Documenting Treatment, Student, and Setting Characteristics 6 Measuring Growth 8 Establishing Cause-Effect Relationships Between Treatments and Outcomes 13 Aggregating Data and Making Effectiveness Comparisons 14 1. INTRODUCTION 17 Legislative History 18 History of Bilingual Education Evaluations 23 Purpose, Objectives, and Scope of This Report 26 2. REVIEW OF CURRENT PRACTICES AND PROBLEMS IN THE EVALUATION OF BILINGUAL PROGRAMS 31 Secondary Analysis of the Quality of Bilingual Education Evaluation Reports 33 Sources of Methodological Problems in Bilingual Education Evaluations 38 Summary 50 Discussion and Recommendation 51 3. A VALIDITY-BASED FRAMEWORK FOR BILINGUAL EDUCATION EVALUATION 57 The Meanings of Research Validity 57 Construct Validity 58 Threats to Construct Validity 64 Internal Validity 68 Statistical Conclusion Validity 76 External Validity 82 Relationships Among and Priorities'of Validities 87 4. TREATMENT, STUDENT, AND SETTING VARIABLES IN BILINGUAL EDUCATION EVALUATION 91 Treatment Characteristics 92 Student Characteristics 102 Setting Characteristics 112 Measuring and/ or Documenting Treatment, Student and Setting Characteristics 115 iii 4 Page 5. MEASURING ACHIEVEMENT AND/OR AFFECTIVE GROWTH 119 Components of Systematic and Random Growth 120 Instrument Selection/Development 135 6. EVALUATION DESIGNS 149 True Experiments 150 Non-Equivalent Comparison Group Designs 153 Regression-Discontinuity Designs 156 Time Series and Quasi Time Series Designs 159 Value-Added Designs 163 Norm-Referenced Designs 164 The Gap-Reduction Design 166 Group Criteria-Mastery Designs 170 Summary 174 7. COMPARABILITY, AGGREGATION, AND A COMMON GROWTH METRIC 177 Effect Size 177 Observed Growth versus Treatment-Related Growth 180 REFERENCES 183 iv List of Figures and Tables Page Table 1. Percentage of Studies Accepted for Review on Effectiveness of Bilingual Education 37 Table 2. List of Characteristics that Should be Documented in Evaluation of Bilingual Education Programs 103 Table 3. List of Student Characteristics that May Interact with Treatments in Bilingual Education Programs... 111 Table 4. List of Setting Characteristics that Should be Documented in Evaluations of Bilingual Education Programs 115 Table 5. Characteristics of Eight Evaluation Designs Considered for Title VII Applications 175 Figure I. Causal relationship among various factors and the technical quality of evaluation practices 51 Figure 2. The regression-discontinuity design showing a substantial treatment effect 157 Figure 3. Different size psuedo-effects resulting from different placements of the cutoff score when linear models are fitted to a curvilinear regression function 158 v ACKNOWLEDGMENTS The first draft of this report was reviewed by eight "external" readers: William H. Angoff - Educational Testing Service Karen E. Banks - Oregon Total Information System Thomas D. Cook - Northwestern University Norman C. Gold - California State Department of Education Luis M. Laosa - Educational Testing Service Robert L. Linn - University of Illinois Beverly B. McConnell - Private Consultant Ann C. Willig - University of Texas at Austin Other reviewers included, from the U.S. Department of Education, James J. English (Project Officer for the study) Janice K. Anderson Keith A. Barer John Chapman Edward J. Fuentes Robert L. Kastner and, from the Council of Chief State School Officer's Committee for Evaluation and In- formation Systems, Gerald E. De Mauro Their comments and suggestions were extremely helpful and are reflected in this, the third revision of the report. There were exactly 100 significant, substantive comments and many more minor ones. We have attempted to respond to all of them, but few of our reviewers (none of the external reviewers) have had a chance to examine our efforts to incorporate their suggest- ions. Whatever deficiencies remain are therefore the sole responsibility of the authors. vii Not all of our reviewers agreed with each other, and there continue to be several points on which the authors themselves are not in total accord. On all of these issueswe tried to reach compromises that we felt were fair to all points of view. The senior authoras the final arbitrator, however, is responsible for any failures to achieve a balanced presenta- tion. We would like to thank all of our reviewers for their thoughtful readings and insight- ful comments. Because of their valued input, this report is substantiallymore complete, more technically accurate, and more in tune with real-world bilingual education issues than our earlier draft. GKT TCML NNG SUMMARY AND RECOMMENDATIONS What we call bilingual education in the United States is quite different from what the rest of the Western world calls bilingual education. Here the term con- notes special programs which are designed for non- and limited-English-proficient, language-minority students and which have two primary objectives: (a) to develop these students' English language skills and (b) to prevent them from falling behind their fully English-proficient peers in other content areas. The students' native lan- guage may or may not be taught as an academic subject, but it often serves as the medium of instruction in classes for students whose proficiency in English is too limited for them to benefit from instruction presented in English. When the native language is taught as an academic subject, the rationale is usually that developing native language proficiency first will facilitate and enhance the subsequent acquisi- tion of English.