A Twenty-Year Survey of Among College Undergraduates

By Chris Impey, Sanlyn Buxner,Jessie Antonellis, Elizabeth Johnson,and Courtney King

Firstresults from a 20-year survey of science knowledge and attitudes toward science among undergraduatesare presented. Nearly 10, 000 students taking astronomy as part of a general education requirement answered "aset of questions that overlap "ascience literacy instrument administeredto the generalpublic by the National Science Foundation. The questions addressed are: What is the level of science literacy among undergraduates,and what variables or attributespredict science literacy? Their attitudes towardscience and pseudo-science were probed by a set of 22 statements coded on a . On the knowledge items, freshmen perform overall. The National Academy of only marginallyhigher than the graduate science plays an Scholars surveyed science curricula generalpublic, with the exception important role in our society. used in bachelor of arts degrees from of largepositive differences in their nyoneIf they teachwho scienceteaches majors,under- the top 50 institutions ranked by knowledge of evolution and the A the they fulfill the need for a technically Big Bang. Gains on any particular U.S. News and World Report; the trained workforce by preparing stu- percentage having science item through the time that students require- dents who are numerate and who have ments dropped from 90% in 1964 to graduateare only 10%-15%, despite inquiring and analytic habits of mind. 34% in 1993, and the percentage with the fact that they have taken two If they teach nonscience majors, they both math or three science courses. Belief in and science requirements often provide the last formal exposure dropped over the same period from runs high, and thefact to the evidence-based reasoning and 36% to 12% (Balch and Zurcher 1996). that the level ofpseudoscience belief ideas that have transfontned our under- A U.S. Department of Education does not correlatewell with the level standing of the natural world. analysis of the "empirical curriculum" ofscience knowledge is particularly Advanced math and science are found that science accounted forjust 7 noteworthy. In addition, no variable voluntary in most high schools; a of the 100 course categories with the in the analysis is predictive of majority of students do take , most undergraduates, whereas a third science literacy.Over the interval but less than a half take , a of all future school teachers do not 1988-2008, there s no detectable quarter take , and only I in 20 take any college-level math (Adelman improvement in undergraduate take calculus (Blank, Langesen, and 2004). In this paper, we consider the scientific literacy. Petersen 2007). At a university, stu- science knowledge and attitudes of a dents face a smorgasbord of choices, cohort that is typical of the 80% of the and despite the popularity of general students at a major Land Grant public education science requirements, sci- university who graduate with degrees ence is being taken less frequently in nontechnical subjects.

Vol. 40, No. 4, 2011 31 Scientific literacy views on change, stem cell of scientific ingenuity without any Science and form an research, nuclear power, and the real knowledge of how they work. "amniotic fluid" around all who live space program. They compete for For good or for ill, science is pro- in the industrialized world and, in- jobs in technology-driven sectors foundly shaping human culture, yet creasingly, in any part of the world. of the economy that didn't exist a any ability to assimilate the insights Americans encounter science in this generation ago. They are consumers of scientists into the natural world country in their roles as citizens, of ubiquitous high-technology de- depends on a foundational under- workers, and consumers. They vote vices and are mostly content to use standing of how scientific knowl- for political candidates with diverse these tools and other manifestations edge is gained, how the scientific

The first part of the two-sided questionnaire administered to students in introductory astronomy classes at the University of Arizona in Tucson.The knowledge-based objective response questions include nine that are part of the instrument administered by the National Science Foundation and reported biannually in the Sci- ence and EngineeringIndicators series.The four open-ended questions were transcribed and coded by newly constructed schema.

"* Are you male _ or female _ ? "• Are you a freshman -, sophomore _-,junior - or senior_ ? "• What is your major _ and estimated GPA ? "* How many science courses have you taken at the Uof A _ ? "* What does it mean to study something scientifically? "• Would you say that is very - sort of , or not at all scientific ? "* The oxygen that we breathe comes from plants.True or False? "• Lasers work by focusing sound waves.True or False? "* Electrons are smaller than atoms.True or False? "* The universe began with a huge explosion.True or False? "* The continents on which we live have been moving apart for millions of years, and will continue to move in the future. True or False? "* Humans, as we know them today, evolved from earlier species of animals. True or False? "• The earliest humans lived at the same time as the dinosaurs.True or False? "* Which travels faster, light __ or sound ___? "• Does the Earth go around the Sun - ,or does the Sun go around the Earth - ? " Does the orbit in the previous question take one day _,one month _,or one year__ ? "* What is DNA? "* What is radiation? "* If the rate of inflation is falling, are prices decreasing -, level -,or increasing _ ? "* Radioactive milk can be made safe by boiling it.True or False? "• Antibiotics kill viruses as well as bacteria.True or False? "• Is all radioactivity manmade __ or does some occur naturally __ ? "* Adoctor tells a couple that they have a one in four chance of having a child with an inherited illness. Does this mean that, a. Ifthey have only three children, none will have the illness.Yes or No? b. If their first child has the illness, the next three will not.Yes or No? c. Each of the couple's children will have the same risk of suffering the illness. Yes or No? d. Ifthe first three children are healthy, the fourth will have the illness.Yes or No? "* Briefly, define computer software. "* Which is the largest contributor to heart disease: smoking -, eating a lot of animal fat -, stress , not getting enough exercise _,or lack of vitamins _?

32 Journal of College Science Teaching Survey of Science Literacy enterprise proceeds, and how to dis- Scientists in different fields might beyond their self-interest. To make tinguish scientific facts from other not agree on a core set of principles informed decisions, Americans need kinds of information-a collection and facts that every citizen should to have assimilated enough from of skills and knowledge commonly know, but they are probably in accord their education to use evidence-based referred to as science literacy. on the fact that science literacy goes reasoning to separate substance from

The second part of the questionnaire has 22 statements, with responses on a Likert scale.The items were newly created for this survey.

Overall, the progress of science and technology has been beneficial to El EII 0l El 1 E I our Civili7;tifnn I

I There are phenomena that physical science and the laws of nature l 1 El 1 El 1 cannot exolain. o1 i I

Sui-us are real anc snloulic ie investigatecl. I L I LiI L I Li I Li I

I Some people possess powers. 1 EI 1 i El El1

There are some circumstances when medical science should not be El El El El El used to prolong life. The Unives ws created in a nro' explo ionblin of years I 011 11111

Some ancient civilizations were visited by extraterrestrials.. El 0 E 11 E Compteswl evenually be nelgn nuht thn l~ikeumans. 11 11 1 lE Technology has too much control over our lives. 0 E El El E Scientists should tak repni itfothe bad effect oftheir theorie 0> 13 011 an netons. L The government should strongly support the manned space program. El [] El El Gen engiens god . i 0 p We should devote more of our money and scientific resources to repair El El El El E damage done to the environment. Pure siece shoudbe uneregarles of it akoImmdaeT1

Science will come up with a way to dispose of toxic waste. El El El El Wehoud xert morecotrol ove thpula epn evlpdb 3 00

Faith healing is a valid alternative to conventional medicine. El El El 11 E We shoud makea cnerted ffottserhfrlfonterpat. 1 0ED0 Scientists should be allowed to do research that causes pain to animals, El E] El o1 El if it helps solve human health problems.

Vol. 40, No. 4, 2011 33 spin and cull corroborated fact from gitudinal data available. The Na- lary of basic scientific constructs, unsubstantiated assertion. The health tional Science Foundation (NSF) (2) an understanding of the process of civic society depends in part on has commissioned a survey of sci- or nature of science inquiry, and (3) science literacy. ence knowledge and attitudes for a level of understanding the impact August bodies have weighed in. 20 years as part of the Science and of science and technology on indi- The Organization for Economic Engineering Indicators series, the viduals and on society. He estab- Cooperation (OECD 2002) defines biannual report to the National Sci- lished a threshold based on (1) cod- it as understanding key scientific ence Board that is used to shape na- ed responses to the question "What concepts and frameworks, the meth- tional research and education policy does it mean to study something ods by which science builds ex- and guide workforce development scientifically?" (2) the recognition planations based on evidence, and in technical fields. Jon Miller, who that astrology is not at all scientific, how to critically assess scientific conducted the surveys on behalf of and (3) correct answers to six or claims and make decisions based the NSF for many years, set the cri- more out of nine knowledge ques- on this knowledge. The American teria of literacy to be (1) a vocabu- tions (Miller 1987). Association for the Advancement of Science (AAAS; 1993) states that scientifically literate citizens should be aware that science, , Responses to subset of items on the science literacy instrument. and technology have strengths and These items form the basis for NSF knowledge-based metric of limitations and are interdependent public sdence literacy. UA freshmen have not taken any science human enterprises; should recog- classes, whereas UA general education students have taken two to nize the unity and diversity of the three science classes. Sample sizes are 1,864 for NSF, 1,275 for UA natural world; and should be able freshmen, and 828 for UA general education; data is from 2006 for to use scientific ways of thinking NSF and 2004-2008 for UA.The anomalous result on astrology-the and knowledge for individual and very low performance of the college age cohort-seems real, because responses to the separate Likert scale are consistent. social purposes (AAAS 1993). The UA = University of Arizona; Gen Ed = general education; NSF = National Research Council (NRC) National Science Foundation. has even more lofty goals for the schools: Education should promote Antibiotics kill viruses as well as bactena. science as one of the pinnacles of (false) human thinking capacity; provide Humans evolved from earlier species a laboratory of common experience of animals. for developing language, logic, and (true) problem-solving skills; and prepare The continents have been moving apart for milions of years. students for a democracy, which de- (true) mands that its citizens make personal All radioactivity is manmade. and community decisions about sci- (false)

entific issues (NRC 2007). Electrons are smaller than atoms. The goal of this paper is to describe (true) o UA Gen Ed a large survey of undergraduate sci- "*UA Freshman Lasers work by focusing sound waves. "*NSF ence literacy at one public university, (false) with homogeneous data spanning 20 years. The data is then used to mea- The earth goes around the sun. (true) sure the level of science literacy of this population and discover which, if The orbit takes one year. any, variables or attributes are predic- (true)

tive of science literacy. Astrology is not at all scientific. (true) Previous The universe began with a huge explosion. There are many ways to frame the (true) general principles described above into a definition and measurement; 0 20 40 60 80 100 therefore, this paper focuses on a Percent Correct survey that tethers to the best Ion-

34 Joural ot College Science Teaching Survey of Science Literacy

Methodology ence literacy? and What variables are graduates' opinions toward science The first author of this paper (CI) has predictive of undergraduates' science and their belief in pseudoscience. been interested in science literacy for literacy? The purpose of this initial Responses are coded on a 5-point 23 years as an astronomy professor look is to establish a baseline of sci- Likert scale, with no opinion instead at the University of Arizona (UA) in ence literacy for this population of of the more traditional neither agree Tucson. Since 1988, he has adminis- undergraduates compared with NSF's nor disagree as a middle item. This tered a survey instrument to the stu- national sample and, more specifically, part of the survey was devised by one dents in general education astrono- to investigate Miller's (2007) conclu- of the authors (CI) when the survey my lecture classes, taught by himself sions about the changes in the national began in 1988. and his colleagues. The introductory sample. This analysis is a precursor to The survey instrument has been astronomy course counts for a third suggestions for the improvement of stable for over 20 years and has been of the science requirement. About teaching science to nonscience majors administered in the same fashion one in five of all undergraduates at at the college level. throughout that span. It should be not- the university take the course, which Although the survey is anony- ed that although there is a substantial is primarily composed of freshmen mous, the following demographic overlap of knowledge questions with and sophomores. Only 5% of the en- information is collected: gender, the NSF instrument, NSF calls respon- rollment consists of science majors. major, self-reported GPA, class stand- dents by telephone at their homes, as The survey is given in the first week ing, and number of science courses opposed to obtaining written responses of class; it is anonymous and volun- taken at the university. The instru- in a classroom setting. Questionnaires tary and doesn't count for any part ment is administered on paper, and were typically given out with classes a student's grade. Typically 10 to 15 subsequent data entry and coding are of 100 to 150 students, using two to minutes are allowed for its comple- done by hand. Figure 1 presents the three classes per semester, giving tion. Only 3% of the students choose knowledge-based part of the question- an average of about 500 completed not to complete the survey. Over 20 naire. Most of the science content questionnaires per year across the 20 years, almost 10,000 surveys have questions are true/false, yes/no, or years of the project so far, with fairly been collected and used for analysis multiple choice. Four are open re- uniform time coverage. in this study. sponse. Nine of the questions overlap The rich data set enables us to with the much more extensive NSF Data and analysis address a wide range of research Science Indicators instrument. Figure The enormous job of data entry was questions; this paper summarizes 2 shows the attitudinal, second half carried out by two of the authors (EJ the survey and provides preliminary of the questionnaire, comprised of 22 and CK) over 18 months from 2007 results to the overarching questions: statements about science, technology, to 2009. The rate of incomplete data, What is undergraduates' level of sci- and society designed to probe under- spoiled or bad data, and illegible an-

Overall responses, by nearly 10,000 undergraduates over 20 years, to 6 out of 24 statements about science and technology.This is the group of items relating to pseudo-science or nonscientific beliefs. These beliefs coexist with solid performance on science knowledge questions.

There are phenomena that physical scence The position of the planets and the laws of have an influence on the Some people possess nature cannot explain events of everyday life psychic powers 6o

.1so

CD40 30

% 20

20

0 iu• 2. 'd5 z 5 i2 e 0 -1 •n 05a

Vol. 40, No. 4, 2011 35 swers or facetious or obviously frivo- Initial results The generally high level of science lous responses amounted to around UA students got an average of 7.2 knowledge is reassuring, but the gain 3% overall. An independent cross- (SD = 1.5) items correct out of the during college amounts to only one ad- check of a sample of the data showed 9 items shared on the NSF survey ditionally correct item among the core that error rate for data entry was below (80%) and an average of 11.2 (SD = set of nine from the NSF instrument, 0.1%. Although the survey included 2.3) correct overall out of 15 (75%; and the gaps that remain are disconcert- both forced-choice and open-response the astrology and the inflation ques- ing. The data show an important gender questions, analysis of the forced- tions were left out). effect, with slightly better performance choice questions is included here. There was an overall stability of by men on the scientific knowledge Descriptive analysis began by questions that were answered cor- questions. However, this is the only looking at responses to each question, rectly over the 20 years. Figure 3 demographic factor or variable that cor- both knowledge as well as belief and compares the UA sample of both relates with performance; it is striking attitude questions, by each demo- pre-general-education requirement that there seem to be no good predictors graphic variable. Next, composite (students equivalent to high school of science literacy. Results from this scores were generated on the basis graduates in terms of science courses), time-limited sample of the UA data are of both overlapping questions with post-general-education requirement representative of the full data in which the NSF survey and scores based on (students having completed at least UA students perform better than those all knowledge questions for each in- two university science courses) to the in the NSF sample but do not show dividual. Additionally, the belief and latest published NSF sample (National a marked improvement after taking attitude questions were grouped into Science Board 2008). The UA sample university science courses. Using a full sets using a combination of expert is truncated (2004-2008) to be more baseline of 1988 to 2008, there's no de- grouping by scientists and science comparable to the NSF sample. Figure tectable improvement in undergraduate educators, exploratory factor analysis, 3 shows that entering freshmen score scientific literacy over 20 years. This and cross-validating through confirma- marginally higher than the general contrasts with Miller's (2007) finding tory factor analysis. Both descriptive public, with the exception of large for the general public and will be the and regression analysis were used to positive differences in their knowledge subject of future analysis. determine the relationship between the of evolution and the Big Bang. On the attitudinal part of the survey, demographic variables and belief and The gains in knowledge on any students are generally well-disposed attitude indicators on the one hand, and particular item through the time that toward science and technology and students' performance on individual students graduate are only I0%-l 5%, 93% agree or strongly agree with the (and aggregated) science knowledge despite the fact that they have taken statement that "Overall, the progress questions on the other hand. two or three science courses by then. of science and technology has been

I;[llF ;IGR5

The analysis of 9,200 questionnaires for true/false reactions to a statement about antibiotics and 9,300 true false reactions to a statement about astrology shows a modest but statistically significant gender effect (fewer students answered the question about antiblotics). here%a larger knprovement with the number of science courses taken on astrology (from a low base) than on antibiotics. Students who have taken four or more science courses are mostly science majors. Soc Beh Sci = Social Behavioral Science Gen Ed = general education.

Antibiotics kill viruses as well as bacteria Astrology is not at all scientific 100 100 90 90 so 80

70 70 ______60 ~60- ~4040

(L 20 2

36 Journal of College Science Teaching Survey of Science Literacy beneficial to our civilization." Belief in although it is piecemeal and barely course-taking, 1972-2000. Washing- pseudoscience runs high. About 40% adequate to make students familiar ton, DC: U.S Department of Educa- believe that the positions of the planets with the major achievements of sci- tion. Unpublished report available affect everyday life, and the same per- ence in the past century. After that, online at www.ed.gov/rschstat/re- centage think some people have psy- college science instruction produces search/pubs1empircurr/index.html. chic powers. About one in six believe incremental gains, but students still American Association for the Advance- that aliens visited ancient civilizations, reach the end of their formal educa- ment of Science (AAAS). 1993. one in four think that faith healing is a tion with substantial holes in their Benchmarksfor science literacy. legitimate alternative to conventional knowledge and understanding of New York: Oxford University Press. medicine, and a quarter think that some science. It is up to science educa- Balch, S.H., and R.C. Zurcher. 1996. numbers are lucky for some people tors and higher education leaders to The dissolution of general educa- (Figure 4). One striking aspect of the decide whether it is an acceptable tion: 1914-1993. Princeton, NJ: analysis is a null result: None of these outcome when a significant fraction National Academy of Scholars. Un- beliefs is strongly correlated with level of college graduates are unaware of published report available online at of science literacy. major tenets of life and physical sci- www. nas.orglpolReports. cfm ?Doc The results for students who have ence and also hold persistent pseu- Id=113. finished all of their science require- do-scienctific belief systems. Blank, R., D. Langesen, and A. Pe- ments are disconcerting. One in three Future papers will closely exam- tersen. 2007. State indicatorsof think that antibiotics kill viruses as ine (1) the relationship between stu- science and mathematics education. well as bacteria, one in four think that dents' beliefs and attitudes and how Washington, DC: Council of Chief lasers work by focusing sounds waves, these attitudes can correlate with State School Officers. one in five think that atoms are smaller science knowledge, (2) the patterns Miller, J.D. 1987. Scientific literacy in than electrons, and one in five either of student belief in pseudoscience in the United States. In Communicating do not believe or are unaware that general and belief in astrology in par- science to the public, eds. D. Evered humans evolved from earlier species ticular, (3) a study of the responses and M. O'Connor, 19-40. London: of animals and that the Earth goes to the open-ended questions and a Wiley. around the Sun in a year. Only one in cross-calibration of undergraduate Miller, J.D. 2007. Civic literacy across five undergraduates say that astrology science literacy with science lit- the life cycle. Unpublished paper is "not at all" scientific, although that eracy in the general public, and (4) presented at the annual meeting of fraction increases from 17% to 34% as separate analysis of the differences the American Association for the they move through the university. The between the way science is viewed Advancement of Science, San Fran- Likert scale item on the position of the by students and by professional sci- cisco, CA. planets is an important cross-check on entists. Another paper will present a National Research Council (NRC). the astrology question because it does new coding scheme for responses to 2007. Taking science to school: not use the word astrology and students the open-ended question "What does Teaching and learningscience in might plausibly confuse astrology and it mean to study something scien- grades K-8. Washington, DC: Na- astronomy. Equally troubling, half of tifically?" in order to derive a more tional Academies Press. all science majors say that astrology is detailed view of the way students National Science Board. 2008. Sci- either "sort of' or "very" scientific (see conceive of scientific information ence and engineering indicators Figure 5). In this case, it is possible that and the process of science. E 2008. Vols. I and 2. Arlington, VA: they are aware of astrology's roots in National Science Foundation. observational astronomy, which means Acknowledgments Organization for Economic Cooperation the astrology item must be interpreted We acknowledge many conversations on and Development (OECD). 2002. with caution. Education majors-the educational issues with Ed Pratherand Education at a glance. Paris: OECD. cohort of future teachers-perform Gina Brissenden. The first author (CI) worse than the average on almost all is grateful.for the assistance of his col- Chris Impey ([email protected]) the individual questions and in terms leagues in the Astronomy Department is a University DistinguishedProfessor in of overall scientific literacy. for over 20 years, who gave up some of the Department of Astronomy, Sanlyn their class time to administer the ques- Buxner and Jessie Antonellis are grad- Summary tionnairein their classes. uate students in the College of Education, Overall, it appears that high school and Elizabeth Johnson and Courtney education and students' exposure References King are undergraduatestudents in the to media and popular culture con- Adelman, C. 2004. The empirical cur- College of Letters,Arts, and ,all at vey a basic knowledge of science, riculum: Changes in post-secondary the Universityof Arizona in Tucson.

Vol. 40, No. 4, 2011 37 COPYRIGHT INFORMATION

TITLE: A Twenty-Year Survey of Science Literacy Among College Undergraduates SOURCE: J Coll Sci Teach 40 no4 Mr/Ap 2011

The magazine publisher is the copyright holder of this article and it is reproduced with permission. Further reproduction of this article in violation of the copyright is prohibited. To contact the publisher: http://www.nsta.org/college