CONTACT Barbara N. Flagg, Ed.D., Director E-mail: [email protected]

EVALUATION SERVICES

For 28 years Multimedia Research has provided consulting services to developers of informal and formal educational products and curricula for children and adults. Our goals are to help organizations improve products and curricula as they are being formed and to evaluate the impact of final products. By collecting systematic feedback from the target audiences during the development phases of a product, we increase the appeal and effectiveness of the final product. Such formative evaluation identifies a product's strengths and weaknesses, so that clients can build on the strengths and eliminate the weaknesses. We also assess the effectiveness of the final product and curriculum in achieving its stated goals in a rigorous summative evaluation.

We work closely with clients to develop a series of national or local studies responsive to their particular needs. We understand the pressures and deadlines of product development, and we try to be both flexible and timely in our evaluation process. Our services are based on three principles:

• The evaluation process should fit the unique needs of the client organization, its products, curriculum, goals and target audiences. Each evaluation is custom-designed and implemented by Dr. Flagg. • Improvement is most likely if evaluation begins early in the development process, even at the proposal stage. • Multiple qualitative and quantitative methods are used to gather reliable and valid diagnostic data.

The evaluation project we design may include:

• Front-end analysis of audience/user/visitor characteristics or early design documents to determine content, objectives, format, interactivity, casting, and production features

• Formative evaluations of experimental materials (e.g. prototypes of software, activities, or exhibits; rough cut video or audio tapes; draft study guides, toolkits and texts) to gather data to guide revisions

• Implementation analysis of the effectiveness and appeal of the near-final product and curriculum in its designated environment with the appropriate target groups

• Summative evaluation (experimental or case study) of the impact of the product and curriculum on awareness, appeal, interest, knowledge, attitudes, behavior, and other intended outcomes.

1 1/18 BARBARA N. FLAGG

Multimedia Research Phone: 631-286-8925 33 Browns Lane Bellport, NY 11713 E-Mail: [email protected]

BIOGRAPHICAL SKETCH After teaching for a decade about design and evaluation of educational technologies in the faculty at Harvard University's Graduate School of Education, Dr. Flagg established a consulting group, Multimedia Research, which specializes in front-end, formative and summative evaluations of technology-based educational products. Clients include filmmakers, television stations, radio producers, software companies, museums and universities. Recent projects include evaluations of giant-screen and planetarium films; public television and radio series; websites for children and adults; after-school and museum outreach materials; and mobile apps.

Dr. Flagg’s doctoral degree was received from Harvard University in Human Development, and her academic research studied how children attend to and learn from media. She is the author of an award-winning textbook, Formative Evaluation for Educational Technologies.

EDUCATION Ed.D. Human Development Harvard University, Graduate School of Education B.A. Psychology Mount Holyoke College

POSITIONS • Director, Multimedia Research (1989-present) National consultants in front-end analysis, formative and summative evaluations in informal and formal educational settings • Assistant Professor, Harvard Graduate School of Education (1980-90) • Reviewer, National Science Foundation (1989-2002) • Consulting Journal Editor, "Educational Technology Research and Development" (1989-2003) • Director, Harvard Eye Movement Laboratory (1974-88)

EVALUATIONS OF CHILDREN'S TELEVISION + ASSOCIATED OUTREACH + WEBSITES

Series & Associated Websites & Outreach: Quality Family Entertainment: Quest Productions: SciSquad : , , 3-2-1 Contact, Square One TV Spiffy Pictures: Nature Cat Thirteen/WNET: Planet H2O, PBS Parents’ Guide to Early Math, Cyberchase (8 seasons) Formative evaluations available at informalscience.org TPT: DragonflyTV (7 seasons), SciGirls (4 seasons) Formative and summative evaluations available for both series at www.tpt.org/science/evaluations WCVB: A Likely Story WGBH: Where in the World is Carmen Sandiego? (2 seasons), , Peep, Martha Speaks

2 1/18 Pilots for Series: HBO: Braingames; Judge Baker Children’s Center: Willoughby’s Wonders; Kratt Brothers Company: Wild Kratts; Lancit Media: Backyard Safari; Nelvana Communications Billie and Miles; Sirius Thinking: The Little Red Caboose [pilot for Lomax: The Hound of Music]; WGBH: Magic 5&10; WNET: Artopia; WTTW: Adventures at Space Academy

Websites only: National Geographic Television: Facing the Wild: Crittercam PBS website: Ready to Learn Reading Literacy TPT: Sparticl (evaluation available at www.tpt.org/science/evaluations) WNET: Cyberchase The Next Frontier (evaluation available at informalscience.org)

EVALUATIONS OF ADULT/YOUTH TELEVISION + ASSOCIATED OUTREACH + WEBSITES

American Society for Microbiology: Intimate Strangers series Blue Penguin Productions: Parenting Works! series Chedd-Angier Productions: Scientific American Frontiers series; Science Out There pilot Education Development Center, Adult Literacy Media Alliance: TV411 series Fisher Television Productions: Savage Yard pilot Meridian Productions; TPT: NOVA: Absolute Zero series Moreno/Lyons Productions: The Mystery of Matter: Search for the Elements Museum of Science, Boston: New England Cable News Sci•Tech Today broadcasts (evaluation available at informalscience.org) Nebraska Educational Telecommunications: NOVA: Secrets Beneath the Ice National Geographic: Alien Deep series Night Fire Films: NOVA: Cracking the Maya Code Thomas Lucas Productions: NOVA: Monster of the Milky Way Twin Cities Public TV: LIFE: Part 2 series WGBH NOVA programs: Galileo’s Battle for the Heavens, Infinite Secrets, Newton’s Dark Secrets, Origins series, Runaway Universe, The Elegant Universe, Einstein’s Big Idea, Forgotten Genius, The Fabric of the Cosmos series WGBH: Building Big series; NOVA scienceNOW series; Making Stuff series WNET: INNOVATION series; Science Insight (pilot); The Human Spark series (evaluation available at informalscience.org) Screenscope, Inc: Journey to Planet Earth series, State of the Planet (evaluation available at informalscience.org) Numerous science/math/social studies Annenberg telecourses for college level and Annenberg teacher training courses with COMAP, EFC, MCET, Pacific Street Films, WNET-TV

3 1/18 EVALUATIONS OF RADIO SERIES American Association for the Advancement of Science (AAAS): Formative and summative evaluation of children’s radio series, Kinetic City Super Crew. (evaluations available at informalscience.org) Format testing of science drop-ins for teens on commercial radio, Sonic Boom

Curved Light Productions: Formative evaluations of StarTalk radio, website (2 seasons)

Earth & Sky: Summative evaluations of many seasons of Earth & Sky; “Edge of Discovery” programming; Nanotechnology series; NOAA series; NASA series. (evaluations available at informalscience.org) Pilot science radio programs for children, Kids' Earth & Sky

National Public Radio, Samanna Productions: Summative evaluations of multiple seasons of Talk of the Nation: . (evaluations available at informalscience.org) Pilot science radio program for adults, The Cutting Edge

SoundVision Productions: Pre-production assessment of show concept and planned programs Science and the Human Quest for Meaning

WNYC: Formative and summative evaluations of series Radio Lab. (evaluations available at informalscience.org)

EVALUATIONS OF COMPUTER-BASED INTERACTIVES Interactive games for websites: Cyberchase Upper elem math Thirteen/WNET-TV DragonflyTV Upper elem science TPT-TV NOVA (program specific) MS & HS science WGBH-TV SciGirls MS science TPT-TV

Interactives for home settings: Farmer Pig's Health Farm Preschool cancer prevention I.S. Grupe Newton's Apple Science TPT-TV, 3M

Interactives for museum settings: Beyond Earth, A Space Adventure Space Science IVSC Earth Over Time Earth Science IVSC The Wonderful World of Medicine Biology IVSC SET QUEST Science Career Awareness Educational Film Center Welcome to the Wrackle Factory Environment Assoc. of Sci-Tech. Centers

Interactives for school settings: Auschwitz: Inside the Nazi State HS social studies KCET-TV Race to Save the Planet Middle school science WGBH-TV Rain Forest Middle school science National Geographic The Great Solar System Rescue Middle school science Tom Snyder Productions

4 1/18 The Great Ocean Rescue Middle school science Tom Snyder Productions The Ways Cells Work Middle school science Sensimetrics Work It Out High school mathematics Envision Interactive Get the Math High school algebra Thirteen/WNET-TV Metropolis College humanities Metropolitan Arts

Interactives for work settings: Software for Management Development Program Harvard Business School 1-2-3 with Multimedia Smart Help Lotus Development Corp.

EVALUATIONS OF MUSEUM-BASED MEDIA, OUTREACH, EXHIBITS Media: Pre-production, formative and/or summative evaluations of 20 giant-screen films; 4 planetarium shows; 1 3D cinema; 1 interactive video theater show (summative evaluations available at informalscience.org) Maryland Science Museum: Formative and summative evaluations of two exhibit-related mobile games (summative evaluations available at informalscience.org) Museum of Science, Boston: Summative evaluation of MOS/New England Cable nanotechnology cablecasts (available at informalscience.org)

Outreach: Museum of Science, Boston: Impact of year-long Research Communication Laboratory on research presentation skills of graduate students (evaluation available at informalscience.org) MIT Museum, Family Adventures in Science and Technology, sponsored by MIT’s Center for Environmental Health Science: Implementation evaluation of Family Day Nanoscale Informal Science Education Network: Front-end lit review on public understanding of nanoscale STEM and nanoscale communication projects for multi-year project involving many museums and research centers using multimodal approaches to communicating nanoscale science to the public. Summative evaluations of nanotechnology awareness of visitors and impact of nanomedicine forum format. (Front-end and summative available at www.nisenet.org & informalscience.org) Pacific Science Center: Origins project: Formative evaluations of museum demonstrations, carts, workshops, community event day and NSTA teacher journals’ insert in support of Origins television series

Exhibits: Association of Science-Technology Centers: Formative evaluation of high level design of kiosk exhibit for environmental traveling exhibit: Welcome to the Wrackle Factory Boston Museum of Science: Summative evaluation of museum kiosk exhibit: Ben's Grille Design advisor for Science-by-Mail Consultant for summative evaluations of symposia, staff presentations, multimedia kiosks associated with U. Mass/Lowell Center for High-rate Nanomanufacturing grant Creative Discovery Museum, Chattanooga, TN: Storyboard evaluation of Schematic Designs of new children's museum, prior to design specifications; Evaluation of prototype exhibits Educational Film Center, NY Hall of Science, COMAP: Careers survey and prototype testing for Science, Engineering, and Technology Careers videotape and museum kiosk: SET QUEST

5 1/18 Interactive Video Science Consortium, Digital Techniques, Inc.: Formative and summative evaluations of high- level design, detailed design, and implementation of three museum kiosk exhibits: Earth Over Time; Beyond Earth, A Space Adventure; Wonderful World of Health (evaluations available at informalscience.org) Maryland Science Center, ScienCentral, Inc.: Formative evaluation of current science news videos via large screens in exhibit galleries and small screens on a handheld delivery system Montshire Museum of Science, Maryland Science Center: Summative evaluation of museum exhibits: Model of the Universe by the Powers of Ten (summative evaluation available at informalscience.org) EVALUATIONS OF MOBILE APPS AAAS: Formative evaluation of Kinetic City Active Explorer app and website

Maryland Science Center: Formative and summative evaluations for PlanetMania and Power Up! apps. (summative evaluations available at informalscience.org)

National Geographic Kids: Summative evaluation of Marine Missions iPad app with Knight Williams Research Inc. (summative evaluation available at informalscience.org)

MISCELLANEOUS National Science Foundation, SGER: Feasibility and Viability of Science Media Review Concept Exploratory research to gather information about American public’s current science media habits, their awareness of and interest in broad areas of science research and the presentation formats most likely to appeal to and reach the public on a regular basis. (report available at informalscience.org)

Girls Interest in Nature and Science Scale In support of a summative evaluation of SciGirls Season Three, Multimedia Research developed, validated and implemented a scale to assess preteen girls’ interest in and science. The work was sponsored by Twin Cities Public Television under NSF Grant No. 1323713. The GINSS is a nine statement Likert scale constructed to reveal girls’ strength of interest in nature and science activities. Researchers and evaluators are encouraged to use this scale to extend its application. See development and validation information at informalscience.org.

BOOKS PUBLISHED Flagg, B.N. (1990). Formative evaluation for educational technologies. Routledge. Received 1991 AECT Award for Outstanding Book in Instructional Development. Available via www.amazon.com (buy it used!) or directly through www.routledge.com.

ARTICLES PUBLISHED SINCE 2000 Flagg, B.N. (2016). Contribution of Multimedia to Girls’ Experience of Citizen Science. Citizen Science: Theory and Practice. 1(2), p.11. DOI: http://doi.org/10.5334/cstp.51

Alpert, C. L. & Flagg, B. (Jan/Feb. 2011). Museum-produced TV News: Small science, big impact. ASTC Dimensions, 12-13.

Flagg, B. N. (2008). Evaluating mass media. In A. J. Friedman (Ed.), Framework for Evaluating Impacts of Informal Science Education Projects (pp. 56-64). Report from a National Science Foundation Workshop available at informalscience.org

6 1/18 Flagg, B. N. (2008). Evaluating learning technologies. In A. J. Friedman (Ed.), Framework for Evaluating Impacts of Informal Science Education Projects (pp. 73-82). Report from a National Science Foundation Workshop available at informalscience.org

Flagg, B. N. (2006, Jan/Feb). Nanotechnology and the public. The Informal Learning Review, 76, 12-17.

Flagg, B. N. (2005, Nov/Dec). Can 90 seconds of science make a difference? The Informal Learning Review, 75, 2- 3, 22-23.

Flagg, B. N. (2005, Spring). Beyond entertainment: Educational impact of films and companion materials. The Big Frame. Giant Screen Theater Association. Also in The Informal Learning Review, 2005, 73, 18-21.

Holland, I. & Flagg, B. (2004). Team games. Museums Journal, 104(12), 28-31.

Saunders, C., Feierabend, J. & Flagg, B. (2002). The effects of television music instruction on kindergarten students’ music performance skills. Bulletin of the Council for Research in Music Education. No. 152.

Flagg, B. N.& Knight-Williams, V. (2002). Marketing science media to the public. The Informal Learning Review. No. 57, November-December issue.

Flagg, B. N. (2001). Gender differences in interest in contemporary science topics. The Leading Edge: Enhancing the Public Understanding of Research. Boston Museum of Science.

Flagg, B. N. (Sept/Oct. 2000). Impact of Science Friday on public radio member listeners. The Informal Learning Review, 44, 6-7.

PRESENTATIONS SINCE 2000 Flagg, B. N. with others (2017, Sept.). “Immediate impacts of giant screen films on audiences.” Presentation as part of panel “Is it more than just a film? How do we know?” Giant Screen Cinema Association conference, Chicago, IL.

Schaller, D. T. & Flagg, B. N. (2013, Apr) “Playtesting PlanetMania: A mobile game for museum exhibits.” Museums and the Web 2013 conference. Portland, OR. http://mw2013.museumsandtheweb.com/paper/playtesting-planetmania-a-mobile-game-for-museum- exhibits/

Flagg, B. N. with others (2011, Oct). “Doing stuff together: A forum on coordinated national outreach partnerships.” ASTC annual meeting. Baltimore, MD.

Flagg, B. N. (2008, Oct). “Summative evaluation of NISE Net’s Forum: Nanotechnology in Health Care.” ASTC annual meeting, Philadelphia, PA.

Flagg, B. N. (2008, Mar). “How DragonflyTV teaches science inquiry.” NSTA annual meeting, Boston, MA.

Flagg, B. N. (2005, Feb). “Mass media and public understanding of current science research.” AAAS annual meeting, Washington, DC.

7 1/18 Flagg, B.N. (2005, Feb). “Communicating current research in museums.” Fourth Annual Japan/US Conference on the Public Understanding of Research, National Academy of Sciences, Washington, DC.

Flagg, B. N. (2004, Sept). Hear the voices of your audience. Giant Screen Theater Association annual meeting, Montreal, Canada.

Flagg, B.N. (2003, July). Listening to the public. Third Annual Japan/US Conference on the Public Understanding of Research, Museum of Emerging Science and Technology, Tokyo, Japan.

Flagg, B.N. (2001, Feb) Gender differences in interest in contemporary science topics. The Leading Edge Workshop, Boston Museum of Science.

8 1/18