Podcasting Lectures Formative t some point in their educations, help them better retain the biomedical students must learn copious information presented. evaluation Aamounts of information. To do The Office of Dental Informatics is this, they use a variety of well-known responsible for the development, imple- strategies helped strategies such as study groups, note- mentation, and evaluation of learning- taking services, and videotapes of lec- technology activities in the school. This identify a solution tures. In fall 2004, a group of first-year includes faculty development, course Web dental students at the University of pages, research with and about learning to a learning Michigan (U-M) School of Dentistry technology, and the formative and sum- asked to have all dental school lec- mative evaluation of learning technol- dilemma tures videotaped and recordings made ogy projects. The office is also respon- available on a Web site. The students’ sible for the acquisition, installation, and doubted their ability to accurately operation of classroom and computer lab By Sarah Brittain, summarize in their notes the quantity equipment. Pietrek Glowacki, of information presented in lectures. In response to the students’ request, the Jared Van Ittersum, The students thought that reviewing a Dental Informatics group applied forma- and Lynn Johnson video recording of each lecture would tive evaluation strategies to determine
24 EDUCAUSE QUARTERLY • Number 3 2006 the ultimate solution. The group deter- views and focus groups. This information Figure 1 mined that podcasting (see the sidebar) guides design decisions. audio recordings of lectures provided a Instructional Design The Design stage consists of creating a better technology solution for the stu- Process prototype, flowcharts, and storyboards. dents’ needs than the originally requested Feedback is gathered by assessing the user video recordings. audience’s attention, comprehension, information retention, personal involve- Formative Evaluation and ment, and user-computer comfort.4 These Instructional Design data guide revisions during production Instructional design and formative eval- of the product, potentially eliminating uation strategies are commonly used in costly and time-consuming changes at a developing instructional products, espe- later stage if the product does not meet cially for computer-based instruction and user needs. other learning technologies. Formative During the Development stage, the evaluation provides information to help product is created, tested, and revised monitor and improve product develop- until the client is satisfied.5 Formative ment to ensure that it meets its intended evaluations occasionally are used outside goals.1 tion. When trying to determine whether of the design, development, and imple- Instructional design consists of several cognitive or behavioral skills have been mentation stages for a product. Often it stages (see Figure 1). In each stage, design- affected, tests are generally used. Occa- is difficult to find funding, resources, and ers use formative evaluation techniques to sionally records and documents, such as support for the development of a product obtain feedback concerning the product server access log files, are used to gather without results showing that it will effec- from clients, subject matter experts, col- information on the frequency of down- tively accomplish its intended goal.6 The leagues, and learners.2 Feedback gathered loads—data that can give an idea of how results of a formative evaluation can serve via formative evaluation in one stage of much a product is being used. as an indicator of a product’s success with the instructional design process is used In the Define stage of instructional its intended audience, which in turn can in subsequent stages to help improve the design, developers begin defining the tip the scale toward securing resources product. Formative evaluation informa- scope of the learning activity, identifying needed for further development. tion is collected in four ways: self-report- learner characteristics, establishing con- Instructors often hesitate to integrate ing, observation, tests, and records.3 straints, and collecting resources. During new products or technology into their Self-reporting, the most commonly used this stage, the learning technology team courses without evidence that it will method, refers to users’ directly relaying members, who have limited knowledge benefit student learning. Information their experiences with or opinions of the of the subject matter and the intended retrieved from a formative evaluation can product, generally by means of a ques- audience, commonly make many of the help them determine whether a product tionnaire, survey, or interview with an design suggestions. They gather infor- should be implemented. evaluator. Observing users’ behavior and mation about the target audience’s prior Once the product has been deployed interaction with the products is another knowledge, interests, and experiences in its intended setting, formative evalua- popular method for collecting informa- with the subject matter through inter- tion can serve several different purposes. Developers can use feedback to make small improvements that were not antici- pated during the initial development, for Podcasting example. Feedback can also help steer Podcasting is a new technology with an evolving definition. The term is gener- future iterations of the product. If the product will be used in a different envi- ally considered to be derived from combining the words iPod and broadcasting. ronment or with a different audience, Podcasting involves making audio and video files available for download on a information gathered during this part of routine basis via subscription. the evaluation can guide reconfiguration Two important characteristics set podcasting apart from downloading audio of the product. and video from a Web site: (1) the routine and regular addition of new content; and (2) the ability to automatically receive new content that you select through Formative Evaluation: subscription. To subscribe to a podcast requires only a single interaction in which A Case Study Formative evaluation strategies are the user chooses to have updates downloaded automatically. Podcasts, like broad- used routinely when developing com- casts, deliver new content regularly. Wikipedia provides a thorough discussion of puter-based instructional programs podcasting (http://en.wikipedia.org/wiki/Podcast). such as patient simulations or tutorials. When students requested that lectures
Number 3 2006 • EDUCAUSE QUARTERLY 25 be videotaped and made available on a tion 3. We examined questions 4, 5, and ing the survey. For records, we looked at Web site, they didn’t get an immediate 6 across all three pilots. server logs to gauge the frequency with decision to grant or deny the request. Certain constraints placed on the proj- which students used each of the three A research university such as the U-M ect contained costs and ensured student media types. Finally, records were kept of values inquiry. This culture made it involvement. First, students were respon- the time spent by the technical staff on natural to apply strategies to system- sible for obtaining instructor permission the project in order to calculate cost. atically determine whether videotaping to record lectures. Second, students had was the best solution. We conducted a to provide their own playback devices. Pilot 1 Results. This pilot focused on series of three pilot studies, using forma- The school provided technical support. A determining the best media format for tive evaluation and instructional design university grant from the provost’s office lectures (question 1). The three formative techniques to guide the process. Flagg7 funded the few additional expenses. evaluation measures—student survey, described four types of formative evalua- focus group, and server logs—showed tion measures, two of which we used in Pilot 1—Media Format that students preferred the audio-only this project—self-reporting and records. The focus of this pilot was to answer the format. Of a possible 105 participants, No tests were administered, nor were question, What is the best media format 30 downloaded media directly from the observations of students conducted. for lecture review? Web server. Server logs revealed that 20 Students initiated and supported the percent of downloads were video, 14 project, so were equal partners from the Pilot 1 Methods. The Advisory Group percent audio synced with PowerPoint, beginning in the formative evaluation selected part of a microbiology course for and 66 percent audio only. The average process. An Advisory Group consisting the pilot because of the difficulty of the time from posting date to download was of five dental students and one repre- content and the dependence on diagrams 16.2 hours. The nature of electronic files sentative from Dental Informatics was and other visuals during the presentation. makes it impossible to determine usage formed to direct the project. The staff Faculty permission was obtained to record among those who obtained the media member supported the project’s techni- the lectures and post the electronic in other ways (from a friend burning a cal activities. presentation files on the course Web site. CD, for example, direct file transfer from The Advisory Group decided to con- The microbiology course met three another user, or media used in groups). duct a pilot in single course, with the times weekly for a total of 3.5 hours per Of the 105 students in the class, 70 (66.67 goal of determining whether video week. The Advisory Group discussed pos- percent) completed the survey. Due to the recordings would be the most benefi- sible media formats and chose three types: formative nature of this project, we report cial media format. The group chose six (1) video, (2) audio synced with the images only the percentage of students respond- questions to answer before making a from a PowerPoint presentation, and (3) ing to a question. In Table 1 the second final recommendation: audio only. The Dental Informatics staff column reports the percentage of survey 1. What is the best media format in member attended and recorded each lec- respondents who selected that answer to which to review lectures? ture, using a digital video (DV) camera. the question, and the third column shows 2. What is the best method to acquire Each resulting DV file was exported as a responses from students who reported the media? video file, audio synced with PowerPoint using the lecture download system. 3. What is the best way to disseminate slide images, and saved as audio only. The results from the students who the media? Two days after the lecture, these files were used some form of media are clear and 4. Which courses would students ben- posted on the course Web site, created confirmed in the server log records. efit from having recorded? using CTools. The U-M’s course manage- Responding users both preferred (66.1 5. What are the support costs in terms ment software, CTools was developed percent) and used (66.1 percent) audio- of staff time and workflow? using the open source content manage- only over both the requested format 6. Does the number of students par ment system Sakai. Because of the uni- of video and PowerPoint synced with ticipating warrant the cost of the versity-imposed file size limit, only links audio. They primarily reviewed lec- project? to the files were posted in CTools; the tures, although a small percentage (9.1 From the beginning we expected that School of Dentistry stored the media files percent) used the online lecture as a one pilot could not answer every ques- on its QuickTime streaming server. It took replacement for attending class. Most of tion. At the same time, we knew we the technical lead on the project approxi- the students reported using the files to needed answers to all the questions to mately 3.5 hours to record, complete the study at home, but some also used them reach a complete solution. postproduction process, and post the files to take advantage of down time when The results of the first pilot would for each hour of lecture. working out at the gym (8.8 percent) or determine the project’s direction and The self-reporting measures for learning during their commute (8.8 percent) to yield additional questions. The first pilot needs were a 12-question survey admin- school. Of the students who used the focused on answering question 1, the istered to the entire class (N = 105) one media, some downloaded their media of second pilot answered question 2, and week after the pilot concluded and a focus choice on a regular basis as soon as it was the third and final pilot answered ques- group of six students immediately follow- available (25.9 percent). More students
26 EDUCAUSE QUARTERLY • Number 3 2006 Table 1 Summary of Student Survey Responses* Percent of Total Percent of Lecture Question and Response Respondents Media Users 1. Which audio/video format did you prefer using? Audio (MP3 or iTunes AAC) 52.9 66.1 Audio synced with PowerPoint (MP4) 10.3 12.5 Video (MP4) 17.6 21.4 I did not use any media type 17.6 N/A 2. Which audio/video format did you use the most? Audio (MP3 or iTunes AAC) 55.2 66.1 Audio synced with PowerPoint (MP4) 13.4 16.1 Video (MP4) 13.4 16.1 I did not use any media type 1.5 N/A 3. Did you use the online media as a review of lectures you already attended, or as a substitute to attending class? Review of lectures already attended 65.7 90.9 Substitute to attending class 7.5 9.1 I did not use the media 9.0 N/A 4. Where did you primarily use the media files? At home 63.2 75.4 At the gym 7.4 8.8 On my commute 7.4 8.8 At school 4.4 5.4 Other 1.5 1.8 I did not use any media type 16.2 N/A 5. What best describes when you acquired the media? (time from posting to download) I downloaded the media as soon as it was available 20.9 25.9 I downloaded the media infrequently 23.9 29.6 I downloaded the media all at once close to the exam period 35.8 44.4 I did not download the media 19.4 N/A 6. Did you find the system for accessing the media easy to use? Yes 69.2 87.3 No 2.3 12.7 I did not attempt to use the media 18.5 N/A 7. Do you feel the use of media had an effect on your exam grade? Yes, a positive effect on my grade 72.7 84.9 No, no effect on my exam grade 9.1 15.1 Yes, a negative effect on my grade 1.5 0.0 I did not use the media 16.7 N/A 8. Of your current classes, which one would be most helpful to have classroom media for? Biochemistry 45.0 45.0 Histology 48.3 48.3 Microbiology 5.0 5.0 Other 1.7 1.7 * Total Respondents (N = 105) = 66.67%
Number 3 2006 • EDUCAUSE QUARTERLY 27 tended to download the files close to the the technical staff member documented students’ strong preference for audio, relevant exam (44.4 percent). the time spent recording, processing, the logistics of acquiring high-quality In addition to an analysis of the server and posting the files. Recording time recordings of lectures at a reasonable logs and the survey, the focus group pro- for one hour of class plus 2.5 hours for price became the focus of pilot 2. This vided more in-depth comments. Four processing time (from DV to the three pilot included two courses, for a total of the focus group participants down- formats) was 3.5 hours total staff time. of six class hours each week. loaded and used media files; two did Additional time costs such as equipment not. The focus group discussion probed set-up were not tracked. Pilot 2 Methods. From the beginning, in more detail the questions asked in the Question 6 asked whether the num- we attempted to contain costs. Because survey. Comments from the group cor- ber of students participating warrants the the iPod would be a low-cost solution, roborated findings from other sources, cost of the project. The dental class has we explored it first as an audio capture specifically that students preferred audio approximately 105 students. Participa- device. Students reported using iPods to as the media choice. tion in the first pilot was solely by word record lectures, and a few students placed The focus group also helped us further of mouth. The server logs revealed that iPods with supplementary microphones explore some of the survey findings. For approximately 30 students (29 percent) on their desks in the front row of the example, in the survey an overwhelming participated voluntarily in the pilot, lecture halls. This method produced number of users (87.3 percent) reported which we considered successful. Also, the unsatisfactory audio quality and was the CTools Web site easy to use. When four students in the focus group who had highly dependent on lecturer position. we explored the same question in the used the recordings were adamant about Because of this, the Office of Dental focus group, however, they revealed that how helpful they found the recordings. Informatics used a Belkin Universal downloading is a cumbersome method Microphone Adapter to connect an iPod of acquiring files from numerous course Pilot 1 Conclusions and Decision. The directly to the lecture hall’s amplified PA Web sites. They suggested a number of results of the first pilot clearly indicated system. Students immediately reported possible improvements (an issue addressed students’ preference for the mobility of the the resulting audio was of extremely in the third pilot). Focus group partici- audio recordings rather than video. The poor quality and almost useless. The pants also reported that the most popular overwhelming support for audio combined inability to accurately monitor audio playback devices were personal computers with the low cost of producing the audio levels for the iPod along with the and iPods. files compared to video resulted in the iPod’s limited recoding frequency (16- Results of the survey, consultation with decision to commit to the project and to bit mono, 8 KHz equivalent to analog the Advisory Group, and responses from conduct subsequent pilots to answer the telephone quality) resulted in extremely the students in the focus group helped technical questions about acquisition and poor recordings. us address which courses, if recorded, dissemination. Next we explored using a computer to would most benefit students (question The Design stage of instructional design capture the audio—an Apple Powerbook 4). Survey options focused on informa- relies on feedback to eliminate expensive G4. The analog audio signal from the tion-dense course content with heavy reli- mistakes. Formative evaluation strategies classroom’s PA systems was fed into a ance on visuals. The course selected by the used to gather student feedback early in computer and captured using Apple’s most students (48.3 percent), histology, this project provided critical information QuickTime Broadcaster. The Dental involves great detail and a large number that redirected the focus of the content Informatics staff added metadata (date, of diagrams. This meant that they could delivery. The result was a satisfactory and course name, instructor, and lecture concentrate on what was being said dur- cost-effective solution requiring less tech- title) to the completed recording and ing the lecture about the visual materials nical support than a video solution. posted the file to a Web site. instead of trying to be stenographers cap- At this stage we had identified a work- Using a computer for both media turing all the information being presented. able, though incomplete, solution. Having capture and processing streamlined the The audio recording served as a safety net found students’ preferred media choice workflow (see Figure 2). This reduced that enabled students to listen to the lec- for reviewing lecture content, we could the time associated with capturing, con- ture repeatedly for information they didn’t have stopped our formative evaluation verting, and posting files. Additionally, record in their notes during class. They at this point, adopted the chosen system, eliminating video as a delivery format identified the next course for recording, and attempted to scale its implementa- removed the need to transfer media biochemistry (45 percent), based on the tion school-wide. However, questions from DV tape to computer. faculty lecturer’s fast speaking rate. The remained about how best to capture and The most expensive component of fast-paced delivery made it difficult to disseminate these audio recordings. any technology project is the technical record all the important information in staff’s time. Thus, pilot 2 attempted to their notes, so the ability to review the Pilot 2—Acquisition Solutions automate the recording process. Once lecture was extremely important. For clarity, we describe pilots 2 and we finalized the computer platform as To determine the support costs in terms 3 as two separate pilots although they the acquisition solution, we turned to of staff time and workflow (question 5), ran in parallel. After pilot 1 revealed automating the work performed by the
28 EDUCAUSE QUARTERLY • Number 3 2006 To determine costs in terms of staff time Figure 2 and workflow (question 5), we added up Audio Acquisition via Computer time required for the steps involved. Reduc- ing the number of media formats lessened the recording, processing, and posting time to two hours per class hour. Automating the process of converting and posting files using Apple’s AppleScript technology also speeded the process. Staff processing time dropped to 15 minutes a week (mostly maintenance on processing machines), and files are available on the Web within five minutes of a lecture’s conclusion.
Pilot 2 Conclusion and Decision. Now that more information has been published about portable audio and iPods, high-quality audio production is clearly the most critical component.8 While the tools used to create the audio files (see the sidebar) are becoming easier to use, technical staff. AppleScript scripts were included a focus group and student e-mail the process still requires professional written to automate the recording, file notifications of problems. For records, we technical expertise. processing, and uploading workflow. We examined server logs and kept records of The results of the second pilot clearly modified the processing script to also cre- the time spent by technical staff on the indicated that a low-cost computer could ate an Advanced Audio Codec (AAC) file project in order to calculate cost. easily capture high-quality audio record- as an audio option for download. AAC is ings of classroom lectures from the room a subset of MPEG-4 and allows for sev- Pilot 2 Results. While the computer PA system. In addition, automating the eral advanced features like bookmarking solution was more expensive than using recording, processing, and posting work- and playback speed changes. It is also the the iPod for audio capture, responses from flow greatly reduced staff time. These two native format for the iPod, which many students and staff clearly indicated the low-cost solutions meant that the school students reported using. superior audio quality using a computer could afford to sustain the project over Five students who volunteered to man- to capture the audio from the room’s time. The automated process combined age these tasks were trained in the three- public address system. That answered with student support allowed the proj- step process. At the beginning of a class question 2, about the best solution. ect to scale up to the point that all lec- lecture they selected an icon from the computer’s desktop that ran a compiled AppleScript application. This launched QuickTime Broadcaster and began the recording using presets designed by the Tools Used technical staff. The script required student Apple Computer: action at the end of the lecture by display- QuickTime,
Number 3 2006 • EDUCAUSE QUARTERLY 29 ture halls now are equipped with lecture obtaining feedback from the focus group. the functionality voluntarily. As access recording capabilities. The students reported equal use of MP3 became more convenient, student usage We could have stopped the evaluation and AAC files, accessed both on personal of the lecture recordings increased from here. While the solution did not represent computers and iPods. Examination of 28 percent to 57 percent of students in the eventual final choice of podcasting, server logs confirmed this, showing equal the class using the service. The marked it did meet the students’ desired media downloads of MP3 and AAC files. In the increase in users and usage helped the format, allowed them to review lecture focus group, students reported using the school commit to the project. content, and eliminated expensive staff- audio book feature of the iPod to speed ing costs. up or slow down a lecture. Students Implementation Realities Stopping at this point would leave sev- specifically pointed to this single feature The three pilots answered the six initial eral student needs unexplored, however. that made the iPod more useful than questions and solidified the U-M School Specifically, students requested that we other audio players. of Dentistry’s commitment to offer pod- examine ways of automatically notifying The Advisory Group concluded that casts of lectures as a teaching and learn- them of updated content and provide bet- the custom Web site, while an improve- ing service. The final step was to make ter tools to navigate available files, along ment over the CTools site, still was not the service routinely available. Two very with enhanced features for working with the easiest method for obtaining the files. significant issues remained, however: long audio recordings. Feedback from the focus group revealed faculty support for podcasting, and the that students wanted a centralized Web institution’s ability to sustain podcasting Pilot 3—Dissemination Solutions site to more conveniently access the audio into the future. The final pilot focused on improving files. While the course Web site served While the faculty involved in the three the process by which students identified adequately for one course’s lectures, the pilots were enthusiastic about participat- and downloaded new recordings. Pilot increasing number audio lectures made it ing in a research and development proj- 3 used the formative evaluation strate- more difficult to access new content. ect, having lectures routinely recorded gies of self-reporting and records: feed- The focus group also suggested creating and distributed raises issues of intellectual back from the focus group and Advisory either a notification system using e-mail property. Conflicts between a lecturer’s Group, and review of the server logs. or a subscription service for automatic intellectual property rights and the need notification when files were posted. We of students to acquire media for review Pilot 3 Methods. After the first pilot, added RSS (Real Simple Syndication) so must be mediated in a way acceptable to the Advisory Group selected a course in that students would not have to check both groups. which the faculty member spoke fast. for new files. Server logs subsequently Once the school chose podcasting as The Advisory Group also requested showed that 50 percent of files were a routine service, the pilot’s makeshift recordings of the Integrated Medical downloaded via RSS. authentication and authorization process Systems (IMS) series because of the large Server logs served to measure student needed to be replaced by a very robust sys- amount of difficult material it contained. participation and thus help us determine tem. Through collaboration with the U-M Students felt that having another source whether the number of students partici- ITCS, the School of Dentistry adopted the to review material would be especially pating warranted the cost of the project U-M’s already implemented authentica- beneficial in this class. IMS lecture series (question 6). The logs indicated no dif- tion and authorization system, cosign. met several days each week, for a total of ference (30 students) at the end of the This guaranteed that only U-M dental 15 hours—a three-fold increase in hours second pilot but an increase to 60 students and dental hygiene students could access recorded from the first pilot. (of a possible 105) at the end of the third the lecture podcasts and eliminated the Once lectures from multiple courses pilot. Students reported that the conve- potential intellectual property issue, thus were available, a focus group revealed that nience of obtaining files via RRS increased helping with faculty acceptance. students would prefer to go to a single the likelihood that they would download While the U-M School of Dentistry Web page to access the lecture recordings files. Server logs confirmed this, revealing had proven that lecture podcasts could instead of moving between numerous a much higher download volume com- be done for a reasonable cost, the devel- course Web sites. Thus, access through pared to earlier in the project. oped system was proprietary to the den- the course management system, CTools, tal school. To guarantee that podcasting was discarded in favor of a custom-built Pilot 3 Conclusion and Discussion. would be sustained at a reasonable cost dynamic Web site with data contained in The third pilot gave solid answers to the over an extended period of time and that a MySQL database. This allowed students project’s final two questions. Because it would continue to evolve, it needed to to more easily sort files by name, media students used both MP3 and AAC formats be offered university-wide. Ideally, the format, and class. equally, the decision was made to provide knowledge, processes, and services devel- both formats. Data also indicated that if oped within the School of Dentistry would Pilot 3 Results. We answered question 3, lecture recordings were convenient—from be shared with the university community the best way to disseminate the media, by a central Web site and/or through RSS—a and beyond. In partnership with Apple consulting with the Advisory Group and significant number of students would use Computer, Inc., a follow-up pilot inves-
30 EDUCAUSE QUARTERLY • Number 3 2006 Special acknowledgement goes to Gordon tigated whether iTunes U could expand dents in the design process and applying Sukwit and Sheri Schultz of Apple Computer, on the services provided by the School proven formative evaluation methods, who had the early vision and recognition that of Dentistry. While still in progress, that the school avoided implementing a sys- mobile devices such as iPods could be valuable pilot’s initial results have proven positive, tem that would have cost more and might learning tools for students and that the School and the U-M and Apple Computer are not have met student needs as well as of Dentistry’s project was an innovative use of the technology. Also from Apple Computer, the now investigating integrating iTunes U podcasting lectures. iTunes engineers Eric Bailey and Sugam Jain into Sakai. If that project succeeds, then The students’ commitment through deserve thanks for their work on the iTunes the School of Dentistry will use the Sakai the entire project was evidenced by their project. Patrick McNeal from U-M Information podcasting software, guaranteeing that enthusiastic participation in focus groups, Technology Central Services is acknowledged podcasting lectures will occur routinely on the Advisory Group, and on the sur- for making the U-M cosign authentication and authorization system work with this project. for an extended period of time and at a vey. From the beginning, students knew We also thank Ruxandra Iacob for her help very low cost. they were guiding the development of the with the analysis and Sharon Grayden for her Faculty and students are interested project, and they sustained their energy editorial assistance. in exploring the expansion of podcast- and commitment through completion. ing services as a separate project with its They now share in the pride of seeing how own analysis, design, and development their efforts benefit their classmates and Endnotes cycle. It is important to draw a distinction the classes of dental students following 1. J. Frechtling, The 2002 User-Friendly Hand- between this deliberate project expan- them. They also share in the presenta- book for Project Evaluation (Arlington, Va.: 9 National Science Foundation, Director- sion and scope creep. Scope creep refers tions, press releases, and writing about ate for Education and Human Resources, to uncontrolled changes in a project’s the project, as in this article. 2002). goals that cause the project to drift away Technology is commonly implemented 2. K. Cennamo and D. Kalk, Real-World from its original purpose. Our purpose is into teaching and learning situations Instructional Design (Belmont, Calif.: to continue to research ways to aid stu- without using instructional design and Thomson Wadsworth, 2005). dent learning. formative evaluation strategies or involv- 3. B. N. Flagg, Formative Evaluation for Edu- We will evaluate requests for new ing students. Faculty and administrators cation Technologies (Hillsdale, N.J.: Erl- baum Associates, 1990), p. 259. features with two intentions: that all (developers) usually make the decisions. 4. Ibid. improvements assist student learning, Unfortunately, many novel and innova- 5. S. M. Alessi and S. R. Trollip, Multime- and that all future developments can be tive projects do not succeed or have disap- dia for Learning: Methods and Development integrated into Sakai. For example, the pointing results. Using both the formative (Boston: Allyn and Bacon, 2001). 10 processing scripts are being written to eas- evaluation strategies suggested by Flagg 6. T. Weston, “Formative Evaluation for ily support complementary media com- and the instructional design process Implementation: Evaluating Educational ponents (PDFs, PowerPoint files, images) described by Alessi and Trollip11 helped Technology Applications and Lessons,” that can be associated with a lecture’s us identify and adjust to unexpected American Journal of Evaluation, Vol. 25, No. 1, 2004, pp. 51–63. audio file. circumstances and develop a successful 7. Flagg, op. cit. technical solution to a learning dilemma. 8. G. Campbell, “There’s Something in Lessons Learned Ultimately, use of these strategies pro- the Air: Podcasting in Education,” EDU- The unanticipated results of this proj- vided the critical data required to ensure CAUSE Review, Vol. 40, No. 6, November/ ect strongly reinforced two lessons that long-term and ongoing support. e December 2005, pp. 33–46,
Number 3 2006 • EDUCAUSE QUARTERLY 31