Activinspire Training Evaluation

ActivInspire Training Evaluation

Kim Ashcraft

EDCI 577

Dr. Leslie Moller

October 12, 2012 2

Executive Summary

Over the last several years, The Summit Country Day School has made a significant financial investment in technology, among which are Promethean ActivBoards (interactive white boards).

Currently there are 35 boards in the school, each carrying a cost of approximately $7,000 for hardware and installation. The observations of the division directors (principals) and the technology department members who are observing in classrooms is that most of the faculty who have the boards are either using them for “glorified white boards,” (writing class notes on the digital board) or as a digital screen to show films or internet content. The few teachers who are utilizing the Promethean software are only using the most basic features and only three teachers are using the software to its full capacity. One of those teachers, Sally Sams, has taken and passed the formal training courses on her own that are offered by Promethean.

In each of the last three years, a Promethean trainer has presented a 90-minute tutorial session (free training). This has had little to no effect on teacher board use and the directors are disappointed in the lack of effective and creative use. The cost of hiring a professional trainer for 2 days is approximately

$600.00, which the school is not willing to pay at this time.

The technology director has instead asked Sally to present to other faculty members who have

Promethean boards in their rooms, a two-day in-service, as it was determined that two sequential days were needed. Sally has asked me to help her design and carry out an evaluation of her presentation since I have had some training in this area. The purpose of this systematic evaluation is to show both the initial and long-term impact of her training. The data will be used to determine whether an allocation of funds will be made for future technology trainings and possibly future evaluations of that training.

I attended a meeting with Sally and the technology director regarding the proposed evaluation of the

Promethean training and full approval has been granted, provided the cost is nominal (less than 3 $50.00). The results of the evaluation will be shared with the faculty, division directors, Head of school, Technology Director and eventually, the Board.

We began to plan the evaluation one month before the fall in-service to be held in October, 2012.

Sally and I discussed the topics, sequencing and main objectives of the Promethean trainings in order to develop evaluation instruments with congruous objectives. The overall goal is for faculty to demonstrate proficiency in creating interactive Promethean flipcharts (file presentations) to support their respective curricula. A sequence of interactive, hands-on sessions, grouped as beginner, intermediate and advanced will be presented. A list of main competencies for each of the three levels has been created and several instruments have been developed to assist in determining the effectiveness of the Promethean in-service and application of teacher learning. See Table Below.

Beginner Intermediate Advanced Session 1 Session 2 Session 3 Thursday PM, October 18th Friday AM, October 19th Friday PM, October 19th

Calibrating the ActivBoard Using the camera tool Creating a magic box Navigating with Toolbars Creating voice recordings Creating containers Creating Backgrounds Creating screen recordings Manipulating Layers Inserting shapes, pictures, text Embedding Media Setting Restrictors Creating speaker’s notes Using Drag and Drop Creating Magic Ink/Erasers Converting from Power Point Hiding objects Registering voting devices Saving as a PDF Creating roll-over IDs Creating ActivEngage quizzes

Kirkpatrick’s model of evaluation was used as a guide in developing this evaluation. Four levels of the framework will be analyzed: Level 1, Reaction, Level 2, Learning, Level 3, Behavior, and Level

4, Results. Level one consists of a survey whose purpose is to measure the effectiveness of the in- service atmosphere, the in-service presenter, the Promethean tutorial flipcharts, the presenter’s delivery method, and the overall participant satisfaction with the Promethean portion of the fall in-service.

Level two utilizes a criterion-referenced test which participants will take at the conclusion of their third

Promethean session. In addition to the test, participants will submit digitally a completed Promethean 4 flipchart to the presenter. For Level three, a peer-teacher system of observation will take place as well as observation by an IT department member one to three months after the in-service has occurred.

Finally, the participants will complete a self-assessment survey three months after the in-service.

Level four will be measured by number of teachers who are creating and using interactive flipcharts in their classrooms. This final observation will be performed by the teachers’ division directors.

Purpose of Evaluation

The immediate purpose of this evaluation is to measure the effectiveness of the Promethean

ActivBoard in-service training. The results will show the number of teachers who are successfully incorporating interactive elements into their Promethean flipchart lessons. Additionally, faculty will have an opportunity to show that they are part of the community of life-long learners and to be role models for their students. The longer-term purpose is that it is hoped that the success of this

Promethean training will convince administrators to invest in future technology trainings. Below is a discussion of the procedures for the Kirkpatrick evaluation framework.

Evaluation Framework

Level 1: Reaction

Level one consists of a survey whose purpose is to measure the effectiveness of the in-service atmosphere, the in-service presenter, the relevance of the Promethean board functions to each participant’s own curricular goals, the Promethean tutorial flipcharts, the presenter’s delivery method, and the overall participant satisfaction with the Promethean portion of the fall in-service. The survey

(see Appendix C), includes questions using a Likert-type scale and includes areas for participants to write in comments on improvements to the course. It will be given to the faculty participants just before the end of each of the three sessions on Thursday and Friday.

The purpose of the survey is to provide immediate feedback for the tutorial and interactive hands-on presentation training. The results of the survey will be collected and analyzed using mean scores. We 5 hope that all faculty participants will see the relevance and importance of these sessions as a means to create more meaningful and interactive lessons of their own.

Level 2: Learning

The first Promethean in-service session will be held on October 18th in the afternoon from Noon-

3:00. This session is intended for new teachers who have never used the Promethean boards or teachers who are still unclear on the basics of setting up the board and creating simple flipcharts.

Tutorial flipcharts will be emailed by Sally to each participant. Participants will bring their laptops to

Sally’s classroom for the sessions. The second and third Promethean in-service sessions will be held on October 19th from 9:00-11:00 AM and 1:00-3:00 PM. A short written test will be given just before the conclusion of session 3 (see Appendix D). Sally will grade the test immediately in order to provide the participants with immediate feedback and review any answers that were incorrect. The recorded scores will be used to determine with whom Sally should follow-up individually during the next week to clarify any remaining questions. An additional performance test will be required of each participant in the form of a completed flip-chart containing the functions that were taught at the in- service (See Appendix E). Some faculty members will have completed this during the in-service while others may need to review the tutorials and submit their flipcharts by the end of the following week.

All test questions were derived from the objectives that were developed in my meeting with and each of these Promethean functions will be not only presented during the in-service training, but will also be incorporated into the tutorial flipchart emailed to each participant prior to the first in-service day. The questions include topics relevant to the three levels of Promethean flipchart creation.

Level 3: Behavior

For Level three, an observation check sheet will be used in a peer-teacher system of observation and an IT department member observation that will take place one to three months after the in-service has occurred (See Appendix F). This will ensure that the in-service participants have had time to develop their flipchart making skills and have become more comfortable with the process. The teacher, peer 6 teacher and IT member will decide upon the best day for each of them to visit the in-service teacher’s classroom. Each of these visits is meant to ensure that the learning transfer has taken place and will also serve to help the teacher prepare for the future observation visit of the teacher’s director.

Additionally, the peer-teacher will also learn by observing others who are going through the same process. The observed uses of the items on the checklist will be tallied to show a measurable behavior change on the part of the in-service teacher. It is expected that the teacher exhibit at least two of the

Promethean functions which were part of the “Advanced” in-service session. Since creating lessons of this type is a creative endeavor, it is not likely that every function learned will be part of an actual lesson, but there should be observable evidence of general competency in the flipchart lesson.

Sally and I will receive the observation sheets which we will then rank as “Proficient” or “Still

Working on Proficiency.” The self-assessment will be completed by the in-service participants three months after the in-service (See Appendix G). The purpose of this survey is to determine how confident the teachers feel about their abilities in creating meaningful, interactive flipchart lessons.

High teacher confidence will show evidence that the in-service presentation was effective.

Level 4: Results

The results of the in-service will be measured by the number of faculty members who are observed as having created interactive Promethean flipcharts (“Proficient”) and who are currently using them in their classrooms to convey their content. Each faculty member’s director will observe this as part of the yearly observation and this will take place approximately five-six months after the in-service training (See Appendix H). This segment of the director’s observation will be given to the evaluator in order to collate the data. The purpose of involving the director is two-fold. One, faculty members are extremely likely to engage in activities on which they are going to be observed by their director. Two, it will provide an opportunity for directors to see first-hand the progress that the faculty members have made in the area of technology. They are likely to carry this information by word of mouth to the

Head of school and Technology Director. 7 Conclusion:

Promethean hardware is very expensive to purchase and install. Administrators had begun to question whether this investment or future investments in technology hardware is wise based on lack of effective teacher use. This evaluation report should show that the benefits of a technology training program are worth the investment of future time and resources in order to train faculty to utilize the technology hardware in the school. In addition, faculty members who are part of a community of life- long learners need to show their superiors that they are modeling this behavior. The results of this evaluation will show that this has, indeed, occurred.

Appendix A: Evaluation Plan

Project ActivInspire Promethean Software Training Evaluation

Scope Evaluation of the 2-day In-Service Program for Teachers

To determine the effectiveness of the Promethean In-Service; Purpose To establish value for future technology trainings

Levels of 1) Reaction; 2) Learning 3) Behavior 4) Results Evaluation

Process

Item Date Due Who Create evaluation plan 9/18/12 Kim A., Sally Sams Design instruments for Levels 1-4 9/19/12-10/4/12 Kim A., Sally Sams Administer/collect level 1 data -1 10/18/12 Kim A., Sally Sams Administer/collect level 1 data -2 10/18/12 Kim A., Sally Sams Administer/collect level 1 data -3 10/19/12 Kim A., Sally Sams Administer and collect level 2 data 10/19/12 Kim A., Sally Sams (Criterion-Referenced Test) Collect level 2 data by means of 10/19/12- Kim A., Sally Sams e-mailed flipcharts 10/24/12 Collect level 3 data By 01/24/13 Kim A. via Sally Sams Collect level 4 data By 4/25/13 Kim A. via Sally Sam Analyze levels 1,2, 3 and 4 data 5/30/13 Kim A. Prepare final report 6/15/13 Kim A. 6/15/13 email; Present results to division 8/20/13 in Kim A., Sally Sams directors. person Evaluation Budget Paper and test materials: $5-10 Evaluator costs: No cost –pro bono work 8

Data Collection Data Item When How Collected From Whom By Whom After each Reaction Survey Faculty Sally/Kim session Post-test and After last Test/Flipchart Faculty Sally/Kim Performance session Behavior Observation Observations, Peer teachers/IT Transfer checklists and Kim A Peer, IT and Self member/self survey Combination of Behavior and Observation Results Directors Kim A Director and Analysis Observation Report of Findings What results are To whom How communicated reported L1 Reactions Division Directors, L2 Test/Flipcharts Written report and presentation – Technology Director L3 Teacher Observations emailed and then face to face Faculty and self-assessments presentation Board L4 Director Observation 9 10 Appendix C: Level 1 Reaction Survey

Name (optional) ______Session # ______

Please help us make this training better by providing your opinions!

Instructions: Please circle the appropriate response after each statement.

Not at all True True Very True 1. I am excited to begin creating my own flipcharts. 1 2 3 4 5

2. The tutorial flipcharts were useful in helping me understand how to create my own. Not at all True True Very True

1 2 3 4 5 3. I feel comfortable creating my own flipcharts after completing this course. Not at all True True Very True

1 2 3 4 5 4. I am not confident that I can create interactive flipcharts without help. Not at all True True Very True

1 2 3 4 5

5. The instructor was knowledgeable about Promethean features and creating flipcharts. Not at all True True Very True

1 2 3 4 5

6. The instructor presented some of the concepts out of a logical sequence. Not at all True True Very True

1 2 3 4 5

7. I don’t think I will be able to use these techniques in my classroom. Not at all True True Very True

1 2 3 4 5 12

7. The length of this session was: Too short Just right Too long

9. Your overall rating of this session: Excellent Very good Good Fair Poor

10 What suggestions do you have to make the course more effective?

Appendix D Level 2 Promethean Test

Name ______

Directions: Circle the best answer to the following questions or provide an answer in the space provided. You have 20 minutes to complete 10 questions. Good Luck!

1. The board is ready to be calibrated when the Promethean flame turns:

a. Red b. Blue c. Orange d. Green

2. In order to use the magic eraser function, you need to manipulate:

a. Layers b. Containers c. Restrictors d. Labels

3. In order to register the ActivExpression voters, you need to: Choose all that apply:

a. Plug in the ActivHub b. Turn on the individual voters c. Click the Configure menu and follow the questions d. Click the Voting Timer button

4. Screen and Voice Recordings are not able to be inserted directly into the flipchart.

a. True b. False

5. You can preset the question timer by:

a. Clicking the vote now button 13

b. Entering the time within the quiz creator screen c. All the questions are automatically set at 30 seconds. d. Opening the quiz browser and typing it in.

6. Choose all functions below that can be inserted using the drag and drop function.

a. Page forward/Page back b. Promethean pen c. Labels d. Eraser

7. When creating a container, if you want the container to contain multiple correct answers, you should choose the ______setting:

a. Nothing b. Anything c. Specific Object d. Keyword

8. What should you select to insert speaker’s notes?

a. Pen function b. “Write Notes” button c. Post-it note icon d. The “insert” link from the menu

9. Which function will allow videos/animations to be embedded into the flipchart?

a. Insert Media File b. Insert Link – html c. Insert Link - website d. Insert Media – My videos

10. When creating magic erasers, on which layer does the eraser function need to be?

a. The top layer b. The middle layer c. The bottom layer 14 15

Appendix E: Level 2 Promethean Function Flipchart Checklist

Name of faculty member who submitted______

The purpose of this instrument is to evaluate the teacher’s initial understanding and competency with interactive Promethean functions and to provide valuable feedback to the teacher if gaps are observed.

Procedure: Please place a checkmark on the left beside all items of which initial competency is observed.

Beginning Level:

Navigating with Promethean Toolbars

Backgrounds, Inserting Shapes, Pictures and Text

Speaker’s Notes

Intermediate Level:

Creating and Saving Voice Recordings and Screen Recordings.

Inserting/Embedding Media: Videos and Animations

Using the Drag and Drop Menu from the Action Browser

Creating Hidden Objects and Roll-Over Identification Labels

Advanced Level:

Constructing a Magic Box and Creating Containers for Objects

Setting Restrictors

Using the Magic Eraser and Magic Ink Functions

Use of Personalized Quiz questions for ActivEngage voting devices 16

Appendix F: Level 3 Observation Checklist for Peer-Teachers and IT Members

Name of faculty member observed ______

Name of observer______Date______

Please place in Sally Sams’ mailbox! Thanks!

The purpose of this instrument is to evaluate the teacher’s effective implementation of interactive Promethean functions and to provide valuable feedback to the teacher. Please place a checkmark beside all items of which you are able to observe evidence. Feel free to write comments to the right of any item.

Beginning Level:

Speaker’s Notes

Intermediate Level:

Advanced Level:

Setting Restrictors

Use of ActivEngage voting devices

Please provide a brief summary of the teacher’s flipchart lesson and your overall impressions of quality

Section 2

1. Please write a brief summary of the teacher’s flipchart.

2. Were there features incorporated into the flipchart that weren’t listed on the checklist? If so, describe.

3. What is your general impression of this teacher’s ability to create and present interactive flipcharts?

4. Are there any areas that you would like to point out as superior or as needing refinement? 18

Appendix G: Self-Assessment

Name (optional) ______Date______

Please consider your comfort level with completing each of the items listed below and write your rating in each blank. When finished, please place in Sally Sams’ mailbox. Thanks!

Ratings are: VC – Very comfortable C – Comfortable NC – Not Comfortable Beginning Level:

______Calibrating your Promethean Board and Editing User Defined Buttons

______Navigating with Promethean Toolbars

______Creating Backgrounds

______Inserting Shapes, Pictures and Text

______Creating Speaker’s Notes for your Flipcharts

______Converting PowerPoint Slides to ActivInspire/Saving files as PDF’s

Intermediate Level:

______Using the Camera Tool,

______Creating and Saving Voice Recordings and Screen Recordings.

______Inserting/Embedding Media: Videos and Animations

______Using the Drag and Drop Menu from the Action Browser

______Creating Hidden Objects and Roll-Over Identification Labels

Advanced Level:

______Constructing a Magic Box and Creating Containers for Objects

______Understanding and Manipulating Layers

______Setting Restrictors and Using the Magic Eraser and Magic Ink Functions

______Registering and Using ActivEngage voting devices

______Creating Personalized Quizzes for ActivEngage voting devices 19

Appendix H: Level 3 Observation Checklist for Division Director

Name of faculty member observed ______

Name of Director______Date______

Please place in Sally Sams’ mailbox! Thanks!

The purpose of this instrument is to evaluate the teacher’s effective implementation of interactive Promethean functions and to provide valuable feedback to the teacher. Please place a checkmark beside all items of which you are able to observe evidence. Feel free to write comments on the right of any item.

Beginning Level:

Speaker’s Notes

Intermediate Level:

Advanced Level:

Setting Restrictors

Use of ActivEngage voting devices

Please provide a brief summary of the teacher’s flipchart lesson and your overall impressions of quality

Section 2

1. Please write a brief summary of the teacher’s flipchart.

2. Were there features incorporated into the flipchart that weren’t listed on the checklist? If so, describe.

3. What is your general impression of this teacher’s ability to create and present interactive flipcharts?

4. Are there any areas that you would like to point out as superior or as needing refinement?