Kristen Barrera (Cuda) 711Th HWP/RHAS

Testing & Evaluation Teleconference 21 February 2014

Roll Call

Kristen Barrera (Cuda) 711th HWP/RHAS Gary Anaya Office of Naval Research (ONRG) Reserve Component (RD) Beth Mutch ONR RC Subhashini Ganapathy Wright State University Jacob Hodges LMITT

Introduction

Discussion:

- This teleconference covered a plethora of issues with the main focus being the need to start assimilating the types of questions that will be asked during the International Military Student Officer (IMSO) and Concept Evaluation (CE).

Assorted Items

Discussion:

- Kristen Barrera mentioned that she has been monitoring the Testing & Evaluation inputs on Google Docs and the due date for these inputs is approximately two weeks away (i.e., 7 March 2014).

 Gary Anaya mentioned that he was having difficulty in accessing Google Docs and thinks the problem may be that he is trying to access it at work. Kristen Barrera mentioned that she was unable to access Google Docs at work and is using a personal email address.

- Subhashini Ganapathy has continued to work on the DMAP. She has input some of the data collection idea and recommendations into the DMAP so that it can be reviewed by the US/ International Team Members. At the current moment it is about 90% completed and will be uploaded to the Google Docs site once all inputs have been provided by the international participants.

 In preparing to upload the DMAP to the Google Docs portal for international participant reviews, Subhashini Ganapathy will send the DMAP out to this T&E Groups by Wednesday, 26 February 2014, so that Gary Anaya and Beth Mutch can take a make recommended changes.

 Gary Anaya and Beth Mutch will review the Data Matrix & Analysis Plan (DMAP) and return it to the T&E Team so that it can be the main focus of discussion at the next T&E Teleconference, 7 March 2014.

Testing & Evaluation Conference Call Minutes – 21 February 2014 Page 1 - Kristen Barrera mentioned that she was looking toward a late-April/mid-May Meeting that was on the calendar in order to have collaboration amongst the three teams (i.e., Concept Evaluation, Research Protocol, and Testing & Evaluation). She was thinking about a 2-day event where the process could be discussed in detail.

- Kristen Barrera pointed out that most of the data decisions will be made by the following dates:

 10-14 March 2014 – DMAP Completed  12 March 2014 – International Consolidated Inputs  14 March 2014 – T&E Teleconference  21 March 2014 - Testing & Evaluation Measurement Plan (Final)  Submission of T&E Strategy to the Management Team for Approval

- Jacob Hodges provided the following comments:

 Based on the teleconference at the beginning of 2014, it was agreed that there was no longer a need to have an April/May Meeting. Collaboration amongst the US/International participants could be conducted via teleconference, Net Meetings and/or Skype. Therefore, the next meeting will be a September 2014 ‘Train-the-Trainer Concept Evaluation Meeting’ prior to the MLS Concept Evaluation (i.e., October/November 2014).

 The T&E Team was reminded of the experience from the Mobile Learning Environment (MoLE) Project where evaluation questions were changed once they were reviewed by the international team due to cultural difference in the use of terminology. Therefore, he didn’t think it prudent to send the set of questions to the Management Team before they had been fully vetted with the international participants.

 The need to ensure both the IMSO and Concept Evaluation process are exactly the same is key if the data collection analysis will provide an opportunity to conduct comparative analysis. This means that the ‘Demographics’ data collection will need to add a question on ‘Country of Origin’ since it may not be possible to identify what country each IMSO volunteer originates.

Concept Evaluation Jacob Hodges

Discussion:

- Jacob Hodges informed the group that he has been in collaboration with Dwayne Eldridge on the Learning Content that will be used for the IMSO and Concept Evaluation. This includes;

 There will be four types of content developed by his group (i.e., video, ePublication, mobile application and web-based).

 The evaluation will cover the same material that he presented at the MLS International Collaboration Kick-Off Meeting (i.e., 2.1: American & Military Cultures Intro; 2.2: Culture; 2.3: Individualism; 2.4: Punctuality; 2.5: Informality; 2.6: Egalitarianism; 2.7: Short-Term Mentality; and 2.8: Daily Life in America).

 Jacob Hodges mentioned that since the MLS Project goal was to ‘develop a capability that ensures IMSPDB students have a positive and successful experience in the U.S. along with their U.S.-counterparts,’ it was agreed at the kick-off meeting that it is important to determine if knowledge transfer occurs via the different mediums. Therefore, confirmation was sought on whom should develop the pre- and post-test for the IMSPDB content. The T&E Team agreed that since the SCETP has expectations from the content they should develop the pre- and post-questions and not the T&E Team.

 Jacob Hodges expressed a concern of the video being 11 minutes in length due to the impact it would have on those volunteers where English is a Second Language. Input will be sought from the international participants since they will be providing the volunteers.

 Jacob Hodges briefly discussed concerns over the length of the evaluation since we are asking each volunteer to evaluation two mediums in addition to the questionnaire process (e.g., pre- and post) when considering that non-native English speakers normally take longer. For example:

Topic Comments Projected Duration Demographics Survey 2-3 mins Volunteer Sign-on Pre-Questionnaire 7 - 9 mins Video, Mobile Application 11 mins Medium Evaluation ePublication, Web Based 11 – 12 mins Post-Questionnaire 7 - 9 mins Final Data Collection Additional T&E Questions 10 – 15 mins Total Duration (based on 2 mediums) 48 - 59 mins

- Before the final Testing & Evaluation Questions have been decided, the US/international Testing & Evaluation Team need to ‘test’ the mediums being developed by the Learning Content in order to ensure that there is no problems noted (i.e., problems in downloading Android/Apple Mobile Applications, ePublication formatting/reading, video download, etc.). This ‘internal test’ is also critical to ensure the questions being asked are relevant to the material since one of the areas of questions relate to usability and ease-of-use.

- In order to ensure the Testing & Evaluation Team has a clear understanding of the Learning Content and how it will be accessed by US/international volunteers, it would be good idea if Dwayne Eldridge could provide a briefing to both the Research Protocol and Testing & Evaluation Team at the next MLS Monthly Teleconference (14 March 2014).

Action Items Status Update

Action items POC Deadline Populate the Data Matrix & Analysis Plan (DMAP) to reflect Kristen Barrera / Ongoing MLS TE Data Collection Strategy Subhashini Ganapathy Provide the updated Data Matrix & Analysis Plan (DMAP) to Subhashini 26 Feb 2014 ONR RC Ganapathy Provide the ONR RC reviewed/edited Data Matrix & Gary Anaya 5 March 2014 Analysis Plan (DMAP) to the US T&E Team Beth Mutch

Testing & Evaluation Conference Call Minutes – 21 February 2014 Page 3 Conduct T&E Teleconference focused solely on the Data All 7 March 2014 Matrix & Analysis Plan (DMAP) Upload the incorporated version of the Data Matrix & Subhashini 18 Mar 2014 Analysis Plan (DMAP) to the Google Docs Ganapathy Upload the incorporated version of the Data Matrix & Subhashini 18 Mar 2014 Analysis Plan (DMAP) to the Google Docs Ganapathy Add Gary Anaya’s personal email address to the Google Docs Jacob Hodges 26 Mar 2014 portal so that he has access to all MLS documents and briefings