Appendix a JMSEL Data Collection Plan

UNCLASSIFIED

Appendix A – JMSEL Data Collection Plan The purpose of the JMSEL v2.6 data collection form is to captured metrics, performance requirements, and other exercise information pertinent to transition activities and evaluation of JTIMS JMSEL (JTIMS Lite) during execution scenarios. Please answer the following questions about your organization’s exercises to the best of your knowledge. Name: (First Name, Last Name) Carl Howell User Group: (i.e. NORTHCOM) NORAD-USNORTHCOM External User Group(s): VIGILANT SHIELD & linked/embedded exercises: JTF- Alaska, Alaskan NORAD Region, Canadian NORAD Region, Continental United States NORAD Region/1st Air Force/USAIRFORCE NORTHERN, USARMYNORTH, MARFORNORTH, JTF-Civil Support & JTF-North,. ARDENT SENTRY & linked/embedded exercises: Commands listed above plus Commands and Interagency partners to include, Canadian Joint Operations Command, Federal Aviation Agency, Transport Security Administration, National Guard Bureau, U.S. Coast Guard, Civil Air Patrol, etc. Role(s): (i.e. MSEL Manager) MSEL Manager for JTIMS/JMSEL

James “Butch” Webb is the MSEL Manager for JCW supported exercises. Presently, these include ARDENT SENTRY plus linked/embedded exercises and VIGILANT SHIELD plus linked/embedded exercises. Main Responsibilities: Oversee the use of JTIMS/JMSEL Application: (i.e. JMSEL 2.6 or JTIMS Enterprise) JTIMS/JMSEL during internal, small scale exercises. JMSEL 2.6 during Large Scale, JCW supported ARDENT SENTRY & VIGILANT SHIELD plus linked/embedded, exercises.

JMSEL Questions Comments Small Exercise Medium Large Exercise Exercise User Load How many users participate in executing JMSEL activities during Small, JCW/JMSEL Average = 30 Medium and Large exercises? (total # of users) 2.6 data How many concurrent users (maximum # of active users executing JMSEL JCW/JMSEL activities simultaneously) are accessing/ executing JMSEL activities on Approx 10 2.6 data average during Small, Medium, and Large exercises? Exercise Duration/Execution What is the average duration for each exercise size (i.e. 7 days)? JCW/JMSEL 3 Days 2.6 data

How many hours during each day do JMSEL scenario activities take place? - NOVEL STRAIN JCW/JMSEL

UNCLASSIFIED (i.e. 8 total hours each day) - (Internal) ARDENT SENTRY - INNER LOOK / HURRICANE ZOE 2.6 data 2x24 hr + 1X12 hr - CYBER STORM 3x12 hr

How many injects are typically released or processed each hour? Average = 2 Does your user group ever execute more than one exercise concurrently Internal small scale HQ exercises are stand (e.g., large exercise and one or more small exercises at the same time)? If alone with no linked or concurrent exercises. JCW yes, do you associate these exercises in order to view them under one supported VIGILANT SHIELD and ARDENT SENTRY umbrella? typically include 6-7 linked or embedded small to medium scale exercises. Inject Creation

How many total injects on average are created for each exercise size? JCW/JMSEL Average = 61 2.6 data What percentage of injects are created during the MSEL Development Approx 60% at the Conference? “Scenario” JCW/JMSEL Development 2.6 data Conference. How long does it take on average to create each inject for the scenario? Approx 1 hr - Initial Draft - Initial Review JCW/JMSEL - Implementer 2.6 data Development - Final Review What percentage of injects are created by copying an existing 0% inject/searching the inject library? The four Small scale exercises JCW/JMSEL contained separate 2.6 data and unique data requirements. Are all injects created processed through completion in the Release Process JCW/JMSEL YES steps? If not, what percentage is processed? 2.6 data Approximately, what percentage of injects are processed in Email mode? Approx 56%

This is a great feature when working properly. We have had varying degrees of success with the “Script Recipient” option, linked to the “Address Book” JCW/JMSEL However, the 2.6 data “Additional Recipient” option for non “Address Book” addressees has never worked for us.

Approximately, how many attachments are uploaded to each size exercise? Average = 21 JCW/JMSEL One exercise had no 2.6 data UNCLASSIFIED

attachments. Average for the three Exercises with attachments was 27. How often are injects deleted? JCW/JMSEL Approx 4 per exercise 2.6 data If “Edit Buffering” enabled, how often do proposed changes occur during Used “Edit Buffering” MSEL development? for approx 6 injects. Proposed changes JCW/JMSEL originated with the 2.6 data inject POC and were approved. What percentages of proposed changes are approved? JCW/JMSEL 100% 2.6 data What percentages of proposed changes are declined? JCW/JMSEL N/A 2.6 data Training Objectives How many training objectives on average are associated to each inject? Average = 2 Are training objectives added or modified once an exercise begins? If yes, Yes. Our JMSEL management business rules allow us how often does this occur? the flexibility to react quickly to short notice direction from the Exercise Director and to accommodate the needs of internal and external exercise control POCs. We have the ability to add/delete objectives and JMSEL injects as required. This occurred twice during Novel Strain and three times during the internal ARDENT SENTRY 10 exercise. Do you associate training objectives from associated events? If yes, how There were no associated events during the four small often does this occur? scale exercises. Additional Questions Comments Inject Search How often do you perform an inject search? Occasionally, when attempting to locate a specific JMSEL number. How often do you use Keyword Search feature? 6 – 10 How often do you use the Advanced Search Options feature to search? 10 - 12 times during the pre-event development process and during execution. What specific criteria do you use to filter your search results? Normally, the following criteria is used - Inject Cell - Inject Day - Inject Status - Key Inject Summary Views How often do you use the inject summary views? Occasionally, during inject development and review when a POC wishes to view all injects associated with a specific theme, training objective or storyline. Used minimally during execution. Which views are primarily utilized: All Injects, Key Injects, Placeholder Injects, - All Injects (Normal View) Deleted Injects, Proposed Changes, Incomplete Implementor, MSEL Summary - Key Injects Report, and Implementor Summary Report? - MSEL Summary (Potential use during MSEL Synch)

Which export capabilities (i.e., Word, Excel) do you typically use for summary - Both Word and Excel views: All Injects, Key Injects, Placeholder Injects, Deleted Injects, Proposed Reports are complete and immediately usable in the Changes, Incomplete Implementor, MSEL Summary Report, and Implementor on-screen format. However, EXCEL products require UNCLASSIFIED Summary Report? reducing the font and/or deleting some columns for a single page printout. The WORD printout is more readily usable with fewer modifications required. Only an occasional “non-wrap around” sentence occurs on WORD reports. Monitoring What percentage of your time, during execution, is spent on monitoring? Approx 80% Which tools do you typically use to view/monitor injects: Calendar, Concept - All Injects Map, Status Board, and Status Report? - Calendar Map - Status Report Which export capabilities (i.e., Word, Excel) do you typically use for injects: - We have used both. However, EXCEL products Calendar, Concept Map, Status Board and Status Report? require reducing the font and/or deleting some columns for a single page printout. The WORD printout is more immediately usable with fewer modifications required. Only an occasional “non-wrap around” sentence occurs on WORD reports. How often do your perform a MSEL Sync during execution? Daily What timeframe do you normally review during each MSEL Sync (i.e. next 12 Next 24 hours hrs, 24 hrs)? Do you use the Status Report during the MSEL Sync to update inject statuses? We have relied on the “All Injects” page to review upcoming MSELs. However, after more closely reviewing the view options, the “Status Report” page seems to be better suited for this purpose. How many injects do you typically review during the course of the Sync During our internal small scale exercises we processed before applying the status change? each inject individually Do you use the Monitor Injects feature? We did not use this feature during our small scale exercises. Face-to-Face coordination between the MSEL Manager and the inject POC’s was conducted within the HQ White Cell. Obviously, this feature would be valuable during exercises involving external participants and a greater number of injects If your answer is Yes to the above, what are the parameters typically set to Refresh Rate: for monitoring injects? Max Email Notices: Time Between Notices: Reminder Notice:

General Comments: We have had limited opportunities to use JTIMS/JMSEL. However, we are well versed on the functionality of the application and look forward to being involved with the transition from JMSEL 2.6 to JTIMS/JMSEL.