Click to Add Title Click to Add Subtitle (If Applicable) Click to Add

Identifying Command, Control and Communication Networks from Interactions and Activities Observations

Georgiy Levchuk, Ph.D., Aptima Inc. Yuri Levchuk, Ph.D., Aptima Inc.

CCRTS, 2006

Challenges of tactical operations Proposed integrated solution concept Focus of this research: adversary organization identification Technical approach Results and conclusions

Conflicts Trends Technology/Information Trends Adversary’s Trends

70 4500 Past 60 4000 Large-size forces of well-known 3500 50 organizational forms 3000 40 Current 2500 Main telephone lines 30 Small- to moderate-size militia 2000 Mobile cellular forces taking many less #, Millions subscribers 20 1500 Internet users organized forms # of US forces ops forces US # of 10 1000 Future: Adaptive Enemy 0 500 Almost unrestricted ability to 1946-1969 1970-1990 1991-2006 0 connect and coordinate 1992 1998 2004 2010 (est) Years Year Can change size and adapt Past Past structure Slow-time conflict Mainly hard-line communications Numbered engagements Current Internet traffic doubles/year Current 630,000 phone lines installed/week Tactical Planning Issues Asymmetric threats and 50,000 new wireless users/day changing missions 650M SIGINT cables generated High manpower needs (~0.001% to products) Takes long time Future: Increased # of Ops 34M voice mail messages/day High info gaps, complexity, Fast-paced engagements 7.7M e-mails sent/min overload Larger number of and Future: Data Explosion Biases of human decisions higher time criticality Data impossible to analyze manually

Semi-automated System CMDR & planning staff

Battlefield Execute attacks against enemy Gather intel INPUT PRODUCT •comm. intercepts predicted enemy C2 vulnerabilities & counteractions, •events structure, CMDR counter-action disruptions & probes •actions roles, & intent impact assessment plan

Outcome: enemy’s performance Challenges degrades Complexity: large data amounts & –incorrect actions –delayed commands solution space to explore –missed critical information and Criticality: limited time for decision engagements Uncertainty: large data gaps

Mission: US SOF assist the army of Urbanistan (AOU) in Suburbistan

their spring offensive against the territories held by the Urbanistan militant rebel coalition called True Sons of Urbanistan

Dizzikh (TSU). Qashcow ● ● Eddiren Gosh ● Situation: TSU irregulars are well concealed in towns and Ruralstan mountains of their Liberated Zone. Extensive system of tunnels and caves. Porous borders with neighboring countries. Gollyadov: Field Commander of semi- Enemy Command and Control: TSU is led by a independent network of several disparate organizations, formation. Staff of 4-7 ranging from informal cells to military structure. competent personnel. Coordinates with Mountain Martyrs.

“Liberated Lions Brigade : 10-12 people, some with FSU Zone” military training. CMDR Abbadirov – Former Soviet Spetsnaz officer. Mountain Martyrs Detachment Uses cell system. Supported by an Qashcow Tribal Council: 10-15 people, control up to 200 influential deputy and 2-3 senior local fighters. Report to Abbadirov but often stays neutral advisers.

Mission: US SOF assist the army of Urbanistan (AOU) in Suburbistan

their spring offensive against the territories held by the Urbanistan militant rebel coalition called True Sons of Urbanistan

Dizzikh (TSU). Qashcow ● ● Eddiren Gosh ● Situation: TSU irregulars are well concealed in towns and Ruralstan mountains of their Liberated Zone. Extensive system of tunnels and caves. Porous borders with neighboring countries. Enemy Command and Control: TSU is led by a network of several disparate organizations, ranging from informal cells to military structure. Original Plan of the offensive: AOU forces penetrate the Liberated Zone along 4 avenues of approach. NLT H+72 seal the borders. “Liberated Zone”

Mission: US SOF assist the army of Urbanistan (AOU) in Suburbistan

their spring offensive against the territories held by the Urbanistan militant rebel coalition called True Sons of Urbanistan

Dizzikh (TSU). Qashcow ● ● Eddiren Gosh ● Situation: TSU irregulars are well concealed in towns and Ruralstan mountains of their Liberated Zone. Extensive system of tunnels and caves. Porous borders with neighboring countries. Enemy Command and Control: TSU is led by a network of several disparate organizations, ranging from informal cells to military structure. Original Plan of the offensive: AOU forces penetrate the Liberated Zone along 4 avenues of approach. NLT H+72 seal the borders. Defeat larger units of TSU with the help of US air “Liberated assets and AOU armor. Zone”

Mission: US SOF assist the army of Urbanistan (AOU) in Suburbistan

their spring offensive against the territories held by the Urbanistan militant rebel coalition called True Sons of Urbanistan

Dizzikh (TSU). Qashcow ● ● Eddiren Gosh ● Situation: TSU irregulars are well concealed in towns and Ruralstan mountains of their Liberated Zone. Extensive system of tunnels and caves. Porous borders with neighboring countries. Enemy Command and Control: TSU is led by a network of several disparate organizations, ranging from informal cells to military structure. Original Plan of the offensive: AOU forces penetrate the Liberated Zone along 4 avenues of approach. NLT H+72 seal the borders. Defeat larger units of TSU with the help of US air “Liberated assets and AOU armor. NLT H+240 force Zone” TSU fighters into several base towns. Isolate and conduct clean-up operations..

Original Plan of the offensive: AOU forces Suburbistan

penetrate the Liberated Zone along 4 avenues of Urbanistan approach. NLT H+72 seal the borders.

Dizzikh Defeat larger units of TSU with the help of US air Qashcow ● ● Eddiren Gosh ● assets and AOU armor. NLT H+240 force Ruralstan TSU fighters into several base towns. Isolate and conduct clean-up operations.. Uncertainties in the Original Plan: - Movement of the enemy units (and rapid concentration of forces in well protected areas) - Attempted points of massive exfiltration of TSU into the Dizzikh region and/or across the border - Events that may influence the outcome (e.g., cross-border instigation of uprising in Ruralstan’s town of Gosh)

Info to Help Reduce Uncertainty : “Liberated Mountain Martyrs CMDR - Name Zone” - Structure of the enemy C2 Unknown - Locations of the enemy leadership Abbadirov - Ties between TSU and groups in Ruralstan’s Gollyadov town of Gosh NetSTARNetSTAR:: UtilizeUtilize friendlyfriendly inteintell capabilitiescapabilities (US(US multi-INTmulti-INT capabilitiescapabilities plusplus UrbaniUrbani IntService’sIntService’s HUMINT)HUMINT) ttoo fillfill inin thethe blanksblanks inin thethe InfoInfo © 2006, Aptima, Inc. 9 Notional Scenario: Support Spring Offensive of Urbanistan Government

Original Plan of the offensive: AOU forces Suburbistan

penetrate the Liberated Zone along 4 avenues of Urbanistan approach. NLT H+72 seal the borders.

Dizzikh Defeat larger units of TSU with the help of US air Qashcow ● ● Eddiren Gosh ● assets and AOU armor. NLT H+240 force Ruralstan TSU fighters into several base towns. Isolate and conduct clean-up operations.. Summary of NetSTAR findings and follow-up analysis: - Chief Leadership is located at Mountain Martyrs Cell - attempt to exfiltrate in/out/by sea is likely - area is likely well protected; - rapid concentration of forces if assaulted is likely Mountain - Secondary echelon of operational command is Martyrs with Lions Brigade (CMDR Abbadirov) Lions - may conduct a massive exfiltration of TSU into Brigade the Dizzikh region or across the border - likely to initiate distraction maneuvers to divert attention from Chief Leadership Projected enemy strategy: - Two main exfiltration brake-through efforts with possible rapid concentration of forces at Mountain Martyrs area of operations

NetSTAR-aided Plan of the offensive: Suburbistan - AOU forces penetrate the Liberated Zone along Urbanistan 4 avenues of approach. NLT H+72 seal the Dizzikh borders. Concentrate own forces to block Qashcow ● ● Eddiren Gosh ● movement of enemy forces towards Mountain Ruralstan NetSTAR-driven Martyrs. Block sea access. Set up ambush plan groups. modifications - Massive air strikes at Mountain Martyrs and Lions Brigade positions - Defeat larger units of TSU with the help of US air assets and AOU armor. - NLT H+240 force TSU fighters into several base Mountain towns. Isolate and conduct clean-up operations. Martyrs

Lions Brigade

Outcome of applying modified Plan: Enemy force concentration prevented, escape routs blocked; Enemy leadership captured/ annihilated; Enemy forces defeated

Battlefield •disruptions & probes plan •observations NetSTAR-driven COA Projected Enemy recommendations: Structure: Disruption Planning Adversary Identification Augment original plan by HWY 1 North Mountain HCT MI X 2 Mountain Martyrs HWY 2 N C2 Structure Mission concentrating own forces to E W CMDR - Name

EPW Unknown DET

RETRANS West Town X Abbadirov block likely enemy BTB BSB

CP 2 (Main)

MI L&R Gollyadov movement. HCT

X X

X X GCS MI East Village GCS EPW Block sea access. Set up MI DET Communication Networks

RETRANS ambush groups. E W Action Assessment

South Mountain Comparison of impact of disruption strategies HWY 1 GCS ΔΔ=41%=41% HWY 3 400 ΔΔ=30%=30% Conduct massive air strikes ΔΔ=18%=18% MI 350 Δ=39% Δ=39%ΔΔ=36%=36% 300 ΔΔ=20%=20% No Disruption 250 Jam 200 Destroy

150 Reinforce Mission time, min min time, time, Mission Mission 100 Strategy Best Best Strategy Best at Mountain Martyrs and Strategy Worst 50 Worst Strategy

Distributed (Multi-Functional) Specialized (Single-Function) Organization Organization ΔΔ=%=% differencedifference fromfrom solutionsolution withwith nono disruptiondisruption Organization Type

Lions Brigade positions ΔΔ=%=% improvement improvement Sensitivity Analysis Algorithm Comparison comparedcompared to to “no “no disruption”disruption” case case Same impact on 160 140 enemy with fewer 140 Δ=20.7% resources spent 120 120 Δ=43.4% # of successful 100 # of successful enemy 80 Δ=62.7% 100 Reduce enemy •action effects/impact enemy •enemy organization operations success with same 60 operations 80 resources 40 20 60 0 No disruption Attack assets Attack assets Attack assets (high capability) (role-based) (role-based) 40 and CMDR 0 20406080100 nodes Disruption type % of friendly resource used in attacking enemy Projected Enemy Strategy: Two main exfiltration brake-through efforts; Rapid concentration of forces at Mountain Martyrs area of operations if needed

actor-node mapping Organization

Communications/Links Node Activity match observation- rank Mapping action Matching Actions/Events data C3I association network Mission Attributes model C3I network state-based Actor Roles activity model sensors, comm intercepts, text analyses, intel collection Organization Library Behavior C2 org Modeling Predictions Rank-ordering

History mission Organization Mission Likelihood DIVISIONAL f 0.65 Model-generated Environment Mission Library DIVISIONAL d 0.18 Analyst hypotheses FUNCTIONAL f 0.22 FUNCTIONAL d 0.12 Org & behavior theory

© 2006, Aptima, Inc. 13 NetSTAR Sample Outputs Output Illustration Inputs Hypothesized candidate C2 networks Organization 1 Flag Rank & Summary Top-ranked Observed resources and activities/events 3 times as likely as the next “best” Observed (categorized) communications alternative F18S – Actor-to-actor UAV F18S Flat C2 structure –Flag likely overloaded with info – Actor-to-resource (i.e., to field unit … F18S FAB Pair-wise coordinating commander) teams Assumptions UAV … Green is likely the most F18A HH60 vulnerable/overloaded Assume known number of actors and resources (need to determine relationships) Organization 2 Flag Rank & Summary Outputs 2nd-best-ranked 1.2 times as likely Command, control, & communication as the next “best” alternative structure F18S UAV F18S Flat C2 structure – Flag – C2 superior-subordinate hierarchy likely overloaded with info … – Peer-to-peer coordination/ F18S FAB Uneven coordinating teams – some isolated synchronization relationships UAV … Blue is likely the most F18A – Communication network(s) HH60 vulnerable/overloaded – Resource/unit control allocation of Organization 3 Flag Rank & Summary adversary commanders 3rd-ranked 2.7 times as likely Roles of adversary actors in C2 network as the next “best” Joint resource utilization relationships alternative F18S Layered C2 structure – UAV F18S Likely longer info turnaround – slower F18S … reaction to specific events F18A FAB UAV Small coordinating teams HH60 … Evenly balanced load © 2006, Aptima, Inc. 14 NetSTAR Experimental Validation

Previous A2C2 Research RED Military Team: Organization Library Human-in-the-loop simulations: Distributed Process/Activity Models Dynamic Decision-making (DDD) Simulator Focus of the Analyses Agent X Reuse readily available data Agent Y Functional Divisional – A2C2 Experiments Agent Z Agent M Detection focus: Mission Library Agent K – Identification of nodes: Communications Activity Log Transcript Single Asset Multi Asset actor-node mapping Agent Start End Action Sender Receiver Start End Content X 01.10 10.43 Explosives Acquisition X Y 00.03 00.07 “Prepare Explosives” – Identification of Y 05.07 12.32 VBIED Attack Y K 00.10 00.12 “I’m on it.” resource allocation: NetSTAR Research: Uncertainty Model control structure COMPARE Uncertainty Model Reverse Engineering Use “uncertainty model” to Message Decoding Observed Events Chatter Levels (assumed known) introduce noise to data: Agent Start End Action Sender Receiver Start End Content X 01.10 10.43 Explosives Acquisition X Y 00.03 00.07 …“explosives”… – Communications ------Y K 00.10 00.12 --- filtered according to C2 Organization security of channel Agent Y NetSTAR Agent X Agent K Reconstruct the (probability of observing organization and Agent M roles assigned to the Agent Z communication) A2C2 team Agent K – Activities filtered according to class of tasks (probability of A2C2 data observing the Missions executed by joint task force organizations involvement in the task) Communications manually coded Number of human-in-loop experimental logs: 40

Comparison of Impact of NetSTAR over Organization Structure of Red Team 90 80 Impact of NetSTAR 70 increases with increased 60 identified organizational complexity relationships) (% of correctly correctly of (% 50

Recognition Accuracy Recognition 40 30 FD I With NetSTAR W/O NetSTAR Organizations (36 samples per point) Sensitivity of NetSTAR Improvement to Data Uncertainty y

90 Curvilinear impact of 80 70 NetSTAR with increased 60 uncertainty in source data 50 40

identified relationships) 30 20 Recognition Accuracy (% of correctl 30% 50% 70% Uncertainty Level = % of missing data With NetSTAR (36 samples per point) W/O NetSTAR

People Resources

Communication intercepts Intel on individuals How can we recognize the SenderReceiver Start End Content Agent Experience Khalid Majed 1400.03 1400.07 “Prepare Explosives” Almihdhar Weapons manufacturing enemy C3I organization Majed Khalid 1400.10 1400.12 “I’m on it.” Moqed Covert ops. planning given uncertain observations Involvement in activities •Actors Agent Time Action INTELLIGENCE •Resources Khalid 1401.10 Explosives Acquisition Majed 1405.07 VBIED Attack •Communication intercepts •Involvement in activities ? •Intel on individual actors & resources

PROBLEM

sensors Objectives text analyses intel collection Inputs: Observations Automatically identify enemy Command, Control, Tracked Actors and Communication networks and CMDR roles Real-time activity log from events observations and communication Real-time comm. intercepts intercepts Properties

Benefits Environment data C3I Speed-up & improve accuracy of analysts’ network decisions by automating hypotheses assessment Handle large complexity of data and high Organization Library uncertainty Structure model C3I Matching network Technical Approach Products: Adversary Predictions & Ranking Real-time hypotheses testing Organization Mission Actor Roles Hypotheses generated manually from users’ inputs and automatically from historic data & organizational theory Attributed relational graph matching model for actor-to-role mapping – network pattern recognition

NetSTAR model supports intelligence analyses by focusing the adversary organization identification on the small number of most relevant hypotheses about the enemy C2 structure NetSTAR finds the rank-ordering of the hypothesized command, control, communication, and task structures of the adversary and the roles of their decision makers and resource nodes NetSTAR algorithms are based on finding the structural match between observed network of interactions of adversary actors and resources – The model uses the notions of structural consistency and maximizes the likelihood that the observed interaction network between the enemy actors and resources has been generated by a specific hypothetical C2 organization – NetSTAR algorithm uses the information about actors/resources’ involvement in actions to improve the identification of their roles in the organization – NetSTAR model works with missing and erroneous data, deceptions from the adversary, and can handle a large complexity of the C2 networks

Cannot effectively predict enemy’s COAs w/o knowing enemy C2 organization Cannot develop effective enemy HVTs & counteractions w/o knowing enemy C2 organization Might entail unintended consequences if the action is taken w/o full realization of the C2 structure and roles of individuals Past Present Event Event US Civil War, 1861-1865 Israel-Palestinian Conflict, 1948-today

Action Action Death of Confederate cmdr Stonewall Jackson, Elimination of HAMAS founder Sheikh Ahmed Yassin 05/1863 and his successor Abdel Aziz Rantisi, 01/2004

Impact Impact Argued by many historians, was a blow to Army of Huge protests Northern Virginia and contributed to its eventual loss HAMAS enters political process through municipal elections, 2005 Reason HAMAS wins Palestinian parliamentary elections Provoked a reorganization of the command from 01/2006 which the army never fully recovered – Resulted in the division of the army into three Reason infantry corps HAMAS reorganized and solidified, gaining more – Two new corps commanders would prove to strength and people’s support be poor choices

Communication and Command Network Overall Organizational Structure decision-makers Flag communication links Flag

F18S CMDR-to-Asset Control Network UAV F18S control links … assets F18S FAB F18S UAV F18S UAV … F18A HH60 … … F18S UAV F18A HH60 FAB Uncertainty in communication messages, CMDR- Asset-to-Asset Task-coordination Network assets asset commands, asset-asset task execution F18S UAV F18S Missing data: unobserved transactions (modeled with miss … probability) F18S UAV F18A FAB Noisy data: wrongly observed transactions (modeled with false task links … alarm probability) HH60 Object Meaning Attributes Observations (real world equivalent) Communication Who talks to whom about Classes of messages Message between actors and its association Link what Control Link What agents control what “Attack” commands Commands sent from CMDR to asset resources Task Link What assets synchronize Classes of tasks Task execution by multiple assets Nodes Agents & Assets Geographic area, functions Task execution by actor or asset (attacks, recon) © 2006, Aptima, Inc. 22 Solution: Structure Matching via Node Mapping

Hypotheses Observations 1 Rank-order every 2 hypothesis- 5 data pair 3 4 Model 1 Model 2 Model 3 Data Match Match Match

Select Best

Model Network 4 Mapping-1 Mapping-2 Mapping-3 1 1 3 2 5 4

5 3 2 3 5 1 4 4 2

Select Best

Objective: Likelihood or A- Max Structural Consistency Posteriori Function 3 Max Observed Links 4 Min False Observations and Miss

5 1

© 2006, Aptima, Inc. 2 24 How to Create Data Network Tracked Actors Observed Interactions Labeled Network Enemy commanders c d1 1,4 c d4 1,2 a c1,3 1,1 d2 c r1 2,3 Units/Assets a3,1 c 4,5 a d3 2,3 a3,2 r2 r4 d5 r a5,3 3 a5,4 Labels Node labels Vector of values for quantitatively Source: area of responsibility, representing multiple relationship performed functions/tasks, expertise types Example: sniper ops; sales of Value weighs the relationship weapons; money laundering

Link labels Hypotheses Networks Link labels correspond to expected Source: types of messages Example: transfer of information; volume of messages action request; synchronization; etc.

Previous A2C2 Research RED Military Team: Organization Library Human-in-the-loop simulations: Distributed Data Process/Activity Models Dynamic Decision-making (DDD) Simulator Agent X Reuse readily available data from Agent Y Functional Divisional human-in-loop A2C2 Experiments Agent Z

–JTF operations; 42 data samples Agent M Mission Library –Communications manually coded Agent K –Events logged in Communications Activity Log Transcript Complexity Single Asset Multi Asset Agent Start End Action Sender Receiver Start End Content –Number of commanders = 6, X 01.10 10.43 Explosives Acquisition X Y 00.03 00.07 “Prepare Explosives” number of assets = 137 Y 05.07 12.32 VBIED Attack Y K 00.10 00.12 “I’m on it.” –Number of events, comms = NetSTAR Research: Uncertainty Model 1000-4000 COMPARE Uncertainty Model Reverse Engineering Data uncertainty model based on Message Decoding probability of miss, deception, & Observed Events Chatter Levels (assumed known) Agent Start End Action Sender Receiver Start End Content error X 01.10 10.43 Explosives Acquisition X Y 00.03 00.07 …“explosives”… ------Y K 00.10 00.12 ---

Detection Focus C2 Organization Agent Y NetSTAR Agent X Agent K Reconstruct the Identification of nodes: actor-node organization and Agent M roles assigned to the mapping Agent Z A2C2 team Identification of resource allocation: Agent K control structure Experimental Comparisons

Compare results of detecting adversarial organizations as produced by human 2-person test team in 1 hour vs algorithm Calculate the impact of information uncertainty on prediction accuracy

Structure Hypotheses (true) Actual Observations Correct Map: Flag Flag 1=GREEN 5=INF 2=BLUE 6=UAV 3=RED 7=HH60 3 2 1 4=F18S 8=F18A F18S UAV HH60 F18A F18A 8 7 4 9=F18A

INF 9 6 5 Correctly Identified Relationships Misidentified Observations (noise) Undetected Relationships Flag =12 Flag =2 Flag =4

3 2 1 3 2 1 3 2 1

8 7 4 8 7 4 8 7 4

9 6 5 9 6 5 9 6 5

# of Random Mappings (permutations for Other Mapping that looks “similar”: actor-agent X resource-asset groups) = 3!*6! = 1=RED, 2=BLUE, 3=GREEN, 4=F18A, 5=F18A, 6=HH60, 4320 7=F18S, 8=UAV, 9=INF If this mapping was correct: # of Random Mappings for Actual # correct relationships = 12 Experiment-8 data =6!*126!≈10237 (w/o asset # misidentified relationships = 3 class info) or ≈10100 (with asset class info) # undetected relationships = 5 Truth: # correct node associations by this mapping=1 © 2006, Aptima, Inc. 27 Anticipated Results

Comparison of Impact of NetSTAR over Organization Structure of Red Sensitivity of NetSTAR Improvement to Data Uncertainty Team 90 90 80 80 70 70 60 60 50

Recognition Recognition 40 relationships) 50 Accuracy (% of identified relationships) identified correctly identified 40 30 20 30 of (% correctly Accuracy Recognition FD I Low Med High With NetSTAR Uncertainty Level in terms of Humans (W/O NetSTAR) Organizational Types With NetSTAR missing data, erors, and deceptions Humans (W/O NetSTAR)

Improvement of NetSTAR Curvilinear improvement of grows with increased NetSTAR with increased organizational complexity uncertainty in source data

Empirical Observations of Human Team Experiment: Humans had difficulty handling the large volumes of data – did not use all available information Humans exhibited confirmatory biases – used first decision and tried to validate it instead of considering other possible solutions Humans exhibited decision biases – used prior knowledge in their decisions, while it did not apply Humans found the problem very complex – did not finish solving all parts of the problem

Algorithm Automated algorithm is less sensitive to information complexity and data uncertainty Initial results suggest significant improvement in recognition accuracy

Next Steps Integrate node mapping and temporal behavior pattern recognition

Focus Compare results of detecting adversarial organizations as produced by – Human analysts team alone – Automated algorithms

Impact of Uncertainty & Organization Type Calculate the impact of information uncertainty on prediction accuracy Calculate the impact of organization type (problem complexity) on prediction accuracy

Recognition Accuracy High-level: Assess correctness of recognizing operating organization (type) Mid-level: Assess correctness of mapping tracked actors to adversarial decision-making nodes Low-level: Assess accuracy of identifying relationships (resource control, communications, command, task coordination)

Procedure Teams were given observed data and tasked with matching it to 1 of 7 hypothetic C2 structures At the end of each trial, teams developed two products – Surveys measuring: Self-reported workload Selection confidence Perceived Fogging Level Perceived Complexity – Mapping between Commanders, Leaders, & Assets Observed

Commanders CMDR CMDR CMDR CMDR CMDR CMDR Mapping Alpha Bravo Charlie Delta Echo Foxtrot

Green x

Hypotheses Blue x

Purple x

Input Example: Network of Intercepted Control Messages from CMDRs to Red x units/assets Orange x

Original hypothesis Accuracy of Org Type Recognition under Medium Uncertainty “Functional” org. (CMDRs controlling assets of the same type) would be easier to 100 recognize by human analysts due to 80 distinction between CMDRs 60 40

Experiment results Selection Accuracy 0 Showed that in fact it was more difficult to Hybrid Functional Divisional recognize this organization type

Reasons All data was fogged – hence no “clear-cut” picture The affecting feature was not a complexity of the baseline but a closeness of other hypotheses – Among 7 hypotheses, 3 lied closely to Functional org.

20 Closeness between Org Hypotheses and True

Experiment results Selection Accuracy Organizations 0 Hybrid90 Functional Divisional Showed that in fact it was more difficult to 80 recognize this organization type 70 60 Distance to F 50 Distance to D 40 30 Distance to Hybrid Reasons 20 10 resources of CMDRs of resources

Problem: of difference Normalized 0 I All data was fogged – hence no “clear-cut” F D Incorrect F2 F3 D3 picture hypothesis too Hybrid Organizational Structures The affecting feature was not a complexity close to baseline of the baseline but a closeness of other hypotheses – Among 7 hypotheses, 3 lied closely to Functional org. Corrective Action: Recognizing the org type is not main objective Give partial credit for correctly mapping CMDRs’ asset control © 2006, Aptima, Inc. 34 Experiment: Effects of Organization Type

20 Closeness between Org Hypotheses and True

Problem: of difference Normalized 0 I All data was fogged – hence no “clear-cut” F D Incorrect F2 F3 D3 picture hypothesis too Accuracy of CMDR MappingHybrid Organizational Structures The affecting feature was not a complexity close to baseline of the baseline but a closeness of other 80 70 hypotheses 60 – Among 7 hypotheses, 3 lied closely to 50 Functional org. 40 30 20 10 0 Divisional Functional Hybrid © 2006, Aptima, Inc. 35 Experiment: Recognition Accuracy

Original hypothesis Identification of Organization Type

Human analysts achieve higher accuracy 2 than random choice 1.5 Performance degrades with increase in Number Correct 1 solution complexity & uncertainty Solutions Per Team 0.5

Experiment results 0 Partipants performance Chance Showed that in fact human performance was higher than random choice Accuracy of CMDR Mapping The uncertainty results were not conclusive 80 70 Reasons 60 50 Between-team testing 40 30 Accuracy of CMDR Mapping 20 10