for Maritime Domain Awareness

Valérie Lavigne, Denis Gouin Michael Davenport Defence R&D Canada - Valcartier Salience Analytics Inc. Québec (QC), Canada Vancouver (BC), Canada

Abstract—Maintaining situation awareness in the maritime domain is a challenging mandate. Task analysis activities were III. VISUAL ANALYTICS conducted to identify where visual analytics science and Thomas and Cook define VA as “the science of analytical technology could improve maritime domain awareness and reasoning facilitated by interactive visual interfaces” [3]. The reduce information overload. Three promising opportunities goal of VA is to facilitate high-quality human judgement with a were identified: the of normal maritime behaviour, limited investment of the analyst’s time. According to them, anomaly detection, and the collaborative analysis of a vessel of people use VA tools and techniques to: interest. In this paper, we describe the result of our user studies along with potential visual analytics solutions and features • “synthesize information and derive insight from considered for a maritime analytics prototype. massive, dynamic, ambiguous and often conflicting data; Keywords-visual analytics; maritime domain awareness; anomaly detection; situation analysis; collaborative work. • detect the expected and discover the unexpected; • provide timely, defensible, and understandable I. INTRODUCTION assessments; In this paper, we report on the results of work conducted by • Defence R&D Canada (DRDC) in the exploration of Visual communicate assessment effectively for action.” [3] Analytics (VA) solutions to real world maritime challenges and VA can help analyze entity behaviour, known patterns, and describe the design of a Maritime Analytics Prototype () the links between them. implementing VA features that will enable better situation understanding. First, we discuss Maritime Domain Awareness IV. MARITIME TASK ANALYSIS (MDA) and its challenges. Results from the task analysis studies identifying VA opportunities are presented, followed by A. Methodology related work. Then, we describe our proposed approach for the MAP along with some of the apps designs for various maritime Task analysis activities were conducted to identify the analysis tasks. We conclude with our plan for future work. opportunities and potential requirements for the application of VA in the RJOCs.

II. MARITIME SURVEILLANCE Our methodology included the review of previous MDA is “the effective understanding of everything on, knowledge elicitation studies from related DRDC research under, related to, adjacent to or bordering a sea, ocean or other projects [4][5]. Then, we organized discussion sessions at the navigable waterway, including all maritime-related activities, RJOCs in Halifax and Esquimalt. We were also able to observe infrastructure, people, cargo, and vessels and other duty personnel in action in their work environment. The conveyances that could impact the security, safety, economy, domain knowledge was gathered through a mix of interviews, or environment.” [1] observations, and group discussions, then documented in a report [6]. In Canada, the Canadian Forces Regional Joint Operational Centers (RJOCs) in Esquimalt and Halifax maintain a 24/7 B. Opportunity Identification Results watch over Canada’s three oceans. The production and The latest requirement elicitation activity identified nine exploitation of the Recognized Maritime Picture (RMP) is an opportunities for applying VA to MDA. This paper summarizes essential part of their work. Watch personnel continually our current R&D activities focused on the first three: interpret the maritime situation with reference to space and time. They need to rapidly process a variety of sources of • Visualizing Normal Maritime Behaviour (VNMB) information in order to develop shared situational awareness. • This is a challenging task because of the large number of vessel Surveillance and Anomaly Detection (SAD) contacts and heterogeneous information sources that must be • Collaborative VA of a Vessel of Interest (CVAV) monitored. This leads to a significant information overload [2]. Fortunately, VA has emerged as a new and multidisciplinary We believe that these applications have the highest field that helps turn this information overload into an potential to gain improvement from VA solutions. However, opportunity. the other opportunities are well documented and could be the basis for future research projects [6]. V. RELATED WORK Visual exploration strategies can be used to extract patterns from past behaviour in a dynamic geo-spatial area of interest. A. Visual Analysis of Trajectories For example, the study of Somali pirate attacks [21] shows that Extensive work on the visual analysis of movement data led the attack pattern changed in response to the creation of the to the definition of aggregation methods suitable for movement Maritime Security Patrol Area. data and to VA ways to visualize and explore the statistical To the best of our knowledge, VA techniques have not been patterns from trajectories [7]. Interactive Kohonen were used in focused analysis of specific entities in the maritime also used for trajectories aggregation [8]. domain. General trends were often identified but the visual Clustering of trajectories helps analyze large group analysis of individuals has received much less attention. behaviours, but it does not easily allow visual identification of anomalous tracks. To this end, [9] produced ship density VI. MARITIME ANALYTICS PROTOTYPE landscapes in which ships that are off historic routes and The intent of the Maritime Analytics Prototype (MAP) is to regular traffic lanes visually stand out. Vessel movement improve the visualization of the recognized maritime picture patterns can also be characterized using hybrid fractal/velocity and help analysts better comprehend a situation and anticipate signatures [10]. As analysts learn to visually interpret these how it could develop. signatures, they can recognize anomalous activities. Innovative tools for geotemporal visualization of trajectories exist [11]. A. Mix of Original and Existing Concepts B. Maritime Anomaly Detection Our proposed Maritime Analytics Prototype (MAP) is a mixture of original innovative concepts and existing VA and With some exceptions (e.g. [9] and [10]), published work information visualization concepts that are not presently regarding visual detection of anomalies is mostly concerned applied to MDA. We may also integrate available academic with network security [12]. The geo-temporal characteristics of and commercial VA solutions such as GeoTime [11], JigSaw vessel trajectory data are very different from computer [22] and nSpace [23]. intrusions, so new strategies are required. Vessel track analysis, for example, is multidimensional and can be affected by factors The contribution of this work is twofold: such as: time of the day/week/year, seasons, meteorological • conditions, economic trends, and special events. Existing theoretical and applied VA approaches have been extended and assembled into a suite that is When considering maritime anomaly detection specifically, optimized for this application. we rapidly find that research has largely focused on developing • automated system that suggests potential anomalies to an New VA innovative concepts have been developed in operator for further investigation. Many of these maritime response to specific maritime domain requirements. anomaly detection systems rely on models of normal/abnormal vessel kinetic behaviour to detect anomalies. B. Apps Design Approach Because of their 24/7 operational context, the RJOCs have Self organizing maps were used and user can be involved in for many years upgraded their information technology through the anomaly detection process using interactive visualizations a series of incremental improvements, rather than through [12]. Reference [13] uses a Gaussian mixture model for system replacement cycles. Accordingly, the MAP will consist maritime anomaly detection while [14] uses a Bayesian of a set of services that could be added incrementally to a network. Spline-based trajectory clustering techniques were service-oriented architecture. The MAP has thus been designed proposed by [15] to represent normal vessel behaviour for as a set of compact software applications (apps), with the coastal surveillance. The use of a neurobiologically inspired following characteristics: algorithm for probabilistic associative learning of vessel motion was suggested by [16]. • Transient: each app can pop up when needed and then Rule-based expert systems rely on artificial intelligence hide away under an icon when not needed. rules derived from human expert knowledge acquisition • Self-contained: apps can be added as a service without activities. Such a system was built at DRDC [17], which we requiring changes to the or to other will use as an input in our prototype. This approach was also applications. employed by [18]. • Single-purpose: users experience each app as doing C. Maritime Situation Analysis one specific job. Although the geospatial trajectory of a vessel is its most DRDC Valcartier and Salience Analytics Inc performed salient signature, maritime situation assessment requires the design studies activities that led to 24 apps concepts [24] analysis of more varied data such as port visit history, owner related to the three selected MDA opportunities. This paper relationships and suspected criminal activities. The advantages describes a subset of these apps. of visualization for data mining applications are outlined by [19] and a number of examples are provided. The tight link C. Workspace and App Icon Collection between VA and data mining is also discussed by [20]. There will be a common operational context in which all the apps will operate. The MAP workflow will allow easy movement between different interconnected views of the Because vessel icons are set by a military standard, we need domain by letting users select, customize, and arrange the an alternative to color coding to show set membership. Bubble views that contribute to their current task, effectively providing sets [26] can group many entities visually into a single blob a user-defined operational picture [25]. Commonly-used without affecting their spatial organization. Considering the arrangements can be saved and deployed. Automatically shape of these blobs can even allow additional analysis configured layouts based on the user role will also be provided. capabilities. Any ship that is far away from the other members of the group will become salient as the blob shape stretches in Portal views may be stacked on a single screen, distributed its direction. This shape could become a feature that can be across multiple screens, or distributed between devices such as easily recognized for known fleets and any change would a laptop, a wall display or a tablet (Fig. 1). Views generated hence be easily noticed. Fig. 2 shows a bubble set used in externally (e.g. a video feed of the harbour) can also share the conjunction with thematic views. workspace.

Figure 1. Multiplatform Maritime Analytics Prototype mock-up.

VII. VISUALIZING NORMAL MARITIME BEHAVIOUR The goal of VNMB is to use VA to produce and record a description of what is normal in an area of interest. The available knowledge about maritime behaviours, trends and Figure 2. Bubble sets and thematic views. patterns is largely built up from individual analyst’s experience and mentoring from more experienced analysts. Once patterns B. Thematic Views are identified, pictures and annotations could be used to visually display this knowledge. The normal pattern of life In order to reduce information overload, we need to allow would then be more easily transmitted to new analysts. users to focus on the specific subsets of information required for their current task. In the thematic views app, visual layers Various elements of information are of interest to help separate the information elements into various themes such as understand normal maritime behaviour such as historical commercial transportation, Vessels of Interest (VOIs) commercial routes and fishing areas. Analysts must also surveillance, sensor coverage analysis, or alerts from automatic understand how external factors, such as seasonal activities, anomaly detection systems. The possibility to hide or show meteorological conditions and economic influences affect the individual layers of information allows the user to view only maritime activities or sensor performance. the relevant information needed for a particular task. Various combinations of visualizations can be proposed. A. Bubble Sets for Fleets We can create generic layouts for tasks or topics that are Exploration of the dataset is critical to support the common and recurrent, as well as allowing users to create their analytical process and develop insight. The number of tracks own custom visual layouts employing the tools that suit their that can be displayed in the RMP is overwhelming. Just as needs. An example of this would be to display together fishing contact data have been resolved into tracks, tracks should be vessels, regulatory fishing zones, meteorological data and some clustered into groups in order to reduce visual clutter. The annotations from a previous normal behaviour analysis (Fig. 2). whole maritime dataset can then be explored one set at the A visualization of tracks on the map along with the boundaries time, while maintaining the possibility to drill down and look at of a region of interest allows direct assessment of their relative individual vessels at anytime. position. This could be useful, for instance, to see if fishing Creating groups of tracks can be done using automatic or ships are actually in fishing zones or to detect ships that have manual selection. Allowing queries to all tracks that meet failed to provide their 24 hr or 96 hr call-in reports. specified criteria (such as time windows, spatial zones, ship sizes, destinations, range of headings) will enable the analyst to VIII. SURVEILLANCE AND ANOMALY DETECTION filter the dataset. A number of tracks can then be selected to An important goal of surveillance is to detect anomalies in manually create a new set. Various automatic clustering the current situation. A huge fraction of the information algorithms were mentioned in the Related Work section. presented to RJOC operators is mundane, from entities going The close-encounters icons can pop-up when we do a about normal, legitimate activities. As a result, important mouse-over on the tracks and instantly reveal if meetings may anomaly information can be masked by the sheer volume of have happened. It can also be included in a vessel summary data. card. For wide-area surveillance, a single button will create pop-ups for all tracks of interest, so that a quick check can be One approach often considered is automatic anomaly done for all tracks (Fig. 4). detection. To this end, a DRDC project [17], has successfully explored the use of rule-based and description logic expert systems. Validation of alerts still needs to be done by human analysts. There are some anomalies however, either difficult to describe in a formal language or simply unexpected, that the automated rules will miss. In contrast, VA detection of anomalies takes full advantage of the human ability to explore, create hypotheses and analyze what is going on. The computational strength of the machine is exploited in another way through the use of clever visualization, automatic clustering, and interactive analytical algorithms. Anomaly detection is closely related to VNMB because an analyst needs to know what is normal to detect what is not. Patterns identified in VNMB can be visually compared to the Figure 4. Close encounter pop-up icons on the map. current situation, or can be used to suppress mundane data, so that anomalies stand out. Even though we will leverage B. Vessel Summary Cards automated reasoning for anomaly detection from previous activities [17], the MAP focuses on innovative visual apps that The information elements related to an individual vessel help the operator detect unexpected anomalies by exploring the may be of different natures (e.g. tracks on maps, crew and maritime situation. passenger lists, flag, , spreadsheets of financial transactions, schedules, textual documents). Visual A. Close-Encounters Pop-Ups representations of these information bits can be assembled to create a visual vessel summary as shown in Fig. 5. If we consider the situation where an analyst wants to check if a rendezvous has taken place, the visualization of tracks on a The vessel summary cards show all the key characteristics map does not allow this fact to be verified rapidly. The operator of a vessel at a single glance and analysts can rapidly flip would need to use a time slider and replay the scene or turn on through a virtual deck of summary cards. Information is the time labels for each contact and mentally compare them. formatted so that the analyst can look for normally present or absent elements rather than having to read each card. The cards The proposed encounter icon in Fig. 3 gives a temporal are updated for each vessel journey and previous versions are overview for the narrow vicinity of a specific vessel. The icon accessible. Details are available on demand by clicking the is centered on the frame of reference of the current ship. A corresponding visual element. simple way to picture this concept is to imagine the icon moving along the ship track while other vessels leave a trace in it as they cross it. This tells the operator instantaneously if any other vessel has been in proximity to this ship anywhere during its complete journey.

Figure 5. Tablet browser for visual vessel summary cards.

C. Route Ribbons Figure 3. Examples of close-encounter pop-up icons. Transcontinental ships follow predictable routes, and any deviation from those routes is noteworthy. Route ribbons visually represent deviation from travel plans, as sketched in Fig. 6. The current track is displayed as a line. The space between this line and the expected route is filled with color, creating the “ribbon”. The ribbon gets larger as the vessel deviates from its normal route, thus increasing saliency as the behaviour becomes anomalous. An analyst can pop-up a temporary route ribbons overlay to get an instant visual indication of how well transcontinental ships are staying on great circle routes. The ribbons also show if vessels are not heading to their stated destination

Figure 7. Mind map with time slider.

By explicitly showing this temporal evolution, we expect that analysts will have better insight into the reasons for past decisions, and will thus be more empowered to correct mistakes when they are discovered. The temporal tagging of all the information makes it easier to later explain the thought process that led to the conclusions presented in reports, helping decision tracking and auditing. It will only be useful, however, if Mind Map data can be collected using a mechanism that requires so little time and energy that operators do not perceive it as an added drain on their time. Figure 6. Route ribbon example. B. Predictive Rendezvous IX. COLLABORATIVE ANALYSIS OF A VESSEL OF INTEREST Fig. 8 shows a simple app for quickly visualizing when and The RJOCs and MSOC agencies do not have enough where a rendezvous or interception might occur, given sparse manpower to fully analyze and interpret every vessel in tracking information. In this example the analyst suspects two Canada’s area of responsibility so they use triage to identify ships of planning to intercept and harass an oil tanker. The Vessels of Interest (VOIs) that are worthy of special attention. time slider at the top left is adjusted to explore a potential intercept time and the app shows: Once a VOI has been declared, multiple agencies and • individuals collaborate to collect, interpret, and disseminate as Where the tanker would be at the specified time, much information as possible about the vessel. Within Canada, • How close the two attacking ships could be at that six federal departments are involved as part of the Maritime time, if travelling at top speed, and Security Operation Center (MSOC). • When an aircraft would have to leave the nearest Detailed analysis of a VOI is necessary to understand the airbase, to arrive on-scene at that time. intentions of the ship and whether it may represent a threat. To this end, the information that need to be visualized is a mixture A key element of this App is the real-time re-mapping of of ship tracks, photographs, schedules, self-reported the scenario as the time slider is adjusted. This provides a information, commercial facts, and intelligence information. dynamic visual component that summarizes a complex what-if decision space. The colored time slider also indicates the A. Mind Map with Time Slider interval in which the vessel trajectories intercept. Commercial VA solutions such as GeoTime [11] have demonstrated the value of a tool in which observations can be collected, hypotheses tested, and a story constructed to interpret the evidence. Our prototype will incorporate such a Mind Map app and will assess the value of explicitly visualizing the time evolution of the analytical story being told. As with other such tools, users can drag and drop into the Mind Map new evidence or snapshots of the current maritime picture. When reviewing the evidence, analysts can choose to display the nodes and links on a timeline (see Fig. 7) to provide the temporal context (past and future elements are faded). Expected or planned events (e.g. the departure of a ship) can also be placed in the future of the Mind Map App. Figure 8. Predictive rendezvous analysis. C. Enabling Collaboration Visual Analytics Science and Technology 2008 (VAST 2008), pp. 51- 58. The VA prototype will be tested in a physical collaborative [8] T. Schreck, J. Bernard, T. Tekusová and J. Kohlhammer, “Visual Cluster environment that combines individual workstations, large Analysis of Trajectory Data with Interactive Kohonen Maps”, Proc of group displays, and network infrastructures with collaboration the IEEE Symposium on Visual Analytics Science and Technology 2008 services such as conferencing. A multi-touch will be used (VAST 2008), pp. 3-10. as a shared workspace where information from different [9] N. Willems, H. van de Wetering and J.J. van Wijk, “Visualization of departments can be brought together to discuss a VOI situation. Vessel Movements, Forum”, In Proc. of EuroVis We expect this advanced collaborative environment to improve 2009, 28(3), p.959-966. the collaboration of co-located and virtual teams. We favour [10] R.A. Enguehard, R. Devillers and O. Hoeber, “ of Fishing Vessel Movement Patterns Using Hybrid Fractal/Velocity use a web based approach, leading to easier multi-device use. Signatures”, GeoViz Workshop, 2011. [11] T. Kapler, R. Eccles, R. Harper and W. Wright, “Configurable Spaces: X. CONCLUSION Temporal Analysis in Diagrammatic Contexts”, IEEE Visual Analytics Science and Technology (VAST 2008), October 19-24, 2008, Columbus, In this paper, we identified a subset of maritime Ohio, USA. applications and tasks where the use of VA has a high [12] M. Riveiro, G. Falkman and T. Ziemke, “Visual Analytics for the improvement potential and focused on improving the Detection of Anomalous Maritime Behavior”, IEEE 12th International visualization of normal maritime behaviour, the detection of Conference Information Visualisation, 2008. anomalies and the collaborative analysis of a VOI. [13] R. Laxhammar, 2008. “Anomaly Detection for Sea Surveillance”, Proc. of the 11th International Conference on Information Fusion (Fusion The VA solutions we propose will offer cognitively rich 2008), 47–54, (June-July 2008). visual representations that will lead to a more efficient [14] F. Johansson and G. Falkman, “Detection of Vessel Anomalies - a exploration and analysis of the current maritime situation. Bayesian Network Approach”, Proc of the 3rd International Conference These tools will also support communication of analysis results on Intelligent Sensors, Sensor Networks and Information Processing, 2007. and teamwork collaboration. With our apps-based design [15] A. Dahlbom and L. Niklasson, “Trajectory Clustering for Coastal approach, we propose new innovative VA capabilities and Surveillance”, Proc of the 10th International Conference on Information apply existing VA concepts to a real world problem. Fusion (Fusion 2007), July 2007. Although all the proposed apps are tailored to address [16] B.J. Rhodes, N.A. Bomberger and M. Zandipour, “Probabilistic Associative Learning of Vessel Motion Patterns at Multiple Spatial specific cognitive overloads that have been identified by RJOC Scales for Maritime Situation Awareness”, Proc of the 10th International personnel, the operational value of each app cannot be assessed Conference on Information Fusion (Fusion 2007), July 2007. until the prototype has been built and controlled evaluations [17] J. Roy, “Rule-Based Expert System for Maritime Anomaly Detection”, done. Proc. of the 12th International Conference on Information Fusion (Fusion 2009), July 2009. Our future work will focus on implementing the maritime [18] J. Edlund, M. Grönkvist, A. Lingvall and E. Sviestins, “Rule Based analytics proof-of-concept prototype. User validation and Situation Assessment for Sea-Surveillance”, Proc of SPIE -Multisensor, evaluation activities will be conducted. They will provide an Multisource Information Fusion: Architectures, Algorithms and opportunity to integrate user suggestions and preferences into Applications, vol. 6242, 2006. future versions. Advancement regarding the VA prototype will [19] D.A. Keim, “Information Visualization and Visual Data Mining”, IEEE be described and discussed further in subsequent papers. Transactions on Visualization and Computer Graphics, 7 (1), 2002. [20] A. Perer, and B. Shneiderman, “Integrating Statistics and Visualization for Exploratory Power: From Long-Term Case Studies to Design REFERENCES Guidelines”, IEEE CGA, pp. 39-51, 2009. [1] National Security Presidential Directive/Homeland Security Presidential [21] UNOSAT, “Pirate Attack Density in the Gulf of Aden”, Product ID: Directive, Maritime Security Policy, NSPD-41/HSPD-13, 2004. 1298 – English, 26 Nov, 2008. [2] W.M. Renaud and A.W. Isenor, “Situational Awareness: Inspiring [22] J. Stasko, G. Carsten and L. Zhicheng, “Jigsaw: Supporting Investigative Operational Clarity”, Canadian Military Journal, 10 (4), 2009. Analysis through Interactive Visualization”, Information Visualization, [3] J.J. Thomas and K.A. Cook, eds., “Illuminating the Path: The Research 7(2), p.118-132, 2008. and Development Agenda for Visual Analytics”, IEEE Computer [23] W. Wright, D. Schroh, P. Proulx, A. Skaburskis and B. Cort, “The Society Press, 2005. Sandbox for Analysis - Concepts and Methods”, ACM CHI, 2006. [4] M. Davenport and S. Franklin, “Visions of a Future Maritime Picture”, [24] M. Davenport, V. Lavigne and D. Gouin, “Design of a Maritime Domain MacDonald Dettwiler and Associates Ltd., DRDC Scientific Authority: Awareness Visual Analytics Prototype (DMVAP), Task 1: Design David M. F. Chapman, DRDC Atlantic CR 2006-038, May 2006. Study”, Salience Analytics, DRDC Scientific Authority: Valérie [5] M. Davenport, “Maritime Domain Awareness Knowledge Management Lavigne, DRDC Valcartier CR 2011-221, July 2011. Requirements”, MacDonald Dettwiler and Associates Ltd., DRDC [25] S. Mulgund and S. Landsman, “User Defined Operational Pictures for Scientific Authority: Jean Roy, DRDC Valcartier CR 2007-174, July Tailored Situation Awareness”, in the Proc of the 12th International 2007. Command and Control Research and Technology Symposium - [6] M. Davenport, “Opportunities for Applying Visual Analytics for Adapting C2 to the 21st Century, Newport, RI, June 19-21, 2007. Maritime Awareness”, MacDonald Dettwiler and Associates Ltd. & [26] C. Collins, P. Gerald and S. Carpendale, “Bubble Sets: Revealing Set Salience Analytics, DRDC Scientific Authority: Valérie Lavigne, DRDC Relations with Isocontours over Existing Visualizations”, IEEE Valcartier CR 2009-227, October 2009. Transactions on Visualization and Computer Graphics (Proc. of the [7] G. Andrienko and N. Andrienko, “Spatio-temporal Aggregation for IEEE Conf. on Information Visualization), 15 (6), Nov/Dec 2009. Visual Analysis of Movements”, Proc. of the IEEE Symposium on