Visualization, Exploration and Data Analysis of Complex Astrophysical Data M. Comparato, U. Becciani, A. Costa and B. Larsson INAF-Osservatorio Astrofisico di Catania, Italy B. Garilli INAF-Istituto di Astrofisica Spaziale e Fisica Cosmica, Italy C. Gheller CINECA - Casalecchio di Reno, Italy and J. Taylor Institute for Astronomy, University of Edinburgh - United Kingdom ABSTRACT In this paper we show how advanced visualization tools can help the researcher in investigating and extracting information from data. The focus is on VisIVO, a novel open source graphics application, which blends high performance mul- tidimensional visualization techniques and up-to-date technologies to cooperate with other applications and to access remote, distributed data archives. VisIVO supports the standards defined by the International Virtual Observatory Alliance in order to make it interoperable with VO data repositories. The paper describes the basic technical details and features of the software and it dedicates a large arXiv:0707.2474v1 [astro-ph] 17 Jul 2007 section to show how VisIVO can be used in several scientific cases. Subject headings: Data Analysis and Techniques 1. Introduction The astronomical community has always dedicated special attention to the growth of graphical and visualization tools, driving their evolution or even being directly involved in the development of many of them. { 2 { At present, the most popular software for astronomers can be subdivided into two main categories: tools for image display and processing and tools for plotting data. Notable among the former are IRAF, by NOAO; ESO-MIDAS, by the European Southern Observatory; SaoImage, by the Smithsonian Astrophysical Observatory and GAIA, by ESO. Many other tools are available, but we refer to dedicated surveys for a complete list. Gnuplot and SuperMongo are popular applications adopted for 2D data plots. A more sophisticated solution is represented by IDL, by ITT Visual Information, which is characterized by a large library of functions specifically developed for astrophysics. Again, for a complete list, we refer to specific surveys. Among the most popular N-body visualization codes used by the community there are: TIPSY, motivated by the need to quickly display and analyze the results of N-body simulations, it is mainly limited to this type of data; ParaView, produced by Kitware in conjunction with the Advanced Computing Laboratory at Los Alamos National Laboratory (LANL), the goal of the project is to develop scalable parallel processing tools with an emphasis on distributed memory implementations; IDL, mentioned above, contains support for N-body data display, but is not free software. A new generation of graphic software tools is now emerging. These tools are designed to overcome the limits and the barriers of traditional software by exploiting the latest tech- nological opportunities. The main challenges and objectives are the following: • High performance and multi threading, in order to exploit multi-core systems, large memories and powerful graphic cards and co-processors. This allows the user to handle large amount of data in real-time. • Interoperability, allowing different applications, each specialized in doing different things, to interact with each other in a coordinated and effective way according to well-defined protocols. The aim is to provide to the user a complete suite of tools to best analyze his/her data. Huge, monolithic and often inefficient tools are obsolete. • Collaborative work. The tools allow several users to work on the same data at the same time from different places, exchanging experience, information and expertise. • Access to distributed resources, via web services and/or Grid protocols. Often, data can no longer be moved from data centres as it is too large and complex. The astronomer must have the tools to access it, independently of his geographical location in a fast and reliable way. Tools like VisIVO, Aladin (Bonnarel 2000) and Topcat (Topcat), have been recently devel- oped in the framework of the Virtual Observatory (www.ivoa.net) to achieve all or some of { 3 { these goals. In this paper we will focus in particular on VisIVO, which stands for Visual- ization Interface for the Virtual Observatory. VisIVO is being developed as a collaboration between the Italian National Institute for Astrophysiscs (INAF) - Astrophysical Observa- tory of Catania and CINECA (the largest italian academic high performance computing centre) in the framework of the FP6 EU funded VO-Tech project. The next section gives a short review of the basic functionality of VisIVO, while section 3 describes PLASTIC, a messaging protocol which allows heterogeneous applications to work together. Part of sec- tion 3 and section 4 are dedicated to presenting several scientific cases in which the support of graphics and visualization is of primary importance. In these sections we show some of VisIVO's capabilities in action, demonstrating how they can be effectively used in practical applications. 2. VisIVO VisIVO is a C++ application specifically designed to deal with multidimensional data. It is free software available both for MS Windows and for GNU/Linux (porting to MacOS is in progress). It can be downloaded from the web site http://visivo.cineca.it. The soft- ware is built on the top of the Multimod Application Framework (MAF)(Viceconti 2004). MAF is an open source framework for the development of data visualization and analysis applications. It provides high level components that can be easily combined to develop a vertical application. It is being developed by the visualization group of CINECA and it can be downloaded from the web site http://openmaf.cineca.it. The framework is based on the Visualization ToolKit (VTK) (Schroeder 2004) library for the multidimensional visual- ization and on the wxWidgets library, a portable Open Source GUI library, for the user interface. It incorporates other open libraries, (for example for data encryption) or drivers for virtual reality devices (3D mouse, gloves, haptics etc.). VisIVO's architecture strictly reflects the structure of a typical scientific application built with the MAF, being mainly developed in the highest layers of the framework. The software exploits, wherever possible, the standard visualization services, views, operations and interface structures provided by the framework and implements all the elements that are specific to the visualization and analysis of astronomical data. Extensions to the basic MAF infrastructure have been developed in order to match astronomy-specific requirements and to provide the highest performance. Internal data rep- resentation is in the form of a Table Data structure, which is composed of a sequence of variables loaded from a data source such as a file or a database. Regardless their original type, variables are all converted to double format. While this incurs a penalty in the appli- { 4 { cation's memory needs, it provides the necessary precision in some of the data processing stages. Once a table is loaded the user can manage and visualize the data. These operations do not increase the memory usage as long as they do not create new tables or new fields: the visualization process is carried out using references to the Table Data with no data replica- tion. In order to visualize data, the user has to set which of the loaded fields will be used as the coordinate system of a Cartesian reference frame. In this way, the software ensures maximum flexibility in data usage. 2.1. VisIVO for data visualization Data visualization is the main target of VisIVO. The software is designed to simultane- ously handle as many properties as possible. Complex tables can be loaded and manipulated, new fields can be derived and finally represented graphically, using points, colours, trans- parencies, surfaces, glyphs and volume rendering. The first step of a working session is usually data loading. Data can be read from files; VisIVO supports different kinds of file formats: standard file formats, like VOTables, FITS, HDF5, ASCII, raw binaries and the native data format of the popular Gadget simulation code (Springel 2000). The VOTable format is an XML standard for the interchange of astronomical data, defined by the Inter- national Virtual Observatory Alliance (IVOA, http://www.ivoa.net). Data is represented as a set of tables, each table being an unordered set of rows, whose format is specified in the table XML metadata. Rows are sequences of table cells, each containing either a primitive data type or an array of such primitives. VOTables can also contain links to external files as a separate data source. VisIVO uses the Savot VOTable parser developed by CDS1 to load and write VOTables. FITS and HDF5 importers are implemented using the published API and libraries. The ASCII table format consists of columns of data spaced with the most common separation characters (space, tab etc.). Raw files are sequences of variables written as binary dumps of the memory. The binary files can be managed by descriptor files which store the associated information (number of variables, data types etc.). VisIVO can also interact with CDS VizieR data service (Ochsenbein 2000), retrieving data directly from remote archives (see par. 2.3). Once data is loaded it can be visualized and analyzed. VisIVO can deal with both structured and unstructured data. The former is represented by fields defined on a regular mesh. The latter is data with no special geometry; it is treated as sets of points. Graph- ically, unstructured data have a default representation as pixels. The points geometrical 1Centre de Donnes astronomiques de Strasbourg { 5 { Fig. 1.| 3D data display of 16 million-particles N-Body simulation and data from VizieR Astronomical Server cone search: the Hipparcos and Tycho Catalogues (ESA 1997) { 6 { distribution is set by selecting three of the fields loaded as Table Data. For example: the x, y, z coordinates of the particles from a N-Body simulation, or the RA (Right Ascension) - DE (Declination) - Vmag (Johnson magnitude V) fields for data from stellar catalogues (figure 1).