PROC. OF THE 11th PYTHON IN SCIENCE CONF. (SCIPY 2012) 23 Python’s Role in VisIt Cyrus Harrison‡∗, Harinarayan Krishnan§ F Abstract—VisIt is an open source, turnkey application for scientific data anal- ysis and visualization that runs on a wide variety of platforms from desktops to Python GUI CLI petascale class supercomputers. VisIt’s core software infrastructure is written in Clients C++, however Python plays a vital role in enabling custom workflows. Recent work has extended Python’s use in VisIt beyond scripting, enabling custom State Control Python UIs and Python filters for low-level data manipulation. The ultimate (Python Client goal of this work is to evolve Python into a true peer to our core C++ plugin Viewer Interface) infrastructure. This paper provides an overview of Python’s role in VisIt with a (State Manager) focus on use cases of scripted rendering, data analysis, and custom application Local Components development. network connection Index Terms—visualization, hpc, python Direct Data Introduction Compute Manipulation VisIt [VisIt05], like EnSight [EnSight09] and ParaView (Python Filter MPI Engine [ParaView05], is an application designed for post processing of Runtime) mesh based scientific data. VisIt’s core infrastructure is written in C++ and it uses VTK [VTK96] for its underlying mesh data model. Its distributed-memory parallel architecture is tailored to Cluster Parallel Data process domain decomposed meshes created by simulations on large-scale HPC clusters. Early in development, the VisIt team adopted Python as the Fig. 1: Python integration with VisIt’s components. foundation of VisIt’s primary scripting interface. The scripting interface is available from both a standard Python interpreter and a custom command line client. The interface provides access to via VTK’s Python wrappers and leverage packages such all features available through VisIt’s GUI. It also includes support as NumPy [NumPy] and SciPy [SciPy]. Current support for macro recording of GUI actions to Python snippets and full includes the ability to create derived mesh quantities and control of windowless batch processing. execute data summarization operations. While Python has always played an important scripting role This paper provides an overview of how VisIt leverages Python in VisIt, two recent development efforts have greatly expanded in its software architecture, outlines these two recent Python VisIt’s Python capabilities: feature enhancements, and introduces several examples of use 1) We now support custom UI development using Qt via cases enabled by Python. PySide [PySide]. This allows users to embed VisIt’s visualization windows into their own Python applications. Python Integration Overview This provides a path to extend VisIt’s existing GUI and VisIt employs a client-server architecture composed of several for rapid development of streamlined UIs for specific use interacting software components: cases. • viewer 2) We recently enhanced VisIt by embedding Python in- A process coordinates the state of the system and terpreters into our data flow network pipelines. This provides the visualization windows used to display data. • client provides fine grained access, allowing users to write A set of processes, including a Qt-based GUI and custom algorithms in Python that manipulate mesh data Python-based command line interface (CLI), are used to setup plots and direct visualization operations. * Corresponding author: [email protected] • A parallel compute engine executes the visualization ‡ Lawrence Livermore National Laboratory pipelines. This component employs a data flow network § Lawrence Berkeley National Laboratory design and uses MPI for communication in distributed- Copyright © 2012 Cyrus Harrison et al. This is an open-access article memory parallel environments. distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, Client and viewer proceses are typically run on a desktop ma- provided the original author and source are credited. chine and connect to a parallel compute engine running remotely 24 PROC. OF THE 11th PYTHON IN SCIENCE CONF. (SCIPY 2012) on a HPC cluster. For smaller data sets, a local serial or parallel compute engine is also commonly used. Figure1 outlines how Python is integrated into VisIt’s compo- nents. VisIt both extends and embeds Python. State control of the viewer is provided by a Python Client Interface, available as Python/C extension module. This interface is outlined in the Python Client Interface section, and extensions to support custom UIs written in Python are described in the Custom Python UIs section. Direct access to low-level mesh data structures is provided by a Python Filter Runtime, embedded in VisIt’s compute engine processes. This runtime is described in the Python Filter Runtime section. Python Client Interface VisIt clients interact with the viewer process to control the state of visualization windows and data processing pipelines. Internally the system uses a collection of state objects that rely on a publish/subscribe design pattern for communication among com- ponents. These state objects are wrapped by a Python/C extension module to expose a Python state control API. The function calls are typically imperative: Add a new plot, Find the maximum value of a scalar field, etc. The client API is documented extensively in Fig. 2: Pseudocolor and Streamline plots setup using the script in the VisIt Python Interface Manual [VisItPyRef]. To introduce the Listing 1. API in this paper we provide a simple example script, Listing 1, that demonstrates VisIt’s five primary visualization building blocks: plot is created to display the scalar field named ’hardyglobal’. The mesh is transformed by a ThreeSlice operator to limit the • Databases: File readers and data sources. volume displayed by the Pseudocolor plot to three external faces. • Plots: Data set renderers. We use a query to obtain and print the maximum value of the • Operators: Filters implementing data set transformations. ’hardyglobal’ field. An expression is defined to extract the gradient • Expressions: Framework enabling the creation of derived quantities from existing mesh fields. of the ’hardyglobal’ scalar field. Finally, this gradient vector is used as the input field for a second plot, which traces streamlines. • Queries: Data summarization operations. Figure2 shows the resulting visualization which includes both the Pseudocolor Streamline Listing 1: Trace streamlines along the gradient of a scalar field. and plots. # Open an example file OpenDatabase("noise.silo") Accessing the Python Client Interface # Create a plot of the scalar field 'hardyglobal' For convenience, you can access the client interface from a custom AddPlot("Pseudocolor","hardyglobal") binary or a standalone Python interpreter. # Slice the volume to show only three # external faces. VisIt provides a command line interface (CLI) binary that AddOperator("ThreeSlice") embeds a Python interpreter and automatically imports the client tatts= ThreeSliceAttributes() interface module. There are several ways to access this binary: tatts.x=-10 tatts.y=-10 • From VisIt’s GUI, you can start a CLI instance from the tatts.z=-10 SetOperatorOptions(tatts) "Launch CLI" entry in the "Options" menu. DrawPlots() • Invoking VisIt from the command line with the -cli # Find the maximum value of the field 'hardyglobal' option starts the CLI and launches a connected viewer Query("Max") val= GetQueryOutputValue() process: print "Max value of'hardyglobal'=", val >visit -cli # Create a streamline plot that follows # the gradient of 'hardyglobal' For batch processing, the -nowin option launches the DefineVectorExpression("g","gradient(hardyglobal)") viewer in an offscreen mode and you can select a Python AddPlot("Streamline","g") script file to run using the -s option: satts= StreamlineAttributes() • >visit -cli -nowin -s satts.sourceType= satts.SpecifiedBox satts.sampleDensity0=7 <script_file.py> satts.sampleDensity1=7 satts.sampleDensity2=7 You can also import the interface into a standalone Python satts.coloringMethod= satts.ColorBySeedPointID interpreter and use the module to launch and control a new SetPlotOptions(satts) instance of VisIt. Listing 2 provides example code for this use DrawPlots() case. The core implementation of the VisIt module is a Python/C In this example, the Silo database reader is automatically selected extension module, so normal caveats for binary compatibly with to read meshes from the input file ’noise.silo’. A Pseudocolor your Python interpreter apply. PYTHON’S ROLE IN VISIT 25 The features of the VisIt interface are dependent on the version • The ability to reuse VisIt’s existing set of GUI widgets. of VisIt selected, so the import process is broken into two steps. The ability to utilize renderers as Qt widgets allows VisIt’s First, a small front end module is imported. This module allows visualization windows to be embedded in custom PySide GUIs you to select the options used to launch VisIt. Examples include: and other third party applications. Re-using VisIt’s existing generic using -nowin mode for the viewer process, selecting a specific widget toolset, which provides functionally such as remote filesys- version of VisIt, -v 2.5.1, etc. After these options are set tem browsing and a visualization pipeline editor, allows custom the Launch() method creates the appropriate Visit components. applications to incorporate advanced features with little difficulty. During the launch,