DD143X EXAMENSARBETE INOM DATALOGI, GRUNDNIVÅ, 15 HP STOCKHOLM, SVERIGE 2016 Heatmap Visualization of Neural Frequency Data Visualisering av neural frekvensdata som värmekarta RODRIGO ROA RODRÍGUEZ ROBERT LUNDIN Supervisor: Alexander Kozlov Examiner: Örjan Ekeberg KTH SKOLAN FÖR DATAVETENSKAP OCH KOMMUNIKATION Abstract Complex spatial relationships and patterns in multivariate data are relatively simple to identify visually but difficult to detect computation- ally. For this reason, Anivis, an interactive tool for visual exploration of multivariate quantitative pure serial periodic data was developed. The data has four dimensions depth, laterality, frequency and time. The data was visualized as an animated heatmap, by mapping depth and laterality to coordinates in a pixel grid and frequency to color. Transfer functions were devised to map a single variable to color through parametric curves. Anivis implemented heatmap generation through both weighted sum and deconvolution for comparison reasons. Deconvolution exhibited a to have better theoretical and practical performance. In addition to the heatmap visualization a scatter-plot was added in order to visualize the causal relationships between data points and high value areas in the heatmap visualization. Performance and applicability of the tool were tested and verified on experimental data from the Karolinksa Institute’s Department of Neuroscience. Abstrakt Komplexa spatiala m¨onster och f¨orh˚allanden i multivariat data ¨ar rel- ativt sv˚ara att identifiera via ber¨akningar men simpla att identifiera vi- suellt. Att visualisera data f¨or denna typ av data-analys anv¨ands ofta i m˚anga olika typer av f¨alt. Detta motiverade utvecklingen av Anivis; ett interaktivt verktyg f¨or visuell utforskning av multivariat kvantitativ data av neural aktivitet. Anivis anv¨ander sig av dataset baserade p˚aexperi- mentell data fr˚an en forskningsgrupp p˚aKarolinska Institutets Institution f¨or Neurovetenskap. Dessa fyrdimensionella dataset best˚ar av m¨atningar fr˚an neuroner i form av deras position, aktivitet i form av frekvens och tidpunkt. Denna data anv¨ands f¨or att generera en animerad heatmap, d¨ar neuroners frekvensv¨arden visas i form av f¨arg. Frekvensv¨ardena om- vandlades till f¨argv¨arden via ¨overg˚angsfunktioner som kopplar numeriska v¨arden till f¨argv¨arden via parametriserade kurvor. Anivis lyckades imple- mentera tv˚aolika metoder f¨or att generera heatmap, viktade summor och dekonvolution. Dessa tv˚ametoder j¨amf¨ordes med varandra, varav dekon- volution visade sig vara den teoretiskt och praktiskt e↵ektivaste meto- den. Utvecklingen av Anivis visade ¨aven behovet f¨or ett punktdiagram f¨or att visualisera f¨orh˚allandet mellan m¨atta frekvensv¨arden och spatial frekvensf¨ordelning i heatmappen. Contents 1 Introduction 1 1.1 Problemstatement .......................... 2 1.2 Scope and limitations . 2 1.3 Disposition .............................. 3 2 Background 4 2.1 Neurons................................ 4 2.2 Reflexes ................................ 4 2.3 Previousstudies ........................... 4 2.4 Visualization . 6 2.5 Color Spaces . 7 2.5.1 Monochrome . 8 2.5.2 RGB.............................. 8 2.5.3 HSL . 9 2.5.4 HCL . 10 2.6 Color transfer functions . 11 2.7 Heatmaps . 12 2.8 Kernelimagefiltering ........................ 13 3 Method 14 3.1 Datastructure ............................ 14 3.2 Visual mappings . 15 3.2.1 Mapping depth and laterality to on screen coordinates . 15 3.2.2 Mapping frequency to color . 15 3.3 Heatmap calculation . 16 3.4 Animation . 17 3.5 Interaction .............................. 17 4 Results 18 4.1 Tools and design choices . 18 4.2 Scatter-plot implementation . 19 4.3 Heatmap implementation . 20 4.3.1 Weightedsum......................... 20 4.3.2 Deconvolution . 21 4.4 Color transfer functions . 23 4.4.1 Grayscale ........................... 23 4.4.2 HSL Rainbow . 24 4.4.3 Cubehelix........................... 25 4.4.4 HCL Heat . 26 4.5 Rendering .............................. 27 4.6 Animation . 28 4.6.1 Animation loop . 28 4.6.2 Frame rate capping . 28 4.7 Interactivity.............................. 29 5 Discussion 30 5.1 Heatmaps in Anivis . 30 5.2 Image quality of deconvolution . 31 5.3 Scatter-plot as a complement to heatmap visualization . 32 5.4 Futureresearch ............................ 32 6 Conclusion 33 1 Introduction In neuroscience, the study of reflex related neural activity seeks to correlate activity patterns with the di↵erent reflexes. Neural activity consists of oscilla- tions made of individual neuron spikes. Experiments by Zelenin, Hsu, Lyalka, Orlovsky, and Deliagina (2014) suggest that the neuron activity overarchingly matches motor response. This makes it desirable to explore the data in terms of spatial and temporal trends rather than numerical values of individual neurons. Stimuli induced patterns of neural activity must be isolated from background activity (Zelenin et al., 2014). Heatmaps could be suitable for this purpose since complex spatial relationships and patterns in multivariate data are rel- atively simple to identify visually but difficult to detect computationally, and particularly in the case of interactive visualizations (Rheingans, 1992). Heatmaps are a popular visual structure for encoding quantitative intensity values spatially as color (Duchowski, Price, Meyer, & Orero, 2012). While originally developed to illustrate financial market information, the visual struc- ture has gained wide adoption in biotechnology and medicine (Akers, 2015). This might be because heatmaps produce similar visualizations to those re- sulting from well-established imaging techniques such as computed tomogra- phy(CT) scans, where color transfer functions can be used to di↵erentiate be- tween anatomical components such as bone, cartilage, muscle and blood vessels by means of mapping densities to di↵erent colors (for an example see Kindlmann et al., 2005). In regards of brain activity, heatmaps produce similar visualiza- tions to positron emission tomography(PET) which assesses neural activity indi- rectly by measuring blood flow or other metabolic processes in di↵erent regions of the brain (Handbook of Laboratory and Diagnostic Tests, 2013). Figure 1: (A) A 3D reconstruction of the brain and eyes from a CT scan, (B) A transaxial slice of the brain from a PET scan and (C) a heatmap visualization of neural frequency data. Sources: (A) By Dale Mahalko - Own work, CC BY-SA 3.0. (B) By Jens Maus - Own work, Public Domain. (C) By Pavel Zelenin - Own work, all rights reserved. Although static visualizations of neural activity data as heatmaps have previ- 1 ously been employed by Zelenin et al. (2014), an exploratory search of current literature in neuroscience failed to reveal any similar attempts by other re- searchers. This visualization is denominated as static because Zelenin et al. (2014) has created individual images rather than an animation or an interac- tive program. While animated visualizations of neural activity were found in current literature, these come from imaging living specimens and do not rep- resent frequency(e.g. Fetcho & O’Malley, 1995; Muto, Ohkura, Abe, Nakai, & Kawakami, 2013). The only interactive visualization of neuron frequency data that was found in literature was an staple diagram that was not animated, did not include spatial information and only presented deviation from mean frequency by individual neurons (Carlis & Konstan, 1998). This project aims to develop spatiotemporal visualization of neural frequency data in the form an interactive animated heatmap. The purpose of this visual- ization is to be used as a tool in data analysis, assisting in pattern recognition and hypothesis formulation. 1.1 Problem statement Complex spatial relationships and patterns in multivariate data are difficult to detect computationally but relatively simple to identify visually, so the goal of this project will be to visually represent multivariate neural frequency data as an interactive animated heatmap. 1.2 Scope and limitations The method of visualization will be theorized first and then implemented as an application. The implementation will be assessed in terms of time and memory complexity and image quality. Furthermore, optimizations and heuristics will be attempted to improve their performance. This project will not analyze or investigate the characteristics of activity dis- tribution between neurons. Although the data originates in a study in neuro- science, this project does not intend to conduct any further research in that field. The dataset’s significance to this project is limited to the numerical val- ues recorded from the experiments. What is of interest in this case is not the interpretation of the values nor their medical and biological connotations, but the structure of the data from a computer science perspective. This project will also not analyze how to best visually represent or highlight distribution trends across neurons nor how the visualization impacts data analysis. Instead, this report will discuss how the data is processed in order to visualize it as a heatmap. 2 1.3 Disposition This report will document the findings from the development of the Anivis program. Section 2 Background will provide underlying the theory in both neuroscience and data visualization on which Anivis is built upon. It will also present the neurological studies behind the neural frequency data. Section 3 Method will describe in detail the structure of the neural data as well as the theoretical reasoning behind the implementation