Art and Science: the Full Experience

Home , Sonification

York 1

Art and Science: The Full Experience

Why Data Visualization and Data Sonification Are Equally Important

______

A Senior Thesis

Presented to

The Faculty of the Department of Music Business, Entrepreneurship and Technology

The University of the Arts

______

In partial fulfillment

of the requirements for the degree of

BACHELOR OF SCIENCE

______

Mandii York

May 2020

York 2

Data organization is and has always been a growing art form. David McCandless describes it as “a fertile and creative medium” (00:05:30). As self-aware creatures we are always searching for information about how the world works. Once we find this information we figure out the best ways to dissect and digest what we discover. We do our best to draw connections to strengthen our understanding. Daniel Kahnman writes “We are pattern seekers, believers of a coherent world, in which regularities appear not by accident but as a result of mechanical causality or of someone’s intent” (115). Kahnman is saying that we don’t accept the idea of random occurrences. We believe that everything is connected. Data is composed of “numbers and disconnected facts, but if you start working with it and playing with it in a certain way, interesting things can appear and different patterns can be revealed” (McCandless 006:00 –

006:04).

There are many ways we communicate the connections we discover. Data visualization is the most common way we convey data. It is so popular because it allows us to easily notice trends at a glance. Take for example graphs and eye catching infographics. Another way to communicate this data is through sonification. Sonification is the art of transforming data into sound. Both data visualization and sonification have their pros and cons and a type of data they can best represent.

Rich Haridy writes “the line between straightforward information conveyance and art is frequently blurred by many artists who create beautiful and compelling forms of data visualization.”. However, this line is also blurred by artists who sonify data. Sonification is just York 3 as valuable and creative as data visualization. A key factor in understanding something is perspective. By mixing data science with art we gain new perspectives. In this essay I will compare and contrast the two, and explain how they are equal in importance and as art forms.

Data visualization has evolved significantly from some of its earliest forms in the 1600s.

These forms include basic cartography and the development of coordinate systems, the introduction of a principle now knows as “small multiples” – which can be defined as small, thumbnail-sized representations of multiple images displayed all at once --, “the systematic collection and study of social data began in various European countries, under the rubric of

“political arithmetic”’ (Friendly 4 -6).

The 19th century is when we begin to see more thematic mapping and statistical graphs expand to different forms. “In thematic cartography, mapping progressed from single maps to comprehensive atlases” (Friendly 9). Statistical graphs grew to include pie charts, histograms, line graphs, scatter plots and more.

Jumping ahead to today, data visualization has grown from being flat maps and basic charts. In an article for New Atlas, Haridy highlighted Fernanda Viégas and Martin Wattenberg as pioneers of data visualization art. One of Viégas and Wattenberg earlier pieces titled Flicker and Flow displays how the colors of the Boston Common change over a year. This piece takes the data and creates an aesthetically pleasing abstract design. Flicker and Flow still displays data in an easily digestible way that makes data visualization well loved. Many things make data York 4 visualization great. Its age, its accessibility, and its relationship with early forms of communication are its strongest attributes.

As explained earlier, data visualization has been around for a very long time. This gives data visualization a leg up from data sonification because there is far more research on it and the fear the comes from introducing new ideas is long gone. Data visualization has also been practiced for centuries making it an average aspect of our society today. We have much more trust in this art form since it has grown and evolved with our culture.

Besides being around for much longer than sonification, data visualization can be more accessible. Our brains aren’t wired to process lines of data quickly. Relating the numbers with images allows us to notice patterns quicker and get a fuller picture of what is being displayed.

This is what makes data visualization favored.

With all of this said, data visualization is not always the answer. Although it is helpful in many ways, data visualization can be a lot to take in for some. Sometimes the information can be seen, but not thoroughly understood. There are many nuances that can be missed in this form of art. As a result, key messages of the data set can be overlooked causing confusion to the objective.

This is when we look to sonification. The origin of sonified data is blurry. There are some who say that it was formalized in 1992 at the International Community for Auditory Display.

However, there are studies from the 1950s that experiment with data and sound but a term was York 5 not coined yet. Andi Schoon and Florian Dombois discuss that “"data music" of the early 1950s was generated by applying sound-synthesis methods. Given its stringently structured form, it can be regarded as a model for the application of sonification to the realm of scientific representation.” They go on to explain, only a decade later in the 1960s, “in an aim to heighten audience awareness of their acoustical environment, artists have devoted themselves to representing data and their phenomena acoustically.”

Shawn Graham describes sonification as “the practice of mapping aspects of the data to produce sound signal”. Now this may sound like something we aren’t familiar with, however there are many forms of data sonification that we hear every day. One example is the ticking of the clock. The ticks and the chimes we hear give “an auditory cue to the listener which represents a piece of data.” (Strickland 00:01:09).

So why should we look into data sonification? How can sonification contribute to the future of understanding data sets? As humans we learn different things from reading something versus listening. When compared to data visualization, data sonification can be believed to have more dimension. I believe that both arts forms have the same amount of dimension. The only difference between the two is that they utilize that dimensions of different senses.

Kelly Kasulis writes “data sonification has the advantage of being an experience”. There are many aspects of sound that can directly correlate with data. These aspects include pitch, timbre, and tempo, but are certainly not limited to those three. Sound art is not an uncharted territory, however its limits have yet to be fully explored. York 6

What we can do with sound art and data sonification currently is quite astonishing. We can use data sonification to analyze DNA sequences to help point out mutations. Mark Temple explains that given the linear nature of DNA, it responds favorably to being sonified. He acknowledges that data sonification is not accepted with open arms and is “typically considered to be an interesting oddity that contributes little analytical benefit.” During his research and experimentation, Temple goes as far as arguing for data sonification “inclusion within the toolkit of DNA sequence browsers as an adjunct to existing visual and analytical tools.”

Shawn Graham covers an experiment conducted by Mark Last and Anna Usyskin where their objective was to “determine what kinds of data-analytic tasks could be performed when the data were sonified.” In their experiments they discovered that data sonification works extremely well with time-series data. Graham writes that this is true due to the “natural parallels with musical sound”.

In a recent article written by Vineeth Venugopal for Science Magazine, Venugopal discusses how data sonification is aiding in the research of Covid-19. Scientists at the

Massachusetts Institute of Technology sonified the data of the protein spikes in the coronavirus and created a 2-hour analytical composition. In technical terms Markus J. Buehler explains:

What you hear is a multi-layered algorithmic composition featuring both the vibrational

spectrum of the entire protein (expressed in sound and rhythmic elements), the sequence

and folding of amino acids that compose the virus spike structure, as well as interwoven York 7

melodies - forming counterpoint music - reflecting the complex hierarchical intersecting

geometry of the protein. (Buehler)

Venugopal goes on to say that researchers find this method more intuitive than conventional methods used to study proteins, such as molecular modeling.

Understanding that data sonification is a fairly new method, when it comes to accurately representing data sets, there are some downsides. A big complaint with data sonification is that it is displayed to the public inaccurately. In July of 2018, NASA published “Sounds of Saturn” which is a piece that represents the light emissions between Saturn and its moons. Many have condemned this piece to being fraud and confusing. It can be argued that the liberal use of audio effects can skew the truth behind the data. The title of this piece was also misleading because it leads the public to believe that this is the true sound of that planet and not a representation of the data collected.

Within their pros and cons, data visualization and data sonification have many similarities. Both art forms are appetizing and can be easily digestible. This is especially true when comparing both to pure data information. Both take something hard to understand, break it down to its truth, then communicates that truth in a way we can quickly comprehend.

Individually, data visualization and data sonification work well. However, when we use both techniques on one data set, we get one step closer to understanding the full picture. An artist who has successfully demonstrated that these art forms can work side by side and give new perspective to the same information is Nathalie Miebach. Miebach works towards bridging the York 8 gap between science and art. Thus giving us beautiful works of art that double as analytical science.

Miebach is an artist who creates intricate sculptures based on weather data. In 2009,

Miebach started a project titled The Weather Score Project. She describes this project as “a series of musical scores entirely based on weather data, which are adapted by composers to musical performance.” (Miebach). Her sculptures are created using basket weaving techniques. One of my personal favorite collections is titled The Sandy Rides. In this collection of sculptural pieces

Meibach was inspired by images of destroyed amusement park rides along the New York and

New Jersey shore after Hurricane Sandy. She uses bright vivid colors and contours to represent these images and to represent the numerical ocean data from weather stations and ocean buoys.

In Miebach’s piece titled Harvey Twitter SOS, the sculpture and the musical score are

“built of weather and twitter data sent during Hurricane Harvey.” (Miebach) While the sculptural piece displays the surface level data, this musical composition adds emotion and other nuances that were previously missed.

I believe the future for data art is a combination of data visualization and data sonification, much like the work of Nathalie Miebach. Whether they be used for scientific research or simply self-expression, when recognizing that these art mediums stand side by side in usefulness, the possibilities to create are endless.

York 9

As stated before, data organization is a growing art form and there is so much for us to learn about it and from it. There are multiple ways to learn and understand data. By limiting ourselves to either data visualization or data sonification, we limit what we can understand.

When we combine science and art we produce a new perspective that allows us to understand data in a way we never thought of before. I cannot stress enough how important it is to look at data from as many perspectives as possible to gain the truth of the data and understand what we can do with it. When we combine sight and sound, data visualization and data sonification we create an experience. I believe we can all learn and understand more through experience than plain science.

York 10

Bibliography

Buehler, Markus J. “Viral Counterpoint of the Coronavirus Spike Protein (2019-NCoV).” SoundCloud, Apr. 2020, soundcloud.com/user-275864738/viral-counterpoint-of-the- coronavirus-spike-protein-2019-ncov.

Graham, Shawn. “The Sound of Data (a Gentle Introduction to Sonification for Historians).” Programming Historian, 7 June 2016, programminghistorian.org/en/lessons/sonification#Last.

Hoffman, Jason. “Pros and Cons of Data Visualization Explained.” WisdomPlexus, WisdomPlexus, wisdomplexus.com/blogs/pros-cons-data-visualization/#messages.

Kahneman, Daniel. Thinking, Fast and Slow. Farrar, Straus and Giroux, 2011. EBSCOhost, search.ebscohost.com/login.aspx?direct=true&db=cat04417a&AN=art.b1216213&site=e ds-live&scope=site.

Kasulis, Kelly. “Data Sonification Lets You Literally Hear Income Inequality.” Mic, Mic, 23 May 2017, www.mic.com/articles/177877/data-sonification-lets-you-literally-hear- income-inequality.

Lovelace, Joyce. “Composing Chaos.” American Craft, vol. 73, no. 6, Dec. 2013, pp. 72–81. EBSCOhost, search.ebscohost.com/login.aspx?direct=true&db=asu&AN=91804737&site=eds- live&scope=site.

Miebach, Nathalie. “Art Made of Storms.” TED, TED, 2011, www.ted.com/talks/nathalie_miebach_art_made_of_storms?referrer=playlist- art_from_data&language=en.

“Nathalie Miebach: Sculpture.” Nathalie Miebach: Sculpture, nathaliemiebach.com/.

Pamela G. Taylor. “Artistic Data Visualization and Assessment in Art Education.” Visual Arts Research, vol. 43, no. 1, 2017, pp. 59–75. JSTOR, www.jstor.org/stable/10.5406/visuartsrese.43.1.0059. Accessed 3 Feb. 2020.

Schoon, Andi, and Florian Dombois. “SONIFICATION IN MUSIC.” SONIFICATION IN MUSIC, 2009, smartech.gatech.edu/bitstream/handle/1853/51415/SchoonDombois2009.pdf?sequence=1 &isAllowed=y.

York 11

Shah, Raivat. “Turning Data into Sound.” Medium, Towards Data Science, 17 Apr. 2019, towardsdatascience.com/turning-data-into-sound-4854a35b3504.

“Sounds Good: Argonne Researcher Represents Data with Music.” New Atlas, 2 May 2015, newatlas.com/argonne-microbial-music/24497/.

Strickland, Jesse, director. Sonification: The Music of Data. Sonification: The Music of Data, YouTube, 9 Apr. 2020, www.youtube.com/watch?v=br_8wXKgtkg.

Temple, Mark D. “An Auditory Display Tool for DNA Sequence Analysis.” BMC Bioinformatics, BioMed Central, 1 Jan. 1970, bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-017-1632-x.

Tu, Chau. “How to Listen to Data.” Science Friday, 7 Feb. 2017, www.sciencefriday.com/articles/how-to-listen-to-data/.

Urist, Jacoba. “How Data Became a New Medium for Artists.” The Atlantic, Atlantic Media Company, 15 May 2015, www.theatlantic.com/entertainment/archive/2015/05/the-rise- of-the-data-artist/392399/.

Venugopal, Vineeth. “Scientists Have Turned the Structure of the Coronavirus into Music.” Science, American Association for the Advancement of Science, 3 Apr. 2020, www.sciencemag.org/news/2020/04/scientists-have-turned-structure-coronavirus-music#.

Yafit Gabay, and Lori L Holt. “Short-Term Adaptation to Sound Statistics Is Unimpaired in Developmental Dyslexia.” PLoS ONE, no. 6, 2018, p. e0198146. EBSCOhost, doi:10.1371/journal.pone.0198146.

Lovelace, Joyce. “Composing Chaos.” American Craft, vol. 73, no. 6, Dec. 2013, pp. 72–81. EBSCOhost, search.ebscohost.com/login.aspx?direct=true&db=asu&AN=91804737&site=eds- live&scope=site.