Big Image Data within the Big Picture of Art History
Harald Klinke
Abstract: The use of the computer in Art History is changing the approach towards our objects of research. Now, we are able to compute more images than a human can see in a lifetime. That, in turn, calls for a new definition of the role of the researcher and the tools being used. The access to large amounts of visual data stands in a tradition of conventional methods of Art History, but also augments them with quantity. This article proposes a theoretical model on which to build an understanding of the meta image with which we interactively derive our conclusions.
Keywords: Big Data, Digital Humanities, Distant Viewing, Information, Media, Meta Image, Methodology, Qualitative/quantiative Research, Semantic Gap
he digitalization of Art History is meaning layer by layer. The goal here Tcreating enormous opportunities. is to embed the individual object into Among the most striking is that we its cultural and historic context. Art now have the opportunity to not only History is, thus, an empirical science look at a single work of art—or the that gathers and interprets visual data. comparison of two—but that we are This is one reason why Art History is, also able to compare more pictures in German-speaking countries, also than a human can look at in a lifetime. called “Bildwissenschaft,” the science This concept is called Big Image Data of pictures. It should be noted that (BID). Panofsky knew that starting with the Taking a look at our discipline’s visual phenomenon does not allow the methodology, the first systematic - ap exclusion of any written records. It is proach that comes to mind is Erwin only natural that scholars in front of Panofsky’s iconography and iconolo- a work of art will acquire knowledge gy.1 According to him, and in contrast from books and compare their findings to all other disciplines in the Human- with those of other scholars, and then ities, art historians take the visual re- add to this shared knowledge through cord seriously and derive knowledge future publications. from it by reconstructing an artwork’s
Figure 1: About 60.000 images taken from the Artemis database (LMU Munich) sorted by average brightness using ImagePlot developed by the Software Studies Initiative Big Image Data
Another art-historical method This capability is exactly what the comes to mind: the side-by-side com- Digital Humanities promise. As the parison of pictures as established by Natural Sciences have used comput- Heinrich Wölfflin.2 What has been ers to yield unprecedented insights done with originals and painted or that were not possible before, such as printed copies before the advent of the sequencing and analysis of the hu- slide projections turned into a central man genome, the Humanities are able method in our discipline through the to make the computer a tool for their use of double slide projectors. Com- research, too. Ever since Roberto Busa parisons make clear the differences as created a lemmatization of the works well as the similarities of the studied of Thomas Aquinas using IBM - com objects. The findings can then be dis- puters in 1951,4 text-based Humanities cussed and converted into insights, have developed many software tools to such as the question of the object being obtain an overview of text corpora em- an original or a copy, questions of dat- ploying statistical methods, to uncover ing (Datierung), or authorship attribu- semantic structures via text mining, tion (Handscheidung).3 A slide library and to develop new insights into lan- containing hundreds of thousands of guage using Corpus Linguistics. The high resolution color images is a pow- visual sciences, the foremost being Art erful means of navigation through the History, started to integrate computa- history of art. tional methods in the early 1980s,5 but progressed slowly for several reasons.6 Both methods show that no other The most important reason is that im- discipline looks at pictures as system- ages are much harder for a computer to atically as Art History. They also show process. This might seem to be an odd that individual scholars and their visu- statement, especially since digital im- al experiences acquired over the years ages consist of nothing beyond a string are at the center of this kind of re- of information units called bits like all search. In both cases, researchers need other digital media. But this matrix of a mental archive of images in order to pixels does not “mean” anything to the identify figures by their attributes and computer at first. to select objects for comparison. Would it not be great to go a step further and While one string of ASCII charac- use computers to be able to consult a ters7 in a text can be compared to an- much greater variety of artworks, to other (search), they can be summed up use algorithms to recommend further in a corpus (frequency distribution) or works that might be of interest to the become the basis of statistical calcu- researcher? Would it not be great to lations (text mining). A string of pix- let the computer do the work of sifting els on a display is nothing more than through a great number of artworks in a sequence of brightness values on order to deliver meta information on the visible light spectrum (red, green an enormous corpus of images? and blue) at first. However, we are
16 DAH-Journal Big Image Data
able to calculate and use these color of images at once and deriving meta in- values as the criterion to sort images8 formation from that corpus of images.12 in a process involving these “low lev- Dealing with a massive amount of data el features” of images (see figure 1). is a tremendous challenge for Art His- This process, however, is limited in its tory. The good news is that there is no epistemic potential.9 In contrast, “high need to start from scratch. Other disci- level features” are what those pixels plines have dealt with images before: represent; the content of the images. medical applications deal with images, Between both of these lies the “se- such as high quality x-rays. Biology mantic gap,” where high level features has developed software to handle pic- are perceived by humans with ease, tures produced by microscopy in order whereas the computer still struggles. to quickly discover clusters or count Information Science—Computer Vi- cells. Astronomy, Geology, and other sion in particular—is working on this similar disciplines incorporated such problem via pattern recognition, deep tools into their daily practices long learning, and other approaches, but ago. In particular, one field of Informa- there is still much research to do.10 As tion Science has advanced quickly of research is carried out, it will increas- late: Big Data. ingly provide the means to work with images that Art History can use for its Big Data has been a buzzword for own purposes. a few years now. In 2012, it was iden- tified as the biggest trend by Bitkom— At this point, we have to distin- the German IT Trade Association—and guish two concepts. Reading a book of in 2014, Big Data was the theme of the literature and writing about it has been CeBit trade fair. The reason for this called “close reading.” Having all works recognition is that it promises unprec- of an author in a digital format and edented insights into social behavior comparing them to those of another and new possibilities for business mod- author by means of statistical analysis els. At the same time, it stokes fears of is an example of “distant reading.”11 It surveillance and espionage. Big Data offers the possibility to use computers seems to be defined as the collecting to work with data in order to analyze a and analyzing of data, with the ulti- greater number of books in a matter of mate result and goal to make money; seconds than an individual could hope Facebook, Apple, and Google come to to read in a lifetime. These concepts can mind when discussing Big Data. With also be applied as standard methods data, companies not only know what of study for Art History, where “close we do, but also things that we might viewing” describes the study of indi- do.13 Some say that data is the new raw vidual reproductions of artwork. In the material.14 And the massive amount of same vein, “distant viewing” describes data is not the problem, but the solu- what algorithms are increasingly able tion. to offer: examining an infinite number
DAH-Journal 17 Big Image Data
Big Data, first of all, is an Informa- a quantity unprecedented in human tion Science concept. Only in combi- history (and that is truer than ever nation with business does it become a today, 22 years later). Since we have concept for profit maximization. It has sciences for language, there must also the potential, in combination with sci- be a science for images (Bildwissen- ence, for “knowledge maximization.” schaft).15 So again, what is an image?16 Moreover, it can become a fruitful An initial approach to answering this tool in the Digital Humanities. While question could be to name the objects “data” is generally used here to denote that we include under the term “image.” values, such as sensor results, the chal- The history of art knows many such lenge for disciplines like Art History objects, and paintings are only one of that use images is figuring out how to them. Photography has arguably been gain knowledge from a mass amount the most important revolution for im- of visual data. How can we understand ages. One can incorporate not only ar- the hidden connections between imag- tistic productions, but also the broad es on a macro scale? How do we deal field of amateur photography. In addi- with what could be called Big Image tion, technical images have to be add- Data? ed because they are mainly used in the natural sciences. Effigies, wax figures, and other such items are also undoubt- What is a Digital edly understood as images.17 Finally, so-called internal or mental images are Image? also a part of the Visual Studies, even if they are of a fundamentally different arge amounts of images can be a nature than the aforementioned exter- treasure trove. An art-historical L nal images. As a matter of fact, in the image database can represent a rele- English language we sometime prefer vant section of art history for specific 18 research questions. But before we are to call external images “pictures.” The able to unearth such a treasure, we first two are closely tied together, as pic- have to understand what kind of data tures cannot be sufficiently explained digital images are. If we want the com- without images. puter to help us generate art-historical knowledge, we have to go a bit further In addition, the term “picture” re- and ask what an image is, generally fers to another important concept. The speaking. conventional panel painting has, for centuries, dominated European art his- “What is an image?” was the central tory. Following its example, art theo- question posed in Gottfried Boehm’s ries have been developing since the Re- 1994 publication that spurred Visu- naissance. But only with the invention al Studies. His thoughts were based of photography—and with approaches on the assumption that we are liv- to make them theoretically compre- ing in a time when we use images in hensible—a new concept has become
18 DAH-Journal Big Image Data
necessary: the concept of the medi- the new image phenomenon tangible; um. The term “medium” here denotes make us able to conceive a history of the carrier of an image that makes visual media and media theory.21 the visual phenomenon possible by its physical configuration.19 The term Today, we are living in a time where is necessary in photography to make revolutionary changes in the hege- clear the relationship between origi- mony of visual media are being made nals and copies. A photograph allows again. A new visual medium has joined for the production of a series of posi- its peers and, similar to photography, tives from one negative. Unlike repro- challenges other media in their artistic duction graphics, the first and the hun- standards and continues to drive the dredth copy do not differ appreciably. democratization of image production This has been made possible by using and lets the flood of images skyrocket. a specific medium—chemical photog- This new visual medium is the digital raphy, in this case. The distinction be- image. tween image and medium makes clear what is new about photography and its The advent of the digital image— demarcation from painting. A picture just think of the spread of digital pho- became not only something that could tography—challenges the older media represent something else (a distinction at least as radically as photography between the picture and what is pic- did in the 19th century. It represents tured, see figure 2), but it could also a new challenge to conceptual tangi- appear with different characteristics in bility, just as photography did. At the its production, distribution, and recep- same time, it is itself the driving force tion. It is noteworthy that after nearly to develop new ideas about what imag- 100 years of image-making practices es are. Just as the photograph required with photography, these crucial con- the concept of media, the digital image cepts were discussed for the first time requires new terminology in order to in 1936 by Walter Benjamin.20 This make its unique ontology understood, opened up the opportunity to make as well. The difficulty in making digital
Image Medium
Object External Image Recipient Mental Image
Figure 2: Scheme of image and medium
DAH-Journal 19 Big Image Data
images understandable comes from the one time, the chemical fixation of the fact that the concepts developed for the volatile image of the camera obscura. older image media must now be chal- Today is quite similar, where digital lenged: What is the image of a digital image captures can be displayed on a image? What is the medium of a digital pixel matrix (see figure 3). image? The distinction of image, what is being pictured, and the medium no Another example of a digital im- longer seems sufficient. Therefore, the age is when you work in a graphics digital image can only be understood program, such as Adobe Photoshop, when new terms are introduced. on a computer. The image is visible on the display and can be changed A few specific examples will make using a graphical user interface that it easier to grasp what has been in- is controlled by mouse and keyboard troduced by the digital image. The peripherals. The possible changes to simplest example of a digital image is brightness and contrast, to collage or probably the digital photograph. It is “paint,” have replaced the convention- noteworthy that even especially high- al processes used in a darkroom.22 The priced digital cameras today imitate “retouched” image can subsequently the mostly conventional Single Lens be sent or shared via email or social Reflex camera (SLR) in design. The film networks and made available to other has been replaced by an image sensor, viewers. such as a CCD chip. The advantage of this technique is the immediate con- A third common form of the digital sideration of the result on a screen, image is—as opposed to digitized—the so much so that in compact cameras, computer generated image (CGI) as the optical viewfinder has largely been used in computer games (first-person phased out in favor of a visual display. shooters, for example), movies, or de- The revolution of photography was, at sign software. The on-screen visible
Media System
A/D CPU CPU 0101
Infor- Algo- Display mation rithm
Figure 3: Basics of digital photography
20 DAH-Journal Big Image Data
image is now generated based on data medium that has a reference to some- and algorithms devised through com- thing), the question arises: what is the putational steps. In addition, this al- medium of the digital image? It is cer- lows for manipulation of the image by tainly not the display alone. In digital the recipient (now called the “user”) in cameras, the memory chip and the files real time. it contains are important elements. When one thinks of Internet applica- These are all undoubtedly examples tions, it is clear that the display unit of images, and there is something they and the file do not even need to be in all share: the visible and the object be- one place. The file is located on a -re ing pictured. Also, digital images rep- mote server and can be fetched via the resent something visible by means of network at any time. the image carrier (the medium). Yet, it is clear that we are dealing with a Strictly speaking, the CCD chip special picture system here whose converts the local brightness values unique ontology becomes clear when of RGB colors via an analog/digital examining the differences with -old converter into numbers, which are er picture systems. The “image” on then compressed using a processor a camera’s memory chip is itself not and written to a file (such as the JPEG visible, comparable perhaps to an ex- format) where they can ultimately be posed roll of film in a sealed opaque presented as on/off states. In reverse, cartridge, carrying the latent images the data is read, decompressed, and only. And yet the “latency” of digital outputted as the brightness values of images on the chip is different because the three primary colors on a display. it is made visible through a different Any alterations via imaging software process than chemical photography. takes place only indirectly on the dis- In image processing, it is clear that the play (the graphical user interface is the image can be changed with more ease interface between users and the data; than performing the same changes in between man and machine).24 In fact, a darkroom. It can be copied without the database is changed by a compu- loss and can become visible anywhere tational process. For example, bright- in the world almost simultaneously. ening an image corresponds to the ad- Finally, computer generated imagery dition of a value to each RGB value of does not need to be in direct propor- the pixel matrix, darkening an image tion to the visible reality; rather, it can is a subtraction, and contrast enhance- be created and modified in real time. ment is actually a stretching of the his- This can be done with such quality that togram. The image is finally generated the impression of immersion may oc- by calculating the numerical values of cur, which was not previously possible the individual pixels. In other words, with any other visual medium.23 the computer knows no image, only If one examines the “image” in the strict numbers. It does not “see”—it only cal- 25 Figure 3: Basics of digital photography sense (what is visible with the help of a culates.
DAH-Journal 21 Big Image Data
The system, as described, makes it memory, input unit, and output unit clear that we have to add a new con- components.26 Every digital camera, cept to image and medium: the concept smartphone, and laptop consists of of information. The science of digital memory, a processor, and a display. images must be understood as informa- The seemingly placeless image, e.g. tion theory of images. The individual a picture posted on Facebook, can be bits of memory contain the information retrieved from different places, upload- that must be interpreted by applica- ed, and downloaded. Such images basi- tions to provide meaningful messages. cally consist of a string of bits in a file A string of bits, like 01000001, can be format, such as JPEG, it is located on interpreted as a decimal (65) or—ac- an Internet server and is transferred on cording to the ASCII table—the letter demand to the local machine. The file “A” or a brightness value of a pixel. In is interpreted there by using programs this way, not only numbers and letters, and made visible by the graphics and but entire pictures, sounds, and myriad display hardware as an “image.” This pieces of data can be coded in binary, entire system can be called the medi- processed, and transported as a signal. um, and therefore its task is to visual- ize the string of bits. But that is not the image. It is the information that is later turned into If we have thus defined and distin- an image through hardware and algo- guished information and media, what rithms; in other words, by an overar- then is the “image”? In summation, the ching system that may be referred to distribution of brightness of the red, as a medium. This process is called “vi- green, and blue values on a screen’s sualizing.” The information can be dis- pixel array results in an impression of played using a computer screen, two contiguous areas, which are detected screens, be presented as printout, etc. as objects based on one’s experience These are only different forms of visu- with the real world. In other words, an alization of the same information. “image” of something. Consequently, the “image” is the visual phenomenon The question “where is the image as perceived by the viewer.27 More spe- in a digital medium?” can now be an- cifically, it is the light projected on the swered: it is not in the memory storage retina, which is then converted into device. The information stored by the nerve signals and—preprocessed by memory device is required to repre- the metathalamus— perceived in the sent something through the use of the visual cortex.28 In short: what is visi- medium. The medium—the system that ble is the visual phenomenon. In order makes the information visible—can to avoid obfuscating the term “image,” be labeled as a computer, which is de- the following descriptors should be signed according to the von-Neumann used: information, media, and visual principles. A von-Neumann-computer phenomenon. consists of controller, arithmetic unit,
22 DAH-Journal Big Image Data
These terms enable a clear descrip- of pigments with a binder (e.g. oil) and tion of the digital images. A digital applied to a substrate (wood or can- camera generates information that can vas). Because of this material config- be visualized on a display or photo uration, a visual phenomenon can be printer. The image processing program experienced, which creates the impres- changes the information (via a graph- sion to refer to objects outside of the ical user interface), which is then sent picture itself. and displayed on devices. To “send a picture” is colloquial and actually Accordingly, in photography, there means to transmit information signals was no need to question the informa- that are interpreted and displayed on tion because the image information the device as a visual phenomenon was irrevocably connected with the recognizable by humans. Computer image carrier. Image data processing games generate sequences of bits us- systems introduced new complexi- ing data and algorithms. These bits are ties to what the information entails. visualized using the medium and are It called for precise descriptions and perceived as visual phenomenon. a more comprehensive understanding of this new imaging system. This con- These terms can be applied to more cept of information can also be traced than digital images. With convention- back historically. As stated above, the al panel paintings, there was no need information is hidden behind the visu- to question the medium because the al phenomenon and defines the bright- visible phenomenon was inseparable ness and color distribution for the out- from the physical medium.29 Image re- put device. Viewed as a whole, these producibility extended the medium’s pieces of information coalesce to form properties and allowed for the new a visual impression—specifically, the concept of the new visual medium to brightness values of RGB subpixels. form. The term “medium” can then be The values associated with each cell of traced historically back to the paint- the matrix are not permanently fixed, ings: the medium “painting” consists because the brightness on a display can
Figure 4: Pigments of a Painting, Silver Nitrate of a Daguerreotype, Pixels of an LCD Display
Source: Micrograph. Light micrograph. Macro shot of an LCD display
DAH-Journal 23 Big Image Data
shift at any point and is controlled by In computer science, this is called the display’s underlying operating sys- a layer model. Among the best-known tem. In a painting, these pieces of color layer models is the OSI model, which information are also present. However, refers to seven perspectives on techni- their values are determined by the art- cal structures in telecommunications. ist using a brush in the form of bound In it, the lowest layer of the physical pigments and are no longer change- layer—copper cable or electromagnetic able after the drying process. The in- waves—transmits the bits in the form formation is thus firmly connected to of signals. The higher layers assure the the image carrier. The situation is sim- correct transmission between sender ilar with photography: silver nitrate is and receiver through a series of trans- changed via light exposure by different mission protocols.30 A similar layered degrees. Each silver nitrate molecule at model can be applied to visual media. any point is a carrier of brightness in- formation that will convey the overall It should be noted that the first lay- visual impression (see figure 4). Again, er consists not only of bits, but also of the information is fixed to the medium. data formats including compression. Only in the digital imaging medium The second layer consists of the- ele are these elements separated and act ments of von-Neumann computers, as properties of the new medium that including memory, on which informa- has added new features and options to tion is stored, processed, and outputted it. The separation of “information” and (e.g. displayed). The third layer consists “media” makes variability, processing of the observable phenomenon that is ability, and lossless copying possible— purely optical. these basic characteristics distinguish the digital image from the convention- On this foundation a system of hu- al image. man perception can be built: an image is only an image when it is viewed by The introduction of the concept of a recipient. Higher layers describe the “information” in image science has steps of assigning meaning. For this, a further advantages: the above triad of proven model of additional layers that information, media, and visual phe- we can make use of already exists: Er- nomenon stands in a hierarchical rela- win Panofsky describes a three-stage tionship with one another. The media general epistemology of visual percep- system requires information to create tion when discussing iconology and a visual phenomenon. This can be rep- iconography. Accordingly, first the ob- resented as follows: jects are named (in his example serving Christian iconography: man, knife), 3. Visual Phenomenon and then the overall image is identified (Saint Bartholomew) and finally placed 2. Media System in its cultural and historical context.31 1. Information
24 DAH-Journal Big Image Data
What Panofsky fails to describe This model describes the picture is that before the cognitive process— medium and its reception by the view- which relies on cultural experience— er in full and is applicable not only to can begin, the sensation must first be the digital image, but any image phe- pre-processed to distinguish shades nomena. It is the relationship between and surfaces in order to identify areas the visual phenomenon caused by a as connected objects. This requisite physical configuration and the view- edge detection, contrast enhancement, er that constitutes what is called an and reduction of information already “image.” It is the human being who happens in part inside layers of the perceives images and fills them with retina and the metathalamus.32 This meaning, and it is the human being pre-processing is necessary to distin- who creates images in order to visual- guish a human from an object based ly communicate ideas. With the digital on the amount of stimuli in a visu- image, it becomes clear: to talk about al phenomenon and can be added to images, one must connect four terms Panofsky’s description as layer zero. In into a relationship: recipient, visual this way, we obtain the following layer phenomenon, media system, and infor- model: mation. In short, an “image” consists of multiple interconnected parts. 3. Iconological Interpretation 2. Iconographic Analysis What is Big Data? 1. Pre-iconographic Description 0. Object Recognition aving defined the term “image,” Hthe question remains: what is “Big Data” and how can we use it, Since the structured image de- especially Big Image Data (BID), for scription as shown here is only com- science? Big Data simply sounds plete with a description of the image like a lot of data. The image database medium, these two layer models can Prometheus (University of Cologne) stacked upon each other: contains 1.5 million images,33 Art Store contains 2 million,34 Google Cultural Institute contains almost 5 million,35 7. Iconological Interpretation and the Artigo project (University of 6. Iconographic Analysis Munich) has generated about 8 million 36 5. Pre-iconographic Description tags so far. A heap of data, however, does not yet represent knowledge. 4. Object Recognition We have to distinguish pure signs, 3. Visual Phenomenon such as bits or bytes that can be interpreted; data, or unstructured facts 2. Media System about certain occurrences in the real 1. Information world; information, or contextualized,
DAH-Journal 25 Big Image Data
relevant meaning; and knowledge, or understanding through experience Knowledge (see figure 5).37 The interpretation of results is still exclusive to humans, but Information computers can help us with the steps leading to that destination. Data Since Big Data promises to acquire new relevant knowledge, it refers to data on one hand, and the analysis of Signs that data on the other. For example, the warehouse chain Target employed a marketing strategy aimed at pregnant Figure 5: The “Pyramid of Knowledge” as 38 women as their target market and tried derived from Ackoff (1989) to address their needs earlier on in their pregnancy than competitors. By searched for. Its real-time analysis al- analyzing the patterns in a customer’s lows for a higher probability of the purchase history, a woman’s pregnan- prognosis. Thus, Big Data means to ac- cy and her child’s birth date could be quire, store, and analyze large amount predicted, sometimes even before the of data that are generated quickly and women knew, it seems.39 Such patterns are not always structured, as with SQL are derived from a combination of age, databases. Open source solutions like place of residence, and purchase his- the Hadoop Framework, or database tory. This analysis discipline is called systems such as Cassandra and Mon- “predictive analytics” and shows how goDB, were all developed to help ana- knowledge about the future can be lyze such data quickly. generated by using the past and pres- ent. An important point should be mentioned here: such predictions are not definitive, rather, they come with What is Big a probability. That means the result of such an analysis is not a statement of Image Data fact, but is instead a fuzzy projection ig Image Data, then, is about a spe- (often with a notably high probability). Bcial kind of data: visual data. Anal- ysis of it can lead—just like ordinary In order to generate knowledge Big Data—to statistical key figures, dia- from pure data, Big Data is character- grams, overviews, or graphs that show ized by three Vs: Volume, Variety, and structures such as clusters or outliers.41 Velocity.40 Volume allows for the access Image data visualization is essentially to a large set of data, and its hetero- images about images—meta images, geneity allows for the drawing from perhaps—since they make a digest of multiple ends in order to find the item many images understandable. How
26 DAH-Journal Big Image Data
that digest is calculated determines the in smartphones. The number of these epistemic content of the image. Simi- images is being generated and accu- lar to text mining, the process could be mulated like never before in history. called image mining, for which algo- Their analysis can generate enormous rithms are currently being developed. insights into societal developments It is a projection from many to one and individual habits. Most of all, visu- and, thus, an information reduction al data speaks to what we see, what we defined by mathematical calculations.42 show to others, and what we think the In short, they reflect other images. Or, world looks like. Art History is still far in the words of W.J.T. Mitchell, “Any away from such repositories, but still picture that is used to reflect on the na- has to answer how a Bildwissenschaft 43 ture of pictures is a metapicture.” makes use of image analysis technol- A few examples of volume in BID: ogy. So how can Art History acquire Apple’s iCloud solution has over 782 data big enough to derive meaningful million users (as of February 2016),44 knowledge about the history of art? and its photo sharing function allows How can we utilize the tools infor- users to upload the last 1,000 photos mation science is offering for our ob- from a device, such as the iPhone, and jectives? Digital Humanities promises sync them with other devices. That that the computer can help us reach means up to 782 billion images can our epistemic aims better. The - com be kept in Apple’s data centers, such puter can support our conventional as the one in Maiden, South Carolina. methods, and with its help, we are able Shortly after the introduction of this to develop new digital methods. What service, the 1,000 image limit was does this mean for Art History? lifted. Starting in 2013, virtually all images can be synchronized. Facebook Apple and Facebook are presumably has over 1 billion users and is said interested in understanding custom- to receive about 350 million photo er behavior in order to match adver- uploads per day.45 The sheer number of tisements with customer interests. To images is not the treasure here, but the achieve such an objective, the collected variety: GPS location integration, likes, data requires the proper structure, and friends, and other contextual data. the challenge comes in how to analyze The velocity is another treasure and the data in order to receive meaning- includes the real-time acquisition of ful insights. Art History is also inter- images, since they are uploaded almost ested in human behavior, but on the immediately after they are taken on a other hand, it is also interested in im- mobile device. age-making processes (as a homo pic- tor),46 in specific images like art works, Big Data content is often acquired and in networks existing between art- in the “Internet of Things” by means work and artist.47 This means that not of sensors. Visual data is acquired only are the number and structure of with visual sensors, such as cameras data relevant, but also their quality.
DAH-Journal 27 Big Image Data
Therefore, sometimes there is another with what we want to find out, and important fourth “V” added: veracity. then ask how computers can assist us Veracity refers to the accuracy of data in this task. As a third step, we should in relation to the real world. For Art ask what data we need, in what kind of History, this begs the question: how structure, what algorithms we should much can we trust a digital reproduc- use in order to analyze the data, and tion of an artwork? Fact is, a repro- what visualizations should be used duction always bears a difference to to show these results and explore the the original. But looking at dozens of data. variations of depictions of the Sistine Chapel in our database makes us won- Data analysis for Art History means der if we are able to comprehend the the introduction of quantitative meth- width of that gap. We need to address ods into a field that has used exclusive- the integrity of our data. Hence, we ly qualitative methods throughout its must first ascertain what data needs existence. However, these quantita- to be acquired to be used as the base tive methods are not a substitute for for subsequent analyses. If we build on conventional methods, but an addi- top of our digitized slide libraries, like tion. Results of data analysis can help our art-historical image databases, we to answer to our research questions, also need to address other quality con- but may also spark new hypotheses cerns apart from accurate color depic- that can be followed using a bouquet tion. The sufficient richness of visual of methods that Art History has. For data remains unanswered: paintings some, Big Data seems to be the end of are not flat and are therefore not ade- theory but one could also argue that quately represented in a JPEG-file, but Data Analytics helps us to see the big they have a relief structure. Paintings picture and create new hypotheses for would be more adequately represent- qualitative research.48 This way, the ed with an additional layer of grazing computer remains a tool in the hands light or x-rays and a 3D model. Such a of the researcher, or as Steve Jobs de- database for visual computing could be fined the role of the computer to the called Rich Image Data. human, “a bicycle for the mind.”49
Once the necessary kind, structure, Figure 6 details how the human and amount of data have been ac- system (i.e. the researcher) makes use quired, how can it all be used to gain of the computer (denoted as the me- knowledge on a higher level? First dia system) in order to access visual of all, art-historical research should data. Such meta images from BID are not be driven by the promises of in- observed by the viewer just like indi- formation technology, but by its very vidual images, except they are a di- own epistemic goals. Digital Art His- gested representation of the amount of tory should therefore not ask what to data, can be interactive, and reference do with all those images, rather start on visual data. The referenced visual
28 DAH-Journal Big Image Data
data relates to the individual physical cess them, visually examine them in or- art-historical object, which the art his- der to create a distant view, to explore torian likes to place in a cultural con- them, and to find them. The processing text. The goal of that epistemic process of low- and high-level features usu- is to form an abstract mental represen- ally takes place in the human retina, tation of developments in the history metathalamus, and visual cortex. With of art.50 In this process, the computer the ability of the computer to process acts as a computational co-processor these plain pieces of data into mean- to the researcher’s mind. To quote ingful information while generating Mitchell once again: “The metapicture meta images, these processes sink on is a piece of moveable cultural appara- the hierarchy of levels into the realm tus, one which may serve a marginal of the media system. Augmenting the role as illustrative device or a central layered model described above with role as a kind of summary image, […] the computer’s visual computation that encapsulates an entire episteme, abilities leads to the scheme shown in a theory of knowledge. […] In their Table 1. strongest forms, [metapictures] don’t merely serve as illustrations to theory; Thus, the computer becomes a they picture theory.”51 powerful tool in the hands of an im- age researcher because it can help us Today, the computer is able to make to access large amounts of visual data. low-level features computable and is At the same time, it leaves cultural increasingly able to compute high-lev- contextualization to the art-historical el features. That is particularly fruitful researcher, who can dive deeper into for Art History when applied to large the individual images. Art is made by amounts of images in order to pre-pro- humans for humans and the computer
Media system Human system
01010100 10101001 CPU
Artworks Visual Data Visualization Recipient Mental Image
CPU Interaction
Figure 6: Scheme of the computer as an epistemic tool in the service of image research
DAH-Journal 29 Big Image Data
can help us cut through the ever-ex- information, and transforms it to a dy- panding forest of mass visual data. namic visual phenomenon. This trans- Such digital methods do not replace formation process can apply different the human researcher, even if the com- algorithms and can be controlled by a puter takes over more and more tasks. number of variables. At the same time, It helps us to focus on those tasks that the user interface allows for the ma- we excel in: interpretation, contextual- nipulation of data via interaction. It ization, and appreciation. The mental allows selection, exploration, and al- archive will remain central and will be teration. The human using the system extended with computers where ap- will see, understand, and comprehend plicable. In short, the processing of in- the visual phenomenon and then form creasingly larger numbers of images by a mental image of the data, which be- the computer (meta images) allows for comes their structure and meaning. deeper interpretation of the individual For a long time, Art History has stud- image by researchers (mesa image). ied the recipient in front of an image. What is being seen is not a passive The computer, used as a tool to ex- process, but shaped by previous vi- plore visual data via meta images, puts sual experiences, cultural norms, and visualization into a core concept of expectations.52 Having the ability to BID. This concept, visualization, can not only contemplate the image, but to be added as the fifth “V” in Big Data. also interact with it, makes the viewer The computer and the data it stores are an active user and designer of data and being used via a graphical user inter- allows explorative access to virtually face—an interface between the human infinite image libraries. and the data. It processes data, reduces
7. Iconological Interpretation 6. Iconographic Analysis Human system 5. Pre-iconographic Description 4. Object Recognition 3. Visual Phenomenon of Meta Image 2. Media System (pre- Media processing, including system object recognition and pre- Table 1: Layers in Generation and iconographic analysis) Understanding of a Meta Image from 1. Information BID in Computer-aided Art History
30 DAH-Journal Big Image Data
BID seems to hold great promise for we have digitized the slide library and Art History. The methods that we are are discussing how to access its data, about to develop are not only for our the physical book library is yet anoth- own scientific needs, but might also er source of information. The next step be used to grasp the billions of imag- will be to combine and connect both es that are created outside the artistic sources of data digitally and make that realm. The methods can be applied to the basis of our research. answer the questions that society has regarding the digital image in general. The acquisition of Big Image Data, Who else can make this contribution data analysis, and visualization will to society if not the discipline that has open up to new methods in Art His- for centuries acquired a great historical tory. We need to team up with infor- and visual experience—Art History? mation scientists, define goals, and develop tools that meet our objectives. The future form of Art History Now is the time to develop such new means more than the analysis of Big approaches. However, we must remain Image Data, but at the moment that critical: data analysis does not replace seems to be the greatest promise and quantitative research; it is an aid to the challenge for the discipline. More than researcher. Results always need inter- any other area of the Humanities, Art pretation; they do not replace hypoth- History deals with images and the eses, but are their premise. And state- analysis of visual data. It is a great op- ments can only be made based on the portunity to modernize the discipline. data present. Meaning today, we teach At the same time, we should not forget our students source criticism, i.e. eval- that, in research, we draw information uating a text by asking who has written from multiple sources—not only imag- what, when and why. In the future, we es. The prototype of an Art History de- need to add critiquing digital resources partment at a university has historical- to the curriculum—data critique must ly had two repositories of knowledge: become a core competence in art-his- the library and the slide library. While torical studies.
Dynamic Experiences
Generation and Graphical User Acquisition of Visual Data Recipient Mental images Interface Data
Interactive Expectations
Figure 7: Scheme of a research workflow via a graphical user interface that accesses visual data
DAH-Journal 31 Big Image Data
We are witnessing a historical shift. visual literacy in the digital age, so to Art historians used to be in the position speak. And that, too, requires digital of image recipients. With data visual- competence. If we want to remain a izations, we have become producers of Bildwissenschaft in the 21st century, we images ourselves. Which discipline has need to embrace digital methods. Not the toolset to understand what kind of only to apply them on reproductions images such visualizations represent, of art or popular culture, but also to if not Art History? Thus, a critique of understand contemporary art that is the digital image should be included in increasingly digital-born. art-historical curriculums as well—a
Notes 1 Erwin Panofsky, “Iconography and Iconology: widely used UTF-8, which is capable of en An Introduction to the Study of Renaissance coding all possible characters, is fully backward Art,” in Erwin Panofsky, Meaning in the Visual compatible to ASCII. Arts: Papers in and on Art History (Garden 8 Sorting of pictures according to their median City, NY: Doubleday, 1955), 26–54. brightness, hue or saturation can easily be done 2 Lena Bader and Martin Gaier/ Falk Wolf (eds.): with ImagePlot developed by the team around Vergleichendes Sehen (München 2010). Lev Manovich: http://lab.softwarestudies.com/p 3 See also Udo Kultermann, ”Der Dresdner Hol /imageplot.html beinstreit,” in Udo Kultermann: Geschichte der 9 R. Datta, D. Joshi, J. Li, and J. Z. Wang, “Image Kunstgeschichte. Der Weg einer Wissenschaft retrieval: Ideas, influences, and trends of the (München: Prestel Verlag, 1996), 136–141. new age,” ACM Computing Surveys, 40(2):1-60, 4 Thomas Nelson Winter, “Roberto Busa, S. April 2008 J., and the Invention of the Machine-Gener 10 See the article by Babak Saleh in this journal. ated Concordance,” The Classi cal Bulletin 75:1 11 Moretti, Franco and Alberto Piazza, “Graphs, (1999), 3-20 http://digitalcommons.unl.edu/cgi/ Maps, Trees: Abstract Models for Literary viewcontent.cgi?article=1069&context=classics History” (Verso, 2007). facpub 12 K. Bender, “Distant Viewing in Art History. 5 See Anna Bentkowska-Kafel, “Debating A Case Study of Artistic Productivity,” Interna Digital Art History,” International Journal tional Journal for Digital Art History, [S.l.], n. 1, for Digital Art History, [S.l.], n. 1, june 2015. june 2015. ISSN 2363-5401. Available at:
32 DAH-Journal Big Image Data
Humby, Address at the ANA Senior Marketer’s buch für Bildkritik Vol. 3, No. 2, Digitale Form, Summit at the Kellogg School (2006), in Michael ed. Margarete Pratschke (Berlin: Akademie Palmer, Data is the New Oil (Nov. 3, 2006), Verlag, 2006), 47ff. http://ana.blogs.com/maestros/2006/11/data_ 26 The Von-Neumann principle describes a is_the_new.html. computer, which is universal, i.e. independent 15 Gottfried Boehm, “Die Bilderfrage,“ in Was of the problems to be processed and controlled ist ein Bild?, ed. Gottfried Boehm (München, by binary coded commands. Fink, 1994), 325-343. 27 Edmund Husserl, “Phantasie, Bildbe 16 The following paragraphs have already wusstsein, Erinnerungen. Zur Phänomenologie appeared in German in: Harald Klinke, der anschaulichen Vergegenwärtigung,“ in “Bildwissenschaft ohne Bildbegriff,” in Bilder Husserliana Band 23, ed. Eduard Marbach (Den der Gegenwart, ed. Harald Klinke, Lars Stamm Haag: Martinus Nijhoff, 1980). (Göttingen: Graphentis Verlag, 2013), 11-28. 28 See Harald Klinke, “The image and the 17 Hans Belting, Bild-Anthropologie. Entwürfe mind,” in Art Theory as Visual Epistemology, für eine Bildwissenschaft (München: Fink, ed. Harald Klinke, 1-10. 2001), 16. 29 See Belting, op.cit., 38. 18 Belting, op. cit., 15. 30 See Peter Stahlknecht and Ulrich Hasenkamp, 19 “Es gibt keine Daten ohne Datenträger. Es Einführung in die Wirtschaftsinformatik th11 gibt keine Bilder ohne Bildschirme” (Claus Pias, ed. (Berlin, Heidelberg, New York: Springer, “Das digitale Bild gibt es nicht. Über das (Nicht-) 2005), 94-97. Wissen der Bilder und die informatische 31 Panofsky, op. cit.. Illusion,” Zeitenblicke Bd. 2 (2003), §53). 32 Specifically in the Corpus geniculatum 20 Walter Benjamin, “Das Kunstwerk im laterale (see Hans-Otto Karnath und Peter Zeitalter seiner technischen Repro duzierbar Thier, Kognitive Neurowissenschaften,rd 3 ed. keit,” in Gesammelte Schriften Bd. 1, ed. Rolf (Berlin: Springer Medizin, 2012), 36-37. Tiedemann und Hermann Schweppenhäuser 33 http://www.prometheus-bildarchiv.de/ (Frankfurt/M.: Suhrkamp, 1980), 431-469. 34 http://www.artstor.org/content/artstor-di 21 Narrations of a general history of media gital-library-features-benefits usually start with the invention of the printing 35 https://www.google.com/culturalinstitute/ press - a text medium. A history of visual media browse/ on the other hand focuses on the visual media 36 http://www.artigo.org/ and therefore begins with the first human image 37 Jay H. Bernstein, “The data-infor mation- artifacts (e.g. cave paintings). The history of knowledge-wisdom hierarchy and its visual media is also not limited to the history of antithesis,“ in Proceedings North American technology. Rather, it is rather concerned with Sympo sium on Knowledge Organization, ed. E. the examination of the cultural consequences K. Jacob, and B. Kwasnik , Vol. 2 (Syracuse, NY, of the development of visual media and the 2009), 68-75. definition of the properties of the medium in its 38 R. L. Ackoff, “From data to wisdom,” Journal historical context. of Applied Systems Analysis 15 (1989), 3-9. 22 See W.J.T. Mitchell, “Realismus im digitalen 39 Charles Duhigg, op. cit. Bild,” in Bilderfragen. Die Bildwissenschaften 40 Douglas Laney, “The Importance of ‚Big im Aufbruch, ed. Hans Belting (München: Fink, Data’: A Definition,” in Gartner, 2012. http:// 2007), 237-255. www.gartner.com/resId=2057415. 23 Oliver Grau, Virtual Art. From Illusion to 41 Harald Klinke, “Heinrich Wölfflin in Immersion (Cambridge, Mass: MIT Press, 2003), Zeiten digitaler Kunstgeschichte,” in Kunstge 15. schichten 1915. 100 Jahre Heinrich Wölfflin: 24 Margarete Pratschke, “Interaktion mit Kunstgeschichtliche Grundbegriffe, ed. Matteo Bildern. Digitale Bildgeschichte am Beispiel Burioni and others (Passau, 2015) , 415-421. grafischer Benutzeroberflächen,“ in Das Techni 42 “Dimension reduction is a projection of sche Bild, ed. H. Bredekamp and others (Berlin: a space of many dimensions into a fewer Akademie Verlag, 2008), 68-81. dimensions – in the same way as a shadow 25 See Frieder Nake, “Das doppelte Bild,“ in Bild of a person is a projection of a body in three welten des Wissens. Kunsthistorisches Jahr dimensions into two dimensions.” Manovich,
DAH-Journal 33 Big Image Data
Lev, “Data Science and Digital Art History.” argued that there is no need for that anymore. International Journal for Digital Art History, A democratic approach would now be possible [S.l.], n. 1, june 2015. ISSN 2363-5401. Available in which every image process would be at:
Ackoff, R. L. “From data to wisdom.” Journal of Applied Systems Analysis 15, 1989, 3–9. Andrew Pole. “How Target Gets the Most out of Its Guest Data to Improve Marketing ROI”, presentation by at Predictive Analytics World Conference in Washington D.C. (October 19, 2010). Bader, Lena and others (eds.). Vergleichendes Sehen, München 2010. Belting, Hans. Bild-Anthropologie. Entwürfe für eine Bildwissenschaft (München: Fink, 2001). Bender, K.. “Distant Viewing in Art History. A Case Study of Artistic Productivity.” International Journal for Digital Art History, [S.l.], n. 1, june 2015. ISSN 2363- 5401. Available at:
34 DAH-Journal Big Image Data
Bentkowska-Kafel, Anna. “Debating Digital Art History.” International Journal for Digital Art History, [S.l.], n. 1, june 2015. ISSN 2363-5401. Available at:
DAH-Journal 35 Big Image Data
2016. doi:http://dx.doi.org/10.11588/dah.2015.1.21631., 30. Manovich, Lev. Manifesto for Democratic Art History, http://lab.softwarestudies. com/2016/02/manifesto-for-democratic-art-history.html. Mitchell, William J. Thomas. “Realismus im digitalen Bild,” in Bilderfragen. Die Bild wissenschaften im Aufbruch, edited by Hans Belting (München: Fink, 2007), 237–255 Mitchell, William J. Thomas: Picture Theory. Essays on verbal and visual representation (Chicago/London: The University of Chicago Press, 1994), 57. Moretti, Franco, and Alberto Piazza. Graphs, Maps, Trees: Abstract Models for Literary History (Verso, 2007). Mundie, Craig. “Data, data everywhere“, The Economist, http://www.economist.com/ node/15557443 , Feb 25th 2010, see also Clive Humby. “Address at the ANA Senior Marketer’s Summit at the Kellogg School.” In: Michael Palmer, Data is the New Oil (Nov. 3, 2006), http://ana.blogs.com/maestros/2006/11/data_is_the_new.html. Nake, Frieder. “Das doppelte Bild“ In Bildwelten des Wissens. Kunsthistorisches Jahrbuch für Bildkritik Band 3, Nr. 2, Digitale Form, edited by Margarete Pratschke (Berlin: Akademie Verlag, 2006). Panofsky, Erwin. “Iconography and Iconology: An Introduction to the Study of Renaissance Art.” In: Panofsky, Erwin: Meaning in the Visual Arts: Papers in and on Art History (Garden City, NY: Doubleday, 1955), 26–54. Pias, Claus. “Das digitale Bild gibt es nicht. Über das (Nicht-) Wissen der Bilder und die informatische Illusion:“ Zeitenblicke Bd. 2 (2003), §53. Pratschke, Margarete. “Interaktion mit Bildern. Digitale Bildgeschichte am Beispiel grafischer Benutzeroberflächen,” in Das Technische Bild, edited by H. Bredekamp and others (Berlin: Akademie Verlag, 2008), 68–81. Stahlknecht, Peter and Ulrich Hasenkamp. Einführung in die Wirtschaftsinformatik, 11th ed. (Berlin, Heidelberg, New York: Springer, 2005), 94-97. Surkemper, Liska. “Zu einer Theorie vom Ende der Theorie.” In: Archplus, 48 (Winter 2015), 66–69. Winter, Thomas Nelson. “Roberto Busa, S. J. , and the Invention of the Machine-Generated Concordance.” The Classical Bulletin 75:1 (1999), 3-20 http://digitalcommons.unl. edu/cgi/viewcontent.cgi?article=1069&context=classicsfacpub. Zweig, Benjamin. “Forgotten Genealogies: Brief Reflections on the History of Digital Art History.” International Journal for Digital Art History, [S.l.], n. 1, june 2015. ISSN 2363-5401. Available at:
36 DAH-Journal Big Image Data
Harald Klinke has a Ph.D. in art history and a Master of Science in Information Systems. Currently he is Assistant Professor at the Ludwig Maximilian University, Munich, and responsible for the doctoral program “Digital Art History”. He conducts research on visual communication, digital media, and Big Image Data in art-historical contexts. From 2008 to 2009, he worked as a Lecturer of Visual Studies (Bildwissenschaft) at the Art History Department of the University of Göttingen. From 2009 to 2010, he conducted research, supported by a grant from the German Research Foundation (DFG), as a Visiting Scholar at Columbia Uni-versity, New York. He has published books on American history paintings, digital images and art theory as visual epistemology.
Correspondence e-mail: [email protected]
DAH-Journal 37