Cultural Objects : an Inquiry Into Embedded Stories

Cultural objects : an inquiry into embedded stories

Thesis submitted in partial fulﬁllment of the requirements for the degree of

MS by Research in Exact Humanities

Priyanka Suresh 201256152 [email protected]

CERTIFICATE

It is certiﬁed that the work contained in this thesis, titled “Cultural objects : an inquiry into embedded stories” by Priyanka Suresh, has been carried out under my supervision and is not submitted elsewhere for a degree.

Date Adviser: Prof. Navjyoti Singh To my family Acknowledgments

This research would not be possible without the inputs of many people. I would like to thank Prof. Navjyoti Singh, for his constant guidance and support. I am very grateful for his patience and wisdom, and for motivating me to think differently and doing my best. I wish to thank Dr. Jeffrey Schnapp and my colleagues at metaLAB (at) Harvard for their encouragement, innovative ideas and friendship. My learning experience there was invaluable. I would also like to thank my parents for their unwavering faith, love and affection. A big hug to all my friends for keeping me sane and the wonderful memories.

v Abstract

Cultural objects are highly interpretive and complex in nature. Though technological advancements have made it possible to digitize and preserve knowledge of cultural value, capturing details of such objects in-situ has rarely been explored. In this thesis, we develop a platform called Curarium for in- depth exploration of cultural collections, within and outside a museum setting. A focus group of 17 researchers gathered cultural collections on a trip to Odisha. The collections contain information pertaining to Pattachitra paintings, Gotipua performances, sustainable crafts, museum objects and processions. The data-sets are unique in nature and content, as they provide for an interplay between different kinds of objects. In this thesis, we examine the cultural value in objects unexplored before - like processions, and what they comprise. To accommodate the different kinds of data, we develop a meta-data structure that would encapsulate all the relevant information. We import the curated data collection into Curarium and utilize the various in-house tools to discover patterns, create annotations, collaborate and visualize the data. Curarium provides a novel approach in delving deep into a cultural object. However, we recognize a gap in cross-referencing objects within its natural environment. In lieu of this, the thesis proposes developing narratives about cultural objects as an approach to capturing information. We develop a platform to generate narratives based on the digital surrogates. We conduct a user study with the focus group to develop narratives on the aforementioned data-sets on a platform called Narratavium. On further research, we recognize these narratives to be of three types - within-object, within-collection, and user generated thematic narratives. Upon recognizing plots within art objects and identifying themes, we account for the human experience in interacting with these cultural objects in their natural environment. This narrows the removed appreciation of the culture by the audience. In this thesis, we utilize sequential art methods to study Pattachitra paintings, Aristotlean narrative theories to study processions and performances, and qualitative analysis to generate themes. By visualizing the narrative information, we gain insights into nuances of the objects and the collections as a whole. We use charts and clusters to extract popular themes in a collection. We develop methods to identify conﬂict points in a narrative and understand transitions in a story plot. We also study information retention in the process of re-narration, and relevant objects in a collection through frequently told narratives.

vi Contents

Chapter Page

1 Introduction ...... 1 1.1 Classiﬁcation of Cultural Data ...... 2 1.2 Narrative visualizations ...... 3 1.3 Outline ...... 4

2 Curarium ...... 5 2.1 Development methodologies ...... 5 2.2 Technical standards for Digital objects ...... 6 2.3 Software architecture and design ...... 6 2.4 Database management ...... 7 2.5 Functional overview ...... 7 2.5.1 Collection Management ...... 8 2.5.1.1 Interacting with collections ...... 8 2.5.1.2 Importing collections ...... 8 2.5.2 Information Development ...... 10 2.5.2.1 Meta-data additions ...... 10 2.5.2.2 Annotations ...... 11 2.5.2.3 Spotlights ...... 12 2.5.3 User Management ...... 13 2.5.3.1 Circles ...... 14 2.5.3.2 Trays ...... 14 2.5.4 Visualizations ...... 15 2.5.4.1 Object map ...... 16 2.5.4.2 List view ...... 17 2.5.4.3 Thumbnails ...... 18 2.5.4.4 Treemaps ...... 19 2.5.4.5 Color Maps: ...... 20

3 Data Preparation ...... 22 3.1 Curation ...... 22 3.1.1 Kinds of data ...... 22 3.1.2 Establishing meta-data structure ...... 23 3.1.3 Exceptions ...... 25 3.2 Importing into Curarium ...... 26 3.2.1 Experiments with data-sets ...... 26

vii viii CONTENTS

3.2.1.1 Record ...... 26 3.2.1.2 Annotations ...... 27 3.2.1.3 Treemap ...... 28 3.2.1.4 Colormap ...... 29 3.2.1.5 Trays ...... 30 3.3 Results ...... 31

4 Narrativium ...... 33 4.1 Motivation for Narratives ...... 33 4.2 Functional aspects of Narrativium ...... 34 4.2.1 Platform development ...... 34 4.2.2 Creating Narratives ...... 35 4.2.3 Importing narratives ...... 35 4.2.4 Interaction with narratives ...... 36 4.2.4.1 Editing ...... 36 4.2.4.2 Forums ...... 37 4.2.4.3 Annotations ...... 38 4.3 Results and Analysis ...... 39 4.3.1 Within the object ...... 40 4.3.1.1 Pattachitra using sequential-art concepts ...... 40 4.3.2 Within the collection ...... 45 4.3.2.1 Analyzing Gotipua using micro and macro structures ...... 45 4.3.2.2 Analyzing processions using Aristotlean Theory of drama ...... 48 4.3.3 Thematic narratives ...... 49 4.3.3.1 Qualities in thematic narratives ...... 49 4.3.3.2 Identifying themes ...... 51

5 Visualizations ...... 53 5.1 For within-object narratives ...... 54 5.1.1 Identifying conﬂict points studying character co-occurrence ...... 54 5.1.1.1 Design choice ...... 54 5.1.1.2 Analysis ...... 55 5.1.2 Analysing transitions ...... 56 5.1.2.1 Design choice ...... 56 5.1.2.2 Analysis ...... 56 5.2 For within-collection narratives ...... 57 5.2.1 Using narrative graphs ...... 57 5.2.1.1 Design choice ...... 58 5.2.1.2 Analysis ...... 58 5.2.2 Information retention through narrative grammar ...... 60 5.2.2.1 Design choice ...... 61 5.2.2.2 Analysis ...... 62 5.3 For thematic narratives ...... 62 5.3.1 Popularity of themes ...... 62 5.3.1.1 Design choice ...... 63 5.3.1.2 Analysis ...... 64 CONTENTS ix

5.3.2 Geoclustering ...... 64 5.3.2.1 Design choice ...... 64 5.3.2.2 Analysis ...... 65

6 Conclusions ...... 66 6.1 Research and contributions ...... 66 6.2 Future work ...... 67

Bibliography ...... 69 List of Figures

Figure Page

2.1 Database diagram of Curarium ...... 7 2.2 Importing a collection into Curarium - 1 ...... 9 2.3 Importing a collection into Curarium - 2 ...... 10 2.4 Display of meta-data ﬁelds in a record ...... 11 2.5 Display of annotations in a record ...... 12 2.6 Creation of Spotlights in Curarium ...... 13 2.7 Display of Circles in Curarium ...... 14 2.8 Display of Trays in Curarium ...... 15 2.9 Visualization tool-kit in Curarium ...... 16 2.10 Object-map visualization in Curarium ...... 17 2.11 List view in Curarium ...... 18 2.12 Thumbnail view in Curarium ...... 19 2.13 Treemap visualization Curarium ...... 20 2.14 Color map visualization in Curarium ...... 21

3.1 Importing our data-sets into Curarium ...... 26 3.2 Record view of Ardhanareshwar ...... 27 3.3 Annotations of the Ramleela Pattachitra ...... 27 3.4 Process of annotating Ramleela Pattachitra ...... 28 3.5 Zoom into Ramleela Pattachitra ...... 28 3.6 Treemap of our Pattachitra sub-collection ...... 29 3.7 Color maps of Konark temple ...... 29 3.8 Color maps of Ganesha paintings ...... 30 3.9 Display of trays ...... 30 3.10 Tray generation tool-kit ...... 31

4.1 UML diagram of Narrativium ...... 34 4.2 Narrative album of our data-set ...... 35 4.3 Importing images for narratives ...... 36 4.4 Re-arrangement of images in narrative ...... 37 4.5 Forums for discussions ...... 38 4.6 Tags in a sub-collection ...... 39 4.7 Pattachitra painting of Chitralekha ...... 40 4.8 Analyzing icons across collections - Nataraja ...... 41 4.9 Identifying icons in Chitralekha painting ...... 42

x LIST OF FIGURES xi

4.10 Studying spatial perspectives in Chitralekha Pattachitra painting ...... 43 4.11 Tracing panel path in Chitralekha painting ...... 44 4.12 Acts in a Gotipua performance ...... 46 4.13 Scenes in a Gotipua performance ...... 47 4.14 A still of Mahishasura Mardini epic ...... 47 4.15 Explanation of Rock edicts of Ashoka ...... 50 4.16 Vyala identiﬁed as theme upon repetition ...... 51

5.1 Visualization for character co-occurance in scenes ...... 55 5.2 Visualization for analysing transitions in Pattachitra paintings ...... 56 5.3 Non-sequitter analysis of Chausat Kala painting ...... 57 5.4 Graph visualization of narratives ...... 58 5.5 Representation of terminal nodes and connection nodes ...... 60 5.6 Narrative grammar representation in narration layer ...... 61 5.7 Narrative grammar representation in performance layer ...... 61 5.8 Narrative grammar representation in re-narration layer ...... 62 5.9 Narrative grammar representation in re-narration layer ...... 62 5.10 Bubble chart for popular themes ...... 63 5.11 Example of geocluster from data-set ...... 64 5.12 Zoomed-in geoclusters ...... 65 List of Tables

Table Page

2.1 Utilitarian chart for artifacts and digital representation ...... 6

3.1 Meta-data ﬁelds validated by CORE ...... 25

xii Chapter 1

Introduction

In the technological age, understanding culture by studying cultural objects has reached unparalleled capacities. We consider cultural memory in a provisional definition as an interplay between the present and past in socio-cultural contexts, which extends to its study by construction of identity through narrative and non-narrative means.[59] Cultural memory through data has become more accessible by digi- tizing tangible and intangible objects, and also by preserving knowledge that is susceptible to transfor- mation and deterioration. A digital archive, as externalized cultural memory helps us establish context for an object in ideation and tracing hybrid knowledge. This effort is extended to both a museum setting and the original environment of the object. In museum exhibitions, the ex-situ preservation means a transfer of specimens from their original habitat to the new site. [18] A challenge in this setting is that the object is removed from its context of use [30], and within an externally curated space. Digitization helps with in-situ preservation, when an object is in its primitive state in an original environment. A challenge in setting is that the information remains abstract and non-relatable for people without prior knowledge about the object or its environment. In this purview, we first place the cultural object on an art-artefact spectrum of classification with its digital representations. We develop a platform called Curarium for curating, analyzing, and exploring cultural collections. We use meta-data curation tools, annotations, publishing tools, and visualizations for this purpose. We then address the uni-dimensional interpretations of stand-alone objects, especially in a museum setting. An example of this is the bas- reliefs studied in the Garbhagriha of Rudreshwara Swamy temple in Warangal, Telangana. All the idols and statues are oriented towards the main idol in their composure, stance and activity. When removed from this original environment, information about the roles of each component of the cultural object is lost. Another example is seen in the Rath Yatra festivities of a temple. In the Puri Jagannath temple, this procession lasts for nine days and contains various decorations, processions, dance and music performances, worship rituals, meals served and public participation. Despite having digital representations of the procession in the form of images, videographic materials etc., it is necessarily different from the procession itself. This is because the digital objects require weaving of the various elements in space and time to represent the original. Thus, we postulate that a cultural object is necessarily different from its digital representation, in its multidimensional complexity of various objects, their coexistence and re-

1 lations. This representation without loss of information requires safeguarding the integrity of the object. A solution that we propose to encapsulate the richness of such complex cultural data sets is developing narratives around these objects. We then represent and visualize them to bridge gaps and gain further insights.

1.1 Classiﬁcation of Cultural Data

The various conceptualizations for distinction of objects include stationary and mobile [21], tangible and intangible, temporal and atemporal [58] objects etc. Various standards for representing such objects as minable information are present. The Nomenclature of Museum Cataloguing [52] contains six hierarchies with ten categories and necessary sub-classifications. The aim is to determine object types for indexing in museums and the principle for classification is to find the original functional context of the objects. A limitation of this system is that it does not account for non-artifact objects. An example from our dataset is the dance performance of Gotipua, which cannot be classified within this system as it is performance art. In comparison, the Art and Architecture Thesaurus (AAT)[4] focuses on objects specifically in the art and cultural heritage domain and is organized into discrete concepts - single or compound words used consistently to describe a generic concept. This makes it highly versatile for cataloguing but the system omits objects that are unique and not part of a generic whole. For example, a historical event or any object that would classify as a proper noun would be excluded from this system. It also specifically only pertains to objects relating to arts and architecture of visual imagery, and not other technical or scientific objects. The Thesaurus for Graphic Materials (TGM)[8] includes a broad range of subjects including activities, objects, events, places etc. in alphabetical order to include subject terms and descriptive terms. But it is limited in the scope that it only deals with still images and has a very narrow terminology for describing the objects. The Cultural Objects Naming Authority (CONA)[46] scheme includes a wide variety of objects classified into built works and movable works, and is constructed as a thesaurus. The objects include performance art, certain kinds of scientific instruments, artists literature and films but are necessarily limited to the fine arts with skilled craftsmanship. To include the different kinds of objects in our classification schema, determine objects on an art- artefact spectrum is based on utilitarian value [51]. In this system, categories are established based on function from aesthetic value to that of utility. The hierarchical categories are: Art (paintings, sculptures etc.), Diversions (philosophy, meals etc.), Adornment (cosmetics, jewelry etc.), Landscapes (architecture, agriculture etc.), Applied arts (furnishings, receptacles etc.), Devices (scientific and musical instruments etc.). The objects are identified based on what they are made of, ie their materiality. This helps us gauge their digital representations [50]. For example, a procession at a wedding can be placed as a diversion as it is a ritual and a performance. Its materiality is performance actions like laughing, rising etc. The digital representation for this would be a videographic material represented in MP4, AVI, MOV formats and/or images in PNG, JPEG formats etc. This distinction helps for seamless coexistence and interaction between objects in a digital archive. For instance, if a collection is comprised

2 of materials with varied digital objects, allowance for visualizations or annotations is easier by accom- modating the different digital formats. Digital representations thereby help us study an object, both from the perspective of a museum space and in its original setting. There are several proprietary and open-source tools that support digital archival and curation processes on the web. Contentdm by OCLC [25] is a powerful and easy cataloging tool, but expensive. Open-source tools such as Omeka [55] provide capabilities limited to meta-data analysis and indexing but have no support for collaboration. Tools like Collective Access [3] requires advanced technological capabilities for curation and collaboration. Pachyderm by New Media Consortium [14] is highly interactive but does not provide cataloguing capabilities and requires download for individual hosting. We identify requirements for cataloguing, as well as tools that enrich meta-data, visual and textual annotations, and analysis tools available for research. To further understand the richness of a cultural object, we need to minimize abstraction and provide context. As our datasets contain media-rich cultural objects in-situ, we explore narrative visualizations as a method to provide relatability. Though a sequence of images provides information in terms of a visual language, certain experimental and non-universal iconography can be explained by expert users or collection providers with more knowledge of the object and its surrounding. This helps with ﬂuency of the visuals in understanding the images as a sequence and to derive meaning out of it as a narrative within a structure.

1.2 Narrative visualizations

As per Gustav Freytag [24], stories of all types have a structure that follows five basic stages of action - exposition, rising action, climax, falling action and resolution. These stories concern characters, a sequence of facts and observations, or themes and arguments. [31] Similarly, a narrative visualization is said to be a frame for data stories with focus on perceptions of the data. Visual structures of representation can either be narrative, presenting unfolding actions and events, processes of change, transitory spatial arrangements; or conceptual, representing participants in terms of their more generalized and more or less stable and timeless essence in terms of class, structure or meaning. [34] Further, conceptual structures are divided into three types - classification, analytical and symbolical. In a visual narrative, the grammar (VNG) posits that images in a sequence have narrative roles within a constituent structure. [34] Based on the method followed, the tools are either dynamic or static. We explored dynamic storyboards and narratives to address the interpretive aspects of the datasets while using more static structures for classification information. Narrative visualizations are often known to be interactive but based on the design decisions, the flow of the story can be moderated. Based on our requirement for pedagogical dialogue between the author and viewer for ideation and verification in research, we picked the martini-glass approach. [56] The processes for a datastory are also argued in [16], as finding insights, making a story and telling it effectively. In this effort, various semiotic modes also need to be explored for depicting information in

3 an effective manner. As per Kress et al [34], both linguistic and visual modes are necessary to achieve conclusions and the audience is crucial as realization possibilities are determined historically, socially and by the inherent potentials of the modes. Various tools and methods for narrative based visualizations exist such as CLUE where users exchange between exploring and presenting Vistories [22], Visjockey [35] , SketchStory [39], Vistrails [12] etc. However, these tools are limited in provisional facilities focussing on cultural data. In cultural research, although there is extensive use of annotations, textual analysis and applied se- mantic analysis like CATMA [45], there is a lot of scope for utilization of visualizations. Paper Machines [23] along with its visualization packages, Curarium Satellite [2], VOYANT [57] are some examples of research towards visualizations in digital humanities. 3DH [44] is another project exploring topics related to visualizations in this space, speciﬁcally related to narratives and narratological annotations. In contrast to analyzing narratives within a corpus, our research studies the process of generating narratives enhanced by visually analyzing them through conceptual methods.

1.3 Outline

• Chapter 2 describes the development of Curarium and its various affordances

• Chapter 3 describes the collection and management of data, and user study in importing it into Curarium

• Chapter 4 proposes development of a narrative platform for interpretive data

• Chapter 5 displays various visualizations utilized to derive analysis and insights out of the narrative data

• Chapter 6 has the conclusions and further work which can be done to truly realize the potential of what has been proposed.

4 Chapter 2

Curarium

We have designed and implemented Curarium [1], a web platform to address the needs of digital curation and knowledge preservation in cultural collections. Through effective moderated curation, interactive visualizations and constant ideation, we intend to examine and enrich objects of this type. The system is developed using the Rails framework and based on four main functional blocks: collection management, information development, user management and visualizations. The collection management block is for users to import new collections and their individual objects and to modify existing ones. It also has an option for viewing the stored collections and objects. The information development module enables enriching existing repository information using annotations, articles, linking surrogate objects, making trays or sub-collections and appending object-specific meta-data. The user management system authenticates users of the platform, so that they can interact with the collections and one another for pedagogical research and information sharing, with defined access controls and profiles. The visualization module helps for analysis of extensive collections through selection of visualization types, properties and filtered meta-data.

2.1 Development methodologies

Curarium has adopted the Agile method of software development, which allowed ﬂexibility and iter- ative prioritization. Using the Redmine project management tool [38], we keep track of system requirements and share progress with key stakeholders. The user requirements are noted, design is explored and behaviour is estimated before and during the development process. Behavious driven strategies are adopted with version prototypes for developers and stakeholders respectively. Focused communication with technical and non-technical stakeholders enable requirements gathering and testing at each stage. Continuous improvement through prioritization of tasks enable different modules of the application to develop from a prototype to a working integration towards the desired results. Feedback from the users of the platform is also facilitated to report bugs or suggest improvements using Redmine. Interested con- tributors can also view version control through the repository on github. All the improvements under consideration are made live in successive releases.

5 2.2 Technical standards for Digital objects

To ﬁt archival themes, the artifact is qualiﬁed as possessing cultural value. In order to do this, we locate objects in a spectrum of aesthetic to utility [51]. The material composition of these objects are determined in order to identify their digital representations. [50] This enables seamless transition between objects in the stage of inspection and further curation. In 2.1, the utilitarian chart helps distinguish the various artifacts in a collection. This helps us develop the platform in order that we can accommodate the technical standards of various digital objects.

Utilitarian type Examples Digital Representations

Art paintings, sculptures PNG, JPEG Diversions philosophy, performance MP4, AVI Adornments jewelry, cosmetics PNG, JPEG Landscapes architecture, agriculture MP4, PNG Applied arts furnishings, receptacles PNG, JPEG Devices Scientiﬁc, musical instruments PNG, JPEG

Table 2.1 Utilitarian chart for artifacts and digital representation

2.3 Software architecture and design

Curarium is a web platform developed using Ruby on Rails. The application has an architectural pattern known as Model-View-Controller (MVC) dividing the functionality into three components. The Model component which implements the business logic and data manipulation is written as Ruby classes. The Views which correspond to the user interface is majorly operated in HTML5 and Javascript. Controllers handle bi-directional interactions between models and views. For example, when a user in- teracts with a view and changes the data on the interface level, the controller enables the appropriate updates in the model, and similarly when the user invokes information, the controller fetches the appropriate data to be displayed. Models implement the Active Record architectural pattern, providing an Object Relational Mapping (ORM) layer which supports a wide variety of Relational Database Man- agement System (RDBMS). The RDBMS that we have chosen is PostgreSQL 9.3+ (with PostGIS 2+ extension and a spatial database template) RDBMS. The dependencies we required to get started with Curarium were Ruby 2.0+ and Rails 4.1 compatible with Ruby on Rails in its current version. We used redis 2.6+ for our server requirements and ImageMagick 6.9+. The web user interface has been devel-

6 oped combining the Ruby’s built-in erb templating system that enabled us to deal with the Asynchronous JavaScript technology.

2.4 Database management

Figure 2.1 Database diagram of Curarium

2.5 Functional overview

As mentioned above, the four main components that Curarium is composed of are: Collection management, Information development, User management and Visualizations. With reference to Curarium, we deﬁne the following terms to help better understand the above components :

• A record as a digital representation of a cultural object.

• A collection is a group of records uploaded by a collection provider.

• An annotation is an explanation, tag or comment for the record by any user of the platform.

7 • A tray is a personalized bin of records, sub-collections, annotations and visualizations.

• A circle is a personal or private group to share insights and facilitate interactions.

• A circle tray is a personalized bin of sub-collections pertinent to a speciﬁc circle.

• A spotlight is a module that allows written articles for insights facilitated by Curarium.

2.5.1 Collection Management

In a museum, where the collections conform to a museum policy, extensive documentation and policies guide the collection management systems in a digital space as well. Each of these collections are also distinguished by a personality and characteristics of their own, which need to be further captured and analyzed. Thus, collection management component is central to the functioning of the platform. It facilitates importing, ﬁnding and following a collection for curation and exploration. The objects in a collection are also accessed with their own uniqueness, adhering to the themes within a collection. Affording cataloging and archival processes efﬁciently need to be considered, especially for big-data cultural collections.

2.5.1.1 Interacting with collections

An overview of the different collections is available to all visitors and users of the platform. In- dividual collections can be inquired further to reveal information about the collection provider, date, description, number of records and associated spotlights with a sample of records at a glance. Records : Individual records in their digital formats are mirrored on a Javascript canvas. Along with this, the meta-data ﬁelds, surrogate objects, a feature to add the record to a personal tray, and associated annotations are displayed.

2.5.1.2 Importing collections

During import, the collection provider follows an organized protocol for meta-data by filling out the fields for general information - name, description, provider detail, a sample of the collection and a zipped file containing all the records in JSON format. JSON format for information representation seems ideal as the key and value fields are parse-able to display the meta-data fields and their corresponding values, pertaining to the specific collection. The records require external hosting to mirror them onto the platform as it is an efficient alternative to data storage and management. The uploaded collection file is unzipped and processed with the help of Javascript to provide a list of name/value objects, thereby converting the information into a human-readable format. It is displayed for validating the format of the meta-data fields. The collection provider is then able to drag-and-drop required values from the parsed JSON data into fields of interest (Examples include Unique identifier, Title, Thumbnail etc).

8 There is also a provision for custom fields, adding flexibility to curation for the meta-data to be very collection-specific.

Importing is done using sidekiq. The JSON that is uploaded when the user imports from the New Collection page, is entered into Redis. Sidekiq extracts the files, creates the records, downloads the corresponding thumbnails, extracts colour information etc. Based on the position of the ”key” in the JSON file, it scrapes the document and extracts the ”value” to be displayed specific to the object itself.

In 2.2 and 2.3, the different fields marking general and important meta-data are displayed on the right. An option to append meta-data with drag-and-drop to pick the fields from the parsed information is also available. This enables the collection provider to display appropriate information. An option to make the collection private or public is present, and upon publishing, the modifications to the imported collection is saved on the platform.

Figure 2.2 Importing a collection into Curarium - 1

9 Figure 2.3 Importing a collection into Curarium - 2

2.5.2 Information Development

2.5.2.1 Meta-data additions

An deep inquiry about individual objects can be obtained through meta-data. It is the descriptions about the objects for purposes including but not limited to discovery and identification. A user of the site can suggest meta-data fields to enrich information or aid development of visualizations. A tool-kit in the record page is provided to append custom fields with the corresponding key/value format. The client-side addition in HTML5 invokes the controller to add a row to the records in the database.

10 Figure 2.4 Display of meta-data ﬁelds in a record

2.5.2.2 Annotations

Annotations are used to make observations about objects or data, within itself. This feature is available to users of Curarium. In the record page, a user is able to create and apppend annotations to make arguments about the object, and these are consolidated and displayed to other users of the platform as well.

Annotations are facilitated by the properties of a HTML5 canvas to section a part of the image by drawing a rectangle. The section is duplicated and can be labelled with a title and description. Hover over the duplicate highlights the corresponding section of the displayed media. The x and y coordinates, width and height for the annotation area are recorded and associated to an annotation id for a particular record. Upon clicking a button to display the annotations by a user, the annotation areas are highlighted.

11 Figure 2.5 Display of annotations in a record

2.5.2.3 Spotlights

To facilitate observations, arguments and publish stories we developed spotlights. Interaction through comments and messages help in prevention of topic obsolescence, pedagogical discussions and collaborative ideation. This occurs on three levels in Curarium - on a collection-level, an object-level and among collections. This distinction is important to observe as it directs the publications and collaboration. The ability to access spotlights occurs in the main collections page, a record page and on the top-menu.

When a user is signed in to generate a spotlight, authentication takes place through Mozilla Persona verifying the user email. Redirection through herokuapp enables the user to arrive at a text-box input with a custom font tool-kit. Based on the privacy level of the spotlight, it is shared either publicly, to speciﬁc users or within a circle. Access to viewing, editing and commenting are also allowed based on the same criteria. Spotlights are also added to ”trays” to make enable textual descriptions within a sub-collection.

12 Figure 2.6 Creation of Spotlights in Curarium

2.5.3 User Management

As a web-based multi-user application, many users can access Curarium simultaneously. They explore collections, interact with other users and import their own collections. They also have various roles within the classroom setting - such as instructor, student and researcher with individual privileges and access. Each user is allowed to perform different levels of operations and access different kinds of information and they may have different roles in different projects as well. User authentication was earlier carried out by a Mozilla plugin called Mozilla Persona but it discontinued in December 2016. Thereafter, we relied on internal authentication systems by adding a database table to keep track of the users Each user has a profile, with the name, email, a username and a short bio, which can be accessed and edited at any time. This information provides insight about the interests, skills and specializations of the user to the others. Along with these functions, the profile page displays circles, the trays created, the collections imported and the spotlights written by the user. Each user’s request to gain access to a specific resource, involves a call to the centralized authorization function, passing along some arguments (for example, the user id) to it. To retrieve these parame- ters, the system has to perform some queries on those database tables that are related to the authorization mechanism. In order to reduce duplicate queries and repeated function calls, we have implemented a

13 caching strategy that allowed us to improve the performances of the system in terms of responsiveness and reactivity.

2.5.3.1 Circles

In order to share collection specific research and interact with a fixed group of people, Curarium provides an option to create circles. Specific flags are given to circles so that they may be accessed by target audience. The three privacy settings of circles are: private, community and public. In a private circle, only invited users can view and join the circle. In the community setting, all users can view the circle but only invited users can join the circle. In a public setting, all users can view the circle and signed-in users can join the circle. The information required while creation of a circle are title, privacy level, users and description. As part of a circle, one can modify collections, interact with each other through messages and create circle-specific trays. Based on the range of privacy levels, the user is authenticated against the user id being a part of the list of circle users to enable appropriate levels of access.

Figure 2.7 Display of Circles in Curarium

2.5.3.2 Trays

To enhance cross-collection research and gather resources speciﬁc to themes, trays are useful. When at a record page, a logged in user can select it and form hybrid sub-collections to explore certain themes. This tray can contain records within and across collections, as well as associated spotlights and annotations. This helps us direct interactions in a thematic manner.

14 For a queried user id who adds an object to a tray, it gets assigned a tray item id which enables us to access the various objects from a tray as well as identify the different trays it belongs to, during exploration of individual records. The same follows a circle tray, which is a feature that allows trays to be added within circle interactions.

Figure 2.8 Display of Trays in Curarium

2.5.4 Visualizations

For representation and consolidation of qualitative information we developed a visualization tool-kit. During exploration of a collection, we enabled ﬁve types of displays for plug-and-play visualizations. They are:

1. Object maps, which help locate objects within a huge collection and get a birds eye view on the interaction between them. This enables easy access of objects, with a dense display and also to understand the thematic aesthetics of a collection.

2. List view, which helps us delve deep into objects one at a time. Each object is given the same real estate on screen, and great detail in meta-data.

3. Thumbnails, are a middle ground between the two above, where the objects are displayed in a grid with their title. Multiple columns provide a greater density of objects, though not a lot of detail in information.

15 4. Treemaps, which provide a hierarchical view of data. The tree branches are represented by rectangles and the sub-branches are represented by smaller rectangles. This proceeds iteratively until the nodes of the tree are reached, thereby enabling deep thematic research.

5. Color maps, where the colors in the objects and their ratios are displayed as a horizontal bar. This helps us observe color themes within collections across eras, cultures, titles etc, identifying important or dominant hues.

We have a visualization controls tool-box for ﬁltering various collection properties like title, medium, location, date, culture etc. that can either be included or excluded, along with deﬁning the number of objects per page, a check-box for colors and the different kinds of displays. A drop-down menu was selected for the toolbox to accommodate the functionality of the control system.

Figure 2.9 Visualization tool-kit in Curarium

2.5.4.1 Object map

Design choice : The objects in the collection are displayed as thumbnails on our visualization page. To generate a map that mirrors the collection, we utilize smaller thumbnails for the cultural objects in an alternate canvas. This map, known as a geomap is placed on the bottom-left as an intuitive design choice. The geomap displays the portion of the map the user is at, at a particular moment. To denote the current region, a red box hovers over the objects. Clicking on an image thumbnail, redirects to the record page.

16 Implementation : A HTML5 canvas is used to suspend the thumbnails on a map with 9 levels of zoom. This enables deeper interaction with the objects within the collection. To generate a geomap, a duplicate of the canvas with smaller dimensions is superimposed on the bigger canvas. The hover is stylized using CSS with transparency.

Figure 2.10 Object-map visualization in Curarium

2.5.4.2 List view

Design choice : For a more in-depth understanding of the collections at a glance, the images of the objects along with their meta-data ﬁelds are displayed. Clicking on an image, redirects to the record page.

Implementation : To make listing efﬁcient, two containers are created for rendering partials in the HTML5 page. One on the left is for media thumbnails and the right is for related meta-data. This also automates equal on-screen space for each object.

17 Figure 2.11 List view in Curarium

2.5.4.3 Thumbnails

Design choice : To view more records within a collection, with minimal additional information, we display the thumbnails on the visualizations page. On the top right corner of the thumbnails, the titles of the objects are placed. The background for this is red as it provides sufﬁcient highlight for the title to be legible. Clicking on an image thumbnail, redirects to the record page.

Implementation : A column class is created to equally space a grid of 6 thumbnails per row, and is then populated with the thumbnails. For related titles, a partial containing this information is present on the right corner of the image thumbnail. The thumbnails are cached for efﬁcient loading.

18 Figure 2.12 Thumbnail view in Curarium

2.5.4.4 Treemaps

Design choice : The objects within a collection are represented as a hierarchy in a tree-structured map. The visualization consists of rectangles in proportion, and ordered as per quantity. The ”property” such as title, culture etc of the tree structures are displayed along with the number of objects that in it. Clicking on each of them generate a tree of objects within them, structured based on the same property.

Implementation : The D3 library[46] is used to develop a recursive treemap based on selected properties. The number of children for a parent node is based on the enumeration of the property counts within limited boundaries. The size of the tiles of the tree are scaled based on the maximum number of children of the parent.

19 Figure 2.13 Treemap visualization Curarium

2.5.4.5 Color Maps:

Design choice : The percentages of color within an image are extracted. They are displayed as a single horizontal bar with the respective color and width percentages. These images are placed as per the thumbnail view and clicking on an image, redirects to the record page.

Implementation : The top colors are extracted using the nearest RGB values of the thumbnail pixels and stored in a hash table. They are arranged in descending ratio of their percentage composition and rounded off to the 4th decimal.

20 Figure 2.14 Color map visualization in Curarium

21 Chapter 3

Data Preparation

A group of 17 researchers collected data in the form of interviews, surveys, images, video and audio clips as part of a ﬁeld study. They visited various cultural heritage sites in Orissa, in and around Puri, Bhubaneshwar and Raghurajpur. Raghurajpur is known as the heritage village of India and is home to various forms of art that dates back to 5th century BC - like the Pattachitra paintings, Gopitua dance, sustainable crafts and a multitude of activities for Puri Jagannath temple.

3.1 Curation

3.1.1 Kinds of data

As a part of this research, we interviewed and surveyed inhabitants of the village, who are all painters and craftsmen tracing back various generations. The information collected pertains to iconography, symbolism, technique of painting, history and development of their respective art forms. The data-sets speciﬁcally correspond to art objects that belong to one of the following categories:

• Pattachitra paintings are a type of painting where the canvas is divided into different panels representing temporal sequences in a story. As the information from a Pattachitra painting would be available to the audience all at once, we chose to gather images of the paintings.

• The dance form of Gotipua, a precursor to Odissi, makes use of various dance and acrobatic poses during the performance of a story. In this case, as the information is unveiled temporally, we chose to record video clips of the performances. For purposes of comparison between the various forms of art, we also collected images of instances from the performances.

• The heritage sites include temples from various Shaivaite, Vaishnavaite, Buddhist, Jain and Jagan- nath cultures. Images and video clips of the sites were gathered as these sites were explored. We also visited Odisha State Museum and restoration center, and Archeological Museum of Konark and gathered images of restored sculptures, fabric, scrolls and coins.

22 • Unique to the data-set is also a variety of other objects - like processions during a wedding, a Rath-yatra ceremony at a temple festival, traditional Odishi meals etc.

As per the CDWA standard [27], we generate meta-data for 500 objects in the collection. According to the nature of our data-sets, we narrow the fields to capture relevant content validated by the Dublin Core [32]. After selection of important fields, they are converted into the respective JSON documents for further research. In order to determine usefulness of a field, we identify the nature of sub-collections and assign flags to extract relevant information. The five sub-collections based on the kinds of data gathered can be classified as :

1. Paintings

2. Performance

3. Architectural

4. Museum objects

5. Processions

A detailed explanation for our choices along with an example from our data-set is as follows:

3.1.2 Establishing meta-data structure

Category Sub-category Example CORE Usefulness Catalog Level Item Yes 1,2,3,4,5 Classification Painting Object/Work Work type Pattachitra Object Date 2016 {Earliest Date, Latest Date} Classification term Odishi Painting Yes 1,2,3,4,5 Classification Object type Title Krishna Leela Yes 1,2,3,4,5 Title Preference Krishna Leela Title language Hindi Creator Yes 1,2,3,4,5 Creator role Painter Identity Creation Creation date 2016 Location Raghurajpur, Odisha Culture Jagannath

23 Styles / Periods / No 1,2,3,4,5 Groups / Move- ments Dimensions 18 x 12 inches Yes 1,3 Materials / Tech- Cured Cotton cloth Yes 1,2*,3,4,5* niques and gum Inscription No 1,3 State No 1,2 Edition No N/A Facture No 1 Orientation No N/A Physical Description No N/A Condition / Exami- No 1,3,4 nation history Conservation / No 3,4 Treatment History Subject matter Legend Yes 1,2,3,4,5 Context No 1,2,5 Descriptive Note No N/A Critical Responses No N/A Related works No Current Location Raghurajpur, Odisha Yes 1,3,4,5 Copyrights / Re- No 1,4 strictions Ownership History No 1,4 Exhibition History No 4 Cataloguing History No 4 Related Visual Doc- No 1,4 umentation Related textual ref- Yes 1,2,3,4 erences Name Yes Name Source Person/Corporate Biography Body Birth Date Authority Death Date Nationality Indian

24 Life Roles Painter Name Raghurajpur, Odisha Yes Name Source Place/Location Coordinates 19 53 7.67 N, 85 49 Authority 35.54 E Place Type Village Context Origin of Pattachitra Generic Term Yes Con- Source cept Context Authority Notes Name Yes Subject Source Authority Context Table 3.1: Meta-data ﬁelds validated by CORE

3.1.3 Exceptions

Fields from the above table are selected based on utility for a speciﬁc sub-collection.

• Each of the Date ﬁelds contains a tuple of Earliest date and Latest date.

• As Curarium mirrors images and thumbnails from the host site, we also add ﬁelds for primary- imageurl and a base image for thumbnails.

• In particular, ﬁelds relating to Copyrights, Ownership, Exhibition and Cataloguing histories are speciﬁc to museum objects.

• For performance based objects, materials and techniques would comprise performance actions like dancing, rising, laughing etc.

• From the datasets, a procession collection is also considered as a performance, where the audience also participates within it.

• With respect to the State ﬁeld, paintings which are incomplete or completed in stages are recorded and in a performance, the scene and act number depicted in the object are noted.

• We also include the object information with respect to the collection like unique identiﬁcation number and other accessibility features.

25 • We generate Unique Resource Names for each of the objects in the collection. They follow the schema ¡URN¿ ::= ”urn:” ¡NID¿ ”:” ¡NSS¿. Here, the NID represents a Namespace ID and the NSS represents Namespace speciﬁc string. In our collection, the NID is a ”uuid” which is a globally unique identiﬁer that we have generated using a java script.

3.2 Importing into Curarium

Based on the above structure, we prepare the data as individual JSON documents. We have curated about 500 records based on the cultural heritage sites, festival processions, art, artifacts and performances. They are digital surrogates of in-situ cultural objects. For the importing process into Curarium, we make a zip of each of these collections. An arbitrary file is picked to select the standard meta-data fields to be displayed. As the zip file gets configured, we drag and drop the necessary meta-data from the canvas into active fields. Based on the collection, we also pick certain recommended fields to enrich the meta-data of the collection. For example, we pick the ”UID” of the object to display the unique ID in the archive, ”baseimageurl” for the thumbnail field, name of the object for ”title” etc.

Figure 3.1 Importing our data-sets into Curarium

3.2.1 Experiments with data-sets

3.2.1.1 Record

We are able to view the individual objects and appreciate them in greater detail by viewing its meta- data and its associated surrogate objects. In the nested JSON, we have a ﬁeld to display ”associateurls” of associated surrogate objects. So when we explore a digital object, the other objects in the collection associated with it also appear. An example from our collection is an object displaying a portion of the ”Ardhanareshwari” painting. To gain a better understanding of it, we see the multiple views through its

26 digital surrogates. Through this setting, we are able to draw relations between objects in the collection which would have been harder otherwise.

Figure 3.2 Record view of Ardhanareshwar

3.2.1.2 Annotations

To explore the information revealed by a Pattachitra painting, we view the record page of the object. Using the annotation module, we are able to isolate parts of the painting for further details. For example, in the panels of Fig 3.3, the story of Ram-Leela is depicted. We are able to study each component of the painting depicting a speciﬁc part of the story. This helps us unveil the narrative displayed within an object.

Figure 3.3 Annotations of the Ramleela Pattachitra

27 The annotated parts of the painting is highlighted with a red translucent box over it. Hover over the box shows a pop up of the textual description to the annotation. As seen in Fig 3.4 we are able to add multiple annotations with their respective titles and descriptions to delve deeper into the object. Further, from ﬁg 3.5 we note that for a more detailed annotation, the canvas enables zoom. This also helps us get a closer look at parts of the digital object as if it were a real canvas. In this manner, we are able to identify subtle elements of the narratives depicted by the painting. Hover over the zoomed annotation, reveals the pop-up providing the description of that part of the digital object.

Figure 3.4 Process of annotating Ramleela Pattachitra

Figure 3.5 Zoom into Ramleela Pattachitra

3.2.1.3 Treemap

Visualizations on the cultural objects helped us gain a deeper understanding of the hierarchy in the data. For example, the treemap ﬁltered by title, displays the important mythological characters that appear in the paintings of the collection. It is interesting to note that in a culture that is predominantly Jagannath/Vaishnava, 15 percent of the icons are Ganesha, which is an important ﬁgure in the Shaivaite culture. From 3.6 we also observe that a majority of the sub-collection contains objects relating to cultural heritage sites - like the Konark Sun Temple, Puri Jagannath temple etc. We gather from the hues that as the darkness of a box in the tree-map increases, the more that topic is populated.

28 Figure 3.6 Treemap of our Pattachitra sub-collection

3.2.1.4 Colormap

The colormap in ﬁg 3.7 is ﬁltered by title to show images from different parts of the Konark temple. The color extracts show similar hues for the stone carvings outside the compound and main temple.

Figure 3.7 Color maps of Konark temple

From ﬁg 3.8 we also notice the colors used in the paintings in which Ganesh is depicted in our sub- collection. We gain an understanding that the majority of Pattachitra paintings on the silk canvas contain shades of black and grey as they are more easier to obtain by the painters.

29 Figure 3.8 Color maps of Ganesha paintings

3.2.1.5 Trays

The users also have a facility to create sub-collections out of digital objects from either the same collection or across collections. As seen in ﬁg 3.9, we create two sub-collections, one for images of Ganesha from one collection, and another with the theme Krishna which contains a Pattachitra painting of Krishna as well as the Puri Jagannath temple (as Jagannath is said to be an incarnation of Krishna). This way, we are able to generate sub-collections and perform explorations and generate spotlights within them. To create a tray, we access the tool-kit from the record page of a digital object.

Figure 3.9 Display of trays

30 Figure 3.10 Tray generation tool-kit

3.3 Results

Curarium is an effective tool to discover insights and relations in rich cultural data. We are able to :

• Understand narratives present within an object like a Pattachitra painting and annotate parts of the digital object to make the painting highly readable

• Use in-house visualizations like colormap to identify dominant hues in paintings and cultural heritage sites

• Use treemaps to understand hierarchy within the data, and also frequent themes

• Make sub-collections of objects from across collections to generate directed collaborative arguments

• Make meta-data changes to suit our collection-speciﬁc needs

However, one major challenge we face in Curarium is the lack of provision for building arguments across different objects in a collection.

• For example, we are unable to identify patterns in the Gotipua performance when viewed as a conglomerate. Though we rely on a sequence of images in the collection but are unable to annotate it as a whole to develop it as a singular object of cultural value.

• Outside the realm of meta-data curation and suggestions, we are unable to capture insights associated with a digital object. A spotlight differs in this respect as it provides references to objects but does not associate itself with the object.

31 • As the curation process makes places the object in an ex-situ environment for analysis, we are unable to capture the entire atmosphere and personality in its natural environment. For example, we are unable to analyze the Rath Yatra processions and all its elements.

32 Chapter 4

Narrativium

4.1 Motivation for Narratives

In Curarium, insights are derived from each object individually on a surrogate level. We perform analysis to gain a better understanding of the various interpretations within an object and its relation to the collection. We are able to access meta-data and use search, ﬁltering and annotations to delve deeper into themes. However, we are limited in the scope of building cross-object relations within a collection and across collections. We are also unable to account for the various elements of the object and the natural environment it belongs to. We address this challenge by developing a narrative layer by developing a platform called Narrativium, and studying objects through relations within them. In a narrative, we identify the concepts that change over time and explore how these concepts relate to each other. In this context, narratives are deﬁned as storylines developed by the user around the cultural objects while interpreting them, understanding them or studying them. For this study, we determine functional aspects of Narrativium as in 4.1, and determine the topology of the narrative layer of study. Further analysis of the narrative structure and content help us widen the scope for research with objects of cultural value.

33 Figure 4.1 UML diagram of Narrativium

4.2 Functional aspects of Narrativium

4.2.1 Platform development

To facilitate creation of a narrative layer, we developed a web platform called Narrativium for our corpus of multimedia content. Narrative structure tactics [56] such as order to the objects and interactions between them, are accounted for in its design. We focus on composition of the narratives such that they consist descriptions and thumbnails of the cultural objects. The narratives are constructed with an order to the images which are aided by textual descriptions. 4.2 displays a narrative about the temples in Odisha. It consists of the images of the temples along with some information with each of those images. The narrative album has an image as the album cover and synopsis of its contents as well. This ﬁgure shows a view of the Jagannath temple in Puri, and the development of temples that began around 11th century AD in Odisha. Interaction with members of the

34 platform occurs through comments and other incentive-based methods like ”thumbs-up”. Suggestions and discussions aid in ideation and veriﬁcation of information.

Figure 4.2 Narrative album of our data-set

4.2.2 Creating Narratives

4.2.3 Importing narratives

As training towards functionality of Narrativium, users are exposed to previously generated narratives. For authentication, they are required to sign in to access full functionality. Upon administrator approval, they proceed to create stories around cultural objects. For this process, users decide the topic of their narrative to determine the method of presentation. They select images from the repository to host on the platform. Upon finalizing the collection, users provide a title and describe their album. For each of the images, they add a description to aid the flow of the story as well. The result is a story plot with logical connections and a flow to deliver the message. Let us suppose a user wants to describe the different aspects of the Konark Sun temple. With this topic in mind, they access ”import” images in the home screen. A pop-up provides various options to upload images - from the computer and other sources from the cloud like Google Photos. The UI provides options to drag and drop as well as select images from a folder. In 4.3, selected images in the narrative of Konark temple are present in the bottom of the pop-up. A provision to add and remove images are also available to the user.

35 Figure 4.3 Importing images for narratives

4.2.4 Interaction with narratives

For the purpose of interactivity on the platform, we develop rearrangements, annotations, forums, comments and incentives.

4.2.4.1 Editing

: To make the narrative album, as users import new images, they get added to the beginning of the album. So they are allowed to arrange the objects in an order they deem to be informative and effective. They can also re-arrange objects to make edits ad-hoc. A provision to add, delete and change order of the objects is present in the ”edit album” feature. Bulk sorting and rotation for images are also provided to users. Descriptions are also added to the images to provide further textual information. This can be carried out either simultaneously, i.e writing a description while importing each image or sequentially, i.e writing a description after importing all the images. Editing the descriptions is also possible in multiple iterations for succinct information and cross-validation.

4.4, displays the editing tool-kit provided to the user describing the Konark Sun temple. The provision for changing the order of the images, image editing tools and sorting options are seen.

36 Figure 4.4 Re-arrangement of images in narrative

4.2.4.2 Forums

The platform is developed with a forum for discussions and comments regarding the narratives. Any viewer of the platform can sign up to access this functionality. It enables the author-viewer communication to enrich the information. It supports ideation, development of narrative and an audience for discussions. The forum is enabled with polls for collecting statistical information and threads for topic- speciﬁc discussions.

In 4.5, we see the topics of discussion such ”Favorite aspects of the trip”. The responses from this thread helped with additional details of the trip that went uncaptured in the narratives generated by the users. The poll on narrative perspectives helped understand the narrative structures inherent in the collections perceived by the audience. An example is the mythological stories during the performance of Gotipua as presented to them. Incentives through likes from the visitors of the platform also help us identify the popular narratives in the collection.

37 Figure 4.5 Forums for discussions

4.2.4.3 Annotations

The narrators use textual annotations and tags for each of the images, and the albums. This helps us identify dominant themes and relations between tags.

In 4.6, we observe the dominant tags in a sub-collection of the dataset. The tag with maximum count is Pattachitra and art, followed by Krishna. This communicates that Krishna is a popular icon in Odishi art.

38 Figure 4.6 Tags in a sub-collection

4.3 Results and Analysis

The narratives that emerged can be classified into three types, based on their structure. We notice the sequence of images that enables a logical transition and flow to the story, with an effective plot. In this case sequences are order in time, cause-effect relationships, patterns and derived insights. Based on this, the narratives are classified as present :

• Within an object

• Within a collection

• Thematic narratives

39 4.3.1 Within the object

An object can contain a sequence among its surrogates, and we identify narratives within the object itself. Instances of this type of narrative from our datasets occurs in the Pattachitra style painting. In this type of painting, an entire traditional or folklore narrative is painted on a single canvas. In 4.7, we see an example of the Chitralekha, a Pattachitra style painting, where episodes in the narrative are distinguished by columns and curves, like a comic strip or graphic novel. The digital surrogates in the repository are individual panels from this larger painting. Arranging these digital surrogates in a sequence displays the narrative present within the object.

Figure 4.7 Pattachitra painting of Chitralekha

4.3.1.1 Pattachitra using sequential-art concepts

We use Will Eiseners [19] definition of comics, as that of sequential art to analyze the Pattachitra paintings from our database. He describes art which qualifies as sequential art if the visual imagery has a deliberate sequence that is juxtaposed spatially. As audience, we understand a lapse in time by this sequence of images placed next to each other, even though all the images are available to us at all times. Art Spiegelman utilizes simplified diagrams to depict condensed thought structures. He describes it as time units of past and present available and decoded simultaneously. Calvi (2001) describes an analytic montage of comics in which a single event could be depicted by a single frame or multiple grids. [13] Pattachitra paintings follow a similar representation to sequences. So, we determine the narrative functions by drawing parallels between Pattachitra paintings and the standard narrative method used in graphic literature. Scott McCloud defines a vocabulary for comics [43] to understand the translation

40 of images to a story. He refers to visual literacy in comics as the interplay between visual and literal elements to convey concepts. We refer to this vocabulary, while analyzing the various components of the Pattachitra painting. For demonstration purposes, we pick the Chitralekha painting above.

Icons : Any image that is used to represent a person, place, thing or idea is known as an icon; and pictures are images that contain icons to resemble their subjects. This could vary from being extremely realistic to very abstract in deﬁning concepts. We look at two images from our repository to analyze icons in 4.8.

Figure 4.8 Analyzing icons across collections - Nataraja

In the 4.8, we recognize the deity depicted to be Nataraja, the Lord of dance. It is a form of Lord Shiva performing the cosmic dance, Tandava. The characteristic of this pose is a bent legs, with one foot raised perpendicularly over the other. The hands are placed such that the right shows Abhaya Mudra, and the left points to the raised foot. This icon is demonstrated in various places across different art- mediums in the data set. In 4.9, we identify the traditional icons that represent Lord Shiva and Lord Krishna. Lord Shiva is traditionally depicted as blue, with a snake around his neck and long loose hair. Lord Krishna is depicted as bejeweled in blue, with a Shank (sea shell) and Chakra (discus). In the painting, the section which depicts Lord Shiva is highlighted with black double-border and Lord Krishna is highlighted with a white border. Similarly, we identify Banasura as having many hands, and Brahma as having three heads.

41 Figure 4.9 Identifying icons in Chitralekha painting

Interactions : The audience is introduced to spatial perspective in the panels of the painting. The depth, location and setting are some of the things represented in the background that enable us to understand what is being conveyed in the painting.

42 Figure 4.10 Studying spatial perspectives in Chitralekha Pattachitra painting

For example in 4.10, during the war, the arrows in air and their direction denote the camps of Krishna and Shiva. The panels also reveal the objects and placement of them around the characters in the paintings. For example, when Usha describes the man who appears in her dreams to her friend Chitralekha, the latter is seen to hold a painting for veriﬁcation to Usha. These panels are highlighted with a white border.

Closure : McCloud introduced concept called closure to denote the role of the reader in understanding events that may have transpired between panels of a comic. It can also be defined as the process of observing the parts but perceiving the whole [43]. Here, the imagination of the audience fills the gaps, as depicted by the gutters between panels and understanding it as a whole. A gutter is defined as the space between panels marked by curves and edges.

We notice that scenes are depicted as stills and the happenings that change between panels are left to the imagination of the audience. In 4.11, we observe the panels 20-23. Panel 20 shows the occurance of war, 21 and 22 show resolution of conflict by the intervention of Lord Brahma, and panel 23 shows the wedding of Usha and Aniruddha. We assume a temporal gap which is filled with preparations of the wedding and approval of Banasura, between panels 22 and 23. This is to say that the mind fills the gaps between between panels 22 and 23 which isn’t depicted in the painting.

43 Figure 4.11 Tracing panel path in Chitralekha painting

Time frames : We study the progression of the story in the painting in terms of time. The current panel represents the ”now” in a time series. However, contrary to comic strips where we follow wither a left-to-right or top-to-down method to study it, there is a variation in the studying the time progression in Pattachitra paintings. For example, we follow a spiral toward a center as the time progresses, as depicted in 4.11. Using panels 17 and 20 as examples for contrast, the change of color in the background occurs due to a change of place (Sonitpur to Vrindavan) or even time if the traditional forms of yellow/blue to show day/night are considered. We also see notice less temporal distance between panels 7 to 13,

44 where Aniruddha and Usha fall in love. Whereas, the temporal gap between panels 20 to 23 would span months. These temporal distances are also assumed by the audience. We contrast engaging different sensory aspects of a comic - such as word bubbles and sound effects to motion in time. The fundamental difference in Pattachitra paintings is that there are no words to aid the scenes or explain them. During the construction of our narratives however, we have a description legend with individual aspects of the painting. Setting the scene : By setting the scene within each panel, the surroundings also invoke a response within the audience. We acquire information about the artists style in the sense of consistency in the characters, embellishments, design, inking. This dictates various aspects of storytelling, context and mood. For example, Aniruddha and Krishna are depicted by the same shade of blue, while Banasura and Usha are depicted by a shade of yellow. This gives the audience a sense of commonality between the characters. Similarly, the characters are shown to wear the same attire throughout the painting. The panels also set a tone for emotional and mental states of the characters through the facial expressions, interactions among characters and subtle body language. For example, in panel 15 of 4.11 where Banasura ﬁnds out about Usha and Aniruddha, he gets enraged and this is displayed by huge eyes pushed closer together. Like in popular graphic literature, the Pattachitra paintings are a sequence of panels. This sequence of images are spatially present for grasping at once, and is up to the audience to cognise. The theme also remains persistent in order to maintain a continuity across the gutter.

4.3.2 Within the collection

Contrary to sequential art where all panels are always available to the audience, the sequences in a ﬁlm occupy the same space but change over time. We explore performing arts to ascertain a structure to the narrative in a Gotipua performance and a Rath Yatra procession. As per Aristotle, the classic narrative structure is said to have three basic components of a beginning, middle and end. [33] Aristo- tle’s Poetics talks about the six basic elements of drama as plot, character, theme, dialogue, sound and spectacle. Field [20] describes a setup, confrontation and resolution. Some other four-fold structures are also explored with the middle as divided in two. Cutting describes the four-fold structure of setup, com- plication, development and climax, with optimal subunits as epilog and prolog. [17] Various popular structures also explore pentalogies such as Gustav Freytag, who talks about introduction, rising action, climax, falling action and denouement. [24]

4.3.2.1 Analyzing Gotipua using micro and macro structures

We adopt similar methods to analyze a traditional Gotipua performance as containing acts to form a narrative as well. There are two kinds of narratives in a Gotipua performance. A macro narrative occurs in terms of what is trying to be conveyed in the performance. A micro narrative is a set of actions

45 that comprise the narrative. [5] We utilize Aristotle’s elements of Drama [33] to analyze the micro and macro narratives of a Gotipua performance. Theme : The theme of most Gotipua performances center around epics glorifying Radha-Krishna and other mythological ﬁgures. The dancers assume the form of Gopikas and perform elements of Tandava and Lasya dances to depict these stories. Plot : A Gotipua dance performance typically constitutes three acts - Mangalacharan, Abhinaya and Bhanda Nritya. In Mangalacharan, the artists perform Vandana as a prayer to mother Earth, to the teacher (guru) and to the audience as gratitude. In Abhinaya, they assume Sakhi-bhaav rupa (as lovers of Krishna) and perform the Pallavi in love songs of Radha-Krishna like Gita Govida, or other epics like Mahishasura Mardini. The Pallavi is a single line from a composition and a performance is based around variations on that line. In Bhanda Nritya, acrobatic yoga poses are performed similar to iconography and visual representations in that of Pattachitra paintings, as seen previously in 4.12.

Figure 4.12 Acts in a Gotipua performance

The various poses in a sequence of the dance behave as iconic stills within a narrative. When images of these stills are represented as digital surrogates and arranged in sequence within a collection, a (macro)narrative is generated. It follows the structure of exposition, rising action, climax, falling action and resolution - as per the narrative structure described by Gustav Freytag [24]. In 4.13, among images in a Gotipua collection, we arrange the surrogates from the Abhinaya into a sequence to form a narrative. In 4.12, we categorize stills from the Gotipua dance into Vandana, Abhinaya and Bhanda Nrutya. This shows the composition percentage of each of the acts in a performance.

46 Figure 4.13 Scenes in a Gotipua performance

The micronarrative is very nuanced in the pallavi section of a Gotipua dance, which is a performance of a popular traditional narrative. We take the 4th still from this sequence for further exploration in the following ﬁgure 4.14. We identify that the macro-narrative within these stills displays the Mahishasura Mardini epic. We identify the micro-narrative structures as actions within it. For example, the performer on the right is seen as the narrator of the epic. His raised left hand near the mouth denotes that he is trying to say it aloud to the audience. The central ﬁgure is shown to hold the hair on the head of the smaller boy. This depicts Kali Ma holding the head of Mahishasura. These iconic stills thereby contain the micronarratives within the actions that are being depicted.

Figure 4.14 A still of Mahishasura Mardini epic

47 Character : All the dancers assume the role of Gopikas. However, as seen above in 4.14, the per- formers hold stills to depict icons in a scene. The dancers assume certain characters in that particular scene and these roles keep changing ﬂuidly between scenes. As there are many scenes within an act, different dancers assume different characters throughout an act. Language : The various acts are performed to songs that are composed in Sanskrit and Odiya. The vocalists are separate from the dancers in the performance, however even the dancers sing certain during certain parts of the Abhinaya. The dancer that assumes the role of the narrator, typically also accompa- nies the vocalist. Music : The musical instruments in a performance consists of a mrudala (percussion instrument), harmonium, gini and vocalists. They are accompanied by violin, bansuri (wind instrument) as well. Spectacle : The dance is performed in any room or stage called a Rangmanch, which is typically circular. The artists bow down to give thanks to this stage before a performance. It can be in an enclosed room or outdoors. The dancers wear a bright costume called Kanchula, which is heavily embroidered and feminine. The dancers are necessarily male, however as they are assuming the role of Gopikas, dress in feminine attire. They adorn jewelry, bindis, alta on their palms and feet, ﬂowers in their hair, Kajal to their eyes and sandalwood on their face. They also wear anklets with bells that enhance the rhythm provided by the musical instruments. This alankaran is necessary and considered sacred as the Gopikas are said to dress up for Lord Krishna.

4.3.2.2 Analyzing processions using Aristotlean Theory of drama

A similar analysis is also carried out for the Rath Yatra procession of the Puri jagannath temple in Odisha. The term Rath Yatra means a piblic procession in a chariot. We draw parallels between elements of a drama as proposed by Aristotle and the elements that comprise the fete. Here, it appears to be an interactive drama as the audience comprise the participants of the play. Theme : According to traditional folklore, a Rath Yatra is carried out as a war declared by the deities against evil forces. The deities are carried on chariots pulled by the inhabitants of the city, which is declared a war-zone. The end of the Rath yatra is declared as a victory of good over evil. Plot : It is an annal festival that contitutes a procession on the second day of the Ashadha month. It commemorates the annual visit of Jagannath, Balabhadra and Subhadra to Gundicha Temple near Balagandi Chaka, Puri. The chariots are taken out in a procession and the deities remain at the Gundicha Temple for nine days. Krishna and Balarama partake in the festival with the representative images of the five main Shiva temples and are called Pancha Pandava. This refers to the five brothers who fought the Mahabharata war. After this, the deities take a ritual bath in the temple with sandalwood, flowers and scents. This is conveyed as a procedure for cleanliness and healing from war wounds. Then the deities return to the main temple. This return journey is called the Bhauda Jatra. On their way back, the deities stop at Mausi Maa temple to pay a visit to their aunt. The deities are then restored in the main temple and decked with jewels and worshipped.

48 Character : The deities of the Puri temple, Jagannath, Balabhadra and Subhadra are carried on the chariots pulled by devotees. The priests are the liaisons between them. They play their parts in the narrative of war against evil. Language : The various rituals that take place consist of chanting Slokas that are usually in Sanskrit or Odiya. However, the public that is a part of the procession is also accounted for and a multitude of languages are encompassed toward this effort. Music : The huge processions accompanying the chariots sing devotional songs, play tambourines, drums, gini and a multitude of other instruments. These people comprise professionals as well as com- moners. Spectacle : The deities are carried in the procession in chariots called Nandighosha, Taladhwaja and Darpalana. The chariots have ﬂags and are heavily ornate. Red, black, yellow and blue are dominant colors for the costumes and ornaments of the chariots. Krishna is the one with yelow robes, Balarama adorns red and blue clothes and Subhadra adornes black. The rangmanch in this case is the streets of the city where the procession takes place.

4.3.3 Thematic narratives

When gathering objects externally in the repository phase, identifying appropriate contexts help build narratives even among different collections. According to Braun and Clarke, thematic analysis is used for identifying, analyzing and reporting patterns within data [10]. Thematic narratives are observed to contain sub-text encapsulating meaning, subtle themes and goals. Tomashevsky describes themes as containing sub-themes or motifs. [40] Hargood et al utilize structuralism in narratology to consist of narrative atoms (natoms), features and motif themes. In this system natoms come together to promote certain themes. [26] Hayes describes thematic analysis as a comprehensive processes that identiﬁes cross-references between data in evolving themes. [29] Especially when confronted with highly interpretive cultural data, qualitative research through thematic analysis helps to narrow down the scope for building arguments around the data. Analyzing the elements of such narratives help us develop themes and richer narratives. It illustrates the data in great detail and deals with diverse subjects via interpretations. [9]

4.3.3.1 Qualities in thematic narratives

Narrative voice : In our data-sets, the kind of stories depend on the narrative voice as being either descriptive or explanatory. Descriptive narratives provide objective descriptions as their content, while explanatory narratives unravel and evaluate the subject in question.

• Personal : The ﬁrst is personal voice, in which case the author talks about experiences with a deﬁnitive sense of self that is relevant to the story. An example of this type of narrative in our data set occurs when the researchers describe the visiting the various temples of Odisha. The personal experiences and thoughts are recorded provide insight.

49 • Omniscient : The second type is omniscient, in which case the author tells the story from an all-encompassing point of view for the story in hand. For example, while describing the Buddha statues at the Shanti stupa, the mudras of the statues are icons that are all pervasive in meaning across space and time.

• Sequential : In the third kind, where the narrator tells the story with knowledge that unfolds with the characters involved. While describing the story of Chitralekha in the Pattachitra painting, the plot progresses as we analyze the painting panel-by-panel.

Characters : For the narratives in our collection, the characters are the cultural objects. Based on the narrative voice, these objects are described in three ways. The ﬁrst is what the narrator says about the object, the second is an object talking about another object, third is an object talking about itself and fourth is what the object does. In the ﬁrst kind of character revelation, there is a certain perception and analysis that is carried out. Here, we have to remember that the narrator could be different from the author of the narrative. Whereas, in the remaining kinds of revelations, the explanations and interpretations are available to the audience.

Figure 4.15 Explanation of Rock edicts of Ashoka

For example in 4.15, there is an explanation of the Shanti stupas in one of the digital objects in the collection. Another example is during a Gotipua performance. In a digital surrogate of video form, the narrator of the performance constitutes the digital object itself, so we derive the information from the object itself. Setting : In our data collections, the geographical locations and time are an important aspect for providing a setting. This is of two types.

• The ﬁrst is integral - which provides details speciﬁc to occurrences at a particular space and time.

• The second is a backdrop setting which is universal and timeless in the narrative.

50 For instance, when documenting the lifestyle of the artists in Raghurajpur, we visit various collections to gather evidence towards the narrative in this context. The village setting remains constant to explain the everyday life of the artists. However, in another narrative that describes the trip of the research group through the various temples of Orissa, the setting changes from location to location.

4.3.3.2 Identifying themes

On these narratives, we generate tags hierarchically to identify themes in the data-sets. Opler (1945) describes themes as visible in the expressions of data, culturally agreed on and subtler themes, and systems that contain inter-related themes. [47] When an object is studied, a themes can be developed. [11] Interview questions are also a frame for developing themes. [15] The act of discovering themes is also known as qualitative analysis. [6] We explore techniques to identify themes that arise from the data and the apriori understanding of the participants as illustrated by Ryan et al. [53] Repetitions : Repetitions are identifying certain themes that reoccur. The frequency of certain themes also helps us gain insights about the collection. An example from our collection with the recurrence of ”Jagannath” tag in the collections. As Jagannath is an important cultural icon in Odisha, and especially Puri, it is a dominant theme in art as well. Indegenious references : Another way to identify themes is the occurrences of unfamiliar objects or terms. [49] An example of this seen in 4.16 is the Vyala. It is a mythical ﬁgure often portrayed as part lion, part elephant and part lion.

Figure 4.16 Vyala identiﬁed as theme upon repetition

Metaphors : Representation of thoughts, behaviors, and experiences with analogies and metaphors are also seen in data-sets to identify themes. [37] An example from out data-sets occurs in the visit to the Shanti Stupas. The different mudras of the Buddha are icons and representations of peace. Transitions : Shifts in content also contribute to themes. This is seen in the narration of the performance of the Gotipua performance by the artists in Raghurajpur and contributes to themes.

51 Theory-related material : Themes that characterize the experience of the informants who are interested in understanding the qualitative data illuminates questions also give rise to themes. [54] An example of this from out data-set are the experiences of our researchers visiting the temples in Odisha. Similarities and differences : Noting similarities and differences between objects in the data-sets through constant comparison gives rise to themes. The various representations of icons of mythology, in the Pattachitra paintings and Gotipua dance is an example.

52 Chapter 5

Visualizations

Narratives, being complex cross-relational data, require effective methods of visualization and presentation to discover patterns and gain further insights. Some systems that focus on narrative visualizations, like ThemeRiver [28], provide a temporal view of changes in a plot. Other systems, like In-Spire [36], use clustering techniques to help identify the themes at a single point in time. StoryFlow [41], an optimization strategy for fast generation of narrative visualizations requires pruning and user interfer- ence for developing constraints. Archive-it allows users to visualize collections of web resources from the Internet Archive. [48] All these tools deal with narratives and their structure however, we identify a requirement for a narrative in the context of a cultural object. We develop Narrativium to address this issue. We develop an API for Narratium to generate visualizations from the gathered narrative information. The rhetoric for the Narrativium visualization tool-kit is to identify the richness of a cultural object through their appearance in narratives. We determine heuristics for what could constitute richness, and identify three kinds of narratives with the metrics that contribute to their interpretability are:

• Narrative within an object - information retention, character co-occurrence

• Narrative within a collection - connectedness, property occurrence

• Thematic narrative generated by users - popularity of themes, thematic clusters

In lieu of this, design choices led us to represent the data as JSON documents and visualize it in D3 [7]. D3 was chosen for implementation due to its ease of plugin in web applications, versatility in handling varied data formats and extensive visualization libraries. The narrative data contains unique identiﬁers of the surrogates and its respective meta-data, the narratives they appear in, textual descriptions and annotations provided by the author. Based on the kind of narratives that exist within the objects, different perspectives are developed to visualize them effectively.

53 5.1 For within-object narratives

5.1.1 Identifying conﬂict points studying character co-occurrence

To identify conﬂict points within a narrative, we depict character co-occurrence from panels of Patta- chitra paintings in a temporal sequence. We identify orientations of characters as protagonist, antagonist and neutral. We attempt to determine conﬂicts by mapping co-occurrence of the different orientations in a scene. The panels in our visualization are arranged left to right to depict a temporal progression, like in the narrative of Pattachitra paintings and other sequential art.

5.1.1.1 Design choice

Color : In order to distinguish between different orientations of the characters (protagonist, antagonist or neutral) in plot progression, we depict a positive character in blue, a negative character in red and a neutral character in black. This distinction enables us to recognize points for conﬂict of interest in the progress of the story. It helps us identify interaction points and add weight to certain panels.

Interactivity : The visualization is interactive as hover over a panel displays the portion of the painting corresponding to it. Clicking on a panel reveals the part of the narrative that it belongs to.

Implementation : For this visualization, we gathered data to be represented in respective JSON documents for each of the narratives. They comprise two arrays each. The ﬁrst array stores character information with their unique identiﬁcation, names, and orientations. The second is for scene information, which stores arrays of character ids present in every scene. On the X axis, different panels of the painting are listed with the co-occurrence of the characters. On the Y axis, the characters of the Pattachitra story are listed and the colors indicate their orientations.

54 Figure 5.1 Visualization for character co-occurance in scenes

5.1.1.2 Analysis

This visualization helps us recognize points for conflict in the progress of the story. A conflict point is defined as clash of interest between characters, usually protagonist and antagonist. In 5.1, we observe the first instance of conflict in scene 6 when Banasura objects to Usha’s request to marry Aniruddha. Similarly, by analyzing scenes with protagonist and antagonist occurrences, we can determine the various points of conflict in a narrative.

We map narrative structures in the plot as character development, rising action, climax and resolution. [24] In this particular instance, 7 panels belong to character development, 9 panels belong to raising action, 4 panels belong to climax and 3 panels belong to resolution of the narrative. By distinguishing narratives in this manner, we identify conﬂict points along the course of a storyline. By analyzing our data-set of 298 records, we recognize that 55-60 percent of conﬂict points occur in the climax, and action that leads to the climax in the narrative.

55 5.1.2 Analysing transitions

Theory : Utilizing McCloud’s theories in Understanding Comics, we depict transition of time between panels to study progression of a story. [43] In Pattachitra paintings, we trace the progression of a story in terms of the changes that occur between the panels. This transition is of six types :

1. Moment to Moment : in which subsequent panels display a transition between a moment or a few moments

2. Action to Action : in which subsequent panels display a sequence of major actions performed by a single subject

3. Subject to subject : in which subsequent panels focus on a single still but various levels of in- volvement within it

4. Scene to scene : in which subsequent panels display transitions across signiﬁcant space and time

5. Aspect to aspect : in which subsequent panels display various aspects of space at a particular instant in time

6. Non-sequitter : in which the different panels seem to have no relationship

5.1.2.1 Design choice

On the X axis, we depict the transitions within a Pattachitra painting and on the Y axis, we depict percentage ratio of their occurrence within it. Implementation : For this visualization, we gathered data to be represented in respective CSV documents for each of the narratives. They comprise transition keys and percentage ratios as values of a dictionary.

Figure 5.2 Visualization for analysing transitions in Pattachitra paintings

5.1.2.2 Analysis

When we conduct the analysis on set of paintings, we ﬁnd that most of the paintings have either action to action or scene transitions. We also understand that within a painting, there is no tempo-

56 ral consistency - it expands and contracts arbitrarily. For example, 5.2 displays the transitions of the Chitralekha painting and Ramleela painting. In both of these paintings, we observe that the temporal gap between panels are not constant. In the instance of Chitralekha, there is a uniform distribution of the painting among all temporal transitions, with a maximum percentage in scene-to-scene, followed closely by moment-to-moment and then action-to-action. However, in the instance of Ramleela, a major percentage of the painting comprises of action-to-action transition followed by scene-to-scene. This denotes that there can be, and often are temporal inconsistencies between panels. We also observe a thematic pattern beyond space and time in the case of non-sequitter transitions. This means that panels within the painting are seemingly disjoint. In the ”Dashavatar” painting, and ”Chausat Kala” painting, the panels seem to be unrelated, but the theme is unveiled through an external fact database. The Chausat kala painting describes 64 arts mastered by Lord Jagannath. But as seen in 5.3, it appears that different people are performing the various arts in each of the panels and underlying theme is not immediately evident.

Figure 5.3 Non-sequitter analysis of Chausat Kala painting

5.2 For within-collection narratives

5.2.1 Using narrative graphs

To depict the direction and flow of the narratives generated, we developed an interactive graph as seen in 5.4. A graph is chosen for representation as the understanding nodes and relationships between nodes are intuitive. Each node is an individual digital object and the link between nodes reflect the relationships between them. We utilize the unique identifier and position of an object in the narrative to display the nodes.

57 5.2.1.1 Design choice

Color : In order to distinguish between different narratives, each one is depicted by a single color. For a node appearing in many narratives, it retains the node color of its first appearance. However, the link emerging from it follows the color of the narrative Interactivity : The visualization is interactive as clicking on a node sequence reveals the associated narrative literature in a narrative explorer on the right. Hover over the node displays the unique identifier of the object. Implementation : For this visualization, we gathered data to be represented in JSON with two arrays. The first array is nodes, which stores object information in a dictionary format. The unique identifier information is stored as key id, and the narrative that it first occurs in is represented by group. The second array is links, which stores relational information between nodes. The source points to the start of the narrative, target points to the next node and corresponding topic represents the narrative id. The distinction between group and topic information is necessary as it handles occurrence of nodes in multiple narratives.

Figure 5.4 Graph visualization of narratives

5.2.1.2 Analysis

Let

• Nx be the xth narrative in a collection

• Oxy be an object contained in xth narrative in y position

For example, O12 is an object that exists in N1 in the second position

58 • Rxy be a relation between nodes (y and y+1) in xth narrative

For example, O31 and O32 are connected by R31

• T shows nodes connected thematically O1y and O2z are any nodes from N1 and N2 respectively. They will be connected thematically if they have the same tags. We identify that while T is an equivalence relation, R is not.

We build a graph from the generated narratives based on the objects from the repository, their position in the narratives and with their thematic annotations. Using this visualization, we can analyze frequency of a narrative, length of the narrative, separation between similar narratives based on the deviation of nodes, gauge proximity of different surrogates and identify objects of interest based on the connectedness of a node.

Frequency of narratives, calculated by Count(Nx) is used to determine the number of times a se-

quence of nodes occur. The length of the narrative, calculated by Sum(Ox), is used to select the narrative

with the maximum number of nodes. Connectedness, calculated by Count(Ry) determines the number of intersections of a narrative at a particular node. Further, a thematic analysis based on k most popular

tags is also implemented. Given k most popular tags, Count(k) in Nx is used to determine the occurrence of popular themes in a narrative. Frequency of co-occurrence of the k popular tags in a narrative is also used. A richly-told narrative is said to be one that is extensive in content, with simple or complex relations

and narrative value. We postulate that the richness of a narrative is Dy+Count(k), as the content is validated by the popular tag occurrence. Based on tags associated with objects within a narrative, we try to identify co-occurrence and the distance between the tags within it, known as proximity. From our data sets, we derive that the longest narrative describes the different art forms in Raghu- rajpur. The most connected narrative is the exploration of all the temples and heritage sites in Odisha. This is also a richly told narrative. The various nodes within this narrative diverge to describe different aspects of each temple, as individual narratives. We validate this by checking tag associations. Tags such as ”Jagannath”, ”Shiva” and ”temples” are the 3rd, 5th and 9th most popular which occur extensively in this narrative. The deviation in a storyline identiﬁes digressions from the narrative including various sub-plots.

Assuming Nx is identiﬁed as the primary narrative, for a particular narrative Ny that is similar to it,

deviation is derived by counting the number of nodes in Ny that is not in Nx. Let us represent this as Dy. Let us consider the description of various mudras of the Buddha installations at Dauligiri. We com- pare it with another narrative about the researchers’ visit to Dauligiri and Khandagiri caves. By studying the diversion of the graph from the former narrative, we can ﬁlter out information about the view from the Shanti stupa. We also include crucial information about the Ashoka edicts and importance of Dhauli as a Buddhist site. In this manner, by studying similar narratives, we can prune out invaluable information and include details that may have been overlooked.

59 5.2.2 Information retention through narrative grammar

Theory : We try to quantify information retention by parsing narratives using narrative grammars. Based on the observations by Mandler et al. [42], grammars of narratives are divided into two types of nodes. The ﬁrst type is terminal nodes, which are represented by States and Events. They may be both internal and external. For example, an internal state could be a happy state of mind, an external state could be a state of chaos in a classroom. Similarly, an internal event could be a realization and an external event could be graduation ceremony. The second type is connection nodes between terminal nodes, are made by rules And, Then and Cause.

• And represents a temporal overlap between the temporal nodes

• Then shows a temporal order and

• Cause denotes that Node 1 provides a reason for Node 2 as necessary and sufﬁcient.

From Fig 5.5, we ﬁnd the various representations of the terminal nodes and connection nodes.

Figure 5.5 Representation of terminal nodes and connection nodes

Data preparation : We identified the narrative grammar in terms of nodes on a written script of the Origin of Gotipua. Further, nodes were identified for a dance performance that was based on this script. On a third level, the audience to this performance was asked to re-narrate the performance and the grammar was identified for this as well. We study the amount of information retained by representing narrative scripts in terms of terminal and connection nodes. For example, we parse the following sentence : ”There is intense dialogue between two friends. On inquiry, the depressed friend tells the reason for his depression”. This sentence is marked as: ”There is intense (S1) dialogue (E1) between the friends. On inquiry (E2), the depressed (S2) friend tells (E3) the reason for his depression”. In this manner, an act of the script is distinguished in terms of nodes. We observe the representations in the performance of this script and represent nodes in this medium as well. The audience is asked to re-narrate the information from the performance and the responses are recorded and nodes are identified again. By comparing the number of nodes between the script-writing and re-narration process, we are able to study information retention.

60 5.2.2.1 Design choice

Visualization of the three sources enable us to predict information-loss through the various stages of re-narration and the nature of this information. Based on the fundamental structure of narrative grammar, we denote the story with a trend plot. A trend plot is chosen for clarity in comparison for the information captured and to identify patterns within a narrative. 5.5 depicts the narrative grammar in terminal and connected nodes as authored by the playwright, 5.6 depicts the dance performance of the script and depicts a sample of the re-narration of the audience to the performance.

Color and text encoding : In all the charts, the X axis is represented by terminal nodes (events as

Ex in orange nodes and states as Sx in green nodes). The subscript x represents the occurrence of the terminal node in the narrative. The Y axis contains the connection nodes with and at 0, then at 1 and

cause at 2. For example, if E1 precedes S1, and S1 is at 2 on the Y axis, it denotes that E1 is the reason

for S1 and so forth. Implementation : For this visualization, the data was represented in respective CSV documents for each of the scenarios, which comprises of two arrays. The ﬁrst array is terminal id, which stores information about the terminal nodes and their occurrence. The array connection stores the information about the connection rules between the nodes.

A function automates calling the information from the three CSV ﬁles to populate three SVG elements representing the grammar from narration, performance and re-narration sources. Based on the x and y co-ordinates, nodes are drawn with a uniform radius of 5 units as per standards of the D3 library. The path is drawn by tracing a line in the distance between adjacent co-ordinates.

Figure 5.6 Narrative grammar representation in narration layer

Figure 5.7 Narrative grammar representation in performance layer

61 Figure 5.8 Narrative grammar representation in re-narration layer

5.2.2.2 Analysis

We analyze the occurrences of the terminal nodes and the connection nodes in the above three cases to derive insights. Only about 40 percent of the information was retained from the narration phase to the re-narration phase in our example. We also note that the ”And” and ”Cause” nodes are consistently reproduced at around 48 percent and 13 percent in all three cases. However, the ”Then” nodes are more prominent in the performance case.

Figure 5.9 Narrative grammar representation in re-narration layer

5.3 For thematic narratives

5.3.1 Popularity of themes

We aim to identify popular tags within our data-sets for extracting dominant themes. During creation of narratives, users were prompted to add tags and annotations to the digital objects. We have repre-

62 sented the tags categorically using a bubble chart in 5.10. A bubble chart was chosen as it represents the relative size of the tag and accommodates multiple categories and subcategories, as opposed to a bar-chart or pie diagram. Major themes on a repository and collection level are identiﬁed and we can determine the usefulness of the ﬁltered data based on the themes.

5.3.1.1 Design choice

Color choice : In the bubble chart, the bubbles represent annotations and the size is determined by the frequency of the tag. The colors of the bubbles represent the categories of the annotations, and the hues of the bubbles represent the subcategories. Interactivity : The bubble chart is also interactive, and upon clicking pulls up all the annotated surrogate objects and hover over a bubble displays sub-themes and number objects within it. Implementation : Scraping annotations from the digital objects, we generate a CSV of the tags and their frequency in occurrence. We have information relating to id, which stores the tag name and the number of objects that the tag belongs to. This follows a hierarchical structure as do the themes. For example, the different art forms in Orissa is a tag, and the tags Gotipua which is a dance form, and Pattachitra painting which is an art technique are sub-classes under this.

Figure 5.10 Bubble chart for popular themes

63 5.3.1.2 Analysis

In the above bubble chart, we analyze the tags on the objects of a narrative. Here, the major Hindu mythological characters are depicted by green, the art forms are epicted by orange and the colors are depicted by purple. This color separation shows the heirarchical categories in a theme, and the labels display the theme of the bubble. The size of the bubble corresponds to the popularity of the tag. We understand that black, white, red and yellow are dominant colors, Jagannath and Krishna are important deities and Pattachitra and Gotipua are popular tags. Upon further investogation, we identify that Ja- gannath is popularly depicted in yellow as he is known as Pitambara. Balabhadra is depicted in red and blue, and Subhadra is popularly depicted in red and black for Shakti.

5.3.2 Geoclustering

We aim to locate density of cultural heritage sites in Odisha by mapping them as clusters on a map layout. These clusters correspond to points of interest based on proximity. The geographical data was a part of the documentation on sites visited by researchers on their trip to Orissa, India.

5.3.2.1 Design choice

Color choice : Larger clusters are displayed in yellow, and upon zoom as the clusters detach, the smaller clusters are displayed in blue. The flags of interest points are pins in red. Hover over the pins reveals the names of the sites. Interactivity : Zooming into the clusters, we find either further thematic clusters or a marker displaying the site. With every zoom value, the clusters detach to reveal more information. Implementation : We utilized the Google Maps API for the layout. To do so, we first generate an API key that allows to validate server requests on a remote repository. We position the center on the tuple with latitude and longitude values at India in zoom level 3. For generate clusters, we list the lat, long values of the points of interest.

Figure 5.11 Example of geocluster from data-set

64 Figure 5.12 Zoomed-in geoclusters

5.3.2.2 Analysis

As some of the narratives were travelogues, we mapped the points of interests visited by the researchers as clusters on the map. This not only helped formulate efﬁcient itineraries, but also helped us understand geographically separated themes. For example, the temples closer to Puri, as seen from A and B ﬂags in 5.10 are Vaishnavaite or Jagannath temples while the temples near Bhubaneshwar depicted by a blue cluster are Shaivite temples.

65 Chapter 6

Conclusions

6.1 Research and contributions

While recording cultural memory in terms of digital cultural objects, they get isolated from their natural environment. Identifying interactions of the cultural object in its surroundings, and encapsulating details through this process is difficult. We are confronted with challenges such as representation of some complex multi-dimensional cultural objects like processions, temple eco-systems etc, which have not been explored before. To address this issue, we develop Curarium with an aim to create a pedagogical environment for knowledge preservation and promote hybrid knowledge through interactions. Our data-sets contain information pertaining to Pattachitra paintings, Gotipua performances, sustainable crafts, museum objects and processions. To accommodate the different kinds of data, we develop a meta-data structure that would encapsulate all the relevant information. We verify this meta-data structure with the Dublic Core standard. We perform a comparitive analysis between meta-data structures like processions and performance art, and in-situ and ex-situ art-objects. To observe the data-collection within the Curarium environment, we import the JSON files of each of the digital objects. Each object has a record page with associated digital surrogates and meta-data information. We are able to zoom into the canvas of an object and publish articles about insights regarding an object. This helps us extend a formal curation process to a cultural object that is not present within a museum setting. Curarium provides a novel approach to examine information within a cultural object. By making suggestions on existing meta-data structures and adding annotations, we are able to enrich typological and topological cultural information. By making sub-collections and representing data as visualizations, we enable creative explorations and a succinct manner of presentation for large cultural data-sets. For example, from our data-set we create a sub-collection of digital objects containing Krishna from Pat- tachitra paintings, Gotipua performances, iconography on temple walls etc. This helps us analyze the significance of Lord Krishna in the Jagannath culture within our collection. We are also able to isolate narratives within objects by developing image annotations for Pattachitra paintings. We perform a visual

66 analysis of the datasets using the treemaps and colormaps to identify hierarchies and extract dominant colors from the paintings and sustainable crafts from our corpus. However, we recognize a gap in cross-referencing objects within its natural environment. For example, we are unable to capture all details of performance art like a Gotipua performance, a marriage ritual or a Rath Yatra procession. In lieu of this, we conduct a user-study where the participants paraphrase narratives around cultural objects to capture information. To generate narratives and perform analysis on them, we build a platform called Narratavium. We recognize that narratives from cultural information are of three types - within-object, within-collection, and user generated thematic narratives. By applying sequential art methods, Aristotlean narrative theories and qualitative analysis on narratives, we are able to gain more information about the data-sets that are not captured within a meta-data structure or curatorial platform. While we recognize plot-lines within art objects, within collections and in the process of identifying themes, we account for the human experience in interacting with these cultural objects in their natural environment. By visualizing the narrative information, we are able to:

• Quantify the richness of an object in particular narrative

• Conﬂict points by studying character co-occurrence in scenes

• Understand the transitions in a story plot

• Identify popular cultural themes

• Study information retention in the process of re-narration

This helps us present the multi-dimensional aspects of information in a cultural object in a succinct manner.

6.2 Future work

This thesis can build towards research in many directions. Utilizing Curarium, we can import large- scale cultural collections in varied digital formats. This helps us explore various kinds of digital surrogates and ideate for hybrid knowledge among a plethora of objects like i-frames, video games, meals etc. We can also explore usability features and tool-kits to expand the functionality of the platform to aid pedagogy. With a more extensive database of cultural objects, we can develop more methods of cultural investigation. For representing this information, we can utilize hypergraph ontology. [] Each hypernode in this graph can contain graphs of information in themselves. An example from our data-collection that can be visualized as a hypergraph is a collection of Pattachitra paintings. Each painting is a node in this graph, and upon delving deeper reveals information about the Pattachitra as a graph in itself. The hypernodes represent the relations between the nodes in the graph.

67 By enabling crowd-sourcing for generating narratives, we open avenues to enrich the database with new information and supplement existing ones. This also provides a platform for preservation of in- digenous information and oral narratives. We can also explore aiding the narrative research pertaining to cultural objects in different languages to capture details. The possibilities for research and avenues for impact in the cultural domain are limitless.

68 Bibliography

[1] Curarium. https://curarium.com/, 2017. [2] Curarium satellite. http://mlhplayground.org/curarium_sat/, 2017. [3] K. J. Applegate. Collective access: An essential tool. Illinois Research and Scholarship Open Access, 2016. [4] M. v. Atten. Brouwer meets Husserl: On the Phenomenology of Choice Sequences. Springer Science & Business Media, 2006. Google-Books-ID: kAw TbpcZkQC. [5] A. Beck. Cultural Work: Understanding the Cultural Industries. Routledge, 2005. Google-Books-ID: OLuAAgAAQBAJ. [6] B. Berelson. Content analysis in communication research. Foundations of communications research. Free Press, 1952. [7] M. Bostock, V. Ogievetsky, and J. Heer. D3 data-driven documents. IEEE Transactions on Visualization and Computer Graphics, 17(12):2301–2309, Dec. 2011. [8] P. Bourcier, N. Taskforce, and H. Dunn. Nomenclature 4.0 for Museum Cataloging: Robert G. Chenhall’s System for Classifying Cultural Objects. Rowman & Littleﬁeld, 2015. Google-Books-ID: xBH0rQEACAAJ. [9] R. Boyatzis. Transforming Qualitative Information: Thematic Analysis and Code Development. SAGE Publications, 1998. [10] V. Braun and V. Clarke. Using thematic analysis in psychology. Qualitative research in psychology, 3(2):77– 101, 2006. [11] M. Bulmer. Concepts in the analysis of qualitative data. The Sociological Review, 27(4):651–677, 1979. [12] S. P. Callahan, J. Freire, E. Santos, C. E. Scheidegger, C. T. Silva, and H. T. Vo. VisTrails: visualization meets data management. In Proceedings of the 2006 ACM SIGMOD international conference on Manage- ment of data, pages 745–747. ACM, 2006. [13] L. Calvi. Hypertext and comics: Towards an aesthetics of hypertext. In Proceedings of the 12th ACM Conference on Hypertext and Hypermedia, HYPERTEXT ’01, pages 135–137, New York, NY, USA, 2001. ACM. [14] T. . E. G. . I. L. S. S. . Z. L. Cheng, Eric ; Craft. Pachyderm by new media consortium, san francisco, 2014. [15] A. Coffey and P. Atkinson. Making Sense of Qualitative Data: Complementary Research Strategies. And Social Thought. SAGE Publications, 1996.

69 [16] N. Cohn. The architecture of visual narrative comprehension: the interaction of narrative structure and page layout in understanding comics. Front Psychol, 5, 2014. [17] J. E. Cutting. Narrative theory and the dynamics of popular movies. Psychonomic Bulletin & Review, 23(6):1713–1743, Dec 2016. [18] S. Dudley, A. J. Barnes, J. Binnie, J. Petrov, and J. Walklate. The Thing about Museums: Objects and Experience, Representation and Contestation. Routledge, 2011. Google-Books-ID: w8GoAgAAQBAJ. [19] W. Eisner. Comics & Sequential Art. Poorhouse Press, 1990. [20] S. Field. Screenplay: The Foundations of Screenwriting. A Delta book. Delta Trade Paperbacks, 2005. [21] M. Flaxman. Using the virtual terrain project to plan real cities: Alternative futures for hangzhou, china. In ACM SIGGRAPH 2002 Conference Abstracts and Applications, SIGGRAPH ’02, pages 275–275, New York, NY, USA, 2002. ACM. [22] S. Gratzl, A. Lex, N. Gehlenborg, N. Cosgrove, and M. Streit. From visual exploration to storytelling and back again. In Computer Graphics Forum, volume 35, pages 491–500. Wiley Online Library, 2016. [23] C. Guldi, Jo ; Johnson-Roberson. Paper machines, 2017. [24] F. Gustav. Die Technik des Dramas. Im Anhang: Wilhelm Dilthey, Die Technik des Dramas. Wis- senschaftliche Buchgesellschaft, Darmstadt, 1982. [25] M.-J. K. Han, S. Bair, and J. Lee. Creating metadata best practices for contentdm users. In International Conference on Dublin Core and Metadata Applications, pages 74–78, 2010. [26] C. Hargood, D. E. Millard, and M. J. Weal. A thematic approach to emerging narrative structure. In Proceedings of the hypertext 2008 workshop on Collaboration and collective intelligence, pages 41–45. ACM, 2008. [27] p. Harping. Categories for the description of works of art. Paul Getty Trust and College Art Association, 1996. [28] S. Havre, B. Hetzler, and L. Nowell. Themeriver: Visualizing theme changes over time. In Information Visualization, 2000. InfoVis 2000. IEEE Symposium on, pages 115–123. IEEE, 2000. [29] N. Hayes. Doing qualitative analysis in psychology. Psychology Press, 2013. [30] K. Hill. Museums and Biographies: Stories, Objects, Identities. Boydell & Brewer Ltd, 2016. Google- Books-ID: gBYABQAAQBAJ. [31] J. Hullman and N. Diakopoulos. Visualization rhetoric: Framing effects in narrative visualization. IEEE transactions on visualization and computer graphics, 17(12):2231–2240, 2011. [32] D. C. M. Initiative. Dublin core metadata registry. Research Center for Knowledge Communities at the University Of Tsukuba, Japan, 2014. [33] A. Kenny. Poetics. Oxford World’s Classics. OUP Oxford, 2013. [34] G. R. Kress and T. V. Leeuwen. Reading Images: The Grammar of Visual Design. Routledge, 2006. Google-Books-ID: wprZmJFXUXIC.

70 [35] B. C. Kwon, F. Stoffel, D. Jckle, B. Lee, and D. Keim. VisJockey: Enriching data stories through or- chestrated interactive visualization. In Poster Compendium of the Computation+ Journalism Symposium, volume 3, 2014. [36] P. N. N. Laboratory. In-spire, 2014. [37] G. Lakoff and M. Johnson. Metaphors We Live By. Phoenix books. University of Chicago Press, 1980. [38] J.-P. Lang. Redmine, 2014. [39] B. Lee, R. H. Kazi, and G. Smith. SketchStory: Telling more engaging stories with data through freeform sketching. IEEE Transactions on Visualization and Computer Graphics, 19(12):2416–2425, 2013. [40] L. Lemon and M. Reis. Russian Formalist Criticism: Four Essays. Bison book. University of Nebraska Press, 1965. [41] S. Liu, S. Member, Y. Wu, E. Wei, M. Liu, and Y. Liu. Y.: Storyﬂow: Tracking the evolution of stories. Visualization and Computer Graphics, IEEE Transactions on, pages 2436–2445, 2013. [42] J. M. Mandler and N. S. Johnson. Remembrance of things parsed: Story structure and recall. Cognitive psychology, 9(1):111–151, 1977. [43] S. McCloud. Understanding comics: The invisible art. Kitchen Sink Press, 1993. [44] J.-E. . K. R. Meister, J.C. ; Stange. 3 dh, 2018. [45] M. G.-E. J. J. Meister, J.C.; Petris. Catma 5.0 - software for text annotation and analysis, 2016. [46] L. of Congress. Thesaurus for graphic materials - about the thesaurus for graphic materials, 2010. [47] M. E. Opler. Themes as dynamic forces in culture. American Journal of Sociology, 51(3):198–206, 1945. [48] K. Padia, Y. AlNoamany, and M. C. Weigle. Visualizing digital collections at archive-it. In Proceedings of the 12th ACM/IEEE-CS joint conference on Digital Libraries, pages 15–18. ACM, 2012. [49] M. Q. Patton. Qualitative evaluation and research methods. SAGE Publications, inc, 1990. [50] M. Portela. Multimodal editing and archival performance: A diagrammatic essay on transcoding experimental literature. Digital Humanities Quarterly, 8(1), 2014. [51] J. D. Prown. Mind in matter: An introduction to material culture theory and method. Winterthur portfolio, 17(1):1–19, 1982. [52] B. Rudolff. Intangibleand tangibleheritage. A Topology of Culture in Context of Faith, 2006. [53] G. W. Ryan and H. R. Bernard. Techniques to identify themes. Field methods, 15(1):85–109, 2003. [54] M. A. Rynkiewich and J. P. Spradley. Ethics and anthropology: Dilemmas in ﬁeldwork. R.E. Krieger Pub. Co., 1976. [55] T. Scheinfeldt. Omeka: open source web publishing for research, collections and exhibitions. Open Source Business Resource, 2008. [56] E. Segel and J. Heer. Narrative visualization: Telling stories with data. IEEE transactions on visualization and computer graphics, 16(6):1139–1148, 2010. [57] G. Sinclair, Stfan; Rockwell. Voyant tools, 2017.

71 [58] M. Van Atten. Brouwer meets Husserl: on the phenomenology of choice sequences, volume 335. Springer Science & Business Media, 2006. [59] S. Young. Cultural memory studies: An international and interdisciplinary handbook, volume 8. Walter de Gruyter, 2010.

72 Datasets

• http://ceh.iiit.ac.in/Odisha Trip/

• https://drive.google.com/drive/folders/0B8J326YJbz0UVHdIcW50VDBTR3M

73 Related Publications

• Suresh, Priyanka and Schnapp, Jeffrey. ”Presenting Curarium and geo-temporal visualizations.” in Beautiful Data, metalab(at)Harvard, Harvard Graduate School of Design and Getty Foundation, Cambridge, MA. 2014.

• Suresh, Priyanka and Magliozzi, Cristoforo. ColdStorage webdoc for ”The Library Beyond the Book” by Schnapp, J. T and Battles, Matthew. Harvard University Press Cambridge, MA 2014 ; presented at Harvard Graduate School of Design, Cambridge MA 2014.

• Suresh, Priyanka and Singh, Navjyoti. ”Collaboration workﬂows in Digital Archives and Collec- tions.” In Digital Humanities Centres : Experiences and Perspectives, University of Warsaw, ss 08, 2016.

• Suresh, Priyanka, and Singh, Navjyoti. ”Meta-Data and Methodology: Standards in the Digital Archive.” In CDH@ TLT, pp. 77-82. 2017.

• Suresh, Priyanka and Singh, Navjyoti. ”Vizception : Visualizing narratives to bridge gaps.” Visu- alization for the Digital Humanities, IEEE Viz, 2017. [Weak Reject]

• Suresh, Priyanka and Singh, Navjyoti. ”Narrativium : cultural data represented as narrativess.” ACM Journal on Computing and Cultural Heritage (JOCCH), Volume 11, Issue 4, 2018. [Sub- mitted]