Int J Digit Libr DOI 10.1007/s00799-015-0140-8 The impact of JavaScript on archivability Justin F. Brunelle · Mat Kelly · Michele C. Weigle · Michael L. Nelson Received: 7 November 2013 / Revised: 12 January 2015 / Accepted: 14 January 2015 © Springer-Verlag Berlin Heidelberg 2015 Abstract As web technologies evolve, web archivists work 12.0 % from 2005 to 2012. We also show that JavaScript to adapt so that digital history is preserved. Recent advances is responsible for 33.2 % more missing resources in 2012 in web technologies have introduced client-side executed than in 2005. This shows that JavaScript is responsible for scripts (Ajax) that, for example, load data without a change an increasing proportion of the embedded resources unsuc- in top level Universal Resource Identifier (URI) or require cessfully loaded by mementos. JavaScript is also responsible user interaction (e.g., content loading via Ajax when the for 52.7 % of all missing embedded resources in our study. page has scrolled). These advances have made automat- ing methods for capturing web pages more difficult. In an Keywords Web architecture · Web archiving · effort to understand why mementos (archived versions of Digital preservation live resources) in today’s archives vary in completeness and sometimes pull content from the live web, we present a study of web resources and archival tools. We used a collection of 1 Introduction URIs shared over Twitter and a collection of URIs curated by Archive-It in our investigation. We created local archived ver- How well can we archive the web? This is a question that sions of the URIs from the Twitter and Archive-It sets using is becoming increasingly important and more difficult to WebCite, wget, and the Heritrix crawler. We found that only answer. Additionally, this question has significant impact on 4.2 % of the Twitter collection is perfectly archived by all of web users [40,43] and commercial and government compli- these tools, while 34.2 % of the Archive-It collection is per- ance [52,53,66]. fectly archived. After studying the quality of these mementos, The web has gone through a gradient of changes fueled by we identified the practice of loading resources via JavaScript increasing user demand for interactivity. Early websites were (Ajax) as the source of archival difficulty. Further, we show relatively static, while continued adoption of web technolo- that resources are increasing their use of JavaScript to load gies has made the pages personalized and more interactive. embedded resources. By 2012, over half (54.5 %) of pages JavaScript, which executes on the client, provides additional use JavaScript to load embedded resources. The number of features for the web user, enabling or increasing interactivity, embedded resources loaded via JavaScript has increased by client-side state changes, and personalized representations. These additional features offer an enhanced browsing expe- J. F. Brunelle (B) · M. Kelly · M. C. Weigle · M. L. Nelson rience for the user. Department of Computer Science, Old Dominion University, Norfolk, VA 23529, USA JavaScript has enabled a wide-scale migration from web e-mail: [email protected] pages to web applications. This migration continued with the M. Kelly introduction of Ajax (first introduced in 2005 [28]), which e-mail: [email protected] combined multiple technologies to give web pages the abil- M. C. Weigle ity to perform asynchronous client–server interactions after e-mail: [email protected] the HTML is loaded. The first wide-scale implementation of M. L. Nelson Ajax was in Google Maps in 2005, but Ajax was officially e-mail: [email protected] added as a standard in 2006 [70]. While archival tools per- 123 J. F. Brunelle et al. (a) (b) (c) Fig. 1 Google Maps as it exists live and as a memento a live version, b archived Dec. 1, 2012, c archived Apr. 30, 2012 formed satisfactorily when the web consisted almost entirely We discuss how these technologies impact archivability in of static resources, archival tools (e.g., Heritrix [51,65], Sect. 9. Many culturally significant artifacts (such as Google WebCite [22], and the wget application [29]) cannot capture Maps, shown in Fig. 1) are unable to be unarchived. Even the more complex, feature-rich pages making up today’s web. if resources are archived, they sometimes become tempo- The ease of archiving a web page (henceforth the archiv- rally inconsistent because JavaScript may load content from ability) is impacted by the migration from web pages to web the live web (referred to as the live web “leaking” into the applications. Over the years the web has shifted from a web archive and discussed in more detail in Sect. 6.4) instead of of documents to a web of applications, and these applica- only loading temporally appropriate content. tions require execution by the client to render what the user Throughout this paper we use Memento Framework ter- experiences. Client-side rendering has not historically been minology. Memento [68] is a framework that standardizes part of the archival process. Even if it were, the applica- Web archive access and terminology. Original (or live web) tions depend on client-side inputs that increase the variability resources are identified by URI-R, and archived versions of between what is archived and what the user experienced [60]. URI-Rs are called mementos and are identified by URI-M. Because of the advancement and increasing adoption of Memento TimeMaps are machine-readable lists of memen- web technologies, the web is composed of an increasing pro- tos (at the level of single-archives or aggregation-of-archives) portion of personalized resources that are difficult to archive. sorted by archival date. 123 The impact of Javascript on archivability This paper makes two contributions. The first contribution refers to the final representation that is not fully realized is a study of live resources and mementos to provide insight and constructed until after the client-side representation is into what makes a resource archivable. We studied two col- rendered; the representation’s final state is deferred until after lections of resources and measured them for archivability. client-side code and events have finished executing. Heritrix, WebCite, and wget are all used to capture a set of As we have mentioned, web browsers use an engine to resources shared via Twitter and a set of Archive-It curated run client-side JavaScript. Web crawlers often do not pos- resources with the intent to determine how well each tool sess such an engine and, therefore, do not have the ability archives the resources. These tools, collections, and result- to execute JavaScript during the archival process. In other ing mementos are compared, and the results are discussed in words, archival crawlers are not equipped with the features depth to show that Twitter resources are less archivable than necessary to execute JavaScript. If Ajax requests additional Archive-It resources, and Heritrix is the best archival tool resources to be embedded in the representation, crawlers will of the three observed. In culmination, the first contribution be unaware of the extra resources required to properly ren- provides an analysis of how well we can archive the web. der the representation and, ultimately, will not archive these The second contribution is a study of how archivability has embedded resources. changed over time. We study the mementos of the URI-Rs Because most web crawlers do not have the ability to exe- in our collections for archival quality and cause of missing cute embedded JavaScript or other client-side technologies, embedded resources. We show that resources are becoming the resulting mementos may only be partially operational or less archivable over time due to the increasing prevalence of incomplete. We discuss crawlers’ current capabilities further JavaScript. In short, the second contribution provides insight in Sect. 4. into how well archival tools have performed in the past. Google Maps is an example of a resource with a deferred representation and where the lack of completeness of memen- tos is easy to observe. Figure 1a shows the live page, con- 2 Deferred representations taining multiple interactive UI elements. The map in the mid- dle is draggable, allowing the user to pan. An archived ver- According to the web architecture [10,23,32], servers deliver sion (from April 30, 2012) of this page in Fig. 1c is missing web resources to requesting clients such as a web browser UI elements and functionality (circled), and the interaction (controlled by a user, such as Firefox [24]) or a web crawler (e.g., panning and zooming) does not function. This is due (e.g., Heritrix or other robot). The content delivered to the to resources that would be loaded when the user clicks, but client by the server is the representation of the web resource that are not preserved by the crawler. Figure 1b shows an (e.g., HTML, PDF). The representation may also include a archived (December 1, 2012) version that gives the façade client-side programming language such as JavaScript which of functionality when, in fact, resources on the live web are will execute when the representation is rendered by a web being loaded. browser and the client-side code is run by a local compiler Figure 2 outlines the role Ajax plays in loading a resource or—in the case of JavaScript [25]—an engine. like Google Maps. When a crawler fetches a web page (1), Executing a technology like JavaScript on the client can it waits for the page and all of the embedded resources to potentially cause the representation to change with or without load to consider it fully rendered. At this point, the crawler subsequent requests to a server for additional resources.