March 2010 www.keep-project.eu

KEEPING EMULATION ENVIRONMENTS PORTABLE Newsletter Abstract

The very success of computing technology, where machines are rapidly superseded, has created a serious and growing challenge Inside this issue of how to preserve access to digital material originally produced on machines which have since become obsolete. Cultural her- KEEP 1 itage organizations are particularly sensitive to the threat of major data loss resulting from technical obsolescence. KEEP Cultural Heritage 2 (Keeping Emulation Environments Portable) will develop an Em- About portability ulation Access Platform to enable accurate rendering of both static and dynamic digital objects: text, sound, and image files; Games and game developers 6 multimedia documents, websites, databases, videogames etc. The Case for Local Preservation You are now reading the second newsletter produced by partners of KEEP project approximately one year after the project’s in- Events 9 ception. In the first half of 2009, research was undertaken into media carriers and available transfer tools. Furthermore, an in- vestigation was carried out into the potential legal issues arising Long-term preservation of digital objects not only entails secure when copying information from original data carriers, and a storage and management, but also includes the development study about metadata standards and approaches in Europe has and execution of strategies to access and render these objects, been produced. Finally, an investigation was conducted into the now and in the future. Such strategies can roughly be divided state-of-the-art in emulation. By the end of 2009, a greater un- into two groups: migration and emulation. Migration is focused derstanding of existing emulation techniques and their applica- on the digital object itself. Emulation does not focus on the dig- bility as strategies for digital preservation was developed. Also, ital object, but on the hardware and software environment in user requirements analysis was carried out in the National li- which the object is rendered. braries of France, Germany and the Netherlands, and a design has been formulated for creating flexible, user-friendly emula- It aims at (re)creating an environment in which the digital object tion services, media carriers and media transfer tools. can be rendered in its original form. This is done by an emula- tor, a software application that runs on a host computer plat- form and recreates the targeted platform. The advantages of Introduction to KEEP emulation is that the original digital object can be left un- touched, no periodic migration cycles are needed, and func- Keeping Emulation Environments Portable (KEEP) is a medium tionality and appearance of the object are preserved by using its scale research project started on 1 February 2009 co-financed authentic software environment. The KEEP project recognizes by the 7th Framework Programme’s ICT-3-4.3 Digital libraries these issues and presents an integrated solution to all challenges and technology-enhanced learning priority. mentioned above. KEEP address all aspects ranging from safe- guarding the original bits from the carrier to offering online serv- Accessing digital material requires the mediation not just of a ices to end-users via a highly portable emulation framework computer but a computer compatible with the preserved mate- running on any possible device. rial. The very success of computing technology where machines are rapidly superseded has created a serious and growing chal- Project KEEP will develop an Emulation Access Platform to en- lenge of how to preserve access to digital material produced on able the accurate rendering of a broad range of static and dy- obsolete machines. Over time individuals and organisations namic digital object, designed for a wide variety of computer build up digital assets comprising a wide variety of digital objects systems, so that they can be securely accessed in the long term ranging from plain text to multimedia applications and websites. Although primarily aimed at Cultural Heritage institutions, this The central goal of KEEP is to provide new tools for accessing Emulation Access Platform will also serve the needs of a wide any digital object both at present and in the long term. range of organisations and individuals due to its universal ap- proach.

KEEP is co-financed by the European Union’s FP7 Programme. The project is coordinated by the Bibliothèque naionale de France 1 March 2010 www.keep-project.eu Cultural Heritage - About portability

We have to wonder about the conditions required to permit Software portability these virtual environments to run on future hardware or, in Emulators raise the same questions as any software in terms of other words, to study the conditions of emulator portability. portability. As we know, runable files are called “binaries”and re- This is one of the gauntlets taken up by the KEEP Project. sult from the compilation of “” files for a certain en- vironment. Then, two main strategies must be distinguished: by Pierre Echegaray and Jean-Philippe Humblot, Bibliothèque nationale de France ▪ Binaries portability This can be illustrated by the Java language: once a program is generated for the Java (JVM), it can be executed The BnF has been using emulators for many years in order to on any environment which disposes of a JVM implementation simulate computer-based machines for which it possesses pub- such as MS Windows, or Mac OS. Its portability on future lications and ensure their communication. Enabling to create vir- environments relies entirely on the capacity to adapt the JVM to tual environments, emulation appears to be the only solution to them but no future compilation is required. untie the link with original hardware, formerly necessary to run these publications. ▪ Sources portability In this scenario, sources are adapted to take into account several In this article, we explain the general principles of software porta- target environments for compilation. Used in conjunction with bility and show how they were applied for writing 10 emulators compiler facilities, the same program code allows to generate used in our Library. As we will see, these emulators have in com- different binaries for different running platforms, operating sys- mon to be open source, written in C or C++, use standardized tems or processors. input/output (I/O) libraries and can be compiled for several - get environments through an open source compiler: GCC.

KEEP is co-financed by the European Union’s FP7 Programme. The project is coordinated by the Bibliothèque naionale de France 2 March 2010 www.keep-project.eu

The first strategy imposes to focus on the definition of a virtual In the following, we will focus on the steps required and prob- machine (VM): portability will rely mainly on its capacities to be lem that may occur during the process of sources of em- adapted to different target platform. In the second case, porta- ulators on a new GCC target. bility focuses on the compilation’s environment required for em- ulators’ generation. This is the GNU Compiler Collection (or Compilation steps GCC)’s approach. The first action of a generation process with GCC is to run a KEEP uses both strategies: it will define a new VM, both fitted to “configure” script on the host platform. This script is generated facilitate its own portability and emulator binaries execution, and by the “autoconf” configuration scripts that can make decisions will study source portability so as to easily adapt emulator’s based on a canonical name for the system type, or target triplet. source code and the compilation chain to create binaries for this VM. The “configure” script is written in UNIX shell and detects all in- formation about the software components which can be used Compilation tool on the host (library, runtime, processor, ...). It generates some “makefile” scripts and “include” files, and sets some macro di- One of the most famous compilation suite, GCC (GNU Com- rectives (#define) adapted to the environment’s capabilities. All piler Collection) is produced by the GNU Project and supports these scripts and files are used to allow GCC to produce the various programming languages. It is a key component of the final executable files or binaries. GNU toolchain which allows portability of applications running on many different platforms, using many different processors, ex- On the other hand, emulators’ sources code is structured to ecuting on all kind of operating systems: portability is achieved embed specific instructions related to each particular target, in in particular by providing the specificities of the target platform terms of or I/O libraries like DIRECTX or SDL. and adapting (or porting) source codes to take into account these properties. Source portability

According to GCC general philosophy, a target is defined by a Knowing this process, let’s explore the requirements and cost of string called a triplet (though it may be composed of 2, 3 or porting existing emulators on a new target ? even 4 terms) like “i686-pc-mingw32” or “-apple-dar- win8.9.0”. GCC is to be modified within the project so as to In a first approach, if we don’t need to access the real hardware allow generation of binary code for the KEEP virtual processor except the processor, the generated code based on the target and environment which will be defined as a new target. We fore- definition is self sufficient : a program running a simple calculus, see to define a new target “keep-virtual-elf” so that the compiler for instance an addition such as 1+1, should execute in the vir- will generate pure code for the virtual machine. tual environment.

It is important to notice that GCC allows to produce binaries But in the real world, software will need to interact with the user for the environment it is running on (the host platform), or to and will call at least basic I/O functions like “open”, “read”, any other environment defined as a possible target. In this last “write”, “close” on some peripherals like screen, disks and key- case, we talk about “cross-compilation”. board. These functions are generally implemented in C standard libraries and give access to the host system functions. So we need to benefit of an implementation of these libraries on the target host platform.

Analysis of 10 existing emulators

The goal of our experiments was to test the completeness of the compilation process on several existing target environments. Fur- ther experiments would lead to test as many documents as pos- sible on each recompiled emulators, in order to check that they work at least as well as their already-compiled distribution.

KEEP is co-financed by the European Union’s FP7 Programme. The project is coordinated by the Bibliothèque naionale de France 3 March 2010 www.keep-project.eu

Ten emulators have been tested and compiled with GCC 4.4.0 ▪ QEMU, a processor emulator that relies on dynamic binary for Windows running on a XP-Pro machine. They are : translation to achieve a reasonable speed while being easy to port on new host CPU architectures; ▪ Basilisk II, an open source software emulator which emulates ▪ MAME, an emulation application designed to recreate the the 680x0-based Apple computer on a variety of op- hardware of arcade game systems; erating systems; ▪ Stella, an emulator for the Atari 2600 game console, that takes ▪ , a portable x86 and x86-64 IBM PC compatible emu- its name from the console's codename; lator and debugger; ▪ VICE, standing for Versatile Commodore Emulator, an emula- ▪ DOSBox, an emulator which emulates an IBM PC compatible tor for Commodore's 8-bit computers; computer running MS-DOS; ▪ FunzyTo7, an emulator for Thomson MO5 and TO7 comput- ▪ UAE, a computer emulator which emulates the hardware of the ers. Commodore range of computers; ▪ PearPC, an architecture-independent PowerPC platform emu- The following figure summarizes some specificities of these 10 lator capable of running many PowerPC operating systems, in- emulators : cluding Mac OS, Darwin and Linux;

Emulates Coded in Version Using Last Update

Basilisk Apple Mac C, C++ 2006 SDL 2001

Bochs IBM PC C, C++ 2.4.1 SDL 2009

DosBox MS-DOS C, C++ 0.73 SDL 2009

E-UAE Amiga C, C++, ASM 0.8.29 SDL 2007

PearPC Power PC C, C++, ASM 0.4 X11, Win32 2005

Qemu Multi-CPU C only 0.10.0 SDL 2009

MAME Arcade C, C++ Sdlmame0135 SDL 2009

Stella Atari C, C++ 3.0.0 SDL partial 2009

Vice Commodore C, C++ 2.1 Win32 2008

FunzyTo7 Thomson MO5,TO7 C only Win32 2004

The results of our experiments show, as regards global results : Conclusion ▪ all compilations succeeded, with a lot of warnings but without These results, and in particular the last one, mean that, in order fatal error; to generate code for a virtual machine, these kinds of depend- ▪ all emulators start and run correctly but need sometimes some encies have to be satisfied. Given the edge between pure ma- extra components like ROM or bootable disk. chine code and hardware/software environment’s interactions, they suggest to adopt one out of two different strategies : As regards compilation process : ▪ in some cases, a fully defined triplet is needed, in other cases, ▪ using an abstract layer such as the SDL approach. In this case, the script considers the default environment as LINUX; portability relies on porting the library itself; ▪ endianness is often considered; ▪ presence of an arithmetic processor is also taken into account; ▪ producing the final executable requires at least a compliant POSIX library and a graphical library (SDL, X-LIB, GTK, DIRECTX, WINDIB ..) for the target environment.

KEEP is co-financed by the European Union’s FP7 Programme. The project is coordinated by the Bibliothèque naionale de France 4 March 2010 www.keep-project.eu

▪ using an existing standard like LINUX and hooking I/O functions to match the actual host platform. This is what does CYGWIN attempting to emulate much of POSIX on Windows through the cygwin.dll.

It also appears that SDL library has been widely adopted by em- Resources: ulator’s developer community and is an efficient choice to face portability issues. http://www.libsdl.org/ Further works on the KEEP project will explore and determine http://basilisk.cebix.net/ which is the best strategy regarding our challenge and maybe http://bochs.sourceforge.net/ find an original way. http://www.dosbox.com/ http://rcdrummond.net/uae/ http://www.qemu.org/ http://rbelmont.mameworld.info/ http://stella.sourceforge.net/ http://www.viceteam.org/ http://www.sylvain-huet.com/hacks/

KEEP is co-financed by the European Union’s FP7 Programme. The project is coordinated by the Bibliothèque naionale de France 5 March 2010 www.keep-project.eu Games and game developers The Case for Local Software Preservation

Assuming that our culture influences what we write, there is focused on software written for the Sega SC3000 microcom- likely to be rich diversity in the software of different countries puter. The project aimed to address a number of the interrelated and regions. ‘challenges’ facing software preservation, namely: legal, media historical, and technical. In this piece – which is an edited extract by Dr Melanie Swalwell of a longer article profiling the full pilot – I make the case for local software and why it matters, before detailing the media- In researching New Zealand’s digital game history, I have turned historical part of the study, in which I prototyped a method for up a remarkable number of unique game artefacts, both hard gathering information on locally written software. and software. This is digital cultural heritage that in many cases is unique internationally (eg, see Swalwell & Davidson, “Malzak”). Why local software matters

Early games (and other software) are artefacts from the road to our becoming digital. To date, digital game history has usually been told with a focus on North America and Japan. The as- sumption that it was the same everywhere does not hold. In the case of early micro-computers, playing a game or running a pro- gram required users to first type in the program code. This ne- cessity helped many users learn how to program, and they honed their skills and wrote their own software.

Assuming that our culture influences what we write, there is likely to be rich diversity in the software of different countries and re- gions (eg. linguistic; digital versions of local games, etc). Fur- thermore, as I have argued elsewhere (Swalwell 2008), 1980s microcomputers were a technology in search of a use. While there were many claims for what these machines could do for home users, keen programmers invented other uses. Often, these programs would be distributed on magnetic tapes, meaning that It includes a significant quantity of locally written software titles. much locally written software was not published (or only pub- In 2004, when I began this historical research, inquiries revealed lished on a small scale). that, with only a few minor exceptions, no cultural institutions were collecting digital game hardware or software. Since then, things have improved slightly, albeit very slowly. Now, there is at least an evolving discourse about digital heritage, however, it is still the case that no institutions have responsibility for collect- ing and/or archiving early software. The threats to the survival of this material are many, and will be well understood by most read- ers. At present, collectors act as custodians of this material. While their efforts to safeguard it are to be applauded, as hobbyists, their resources for undertaking preservation work are limited.

This situation prompted the assembly of a multidisciplinary team of researchers in 2005, to try to bring our legal, technical, and media-historical expertise to bear on the preservation of these digital artefacts. In 2007, our ‘NZTronix’ team received a small amount of funding for a pilot digital game preservation project,

KEEP is co-financed by the European Union’s FP7 Programme. The project is coordinated by the Bibliothèque naionale de France 6 March 2010 www.keep-project.eu

New Zealand’s software history is remarkable for the numbers of The Database has entries on 47 titles. While this number is mod- people who were involved in writing software (Swalwell 2008), est, I think it a good result from one media appeal (it also gen- and the numbers of locally published titles (Wheeler & David- erated many email contacts). Certainly, it serves to demonstrate son). While software was rarely developed in total isolation, local the usefulness of the method. I have plans to expand and rede- structural conditions kept influences from abroad in check. As ploy the tool in a new project on Australian software history, and well as being geographically remote, trade restrictions further I’d be happy to hear from anyone else who can see a use for the moderated influences from the US and elsewhere, while govern- tool in their projects. ment incentives for the electronics industry supported the de- velopment of local games production. Inquiring into local Reading the entries in the Database helps to assemble a richly conditions of production and reception usefully complexifies ex- textured picture of software writing activity in 1980s and 90s isting, taken-for-granted histories of game and software devel- New Zealand, ranging across commercial, educational, artistic opment, revealing hidden networks of friendships and influence. and hobbyist spheres. This is unusual: many histories of software are written from a single perspective, such as media arts, games, Locally produced software valuably assists in understanding what or business history, and attend only to software in that ‘class’. people did with early computers; often these are programs and uses that we have forgotten. They are important to record and re- For example, a business history such as Campbell-Kelly’s From member if we want to know how computers changed our lives Airline Reservations to Sonic the Hedgehog focuses on compa- and our cultures. Local software tends not to be of interest to nies and titles that became industry ‘leaders’ (and those that larger preservation projects, so if it is to be preserved, local peo- went under in the process), with little on titles that did not ple need to do this. In the absence of dedicated media museums, achieve significant market penetration. The only written account existing cultural institutions are probably best placed to under- of the New Zealand software industry that I have been able to lo- take software preservation work. cate from the period is similarly business oriented.

Early New Zealand Software Database By contrast, the Database includes information on: shareware; ports; titles that were locally oriented or produced in low volume One significant challenge to preserving local software is in know- (eg. the game “Maze”, one of 6-8 games contained on a cas- ing what software was written. Media historical research can pro- sette for the ZX81, written by John Barton of Timaru; and the vide guidance as to what is most historically significant, so that “Scheduling System”, by Alec Utting, originally written for North preservation efforts can be prioritised. To address this gap in Harbour Netball). Titles that were commercially successful in- knowledge, I prototyped a method for collecting information clude “Lockie’s Dispensing Software”, for New Zealand and Aus- from the public, via a web interface. Reasoning that the best way tralian Pharmacies, of which more than $1 million worth was to find out what software was written was to ask people who exported to Australia, and the games published by New knew, I did an interview and appeal on Radio New Zealand Na- Zealand’s Acid Software. tional’s breakfast show to ask people to contribute information to the “Early New Zealand Software Database” (Swalwell 2007). Custom written training software for Medical and Dental stu- dents at the University of Otago is included, as are a variety of The Database consists of a web form, database, search tool and titles which were never published, either because they were hob- backend interface. The site is written in PHP and utilises a MySQL byist creations, written by juniors (eg. “Twist-a-Plot's Friend” database to store information submitted by users. Information is (1986) written by Michael Clark, Form 4, Katikati College), or collected in 9 fields: Title, Year, Author/Developer, Publisher, Sys- designed as conceptual art pieces, such as Stephen McGregor’s tem, Description, Notes, with the two remaining fields requiring “Incorrect Sums” and “Before the Day of Wrath” (c. 1984). In the name and email address of the person contributing the in- these, McGregor seeks to ‘humanise’ the computer by making it formation (these details are not publicly visible). add up incorrectly and read poetry.

Title and Description are the only mandatory substantive fields. The enjoyment and pleasure which people derived from playing This is to encourage those who only know, or can only recall, early digital games makes them attractive targets for preserva- partial details to contribute these. A “Do you know more about tion. Not only does the popularity of games mean that they will this entry?” link invites others to help complete entries. There is have a ready user base for preserved examples, cultural institu- a facility for uploading files and source code, where this is pos- tions doing such work will be well positioned to reach out to sible, and nominating Creative Commons licences for software these (potentially) new audiences. and other assets, again, where appropriate.

KEEP is co-financed by the European Union’s FP7 Programme. The project is coordinated by the Bibliothèque naionale de France 7 March 2010 www.keep-project.eu

(Note: This is an edited extract of a longer article “Towards the Preservation of Local Computer Game Software: Challenges, Strategies, Reflections”, which appeared in Convergence : The International Journal of Research into New Media Tech- nologies, vol. 15, no. 3, 2009: 263-279. The full text is acces- sible from http://con.sagepub.com/)

Works cited

Campbell-Kelly, M. (2003) From Airline Reservations to Sonic the Hedgehog: A history of the software industry, Cambridge, MA: MIT Press. Swalwell, M. and M. Davidson (forthcoming) ‘Malzak’, in M. Bit- tanti and I. Bogost (eds) Ludologica Retro: Volume 1: Vintage Arcade (1971-1984), available online at http://nztronix.org.nz/malzak.php Swalwell, M. (2008) ‘1980s Home Coding: the art of amateur programming’, in Stella Brennan and Susan Ballard (eds), Aotearoa Digital Arts: New Media Reader, pp. 193-201, Auck- land: Clouds/ADA, available http://melanieswalwell.back- packit.com/pub/1418682-1980s-home-coding-the-art-of-amate ur-programming Swalwell, M. (2007) “Early New Zealand Software Database” http://nztronix.org.nz/main.php Wheeler, Aaron and Michael Davidson (updated June 2008) “SC3K Tape Software List”, http://homepages.ihug.co.nz/~atari/SC3K08.html

Biography

A scholar of digital media arts, cultures and histories, Melanie Swalwell is the author of chapters and articles in both traditional and interactive formats, in journals such as Convergence, Vectors, and The Journal of Visual Culture. Her anthology, The Pleasures of Computer Gaming: Essays on Cultural History, Theory and Aesthetics (co-edited with Jason Wilson) was published in 2008 (McFarland). She is a Senior Lecturer in Screen & Media at Flinders University, Australia. Contact: [email protected]

KEEP is co-financed by the European Union’s FP7 Programme. The project is coordinated by the Bibliothèque naionale de France 8 March 2010 www.keep-project.eu Events

what we can do, individually and together, to preserve our per- GDC 2010 sonal and shared collections. http://www.ala.org/ala/mgrps/divs/alcts/confevents/preswk/ind The Game Developers Conference® (GDC) is the world’s largest ex.cfm professionals-only game industry ALA will also organize the annual conference focused on digital event. It is the essential forum for preservation in June 24-30, 2010 in Washington, D. C. learning, inspiration, and networking http://www.ala.org/ala/mgrps/divs/alcts/confevents/upcom- for the creators of computer, console, handheld, mobile, and on- ing/ala/ac10/index.cfm line games. The conference was held on 9 - 13 March 2010 in San Francisco and in Europe will be held on 16 - 18 August in Cologne, Germany as its second year and again in conjunction Paris Game AI Conference with gamescom, Europe's largest computer and video games trade show for consumers and professionals. 2010 http://www.gdconf.com/

The Paris Game AI Conference is the most passionate event 8th European Conference about artificial intelligence in game development, and the largest independent gathering on the topic. In 2009, it brought together around two hundred people from all over the world, including on Digital Archiving programmers, designers, students and enthusiasts. In 2010, by popular demand, the conference is returning for its third year The 8th European Archives Conference will take place in Geneva on June 23rd and 24th. from 28 - 30 April, 2010. Designed to define the current and fu- http://gameaiconf.com/ ture challenges of the profession, the conference is being de- voted to digital archiving as the general theme for this 8th edition. They wished to innovate by focusing on the function of ECDL 2010 archives – the archiving – and no longer on the institution. The European Conference on Research and Advanced Technol- http://www.ica.org/en/2010/04/28/8th-european-conference- ogy for Digital Libraries (ECDL) is the leading European scien- digital-archiving-geneva-2010 tific forum on digital libraries and associated technical, practical, and social issues, bringing together researchers, developers, con- tent providers and users in the field. ECDL 2010, the 14th con- ALA Preservation Week ference in this series, will be organized by the University of Glasgow on 6-10 September 2010. and Annual Conference http://www.ecdl2010.org/

In case you would like to contribute to The American Library Association will launch its first Preservation our next issue of KEEP newsletter, please Week May 9–15, 2010. The week will highlight " collecting and send an email to us at preservation in your community." ALA encourages libraries and [email protected] other institutions to use Preservation Week to connect our com- or send a message through the website munities through events, activities, and resources that highlight www.keep-project.eu

The newsletter is freely usable, KEEP consortium must be notified and credited when contents are used for any public work. No commercial use permitted. Subscription to the newsletter is possible online at www.keep-project.eu Newsletter editor: Iveta Mikolaskova (Cross Czech a.s.) Newsletter reviewer: David Anderson (UPHEC)

KEEP is co-financed by the European Union’s FP7 Programme. The project is coordinated by the Bibliothèque naionale de France 9