The Following Essay First Appeared on the BBC Domesday Project National Disc, First Published November 1986

The following essay first appeared on the BBC Domesday Project National Disc, first published November 1986.

THE DOMESDAY PROJECT

An Overview of the Production of the Domesday Discs By Peter Armstrong. Editor, BBC Domesday Project , 1986

A NEW AND UNIQUE MEDIUM The idea for a 20th Century Domesday Book, produced in a form that would make it widely available, was only possible because of recent developments in optical storage. This technology was developed in the 1970s and depends on a laser beam reading millions of ‘pits’ representing pictures, sound and data in the spiral track of a plastic disc. It is the density of this data storage and the fact that discs can be pressed rather than recorded that has opened up the new possibilities of electronic publishing.

In particular, the interactive video disc in its most advanced form offers to anyone interested in the exploration and communication of ideas through words and pictures a medium that is cl ose to ideal. It can store film, sound, still photographs, text, data and computer software on the same surf ace and can be cheaply replicated. And, in spite of its huge volume of 110,000 still images and 648 megabytes of data per disc, it can be accessed quickly, so that users can move straight to the section t hey want without being burdened by the sheer mass of material at their disposal. This is made possible by the combination of replay hardware, indexes and retrieval software that the new generation of interactive video discs brings together. How should one think of this new medium? In some ways it is l ike an electronic book, with screen pages that can be arranged into chapters, an index and a table of contents. But most books have a beginning, a middle, and an end, whereas the disc offers random access: you can start at any point and move t o any other. Does this make it more like an encyclopaedia? In a way - though this suggests fixed sections related to topics, while the video disc can put the same material into an almost infinite series of patterns in response to different enquiries. Seen in this way, therefore, it is closer to a data base with pictures, or a set of television programme elements, put together by the disc user, taking the place of the television producer. And then again, interactive video can be seen as operating in the same way as a teacher or expert, training or informing the person using the disc. My own preference is to see the Domesday Discs as a kind of exhibition - an electronic exhibition of British Life in the 1980s. This exhibition tries to reflect many of the aspects of life today in just the same way that, for example, the Great Exhibition of Victorian Britain or the Festival of Britain of 1951 did for their ti mes with traditional means. Like them it is highly selective: it does not aim at the comprehensiveness of an encyclopaedia. It offers not general truths but insights and images that we hope are characteri stic of our era. And the visitor to this electronic exhi bition can wander freely around, focusing on those elements that cat ch the eye.

Who is the exhibition for? The advantage of an electronic exhibition is that it can claim to be for everyone. We hope the Domesday Discs will be of real value today particularly to the educational community, as a way of taking stock of the state of the country under many headings. But if our exhibition stands the test of time, it can equally be visited in the coming centuries by those who want to look back at what we thought, how we looked, what we did in the 1980s, and also - from what we deci ded to include in the exhibition - what we thought important. It was this idea of the creation of a multi -media record of our times for use in the future that was the primary motivation in the creation of the Domesday Discs.

Who has compiled the exhibition? The BBC has taken the lead, regarding video disc production of this type as an important extension of its public service broadcasting role, and as one of the first steps into the fully interactive television and information systems of the 21st Century. It is appropriate for a broadcasting authority to become involved since we combine editorial, journalistic, and technical skills with the resources of film and picture libraries and a long tradition of popular education and information. So the creation of the discs has been centered on a BBC project team about fort y strong at any one time.

But we saw Domesday as a national undertaking - it could not reflect a particular view of Britain. So a system of advisory commit tees was set up (described in section 3). Around these we built up concentric rings of contributors - ranging from full time university teams, through hundreds of voluntary area organizers and contributors, to the million or so people carrying out the surveys on the ground.

Our aim has been to produce what I called at the outset of the project a people’s database - not only something that ordinary people could use, giving them access to much of the data about vital issues that is normally difficult to obtain, but also a database that they themsel ves have helped to create. Our portrait of Britain in the 1980s is largely a self - portrait. The two-year process by which the Domesday Discs have been created has not been a centralised operation by a group of people with strong ideas about British life and how it should be reflected. Our role has been rather to gather and shape all the material offered so enthusiastically by thousands of people - photographers, academics, journalists, government statisticians, teachers and pupils, members of conservation groups and political parties, and many many more. And the process is not over. Our hope is to produce further editions of the Discs incorporating feedback and additional material offered by people using them. Only then will I feel justified in calling it a people’s database.

HOW THE PROJECT AROSE

The first idea was for a television series presented by Michael Wood to mark the 900th anniversary of the Domesday Book with a set of historical essays on Britain from 1086 to the present day. This in turn led me to propose the creation of the Domesday Discs as a twentieth century equivalent of William the Conqueror’s Domesday Book. It was to be a national project based in the first instance on the network of distributed computer power represented by the tens of thousands of micro computers in schools. I spent a week in May 1984 thinking the idea through and writing a detailed proposal with details of content, team, budget and timetable. This proposal was put to BBC Television management and received whole- hearted support. BBC Enterprises agreed to provide the first one million pounds of funding.

We then took the idea to the Rt. Hon. Kenneth Baker, then Secretary of State at the Department of Trade and Industry, who agreed a further half a million pounds - in furtherance of their aim of promoting interactive video technology in this country. The D. T. I. in turn - in line with government policy - recommended private sector funding, so we approached Philips Electronics for the f inal half a million pounds. They were to become central part ners in the project as it became clear that a completely new video disc player would have to be designed and manufactured in order to make possible the digital content of the Domesday Discs. Our fourth partner was Acorn Computers, manuf acturers of the BBC micro.

With capital equipment and a building loaned from other parts of the BBC, we were ready to begin in the autumn of 1984. That gave us exactly two years to plan and complete the project, including data, picture and text collection, and hardware and software development from scratch. We all knew it was at least a year too little, but the anniversary could hardly be moved. I have been constantly criticised for not coming up with the idea a year earlier! This time pressure has put enormous strain on all of us - particularly for the schools operation. But it has also provided an impetus and a spur, turning deliberation into action faster than is usually the case.

THE PROJECT TEAM AND ASSOCIATED COMMITTEES

The project team has been made up of between 30 and 60 members for the two years of the project. Some were seconded from different and diverse parts of the BBC, while others came in on contract. The team has included an extraordinary range of skills from social science researchers to systems analysts, from video disc producers to copyright specialists. They have worked harder (often at week-ends and through many nights) and with a spirit of personal dedication more striking than on any other project in my personal experience. For the f ull list of team members and associates please refer to the article called ‘Credits and Acknowledgments’. Internally the team reports to a steering committee of BBC senior management. From the outset I was clear that, in spite of the common prejudice agai nst committees, this was a project that would depend crucially on a hierarchy of committees - many of them working on a voluntary basis. The Education Committee advised on the schools’ participation on the project. The Software Committee discussed the ret rieval soft ware specification. The Technical Liaison Committee covered all aspects of the recording and replay hardware. The Consultative Committee represented the interests of the major part ners in the undertaking. The Marketing Committee.. . you’ve guessed it .

The Central Committee (choosing my phrase carefully) was the Editorial Board. It consisted of leading practitioners in the relevant fields - social science, geography, education and so on. The Board met monthly in order to resolve the policy issues of what should be put on the discs and how this could best be achieved. It made, for example, the decision on how to split the material between the two discs: the Community Disc with its geographical basis and the topic-based National Disc drawn from more centralised information sources.

EDITORIAL POLICY

It was clear to all of us that balance and objectivity of the kind the BBC demands of its programmes would be equally important for the discs. It could not be a picture of the United Kingdom from a single vi ewpoint. But it was equally clear that this was going to be very difficult to achieve. Although there was room for a mass of material, everything still had to be selected by someone. Moreover, since it is impossible in practice for a user of the disc to get a sense of the complete contents, he or she could find their browsing around a particular topic throwing up a very unbalanced mixture of material - depending on the particular keywords they had used. In fact the keywords themselves often turned out to be more sensitive than the things they referred to. An article about inner city violence, for example, might be balanced in itself, but should it be indexed as ‘riots’, with all that implies, or ‘social deprivation’, with it s alternative view?

These questions of policy were talked through at the monthly meetings of the Editorial Board, and the detailed lists of proposed material under each topic heading arose from these discussions. What is on this first edition of the discs represents our best attempt at a rounded and bal anced, yet colourful and lively, portrait of Britain in the 1980s. We hope that the detailed reaction of those who use the discs will lead to improvements and additions for the next edition.

It is import ant to stress again that our aim was not to produce any form of encylopaedia. The discs do not contain comprehensive information across the whole spectrum of human knowledge! There is next to nothing at all, for example, about astro-physics, the workings of the brain or how to wire a plug. Our subject is strictly ‘British Life in the 1980s’ and we have often had to be very selective in what material to include as characteristic of our times. Nor do we endorse the accuracy of what is on the discs. An article or a dataset carries the authority of the source from which it comes and if it proves erroneous, that fact may itself be of interest in the future. This was how we saw ourselves in the 1980s.

THE SCHOOLS PROJECT

The 14,000 schools that volunteered to take part in the project were the spearhead of community participation that also included hundreds of branches of the Women’s Institute, scores of scout troups and many other voluntary bodies throughout the country. We saw their contribution as vital to reflect a picture of how people themselves saw the state of the nation. They were given only the most general guidance on what they could write about. We wanted their priorities, their way of putting things and, for Welsh and Gaelic speakers, their national language to come through. For this reason contributions were not edited, except for legal considerations. Even spelling was not corrected, so as not to impose any central orthodoxy and to preserve a body of vernacular text that night also be of interest to future scholars.

Inevitably, some contributions were better than others, but overall the quality of the work that was produced and the enthusiasm that was engendered was a great tribute to the British educational system. Many observers from overseas, who are hoping to organise their own national surveys, wonder if such large-scale, voluntary co-operation will prove possible outside the United Kingdom.

The same free choice of material applied to the photographs which each group supplied. However, the survey data was strictly standardized so that the results could be plotted on a countrywide basis. The amenities counted were those recommended by the Editorial Board and include features never previously counted on this scale. The land use survey follows the principles pioneered by Dudley Stamp in the 1930s and Alice Coleman in the 1960s, though we were forced to work to a much faster time-scale and at a coarser resolution. Stamp, for example, took seventeen years rather than one term for hi s survey. Nevertheless, we believe our approach has produced some valuable results, and provides a basis for continuation and extension. The most serious limitation is that the local coverage is patchy. My original plan saw 12,000 schools taking part and each of them covering two of our 4 by 3 kilometre blocks. This would have covered the 23,000 blocks which make up the United Kingdom. However it soon became clear that, in the year of industrial action by teachers, schools could not cover more than one block. Even worse, they were not able to travel far from the school base to survey more distant blocks. This meant that the 14,000 schools often doubled up on near by blocks while distant ones could not be allocated. We tried many ways to supplement this coverage, appealing to tourist centres in the highlands of Scot land and parish councils in the shire counties, but in spite of the best efforts of our team and the 125 local authority co-ordinators who worked so hard to enlist local support, we could not extend coverage beyond 9000 blocks. In practice this means that most of the centres of population are well covered by text and pictures, but it does leave holes in the amenity data maps. This may be something that can he addressed in the second edition of the Domesday Discs.

COLLECTING STILL AND MOVING PICTURES

Video disc i s an ideal medium for the reproduction of colour photography, which is often prohibitively expensive in book form. But although the disc has space for tens of thousands of images, the pictures editor decided on a policy of quality rather than quanti ty. The 22,000 images that her team have brought together must represent one of finest collections of contemporary phot ography ever made available to the public. This in turn was made possible by the generosity of more than one hundred individual photographers and picture agencies who contribut ed photographs to the disc. They are acknowledged in the descriptive text that accompanies their pictures. Since we do not see these discs as encylopaedic, we have not included pictures on every possible topic, but have concentrated on those areas where the visual dimension adds understanding and excitement.

We were also delighted at the response to our photographic competition. We hoped this would be a means of finding pictures that reflected the more personal, quirky and unusual aspects of everyday life that official collections can miss. Our hopes were certainly realised. There was a magnificent public response and we have been able to include more than 5, 000 excellent pictures on the National Disc.

We have arranged all the pictures in 512 picture sets. But this is only one possible way of arranging them.

Additional soft ware can easily create new sets with different emphases by linking individual pictures from different parts of the disc. For reasons of space and time, pictures on the National Disc, unlike those on the Community Disc, are not individually referenced in the main index.

In gathering film and video clips for the disc we hit up against two clear constraints. Moving film quickly eats up disc space and we could not allocate room for more than an hour’s worth. Rather than use this for a lot of tiny clips which we felt might be unsatisfactory to viewers, we decided to create montages of characteristic images for each for the years of the 1980s up to August 1986.

The second restraint was that copyright agreements made it virtually impossible to include musical or dramatic material. We were therefore forced to centre our choice of moving images on the idea of the ‘events of the 80s’, from news to sport, with only a sprinkling of other topics.

THE WALKS

The idea of surrogate travel by comprehensive photographic coverage of a location is unique to the video disc medium. We wanted to include on the disc some characteristic places and buildings, which we believed would quickly gain in historical value if they could be covered in sufficient detail. The location of each walk (with exception of the town centre of Brecon) is not identified, i n order to protect the owners who kindly allowed us to record for posterit y the exact look of their houses.

THE GALLERY

This is a new idea for visually representing the contents of the disc. It allows people who do not have specific enquiries simply to browse. Originally we had intended that it would encompass all the types of material on the disc, but the computer model of the data landscape would have been impossibly large for present technology. Moreover, much of the material (for example, thousands of detailed data sets on unemployment) would have proved irritatingly complex for the casual browser. We therefore decided to concentrate on the visual aspects of the disc content, adding our ‘overview essays’ for completeness. The idea of the Gallery, following leads from the Massachusetts Inst itute of Technology, is to off er users a way of picturi ng the contents of the disc - an i nternal map that can function as an aid to memory. We hope it will also make the di sc more fun for children to use. COLLECTING TEXT

We have taken two approaches to the text we have chosen for the National Disc. Articles have been commissioned from leading authorities on forty-five key aspects of British life, so that the disc user has available an overview of the particular topic. For further detail within each topic, however, we decided to use a scrapbook approach. This means that we have culled textual material from many different kinds of existing published sources - from Fleet Street to Hansard. We hope this serves a double function of reporting ‘facts’ and also showing how they were perceived by dif ferent people using different media in the 1980s. Do not assume that any one piece of text carries an imprimatur - each is a fragment of how this society wrote about itself. Where an issue is controversial, we have aimed to include pieces on both sides of the argument.

Unlike the collection of data, where a significant portion of what exists in certain fields can be put on the discs, it is out of the question to include more than a tiny fraction of the mass of material that is published in Britain even in a single week.

Authors, journalists and publishers were generous in the access they gave us to their material, but since each piece had to be typed and indexed individually, there was a strict limit on what could be included. Our text editor has tried wit hin these constraints to produce a varied and readable, as well as authoritative, collection of text that reflects most of the issues in the forefront of our concern in the l98Os. COLLECTING DATA

Our original idea was to include only data that was already in machine-readable form. Our contacts with the ESRC National Data Archive, for example, had established just how much fascinating data was included in their holdings, to which only academics enjoyed regular access. We then made contact with the Centre f or Urban and Regional Development Studies at Newcastle University, Birkbeck College, University of London and the Institute of Terrestrial Ecology, Natural Environment Research Council, for additional data and expertise on analysis and presentation. A common data format was established, contracts with the copyright owners were concluded and - with prodigious efforts on the part of our university and other colleagues - more than six thousand data sets were assembled on the disc, each containing up to ten thousand cell values.

During the assembly process, however, it became apparent that important data would be missed unless we were prepared to enter by hand other sets which did not exist in machine- readable form. Once again the problem was to know when to st op, but we hope that those aspects of British life that have been interestingly quantified are covered by data sets on the discs. We did not feel any necessity to include data for its own sake on a subject (for example sculpture) that could best be covered in another way. Conversely, economics is covered mostly in data form, with some text and almost no pictures.

SOME STATISTICAL WARNINGS

Data is held on the discs in as raw a form as possible. This leaves all the options for manipulation and interpretation open to the user. We hope this will lead people to enjoy ‘playing’ with the data sets, looking for interesting patterns and comparisons. To this end some quite powerful types of statistical analysis are provided in our software, and others may be added i n the future. This does, however, give rise to the possibility of wrong- headed conclusions, if users are not aware of the limitations of some of these techniques.

In particular we would wish to warn you about three possible traps. First , subdividing a dat a-set may produce such a small sample that no wide-ranging conclusion should be drawn from it. Second, when using rates, ratios or percentages, you must ensure that the numerator is genuinely a sub-set of the denominator. Third, ratios calculated and mapped for a geographical area may not be true for any part of the area. These warnings are explained in detail in the text accompanying the relevant data sets. Some of our critics have expressed fears about making raw data and statistical soft ware available t o untrained members of the public. So our message must be - enjoy experimenting with the data on the discs, but be cautious in the conclusions you draw from it.

MAPS AND REMOTE SENSING IMAGES

Domesday would have been impossible i n its present form without our partnership with Ordance Survey. They have provided us with their latest maps at the 1:625000, 1:50000 and 1:10000 scales. In the case of the Channel Islands no appropriate map was available, so Domesday commissioned a new 1:50000 sheet, which is now in the Ordnance Survey catalogue. The maps were physically cut and pasted together, then photographed in around 25, 000 frames by the computer-controlled video rostrum camera at the BBC Open University Production Centre. Ordnance Survey also provided their gazetteer of 250,000 British place names and 600 aerial photographs. Our satellite imagery on the Community Disc derives ultimately from the American Landsat satellites, via the Royal Aircraft Establishment at Farnborough. Their data tapes were transformed to our specification at the GeoData Unit of Southampton University and the Geography Department of Sheffield University, so that they matched our 40 by 30 kilometre maps. Case studies showing other, more detailed, analytical work on remote sensing imagery are contained on the National Disc.

TECHNICAL PROCESSES

As a new and relatively untried medium, these two video discs have been far from easy to produce. The tens of thousands of frames of stills, film, maps, walks, titles and so on, had each to be produced, then transferred to the correct single frames of video tape. This process was carried out by the BBC Open University Production Centre. It demanded inventiveness, as new techniques were required, meticulous attention to detail, as the numbers moved into six figures, and a great deal of energy and goodwill , as optimistic deadlines were missed and new ideas were introduced.

THE SOFTWARE

Our original intention was that the retrieval software by which you are now reading this article would be written by Acorn Computers Ltd. In the event , their commercial problems in 1985 made this impossible and the work was undertaken by Logica UK, as sub-contractors to the BBC. Taking seven person-years to write, it must represent the largest body of code (some 60, 000 lines) yet written for the BBC micro. The elegance and sophistication of the software derives particularly from the contributions of the Logica team leader, Jardine Barrington- Cook, and Dr Martin Porter of Cambridge, upon whose earlier work on large scal e retrieval systems (for example, Muscat) we have drawn extensively.

At the same tine our BBC software team has been concerned with the massive task of pre- mastering on a Vax 11750 all the digital material (data, text, indexes and software) in the ridicul ously small number of months available to us. The new form of mastering process required by video discs which contain large amounts of digital data has been successfully developed by Philips at its plants in Eindhoven and Blackburn.

The Domesday Discs may be the last to be produced in this way. This is because they are primarily intended as historical records. A comprehensive home and institutional knowledge base kept up to date by datacasting techniques will be a different step, but one that I personally hope that, as a result of the Domesday Project, the BBC and others will want to explore.

In the final analysis, however, what these discs have to offer is not a given quantity of information but the means to manipulate old data into new patterns, and thus to see our life in this country with new eyes. The original Domesday Book, 900 years a go, was designed to collect information; but that information was collected for a particular purpose, to help consolidate the economic and political power of the conqueror. As many historians have pointed out, it laid one of the practical foundations of the new feudalism.

With the new Domesday, by contrast, the intention is not to affirm a centralising power by the further centralisation of information. Instead our hope is that it will help to democratize information. The people help provide and the people can call up information; and, crucially, the people can themselves see whatever patterns in British life become visible when disparate fragments are brought together.

Benjamin Woolley put this very well in his List ener article of 25th April 1985. “Uniquely, the picture will be compiled by the survey’s user, rather than its compiler. The interactive video-disc technology allows the minimum of structuring to be done at compilation stage. On the disc, the survey will just be an apparently randomly ordered series of maps, pictures, pages of text and computer programs. Only when these are brought together by the computer, according t o criteria supplied by the user, will they make sense. The Domesday Project will bring together all the disparate data about the modern economy and society that, properly integrated, will create a completely new and unpredictable picture of the nati on. Who knows what British cult ure will look like constructed from such a mass of disparate data? It could be like a tower in a flat land, offering a view never before imagined or imaginable”.

Well, the tower i s bui lt. We hope you enjoy the view.

DEDICATION

The Domesday team have kindly agreed t o a personal dedication of our joint work. The dedication is to the memory of my father, William, Lord Armstrong of Sanderstead. It is from his lifelong work in the Civil Service that I have learned what ever insights I have brought to the Domesday Project about the importance for our national life of information that is detailed, impartial and freely available.