Thomas Virgona Doctoral Dissertation: Defense
September 11, 2001:
A Study of the Human Aspects of Disaster Recovery Efforts for Wall Street Financial Services Firms
A Dissertation
Submitted to the Faculty
Of
Long Island University
By
Thomas James Virgona
In partial fulfillment of the requirements for the degree
Of
Doctor of Philosophy in Information Studies
Spring 2008
Thomas Virgona
74 Waverly Avenues
East Rockaway, New York 11518
516-599-2890
This dissertation is dedicated to public use; copying and reprinting are encouraged.
Page: 1 of 237
Thomas Virgona Doctoral Dissertation: Defense
Dedication
To my loving wife of over twenty years, for her unwavering support and confidence. Without Denise, who has been my emotional anchor through not only the challenges of my doctoral studies, but my entire adult life, this achievement would hold no value. To my mother, for instilling the importance of hard work and higher education.
To my late father, who has been my role-model for persistence and personal sacrifices, and who instilled in me the inspiration to set high goals and the confidence to achieve them.
To my children, TJ, Nicole and Joey, who have grown into three of the most special people any parent could wish.
To my colleagues in the program, especially Dan and Chris, who have provided (largely useless) advice and support during the coursework, comprehensive exams and dissertation process.
I wish to thank my committee members who were more than generous with their expertise and precious time. A special thanks to Dr. Hunter, Dr. Knapp, and Dr. Hildreth, for their countless hours of reflecting, reading, encouraging, and most of all patience throughout the entire process. Thank you Dr. Westermann and Dr. Koenig for agreeing to serve on my committee.
Page: 2 of 237
Thomas Virgona Doctoral Dissertation: Defense
Table of Contents 1 INTRODUCTION AND CONTEXT ...... 8 1.1 A SUCCINCT STATEMENT OF THE PROBLEM UNDER INVESTIGATION, INCLUDING ITS IMPORTANCE TO THE DISCIPLINE ...... 8 1.2 THEORETICAL FRAMEWORK ...... 20 2 LITERATURE REVIEW ...... 30 2.1 INFORMATION SYSTEMS TECHNOLOGY – THE BEGINNING ...... 31 2.2 THE GROWTH OF INFORMATION SYSTEMS TECHNOLOGY ...... 33 2.3 CHANGES TO DISASTER PLANNING CAUSED BY THE COLD WAR...... 36 2.4 CHANGES TO DISASTER PLANNING IN RELATION TO OTHER “DISASTERS” ...... 42 2.5 INFORMATION SYSTEMS AND TECHNOLOGY – THEORIES AND METHODOLOGIES...... 48 2.6 THE HUMAN COMPONENT OF INFORMATION SYSTEMS TECHNOLOGY...... 56 2.7 THE RELATIONSHIP OF THE EVENTS OF SEPTEMBER 11, 2001 TO INFORMATION SYSTEMS...... 61 2.8 SCHOLARLY LITERATURE; THE IMPACT OF SEPTEMBER 11, 2001 IN VARIOUS DISCIPLINES...... 64 2.9 SEPTEMBER 11, 2001: INFORMATION TECHNOLOGY AND DISASTER RECOVERY...... 74 3 STUDY DESIGN AND METHODOLOGY...... 89 3.1 RESEARCH QUESTIONS...... 89 3.2 OPERATIONAL DEFINITIONS AND LIMITATIONS...... 94 3.3 A SPECIFIC METHODOLOGY AND JUSTIFICATION...... 97 3.4 SAMPLE DATA-GATHERING IMPLEMENTS ...... 98 3.5 A STATEMENT IDENTIFYING POTENTIAL ANALYTICAL METHODS AND EXPECTED RESULTS ...... 105 3.6 MAPPING OF METHODOLOGICAL TECHNIQUES TO RESEARCH QUESTIONS...... 106 4 RESULTS...... 108 4.1 UNSTRUCTURED INTERVIEWS ...... 108 4.1.1 Senior Technology Manager...... 108 4.1.2 Help Desk Manager ...... 110 4.1.3 Application Manager...... 113 4.1.4 Network Engineer ...... 116 4.1.5 Business User...... 118 4.1.6 Database Administrator ...... 121 4.1.7 Summary...... 123 4.2 DISASTER RECOVERY TEST OBSERVATIONS ...... 124 4.2.1 Summary...... 127 4.3 FOCUS GROUP ...... 128 4.3.1 Summary...... 130 4.4 ARTIFACT ANALYSIS ...... 131 4.4.1 Asset Sales System...... 131 4.4.2 Global Technology Department...... 136 4.4.3 Funds Transfer Application...... 140 4.4.4 Globally Deployed Application...... 140 4.4.5 Loss of a Building ...... 144 4.4.6 Summary...... 150 5 DATA ANALYSIS AND FINDINGS (AND RELATIONSHIP TO PRIOR RESEARCH)...... 152
6 CONCLUSIONS AND RECOMMENDATIONS FOR FURTHER STUDY...... 167
7 REFERENCES...... 172
8 GLOSSARY ...... 202
Page: 3 of 237
Thomas Virgona Doctoral Dissertation: Defense
9 APPENDICES ...... 205 9.1 APPENDIX A: SDLC DELIVERABLES AND APPROVERS...... 205 9.2 APPENDIX B: TYPES OF RESEARCH DESIGN ...... 208 9.3 APPENDIX C: INFORMED CONSENT FORM...... 209 9.4 APPENDIX D: INSTITUTIONAL REVIEW BOARD APPROVAL ...... 210 9.5 APPENDIX E: INTERVIEW SCRIPT...... 212 9.6 APPENDIX F: DEFENSE ACCEPTANCE FORM ...... 214 9.7 APPENDIX G: INDIVIDUAL INTERVIEW TRANSCRIPTS ...... 216
List of Tables
Table 1 – Dynes’ Four Types of Tasks ...... 43 Table 2 - Definition of Terms Used in this Dissertation...... 94 Table 3 - Bergs Four Steps for Conducting a Focus Group...... 99 Table 4 - Methodical Techniques Used to Address Research Question ...... 106 Table 5 - Recovery Strategy...... 138 Table 6 - Loss of staff ...... 139 Table 7 - Assembly Points...... 145
Page: 4 of 237
Thomas Virgona Doctoral Dissertation: Defense
Abstract
Single events have impacted information technology and the general society. The effects of Guttenberg’s printing press were extraordinarily far-reaching: speed, uniformity of texts and relative cheapness (Eisenstein 1979). Another groundbreaking event in the field of information technology was the Soviet’s launching of Sputnik in October 1957. As a result, the American government, educators and general society placed heavy focus on organizing scientific information, increasing science and technology education, and established a national center for scientific and technical information. A more recent phenomenon is the diffusion of computers, and more specifically, the Internet. The Internet is the fabric of our lives, a ubiquitous presence. Information technology is the present-day equivalent of electricity in the industrial era.
This research examined the changes to disaster recovery plans for financial firms located in the Wall Street area since the events of September 11, 2001. The literature indicates that disaster recovery plans usually rely upon human capital and expertise, which has required critical information service providers (e.g., the financial services industry) to reexamine existing contingency plans.1 This dissertation will investigate the role people played in the disaster recovery efforts and subsequent updates to disaster recovery plans to account for these roles and tasks.
Why study disasters at all? There is one main reason: The scholarly study of disasters helps answer important questions as societies try to maintain order in the face of uncertainty
(Robert Stallings in Quarantelli 1998). For this dissertation, non-experimental design was used (exploratory and descriptive). Qualitative analysis is intended to produce an explanation of a phenomenon, particularly an identification of patterns. Specifically, the study included a
Page: 5 of 237
Thomas Virgona Doctoral Dissertation: Defense
focus group, unstructured interviews, and observations of a disaster recovery test and content analysis of disaster recovery plans. The research goal was to uncover what, if anything, we have learned from the events of September 11, 2001.
The devastating loss of life on September 11, 2001 was concentrated in the financial industry. Fatalities in that industry represented over 74 percent of the total civilian2 casualties in the World Trade Center attacks, and one firm, Cantor Fitzgerald, lost 658 employees
(General Accounting Office 2003). Ironically, September 11, 2001 was not the first such attack aimed at Wall Street. On September 16, 1920, a horse-drawn wagon carrying hundreds of pounds of explosives was detonated at the corner of Wall and Broad Streets in lower
Manhattan, killing thirty people and causing the New York Stock Exchange to close. The exchange reopened the next day and banking and financial activity quickly returned to normal
(Brooks 1969). One difference between the incident in 1920 and 2001, aside from the magnitude of the loss of life, is the reliance on technology. Lacker calls this a “technology shock”, a significant damage to operational capability due to either the inoperability of physical capital or the loss of staff (Lacker 2003).
The events of September 11, 2001 literally struck at the heart of America’s financial information center, causing both immediate and long-term adjustments. Information technology professionals on Wall Street on September 11, 2001 were placed in a unique position at the center of a disaster, concerned for family members and asked to recover information system with little or no “status” information. These professionals performed the
1 The penalty of a late information, even only a few minutes late, can be catastrophic to a company (Horton 1985). 2 In this specific context, fire fighters and police are not considered “civilian”. Page: 6 of 237
Thomas Virgona Doctoral Dissertation: Defense
task to the best of their abilities despite enormous distractions. What was learned during this process may be critical for future designers of information systems: dependence on human resources is critical during emergencies, recovery plans may provide little or no value during an actual emergency, assumptions about travel or communications may all be invalid and the dependency on other firms may be essential.
Page: 7 of 237
Thomas Virgona Doctoral Dissertation: Defense
1 Introduction and Context
1.1 A Succinct Statement of the Problem under Investigation, Including its Importance to the Discipline
Information systems and technology have changed as a result of both major revolutionary events and smaller evolutionary adjustments. Scholars have devoted a tremendous amount of focus to three “events” (the printing press, the launch of Sputnik and the Internet) that had dramatic effects on how information technology impacted the general society. In the late 15th century, the reproduction of written materials began to move from the copyist’s desk to the printer’s workshop because of Guttenberg’s invention of the printing press in Mainz, Germany. The effects of Guttenberg’s invention were extraordinarily far- reaching: speed of reproduction, uniformity of texts and relative cheapness (Eisenstein 1979).
The invention of movable type has been discussed as one source that ignited the Protestant
Reformation. There is a tendency to forget the awesome power the church had in the area of information control. The weekly church sermon provided news, real estate transactions and other mundane matters. The Roman Church had moved against Bible-printing and even developed a new form of censorship. Guttenberg’s invention represented a fundamental informational control shift away from the church to other disseminators of information.
Some of the subtle nuances of the introduction of the printing press included the ability for dispersed readers simultaneously to read maps, images, text and diagrams. Books that were known to be banned had a built-in attraction (Eisenstein 1979). Eisenstein noted the new kinds of medical self-help that ensued, as physicians were killing more patients than they saved (Eisenstein 1979)!
Page: 8 of 237
Thomas Virgona Doctoral Dissertation: Defense
Possibly no social revolution in European history is as fundamental as that which saw book learning (previously assigned to older men and monks) gradually become the focus of daily life during childhood, adolescence and early adulthood. It also widened the gap between literate and non-literate cultures in a manner that placed the well-read adult at an increasing distance from the unschooled small child (Eisenstein 1979)3. There are arguments for regarding Guttenberg’s invention as part of a continuously unfolding process. For at least
50 years after the introduction of movable type, there was no striking evidence of a cultural shift. Eisenstein (1979) argues that one must wait a full century to see evidence to emerge into full view.
The printing press also introduced some unique organizational shifts. Early printers were not only responsible for printing reference guides, but also compiling them (Eisenstein
1979). The growing power of the press as an independent group empowered all classes.
Gifted boys who might have become preachers became publicists.
“Perhaps no social revolution in European history is as fundamental as that which saw book learning gradually become the focus of daily life during childhood, adolescence and early manhood” (Eisenstein 1979, page 432). One of the most important cultural shifts resulting from the printing press was learning by reading, with the transmission of knowledge becoming much more efficient. “The nature of collective memory was transformed”
(Eisenstein 1979, page 66). The act of putting the Bible in everyone’s hand did encourage splintering of congregations and a new tendency towards religious self-help.
3 "The prospect of tackling this subject is far too vast to be assessed by any present or future assemblage is apt to even daunt the most audacious individual. If it is too vast to be handled by any single scholar, however, it is, by the same token, also too vast to be avoided by any single scholar" (Eisenstein 1979). Page: 9 of 237
Thomas Virgona Doctoral Dissertation: Defense
Before the printing press, scientific knowledge was slow to spread. Charting of the planets, mapping the earth and codifying laws, synchronizing chronologies and compiling bibliographies were all revolutionized by the printing press (Eisenstein 1979). Main centers of knowledge dispersed. The local storyteller was replaced by the literate villager. With the new editions of books and dictionaries, scholars found teaching to be an easier task, matching information from books to lectures (Eisenstein 1979). Journals speeded up circulation of scientific news and enabled scattered scholars to keep abreast of each others’ work. Hebrew and Arabic studies gained new momentum, as Medieval Bible studies had depended upon oral contact with the Jews and Greeks. The result was a move from manual to mental labor. The printer’s workshop attracted scholars of the day. Easily transmitted information enabled each subsequent generation to probe deeper into the past and advance beyond the position of its predecessor. The new format of technical literature increased scholar’s ability to cite prior works. One question still remained: “Had technology gone to press or was it still largely concealed?” (Eisenstein 1979, page 555), or as stated in Polyanyi’s maxim: “we know more than we can tell” (Srikantaiah and Michael 2000).
In the 15th century, the Roman Catholic Church was an integral part of people’s lives, in many ways shaping thoughts and perceptions. Many changes in society are caused by the introduction of technology. Today, people have integrated computers into their thought processes, into the way they work and think. As the printing press had a large impact on the way information was used throughout society in the 15th century, will the events of September
11, 2001 have a similar impact on information service providers in the area of disaster recovery, specifically the firms on Wall Street? Will “owners” of the financial information systems be forced to change their views on the protection of information, not unlike the
Page: 10 of 237
Thomas Virgona Doctoral Dissertation: Defense
changes the Roman Catholic Church went through as a result of the introduction of the printing press? As Postman said (1992), new technologies alter the structure of our interests; the things we think about. Modern computer technology had made information available in many forms and virtually instantaneously. As a result, critical information service providers need to respond to outages or users may go elsewhere for information, just as the Roman
Catholic Church discovered.
A second significant event in the field of information technology was the Soviet’s launching of Sputnik in October 1957. Kippenberger looks at the beginning of the e- commerce boom and its worldwide growth, and believes it all began after the Soviet Union launched the Sputnik in 1957 (Kippenberger 1999).Since Sputnik, the growth in American technology has been unprecedented. When the Russians astonished the world with Sputnik I, even the general public became concerned. Governments listened as well and provided funding using the following rationale: Because science and technology are strategically important for society, all efforts that help them, information activities in particular, are also important (Saracevic 1999). President Eisenhower was much less concerned about the Soviet actions than was the general public but nonetheless substantially altered many defense programs in order to meet perceived public demands. The President acknowledged privately that at least two-thirds of a spending supplement was used to meet public fears, not real security needs (Payne 1994).
On October 9, 1957, Eisenhower faced the press for the first time since the launch. Seeking to calm Congress and the public, he assured reporters that Sputnik contained "no additional threat to the United States," adding that "from what [the Soviets] say, they have put one small ball in the air." When asked how his administration could have let the Soviets be first in space, Eisenhower said that "no one ever suggested to me . . . a race except, of course, more than once we would say, well, there is going to be a great psychological advantage in world politics to putting the thing up, but . . . in view of the real scientific character of our development, there didn't seem to be a reason for just trying to grow hysterical about it." He added that he had provided the
Page: 11 of 237
Thomas Virgona Doctoral Dissertation: Defense
U.S. satellite and missile efforts with funds "to the limit of my ability . . . and that is all I can do." (Anonymous 1957)
The launching of the Sputnik satellite into space by the Soviets in October of 1957 prompted ample public interest in the U.S.A. and led to a mini-explosion in Soviet studies.
New Russian and Soviet study programs were established at many universities and the few already existing programs got a second wind (Zilper 2002). In the U.S., the National Science
Foundation (NSF) Act of 1950 established NSF and provided funding for a number of mandates, among them “to foster the interchange of scientific information among scientists in the U.S. and foreign countries” and “to further the full dissemination of [scientific and technical] information of scientific value consistent with the national interest”. The 1958
National Defense Education Act (also known as the “Sputnik Act”) enlarged the scope of the
National Science Foundation to include a task to develop methods, including mechanized systems, for making scientific information available”. By those mandates, an NSF division, which after a number of name and direction changes is now called the Division of Information and Intelligent Systems (IIS), has supported research in these areas since the 1950s.
Importantly, the field-defining studies that NSF supported included, among others, Cranfield
IR4 evaluation studies in the 1950s and 1960s, large potions of SMART studies from the
1960s to 1990s, and now the Digital Libraries Initiatives (Saracevic 1999). Information
Science developed and flourished, as did many other fields, due in large part due to government support by a host of agencies. Historically, the support was a success—it was instrumental in creation of the whole enterprise of information science and even of the information online industry based on Information Retrieval (IR) (Hahn 1996). But to the credit of information science it kept growing on its own even after government support slackened substantially. This cannot be said of a number of other fields or areas that
Page: 12 of 237
Thomas Virgona Doctoral Dissertation: Defense
floundered after the government stopped being their main source and resource (Saracevic
1999).
Workers are now technically educated and able to take initiatives, which leads to a third growth event in information technology, the introduction of the Internet. Of all the technical innovations featured in the ARPANET, forerunner of the Internet, perhaps the most celebrated was packet switching. The Internet is the fabric of our lives, a ubiquitous presence.
Information technology is the present-day equivalent of electricity in the industrial era. In reality, the social impact of cyberspace upon the individual is only beginning to be understood
(Conway et al 2003). Winner suggests the most significant challenge posed by the linking of computers and telecommunications is the prospect that the basic structures of political order will be recast (Winner 1986).
The present time could be characterized as the era of the Internet. The Internet seems to have some positive effect on social interaction, and it tends to increase exposure to other sources of information. The Internet provides a tool to give a voice to people who would otherwise find difficulty in obtaining that voice. However, the Internet still has difficulty in attracting the most deprived and socially excluded in society. It is in these respects that the
Internet, rather than providing a vehicle for liberation, serves to reinforce the prevailing control, as the more powerful have the louder and more eloquent voices (Conway et al 2003).
Indeed, wider access and participation in the information society is paramount for broader issues of social inclusion. Many theorists reject any suggestion that the “information revolution” has overturned everything that went before. On the contrary, they come to explain that it is a primarily an outcome and expression of established and continuing relations (Webster 1995). Herbert Schiller suggests that the information explosion of the
4 The Cranfield experiments were designed to evaluate the performance of various indexing languages in retrieval. Page: 13 of 237
Thomas Virgona Doctoral Dissertation: Defense
post-war years is the consequence, for the most part, of corporate capitalism’s inexorable march (Webster 1995).
The Internet is not just a technology, but a technology of freedom. The fundamental digital divide is not measured by the number of connections to the Internet, but by the consequences of both connection and lack of connection (Castells 3003). Technology is a vitally important aspect of the human condition. Technologies feed, clothe, and provide shelter for us; they transport, entertain, and heal us; they provide the bases of wealth and of leisure; they also pollute and kill (Mackenzie and Wajcman 1999). Castells defined the digital divide by the following categories: Income, Education, Age and Ethnic.
Littlefield researched the impact of the Internet on real estate sales. Internet access provides convenience and opportunity for home buyers, providing the ability to search the
Internet for house-related information. The younger generation of home buyers appears to use the Internet to aid their home purchases more than older generations do (Littlefield, et al
2000).
It is critical to note that these three events (the printing press, Sputnik, and the
Internet) were protracted historical events and did not occur as a singular phenomenon.
Fischer refers to the reduction of these extended trends to momentary transformation as
“telescopic fallacies” (Fischer 1970). An exhaustive study of any or all three of these innovations and their origins would demonstrate an extended development timeline.
Eisenstein (1979) favors the gradualist, evolutionary approach. However, the use of these singular phenomena is meant to show the impact to information systems and not make a long story short. Historians also find similar telescoping problems when survey respondents recall events (Fischer 1970).
Page: 14 of 237
Thomas Virgona Doctoral Dissertation: Defense
The underlying issue of this research is that on September 11, 2001, the terrorist attacks that struck downtown Manhattan rendered Wall Street area financial services unable to provide critical information services. This research will investigate the role people played in the disaster recovery efforts and subsequent updates to disaster recovery plans to account for these roles and tasks. The research indicates that the disruption to the workings of the financial information systems rendered them unusable by customers and clients. It is important to remember that many of the system failures and outages that occurred on that fateful day are not public knowledge and are treated as confidential information.
Social scientists generally agree on what disasters are and how they are distinguished from other social phenomena (Kreps, and Kroll-Smith/Gunter in Quarantelli 1998). For this study of September 11, 2001, the Porfiriev definition is sufficient (in Quarantelli 1998). A disaster is a condition destabilizing the social system that manifests itself in a malfunctioning or disruption of connections and communications of a social unit, partial or total destruction, making it necessary to take extraordinary or emergency countermeasures to reestablish stability (Kreps, and Kroll-Smith/Gunter in Quarantelli 1998).
Gilbert classifies disasters into three paradigms (Quarantelli 1998). The first is a catastrophe imputed by an external agent or human communities reacting against an aggression. Gilbert calls this a duplication of war. The second disaster is an expression of social vulnerabilities. The third disaster is an entrance into a state of uncertainty.5 September
11, 2001 would fall into the first paradigm – duplication of war. These events involve considerable harm to the physical and social environment. They happened suddenly and something might have been done to mitigate their effect before or after they happened (Kreps in Quarantelli 1998).
Page: 15 of 237
Thomas Virgona Doctoral Dissertation: Defense
The devastating loss of life was concentrated in the financial industry. Fatalities in that industry represented over 74 percent of the total civilian casualties in the World Trade Center attacks, and one firm, Cantor Fitzgerald, lost 658 employees (General Accounting Office
2003). It is not specifically known how many held positions in information technology or were responsible for disaster recovery tasks. Terrorist attacks on physical infrastructure are capable of interrupting major financial, banking and payment functions. Ironically,
September 11, 2001 was not the first such attack aimed at Wall Street. On September 16,
1920, a horse-drawn wagon carrying hundreds of pounds of explosives was detonated at the corner of Wall and Broad Streets in lower Manhattan, killing thirty people and causing the
New York Stock Exchange to close. The exchange reopened the next day and banking and financial activity quickly returned to normal (Brooks 1969). One difference, aside from the magnitude of the loss of life, between the incident in 1920 and 2001 is the reliance on technology. Lacker calls this a “technology shock,” a significant loss of operational capability due to either the loss or malfunction of physical capital or the loss of staff (Lacker
2003).
One of the most visible disruptions was that the New York Stock Exchange (NYSE) ceased all operations for four business days. Although not a direct target of the attack, the dependency on other financial systems (e.g., inter-bank payments and the Federal Reserve
Bank) made normal business operations impossible. However, at the core of the issue was the disruption of inter-bank payment systems (Lacker 2003). On that Wednesday (9/12/2001),
5 Disasters need to be studied within human groups, and not the results of external factors. The framework for disaster is not conflict or external attacks, but the results of upsetting human relations. Page: 16 of 237
Thomas Virgona Doctoral Dissertation: Defense
Richard A. Grasso, chairman of the New York Stock Exchange, vowed that U.S. stock trading would resume no later than Monday (9/17/2001) (Blustein and Day 2001). The decision to shut down the NYSE and when to return to operation was a difficult one, fraught with risk. There was a risk of bringing the markets back too soon if too few participants were functioning again. Also poor liquidity could hamper trading and exacerbate the expected price declines. Moreover, physical conditions in Lower Manhattan were unpleasant and potentially harmful. Conversely, the symbolic value of a return to normalcy was very attractive. The questions remain unanswered: Where these plans fully tested in regularly scheduled Disaster Recovery simulations? Were operational and technology decisions on
September 11, 2001 dependent on individuals.6
From an economic and operational perspective, the banking system was in relatively healthy condition on September 11, 2001. From a geographical perspective, it was a true disaster. The facilities of the New York Board of Trade in Four World Trade Center were destroyed. Several firms, including the Federal Reserve Bank itself, were forced to relocate to disaster recovery sites. Regional stock exchanges, the NASDAQ Stock Market, the Chicago
Board of Trade, the Bond Market Association and the Chicago Mercantile Exchange all closed as well. European markets remained officially open but from a “human” perspective, traders found it difficult to do much business. Connections to the Bank of New York (BoNY) were lost for part of the week and as a result the bank did not know what securities and cash it had received, and it was unable to transmit settlement instructions (Costa 2001).
6 In some scenarios, individuals do not make decisions to implement disaster recovery plans. For example: During a power outage, computer systems cutover to auxiliary power without human intervention. Page: 17 of 237
Thomas Virgona Doctoral Dissertation: Defense
On the Federal Reserve’s Fedwire Funds Transfer System, payments are initiated by the sender of funds, but the major banks’ inability to send funds transfer payment instructions following the September 11, 2001 attacks meant funds accumulated in that bank’s account. At one point during the week after September 11, BoNY publicly reported to be overdue on $100 billion in payments (Beckett and Sapsford 2001). The Moscow International Currency
Exchange (MICEX), which used BoNY as a business partner, suspended trading due to
BoNYs problems.
Banks with excess balances found it difficult to locate borrowers. The general disruption in payment flows meant uncertainty for many banks about whether scheduled incoming payments would be received as planned (McAndrews and Potter 2002). This lack of clarity on bank finances at a macro level caused trickle down concerns at a microeconomic level. Reports of increased cash withdrawals by bank depositors were common. Currency in circulation increased by $4.4 billion from Monday (9/10/2001) to Wednesday (9/12/2001)
(Lacker 2003). The government securities market also was hit particularly hard because many critical market participants were incapacitated, in part because the government securities market opens earlier that the stock market. Trading in U.S. government securities starts at 8 a.m. in New York. Dealers in U.S. government securities trade with each other through inter-dealer brokers (IDBs). Cantor Fitzgerald, which suffered tragic losses, was the largest IDB prior to the attack.
Two hundred thousand voice access lines went out, 100,000 PBX/Centrex lines went out, 3.6 million data circuits went out, 10 cellular towers were lost or damaged, and approximately 14,000 businesses and 20,000 residential customers were affected (Lacker
2003). The disruption to communications links impaired many institutions’ ability to initiate payment instructions. The failure of many communications links between government
Page: 18 of 237
Thomas Virgona Doctoral Dissertation: Defense
securities dealers and the market clearing and settling institutions was also a source of major disruption. The voice communication systems which replaced e-mail or computer system communication to conduct business as a contingency proved to be unreliable (Lacker 2003).
In New York and Washington, bank branch closings were widespread, but many banks outside those cities closed branches briefly as well. Some state banking agencies and the Office of the Comptroller of the Currency issued statements allowing banks to close at their discretion (Lacker 2003). Keeping in mind that Wall Street is a major information service provider, the financial world was literally in a state of flux immediately after
September 11, 2001. There was no reliable method of knowing your bank balance or if checks were processed, as well as your stock and bond portfolio status.
What happened the first day the financial markets re-opened? The Dow Jones
Industrial Average experienced its largest single-day drop ever, with the Industrials tumbling more than 680 points (7%). By comparison, on Tuesday October 29, 1929, Wall Street witnessed a 13% drop, known in financial mythology as “Black Tuesday”.
The events of September 11, 2001 literally struck at the heart of America’s financial information center. What impact will September 11, 2001 have on Information
Systems and Technology, specifically disaster recovery planning and implementation? Will it be revolutionary or evolutionary? What were the human factors encountered that day? Has the role of individuals been accounted for in updated disaster recovery plans? In many ways,
September 11, 2001 was the first implementation of full disaster recovery effort, thereby exposing deficiencies in recovery plans. As scholars, it is imperative to investigate the root causes and impact of these changes, and future implications for the discipline as a whole. “So it is natural to ask how the events of 11 September will affect our profession in the months and years ahead (Hayhoe 2002).” Scholars believe these “human” factors present one of the
Page: 19 of 237
Thomas Virgona Doctoral Dissertation: Defense
most unpredictable areas for disaster recovery researchers (Sikich 2003). How will humans react to unfolding events? Sikich also puts forth the definition of human factors7 in the context of business continuity (Sikich 2003). Questions that are now relevant include:
• How well do you know your workforce?
• What is the extent of background checks that are part of the screening
process?
• Can someone, either overtly, clandestinely, or unwittingly, be
compromised into creating an exposure that puts the firm at risk?
• How can you implement checks and balances so that critical information is
not subject to compromise.
1.2 Theoretical Framework
Several theoretical frameworks are available for this research. As Geoges Anderla said (in Horton and Lewis 1991) many disciplines, especially economics, land themselves in trouble whenever they attempt to integrate technology innovation into theoretical frameworks.
Everett Rogers has developed the very versatile diffusion theory and has already applied the theory to information communication on September 11, 2001 (Rogers 2003). The events were so shocking that people felt they needed to share the news and their reactions with others. In a survey, 88 of the 127 respondents (69%) communicated the news to others
(Rogers 2003). Those 88 people communicated with 418 others (average of 4.8 people
7 Not to be confused with human ‘aspects.’ Page: 20 of 237
Thomas Virgona Doctoral Dissertation: Defense
each). Despite advanced technology, this is a clear example of the reliance of human communications networks.
Control of information has always been dictated by technology. Frederick Ferre stated that the definition of technology is the practical implementations of intelligence (Ferre 1988).
There is a tendency to forget that sermons used to couple news, real estate transactions and other mundane matters (Eisenstein 1979). The Sunday paper has replaced church going as an information source. Until Gutenberg, the church had censured ideas more than texts. In the big cities, newspapers succeeded in reaching the general population, whose cultural and educational level was low (Martin 1994). The printing press was such a major technological advancement that Sir Francis Bacon said it was one of the three inventions (printing, gunpowder, and the compass) that changed the state of the whole world (Eisenstein 1979).
Christopher Burns (in Horton and Lewis 1991) raises the information control issue in a more recent event – “Three Mile Island.” What went wrong at Three Mile Island is classic information management collapse that raised critical questions about how to control the new information environments. As with September 11, 2001, communication lines were also unavailable or simply broken.
More specific to this research effort would be Hollan’s views on Distributed
Cognition. The theory of distributed cognition seeks to understand the organization of cognitive systems (Hollan, Hutchins and Kirsch 2000). Unlike traditional theories, it reaches beyond what is considered cognitive and beyond the individual to encompass interactions between people and with resources and materials in the environment. In this particular scenario, technology was the material and the environment is September 11, 2001. It is distributed by placing memories, facts, or knowledge on the objects, individuals, and tools in our environment. Distributed cognition is a useful approach for (re)designing social aspects
Page: 21 of 237
Thomas Virgona Doctoral Dissertation: Defense
of cognition by putting emphasis on the individual and his or her environment. Distributed cognition views a system as a set of representations, and models the interchange of information among these representations. These representations can be either in the mental space of the participants or external representations available in the environment. This model establishes a new foundation for human-computer interaction research. In the more traditional views of cognitive factors, the boundaries are those of the individuals. One aim of cognition, or the mental process of knowing, is to create knowledge through perception, reasoning, or intuition. In Hollan’s view of distributed cognition, one expects systems to configure themselves dynamically to align subsystems and accomplish detailed functions
(Hollan et al 2000). Christopher Burns (in Horton and Lewis 1991) noted the importance of team processing in technology. He recognized that the old industrial approach of segregating work and assigning it to specialists does not produce the best result when information work is involved. A carefully engineered collaborative approach is more effective.
When applied to observing human activity in its natural setting, three types of distributive cognitive processes become apparent (Hollan et al 2000). In order to understand and design effective human-computer interactions, it is critical that these processes are understood.
• Cognitive processes may be distributed across the members of a social group.
They involve trajectories of information (transmission and transformation) and are
reflective of an underlying cognitive architecture. The broader context includes
phenomena that emerge in social interactions as well as the interactions between
people and the structure in their environments. In this research, the application of
the trajectories is the design and execution of disaster recovery information and the
subsequent human interactions.
Page: 22 of 237
Thomas Virgona Doctoral Dissertation: Defense
• Cognitive processes may involve coordination between internal and external
(material or environmental) structures. This is an essential fact of cognition
that people are designed to use. It is these specific undocumented interactions
during September 11, 2001 that require investigation to determine gaps in disaster
recovery plans on that day. “Well-designed work materials become integrated into
the way people think, and control activities, part of the distributed system of
cognitive control” (Hollan et al 2000, page 178).
• Process may be distributed through time in such a way that the products of
earlier events can transform later events. Culture accumulates solutions to
frequently encountered problems. The meeting of cognition and culture is the
concept that a person’s environment is a reservoir of resources for learning,
problem solving and reasoning. One of the goals of distributed cognition is to
return culture, context, and history to the cognitive view. One aim of this study is
to learn from the disaster recovery failures8 on September 11, 2001 and to design
better controlled and more effective plans.
Since the cognitive properties of the entire system are larger than any one individual’s activity, cognitive ethnography9 must be event-centered (e.g. September 11, 2001). When speaking to experts on the lessons of financial information systems on September 11, 2001, one must know the design structures and how they were organized. Hollan believes this forces us to look at the barriers between what is defined as inside and outside, forcing exploration of interface components. A rapport must be established with the participants.
What processes and tasks were people engaged in, and what actions performed during the event (September 11, 2001) were meaningful? Hollan believes this is invariably revealing
8 Learning can also happen from success; however, this study is focus on failures on September 11, 2001. Page: 23 of 237
Thomas Virgona Doctoral Dissertation: Defense
and surprising. “As we build richer, more all-encompassing computational environments it becomes more important than ever to understand the ways human agents and their local environments are tightly coupled in the processing loops that result in intelligent action”
(Hollan et al 2000, page 186). After all, we are constantly reorganizing the work environments to optimize performance. Individual work tasks are no longer confined to a desk, but reach into the global networked world. Distributed cognition is tailored to understand the interactions among people and technology. The framework requires the observation of human activity, the analysis of cognitive processes of social groups, coordination of internal and external structures, and how products of earlier events can transform the nature of later events.
Distributed cognition is a popular framework for researchers. A recent Google search indicated 249 scholarly articles citing Hollan’s work. Rogers (2004) wrote that the distributed cognition approach has been used primarily by researchers to analyze a variety of cognitive systems, including:
• Airline cockpits: As with supporting financial system on Wall Street, flying a
modern jet transport is a job that cannot (at least not in current practice) be done by an
individual acting alone. The distribution of access to information is an important
property of systems of the distributed cognition theory. The shared understanding of
the situation is known as an inter-subjective understanding. A cockpit provides an
opportunity to study the interactions of internal and external representational structure
and the distribution of cognitive activity among the members of the crew. The
properties of the larger system emerge from the interactions among the members of
the crew and the contents of those communications. Interpretations are determined in
9 A cognitive ethnography study investigates information practices in experimental life sciences research (Peter Jones 2005). Page: 24 of 237
Thomas Virgona Doctoral Dissertation: Defense
part by the access to information of the crew. Through an analysis of audio and video
recordings of the behaviors of real airline flight crews performing in a high fidelity
flight simulator, Hutchins and Klausen were able to demonstrate that the expertise in
this system resides not only in the knowledge and skills of the human actors, but in the
organization of the tools in the work environment as well (Hutchins and Klausen,
1996).10 The analysis reveals a pattern of cooperation and coordination of actions
among the crew. Ironically, one pilot remarked, the cockpit is a poor classroom, a
considerable amount of training takes place there.
Hutchins and Palen also studied distributed cognition in a cockpit environment. Their research looked to supplementing the meaning of verbal communications with Space and
Gestures.11 The gestures acquire their meaning by virtue of being superimposed on the meaningful spatial layout of the control panel. The same gestures produced in the absence of the panel would, of course, be quite meaningless (Hutchins and Palen 1997).
• Call centers: Ackerman and Halverson studied organizational memory in the
framework of distributed cognition in a telephone hotline group. It was noted that
most studies of organization memory have largely focused on the technology
systems designed to replace human and paper-based memory systems (Ackerman
and Halverson, 1998). Telephone hotlines12 in general, are good places to study
memory in an organization, because their operation is so information intensive. In
the telephone hotline study, memories were complexly distributed, interwoven,
10 This research included analysis of cockpit transcripts. 11 The Hutchins and Palen research employed the use of videotapes. 12 The Ackerman and Halverson study used several data collection methods: direct observation, video, semi-structured interviews and social network analysis. Page: 25 of 237
Thomas Virgona Doctoral Dissertation: Defense
and occasionally overlaid, which makes telephone hotlines a good research area
for distributed cognition. It was found that memory is both an artifact that holds
its state and an artifact that is embedded in many organizational and individual
processes.
• Software teams: Software development can be a highly social activity involving
frequent interaction between programmers and with their development tools in the
performance of a task (Flor and Hutchins, 1992). The development and
comprehension of a computer program is a function of how well the system
performs as a whole. Other system-level variables include how well programmers
communicate inside and outside the group and the use of development tools. The
system of system-level properties is very complex, yet, is too difficult to ignore.
The system performs the task, not any individual!
• Control systems: In a study of operations in emergency resource centers, public
displays (e.g. a flip chart) were noted to perform central roles in indicating status
information and facilitating discussions among decision makers (Garbis & Waern,
1999).
• Engineering practice: Rogers studied how networking technology has changed
the working practices of an engineering company (Rogers 1994). Rogers
specifically examined how a close-knit group of engineers attempt to collaborate
when managing a networked system, while at the same time trying to maintain
coordination of their interdependent activities. Through a Distributed Cognition
Page: 26 of 237
Thomas Virgona Doctoral Dissertation: Defense
analysis, Rogers was able to reveal various breakdowns that occurred in the work
activities and the mechanisms by which the group had adapted its working practice
to overcome them.
One of the main outcomes of the distributed cognition approach is the discovery of complex interdependencies between people and artifacts in their work activities (Rogers
2004). In this sense, the distributed cognition approach is difficult to apply, since there is not a set of clear features to look for, nor is there a check-list that can be easily followed when doing the analysis. The distributed cognition framework can provide insights for changing a design to improve user performance, or more generally, a work practice (Rogers 2004).
Distributed cognition is the appropriate theoretical grounding for this study, describing how systems technologists worked together during the disaster. Technical coordination and systemic processes occur by means of shared practices, beliefs, values, and structures of interaction, which are institutionally based. The cognitive processes studied here are distributed across the members of a social group – technology professionals located at Ground
Zero on September 11, 2001. The cognitive processes to be studied involve coordination between internal (corporate communications) and external (news of that day) structures. Did the tasks performed during that horrific day transform the nature of later DR events?
Distributed cognition provides the grounded theory framework that will be most fruitful for this research.
The expected outcome of this research will be a better understanding of the reliance on
“humans” during information technology disasters. Specifically, when disaster recovery plans are compiled and tested, what is the “new” (or modified) role of individuals during a real disaster scenario? Although the research will focus on Wall Street firms directly impacted by the events of September 11, 2001, all information technology professionals
Page: 27 of 237
Thomas Virgona Doctoral Dissertation: Defense
entrusted to build and maintain information systems can use the findings to enhance existing recovery plans. Disaster recovery plans are no longer a “second thought” or “we will get to that later.” As a result of the events of September 11, 2001, information systems must be constructed with functional and tested disaster recovery plans. Usability during a disaster is now a critical component of the Systems Design phase of the Systems Development Life
Cycle (SDLC). Additionally, auditors and regulators will include the Disaster Recovery (DR) design and testing results in reviews.
Below is a ”Waterfall” Depiction of the Research Problems’ relation to the discipline of Information Studies and how the final research questions are a subset of the Information
Science discipline:
Information Science: The theoretical discipline concerned with the application of mathematics, systems design, and other information processing concepts; it is an interdisciplinary science involving the efforts and skills of librarians, logicians, engineers, mathematicians and behavioral scientists. The application of information science results in an information system (Borko 1968).
Systems Analysis: Systems analysis is a means of viewing circumstances realistically
and designing practical solutions (Osborne and Nakamura. 2000). This includes a set
of guidelines and techniques that assists a systems analyst in stating functional
requirements of a system in logical terms (Yourdon 1979).
Systems Design: Systems design is a set of guidelines and techniques that
assists a systems designer in determining which modules, interconnected in a
way, will best solve a well-stated problem (Yourdon 1979).
Page: 28 of 237
Thomas Virgona Doctoral Dissertation: Defense
Disaster Recovery: Any system that relies on a computer13 should
include a plan to cope with the loss of that computer. It should be
written to take into account several levels of a disaster and should be
reviewed at regular intervals. Some responses may include: revert to a
manual system, create a special temporary system, maintain a second
backup computer, move operation to another location, or stop
operations (Osborne and Nakamura 2000).
Human Factors: Human Factors is the scientific discipline
concerned with the fundamental understanding of interactions
among human and other elements of a system, and the
application of appropriate methods and theory to improve
human well-being and overall system performance (Karwowski,
2001).
13 All information systems do not rely on a computer information system. Page: 29 of 237
Thomas Virgona Doctoral Dissertation: Defense
2 Literature Review
The events of September 11, 2001 represent one of the most significant moments in
American history. The terrorist attacks that occurred that day impacted every area of
American life. Financial markets were closed and skewed for months. Simple everyday tasks, such as commuting to work or going to an airport, were changed immediately. Many citizens were psychologically impacted and lived with a fear of another attack. Beniger once stated that important transformations of society rarely result from a single discreet event (Beniger
1986). September 11, 2001 may be that rare exception.14 Time will determine if the events of
September 11, 2001 will rise to the level of true information disaster. Barker (2007) listed what he believed to the biggest information technology disasters to date:15
1. Faulty Soviet early warning system nearly causes WWIII (1983)
2. The AT&T network collapse (1990)
3. The explosion of the Ariane 5 (1996)
4. Airbus A380 suffers from incompatible software issues (2006)
5. Mars Climate Observer metric problem (1998)
6. EDS and the Child Support Agency (2004)
7. The two-digit year-2000 problem (1999/2000)
8. When the laptops exploded (2006)
9. Siemens and the passport system (1999)
10. LA Airport flights grounded (2007)
One discipline clearly impacted, but not yet researched is the field of information technology. The fallout from the September 11, 2001 investigations indicates a vast array of
14 Pearl Harbor probably was another.
Page: 30 of 237
Thomas Virgona Doctoral Dissertation: Defense
information failures, both on a human and machine level.16 As business skills change, decision-makers must now seek out, retrieve, reorganize, assimilate, interpret, and utilize data
(Horton 1994a). As Gray and Altmann wrote in 2001, information in the world is useful only if we can find it when we need it. This will be a critical theme of September 11, 2001 – people now think of information as an all-pervasive/universal resource (Horton 1994a).
Single events can have a significant impact on this growing field. Horton and Lewis (1991) stated that in many information cases the senders of specific messages were either uninformed, misinformed, or if they were informed, therefore not able to fit the information into preconceived stereotypes, value systems, belief systems or attitudes. Several events that depict information disasters, according to Horton and Lewis are war (Hitler’s decision to attack the Soviet Union in 1941, Civil War Intelligence and Signals in 1864), tragedy (Three
Mile Island, and The Tacoma Bridge Disaster) and business (the stock market crash of
October 1987).
2.1 Information Systems Technology – The Beginning
In relative terms, information science does not have the deep and rich history of other disciplines, such as mathematics or astronomy. Shera wrote about the origins of information science.17 Like librarianship, information science is still largely an agglomeration of knowledge and technologies drawn from other areas (Bennet 1988). During the Second
World War, an outburst of scientific and technical innovations occurred, including the reliance on information. In the same general period, Vannevar Bush (Bush 1945) published his landmark article, including the theoretical Memex machine, which transformed thought
15 http://resources.zdnet.co.uk/articles/0,1000001991,39290976,00.htm 16 Horton and Lewis (1991) addressed this topic by bringing together examples of “how information mismanagement led to human misery, political misfortune and business failure.”
Page: 31 of 237
Thomas Virgona Doctoral Dissertation: Defense
and human creative activity. When the Russians astonished the world with Sputnik I, even the general public became concerned (Bennet 1988). Technology arose in a world of bizarre contradiction: starving people vs. bigger bombs (Winner 1986). Early in February 1958,
Allen Kent and his colleagues organized a national meeting at Case Western University in
Cleveland to discuss a proposal to establish a national center for scientific and technical information. Many U.S. scientists suggested that one of the reasons for the Soviets’ taking the lead in the space race was the existence of their Institute of Scientific Information (Kent in
Hahn 1998).
Scientists themselves became national heroes as the nation’s strength came to be determined equally by military might and by scientific capability (Bowles 1998). Information systems for science and technology had a privileged existence because of industrial and military needs and government policies. In the beginning, much of the pioneering was by individuals trained in chemistry (Buckland 1998). Eventually, information science grew and evolved into a practical discipline. The demarcation between the old and new ways of searching for information was the change from paper indexes and card files to online database
(Hahn 1996). The three groups18 of pioneers were:
1. Programmers, system designers and researchers
2. Large scale commercial and government operations
3. Early PC Users; their perspective provides yet another angle to judge success and
failures and to measure the rate of development.
17 The Information Society Concept dates back to Fritz Machlup in 1962 (Beniger 1986). Despite Machlup’s writings, there is a an ongoing scholarly debate on the we are an “Information Society”. 18 These are not exclusive categories and do have overlap. Page: 32 of 237
Thomas Virgona Doctoral Dissertation: Defense
2.2 The Growth of Information Systems Technology
Americans have enthusiastically embraced new information technology that has come along (Chandler and Cortada 2000). The development of online systems and services is not just a story of tapes, terminals, telephones, search engines, algorithms and downtime; it is also a story of people. Acquisition of information technology is associated with social privilege
(Chandler and Cortada 2000).
The leaders of the online age can be divided into three groups: the developers, the managers, and the users (Hahn 1998). The developers were diverse in their geographic and disciplinary backgrounds and their underlying goals. They were aggressive, competitive and imaginative in creating opportunities to exploit the latest hardware and software of the initial period. The second group, managers and trainers, demonstrated the problems of online systems. With zeal, perseverance, charm and even chicanery, they recruited and trained the first users. The users were the third group, playing a critical role in evaluating new systems, testing documentation, and assessing training programs (Hahn 1998).
Measurements can be used to demonstrate the growth and change of information systems. Moore’s law is a simple example of the volatility of the discipline. Moore's Law states that processor power doubles every 18 months (Otto, Cook and Chung 2001). A 2004 study at Berkeley states that annually we are generating about 4.5 Exabytes of magnetically stored information. This is equivalent of 34,000 Libraries of Congress! (Scholl 2004). The changes are fast and significant. A historical comparison can be made to another technology
– railroads. In the middle of the 19th century, there were 7 distinct track gauges for railroads operating in North America. This infrastructural impediment to the flow of goods had demonstrable effects on economic development, to say nothing of the additional costs of
Page: 33 of 237
Thomas Virgona Doctoral Dissertation: Defense
supporting such a rail network. It is important to note that nobody stopped the trains to wait for the tracks to become the same width (Weibel 1997). The analogy is clear. Current attempts to standardize information technology have not always been successful. Other information age issues include the misinterpretation of market data, complex systems going awry, and making difficult decisions with the information “at hand” (Burns in Horton and
Lewis 1991).
Information technology is the present day equivalent of electricity in the industrial era.
The Internet is the fabric of our lives (Castells 2003). What made it possible for the Internet to embrace the world at large was the development of the World Wide Web, an information sharing application developed in 1990 by an English programmer named Tim Bernes-Lee.
The Internet was purposely designed as a technology of free communication (Castells 2003).
The origins of the Internet are found in the ARPANET in 1969. The diffusion of the Internet provides a platform for a vast array of the changes19 in information systems and technology.
The Internet continues to evolve. Web portals are seen as positive potential frameworks for achieving order out of chaos (Lakos 2004). Key principles govern any portal design: simplicity, dependability, quantifiable value, personalization, and systematic management. Structured development concepts now include usability, self-navigation, self- sufficiency, personalization, and identifying content that is vital to the users. From many points of view, hypertext, and hypermedia have been a success (Jones and Willett 1997).
People pass around URLs as a way of sharing experiences (Brown et al, 2002).
19 Changes include the number of internet users, uses for software, new ways to communicate and shop, etc. Page: 34 of 237
Thomas Virgona Doctoral Dissertation: Defense
The WWW URLs are a unique interface and security is major, but not a singular, concern. The direct manipulation interface with its click-and-point modus operandi has made non-sequential reading of an information resource easy and productive (Jones and Willett
1997). Copyright and patent rights present issues (Baeza-Yates 1999) and are growing global concerns. The web then leads to other concerns: distributed data, high percentage of volatile data, large volume, unstructured and redundant data, quality of data and heterogeneous data.
Also, problems continue to arise from internal politics and the “war for screen real estate”
(Tennant and Michalak 2004). Two types of changes lead to web page and web site mortality: content and structure (Koehler 1999). Almost without exception, over the period of a year, all web documents change (Koehler 1999). Web sites demonstrate a great deal of volatility (and variability). Networked electronic information is often transitory, without quality control or stability (Velluci 1998). Near term challenges for the Internet include, but are not limited to security, ownership and structure (Rowley and Farrow 2000).
There sometimes is a cultural preference for paper over screen information. Why do air traffic controllers prefer paper to electronics? Flexibility in spatial layout, ease of manipulations, easy and direct marking and information at a glance are a few noted benefits.
When observing police officers trying to use laptops, it was discovered that laptop design, shape, input methods, and software did not support interweaving of the police officer’s computer activity with a conversation with the crime victims (Sellen and Harper 2002).
Page: 35 of 237
Thomas Virgona Doctoral Dissertation: Defense
2.3 Changes to Disaster Planning Caused by the Cold War
The 1950s and 1960s saw an emergence of policies and plans dealing with the threat of a nuclear attack resulting from the Cold War between the United States and the Soviet
Union (Fagan et al 2005). The first federal disaster planning administration, the Federal Civil
Defense Administration (FCDA), was created by President Harry Truman in 1949 after the
Soviet Union detonated its first atomic weapon. Congress remedied the fact that there was no instrument in place to offer direct federal aid to state and local governments during an emergency by passing the Federal Civil Defense Act of 1950. The FCDA later was made an independent agency (Fagan et al 2005). It then took over the responsibilities of what was once the National Security Resources Board (NSRB) which was created by the National
Security Act of 1947. This board was created to advise the President on coordinated mobilization of the United States during times of war.
There was tension within all levels of government at this time about the difference between civil defense activities during times of war and natural disaster relief efforts and what types of aid and activities were to be used for each. Also during this time, civil defense planners were creating mass evacuation policies for assumed targets of the USSR, on the belief that major cities and installations would become prime targets for nuclear missiles
(Fagan et al 2005). The Federal Civil Defense Act was modified in 1958 to allow the government to allocate money for civil emergency preparedness. During the 1960s, the
Office of Emergency Planning (OEP), which was renamed to the Office of Emergency
Page: 36 of 237
Thomas Virgona Doctoral Dissertation: Defense
Preparedness, became the lead organization for the coordination all of civilian emergency preparedness events (Fagan et al 2005). These activities included disaster relief, post-attack analysis, financial stabilization, resource deployment, and continuity of government functions.
During the Cold War era,20 disasters became more important to practitioners and scholars (McEntire 2004). The threat of “mutually assured destruction” reached its pinnacle during the Cuban Missile Crisis in 1962. Civil defense grew during this time to organize air- raid precautions, shelters and alarms for everyday citizens. Civil defense during the Cold War
(1948-1989) included the development of plans to relocate large civilian populations in the event of a threatened nuclear attack (Alexander 2002).
One element of civil defense in the nuclear era has been a strategy to preserve a functional government by protecting key political and military leaders. Underground bunkers were set up with dedicated communications systems and food stockpiles (Jackson 1994).
While civil defense was being ingrained in the institutional fiber of the American government, military leaders wondered how the populace would react after a nuclear exchange. Because it would be impossible and unethical to run a test on humans, the government looked to scholars for assistance. Millions of dollars were poured into the social sciences (particularly Sociology): academic institutions (such as the well-known Disaster
Research Center) were created to answer the questions (McEntire 2004). Although people’s responses to disasters had been studied for years, scholars were now able to conclude that victims generally exhibited rational behavior in natural disasters (McEntire 2004).
20 Russett (1993) defined the Cold War era as a period of escalated tension and hostilities between the Unites States and the Soviet Union in the areas of politics, military, ideology, etc Page: 37 of 237
Thomas Virgona Doctoral Dissertation: Defense
It is also critical to note the role and responsibilities that the military can take during a national crisis. Military involvement in direct law enforcement activities is normally prohibited by the Posse Comitatus Act (Brake 2003). This act prohibits the use of the military in activities such as: arrest; seizures of evidence; search of persons; search of a building; investigation of a crime; interviewing witnesses; pursuit of an escaped prisoner; search of an area for a suspect and other like activities. The Posse Comitatus Act, however, does not stop the military from providing logistical support, technical advice, facilities, training, and other forms of assistance to civilian law enforcement agencies. Courts have held that providing assistance falls in the “passive” category and does not violate the Posse Comitatus Act.
Technical support activities such as explosive ordinance disposal, providing specialized equipment, and expert advice on weapons of mass destruction (WMD) devices also do not violate the act (Brake 2003).
The military has now started dispensing its crisis management expertise to civilian groups through the use of simulators. The University of Central Florida, in conjunction with the U.S. Army and the Orange County (Florida) Fire Rescue Department, has developed and fielded a series of simulations for conducting disaster exercises and training public safety personnel to respond to disasters (Kincaid et al 2003). Researchers are also gathering persuasive evidence that training effectiveness is substantially improved by the use of simulation as compared with traditional field exercises. The training is in two distinct areas: emergency management incident command and emergency medical care performed in the field. The simulated scenarios include treatment of battlefield casualties and crisis management. It is interesting to note that police officers with military training experience the same levels of stress as police officers without military training (Patterson 2002).
Page: 38 of 237
Thomas Virgona Doctoral Dissertation: Defense
A goal of Cold War civil defense was to enable the greatest number of Americans to survive a Soviet nuclear attack should one occur, with a clear focus on the people as a strategic national resource (Dory 2003). Through outreach and educational events, the government provided the public with a basic understanding of the nature of the Soviet threat, the nation’s vulnerability to nuclear attack, and potential consequences if one were to occur.
The federal government developed some contingencies defining the roles and activities that agencies would perform under grave scenarios (e.g., Soviet sneak attack on Washington, and nuclear war) (Carafano 2006).
As cuts in military spending started at the end of World War II, military planners
(along with some civilian supporters) proposed a new understanding of military forces. Rather than rely upon rapid mobilizations following the outbreak of war, these planners argued that it was necessary to permanently prepare for unannounced attacks: what Sherry called an
“ideology of preparedness.” (Sherry 1977). Technological developments of the atomic bomb and the long-range bomber rendered obsolete the traditional reliance on oceans for a defense in geographic isolation. Instead, these planners believed that a new era of “total war” had begun in which “the battle was not confined to the front lines but extended to the home front as well.” E.B. White (White 1949) contemplated nuclear attacks at the beginning of the Cold
War: and his words seem eerily prescient in relation to September 11, 2001. He wrote:
The subtlest change in New York… is something people don't speak much about but that is in everyone's mind. The city, for the first time in its long history, is destructible. A single flight of planes no bigger than a wedge of geese can quickly end this island fantasy, burn the towers, crumble the bridges, turn the underground passages into lethal chambers, cremate the millions. The intimation of mortality is part of New York now: in the sound of jets overhead, the black headlines of the latest edition. All dwellers in cities must live with the stubborn fact of annihilation; in New York the fact is somewhat more concentrated because of the concentration of the city itself, and because, of all targets, New York has a certain clear priority. In the mind of whatever perverted dreamer who might loose the lightning, New York must hold a steady, irresistible charm
Page: 39 of 237
Thomas Virgona Doctoral Dissertation: Defense
Records management also underwent a transition and growth during this period.
Executive Order 9784 in 1946 required all executive branch agencies to implement records management programs and expanded the management authority of the National Archives
(Cox 2000). The mandate was better defined in the 1950 Federal Records Act. Records management developed into records creation, maintenance, and disposition. The act also required each agency to establish an ongoing program for records management and to work in concert with the National Archives. During the 1950s, vital records program originated as part of the “Continuity of Government” program. The initial focus of vital records programs was the continuation of Federal agency operations under national emergency conditions, including a possible enemy nuclear attack upon the United States, and the reconstitution of normal agency activities at the emergency’s conclusion. The Bureau of Budget established requirements21 for vital operating records protection programs. Executive Order 10346 in
1952 made each Federal department and agency responsible for carrying out its essential functions in an emergency. Subsequent presidents have issued various executive orders that have modified Federal continuity of government and emergency preparedness responsibilities.
The vital records program has increasingly been dedicated to meeting the challenges Federal agencies encounter in continuing their operations and protecting their records in the face of natural disasters and terrorism (National Archives and Records Administration 1996).
During this period, the military spawned a new industry for the private sector.
Companies were now bidding on defense contacts for the United States and NATO military forces. Adding to the complexity were new multi-national corporations that campaigned to strengthen international economies (Latham22 in Schain 2001). Much of the debate at this
21 Bulletins No. 51-14, May 22, 1951, and No. 52-5, September 6, 1951. 22 Chapter 3: Cooperation and Community in Europe: What the Marshal Plan Proposed, NATO Disposed. Page: 40 of 237
Thomas Virgona Doctoral Dissertation: Defense
time centered on the Marshall Plan and the financial aid provide to Western Europe at the end of World War II. Some in the business community expressed concern with the level of government involvement in global economics. Others, such as General Electric president
Philip D. Reed, believed the plan would open more markets to American companies
(McGlade23 in Schain 2001). Citing fears of industrial espionage, major firms such as DuPont and General Electric began to restrict, and in some cases prohibit, United States Technical
Assistance & Productivity Program24 teams from visiting (McGlade in Schain 2001).
Information resources are now embedded in new services in such a way as to appear indistinguishable from the product itself25 (Horton 1994a). Disaster recovery requirements for these services also created new opportunities for the private sector. One such company is
Iron Mountain.26
Iron Mountain was founded in 1951 in Livingston, NY, 125 miles north of New York City. In 1936, Herman Knaust purchased a depleted iron ore mine and 100 acres of land for $9,000 because he needed more space to grow his product. But by 1950, the mushroom market shifted and Mr. Knaust was looking for alternative uses for his mine, which he had named Iron Mountain. After World War II, Mr. Knaust sponsored the relocation to the United States of many Jewish immigrants who had lost their identity because their personal records had been destroyed. During that same period, the world was embroiled in cold war apprehension about atomic security. Both factors impressed upon Mr. Knaust the need to protect information from the havoc of wars or lesser disasters. In 1951, Iron Mountain Atomic Storage, Inc. was founded. Mr. Knaust opened the first "vaults" inside Iron Mountain and put a sales office in the Empire State Building. Having a knack for publicity, he persuaded luminaries such as General Douglas MacArthur to visit Iron Mountain. The attendant publicity was the extent of the new venture's marketing program. Iron Mountain's first customer was East River Savings Bank, who brought microfilm copies of deposit records and duplicate signature cards in armored cars for storage in the new mountain facility. Other corporate customers soon followed as New York-based companies began to see the need to protect their vital records.
23 Chapter 10: A Single Path for European Recovery? American Business Debates and Conflicts over the Marshall Plan. 24 USTA&P was started in 1948 as “exchange of persons in industry” program. 25 Horton cited classic examples: Merrill Lunch Cash Management Account, Federal Express Zap Mail and MCI Mail (Horton 1994). 26 http://www.ironmountain.com/company/history.asp Page: 41 of 237
Thomas Virgona Doctoral Dissertation: Defense
Today, the focus of many governments has shifted to terrorism. Modern disasters are complex enough to require the utmost flexibility in their management. From the 1970s onwards, disaster research stressed non-military models of civil protection, such as the incident command system (ICS). Civil protection later emerged as demand increased under the duress of more serious, civilian disasters such as earthquakes, hurricanes, floods, and transportation crashes (Blanchard 1984). The ICS is different from the traditional command- and-control model derived from the direction of troops during combat, as it relies on information sharing and collaboration among task forces (Irwin, 1989). Decision making is a major problem in disasters. Other areas for concern during disasters include bureaucratic politics/procedures, groupthink and misperception (McEntire 2004).
2.4 Changes to Disaster Planning in Relation to Other “Disasters”
Although September 11, 2001 has spurred disaster recovery planning, disaster recovery and continuity of business plans can be implemented for a variety of reasons. The reasons can be natural (flooding, hurricane, etc) or human (terrorism, war, blackout, etc.). In both cases, uncertainty is at the core of the problem (Hewitt in Quarantelli 1998).
Russell Dynes has developed a disaster topology to describe a tasks performed pre-and post-event. The purpose of the topology here is to provide a framework for describing extraordinary efforts and judgments during a disaster. The four types of tasks are (Dynes in
Quarantelli 1998):
Page: 42 of 237
Thomas Virgona Doctoral Dissertation: Defense
Table 1 – Dynes’ Four Types of Tasks
Tasks
Routine Non-routine
Same as pre- Type I – Established Type III – Extending
Organizational disaster
Structure New Type II - Expanding Type IV - Emergent
• Type I – organizations carry on the same tasks with the same structure but often
expand their conventional efforts by extending the workday and double shifting.
• Type II – Organization expands their structures to carry out anticipated disaster
tasks. These organizations anticipate the involvement and use volunteers to cope
with the extraordinary effort.
• Type III – Organizations with no anticipated emergency responsibility, but may
become involved because they possess manpower and other resources.
• Type IV – These organizations do not exist before a disaster. They become
involved with new tasks and develop a structure to deal with the assigned work.
Tucson, Arizona, experienced two large-scale floods in October 1983 and January
1993. McHugh’s research into the human response to the 1983 event, found that the community's emergency co-ordination center was ineffective and isolated from the public safety response network (McHugh 1995). Local government mitigated these deficiencies before the January 1993 flood in two ways. First, the community's emergency management agency merged into the Sheriff's Department and second, through consensus building and training, the community institutionalized an effective disaster response organizational Page: 43 of 237
Thomas Virgona Doctoral Dissertation: Defense
structure (Type II). During the Mexico City earthquake in 1985, the federal government did have a plan for disasters – assignment of responsibility to the Mexican military (Kreps in
Quarantelli 1998). There had been no formal planning for disasters of any kind. Much of the
Mexico City response was tied to specific locations and not centrally controlled (Type IV).
During the Chernobyl reactor accident in 1986, initial attempts to monitor and control radioactive contamination were largely improvised (Type IV) (Kreps in Quarantelli 1998).
Simply put, being on-site or responding to the accident was likely to result in acute radiation sickness.
Regardless of all we know, we acknowledge the impossibility of predicting future events. The private sector also recognizes the problems of predicting the future and planning for inevitable disasters. Consequently, companies should not use time and resources attempting to plan for patterns that are simply unpredictable (Day, et al 2004). Rather, it is critical that companies pay attention to contingency planning (such as crisis management plans). Crisis management plans must be robust enough to handle all forms of the unexpected. As events arise that give us insight into the unforeseen, it is essential that organizations reexamine their crisis management plans to see if they were designed effectively enough to handle the unique features of our evolving environment (Day, et al
2004). Decision-making under pressure requires certain capabilities and the factors that shape decisions under pressure are quite different from those in on-disasters circumstances (Childs
2004). When researching firefighters specifically, “freelancing” decisions during a disaster presents other problems, including endangering lives. Issues encountered during a disaster, which create dysfunction under intensely stressful “battlefield” conditions, can be mitigated by repeated practice under more realistic conditions (McHugh 1995). This practice enhances performance and adaptability to varied conditions.
Page: 44 of 237
Thomas Virgona Doctoral Dissertation: Defense
The first systematic efforts by the United States federal government to give some assistance after a disaster were during the Dust Bowl Years in the late 1920s and early 1930s after farmlands were devastated. The onslaught of WWII and the development of missiles capable of traveling several hundred miles by Germany was the catalyst for the federal government to develop a federal civil defense system (Fagan et al 2005). In 1979, the Federal
Emergency Management Agency (FEMA) was created by President Jimmy Carter to house civil defense emergency preparedness functions together in one organization. For the next two decades, FEMA would be the center for state and local emergency preparedness.
In the 1980s, an idea known as “Comprehensive Emergency Management” (CEM) developed within FEMA’s civilian programs. CEM refers to the responsibility for managing responses to all types of disasters and emergencies through the coordination of multiple agencies or entities. It is in this process that many feel the government failed during Hurricane
Katrina. Before September 11, 2001, there was no comprehensive federal emergency response plan available that integrated all the federal agencies and their respective roles. To remedy this, the Department of Homeland Security created the National Incident
Management System (NIMS), which works as a guide for the federal government, as well as the state and local governments (Fagan et all 2005). NIMS is an emergency response system aimed at providing flexibility and standardization throughout the life cycle of an incident. The main goal of NIMS is to provide effective and efficient coordination among the various levels of government during an emergency. NIMS is designed to function regardless of the size or difficulty of the incident; it uses a standard language to unify the response effort.
Hurricane Katrina provides tremendous lessons in disaster recovery, as the federal government has been criticized for its response. Katrina once again showed the reliance on technology and communications. The aftermath of Katrina has reinforced the role of
Page: 45 of 237
Thomas Virgona Doctoral Dissertation: Defense
communication networks and information management in providing effective response to a large-scale disaster (Banipal 2006). Breakdown of phone circuits, flooding of Public Service
Transmission lines and disruption of electricity contributed to failure of communication systems. Overall wireless voice and data networks had faster recovery time and performed better than the landline networks. The absence of inter-agency information system contributed to delayed response. Banipal (2006) specifically targets the design aspect in lessons learned from Katrina. Officials will need to refocus on the design of networks and information management systems so as to improve inter-agency communication, speed up recovery efforts and limit loss in business value. It is imperative that organizations involved in the disaster recovery process have all the information they need – quickly and accurately. Quick response to disaster has the potential to reduce total loss significantly.
In the specific case of records management, many problems that were identified following Hurricane Katrina (Ritzenthaler 2006). During the storm, records were exposed to sewage, petrochemicals and coroner lab contaminates. Building issues included power, access and security. The records recovery followed these steps: vacuum freeze dry items, sterilize item, clean and then reformat the record. For example, the Orleans Parish District
Attorney’s Records contained 785 cubic feet of archives and 36 computers (Ritzenthaler
2006). The critical lesson learned from Katrina from the records management viewpoint was
“know your records from several perspectives: vital, permanent, media, locations.”
Quarantelli (2005) wrote of the “catastrophe consequences” that can be learned from
Hurricane Katrina. In Katrina, there was across-the-board and almost total disruption of community functions. Most of the community infrastructure was heavily impacted. Local officials were unable to undertake their usual work role, and this often extended into the recovery period. In catastrophic situations local personnel are often unable for some time,
Page: 46 of 237
Thomas Virgona Doctoral Dissertation: Defense
both immediately after impact and into the recovery period, to carry out their formal and organizational work roles. This is because some local workers either were unable to communicate with or be contacted by their usual clients or customers and/or were unable to provide whatever information, knowledge or skills, etc. they usually can provide (Quarantelli
2005). Help from nearby communities could not be provided. (In many catastrophes, not only are all or most of the residents in a particular community affected, but often those in nearby localities are also impacted.) Most, if not all places of work, recreation, worship and education such as schools were totally shut down and the lifeline infrastructures was so badly disrupted that there were extensive shortages of electricity, water, mail or phone services as well as other means of communication and transportation. One of the more important Katrina consequences was the media activity. With Katrina, there was far more diffusion of rumors that usually occurs in disasters (Quarantelli 2005). The media were not always accurate in reports of looting in the post-disaster time period (Barsky et al 2006). While looting did occur, which is atypical for disasters, the anti-social behavior was widely depicted as out of control.
The question of “who is in charge?” was reiterated over and over again, depicting the command and control model as inept (Quarantelli 2005).
One example of a positive post-Katrina lesson is the New Orleans Veteran’s
Administration Medical Center. With a new Computerized Patient Record System, all patient records, prescriptions, and laboratory and radiology results on every New Orleans VA patient are now available at any VA medical center and by any VA physician nationwide
(Anonymous 2005).
A study into the role of social capital in the post earthquake reconstruction programs in two cases (Kobe, Japan and Gujarat, India) demonstrated the reliance upon people during a disaster. Social capital refers to the trust, social norms, and networks which affect social and
Page: 47 of 237
Thomas Virgona Doctoral Dissertation: Defense
economic activities. It is not a new idea that trust and networks help reduce transaction costs and make things easier. The Kobe case study shows that the community with social capital and with a tradition of community activities can pro-actively participate in the reconstruction program, and thereby make a successful and speedy recovery (Nakagawa and Shaw. 2004).
2.5 Information Systems and Technology – Theories and Methodologies
The growth of Information Technology is remarkable when one considers the age of the discipline. Early approaches emphasized the “waterfall”27 approach. One flaw was that critical requirements often emerge during system development and cannot be anticipated.
Brooks concluded that software designers should plan to throw one version of the software away (John Carroll in Baecker, Ronald, et al. 1995). That lesson continues, and design is now seen as opportunistic, concrete, and necessarily iterative.
Austrian biologist Ludwig von Bertalanffy defined a system as an entity which maintains its existence through the mutual interaction of its parts (Bertalanffy 1968). The environment is the part of the world that can be ignored by the information system itself, except for its interaction with the system. In order to understand the relationship among other systems, inputs, outputs and processes, one needs to understand the environment in which all of this occurs. The environment represents everything that is important to understanding the functioning of the system, but is not part of the system. It includes competition, people, technology, capital, raw materials, data, regulation and opportunities. Prescriptive information system methodologies are unlikely to cope well with strategic uncertainty, user communication or staff development (Middleton 1999). Middleton’s recommendations are to
27 The waterfall approach emphasizes feedback loops between the following development phases: System feasibility, Requirements, Design, Coding, Integration, Implementation and Operations/Maintenance (Boehm 1988). Page: 48 of 237
Thomas Virgona Doctoral Dissertation: Defense
focus more on soft organizational issues and to use approaches tailored to each project. While each technology project progresses through the System Development Life Cycle (SDLC), each project needs to ensure the disaster recovery designed for the developing application is in alignment with the organizations needs.
From a pragmatic point of view, the traditional System Development Life Cycle is one of the most critical methodologies in information technology. Disaster recovery is dependent on the SDLC for ensuring disaster recovery planning is integrated throughout the technology development process: the requirements for the system’s recovery are defined in the analysis phase, the system is designed to provide service during a disaster within the specified timeframes and testing the recovery capabilities is part of the creation of the project, thus ensuring continued use during a disaster.
Over the years, the basic SDLC has been modified for newer technologies, such as object oriented design, Unified Modeling Language (UML) and prototyping, but the basic construct remains in place. The steps in modern systems analysis are: problem definition; data collection and analysis; analysis of alternatives; feasibility determination; systems proposal; system design; pilot study; implementation; system review and evaluation. Systems analysis is a means of viewing circumstances realistically and designing practical solutions
(Osborne and Nakamura 2000). However, there is no guarantee the solution may be found.
The drawbacks of the “waterfall” approach have been well documented: managing ever- shifting requirements, poor relationships with the users and the emergence of serious problems late in a project (Middleton 1999). Poor quality is largely attributed to design problems (Cole 1981), which can be avoided by paying attention to quality problems during design, understanding customer requirements, and designing modularized objects for reuse.
Participation of users, vendors, and developers in the core design and development process
Page: 49 of 237
Thomas Virgona Doctoral Dissertation: Defense
promotes mutual understanding of issues and constraints to be addressed to improve quality.
Project management problems can be quickly summarized as (Middleton 1999):
1) Users did not know what they wanted.
2) Users did not know the possibilities of the technology.
3) User’s perceptions changed while the system was being developed.
4) The developers did not understand the intricacies of the user's work.
5) There were constant changes in the external environment that were
not anticipated.
In recognition of these issues, information system professionals enhanced the SDLC to include the use of:
• Flowcharts: Diagram that shows the operations performed in an information
processing systems and the sequence in which the operations performed.
Flowchart symbols are used to represent the operations and sequence of
operations (IBM 1969).
• Data dictionaries: A data dictionary is a set of metadata28 that contains
definitions and representations of data elements (Yourdon 1989).
• Decision tables: This table describes terms of conditions that must be satisfied
in order to carry out the action specified in the decision table. With every
decision table a set of decision rules, called a decision algorithm can be
associated. It is shown that every decision algorithm reveals some well-known
probabilistic properties, in particular it satisfies the total probability theorem
and the Bayes’ theorem. These properties give a new method of drawing
28 Hurley definition will be used as the working definition: Metadata is the data that in some manner describe the content of the object but is separate from the content (Dentinger 1998). Page: 50 of 237
Thomas Virgona Doctoral Dissertation: Defense
conclusions from data, without referring to prior and posterior probabilities
(Pawlak 2000).
• ER diagrams: Entity-relationship modeling is a method used to present a
system and its requirements in a top-down approach. This approach is
commonly used in Database design. The diagrams created using this method
are called ER diagrams (Chen 1976).
• UML: The Unified Modeling Language (UML) is a family of graphical
notations backed by a single meta-model that help in describing and designing
software systems, particularly software systems built using the object-
orientated (OO) style (Fowler 2004).
• GANNT chart: This chart displays the time span of each task as indicated by
the length of a line on an adjacent calendar (Carter 1987).
• DFD: A data flow diagram (DFD) is a graphical representation of the "flow" of
data through an information system. A data flow diagram can also be used for
the visualization of data processing (structured design). It is common practice
for a designer to draw a context-level DFD first which shows the interaction
between the system and outside entities. This context-level DFD is then
"exploded" to show more detail of the system being modeled (Yourdon 1979).
CASE tools are being used by larger firms to emphasize the prototyping and code generation facilities and to build completed systems. Smaller firms are primarily using the tools for analysis and design and to share development work across teams. Support for data flow diagrams and the data dictionary were revealed as key factors for improving productivity
(Post, Kagan, Leim, 1998).
Page: 51 of 237
Thomas Virgona Doctoral Dissertation: Defense
With the birth of object oriented programming, information technology was introduced to another subtle change to the SDLC methodology. It is important to understand the ongoing changes to information technology. Object Oriented (OO) analysis and design is not as mature as other structured techniques. In general, the OO style is to use several little objects29 with many small methods.30 The technique of keeping data with objects and, if necessary, providing techniques for making it available is called encapsulation and has been part of OO since its inception. Another core concept of OOP is polymorphism, the idea that a super-class defines a generic behavior, while specific instances of that behavior are refined when that super class is referred to by a class (Osborne and Nakamura 2000). This style is very confusing to people used to long procedures; indeed, this change is the heart of the paradigm shift of object orientation. Unified Modeling Language (UML) can be used to support Object Oriented Project Development initiatives. It is important to note that the UML is a modeling language, not a methodology. The UML has no notion of process, which is an important part of a methodology. Models based on objects provide a different perspective since they are structured around real-world objects. The benefits of object oriented analysis and design are reusability, reliability, seamless integration with a graphical user interface
(GUI), and speedier design. The suggested framework for Object Oriented Analysis and design is (Osborne and Nakamura 2000):
• Prototyping
• Diagramming tools
• UML
Designing the system includes:
• Functional Specification
29 There is much debate over the definition of an object. In OOP, an object is a finite set of components (Xing 2003). Page: 52 of 237
Thomas Virgona Doctoral Dissertation: Defense
• Determination of alternatives
• Conceptual design (inputs, outputs, processes, files)
• System integration
An example of a highly successful conceptual model based on an object is the spreadsheet (Winograd 1996). The first spreadsheet was designed by Dan Bricklin and was called VisiCalc. The main reasons why the spreadsheet has become so successful are that
Bricklin understood what kind of tool would be useful to people in the financial world (like accountants), and he knew how to design it so that it could be used in a way that these people would find useful (Preece 2002).
Methodologies developed over the last half century are more examples of changes and growth to the field of Information Science. Although not born or derived from disasters or singular events, they do represent the evolution of the field. The SDLC may be the most widely known, but it not the only contribution.
A mental model is one's way of looking at the world, a framework for the cognitive processes of our mind. In other words, it determines how we think and act. Much of the work involving mental models comes from Chris Argyris and his colleagues at Harvard University.
The object of activity theory31 is to understand the unity of consciousness and activity. The concern is that activity theory is hard to learn, and, because we have not seen its actual benefits realized in specific empirical studies, the time spent learning it would be of dubious benefit (Nardi 1995). The GOMS methodology involves: Goals, simple Operations, Methods of accomplishing a goal, and a Selection rule for alternatives. Cognitive modeling is the
30 Method is the body of a programming procedure for an object (Fowler 2004). 31 Vygotskian activity theory. In this approach the main feature of the psyche is the active position of human beings toward the world in which they live. Humans are continually changing the objects and creating artifacts – tools (Verenikina and Gould 1998). Page: 53 of 237
Thomas Virgona Doctoral Dissertation: Defense
application of cognitive theory to applied problems (Gray and Altmann 2001). Models vary in their concern with generality versus realism.
Systems’ thinking is a set of tools, a unique perspective on reality, and a specific vocabulary. It dates back to the 1940s and 1950s when thinkers such as Wiener, von
Bertalanffy, Ashby and von Foerster founded the domain through a series of interdisciplinary meetings (Heylighen, Joslyn, and Turchin. 1999). Systems theory or systems science argues that however complex or diverse the world, we will always find different types of organization in it. Such organization can be described by concepts and principles which are independent from the specific domain at which we are looking. The steps in systems thinking are the following: specify a problem/issue, construct a hypothesis, test the hypothesis, and implement changes (included looping feedback). Systems skills thinking requires: dynamic thinking (framing a problem of behavior over time); systems as a cause thinking (placing responsibility for a behavior on the internal actors who manage the policies and plumbing of a system); forest thinking (believing that, to know something, you must understand the context of relationships); operational thinking (concentrating on getting causality and understanding of how a behavior is actually generated); closed loop thinking (viewing causality as an ongoing process, not a one time event, with the effect feeding back to influence the causes, and the causes affecting each other); quantitative thinking (accepting that you can always quantify, but you can't always measure); and scientific thinking (recognizing that all models are working hypotheses that always have limited applicability) (Richmond 2000). The benefits of systems thinking include:
• More effective problem solving
• More effective leadership
• More effective communications
Page: 54 of 237
Thomas Virgona Doctoral Dissertation: Defense
• More effective planning
• More effective organizational development
• Avoiding Founder's Syndrome
Founder's Syndrome occurs when an organization operates primarily according to the personality of one of the members of the organization (usually the founder), rather than according to the mission (purpose) of the organization. When first starting their organizations, founders often have to do whatever it takes to get the organization off the ground, including making seat-of-the-pants decisions in order to deal with frequent crises that suddenly arise in the workplace. As a result, founders often struggle to see the larger picture and to plan effectively in order to make more proactive decisions. Consequently, the organization gets stuck in a highly reactive mode characterized by lack of funds and having to deal with one major crisis after another. The best "cure" for this syndrome is a broader understanding of the structures and processes of an organization, including an appreciation for the importance of planning (McNamara 1999).
McNamara describes how an organization seems to experience the same kinds of
problems over and over again. The problems seem to cycle through the organization. Over
time, members of the organization come to recognize the pattern of events in the cycle, rather
than the cycle itself. Parents notice this as they mature as parents. Over time, they recognize
the various phases their children go through and consider these phases when dealing with the
specific behaviors of their children (McNamara 1999). Systems that do not interact with their
environment (e.g., get feedback from customers) tend to reach limits.
Prescriptive information system methodologies are unlikely to cope well with strategic
uncertainty, user communication or staff development. Middleton’s recommendations are to
focus more on soft organizational issues and to tailor approaches to each project. All
Page: 55 of 237
Thomas Virgona Doctoral Dissertation: Defense
elements of the organization need to be developed in order to attain quality goals; piecemeal adoption of selected quality management practices are unlikely to be effective (Middleton
1997). This is essential in disaster planning, as critical components of the organization must continue to function in harmony with other organizational functions. Fleshing out the key areas during the initial system design will avoid haphazard decisions during a disaster.
2.6 The Human Component of Information Systems Technology
Despite the evolution and advances in information systems and technology, it is an almost universal finding in studies investigating human information behavior that people choose other people as their preferred source of information (Johnson 2004). Studies of academic researchers in both the sciences and the humanities have revealed the importance of consulting with colleagues at different stages of their research (Johnson 2004). Professionals, such as engineers, nurses, physicians and dentists rely on co-workers and knowledgeable colleagues in their search for work-related information (Leckie, et al., 1996). People are also among the most important sources consulted by chief executive officers during their environmental scanning (Choo 1993). Studies of ordinary citizens' preferred sources of information also confirm the importance of personal contacts in information seeking behavior
(Warner 1993). The poor, as well, prefer people over other sources of information (Agada
1999). The explanation for the use of people as information sources has often been that they are “typically easier and more readily accessible than the most authoritative printed sources”
(Case 2002). Immigrants are generally perceived to be information poor, meaning they face major challenges with finding and using greatly needed everyday information (Agada 1999).
Research findings suggest that personal networks were used more readily than any other type
Page: 56 of 237
Thomas Virgona Doctoral Dissertation: Defense
of information source (Fisher 2004). The ability of these populations to establish themselves independently is limited and often restricted by barriers of language and influence. There is a negative spiral effect for these populations as they work to improve their socio-economic situation while being unable to operate outside of the community information system they have established for themselves (Fisher 2004).
Human Computer Interaction (HCI) is a growing and maturing field. Although not the primary focus of this dissertation, the discipline of HCI has an impact on systems design, including disaster recovery planning. A unique aspect of the field is that HCI treats the computer and its operator as equals (Verenikina and Gould 1998). HCI research has investigated usability and where it fits into the concept of systems design. For good design, the designers need to know the users and their tasks (Karat and Karat 2003). Developing new systems is always done within a context of design trade-offs and limited resources (Karat and
Karat 2003). Since the events of September 11, 2001, usability may have taken on new context, especially for information providers, such as Wall Street financial firms. External parties, vendors and internal staff who use financial information to make critical economic decisions, require that data be available when needed. System analysis and design now includes Disaster Recovery and Continuity of Business as critical components of the phase
(Osborne and Nakamura 2000). Perhaps Everett Brenner put it best in “Brenner’s Law”:
Determine the best system you can foresee before designing the system you can afford (Hahn
1998).
Donald Norman has documented a common sense approach to usable design.
Designing well is not easy and it usually takes five or six attempts to get a product right. If an error is possible, someone will make it. In keeping with the human element theme of
September 11, 2001 and design problems, Norman cites an incident aboard a Lockheed L-
Page: 57 of 237
Thomas Virgona Doctoral Dissertation: Defense
1011 airliner flight to Miami as an example of poor design during disasters. The pilots were too busy to instruct the flight crew properly, so the passengers were not given safety instructions by the personnel (Norman 1998). This is an example of a disaster plan that relied on humans to convey information during a crisis, despite those people being occupied with other critical tasks.
Norman introduces several concepts that he uses in his analysis of both good and bad design: affordances (buttons are for pushing, menus are for choosing); constraints (logical relationship between the functional layout of components); conceptual models (a good conceptual model allows us to predict the effects of our actions); mappings (relationship between the controls and the results); visibility (the system state should be visible and interpretable); and feedback (sending information back to the user about what action has actually been done and what results have been accomplished). When applied to information systems, these concepts mean that computer systems must be capable of making things visible
(or audible). Norman also points out a critical linkage between usability and design (Norman
1988). Usability is rarely a consideration when purchasing, in fact, the purchaser is rarely the user.
Norman’s key concepts of user-centered design are (Norman 1988):
• Make it easy to determine what actions are possible at any moment.
• Make things visible, including the conceptual model.
• Make it easy to evaluate the current state of the system.
• Follow natural mappings between intentions and the required actions.
Landauer believes these contributions are marginal. Useful theory is impossible, because the behavior of human-computer systems is chaotic or worse, highly complex, dependent on many unpredictable variables, or just too hard to understand. Middleton (1999)
Page: 58 of 237
Thomas Virgona Doctoral Dissertation: Defense
also questioned the strict methodologies. Prescriptive information system methodologies are unlikely to cope well with strategic uncertainty, user communication or staff development.
His recommendations focus more on soft organizational issues and to use approaches tailored to each project.
With the Internet and many other sources available online, there is a need to ensure that people who are information technology savvy do not confuse this with having information literacy skills. There is more to information seeking than just knowing where to find information; that is, it also includes problem solving and evaluation of sources. The ability to validate sources is probably even more important today with the volume of information available on the Internet (Kerins 2004). The theory of social capital, however, suggests that the use of people as information sources is not necessarily an easy option, but may also require a considerable effort (Johnson 2004). The Internet seems to have a positive effect on social interaction, and it tends to increase exposure to other sources of information.
The body of evidence does not support the thesis that the Internet leads to lower social interaction and cause greater social isolation (Castells 2003).
Human information behavior is a highly active area of research within Information
Science and other fields. Research that has been carried out to date has contributed greatly to our understanding of human-information interaction. Yet, Fidel states that very few studies have generated results that are directly relevant to the design of information systems (Fidel
2004).
In recent years, researchers in HCI have criticized the gap between research results and practical design. There is an emerging consensus among researchers that the cognitive approach to HCI may be limited (Uden and Willis N.D.). Landauer writes that useful theory is impossible, because the behavior of human-computer systems is chaotic or worse, highly
Page: 59 of 237
Thomas Virgona Doctoral Dissertation: Defense
complex, dependent on many unpredictable variables, or just too hard to understand.
Theories have minor impact, such as Fitt’s Law32 and Hick’s Law33 (Landauer 1991). Even the best applications of theory have produced only small quantities and/or local gains in productivity. The few successful computer and HCI inventions to date have come from lucky hunches and produced mundane results, such as rules of thumb for the use of color or and empirical generalizations of user needs and characteristics.
Clearly, information systems would be most effective if their design is informed by an understanding of the human-information interaction of their intended users (Fidel and
Pejtersen 2004). Yet, information systems have been designed and widely used almost completely unaffected by results of studies in human information behavior. It is important to examine how human-information behavior research could inform design. A variety of reasons have probably motivated systems designers to ignore this research, such as pressure to design systems quickly, no obvious relevance of research results to design, and lack of appreciation of soft research. Instead of analyzing these reasons, Dervin thought it might be useful to examine how results of human information behavior research projects can increase their applicability to systems design. This will address a standing concern bridging the gap between designers and researchers, and increasing the relevance of academic research to the practitioners' work (Dervin. 2003).
The information systems themselves – not the people – can become the stable structure of the organization (Srikantaiah and Koenig 2000). This in-turn, will remove some of the reliance on human resources to solve problems during disasters. As Dombrowsky
32 Fitts's law is a model of human movement which predicts the time required to rapidly move to a target area, as a function of the distance to the target and the size of the target (e.g., on computers with a mouse) (Fitts 1954).
33Hick's law is a model of human-computer interaction that describes the time it takes for a user to make a decision as a function of the possible choices he or she has (Hick 1952). Page: 60 of 237
Thomas Virgona Doctoral Dissertation: Defense
wrote, it is foolish to intervene in systems upon which people depend without knowing how the systems work and how they will react (Quarantelli 1998).
2.7 The Relationship of the Events of September 11, 2001 to Information Systems
Humans have deployed technology to combat disaster since the beginning of recorded history. The cradle of Western civilization, the Tigris-Euphrates river valley, was settled and urbanized through an extensive flood control infrastructure that stabilized the flow of water to fields while also protecting fixed settlements (Moss and Townsend 2006). Over the past century, the role of technology has expanded from just mitigating the impacts of natural disaster to producing disaster itself. The devastating effects of aerial bombardment of cities during 20th century may well have killed more people than all natural disasters in history combined. Chernobyl (1986) and Bhopal (1984) demonstrate the potential for chemical and nuclear industrial accidents to cause major disasters (Moss and Townsend 2006).
A disaster is an unexpected occurrence inflicting widespread destruction and distress and having long-term adverse effects on society. An emergency is a situation or occurrence of a serious nature, developing suddenly and unexpectedly, and demanding immediate action
(power failure and minor flooding) (Hunter 1997). The events of September 11, 2001 can be defined as both an emergency and a disaster.
In reviewing the September 11, 2001 investigations, a common and tragic theme is the failure of information, both in quality and communication as documented in The Complete
Investigation; The September 11, 2001 Report: The National Commission on Terrorist
Attacks Upon the United States. On the morning of September 11, 2001, the existing information infrastructure design was unsuited in every respect for what was about to happen. Page: 61 of 237
Thomas Virgona Doctoral Dissertation: Defense
Problems with information that morning were vast. Even the President told investigators he was frustrated with communications. The airlines were facing an escalating number of conflicting and, for the most part, erroneous reports about other flights, as well as a continuing lack of vital information from the Federal Aviation Administration (FAA) about the hijacked flights. Several FAA air traffic control officials told investigators it was the air carriers’ responsibility to notify their planes of security problems. Most federal agencies learned about the crash in New York from CNN. Some startling revelations about information communication breakdowns on the fateful morning:
8:34; The command center tried to contact a former alert site in Atlantic City, unaware
it had been phased out.
• 9:42; The command center learned from a news report that a plane had struck the
Pentagon.
• The Secret Service was relying on projections and was not aware the plane was
already down in Pennsylvania.
• The Vice President was mistaken in his belief that “shoot down” authorization had
been passed to the pilots flying at NORAD’s direction.
• The Langley pilots were never briefed about the reason they were scrambled.
• The FAA “no-fly” list contained the names of just 12 suspected terrorists.
• The FDNY chiefs in the increasingly-crowded North Tower lobby were confronting
critical choices with little or no information. According to one of the chiefs in the
lobby, “One of the most critical things in a major operation like this is to have
information.”
Page: 62 of 237
Thomas Virgona Doctoral Dissertation: Defense
• A major planning failure was that the Mayor’s Office of Emergency Management,
created in 1996, was located at 7 World Trade Center. Some questioned locating it so
close to a previous terrorist target.
The FBI also had information problems. Analysts had difficulty getting access to the intelligence they were expected to analyze. The FBI’s information systems were woefully inadequate, lacking the ability to know what it knew. In July 1995, Attorney General Reno issued formal procedures aimed at managing information sharing between Justice Department prosecutors and the FBI. These procedures were almost immediately misunderstood and misapplied. Even the best information technology will not improve information sharing so long as the intelligence agencies’ personnel and security systems reward protecting information rather than disseminating it.
FBI Director Robert Mueller, who was in Birmingham, Alabama, said he was
"frustrated by the delays. I am frustrated that we do not have on every agent's desk the capability of a modern case management system." Sen. Patrick Leahy of Vermont, the ranking Democrat on the Senate Judiciary Committee, called the program "a train wreck in slow motion." Top FBI officials cited a wide range of reasons for the software-development problems (http://www.cnn.com/2005/US/01/13/fbi.software/index.html). The rapidly changing state of technology was insufficiently understood, and an entire system was developed to replace the antiquated FBI computer and record management systems.
With respect to communications that day, the September 11, 2001 Commission stated:
“Almost all aspects of communications continued to be problematic, from initial notification to tactical operations. Cellular telephones were of little value… Radio channels were initially oversaturated. Pagers seemed to be the most reliable means of notification when available and used, but most firefighters are not issued pagers.” Conversely, Al Queda managed
Page: 63 of 237
Thomas Virgona Doctoral Dissertation: Defense
information very carefully. Where electronic communications were regarded as insecure, Al
Queda relied even more heavily on couriers. This included personal transmission of the
September 11, 2001 attack date to Bin Laden, using courier ‘encryption:’ two sticks, a slash and a lollipop (11/9 or September 11, 2001).
2.8 Scholarly Literature; The Impact of September 11, 2001 in Various Disciplines
The full impact of the events that occurred on September 11, 2001 are impossible to define, and certainly is not the goal of this investigation. However, a literature review finds studies from virtually all areas of scholarly research. In the aftermath of the terrorist attacks and the anthrax threat, scientists and government officials have been forced to modify their activities in numerous areas. Delays in obtaining visas and reluctance to submit to some security measures have caused some scientists to miss important conferences. It has also led a significant number of international students to decide to study outside the United States.
Scientists are also disturbed by heightened restrictions on the distribution of information, a cornerstone of scientific research (Price 2004).
For planners and strategists, a common consequence of a major shock event like
September 11, 2001 is loss of perspective. The tendency is to over-estimate impacts in the short-term and under-estimate (or even ignore) impacts once the immediate shock recedes.
Yet achieving and maintaining a sense of balance in perspective is critical – all the more so with every indication that the world is headed into a prolonged period of acute turmoil and uncertainty (Kennedy et al 2003). The following sections (panic, altruism, disaster shock and psychological dependency, media, martial law, local decision makers, looting and price-
Page: 64 of 237
Thomas Virgona Doctoral Dissertation: Defense
gauging, and estimates of damage, injury and death) categorize the human aspects and reactions immediately following a disaster.
Panic: Even though the word “panic” was often used by mass media personnel and lay people to describe the escape of many running from the Twin Towers as they collapsed, a careful examination by Kennedy (2003) (via content analysis of the live video footage) of the behavior of these survivors indicates that they were rationally moving away from the obvious danger. Did they experience grave fear? Undoubtedly. However, they were not in a state of panic. They were rationally moving from point “a” to point “b” or from danger to a safe place.
Furthermore, conversations (while not a random sample) with survivors, who descended the stairways in the Twin Towers prior to their collapse, indicate that these individuals behaved in a very orderly, altruistic fashion. They helped one another down the steps. They preceded according to previous evacuation plans. They were calm and followed directions (Kennedy et al 2003).
Altruism: The New York Fire Department (NYFD) lost many brave members when the towers collapsed. Responding to the call as they were trained, these fire-fighters ascended the stairwells in the towers – and died doing their job. Others kept coming. In subsequent days, fire departments from around the country sent personnel too numerous even to be used at
“ground zero.” Other NYFD members did not want to leave the impact area and resisted efforts to give them relief – they stayed on the job, often to their own detriment. Individual citizens throughout the U.S.A. donated financial resources to help the victims. Citizens from varied backgrounds converged to offer their help, (e.g. medical personnel, counselors and therapists, average people seeking to help in any way they could). Altruism was extremely
Page: 65 of 237
Thomas Virgona Doctoral Dissertation: Defense
evident in the immediate post-impact period and the longer recovery period (Kennedy et al
2003).
Disaster shock and psychological dependency: Anecdotal evidence suggests that survivors assessed information as they obtained it and sought to take charge of their individual situations, not waiting for others to direct their behavior – except where it became necessary to do so. For example, one teaching assistant in a local elementary school received a cell phone call from her husband, who worked at the Trade Center, shortly after impact. He indicated that two planes had hit the Twin Towers and that he had evacuated to a safe site
“down the street.” As she was talking to him the towers collapsed and their phone line went dead. She was not certain of her husband’s fate (he survived), but refused to go home to wait for word about her husband, preferring, as she said, to “stay on the job where she is needed – with the children” (Kennedy et al 2003).
Media: The major network news reporters based in New York City (e.g. NBC’s Today Show personnel and the NBC Evening News personnel) functioned very much as local media do generally during disaster events – suspending normal programming and focusing on providing information about and for local people and organizations for local citizens and organizations
(live broadcasting from New York City was from both local stations and well as national networks) (Kennedy et al 2003).
Martial law: Even in this tragic event involving a “new” weapon of mass destruction, martial law was not declared. While the military offered to relieve the city of the burden of responding to the event, New York City officials declined the offer and continued to coordinate the organizational response. The original Emergency Operations Command
Page: 66 of 237
Thomas Virgona Doctoral Dissertation: Defense
(EOC) no longer existed, as it had been in one of the towers, and the city had to establish a new, emergent EOC. Despite all the challenges New York City faced, its response was heroic in proportions (Kennedy et al 2003).
Local decision makers: Local decision makers, e.g. the mayor’s office, sought to establish a command center (and, as noted above, they had to re-establish their EOC), designate a spokesperson to interact with the media, and update the community and nation at regular intervals. Perhaps benefiting from prior training sessions and drills, the mayor’s office in particular mastered the ideal model of providing regular briefings to the press (feeding them), during which they delineated what they currently knew and did not know. It would appear that the mayor became a role model for future decision makers in his gathering and dissemination of information (Kennedy et al 2003).
Looting and price gauging: The mayor’s office reported that crime in general was dramatically down during the aftermath of the terrorist event, despite a few reported instances of looting (Kennedy et al 2003).
Estimates of damage, injury and death: The usual pattern of overestimating the death toll is evident in the New York City Twin Tower collapse. The actual death toll appears to be less than half of the initial estimate, which is consistent with research findings for natural and technological disasters generally (Kennedy et al 2003).
In addition to Kennedy’s study cited above, there have been investigations of the role of safety distributors, manufacturers, and government agencies in ensuring the safety of rescue and recovery workers after the attack on the World trade Center (WTC) and the
Pentagon. One problem in the WTC collapse was the loss of an emergency management
Page: 67 of 237
Thomas Virgona Doctoral Dissertation: Defense
command center for the city. Additionally, in order to respond to major catastrophes, manufacturers need to produce the equipment quickly (respirators). It was recommended that distributors should be assisted with resources and the capability to get safety products to the right place, to the rescue workers (Herring 2002).
One obvious area of continuing investigation is architectural design. As an indication of how these events affected the entire globe, the future of tall concrete buildings construction following September 11, 2001 was discussed in 2002 at a seminar in Sydney, Australia. The seminar was jointly presented by the Concrete Institute and the Australian Pre-Mixed
Concrete Association. The presenters stated that structural engineers alone cannot solve the problem of protecting tall buildings against acts of terrorism and that it was more of a security issue (Woolnough and Paul 2002).
The health care field has been at the forefront of many of the pertinent studies. In the wake of the tragic events of September 11, 2001, there is an awareness of the critical role of health professionals in disaster response. Hospitals are one particularly important component of the public health system. How prepared are hospitals for a biochemical terrorist event?
What does the Internet offer health professionals for disaster readiness? The Joint
Commission on Accreditation of Healthcare Organizations (JCAHO) requires all hospitals to have an emergency management plan, and state licensure regulations often require a hospital's disaster plan to be comprehensive and filed with a state agency.
From an informational perspective, medical librarians need to be prepared to respond to the emergency information needs of patrons during such an event: the requests often are unique, urgent and sustained. In this case, a library "plan" and Internet resources are good starting points. When a disaster strikes, it's too late to start searching for the authoritative resources; they need to be ready and at your fingertips. Using a "checklist for medical
Page: 68 of 237
Thomas Virgona Doctoral Dissertation: Defense
librarians" or an "algorithm for disaster information preparedness," a medical librarian can assure readiness and offer resources and services that are coordinated and instantaneous
(Volesko 2002).
The collapse of the World Trade Center presented a health risk, as the buildings generated large amounts of dust and smoke that settled in the surrounding indoor and outdoor environments. Two to three years after September 11, 2001, survivors of buildings that collapsed or that were damaged reported substantial physical and mental health problems
(Center for Disease Control and Prevention 2006). The long-term ramifications of these effects are unknown. Many survivors were caught directly in the dust and debris of collapsing towers, a dense cloud of particulate matter that might have produced or exacerbated these health effects. During December 2001, a field study of 183 clean-up and recovery workers at the World Trade Center (WTC) disaster site was conducted to assess respiratory health effects potentially resulting from their work. A questionnaire was administered to assess upper respiratory symptoms and lower respiratory symptoms, including cough, phlegm, and wheeze, as well as indices of exposure, including number of days worked at the site and job category.
Of this group, 34% developed cough, 24% developed phlegm, and 19% developed wheeze
(Herbstman et al, 2005). The Center of Disease Control and Prevention is conducting a longitudinal study on the effects on the survivors of World Trade Center collapse. The report includes data from health interviews conducted from September 5, 2003, to the close of the
World Trade Center Health Registry (WTCHR) enrollment on November 20, 2004. The
WTCHR will be used to monitor the mental and physical health of 71,437 enrollees for 20 years. Though the analysis includes 8,418 adult survivors of collapsed buildings, the total number of survivors is unknown. (Center for Disease Control and Prevention 2006). A total of 62.4% of survivors of collapsed or damaged buildings were caught in the dust and debris
Page: 69 of 237
Thomas Virgona Doctoral Dissertation: Defense
cloud of the WTC towers, and 63.8% experienced three or more potentially psychologically traumatizing events. Injuries were common (43.6%), but few survivors reported injuries that would have required extensive treatment. More than half (56.6%) of survivors reported respiratory symptoms after the attacks, 23.9% had heartburn/reflux, and 21.0% had severe headaches. At the time of the interview, 10.7% of building survivors screened positive for serious psychological distress (SPD). Building type and time of evacuation were associated with injuries on September 11, 2001 and reported symptoms; building type (collapsed versus damaged) also was associated with mental distress (Center for Disease Control and
Prevention 2006).
The field of physical security has also grown dramatically after September 11, 2001.
Biometric technologies are currently at various stages of development (e.g., integrating automated face recognition systems with other biometric technologies). Event operators will be able to provide higher levels of security for spectators and participants. The ongoing threat of terrorism targeting high profile events must be framed within the context of “when,” not
“if.” Venue security will continue to be a critical component of the spectator’s experience
(Whisenant 2003). While it may be impossible to eliminate the distractions caused by increased security at venues, those distractions can be minimized over time. Venue operators have a moral obligation to ensure their facilities are as safe as possible for both spectators and participants (Kalinsky 2002). By educating all personnel involved, both spectators and venue staff, the level of security at events will be enhanced. While a venue cannot be both open and secure, the operator can provide an “open looking” venue.
The ferocity and size of the terrorist attacks overcame any reluctance by governments to commit to legislation against money laundering (Johnson 2002). In the aftermath of the
September 11, 2001, terrorist attacks, Congress and the President acted swiftly to pass new Page: 70 of 237
Thomas Virgona Doctoral Dissertation: Defense
legislation to make it easier for federal, state, and local law enforcement to investigate and avert suspected acts of terrorism. On October 26, 2001, President Bush signed into law the
USA Patriot Act of 2001. The Patriot Act had the paramount goal of enhancing the ability of
America's intelligence and law enforcement communities to prevent terrorism. The Act provides scores of amendments to existing federal statutes, all of which, to varying degrees, remove obstacles to investigating terrorist acts and acts related to or in furtherance of terrorism.
At the same time, the Patriot Act's broad amendments present a number of new concerns for all businesses operating in the Internet age. As critical financial and technological infrastructures continue to serve legitimate and illicit needs (from legitimately driving America's economy to serving as the platforms for illegal international money laundering by terrorists) business owners face the threat of a dragnet approach to surveillance and investigation of suspected terrorists and terrorism. To ensure they are prepared to withstand the tightrope walk of balancing overwhelming national security concerns with their own network integrity and privacy issues, all businesses that rely in any way on computers should understand the full implications of the Act (Rush and Paglia 2002).
The Patriot Act continues to be very controversial. The American Civil Liberties
Union (ACLU) has challenged the Patriot Act in court with two cases: one involving an
Internet Service Provider (ISP) and the second a group of librarians. In both cases, the judges ruled that the gag rules were unconstitutional.34 Monitoring of Internet data is now legalized.
All major ISPs have installed surveillance software to monitor e-mail messages and store records of Internet activity by people suspected having suspicious foreign contact (Nijboer
2004). The Patriot Act expands the kind of information the authorities want and they can
Page: 71 of 237
Thomas Virgona Doctoral Dissertation: Defense
force ISPs, libraries and other organizations, without court approval, to hand over Internet data, such as records of Web sites visited. The act has a section that allows ISPs to disclose the content of their customer’s messages at the request of federal or local authorities if, “in good faith” they think this will prevent death or serious injury – no court involved, a kind of self regulation, in which the ISPs decide when and what to disclose to the authorities. A key concept is “good faith” rather than “reasonable belief”. The same section allows police to record, without permission, any message sent or received by a “protected computer” which is under attack. The new law allows a person's record in libraries, universities and health organizations to be accessed by the FBI. The organizations have to cooperate and are not permitted to inform the involved employee or client. Activists and dissidents are quite worried about the far reaching authority of the FBI (Nijboer 2004). Nijboer illustrated some examples below:
• Marc Schultz, bookseller in the Chapter 11 store in Atlanta went to a coffee shop
before going to work. He was standing in line for a cup of coffee and reading an
article “Weapons of Mass Stupidity” in the Weekly Planet of Tampa. Somebody
behind him saw what he was reading and called the FBI. Three days later two FBI
agents visited the bookstore.
• A bookstore owner purged customer files because he wanted to protect them against
the Patriot Act.
• Student Andrew O'Connor was interrogated by Albuquerque police and the Secret
Service in February 2003. O'Connor was removed from the college library by police
after he made negative comments about President Bush in an online chat room. He
34 Available at: http://www.aclu.org/safefree/nationalsecurityletters/index.html. Page: 72 of 237
Thomas Virgona Doctoral Dissertation: Defense
ultimately was released without being charged. What he said, how the police and
Secret Service knew he said it, and the gag order on the college to keep people from
talking about his arrest, are all shrouded in silence.
The September 11, 2001 terrorist attacks on the World Trade Center and the Pentagon and the subsequent advice of the U.S. Attorney General encouraging U.S. companies immediately review and reinforce their security programs has raised a number of questions regarding the status and viability of existing life-safety and business continuity planning processes within the public and private organization infrastructures of the U.S.A. The following Computer Security Institute checklist attempts to clarify answers to several of these questions for assessing current continuity and emergency response planning, and suggests approaches to undertaking a rapid development effort and/or improving existing implementations (Jackson 2002).
Specifically, for finance and banking, the checklist includes:
• Financial institutions that recognize the effects of concentration and interdependence
in the sector will be better placed to manage the operational implications of disasters
in other parts of their sector.35
• Comprehensive business continuity plans that include evacuation, electronic and
physical backup arrangements will enable most organizations to relocate and re-
establish operations following a disaster.
• The business continuity planning of financial institutions will be improved by
accounting for disaster scenarios that involve consequences that could harm or render
inaccessible critical employees.
35 This is in contrast to what Horton wrote in 1985: Control systems rely more heavily on internal data company data than external data. Page: 73 of 237
Thomas Virgona Doctoral Dissertation: Defense
• Institutions that address the information, communication and personal needs of
employees during a crisis will further the efficiency of their business resumption
plans.
• Business resumption plans and hot site models that take into consideration disasters
involving broad geographic areas (such as a district, city or region) will be better
equipped to cope with the impact of such disasters on business recovery efforts.
• Stakeholders who share recovery plans and contact information with one another will
be better able to establish crisis management communication in the event of a disaster.
• Advanced planning and communication between sectors will help to minimize the
impact of interdependencies on business continuity plans.
As this checklist circulates throughout the finance and banking sector, further research will be required to determine how effectively these suggestions were implemented.
2.9 September 11, 2001: Information Technology and Disaster Recovery
Following the 1993 bombing of the World Trade Center (WTC), terrorism and security experts agreed that the U.S. financial services industry was a prime target for future terrorist attacks. Experts warned that the financial industry’s disaster recovery plans were out-of-date, designed primarily to withstand natural disasters, and were no match for the destructive power of an intentional terrorist attack (Beacham and McManus 2004). Sikich believes one underlying vulnerability issue for organizations continues to be the assumption that threat,
Page: 74 of 237
Thomas Virgona Doctoral Dissertation: Defense
hazard, risk, and consequence-assessment are one and the same. These elements are intertwined but are distinct and different (Sikich 2003).
While the tragic events of September 11, 2001 confirmed experts’ foreboding predictions of attacks on the U.S. financial system, was the financial services industry inadequately prepared to recover from such an attack? As the financial services sector, and the securities industry in particular, were heavily concentrated within the World Trade Center towers, several such firms have emerged as those “hardest hit” by the September 11, 2001 attack (Cantor Fitzgerald; Keefe, Bruyette & Woods; and Sandler O’Neill & Partners). While the financial services industry as a whole made great strides in recovery and continuity planning regarding data and data systems, the attack on September 11, 2001 revealed inconsistency in the level of disaster recovery preparedness at individual companies. While
Cantor Fitzgerald had duplicate systems in place so that its data system never went down, smaller companies, like Sandler O’Neill were less prepared, and had to rebuild their IT system from scratch. What the attack on September 11, 2001 made tragically apparent, however, was the industry’s grossly inadequate preparation for the tremendous loss of human capital. While existing recovery plans assumed the safety of company personnel, on
September 11, 2001 several companies literally lost their entire disaster recovery team. The bottom line is that the attack on the World Trade Center exposed both the financial services industry’s reliance on human capital and its inadequate preparation to recover from such a loss (Beacham and McManus 2004). Johnston and Nedelescu (2006) wrote that the economic consequences can be largely broken down into short-term direct effects; medium-term confidence effects and longer-term productivity effects. The direct economic costs of terrorism, including the destruction of life and property, responses to the emergency, restoration of the systems and the infrastructure affected, and the provision of temporary Page: 75 of 237
Thomas Virgona Doctoral Dissertation: Defense
living assistance, are in the short run. The medium-term impact is the indirect costs to affect the economy by undermining consumer and investor confidence. Over the longer term, there is a question of whether the attacks can have a negative impact on productivity by raising the costs of transactions through increased security measures, higher insurance premiums, and the increased costs of financial and other counterterrorism regulations.
Connell (2001) discussed the lessons learned form the 1993 World Trade Center terrorist attacks and the impact on the September 11, 2001 event. Panic was not widely observed during the evacuation of the Twin Towers, as the evacuation experiences of many of the workers in the buildings in the 1993 attack may have had an impact on their decision- making process during the September 11 disaster. Improvements made following the 1993
World Trade Center attack contributed to a more successful evacuation. These improvements included the addition of battery-powered lights and glow-in-the-dark paint in the stairwells, the appointment of floor marshals to guide the evacuation process, and redesigned emergency plans (Connell 2001). Many survivors cited the improved conditions in the stairwells during the September 11 evacuation. As one survivor observed, despite the magnitude of the recent terrorist attacks on the WTC, the evacuation process did not seem as dire as the evacuation following the 1993 attack due to the improvements in ventilation and lighting. Many organizations located within the WTC significantly improved their evacuation plans following the 1993 terrorist bombing. Individuals were more likely to decide to evacuate the premises if they experienced visual or sensory clues that suggested the dangerous nature of the event.
Examples of visual or sensory clues that were cited by survivors in their accounts included smoke, fire, water from the sprinkler systems, debris, structural failure, shattered glass, and the impact of the plane collision (Connell 2001).
Page: 76 of 237
Thomas Virgona Doctoral Dissertation: Defense
Over the past two decades, information technology has become increasingly integrated into the day-to-day operations of most financial service organizations. A common phrase today is “ubiquitous” computing. The dependability and continuity of information infrastructures can be a determining factor in how well an organization will be able to respond to a catastrophic event. Although many lessons can be identified, they emphasize three general principles: the establishment and practice of comprehensive continuity and recovery plans, the decentralization of operations, and the development of system redundancies to eliminate single points of weakness (Seifert 2002).
The events of September 11, 2001 also highlighted an increased need for information technology security not only for New York/Washington business end federal executives, but for other state government executives as well. This increased urgency and heightened awareness left many of Virginia's government executives asking the question, “How secure and prepared is the Commonwealth to deal with information security attacks” (Redwine
2002). Even more alarming is that in 1993, the World Trade was the primary target for another terrorist attack. Yet, many organizations were still unprepared.
Looking back to another disaster, the 1994 California earthquake, the reliance on human resources was obvious. As one information systems manager stated afterwards, “No matter how well you plan it, you assume that people are going to follow the instructions per the plan, but what people think about first are their families” (Bozman 1994). The earthquake, registering 6.6 on the Richter scale, hit the suburbs north of Los Angeles. Charles
Peruchini, director of information systems at aluminum wheel manufacturer Superior
Industries International, Inc., worked less than five miles from the quake's epicenter, and the company was completely shut down for two days.36 Six U.S. sites depended on that data
36 Article available at: http://www.computerworld.com/news/1994/story/0,11280,17647,00.html. Page: 77 of 237
Thomas Virgona Doctoral Dissertation: Defense
center for computing services. Peruchini was the only IS staffer to make it into the data center that morning. The systems manager had slashed his leg on broken glass, and the programming supervisor could not drive her car out of a smashed garage. Others could not be reached by phone, and power was out throughout the area. His “day” ended 39 hours later.
The September 11, 2001 attacks destroyed 13.4 million square feet of World Trade
Center office space from the Lower Manhattan market, space that is not likely to be replaced in this decade. This inventory destruction, when combined with the sheer physical disruption to corporate operations and the traumatic human losses, is likely to cause renewed long-term decentralization in the region (Hughes 2002). A new location strategy, particularly for the region's major financial firms, appears to be emerging: the avoidance of excessive concentration of human and physical resources (Hughes 2002). Wall Street in many ways is a unique environment. Logistically speaking, major firms are co-located in close proximity to one another - in some cases, in the same building. Also, despite intense business competition, there is a strong inter-firm dependency on each other to conduct business. Companies have learned to integrate internal and external information efficiently (Horton 1994a). Many large deals require several firms to be involved, and therefore, there is a reliance on other firm’s technology for information and funds distribution. This implies a technological and regional decentralized framework that goes beyond the dispersion of back office functions to the segmentation of a firm's higher-level activities. This is a different strategy of place, one of distributed workplaces on different power and telecommunication grids to ensure business continuity in the face of disaster. The suburban ring, Midtown Manhattan and select core locations such as Jersey City, should be the economic beneficiaries of this change, adding pressure on the suburban ring's congested highway infrastructure as well as on the overburdened rail lines converging on Midtown Manhattan (Hughes 2002). Page: 78 of 237
Thomas Virgona Doctoral Dissertation: Defense
Business continuity and disaster recovery have been become a higher priority for financial services firms in the years since September 11, 2001. Terrorist threats pushed-up global institutions' projected IT spending on operational recovery. The industry closed operational gaps since the destruction of September 11, 2001. These deficiencies included serious weaknesses in business continuity plans, including the need for geographic dispersion of offices, employees and business processes, as well as redundancy of supporting infrastructure, like telecommunications networks and power supplies. Money has been spent on backup systems, storage units, and remote-mirroring technologies. Many have set up remote-workforce operational-resilience plans to ensure that work can be done at satellite offices and other sites (Krebsbach 2004). Oz researched firms directly impacted by the terrorist attacks (Oz 2003). The impact of not having a disaster recovery plan is clear: two of the four companies that did not have a business continuity plan have not regained their business potential.
The need for every firm to have a disaster recovery plan would seem clear - and yet
Michael Breier, a CPA at the Chicago-based firm of Shepard Schwartz & Harris, believes that the importance of these plans is often understated. Mitchell Freedman founder of Mitchell
Freedman Accountancy Corp. in Sherman Oaks, California, points out that many parts of the country have been subjected to natural disasters for eons (Kahan 2005). Since the terrorist attacks of September 11, 2001, Freedman said that it has become undeniably clear that any part of the country can be subjected to a disaster, including man-made ones. Freedman noted that inasmuch as there could be a loss of office space, important papers, computer equipment and electronic data, not to mention key personnel, it is vital that owners and managers do themselves, their customers, their staff and the economy a service by planning for the possibility of a disaster, so that they can be up and running as quickly and practically as
Page: 79 of 237
Thomas Virgona Doctoral Dissertation: Defense
possible should a catastrophe occur (Kahan 2005). Organizations need to take responsibility for their own protection, achieve cooperation with other parties involved, and understand that they are facing an increasingly dangerous e-commerce environment requiring them to have knowledge of the weaknesses of the systems upon which the firm is dependent.
The law firm Sidley Austin Brown & Wood LLP (SAB&W) was hit directly and is an example of a firm that did not have a comprehensive disaster recovery plans in place. By the afternoon of September 11, 2001 a partner had to locate additional office space in the Third
Avenue building to accommodate 600 additional people. The purchasing department got on the phone and bought new computers from Dell Systems to be delivered to the Third Avenue office. They also acquired used furniture for offices and hotel space in New York for out-of- town personnel and those unable to return home (Barr 2003).
The human aspect of restoring systems to financial services may best be illustrated by
Lehman Brothers. Lehman Brothers had three floors of IT developers at 1 World Trade
Center, the North Tower, as well as a 20,000-square-feet, at nearby 3 World Financial Center.
"Part of what I'm dealing with here is that I had a 150-person technical staff in the Trade
Center," said Bob Schwartz, CTO and managing director for Lehman Brothers (Grygo 2001).
"Those people, including myself, were in the building struck by the first plane. We had to march down 40 flights of stairs and then escape." The other 150 IT staff members watched from across the river at Lehman's second data center in Jersey City, N.J. "So, I have 300 fairly traumatized people, all of whom are now working very, very hard, close to 24 hours a day, to insure that everything runs the way it's supposed to." There were minor amounts of data that were lost, but what they hadn't counted on were the losses in human expertise from the IT and business sides of their operations (Grygo 2001).
Page: 80 of 237
Thomas Virgona Doctoral Dissertation: Defense
This will become more important with the international expansion of Internet access.
The explosive growth of the Internet suggests an equally explosive growth in the ranks of those with the skills required to cause significant harm to information systems (McCrohan
2003). This international expansion of Internet use, combined with the increasing power of hacker tools and their increasing ease of use, argues for the permanence of a constant cyber threat. In this environment, the technologically sophisticated nations of the world and their corporations, will increasingly find themselves under attack from political malcontents (as described by the September 11, 2001 Commission Report), for whatever real or perceived wrongs. Added to this mix will be the commercially or economically motivated, the unscrupulous, and the cyber terrorists. These threats to the expansion of e-commerce are present and real and will exist without the emergence of a major countervailing power to the
USA. In that it does not appear that any system can be truly secure, the involvement of senior managers in risk awareness and risk assessment initiatives, along with increased cooperation with the public sector, is required to achieve a realistic level of security (McCrohan 2003).
September 11, 2001 changed how Greg Burnham thinks about things as simple as paper and as priceless as people. Burnham, chief technology officer at the Port Authority of
New York and New Jersey, walked down 70 flights of stairs that day to escape his office in the north tower of the World Trade Center (Garvey and McGee 2002). The disaster made
Burnham realize just how unprepared the Port Authority was, despite extensive disaster planning. Half of companies in the Garvey and McGee study have significantly altered their approaches or policies related to business-continuity planning since September 11, 2001.
Some businesses are like the Gates Group of Companies in Denver, which is completely rethinking its approach to minimize disruptions to its business (Garvey and McGee 2002).
Page: 81 of 237
Thomas Virgona Doctoral Dissertation: Defense
At the Financial Industry Summit on Business Continuity, Roger Ferguson, vice chairman of the Federal Reserve Bank of New York, outlined three areas of vulnerability that regulators and financial firms have found following September 11 (Wall Street Technology
2002).
1) Contingency planning generally did not account for region-wide events and some
firms found they lost both primary and back-up sites. In addition, there were
concerns about the loss or inaccessibility of staff.
2) Market-based and geographic concentrations were readily evident and became a
source of vulnerability.
3) Critical interdependencies across the industry were apparent. This became evident
in the impact of key infrastructure providers on a wide range of financial
institutions. Even institutions removed from the New York City area were
significantly affected by interdependencies.
Some questions discussed with participating firms in the summit, in order to produce a
"sound-practices" approach to Business Continuity Planning (BCP) included:
• What are the costs and benefits of the different approaches to BCP and what would
lead firms to adopt one or the other?
• What are the key practices on which the industry should focus in the effort to
strengthen BCP?
• What are reasonable recovery times for critical operations and do these need to be
consistent across the industry?
• What decisions are firms making now and how prepared is the industry today?
• What can be done to improve crisis management?
Page: 82 of 237
Thomas Virgona Doctoral Dissertation: Defense
Testing execution and test walkthroughs and for the management of the test is recommended (Neal 2003). “Disaster recovery” tends to conjure up ideas of major disasters
(terrorist attack, hurricanes and floods), but often it is a mundane incident such as a power failure that can bring down a business. Fundamentals, such as regular backups of data and storage systems, to ensure you can continue with business as usual, whatever hits you. The thread that runs through all forms of disaster and business continuity planning is the need to ensure continued access to information and data. In practice for global corporations, this translates into managing some form of backup or replica copy of that data, which inevitably means addressing the potential impacts of the United Kingdom Data Protection Act (DPA)
1998 (and related legislation).
The Data Protection Act of 1998 came into force on March 1, 2000, and regulates the
"processing" of "personal data" (Wingfield 2002). "Personal data" means any information whereby a living individual can be identified. "Processing" means virtually any activity, such as obtaining, recording or holding the data, carrying out operations or sets of operations on the data, organization, adaptation or alteration of the data, retrieval, consultation or use of the data and alignment, combination, blocking, erasure or destruction of the data (Wingfield 2002). It applies to data processed electronically, and to some manual (e.g., physical) records (Crichard
2004). The act places restrictions on organizations which collect or hold data which can identify a living person. Data collected by any person or organization may only be used for the specific purposes for which they were collected. Personal data may only be kept for an appropriate length of time and must not be disclosed to other parties without the consent of the data owner. Schools, for example, may decide to keep information on former pupils for no longer than ten years. The UK Data Protection Act is large and has a reputation for complexity. The requirement that researchers use “intermediaries” to obtain consent from and Page: 83 of 237
Thomas Virgona Doctoral Dissertation: Defense
recruit subjects for studies increases the risk of selection bias, may expose the practitioner to ethical difficulties and may compromise the external validity of trial results. There is also a danger that research costs will soar when the Data Protection Act (1998) is fully realized
(Redsell and Cheater 2001).
The need to improve the reliability of IT systems, and in particular the ability of companies to recover their data following a problem, has driven companies to innovate around disaster recovery and backup technologies. Innovations such as snapshotting,37 shadowing, backing up data to disk rather than tape and the development of mirrored systems have all contributed to greater availability and reliability in data centers.
Businesses correctly plan for a catastrophic loss, but in practice, a number of smaller business process or IT systems failures can be almost as damaging in terms of lost business or reputation, especially when customer data is exposed. For example, the United Parcel Service lost a box of computer tapes containing information on 3.9 million Citigroup customers while the tapes were in transit to credit reporting agency Experian in Allen, Tex from Weehawken,
NJ. This resulted in “reputation” damage to Citigroup.38 It is the smaller incidents, left undetected and under-prepared for, that can undermine business continuity plans. Frequently, data integrity is threatened during periods of organizational change. Thus, BCP becomes inherent in all business planning. Business continuity plans cannot solely be the responsibility of the IT function. Business leaders need to establish objectives and priorities for recovery and downtime that can then be used to inform infrastructure investments. And as with all plans, it is important to test them. Again, this involves groups across the organization, and not just the IT department. At the same time, limited budgets make
37 Snapshotting, shadowing and mirroring are all techniques for replicating data onto an alternative sites. Snapshots of the database may be sent to backup hardware during pre-determined intervals. Shadowing and mirroring database functions will copy transitions to another server either simultaneously or after the original transition has been completed. 38 Article available at: http://select.nytimes.com/gst/abstract.html?res=F30E1EF7395C0C748CDDAF0894DD404482. Page: 84 of 237
Thomas Virgona Doctoral Dissertation: Defense
mitigating such risk a major challenge. The chances of a business-crippling catastrophe, such as the data center burning down, may be small, while the inevitability of a server or disk drive failing at some point needs to be balanced against the likely lower impact on the business.
Promoting the disaster plan must be central to disaster preparation (Muir and Shenton 2002).
The most useful part of the plan is its contacts list.
Spending on business-continuity and disaster-recovery planning is poised to grow following terrorist threats against financial-services firms. Unlike other IT areas, where growth in spending by financial-services institutions tends to be stable, business-continuity spending spikes as a result of crises (Marlin and Garvey 2004). The financial-services industry swiftly responded to the elevation of the terrorist threat, setting in motion a full-scale crisis-management plan. BITS, a banking-industry group that has taken a lead in formulating crisis-management plans, is taking numerous steps to bolster its already strong business- continuity efforts in the event of a large-scale disaster (Marlin and Garvey 2004).
Crises are likely to become increasingly common events in the life of organizations
(Seeger 2005). Managers, therefore, will be required to respond to the radical changes, disruptions and uncertainty imposed by a crisis. Conventional wisdom suggests that crises are primarily negative events creating severe hardship and organizational decline. The ability of organizations to survive, rebuild and even renew themselves will depend on the ability to learn from these events and create a new sense of normal (Seeger 2005). This discourse of renewal creates an opportunity after a crisis to fundamentally re-order the organization down to its core purpose. Crises, in these cases, create not only severe devastation, but a unique
Page: 85 of 237
Thomas Virgona Doctoral Dissertation: Defense
opportunity for systemic change and fundamental re-invention (Seeger 2005). In normal times, such fundamental change would require long-term strategic efforts as well as major investments of time and resources without guaranteed success. “Crises, however, disrupt the status quo in basic ways allowing for new assumptions, methods and organizational values to emerge. Many outdated assumptions, impediments, inertia and political resistance to change are removed during a crisis (Seeger 2005).” Attention and energy are focused on the immediate and obvious need. While Cantor Fitzgerald remains a company cut from the tough tradition of Wall Street, its culture and mission have been fundamentally altered by the events of September 11, 2001 and the resulting effort to provide support to the families.
September 11, 2001 was not only a time of destruction and loss, but also a time of change and renewal (Seeger 2005).
Looking back in history, the concept of a corporation possessing security forces and even carrying out military operations is best exemplified by the British East India Company
(Roukis 2004). It dealt with a variety of challenges, including military conflict with Portugal,
Holland, and France over territorial acquisitions in the East Indies and the Indian subcontinent, addressed global piracy (the global terrorism issue of the 16th and 17th centuries) and successfully suppressed the feared Thuggee organization (1830-1837). The company administered an effective information gathering system, ran spies, conducted diplomacy and understood the importance of informers in rooting out the terrorists. The
British could not deploy regular military forces to the subcontinent because of events on the
European continent, and thus saw that company employees, capable of self-defense in case of war and financially independent, were vital. This became the operational foundation of the
British East India Company. One important lesson was learned: Trade must be protected by
Page: 86 of 237
Thomas Virgona Doctoral Dissertation: Defense
soldiers, guns and armed fleets (private forces) (Roukis 2004). In the post September 11,
2001 world, where a premium is placed on timely information and where networks and media are driving forces in the transformation of organizations and society, corporations will assume greater responsibility for functions. These functions include intelligence acquisition, law enforcement, and military projection (Roukis 2004).
Today’s banking and finance firms face new threats to safety and business stability
(Sikich 2003):
• Deregulation and increased competition that can lead to the capability to
address security concerns.
• Convergence of computing, networking, communication and encryption
technologies increases flexibility, but can also add vulnerabilities.
• International commerce gives non-domestic entities unprecedented access
to Unites States information.
As a result, the new challenges are:
• Making policy tradeoffs to balance competing markets and regulators and
security.
• Adopting the two technology revolutions: information and
communications.
• Defining new assets that need protection.
September 11, 2001 will be indelibly stamped upon all our memories. It taxed our emotional foundation and tested the very mettle upon which our country was built. It also Page: 87 of 237
Thomas Virgona Doctoral Dissertation: Defense
provided a most compelling reason for having a viable and executable process. No longer will most corporate executives question the need for recovery planning. In fact, the very opposite is unfolding in corporations around the world. Boards of directors are now asking for an assessment of recovery capabilities and want to understand their corporate exposure. With this new emphasis on business continuity planning and disaster recovery (traditionally the information technology or IT portion of business continuity planning), there is a sense of urgency to take steps to be prepared for the next crisis. What we learned from September 11 should serve as a guideline for addressing these requirements. What worked and what did not work are lessons that should be examined and applied to our planning process (Berman 2002).
In summary, historical events have changed information systems numerous times.
Three examples are cited here: the printing press, Sputnik and the Internet. The events of
September 11, 2001 have had a far reaching impact on everyday life in America; information technology is not exempt from these changes. In many respects, September 11, 2001 represents the first “true” disaster, as many of the situations were never encountered before.
Despite previous terrorist’s attempts to strike at the financial center of America, Wall Street was generally unprepared for the devastating events of that day. Wall Street financial firms, which are critical information systems providers to the globe, were forced to re-evaluate disaster recovery plans to ensure continuity of business during systems outages.
Page: 88 of 237
Thomas Virgona Doctoral Dissertation: Defense
3 Study Design and Methodology
3.1 Research questions
This research proposes to examine the reliance on information systems technologists on September 11, 2001 and the impact this reliance may have had on disaster recovery. The research indicates that disaster recovery plans were heavily reliant on human capital and expertise, and required critical information service providers (e.g., the financial services industry) to reexamine existing contingency plans.
This research proposes to examine the following question and map the finding and map the finding to Distributed Cognition theory:
• Were information systems on Wall Street negatively impacted by the events of
September 11, 2001?
• What happened to the systems that day and how did information systems technologists
react?
• What changes to the SDLC (specifically humans role in disaster recovery design
planning have been implemented since September 11, 2001?
• What lessons were learned?
For this dissertation, the research question and sub-questions will be addressed via a
“grounded theory” approach, not to be confused with theoretical grounding. Grounded theory builds up to a systematic theory inductively, “grounded” in, or based upon, the observations.
The observations are summarized into conceptual categories, which are tested directly in the research setting with more observations (Schutt 1999). A grounded theory approach is to be
Page: 89 of 237
Thomas Virgona Doctoral Dissertation: Defense
used when there is no testable hypothesis. No formulated testable hypothesis is appropriate since there is insufficient information available. A research problem (Sproull 2002):
• Is prompted by recognition of some existing problem for which insufficient
information is available (e.g., Were there disaster recovery issues on September
11, 2001 for financial services firms relating to personal?).
• Stems from the desire to improve the status quo (e.g., The current Systems
Development Life Cycle includes disaster recovery planning in the design phase.
How should this be updated based on the lessons learned on September 11, 2001?).
• Originates from the process of short- and long-range planning for the future (e.g.,
Do people and their skills need to be specifically accounted for in disaster recovery
planning?).
As the System Development Life Cycle continues to evolve into a mature methodology, phases of the Systems Development Life Cycle will change over time. One of the critical SDLC Design phase components, the Disaster Recovery or Continuity of Business plans, may have been impacted by the events of September 11, 2001. Substantial information systems failures are more common than information system professionals would like to admit
(Lytle in Horton and Lewis 1991). While building information systems, Lytle points to these development “disasters:”
• Develop of non-strategic systems.
• Focus on technical issues.
• Define few requirements.
• Confuse user and technical responsibilities.
• Use computers to solve management problems.
Page: 90 of 237
Thomas Virgona Doctoral Dissertation: Defense
• Select hardware first.
• Buy software package first.
• Hire a vendor on time and materials.
• Implement with a big bang.
In many respects, those practices represent the first true “disaster” encountered by the financial services industry. Previous scenarios, such as weather, market conditions, and even blackouts were thoroughly tested and vetted within the Wall Street firms. However, the terrorist attacks on September 11, 2001 presented uncharted challenges never planned for in existing disaster recovery or contingency plans. Additionally, Wall Street financial firms are key global information (market data, interest rates, personal financial information, etc.) service providers. Some information is more critical or central to the fabric of a society
(Richard A.V. Diener in Horton and Lewis 1991) and needs to be provided on-demand globally in a timely and accurate manner. The events of September 11, 2001 rendered Wall
Street financial firms incapable of producing and disseminating that information.39
The research has indicated that one of the major shortcomings in the Disaster
Recovery (DR) or Continuity of Business (COB) design was the reliance on humans to ensure that company’s information infrastructure was restored to an operational status.
Subsequently, when people could not be located, or in some cases, entire DR/COB departments were killed, restoration of these services failed. Much of the information about the extent of the disaster is still not in the public domain. Oz, in a previous studied found four major reasons for this situation (Oz 2003):
• The organization has a policy not to participate in any research survey.
• The organization considers the data confidential despite confidentiality guarantees.
Page: 91 of 237
Thomas Virgona Doctoral Dissertation: Defense
• There is a lack of time to fill out the questionnaire.
• The data are not available.
The purpose of this research is to identify if there was a failed reliance on human resources in disaster recovery plans on September 11, 2001. As there is insufficient information to form a hypothesis, only research questions will be addressed in this study.
This research specifically explores the “human” factors involved in information technology disaster recovery design as a result of September 11, 2001. As a result of
September 11, 2001, many technical changes were also made to the design phase of the
SDLC. For example, where can a data center’s disaster recovery site can be located? This research will not address technology related changes, but will focus on the human factors involved in designing recovery of information systems during a disaster.
Kenneth Hewitt has studied disasters in recent years and has found the most important insights come from the workers on the ground (Quarantelli 1998). Specifically, the most knowledge comes from those on the front-line of a disaster reflecting upon field conditions.
Those who speak the language and have some depth of knowledge of the culture provide essential insight (Quarantelli 1998). The methodological challenge of disaster recovery studies is to pay attention not just to the local conditions, but to the voices of the persons involved. Robert Stallings described the challenge of researching disasters: there are many of empirical studies, but less certainty as to what they add up to (Quarantelli 1998).
39 John Weitzel and Donald Marchand documented another major system malfunction during the US Stock Market Crash of 1987 which impeded information flows, which played a major role in bringing the financial markets to the edge of collapse (Horton and Lewis 1991). Page: 92 of 237
Thomas Virgona Doctoral Dissertation: Defense
The work of recovering and listening to oral testimony from the victims of a disaster goes to the heart of the problem of finding the human and social risks. Meyer and
Poniatowska (1988) wrote of how oral testimony identifies humanitarian perspective of harm:
“oral history not only shares the fate of the vanquished, but is also born at the moment of disaster and of collective social forgetfulness” (Meyer and Poniatowska 1988). The nuances and symbolic meaning require careful, and cautious, attention (Hewitt in Quarantelli 1998).
To focus on their words is to recognize the plight of the victims of a disaster (Hewitt in
Quarantelli 1998).
This research intends to explore the September 11, 2001 disaster recovery events on
Wall Street and identify any patterns observed. Categories will be discovered, if they exist, by examination of the data. Previously unstudied areas cannot be adequately researched with a structured set of questions. Therefore, a qualitative study is best suited for this purpose.
The focus group and interviews will provide a forum to provide a consistent understanding of the terms and identify the full range of events of that day. The study will also be grounded on
Hollan’s views on Distributed Cognition.40 Distributed Cognition recognizes the critical human elements in the field of human computer interaction, the critical foundation of this research.
40 Section 1.2 Theoretical Framework describes this theory in detail. Page: 93 of 237
Thomas Virgona Doctoral Dissertation: Defense
3.2 Operational definitions and limitations
Operational definitions document the intended meaning of a concept in relation to a particular study and provide some criteria for measuring the empirical existence of that concept. In defining a term or concept, a researcher is declaring what the term is to mean throughout the research (Berg 2001). As is the example is the various definitions of” poverty:”
• Subjective poverty: “Would you say you are poor?”
• Absolute poverty: Family Income / Poverty threshold
The definitions that will be used in this study are presented below:
Table 2 - Definition of Terms Used in this Dissertation Term Definition
September 11, Events related to the terrorist attacks on the World Trade Center and
2001 Pentagon on September 11, 2001. The time period to be reviewed is the
one week recovery period following September 11. The rationale is that
this was the time required for the New York Stock Exchange to re-open
and resume operations.
Page: 94 of 237
Thomas Virgona Doctoral Dissertation: Defense
Term Definition
Call Tree A document that depicts the calling responsibilities and order used to
contact management, employees, customers, vendors, and other key
personnel in the event of a disaster.
Continuity Of Business continuity is a plan that contains a detailed specification of the
Business (COB) main IT systems and the supporting local and wide area network
Activity infrastructure. The documentation should make it clear which key
business processes and functional activities are dependent on each of the
systems. Because IT changes fairly often and fairly rapidly, it is
important that the system specifications and configurations are kept up to
date as systems are upgraded or modified.41
COB Awareness In order to enhance and strengthen the Continuity Of Business (COB)
Training awareness, the training covers general COB concepts with which all
employees should be familiar with, as well as information on available
resources and key contacts.
41 Existing, normal IT back-up and recovery procedures should be documented within the business continuity plan along with details of any off-site storage arrangements for data/media. For each key process/activity/system, it is necessary to determine the type of back-up process which is appropriate – see below. At a different level, for a simple administrative process, it may be sufficient to back up the computerized process with a manual process (perhaps supported by stand-alone PCs). Of course, the nature of the recovery process will determine its speed. This planned speed of recovery must be appropriate to the time-band assessed impact study carried out earlier. IT systems may need to be recovered/restored in the event of an IT-related incident – or in a more general, premises-based one. Any significant premises-based incident (power outage, fire, flood, etc.) is almost bound to have some effect on either the underlying IT infrastructure or the major systems themselves (Savage 2002). Page: 95 of 237
Thomas Virgona Doctoral Dissertation: Defense
Term Definition
Disaster The exploitation of tragic images and heart rendering words by officials
Pornography and media for the promotion of their own organization (Hewitt in
Quarantelli 1998).42
Length of The amount of time between the identification of hazardous conditions
Forewarning and the actual onset of effects at particular locations (Kreps in
Quarantelli 1998).
Magnitude of The severity of social disruption and physical harm (Kreps in Quarantelli impact 1998).
Scope of Impact The social and geographic boundaries of social disruption and physical
harm (Kreps in Quarantelli 1998).
Duration of The time lag between the onset of the social disruption and physical
Impact harm and when the disaster is no longer defined as producing these
effects (Kreps in Quarantelli 1998).
Technology field Building, maintaining and supporting computer, voice and data systems
for business, commerce, government or educational entities.
Logistics
The time of this study was Summer 2007 through Fall 2007 and the research will be based at Long Island University, the C.W. Post Campus, in Brookville, New York. Since humans are involved, an Institutional Review Board (IRB) approval is required, as per
Kathryn S. Rocket, Director - Office of Sponsored Research, LIU.
42 This study is aware of the September 11 conspiracy theories and will not address them. Page: 96 of 237
Thomas Virgona Doctoral Dissertation: Defense
3.3 A specific methodology and justification
Research methods are varied and have inherent benefits and risks. For this dissertation research, non-experimental design have been used. Specifically, qualitative analysis techniques were used – exploratory and descriptive. The research goal was to uncover what we have learned from the events of September 11, 2001. Qualitative analysis is intended to produce an explanation of a phenomenon, particularly an identification of any patterns observed. Qualitative research methodologies can include structured observation,
"On the spot" Interviews, Verbal Protocol Analysis (Thinking Aloud), Focus-Group
Interviews, Diaries/Journals-Analysis, Case Studies, Experiential, Contextual, Engagement,
Immersion, "Walking in their Shoes," "Seeing it in their Eyes," or Ethnographic study.
Conversely, quantitative analysis is best suited for: breadth/mass data and a highly structured approach.
Qualitative analysis involves the close examination of deliberately selected cases (in this case, September 11, 2001) to yield insights into phenomenon that might not otherwise be discovered. In the social sciences, qualitative analysis is used predominately with human subjects to uncover patterns of social interaction. While many see the qualitative and quantitative paradigms as opposites, increasingly scholars understand the two as complementary. Where the quantitative paradigm allows generalization of specific empirical observations to a population, the qualitative paradigm facilitates the discovery of interactions that might otherwise go unnoticed. Qualitative analysis, particularly case study, is a powerful means for analyzing phenomena that are little understood, which can lead eventually to the development of hypotheses that can be tested in the quantitative paradigm. On the other hand, qualitative analysis is a critical means for discovering cultural influences that affect
Page: 97 of 237
Thomas Virgona Doctoral Dissertation: Defense
phenomena under study (Smiraglia 2002). Observational methods are best suited for describing and understanding behavior as it occurs. They are less effective for gathering information about a person's beliefs, perceptions, attitudes, etc. Questionnaires and interviews are frequently used for obtaining the latter type of information. In this scenario, a qualitative approach is justified.
We will study a sample, a subset of the population of information technology professionals that contributed to recovery efforts after September 11. The purpose of concentrating in one place, and on a well defined period, is to make the subject manageable
(Clanchy 1991). While many collective behavior studies have focused on processes in disaster events, the events that these studies investigated do not approach the magnitude of the
September 11 attacks on the WTC (Connell 2001). Clearly, an examination of collective behavior during the events on September 11 can provide an important perspective on the existing knowledge regarding collective behavior in disaster events.
3.4 Sample data-gathering implements
There was strong temptation to use a Delphi Study for this research. The ability to gather a consensus among experts in disaster recovery has strong appeal. However, the drop off in participation in later rounds of questioning posed a risk to the research. For this reason, a focus group data gathering technique has been selected.
Berg cites many advantages to the focus group technique that apply to this study (Berg
2001). When focus groups are administered properly, they are extremely dynamic, spurring interaction among group members. This has an important advantage over a face-to-face interview. The Berg terminology for this type of session is called bracketing: investigating a
Page: 98 of 237
Thomas Virgona Doctoral Dissertation: Defense
phenomenon up close for careful inspection. In this case, the reliance on human resources during a true disaster recovery scenario will be studied.
One disadvantage of a focus group is the need for the moderator to take copious notes.
Since most communication will be verbal, detailed note taking presents unique issues. To reduce the risk of missing critical information, the participants will be asked if it is acceptable to audio tape the session. The information can be transcribed at a later date to ensure accuracy.
According to Berg, there are four steps in conducting a focus group:
Table 3 - Bergs Four Steps for Conducting a Focus Group Step Action
1. Locate a population The focus group participants will be selected from people from which to select partaking in Wall Street Financial services firm’s disaster participants. recovery tests. Since the researcher is a regular participant in
these tests and is responsible for Disaster Recovery / Continuity
of Business for his technology unit, he is privy to names of
people involved with the tests. These individual are responsible
for testing disaster recovery and contingency plans developed
during the Systems Design phase of the Systems Development
Life Cycle. Typically, these individuals have project
management and information technology testing experience. As
a result of the high interconnectivity among financial services
firms, testing a contingency plan requires resources from many
firms in the immediate geographical area.
Page: 99 of 237
Thomas Virgona Doctoral Dissertation: Defense
Step Action
2. Contact potential Individuals will be contacted during the planning of a disaster participants and convince recovery test. These tests are scheduled every month and them that their involve staff from major firms in the Wall Street area. There is participation is important no reason to believe that permission from the participants’ firms and necessary. is required for this research. Berg states that there is no
common consensus on the “perfect” small group for focus
group. The range is generally six to twelve participants.
According to George Miller (Miller 1956) the magical number
seven (plus or minus two) may be the human limit for our
capacity for processing information. The goal of this project is
ten or twelve participants.
Dealing with sensitive issues:
The events of September 11, 2001 are still fresh in the minds of
many people. Discussing the changes precipitated by the events
of that day may illicit emotional responses. All participants will
be asked prior to the actual focus group if they may be
uncomfortable discussing this event in a public forum. If any
participants express any discomfort with the research, they will
be thanks for their time and participants will be selected.
Page: 100 of 237
Thomas Virgona Doctoral Dissertation: Defense
Step Action
3. Hold the focus group Introduction and introductory activities:
The moderator will explain the research topic to the subjects and
how a focus group operates. It will also be explained that there
is a recording device in the room to ensure all information is
collected accurately. It will be hidden so the participants do not
feel uncomfortable. It also will be stated again that all
information will be confidential and that their names will not be
published in any documentation.
The information sought will be if usability requirements have
changed disaster recovery design since September 11, 2001.
The specific focus will be on human, not technical, factors. The
group will introduce themselves with name, background and any
other special information about themselves.
The basic expectation of the focus group session will be for an
open, honest, polite and orderly exchange of thoughts. The
planned duration of the session is 2 hours.
Short Question and Answer Discussion:
No set script will be prepared. There will be a couple of starter
questions: On September 11, 2001 a disaster was declared and
many technology systems went into disaster recovery mode.
Who in your company made that declaration to cut over to DR?
How were you notified?
Page: 101 of 237
Thomas Virgona Doctoral Dissertation: Defense
Step Action
The atmosphere will be comfortable and casual. The setting
will be the home of Thomas Virgona. The expected duration
will be 2 hours in total.
4. Follow Up A thank you e-mail will be sent to all of the participants. They
will also be asked to communicate anything else they may have
left out of the discussion but would like to include in the
research.
The second qualitative technique used for this research will be unstructured interviews. Face-to-face interviews will be conducted with Wall Street individuals currently tasked with ensuring continuity of business operations for financial firms in the following areas: Database Administrator (DBA), Network Engineer, Application Manager, Help Desk,
Senior Manager, and a Business User. All interviewee’s will be knowledgeable in their field.
The rationale of using this technique is that it is effective for eliciting opinions, attitudes, values and beliefs on a topic (Sproull 2002). Discrepancies can also be clarified immediately. In addition to obtaining information directly from people, it allows the opportunity for deeper probing. Using an unstructured technique allows the feedback to vary from one interview to another. Interviews are also best in the cases where people’s opinions are a valuable source of information and the desired information is complex in nature. The known disadvantages are that interviews are time-consuming in nature and subject to inaccuracies due to the participant’s lying or bias.
The third qualitative technique that will be utilized is observation. The researcher will observe an actual disaster recovery test conducted by Wall Street financial firms. These tests
Page: 102 of 237
Thomas Virgona Doctoral Dissertation: Defense
are scheduled on a regular basis and will provide valuable insights into disaster recovery tasks as they are performed in their natural setting. The researcher is a member of the Disaster
Recovery team for one of the firms and requires no additional training. The goal is to see things as they happen without disrupting the participants.
The fourth and final qualitative technique that will be utilized for this dissertation is artifact analysis. Disaster recovery plans from the year 2000 will be compared to current disaster recovery plans. A sample of five business applications plans will be reviewed. One major advantage of this technique is that the data already exist and the researcher has access to the information. The documentation resides in electronic format in the central testing repositories. The firm names will not be published and any sensitive data will not be included in the review. The assumption is that plans have become more detailed and thorough.
By definition, a sample is “the subset of people or other entities selected for a study from a larger population (Schutt 1999). Since the “larger population’ of technology professional impacted by the events of September 11, 2001 is not known, a random sample is not possible. The participants for the focus group and interviews will be selected based on subject matter expertise and availability. The subjects will be contacted and recruited directly by the researcher. As the focus of the study is on human aspects of the September 11, 2001 attacks, the companies for whom the subjects work do not need to provide approval since no information relating to a particular firm will be included in the findings or results. The only identification to any firm will be included in the literature review, which is publicly available.
The criteria for inclusion are simply the willingness to participate in the study once contacted by the researcher. Exclusion from participation will be granted to anyone who does not care to participate.
The initial contact script will be as follows:
Page: 103 of 237
Thomas Virgona Doctoral Dissertation: Defense
You are being asked to volunteer in a research study called September 11, 2001 - A
Study of the Human Aspects of Disaster Recovery Efforts For Wall Street Financial
Services Firms. The purpose of the research is to examine the impact of the
September 11, 2001 events on Information Systems Technology. You will be asked to
take part in an [individual or group] interview process to develop an understanding of
information technology changes precipitated by the events of September 11, 2001.
These tasks will require approximately 2 hours. The day will be scheduled at your
convenience at my home. There are no foreseeable risks or discomforts associated
with your participation in this study.
Page: 104 of 237
Thomas Virgona Doctoral Dissertation: Defense
3.5 A statement identifying potential analytical methods and expected results
Qualitative data needs to be reduced and transformed in order to make it more accessible, understandable, and to draw out various themes and patterns (Berg 2001). This will be achieved by grouping common themes and concepts provided by the participants across all four qualitative methods. The results will be collected and common themes and terms will be grouped together - triangulation.
The primary data analysis technique will be term frequency and identification of common themes. In addition to the keyword analysis, verbal trends will be reviewed for previously undocumented patterns. If needed, follow up discussions may be performed on an individual basis to clarify ambiguous terminology or to obtain better definitions of a pattern or trend.
Data gathering analysis will also attempt to define “social groups”, identify internal
and external structures, and indicate how products of earlier events can transform the nature
of later events.
We expect that this September 11 research will demonstrate the human aspects during a disaster. However, the subtle impact will influence several phases of the system development life cycle. To continue business functions, system designers will have to account for external data flows (e.g., anti-money laundering), disaster recovery data center locations, and dependency on governmental agencies to continue business functions (e.g.,
Federal Reserve for money transfers).
Page: 105 of 237
Thomas Virgona Doctoral Dissertation: Defense
3.6 Mapping of methodological techniques to research questions
To ensure the research goals are met and each area is addressed, a set of initial probes will be defined prior to the focus group sessions. The purpose is to initiate a discussion relating to a specific research question, but not to restrict the conversation to that particular topic. The initial probe and survey query for each research question is as follows:
Table 4 - Methodical Techniques Used to Address Research Question
Research Question Methodical Techniques
Were systems on Wall Street negatively The focus group and unstructured interviews will impacted by the events of September uncover actual problems encountered that day and
11, 2001? what the deficiencies were in the previous DR/COB
plans. The focus group will allow participants to
interact with each other and trigger thoughts on the
topic that an unstructured interview may miss.
What happened to the systems that day The focus group and unstructured interviews will and how did information systems describe the tasks performed that day while technologists react? correcting the problems that were encountered.
Page: 106 of 237
Thomas Virgona Doctoral Dissertation: Defense
Research Question Methodical Techniques
What changes to the SDLC The review of the disaster recovery plans will
(specifically human’s role in Disaster indicate the current focus of recovery plans are recovery design planning) have been now, and how they evolved since September 11, implemented since September 11, 2001. Additionally, by observing an actual disaster
2001? recovery test, it is reasonable to expect that many of
the participants will make reference to changes
since September 11, 2001.
Are there a lessons learned? All four qualitative methodologies will provide
critical feedback on lessons learned after the events
of September 11, 2001.
Page: 107 of 237
Thomas Virgona Doctoral Dissertation: Defense
4 Results
4.1 Unstructured Interviews
The unstructured interviews were conducted during the Summer 2007 in the home of
Thomas Virgona.
4.1.1 Senior Technology Manager
The first interview was with a senior technology manager. His background included over 20 years in application development and management in both London and New York.
At the time of the terrorist attacks on September 11, 2001 his area of management included 6 production applications. On that morning, he was walking on Wall Street when the first plane hit the World Trade Center. His first recollection after the crash was being hit with burning debris and paper. His initial reaction was worrying about the safety of his staff that may have been in transit. Once inside his building, he advised his staff not to leave the facility, as another strike might be pending. His rationale for this fear is grounded in his experiences in
Great Britain and the IRA attacks that came in pairs. He also contacted his wife who was commuting from upper Manhattan and told her to return home.
By mid-morning, he was faced with a number of disaster recovery / continuity of business decisions: Should the production applications be cut over to the disaster recovery machines? Should the Wall Street site contingency be invoked? What is the status of the
“financial world?” The decision to use the backup machines for the applications was quickly rejected. The data center, although just a few blocks north of ground zero sustained no damage and the auxiliary power and communications were functioning. The DR plans in
Page: 108 of 237
Thomas Virgona Doctoral Dissertation: Defense
place had never been successfully tested and no plan existed for migration back to the production machines from the DR machines.
The second decision involved the Wall Street site since it became apparent that the immediate area would not be accessible for an undetermined period of time. The discussion centered on safety and enabling the business to perform. All programming and development tasks immediately ceased. Technical staff with better “people and soft skills” were placed with the business users at the mid-town Manhattan site to assist with any technical issues that may be encountered.
The remaining staff were asked to work from home and provide technical expertise for the issues that were encountered, such as connections with the Bank of New York43. Since no plans existed for these issues, the brut force approach was used, and in some cases trail and error. In the specific BoNY case, transmissions were normally prioritized by name of the institution in alpha order. On the days following September 11, 2001, processing was extremely slow. The lesson learned from this event was to bring in experts and leverage existing relationships.
One mitigating factor in the days following the attacks was a Jewish holiday. That reduced the number of available personnel across all areas. In this particular scenario, communication with his manager was a one-way process, as he could call his mangers home with status updates, which enabled his manager to listen to the incoming phone message.
Return phone calls from the manager were not allowed for religious reasons.
Because of his experience and title (1st Vice President) this person became the de facto leader for many applications when the management for other groups was not on site for logistical reasons. Few people were allowed on Wall Street in the days immediately
43 The Bank of New York is specifically named as this incident is cited in the literature review. Page: 109 of 237
Thomas Virgona Doctoral Dissertation: Defense
following the attacks, and he was struck by the number of people who called asking him to water their office plants! Communications that day were extremely difficult, especially voice communication. Phone lines inside the borough of Manhattan were overloaded. At one point, calls were placed to the London office and those people were asked to distribute information via phone or e-mail. For this particular firm, people working in downtown Manhattan were able to use the fitness center facilities for showers and the cafeteria for food. The facilities were also used by emergency workers, as the lobby was transformed into a triage unit.
4.1.2 Help Desk Manager
The second interview conducted was with help desk manager.44 His background includes over 20 years in development and management on financial applications in the Wall
Street area. At the time of the terrorist attacks on September 11, his area of responsible was to provide on-site information technology support to the fixed income trading (also known as
Bonds) floor. From a logistical perspective, he was located approximately 5 blocks north of the World Trade Center.
His primary responsibilities were to provide any technical assistance required by bond traders on the trading floor itself. Tasks ranged from software to hardware to phone support.
Bond trading is very “relationship” based and phones are as critical as any other technology to a bond trader. The bond trading market was open at the time of the first attack. The New
York Stock Market had not yet opened for business.
44 This person is the person that first notified this researcher of the attacks. The message was sent to an alias for the group. Page: 110 of 237
Thomas Virgona Doctoral Dissertation: Defense
The trading floor was configured with approximately 30 televisions. The trading floor first heard of the attacks by viewing the television monitors which were tuned to CNBC.
Senior managers called the trading floor to say they felt the rumble of the first plan flying over the building. For the period between the first and second attack, the majority of the traders viewed the television monitors. Taller buildings between this location and the World
Trade Center did not allow for a direct view. When the second plane hit the second World
Trade Center tower, the head of the Fixed Income business screamed to the traders to evacuate the building. No speaker system was used. The information was also not sent via e- mail or phone system. The streets were in chaos as all the immediate buildings were also evacuated. No formal disaster plans were invoked.
Phones were not functional but Blackberry’s did perform well.45 At that time, he sent messages to colleagues in New Jersey to call his family and notify them that he was not injured. He was also amazed at the number of people in the street calling 911 after the second plane hit.
From his vantage point, the impact zones were clear to view as he was north of the trade center and the wind was blowing the debris east towards Brooklyn. He remembers being horrified by bodies falling through the debris. He later found comfort in being told that the people were unconscious as they descended.
The screaming was “incredible” as the first building fell. Walking towards West
Street, he encountered his managing director who told him to go to the mid-town office. The
45 SAN JOSE, California (AP) -- The company that makes the ubiquitous, addictive BlackBerry smart phones said Tuesday it was still looking into what caused the second widespread service disruption in less than a year. Research in Motion Ltd. said customers in the United States and Canada "experienced intermittent delays" for about three hours Monday beginning about 3:30 p.m. EST. RIM said no messages were lost, and voice and text messaging services were unaffected. http://www.cnn.com/2008/TECH/02/12/blackberry.outage.ap/index.html?iref=newssearch Page: 111 of 237
Thomas Virgona Doctoral Dissertation: Defense
walk up the West Side Highway took about 1 hour. By the time he made it to the midtown office, the business status was still unknown.
The Chief Operations Officer for the firm was at the midtown office trying to determine status, what was running, who was making decisions, etc. He was speaking to key people but had difficulty in obtaining basic information. As the day went on, little credible information was known. Simply commuting home to Long Island was difficult as the railroad had shut down operations with no alternative in place. A call tree was invoked to locate all staff: all staff contacted by their managers manually.
He worked from home the next day with no interaction with business group he supported. He was told that the firm was invoking their disaster recovery plans. He could not recall who told him DR was invoked. He also recalled that he did not reference a disaster recovery documentation or continuity of business plan for the entire September 11, 2001 period.
The downtown building was temporarily unavailable as the lobby was being used as a morgue. He spent the rest of the September 11, 2001 week in New Jersey setting up desktops for the business to use the following Monday when trading resumed. This required the displacement of many operations and technology staff. Traders were totally distracted on the first trading day. Business transactions were difficult to perform as many of the traders were unable to locate people in others firms they normally contacted to conduct normal business.
The following are some anecdotal items of interest:
• He recalled the sight of two beams extending upwards for just a few seconds as the
second tower fell.
Page: 112 of 237
Thomas Virgona Doctoral Dissertation: Defense
• He recalled the site of Emergency Medical Service staff vomiting and it struck him
how these were people familiar with these situations but they were still affected.
• Before September 11, 2001, managing COB or DR tasks meant that you had “one foot
out the door”. Nobody took the job seriously.
4.1.3 Application Manager
The third unstructured interview was conducted with an application manager. On the day of the terrorist attacks, he managed the global loan applications and his office was approximately 5 blocks north of the World Trade Center. He was notified of the first attack via e-mail.
The applications that he supported were not “heavy”’ volume systems but the transactions were for large dollar amounts (averaging over $100,000). The business was driven by major business acquisitions and not by financial market. The main group of users for this system was in Delaware performing back office and accounting functions.
After the second plane hit the Trade Center, he and his team (approximately 5 programmers) went onto West Street to view the towers. Two of the programmers were visibly shaken and had to go back into the building. After a few minutes, he and the reminder of his team tried to return to the building, they were told the facility was closed. He then called the members of his team who had gone into the building to retrieve wallets and medications for people who were not allowed back in the building. At this point, there was no corporate direction or communications of any kind. One person was still unaccounted for, as she usually walked up from the PATH train at about 9 a.m.
While the streets quickly filled with people, the lack of credible information was unsettling to everyone. He first attempted to take the subway to Penn Station to get on the
Page: 113 of 237
Thomas Virgona Doctoral Dissertation: Defense
Long Island Railroad, but the E, F and A trains were not running. As the debris was blowing towards Brooklyn, attempting to take a subway to Atlantic Avenue for a Long Island Railroad connection did not seam feasible. At this junction, his team had totally dispersed and he decided to walk to Penn Station. While walking up Broadway, the first tower collapsed. He was able to get a cell phone connection to his mother and asked her to call his wife and tell her that he was safe. Walking through the city at that point became extremely difficulty as chaos erupted in the streets in the area of New York University. After making it to Penn
Station, he was notified that the Long Island Railroad had invoked its “contingency” plan. To this day, he has no idea what that meant, but essentially, there was no train service. At that point, he walked to the firm’s midtown office.
The company had increased security at the entrance to allow only employees into the facility. The cafeteria was opened up and food was free, but people mainly looked for water after walking from downtown. An e-mail was received that technology had set up a temporary “war room” on the second floor, and all managers were to provide a status update if they were in the building. The room had several senior managers and they were clearly groping for information.
The first question he was asked was “Is your staff counted for?” He said one resource had not been located. He then was asked to provide a status for his applications. He told them that his systems were up and running, and since his systems had interfaces to faxing software and funds transfer systems, his applications could be used by other business lines for these utilities. Once he left the room, he wondered why he said that, as he had absolutely no knowledge of what state his applications were in. His biggest worry was that he would be called back into the room and be asked to provide those services.
Page: 114 of 237
Thomas Virgona Doctoral Dissertation: Defense
As the day went on, he was on the phone with Delaware wondering what would transpire over the next few days. The existing disaster recovery plans were of absolutely no value, as they mainly addressed loss of power to a building. These documents were never referenced during the September 11, 2001 timeframe. Delaware Operations was told that they would need to house additional staff from the City, but it was unclear how the staff would get there or how long the site would be needed. He went home that night still worried about the one missing person and how to support the business the next few days. About 10pm the one missing staff member called his home. She did not have his home number and needed to call directory service for it.
On September 12, he was asked to provide one person for Delaware and one for the midtown office. Delaware needed a technologist with good soft skills and communication to assist with the staff driving down to Delaware and provide application support and technology needs. The person needed for the midtown office needed deep networking skills, as laptops, printers and PC had to be added to the network and engineering was short on people. These staff were easy to identify, but asking a person to leave his or her family to work in Delaware for an extended period was difficult. The staff person was told to stay for a few days and return on the weekend.
The main concern for his loan applications was the concept of a “business day.” Since no deals were conducted on September 12, 13, or 14, how would this impact the accounting systems? Would the batch feeds to the general ledger work? Was accounting out of balance?
Nobody had these answers. The batch scheduler continued to process feeds with headers and trailers but with no data. Another concern for all application managers was the supply of diesel fuel. Deliveries were not allowed below Canal Street and the data centers were already running on backup power supplies. How long before fuel became an issue?
Page: 115 of 237
Thomas Virgona Doctoral Dissertation: Defense
In the following days, most of the questions were answered with improvised answers.
The corporate books were out of synch, but were adjusted manually. Emergency shipments of fuel were allowed.
With the exception of the person who had gone to Delaware, the team was back at the downtown building the following Monday. Exiting the subway required a corporate ID and the name had to be on a list held by the U. S. Army. The purpose of returning to the downtown office was to restart business as usual. Simply stated, that was not possible.
Phone lines were still not operational.46 Most people still were still dispersed and difficult to locate. And most of all, the eerie silence of no traffic in the area was nerve wracking.
The was one additional anecdotal item of interest: During the blackout in the
Northeast two years later in August 2003, he recalled using UPS and FedEx drivers as sources of information, since their communications were more reliable than other sources.
4.1.4 Network Engineer
The fourth unstructured interview was conducted with a Network Engineer. On
September 11, 2001, this person was a computer architect for a firm on Wall Street. Since then, his role has changed to manager of technology control and risk. He has 30 years experience in the information technology field.
Due to a leg injury, he was working from home in Brooklyn on Tuesday, September
11. He was on a management conference call when the first plane hit the World Trade
Center. A person on the call announced what she saw on the call as she had a direct view of the trade center. Since the network engineer’s wife and oldest child had just left for work and
46 Many different, alternative, forms of communication helped cope with the September 11 crisis in the most tragic of times (Noll 2001). Page: 116 of 237
Thomas Virgona Doctoral Dissertation: Defense
school, his first reaction was to get his family back in the house. His daughter was already on her way to school, but his wife came back into the house and described the loud explosion.
Minutes later the second plane hit. Within just a few minutes, the network engineer was contacted by his manger and told to contact his staff to ensure everyone was accounted for. One person could not be located and was one of the people who died during the attacks.
Another staff member was in a state of shock as she witnessed bodies falling from the buildings.
By mid-day, the debris from the attacks was making the air in Brooklyn difficult to breathe. The network engineer picked his kids up from school and brought them home. Later that day, he was told that the department’s COB plan had been invoked, and this essentially meant “go home and stay home.” As an architect, he recalled thinking that none of the DR plans have ever envisioned a disaster of this magnitude - where communications had stopped, travel had ceased and there was no concept of making decisions based on an unknown status
(such as, is today a business day?).
The network engineer realized that the ‘brut force’ approach would be used, aligning skilled people against the recovery tasks. His observations included people testing connections and “hoping”’ they would work. When connections did not work, it was nearly impossible to determine why or the true root cause (Was the connection bad? Were the systems on the other end up and running? Had the entitlements protocols locked them out?
Were the interfacing systems now on a new IP address for backup and he was looking at the wrong system?) While “most” of his firms systems were functioning in three days, they certainly were not operational since most of Wall Street was still not working. He also recalled seeing Verizon47 Vice Chairman Larry Babbio physically handing out brooms to his
47 Verizon was named since this incident is now public knowledge: http://www.cwa1101.org/wtchome.html Page: 117 of 237
Thomas Virgona Doctoral Dissertation: Defense
workers in a Verizon building, as the debris had accumulated in stairwells and made climbing the stairs difficult.
In his current role as manager of risk and control, this subject now views COB and DR planning in a new light. Based on his experience on September 11, 2001, documented plans during a disaster offer a comfort level for a “predictable” crisis, but the unknown or unpredictable is the real risk. He cited New Orleans after Katrina: “What DR plan accounts for a large percentage of police officers resigning during a crisis?” One negative impact of the increased focus on COB testing is the resulting outages to production systems. Many times, when testing is conducted on weekends, the production systems used by the business units are not put back in the “ready state” for start of business on Mondays.
4.1.5 Business User
The business user selected for this research was a trading analyst with over 20 years of experience. She had worked in two other Wall Street firms prior to accepting her current position. On the morning of September 11, 2001, she was in her office reviewing market data
(Moody’s and Bloomberg) in preparation for the market opening. The window of her office overlooked Wall Street and the moment she noticed debris in the air there were several news reports describing a plane that had crashed into the World Trade Center. As she was in the
WTC during the first terrorist attack in 1993, she knew this event was not an accident. Her first reaction was to check on her family. Her husband was safe in his midtown office and she was confident her two children were already in school but called to verify. Her mother was in a nursing home on Long Island and she called to tell her she would be leaving immediately
Page: 118 of 237
Thomas Virgona Doctoral Dissertation: Defense
and not to worry (phone service was lost right after the call). She gathered her belongings and went onto Wall Street.
Once she was on Wall Street, the second plane hit the WTC. Although she heard the crash and the ensuing screaming, actually seeing the impact was not possible due to the debris and the obstructed view to the WTC. By this time it was approaching 9:30, the normal start of the trading day. The streets were packed with workers from the New York Stock
Exchange: she had heard that the market would not open and employees were told to evacuate the area.
The subject went back into her office to get her pager and blackberry and was struck by someone reading a disaster recovery plan. She immediately thought is was ridiculous that someone would read a 300 page manual while the immediate area was under attack. As she had never participated in a disaster recovery or continuity of business test, she did not know of any meeting places or escape routes. She had a car available in a nearby parking garage and had intended to pick up her husband in mid-town and drive home to Long Island. The problem was that traffic was now at stand still and she could not contact her husband. She decided to get her car out of the garage and “grind-it-out” into midtown, in the hopes that she would eventually be able to contact her husband while in route. Luckily that plan worked.
Once home, she was instructed to follow the disaster recovery plan, which she did not have and had never read. She always thought the disaster recovery tests were too “staged”, with three months of preparation and they still failed. In her mind, for her job function, she simply needed contact names and market data. On the morning of September 12, 2001, she realized that even those simplistic needs would not be met. Although she had the internal and external customers’ names and “normal” contacts, she had no idea where these people were
Page: 119 of 237
Thomas Virgona Doctoral Dissertation: Defense
currently located. Using the corporate network from home was not allowed except for recovery reasons, so obtaining a true business picture was not possible.
By mid-day, the subject was able to get a copy of the DR plan on her Blackberry and she forward that to her personal e-mail to it could be printed out. The plan indicated that her department was to work from a DR site in New Jersey, which was not feasible given travel restrictions. Her manager informed her that every effort needed to be made between then and
Sunday to determine how the market would react when the NYSE re-opened that Monday.
This proved to be a difficult task without the ability to see how the foreign markets were reacting (except for news accounts) and no communications with her normal business contacts. As the week progressed and pieces of information were gleaned from a variety of sources, it was established by that Sunday that the market would take a “sizeable” hit on the opening bell. She was told to report to the New Jersey office for the re-opening of the market, as connections and communications on Wall Street were still not fully operational.
Her most vivid memory of the first business day back was the disappointment in the large volume of selling that morning, as she thought people would be more “patriotic.” As a disaster recovery site, the New Jersey building worked well: as people were easy to locate and the information required to do her job was readily available. She moved back to her regular location on Wednesday.
A few days later a departmental meeting was held to discuss improvements to the disaster recovery plans. Her input was that everyone needed to participate, not just a selected few. Also, the plans themselves should be updated on regularly with current contact information and hard copies should be left at home. She also recalled, without great detail, one colleagues input on the lessons learned. She recalled one manager stated that disaster recovery site, although handicap accessible, did not meet all of the business needs. Some
Page: 120 of 237
Thomas Virgona Doctoral Dissertation: Defense
users required specific ergonomic equipment (e.g., keyboards and monitors) for special needs and one user required voice recognition software. The disaster recovery plans and site did not address these requirements on September 11, 2001.
4.1.6 Database Administrator
The final unstructured interview was conducted with a database administrator, frequently referred to as a DBA. The DBA interviewed specialized in Oracle, but also supported Sybase and DB2. This person currently has over 10 years experience in the field and was working on Wall Street for a major bank on September 11, 2001.
On the morning of September 11, this DBA was at work by 7a.m. His normal
“workload” was the care and maintenance of more than thirty production databases. Some, but not all, of the tasks were: performance monitoring, load balancing, database version upgrades, table configuration, addition of columns and indexes. To monitor the status of the production databases, he was provided a “Dashboard Monitor” which at 25 inches was much larger than a standard personal computer screen. The dashboard also displayed status visually
(Red, Yellow, Green) for the critical performance measurements.
When the first and second plane hit the WTC, there was absolutely no change of status to any of the production databases. He heard of the attacks from a colleague. He is divorced with no children, but was worried for his ex-wife’s safety, as she was a trader on the floor of the New York Stock Exchange. She was coming out of the #4 subway when the first plane hit and she immediately went into the stock exchange building to avoid the debris, using the
Broad Street entrance. Ironically, security up until that day was much looser than it is today and entering the building only required a photo ID at the time.
Page: 121 of 237
Thomas Virgona Doctoral Dissertation: Defense
Despite the normal dashboard status, he sensed that there would be problems, and by
9:15 the issues had started to trickle in via e-mail. The New York Stock Exchange announced that trading would be suspended that day. Application managers had started e-mailing asking him to call them to discuss disaster recovery plans. At this point he realized two critical issues:
• As a group, the database administrators do not “own” disaster recovery or continuity
of business plans. The database plans are embedded in the application plans, which
makes DBAs a largely dependency for other groups and a successful cut-over to
DR/COB.
• The underlying assumption was that one, or maybe two, applications would fail at a
single time and the workload would be manageable. Nobody had envisioned a DBA
supporting 30+ applications moving to the DR/COB environment. The current staff
would be insufficient. He commented that it was similar to Hurricane Katrina, where
the DR plans never envisioned a major percentage of the police force resigning during
the crisis.
Once the first WTC tower collapsed, his dashboard immediately turned from green to
“bleeding” red. The root cause of the status change was the loss of critical communication lines, which necessitated a move to the contingency databases. His initial reaction was that under normal DR scenarios, moving a database to a new environment can be complex.
Moving over 30 databases would prove to be a real challenge. As a very general statement, having an application point to a contingency database is a very simple procedure and is tested on a regular basis. The issues that were encountered included transactions lost in transit
Page: 122 of 237
Thomas Virgona Doctoral Dissertation: Defense
during the cutover, insufficient access rights to a firewall or machine to perform configuration setups, applications software not recognizing the new database, etc.48
The one phrase he recalled vividly was that “COB trumps IS.” The meaning behind that statement is that brut force was needed to get the applications up and running correctly, and the normal maker/checker and approval procedures simply had to be discarded. That is not to say there were not many arguments with management or colleagues, as these safeguards were implemented to ensure sensitive information was under lock-and-key and in a controlled environment. To perform the tasks required following September 11, 2001, the normal processes simply were ignored. He also recalled encountering an issue that was not documented or even considered up until that day: “How do you get all of the databases back onto the production machines and synched up with the application and other system?” As he later found out, it was a manual process performed one at a time.
During the week of September 11, 2001 (Tuesday to Friday evening), he never left the office, estimating he slept about 12 hours during that timeframe. He conservatively estimated that he made 200 manual configuration errors during that period due to stress and lack of sleep. He also stated that despite all of the lessons learned from September 11, 2001, the database issues would probably re-occur as DBAs do not own the DR/COB plans and do not have direct input into the DR documentation.
4.1.7 Summary
In listening to the subjects that were interviewed for this dissertation, there were several points that were universally discussed and agreed upon. First and foremost, every
48 There is a patent pending (Moore et al) to provide a method to reduce the time that a broken database is unavailable by processing recovery input data in parallel and recovering multiple database data sets, including areas, simultaneously. Page: 123 of 237
Thomas Virgona Doctoral Dissertation: Defense
interviewee had the same initial reaction when they were notified of the September 11 attacks: safety! Safety for themselves and family members was universally stated as the biggest concern that morning. The inability to contact family members and colleagues was extremely frustrating. All other issues were described as a secondary problem. The focus group discussions will validate this finding.
Problems specifically related to disaster recovery were common among all respondents. The inability to determine basic operational status, specifically if September 11 was a business day, caused additional confusion. Test plans were not used by one interviewee for a variety of reasons: the document was unavailable, pre-conceived notion that the document was stale or useless, and the plan that was in place did not account for a disaster of this magnitude. The review of the disaster recovery artifacts will help to define the exact problems with the disaster recovery planning and documentation.
4.2 Disaster Recovery Test Observations
The second qualitative technique used was an observation of an actual disaster recovery test conducted by Wall Street financial firms. The test was conducted on a Thursday evening in July of 2007. The goal of the test was to validate the ability of multiple firms to use applications (used by multiple firms) housed in a specific data center in lower
Manhattan.49 A “War Room” floor was set up for coordination and administration of the test in building that will be used as the disaster recovery site. A separate floor was set up for the application support teams. The floor was an open space with low rise cubicles including a phone, PC and a laptop connection. Over each cubicle was a sign indicating the company and
Page: 124 of 237
Thomas Virgona Doctoral Dissertation: Defense
application to which the space was assigned. The researcher spent the night going from the
“War Room” to the technology floor.
As a general statement, every phone was dialed into the conference line that was used for status updates and communication of open issues. The first issue noted was that a higher number of users were logging into the test and the systems performance was not acceptable.
Once the test started, the administrators of the test encountered a technology problem: the website that was to be used for problem tracking was not operational. The root cause could not be identified immediately and the process was updated to announce issues on the open call: they would be logged manually into an Excel spreadsheet.
The test also used an automated distribution system to disseminate information, such as participant notifications and disaster recovery documentation. Once the test began, all business and technology participants were to be notified via a phone call and e-mail
(including an attachment with the test plan). As some of the data in the system was no longer valid, many parties were not notified that the test had started and relied on personal copies of the test plan. As is standard practice with most Disaster Recovery tests, the initial phase included individual application check-outs. Technologist will ensure the system is up and running and connected properly to the required networks and databases. Business users will then log in and validate the data are accurate and appear reasonable. As the July test was planned for several months, this phase of the test was expected to go quickly and it, in fact lasted less than 30 minutes.
The next phase was for the actual users of the applications to enter transactions onto the systems. This phase of the test requires connections to other systems and companies for market data, funds transfers, faxes and validations. At this point in the evening, the noise
49 For confidentiality reasons, the exact location is not provided. Page: 125 of 237
Thomas Virgona Doctoral Dissertation: Defense
level rose dramatically. As transactions were being entered by the users in disperse geographic locations, many encountered errors. This resulted in many calls to the technology floor requiring the technologists to investigate the issues. In some cases, the fix was simply switching a parameter or educating a user. In many cases, the problem was with the interface to other systems (e.g., system was not ready, interfacing application was not participating in the test and the test pointed to a production system, etc.) If the interfacing application (or company) was on the technology floor, participants would walk over the cubicle of the application and discuss the issue. In some cases, phone calls were placed to the application support teams listed in the documentation, which led to frustration as the information was not always accurate. Many technologists met in the cafeteria to share solution to the common problems that were arising.
The final phase of the test was conducted at the 5 hour point, when the manual entry of data ceased and the overnight batch systems started processing. This segment of the testing is extremely complex as it relies heavily on feeds from interfacing systems sent specific intervals. Additionally, many institutions have reconcilement checks that will abort processing if the information fails data integrity checks.
While the focus of this dissertation is on “human aspects” of disaster recovery efforts, it is critical to take note of the issues that did occur during the test:
• Retired System: Many applications were expecting to receive a feed to/from a system
that had been replaced since the last test. The documentation did not reflect the new
system and testing was not possible.
• Password reset: When systems encountered minor issues, such as a batch job that
required a restart, the normal process of calling the help desk to obtain the password
failed. In most cases, the help desk staff can only change production system
Page: 126 of 237
Thomas Virgona Doctoral Dissertation: Defense
password, and the application in the COB/DR environment were still classified as non-
production. The help desk was unable to reset the passwords.
• Test Versus Production Mismatch: During the test, an overriding, yet incorrect
assumption is that if the data center tested that evening went into DR mode, all
application were also in DR mode. This caused problems with virtually all feeds that
were not directly involved with the test. For example, market data that is usually
streamed in production databases (e.g. Bloomberg) failed the test as the COB system
were attempting to interface with their COB counterparts, which were not operational
that evening.
• Information Security: Firewall issues between systems did not allow connectivity
and required configuration/parameter “rule” changes during the test.
4.2.1 Summary
In observing a disaster recovery test and comparing compare it to the conclusion the in the interviews, it appears that even in 2007 there is a great deal of reliance on the "human aspects." Despite trying to minimize the need for human intervention following September
11, the need is still is there. The System Development Life Cycle places emphasis on building technology in a control manner and this process includes documenting the disaster recovery process. Feedback from the interviewees and observing a disaster recovery test raises the value of the documentation. During September 11, none of the interviewee’s uses any documentation to assist with the system recovery effort. Similarly, during this disaster recovery test, no documentation was used to resolve any problem that was encountered. Each and every instance of a system problem that evening was resolved in the same manner: contact the ‘expert’ (either by phone or looking for the person on the floor or lunchroom).
Page: 127 of 237
Thomas Virgona Doctoral Dissertation: Defense
What should also be of concern to financial firms is that once these issues are uncovered during a test, and the problem is managed to resolution, there appears to be no updates to the documentation to reflect the change. This issue will be raised in the focus group.
4.3 Focus Group
A focus group to discuss the human aspects of the September 11, 2001 disaster recovery efforts was held on Wednesday, August 8, 2007. In a somewhat ironic twist, Long
Island was flooded with torrential rains that day, causing site contingency plans to be invoked for several members of the focus group since the Long Island Rail Road and New York City
Subways system were not operating. The session was originally scheduled to begin at 10 a.m., but was delayed until noon so the participants could address business issues and make the longer than anticipated drive to the researcher’s home.
There were a total of nine participants from three different Wall Street firms and one government agency. No tape recorder was used. To start the discussion, attendees were asked to introduce themselves and describe their current roles and their role on September 11
2001. For this session, four programmers, three managers and two legal/compliance managers participated.50 All members of this focus group had recently (within 6 weeks) participated in a disaster recovery test. All members of the group were in their work locations on Wall Street when the first plane hit.
As with the one-on-one interviews, all participants described shock at the unfolding events on September 11, 2001 and unsuccessful attempts to contact family members. Once the second plane hit, evacuation from the immediate area was ordered. Two members of the
50 No member of the focus group participate in any other phase of this research. Page: 128 of 237
Thomas Virgona Doctoral Dissertation: Defense
focus group made the logistical mistake of heading south to Battery Park, essentially cutting themselves off from the rest of Manhattan. Ferry service was the only alternative available to those two people.
The programmers stated that they were totally disconnected from information sources since they did not have Blackberries and cell phones were inoperable. One manager stated the irony of having a disaster recovery plan that requires them to call a programmer, with limited communications available.
The general consensus of the group was that the “fail-over” plan for hardware and applications performed well. However, to state that the DR or COB plan went well would be simplistic, and more pointedly, inaccurate. Once the disaster struck, the integrated networks between the firms (and globe) essentially transformed applications into individual silos that had to be connected manually to the “new” technology world. To do so required contacting business partners to find out the location of their “bunker.” Inter-personal relationships played a key role, as there had to be a major trust level among staff for password sharing. The major obstacle was that most DR plans did not have this information.
The managers who participated in the focus group highlighted the problem of having backup systems in 7 World Trade Center. These backup systems not only included physical data tapes, but alternative seating for displaced resources. To the managers in the group, noting was more frustrating than the lack of known status. Was September 12, 2001, a
“business day?” Where are the business contacts? Who is making the decisions? The lack of a centralized coordination point for the firms was cited as a critical need during disasters.
There was complete agreement that the term “Disaster Recovery” planning was not longer applicable. The events on September 11, 2001 were so “diabolical” that all previous plans became irrelevant. Current disaster recovery or continuity of business planning focuses
Page: 129 of 237
Thomas Virgona Doctoral Dissertation: Defense
on predicable problems and makes assumptions that the disaster is contained. For example, as a result of the events on September 11, 2001 air travel was halted and transportation in and out of New York City was not permitted. Before that date, no published plans made the assumption that travel came to a complete stop. The focus group agreed that current DR and
COB efforts plan on system outages or blackouts, but in no way are preparing for a disaster.
The most significant portion of the discussion centered on records management. In the days and weeks following the terrorist attacks, federal regulators and agencies requested status and information. The most common request was to provide detailed descriptions of which records were destroyed and of those records, which records could not be reproduced.
Every member of the focus group cited several examples of backups that were destroyed or useless. Many original documents stored in the WTC facility were destroyed. The definitive number of “books and records” lost were impossible to determine. Attempts to retrieve the information electronically led to more issues, such as corruption of databases with data in incorrect formats and incorrect header and detail records. Given that most firms were in no position to determine what physical and electronic records were missing, the federal agencies and regulators were extremely cooperative with the compliance departments.
4.3.1 Summary
The focus group confirmed many of the findings of the one on one interviews and raised other concerns. The focus group confirmed a lack of confidence in disaster test documentation and do not consider these documents to have value during a disaster. The group discussed that the naming convention itself is no longer applicable since these documents do no provide recovery information during a disaster. The "lessons" of
Page: 130 of 237
Thomas Virgona Doctoral Dissertation: Defense
September 11, 2001 may have been "learned" in the short term but already are slipping from the institutional memory.
The focus group also raised a major information security issue not discussed in the literature review or during the one-on-one interviews. Once a disaster scenario has been encountered, many information security controls are by-passed in an effort to expedite recovery time. In some cases, passwords are provided with no official authorization and maker-checker controls are often ignored. This issue required further investigation as to its potential ramifications.
4.4 Artifact Analysis
To compare disaster recovery plans from before September 11, 2001 to current plans, members of the focus group were asked to bring copies for their disaster recovery plans to the focus group meeting. The analysis of the plans presented below. The plans were reviewed by Thomas Virgona and follow up questions were answered by the focus group member who provided the document.
4.4.1 Asset Sales System
The first plan reviewed was for an asset sales system. For all practical matters, the pre-September 11, 2001 disaster recovery plan did not exist for this particular application.
The only “plans” were part of a larger data center plan, which partially explains the technical jargon in the document. The only reference to the application itself was in a table of IP addresses that contained Production IP addresses and DR IP addresses.51
======
51 All company sensitive information has been replaced by xxxxxxxxx. Page: 131 of 237
Thomas Virgona Doctoral Dissertation: Defense
Application Name : xxxxxxxxxxx App Short Name : xxxxxxxxxxx App ID : 31330 Phase (Dev/Stg/Prod) : DR Environment (Old/New) : New IPLEX/IPOD : IPOD
Webserver Info ------URL : xxxxxxxxxxxxx Webserver:Port : webgmsn3r:20872 Webserver Ver : 4.1 BigIP(Yes/No) : Yes
App Server Info ------App Server:Port : webweia6r:10872 App Server Ver : 3.01
Developer Info ------Developer Contacts : xxxxxxxxxxxxxxx Dev Group E-mail : xxxxxxxxxxxxxxx
LDAP Info ------LDAP (Yes/No) : No LDAP Group Name : LDAP Admin Name :
xxxxxxxx Info ------xxxxxxxxx (Yes/No) : No Platform (UNIX/NT) : Domain Name To Access : Base URL Protected : Root directory Protected:
Security ------SSL (Yes/No) : No Firewall Requests(Yes/No) : No Machine Name:Ports to Open : System Admin Info ------App UID : xxxx ftp Login : xxxxxxx
======
The post September 11, 2001 disaster recovery plan was far more robust. It should be noted that the template and information had evolved over a 5 year period. The document starts with a descriptive of the purpose:
Page: 132 of 237
Thomas Virgona Doctoral Dissertation: Defense
The Application/System Recovery Document has been designed for use by all technology areas to document detailed instructions for switching a system or application to its Contingency/Disaster Recovery Environment and back again to its Production Environment. This is required for all Distributed (non-mainframe) applications, and may be used to document special considerations for mainframe applications. It is the responsibility of the Product Manager of the application to maintain these procedures in this standard format. Procedures are to be reviewed at least annually. Storage of these procedures is determined by each Development or Systems area.
The document control section52 indicates the 5 year history of the document and what company resources update the information and on what date. The history shows that the document was created in December of 2001 and that the document was updated twelve times, including the creation of the document in 2002 (seven times in 2002, four times in 2003 and once in 2005). The application and systems and recovery procedures contain 23 sections:
1. ENVIRONMENT INFORMATION 2. SOFTWARE INFORMATION 3. HARDWARE INFORMATION 4. PACKAGE/WRAP/URL INFORMATION 5. INTERPLATFORM CONNECTIVITY CONSIDERATIONS 6. NETWORK/FIREWALL CONSIDERATIONS 7. STEPS FOR SWITCHING APPLICATION TO CONTINGENCY ENVIRONMENT 8. STEPS FOR SWITCHING FROM CONTINGENCY TO PRODUCTION ENVIRONMENT 9. DATABASE/DATA SERVER SWITCHING CONSIDERATIONS 10. SECURITY/SYSTEM INFRASTRUCTURE CONSIDERATIONS 11. BATCH RE-START CONSIDERATIONS 12. WORKFLOW CONSIDERATIONS 13. SCHEDULING CONSIDERATIONS 14. SUPPORT AND CONTACT INFORMATION 15. THIRD-PARTY DEPENDENCY INFORMATION 16. MAJOR DEPENDENCY INFORMATION ON OTHER APPLICATIONS – PREDECESSORS 17. MAJOR DEPENDENCY INFORMATION ON OTHER APPLICATIONS – SUCCESSORS 18. ARCHITECTURE DIAGRAM 19. NETWORK DIAGRAM
52 The document control section of a document indicates the historical changes made to the information, including date, author and summary of the change. Some organization also track document ownership and location (physical or electronic). Page: 133 of 237
Thomas Virgona Doctoral Dissertation: Defense
20. MANUAL PROCEDURES 21. AVAILABILITY OF THE APPLICATION OR SYSTEM (INCLUDING GREEN ZONE) 22. SERVICE LEVEL AGREEMENT 23. MISCELLANEOUS
The fourteenth section, Support and Contact Information, has the names of the resources that work on the application and their work phone numbers. For this reason, and others, the document is marked as confidential as it contains personal information. The fifteenth section contains third party dependency information, such as interfacing partners,
DBA’s and systems administrators. Section twenty-two requires the system to be fully recovered in two hours. Some of the terms used in the document include:
• BCV: Business Continuity Volume. A term used to describe mirrored
DASD.53 The continuity volume is the contingency or DR storage, mirrored
from production.
• FDR: Fast Dump and Restore product for mainframe DASD and data set
management, produced by Innovation Software.
• Greenzone: The designated period of time mutually agreed to by the Business
area, Development area and Operations when changes can be introduced into
the Production and Contingency environments. This implementation window
is defined as a period of lowest risk where customers will not be adversely
impacted.
• Hot Recovery Method: Method for recovery of a system in the event of
production failure includes a hot or live environment which automatically
takes over if one environment fails. Both environments can be considered
53 IBM Disk Drives Page: 134 of 237
Thomas Virgona Doctoral Dissertation: Defense
production. Methods to synchronize both environments are in place, such as a
mirroring.
• Recovery Time Objective (RTO): The maximum time allowed that the
business could sustain without access to required applications before the
business experiences a significant impact (operational, financial, regulatory,
and/or compliance).
• Recovery Point Objective (RPO): The age of the data that could be restored
successfully in the event of a disaster. For instance, if RPO is “start of the
day,” systems need to be restored to the state they were in, as of at the start of
the business day – no later than that. Any data created, deleted or modified
within the recovery point objective could be lost or need to be recreated during
the recovery process.
• Remote Recovery Data Facility (RRDF): Minimizes data loss and service
outage time in the event of a disaster by logging database system update
information in real time at another site. Log or journal data and other recovery
information is duplicated at the other site and can be used to recover databases
to within seconds of the disaster, typically within one second.
• Semi-Hot Recovery Method: This method for recovery of a system in the
event of production failure includes an environment ready to take over for the
production system. Methods to refresh or synchronize the contingency
environment are in place, such as a daily or weekly refresh of the environment.
• Symmetrix Remote Data Facility (SRDF): A remote storage replication
solution available from EMC for disaster recovery and business continuity.
Page: 135 of 237
Thomas Virgona Doctoral Dissertation: Defense
• Traditional Recovery Method: This method for recovery of a system in the
event of production failure includes building a contingency environment (from
a development or Users Acceptance Testing environment) using backups from
the production environment. No dedicated environment is available.
4.4.2 Global Technology Department
The second set of disaster recovery plans reviewed was for a global technology department. As this particular firm has many organizational changes and acquires other companies on a regular basis, a direct comparison of plans for the specific department was not feasible. Additionally, the owner of the current document stated she did not believe plans existed for “department” before 2003. The recovery plan reviewed was for the United Status only. The document is 153 pages.
The purpose of this plan is to define the actions required to minimize risk that might otherwise result from a business interruption regardless of the cause or duration, and to ensure the timely and orderly restoration of business activities. The business continuity goals of the division, in the event of a major interruption of significant duration, are:
1. Provide for on-going production support for the applications that are used by the
business.
2. Provide for on-going production support for applications that are used by the
administration human resources, accounting, and travel).
3. After insuring the above goals first, the last goal is to provide for on-going
applications development across various engagements.
Page: 136 of 237
Thomas Virgona Doctoral Dissertation: Defense
In speaking with members of the focus group during follow-up interviews, these goals are common across many firms. During a disaster, the first priority is consistently stated as ensuring production applications are up and running. Staff is re-deployed to assist with assisting with restoring production technology and development projects will have the lowest priority. The plan turns next to the key recovery priorities for the division. The priorities are presented in order:
1. Insure the safety of the staff affected by the business interruption, irrespective of
cause, time of the event or duration. In the event of an incident, designated staff will
be requesting and monitoring business specific impact and status information
pertaining to the interruption. Depending upon the nature of the disaster, the
Management Team will usually be informed by and take direction from Crisis
Command Center.
2. Insure that Production Support is restored from our remote recovery locations for the
critical applications supporting businesses;
3. Insure that the Production Support is restored from our remote recovery locations for
the balance of the applications supporting businesses;
4. Insure that the on-going development teams are operating from their remote recovery
locations; and
5. Insure that we are in a position to return from CoB once Realty Services have
recovered our “business as usual” location or established alternate permanent
facilities.
To achieve these goals, the organization has addressed the primary goal of restoring production applications and the lower priority task of continuing application development. It is interesting to note that the technologists do not have a back up location in the plan. When
Page: 137 of 237
Thomas Virgona Doctoral Dissertation: Defense
speaking to the owner of the document, she stated that the priority is to ensure the business operation is functioning first, and technologists are better suited to work from home.
Table 5 - Recovery Strategy Production Support Recovery Strategy
Relocate to Disaster Recovery • Work from home unless otherwise Location instructed, or; • Relocate to Recovery Location beyond a 30-day outage at instruction of Crisis Management team.
Shift workload to an alternate site. • Shift Interagency White Paper (IAWP) workload to alternate out of region (OOR) staff in other states.
Application Development Recovery Strategy
Relocate to Disaster Recovery • Work from home unless otherwise Location instructed, or; • Relocate to Recovery Location beyond a 30-day outage at instruction of Crisis Management team.
One section specifically addresses the potential loss of human life. In the event that staff is lost, other department heads will assume the lead on crisis management activities. It is at that time that the regional Crisis Management Teams would determine next steps. In some respects, this scenario already was accounted for with the technology staff already geographically dispersed (India, London, Other US States). A Loss of Staff Strategy is provided below. One interesting note on the approach is that the London office has a similar plan for loss of staff and the plan was invoked during the London subway bombings in July Page: 138 of 237
Thomas Virgona Doctoral Dissertation: Defense
2005 (when staff were difficult to locate). The major issue with approach was the five hour time difference. Staff in New York were either sleeping or commuting when the plan was invoked. Additionally, virtually all of the staff contacted was unaware that they were backups for London production systems.
Table 6 - Loss of staff
NY Production Support – Loss of Staff Strategy Primary and Level 1 -Up Level 2 - 2-7 Level 3 - 7- Level 4 - Greater Secondary to 24 hours days 30 days than 30 days location Scenarios Loss of Primary Location and Staff (Loss of skilled London London London London Takes Over staff) Takes Over Takes Over Takes Over
References to automated call tree tools were made throughout the document. The automatic dialing system is to be tested twice a year and the names and phone numbers are to be included the document. As a result of the recent shootings on the Virginia Tech campus, automated dialing systems businesses have seen an increase in sales as organizations see this tool as an effective method to communicate during disasters.54
In many respects, this plan is an example of why members of the focus group stated disaster recovery plans are useless during a crisis. Aside from being very long (153 pages), the information is very high level (production applications are the highest priority during a crisis but does list the specific applications) and has never been reviewed by staff that were assigned responsibility in the plan. The names and contact information included in the document was outdated and could not be used during a crisis.
54 Article available at: http://www.dallasnews.com/sharedcontent/dws/bus/stories/DN-security_07bus.ART.State.Edition1.3658c92.html. Page: 139 of 237
Thomas Virgona Doctoral Dissertation: Defense
4.4.3 Funds Transfer Application
The third disaster recovery plan reviewed was for a funds transfer application. The plan is updated based on the SDLC design document. The format and sections of the applications disaster recovery plans have not changed since 2000. The owner of the document stated due to heavy legal, audit and regulatory reviews of this system, maintaining an up to date disaster recovery plan is a top priority. Also, as an IBM mainframe application, the hardware and data center environment is extremely stable.
Two items makes this document unique: (1) the number of pages exceeds 250; and (2) the document control section of the document was seven pages in length. Every update to this document includes a reference to a specific technology project, including the project number and date. For example, one project was to migrate the production system hardware from downtown New York City to a data center in the Midwest. The document was changed to include all new data center contact numbers, regional IBM support, updates to firm staff who have changed since the last update and new IP / router configurations. A typical revision to the disaster plan was the addition of a new interface partner and the associated system and resource information. The other item of note in this document is a hyperlink to an internal website that lists the issues from the last DR test and how the problem was resolved.
4.4.4 Globally Deployed Application
The fourth disaster recovery artifact reviewed was for a globally deployed application with components in several countries, including the United States. The disaster recovery plan prior to September 11, 2001 was a single worksheet in a spreadsheet listing all of the
Page: 140 of 237
Thomas Virgona Doctoral Dissertation: Defense
production and database servers globally, with the IP address of the backup / contingency server. That was the totality of the plan. The post September 11, 2001 plan is more extensive.
The post September 11, 2001 document is more extensive, totaling 104 pages. The table of contents includes the following sections:55
1 Application and System Recovery Procedures 1.1 Environment Information 1.1.1 Application Configuration 1.2 Software Information 1.3 Hardware Information 1.4 Package/Wrap/URL information 1.5 Interplatform Connectivity Considerations 1.6 Network/Firewall Considerations 1.7 Steps for Switching Application to Contingency Environment 1.7.1 Preparing for the Recovery 1.7.2 Application Configuration 1.7.3 Oracle recovery: 1.8 xxxx Configuration: 1.7.4 Filenet Recovery: 1.7.5 xxxx Configuration 1.7.6 xxxx Recovery 1.7.6 xxxx Recovery 1.7.7 xxxx INTERFACES 1.7.8 Registry information 1.7.8 Asia Regional xxxx Recovery 1.7.9 Configuration Changes 1.8 Steps for Switching from Contingency to Production Environment 1.9 Database/Data Server Switching Considerations 1.10 Security/System Infrastructure Considerations 1.11 Batch Re-start Considerations 1.12 Workflow Considerations 1.13 Scheduling Considerations 1.14 Support and Contact Information 1.15 Third-party Dependency Information 1.16 Major Dependency Information on other xxxx Applications – Predecessors 1.17 Major Dependency Information on other xxxx Applications – Successors 1.18 Architecture Diagram 1.19 Network Diagram 1.20 Manual Procedures 1.21 Availability of the Application or System (including Green Zone) 1.22 Service Level Agreement
55 All company sensitive information has been replaced by xxxxxxxxx. Page: 141 of 237
Thomas Virgona Doctoral Dissertation: Defense
1.23 Miscellaneous
In reviewing the content of the artifact, the last date the information was updated was
2/24/2005, approximately 2.5 years ago. Additionally, the document controls sections are missing critical names. This is contrast to the previous plans reviewed, which have been recently revised.56 Additionally, since there is no audit trail of who changed the document, and it is not possible to determine who made what changes and when. In a follow up discussion with the manager of this application, this document was cited in a recent (2007) audit as deficient and will be revised.
a. Document History
The following table records information regarding released versions of this document and briefly describes the changes made to them.
Comment / Changes from Prior Version Date Author Version 1.0 24/02/2005
b. Document Approvals
This edition must be reviewed or approved by the following individuals. An “X” indicates whether the individual is a reviewer or an approver. Note: Please tailor the positions below appropriate to your organization.
Name Position Reviewer Approver Product Manager/Owner X X Applications Development COB Coordinator X X Software Quality Manager X X
The global components of the application are accounted for by region. For example:
The xxxx Asia Production servers are located at the Asia Pacific data center. The backup servers are located at the xxxxx data center in Asia. The xxxx Asia
56 Recent revised does not indicate that all of the information is up to date or current. Page: 142 of 237
Thomas Virgona Doctoral Dissertation: Defense
Production environment consists of three HP servers, the application server, the filenet server, and the database server. xxxxx Asia uses EMC/SRDF to replicate data to the COB site.
In the section labeled Steps for Switching Application to Contingency Environment, a critical step in moving to the contingency environment required a database password.
Instructions are included in the procedures.
• The database update password must be known.
Update password for database can be obtained by calling filenet support number 1-800-xxx-xxx (xxxx customer number is #xxxx). This password changes every month.
The cutover to the contingency system requires a fair amount of programmers’ involvement. Specific programming and configuration steps are required a specific intervals to ensure the application is started correctly and is ready for operation. A sampling of the tasks people must perform to start this application in contingency are below. Many of these commands must be performed from the operator console.
• These changes are made for the cache server to run in recovery mode. Immediately
after the test, these changes need to be rolled back to restore the cache server in
production mode. Copy /opt/xxxx/prd/xxxxprd.env to
/opt/xxxx/prd/xxxxxprd.env.YYMMDD.
• Edit the ksh files to include following changes. Values for Application Server (agni),
Filenet Server (pasha) and Database Server (Shiva).
• In the [mhsServer] section, changes are required for following tokens. These changes
are only for COB testing. Incase of actual disaster, values for REMPADDR and
LCLPADDR must be set to the same values as that of the region. Values shown below
Page: 143 of 237
Thomas Virgona Doctoral Dissertation: Defense
are only for NA COB test. So the values for these tokens will be supplied separately
depending upon the region and actual COB vs. testing COB.
• An Oracle DBA runs the Oracle recovery. Typically the dba will have logged in
under the UNIX id of the user that owns the Oracle database and related files.
• Run 'fn_setup' to rebuild the system configuration files.
• Dazel is started by the system operator from Trims Operator Console menu “4. Dazel
Administration /2. Start Dazel, /a All.”
• Start Dazel Components;57 Dazel System components consists of Primary Servers and
Delivery Servers that manage the physical destination (printer, fax modem, file
system, pager etc.).
Ironically, the Manual Procedures section (1.20) of the plan focuses on what is
referred to as “followed the sun” operations. This is a term for applications used globally,
ensuring their business days are aligned correctly. In this scenario, technology is to ensure
the business heads globally synch up their business day so the data flows correctly
throughout the application. While scanning this document during the focus group, the
technology owner of the application seemed surprised at the manual steps needed to ensure
the backup could run.
4.4.5 Loss of a Building
The fifth and final disaster recovery document reviewed was for a loss of a building.
The document was referred to as a “Crisis Management Playbook.” The specific purpose of the playbook is to provide guidance to the company’s staff during a crisis that renders a
57 The application manager has no idea what this component is and believes it may have been a cut and paste error by one of the authors. Page: 144 of 237
Thomas Virgona Doctoral Dissertation: Defense
particular facility unusable. Documentation for this plan was unavailable for the period prior to September 11, 2001.
The first section of the playbook explicitly lists assembly points, depending on the severity of the crisis. The document includes a map depicting the locations of the 5 assembly points. The management scenarios and responses are defined as follows:
Table 7 - Assembly Points Shelter in place Major crisis Forced of staff situation only evacuation due to crisis Impacting Crisis of entire situation in Campus and no Scenario Management building neighborhood one else. Scenario • Localized • Explosive • Chemical • Terrorist Examples floor fire devices found incident near action taken • (evacuate to in building. building causing major next re-entry • Building • Car bombs building floor) utilities lost throughout damage and • False alarm the area injuries. • Severe • Forced Weather Evacuation South of Canal Street.
Page: 145 of 237
Thomas Virgona Doctoral Dissertation: Defense
Shelter in place Major crisis Forced of staff situation only evacuation due to crisis Impacting Crisis of entire situation in Campus and no Scenario Management building neighborhood one else. Response to Option 1 Option 2 Option 3 Option 4 Situation Alert xxx of Activated x Activated x Activated incident and and and xxxx and place on stand-by assemble at assemble at xxx assemble at xxx. Activate xxxxx or if other Deploy selected floor fire cafeteria locations dial in transportation wardens and Dispatch to bridge line buses searchers recovery team to Broadcast to city approved xxxxx information to staging Communicate to Critical employees area. employees recovery go to Activate of situation xxxx Communicate to Work Area Place Work Deploy employees Recovery sites area recovery transportation of situation Transport sites on stand-by buses Place Work recovery staff Place To xxxx area on standby Send Command Activate Place general Center on Stand- Work Area Transportation employees by Recovery sites on home Place Transport stand-by and Broadcast to Transportation recovery staff stage local buses employees on Conditional Work with Interact with stand by A. Send xxx public sector for general for logistics injured status employees requirements Activate home Activate Employee B. Stage Shelter in Place assistance and employee at protocols. accounting assemble process point Conduct Broadcast to damage employees assessment Work with Work with xxxx xxx for logistics for logistics requirements requirements
Page: 146 of 237
Thomas Virgona Doctoral Dissertation: Defense
The General Status update is to include:
• Security – Physical access / City update / restrictions • Facility – Damage assessment / restoration estimates • General Services – Transportation / Food & Logistics • Technology – Infrastructure assessments / restoration estimates • Human Resources – Employee accounting / assistance • Communications – Media inquiries / employee communications • Industry – Other firm actions / regulations updates • Recovery – Command Post update • Impact to business operations • Customer Inquiries • Open issues • Employee concerns • Prioritize issues • Assign resolution owners
The corporate protocols are as follows:
Core DR / COB Team - Event Response Questions • Which, if any, businesses do you direct to evaluate the impact mitigation process? • Should there be any dispersion of the key management to reduce the concentration risk? • Which, if any, regulatory entities need to be contacted? Who does it? • What regulatory relief should the business seek? • What blanket communication for employees needs to be prepared? • How do we deal with rumors/employees concerns at this time? • What services are available to help employees deal with renewed levels of stress/anxiety as a result of the escalating events? • How do we leverage our regions? • What needs to be done to secure accurate information from the authorities? • What business functions need to be managed outside normal business-as- usual locations? • What command centers need to be activated? Business Team - Event Response Questions • What should managers do to insure they have accurate accounting of employees? • Should any steps be taken to activate impact mitigation processes (i.e. splitting staff, transferring work, prioritization of processes)? • What should be done to let clients and counterparties know that xxxx is open for business despite the events? • What coordination should be implemented with vendors, customers, regional counterparts?
Page: 147 of 237
Thomas Virgona Doctoral Dissertation: Defense
• What systems are in place to advise personnel to which alternate work strategies they should report or if they should work from home? • Are there business materials you will need from the BAU office to conduct business at the alternate workareas? • What needs to be done to insure the vital records that support critical functions are available? • What do we need to tell our staff at other BAU locations? • How do you assure employees, clients, etc. that CIB is still functioning and able to provide levels of service? • How are you managing the problem of employees that no one has been able to contact? • What capability support is needed for next day start of business, e.g. facilities, communications, transportation, systems, personnel, etc.? • How do you determine which, if any, employees require EAP or other assistance? • What level of business do you reestablish, or do you try to reestablish 100% of business levels? • At what threshold does the decision to exit a market or cease business occur? • How does the business load balance activities across regions? Support Team - Event Response Questions • Do you make any statement to the media after this event? • What blanket communication for employees needs to be prepared? • What needs to be done to prepare for the possibility of casualties among the potential missing xxx employees? • What should be prepared for the potential activation of alternate workareas (e.g. recovery sites, transfer work, regional counterparts) for critical business functions? • Should the procedure be activated to ensure staff are accommodated with food, shelter, etc.? • What is the current capability of Remote Access? • What corporate activities should we kick off at this time? • How should support resources be prioritized at this point? • What entities should we be talking to at this time? • What statement do you make to the media about the firm’s readiness? • How will employees be tracked? • What will need to be done to secure access to the impacted or at risk facilities? • What services are available to help employees deal with renewed levels of stress/anxiety as a result of the escalating events? • How should we prioritize service requests in the face of competing demands? • Are technology capabilities supporting the business’ changing needs? • How will Help Desk staffing be augmented to meet the expected increased call volumes? • How will coordination with government response organizations occur? Page: 148 of 237
Thomas Virgona Doctoral Dissertation: Defense
• What logistics support will be needed to insure the alternate sites are ready for business tomorrow? • What steps need to be taken to secure prompt delivery of necessary replacement technology equipment at the alternate sites? • How should we prioritize service requests in the face of competing demands? Procedures for Communicating with Customers • The Crisis Management Team will ensure that the subject of customer communication is given priority on its agenda at all times. • Timing and means of communication will be determined by the nature of the crisis and will take full account of the fact that customers may need adequate time to make their own financial decisions to respond to the particular circumstances. • At all times communications to customers will be in the best interests of our customers. Where this may appear to be in conflict with that of the bank’s, legal counsel will be involved in the decision making process. Procedures for Communicating with the Media • Standard procedures throughout the bank require that any contact made by the media be referred directly to the Business Group Head and Corporate Communications who will jointly determine what action to take dependant on the situation. • No employee should reply to a reporter’s enquiry without prior clearance from Corporate Affairs. • Corporate Affairs will follow their own procedures in determining the nature and extent of any communication to be made to the media. At all times the Business Unit Head will be involved in the decision making process. • Corporate Affairs are also responsible for analyzing any comment in the press concerning particular events and for responding accordingly. External Communications • Coordinate all press contact • Coordinate press messages with xxxx • Coordinate employee messages with xxxx Internal Communications • Advise staff not to communicate with the media Internal Communications • Urgent CIB messages are communicated to employees via several channels via xxx Communications department: • xxx Central Intranet Sites announcements • ‘Firm wide Broadcast’ e-mail messages Medical Protocols To the extent possible in a crisis situation, the Medical Department will assess and stabilize employees. The department will coordinate private and public ambulance services that are available for treating and transporting injured employees to the emergency treatment centers. As a division of Human Resources, the medical staff will utilize available resources to coordinate identification and notification of
Page: 149 of 237
Thomas Virgona Doctoral Dissertation: Defense
emergency contacts. The senior medical staff will notify appropriate members of the Crisis Management Team of developing news, status, or recommended actions.
The playbook is 31 pages in length and is to be in the possession of disaster recovery coordinators at all times. It contains sensitive corporate information (lists of critical resources) and account information (such as hotel registration codes). Although the document itself is entitled “Loss of Building,” just one section is fully dedicated to actions to taken in the event a building is lost. That may the reason the owner of the artifact refers to the document as “Crisis Management Playbook.” The majority of the information is a checklist of actions to perform while dealing with the press. Ironically, nowhere in the document was a step describing who will invoke the plan, when will the plan be tested and how will the information be dispersed to staff before a crisis.
4.4.6 Summary
The review of five disaster recovery documents validates comments made during the one-on-one interview and focus group session. The artifacts reviewed had stale information and had not been updated in a timely manner. Keeping the documentation is more and more difficult as organizations continually change and outsource to of the regions of the globe. The information, specifically the detailed technical steps, require intimate working knowledge of the application and generally speaking, can only be performed by a subject matter expert that is familiar with the environment. Ironically, it does not appear that any subject matter expert has written or reviewed these documents. As stated in the focus group, most of the staff is unaware of disaster plans inside the group (e.g., organizational or building) or outside the group (e.g., being names as a backup resource in other regions). It is evident that these plans
Page: 150 of 237
Thomas Virgona Doctoral Dissertation: Defense
have been tested, but the documentation has not been updated to reflect issues that were encountered during the test. This was clear during the disaster recovery test in July, when there was no visible use of the documentation.
In this particular case, it is not imperative that the researcher fully understand the terminology used in the documentation or artifacts. The document was tailored for a specific audience with the assumption that the users of the document will understand the terminology.
The larger question is: do the users of the disaster recovery plans find the information useful during a crisis? Evidence from the focus group, interviews and disaster recovery test observations indicate that these documents have little or no value to the technologists tasked with recovery information systems. Another issue raised relates to the document control section of the artifacts (where that section was included). Some of the roles included original writer, reviewer and approver. In the case where an approver was listed, that person approved the document knowing that the disaster recovery plan test had failed. A follow up discussion with that person indicated she signed the document for ‘audit reasons’ knowing the plan would not work.
Page: 151 of 237
Thomas Virgona Doctoral Dissertation: Defense
5 Data Analysis and Findings (and relationship to prior
research)
A great deal of data have been collected that describes human aspects of disaster recovery in this study. The design of this study was constructed to extract information about people during a crisis. Rather than reading the “plans,” the research design went several steps further to allow for more freedom of thought during interviews and a focus group, therefore providing an close view of what technologists did during recovery effort on September 11,
2001. The design was intentionally open to invite the unknown. The results and insights documented in this study and valuable because they provide critical insights on how technologists interact and perform during a disaster. In addition, the results report on how large organizations plan for emergencies. These data clearly shows the important role people take in the recovery of technology system during a disaster or crisis. The complexity of information systems on Wall Street requires intimate knowledge of business and technology processes that have not been sufficiently captured in designing disaster recovery processes.
Additionally, current documentation has little of no value to staff during a crisis or disaster.
As was noted in this study, the assignment of preparing for a disaster appears to be secondary to the primary function of supporting profit making functions. Having the “best-practices” in disaster recovery does not appear to provide any competitive advantage. Additionally, disaster recovery strategy appears to be largely reactive to market conditions and demographic factors (e.g., outsourcing).
The research model presented four studies among various aspects of disaster recovery on September 11, 2001. One of the main purposes of this study was to explore empirical data
Page: 152 of 237
Thomas Virgona Doctoral Dissertation: Defense
that would help in understanding the human aspect of disaster recovery. The present study investigates below the general planning level and identifies the link between technology recovery and the human action required to achieve that task. Though exploratory and descriptive in nature, this research can be used a starting point for further technology studies that focus on providing information services during a disaster.
This study also identifies where knowledge should be captured and is not. People tend to accumulate information and knowledge informally and may not be aware of its value.
The data described in this study also suggests modifications to the initial research model.
The most significant change is to interview a “unit” individually, as a group and to review the “units” plans. The “unit” can be described as business users of an application,
Database administrators, programmers, managers, network administrators, etc. and technology that supports the application. It would be interesting to note how each unique member of the unit views disaster recovery and their role from individual viewpoint, discussing it as a group to see if there is agreement in the assumed roles and then to review the documented plans to determine if the roles and responsibilities match what was presented by the “unit.” These results could then be triangulated against the other groups to determine if there were consistent opinions that surface among the group, what is emphasized, or is an issue for one unit and not another unit. The time to perform this type of study is too large for a single dissertation and would need to be coordinated across several studies.
The Institutional Review Board approval for this dissertation lasted several months and was finalized on July 30, 2007. Ironically, this long (and somewhat painful) process provided additional insight for triangulating the data from the multiple qualitative techniques.
During the meetings and conference calls, it was beneficial to articulate how each qualitative technique would validate or contradict other parts of this study. By the end of the approval
Page: 153 of 237
Thomas Virgona Doctoral Dissertation: Defense
process, there was a clear mental model of how the data from the four independent studies would be triangulated. One simple, but clear example: one disaster recovery document required staff to work from home but many resources went into their offices to lend technical assistance.
The primary analysis performed was term occurrence, with the goal of producing a pattern or common trend. Correlation analysis is not applicable in this case. Results emanated from the following analysis (Schutt 1999):
• Reviewing research notes to identify important statements and possible ways of
coding the data.
• Determining how many people made a particular type of comments?
• Determining if and how often did the social interaction lead to
arguments/disagreements?
Interviews, a focus group, artifact analysis and an observation of a disaster recovery test all indicated the same two ‘reliance’ themes: disaster recovery documentation is not relied upon during a disaster and the recovery of information systems is heavily reliant on humans.
During the analysis of the interviews and focus groups, a number of consistent themes emerged. Undoubtedly, the most common, and passionate statements made during the research involved the immediate desire to check on the safety of family members one the attacks started. These observations signal the importance of recognizing the primacy of relationships people have with their families (Paton 1997). The focus on family has therefore created communication and coordination difficulties with public officials and other organizations in certain situations (Bolin and Borton, 1986). A framework should be developed that accommodates these needs (Paton 1997).
Page: 154 of 237
Thomas Virgona Doctoral Dissertation: Defense
There was universal agreement across all research participants (the ten focus group participants and 6 interviews) that during September 11, 2001, disaster recovery plans were not used. The rationale was varied, including availability of the plans, lack of confidence in the plan themselves and the “ridiculous idea” of reading a plan while building are falling. The research has shown that organizations often create elaborate emergency operations plans, but they fail to develop the capability to implement these plans (Auf Der Heide 1989). Disaster plans are important, but they are not enough by themselves to assure preparedness. They can be an illusion of preparedness if they are not tied to training programs, not acceptable to the intended users, not tied to the necessary staff or other resources, or not based on valid assumptions. This illusion is called the “paper plan syndrome” (Auf der Heide, 1989). The terms “lack of confidence” and “incomplete” were the term most commonly used by every interviewee and member of the focus group as a rationale of not using the documented DR plans. Those preparing for disasters should therefore ensure that their plans are realistic and achievable in practice. The most memorable comment was: With a building falling down, who will locate or print a 300 page document and start reading? Of course, proper management of the disaster recovery plan will ensure all staff are aware of the plans before a disaster strikes.
This study has demonstrated a strong link between people and the recovery of technology. Disaster recovery plans, at best, can be described as inadequate. Specifically, documentation on September 11 was either ignored or useless. As a result, staff required to restore production application relied heavily on their personal relationships with business contacts, DBAs and management. The must fundamental question of the day could not be answered: are we open for business today? This validated the Kasten (2001) study that informally developed teams are more effective than formal teams. This was evident during
Page: 155 of 237
Thomas Virgona Doctoral Dissertation: Defense
the disaster recovery test, where the formal plans and assignments in the DR plans were largely ignored, and information and problem solving was performed in the coffee room.
Rather than search for the information needed, technologists utilized their personal networks to solve problems. These personal relationships are critical during a crisis and are very informal in nature.
The interviews and focus group provided three recommendations businesses can immediately implement to improve disaster recovery efforts. Also, there was an informal recommendation (also called the “wink/wink” idea) related to information security.
1. Disaster recovery plans and emergency status should not be filed “locally’ or kept in
silos. During a disaster, the firm’s status and tasks should be made readily available
on the company’s intranet site (not internet which is available to the general public).
The site should be structured so the reader can drill down to the required information
in a short timeframe. The information needs to be concise (e.g., today is or is not a
business day) and updated in a timely manner. It was universally agreed that
extensive DR documents are useless.
2. Why wait until a crisis start to activate a DR plan? Can contingency activities start
before a disaster strikes? Disaster recovery planning, for the most part, starts once a
crisis starts. Why? If the staff can work from home during a crisis, people should be
able to work from home any day. It raises the question; Can a firm have a small
percentage (10 to 20%) of employees working at alternative locations everyday? If
the answer is yes, would they not be better prepared if a disaster were to unexpectedly
strike. This approach would also help to flesh out communication and capacity issues
in real time, instead of a canned test or during an actual crisis.
Page: 156 of 237
Thomas Virgona Doctoral Dissertation: Defense
3. One unofficial information security recommendation was agreed upon, but the
language was debated, leaving the focus group and interviewees to name this item the
“wink wink” recommendation. Information security is critical in large firms and is
often very cumbersome. The reality is that during a crisis, “brut-force” measures
require some policies to be circumvented, such as 48 hours between a change approval
and the implementation, no direct database updates via SQL tools and password
management. All agreed that information policies can’t be stopped during a crisis, but
they need to be relaxed. Due to the human elements and personal relationships, firms
need to realize that information system will be changes in a un-controlled manner
during a disaster. How these changes conflict with existing information security and
change control policies presents an issue for firms.
All of the disaster recovery plans reviewed in this study were for large firms and organizations. Logic and necessity dictates these firms require many levels and types of recovery plans. In reviewing just this small number of artifacts in this study highlighted the complexity of disaster recovery and the need for coordination and communication among the disaster recovery planners. One major example: The department and building recovery plan reviewed in this study both instructed technologists to work from home in the event of a disaster. This would appear to be a logical approach. However, a contradiction appears when the recovery plan for a production application was reviewed. That plan requires the technologists to work from the business (or operations) location to provide on-site support.
Assuming that the staff were aware of all the plans, in the event of a disaster, where should the technology staff work – home or at the business unit they support. As a manger, do you know where you staff would be and how to account for their safety? This also validates
Kasten’s (2001) finding that organizational structure can foster silos and will have a definite
Page: 157 of 237
Thomas Virgona Doctoral Dissertation: Defense
impact on knowledge flow and sharing. Stating the obvious, knowledge sharing can be difficult in an large organization.
Although this study was primarily focused on the events of September 11, 2001, several member of the focus group expressed concern with the growing trend in disaster recovery. The “worst-case” has occurred and it seems that the planning is now geared for an all-out terrorist attack and ignore the medium sized crisis that happen on a weekly basis. The topic was probably raised due to the fact that the focus group was held on a day when torrential rain forces mass transit in New York City to a virtual stop. Many people could not make it to their offices that day, and remote access capabilities (the ability to work from home) for many firms could not handle the volumes. Another example occurred on February
12th and 13th, 2008. New York City experienced an ice, snow and heavy rain event. As a result, many staff left early to avoid commuting problems.58 A quick discussion with two member of the focus group indicated that the remote access capabilities could not handle the volume that evening. It calls into question how viable is the plan to have entire divisions work from home during a crisis.
Hurricane Katrina (September 2005) occurred four years after September 11, but many member of the focus group and interviewee’s compared the events and how they differed. Much of the discussions centered on advanced warning and when the immediate danger ended. Kreps wrote of the suddenness of the disaster as in the case of September 11
(Kreps in Quarantelli 1998). There was no advanced warning (“Length of Forewarning”) to
58 -----Original Message----- From: [email protected] [mailto:[email protected]] Sent: Tuesday, February 12, 2008 9:11 PM To: [email protected] Subject: Service Advisory The LIRR is experiencing scattered residual delays of 15-20 minute delays system wide. Earlier, service on the Ronkonkoma Branch was suspended when a train struck an unauthorized vehicle at East Little Neck Road and service on the south shore was impacted by a disabled train in Valley Stream. Feb-12-2008 09:07:20PM Page: 158 of 237
Thomas Virgona Doctoral Dissertation: Defense
the general public and while an assessment of the first crash was underway, another plane struck the second tower. One member of the focus group dissented with this opinion, stating the 1993 attack on the World Trade Center was sufficient warning. In the case of Hurricane
Katrina, there was ample warning of the storm, giving planners several days to prepare. It is interesting to note that most of the destruction occurred after the event itself; the World Trade
Center Collapsed hours after being struck and the levees in New Orleans failed almost a full day after Katrina has passed. Members of the focus group spoke of timing (“Duration of
Impact“) on September 11: were there more attacks on the way and was the disaster over.
Should disaster recovery plans be invoked or will future attacks require a change to those plans? In the case of Katrina, it was known when the danger from the hurricane had passed and when the recovery could start.
Leadership was another difference noted between Katrina and September 11. Drabek illustrated the importance of leadership skills and abilities (Drabek 1987). effective emergency managers are able to motivate others and harness their knowledge and contributions for disaster preparedness. Capable emergency managers are also able to compromise, mediate and facilitate in difficult situations. Finally, strong emergency managers communicate effectively, are highly organized, and are able to maintain control under stressful situations. Five of the six interviewee’s felt New Orleans Mayor Ray Nagin did not manage the disaster well. Public finger pointing, conflicts with state officials and lack of a strong presence added to the crisis. The question of “who is in charge?” was reiterated over and over again, depicting the command and control team as inept (Quarantelli 2005).
Conversely, the same five interviewee’s stated that Mayor Rudolph Giuliani was such powerful force that the public easily rallied around his leadership. As a result, the crisis
Page: 159 of 237
Thomas Virgona Doctoral Dissertation: Defense
propelled Mr. Giuliani to the national stage and at one point was a leading candidate for the
Republic presidential nomination.59
The concept of community/government support was also discussed among the participants. In the days immediately following September 11, there was an outpouring of support across the globe. This support translated into real actions, as the focus group recalled how the Jacob Javitz Convention Center became a warehouse for supplies. One interviewee recalled the skids of dog food (used for search and rescue dogs) visible from 10th Avenue and the license plates from different states. Quarantelli (2005) wrote that with Katrina, help from nearby communities could not be provided. In many catastrophes, not only are all or most of the residents in a particular community affected, but often those in nearby localities are also impacted. As one interviewee stated, what plans account for 25% percent of a police force resigning immediately after a disaster?
One area of great debate in the focus group was the role of government surveillance in information systems. It is what I refer to as: “information management versus Information legislation” debate. The often heated discussion centered on the concept of expanding government surveillance of information systems. The argument for increased surveillance and government oversight was based on the governments needs to be proactive in an ever changing technology world. Terrorists are very familiar with banking and financial rules and have become creative in circumventing the rules to avoid detection. The argument against more legislation, such as the Patriot Act, is that the government already had sufficient information on September 11th that an attack was imminent – the information was simply not managed or communicated properly. It was also interesting to note that the debate did not
59 Mr. Guiliani has since left the presidential primary process. Page: 160 of 237
Thomas Virgona Doctoral Dissertation: Defense
follow political party affiliations. Make no mistake about this topic; people on both sides of the argument were passionate in their beliefs.
The purpose of disaster planning is to ensure a company’s survival, not just to recover computer systems. But in many industries information is the mainstay of the business, and re- creating the computer systems may be the most essential part of recovery (Edwards 1994).
This research demonstrated clear and heavy dependence on human intervention in the recovery of information systems. Many participants used the term “brut force” to describe the
Herculean effort it took to re-establish the Wall Street IT infrastructure. I can conclude with little risk of contradiction that the present study highlights the lack of confidence information technology staff shows in disaster recovery planning.
One important research discovery is that most disaster plans and preparedness activities have been based on false assumptions of human and organizational behavior
(Quarantelli 1984). Locating subject matter experts during disasters and DR testing underscored the fact that the strong interdependency still exists. Studies have demonstrated the benefit of testing to document and test these needs. Testing of DR plans establishes the responsibilities of key players (e.g. community officials, state officials, outside agencies, municipalities, first responders, hospitals, etc.) for disaster response. A joint planning and disaster rehearsal activity, for example, facilitates coordination and strengthens personal relationships among participating agencies (Dynes 1994, p. 147).
The events on September 11, 2001 also demonstrated the fluid nature of management skills during a crisis. In the days and weeks following September 11, 2001, managers were called upon to perform new and unique tasks.60 A study of professional emergency managers illustrated the importance of leadership skills and abilities (Drabek 1987). The survey
60 John Weitzel and Donald Marchand documented ad-hoc procedural decisions during the technical problems of the US Stock Market Crash of 1987 (Horton and Lewis 1991). Page: 161 of 237
Thomas Virgona Doctoral Dissertation: Defense
indicates that effective emergency managers are able to motivate others and harness their knowledge and contributions for disaster preparedness. Capable emergency managers also are able to compromise, mediate and facilitate in difficult situations. Finally, strong emergency managers communicate effectively, are highly organized, and are able to maintain control under stressful situations. Emergency managers will be required to make decisions with incomplete or inaccurate information in a period of changing and possibly hazardous conditions. In addition to the “disaster itself”, this research documented that inaccurate information will be disseminated during a crisis, thus compounding an already difficult situation. Previous studies have discovered that information outside of official channels may be lacking or inaccurate (Britton 1989). Sensationalizing, misreporting, or generating rumors about the response and/or how it was managed are prominent stressors in this situation (Patton
2003).
The disaster recovery test demonstrated the unique character humans play in disaster recovery, specifically the distinction between formal and informal information networks. By design or not, the biggest benefit from conducting a disaster recovery test appears to the tacit61 knowledge the staff obtains. Frederick Ferre stated that the definition of technology is the practical implementations of intelligence (Ferre 1988). Cultures accumulate solutions to frequently encountered problems. At no point during the simulation did the process require the participants in the test to transfer the informal knowledge gained into the formal test plans.
Process may be distributed through time in such a way that the products of earlier events can transform later events. The meeting of cognition and culture is the concept that a person’s environment is a reservoir of resources for learning, problem solving and reasoning. One of the goals of distributed cognition is to return culture, context, and history to the cognitive
61 Tacit knowledge is normally held by the organizations people while explicit knowledge is usually found in either documentation or other embodiments (Kasten 2001). Page: 162 of 237
Thomas Virgona Doctoral Dissertation: Defense
view. Lessons learned should provide guidance for future decisions and that does not appear to be the case in this instance. One critical concept of the distributed cognition approach is the complex interdependencies between people and artifacts in their work activities (Rogers
2004). The culture appears to rely heavily on tacit knowledge of staff rather than ensuring critical information is readily available when it is needed. Although, not everyone learns well from a book and staff will acquire knowledge once they are certain it is necessary and useful
(Kasten 2001).
This study used Distributed Cognition as the theoretical framework. Distributed cognition is tailored to understand the interactions among people and technology. The theory of distributed cognition seeks to understand the organization of cognitive systems (Hollan,
Hutchins and Kirsch 2000). An organization is a system with internal and external interactions, including feedback looping. Unlike traditional theories, it reaches beyond what is considered cognitive and beyond the individual to encompass interactions between people and with resources and materials in the environment. It is distributed by placing memories
(e.g., information learned during disaster recovery testing), facts (e.g., intimate knowledge of the applications batch process), or knowledge on the objects (e.g., is a business-day important), individuals (e.g., who can grant entitlement to a particular function during a crisis), and tools in our environment. As stated above, the cognitive properties of the entire system are larger than any one individual’s activity, cognitive ethnography must be event- centered. There was not one statement or observation in this study where an individual did not express a need to obtain critical information from another person or tool. Individual work tasks are no longer confined to a desk, but reach into the global networked world.
Distributed cognition views a system as a set of representations, and models the interchange of information among these representations. There is no better single example of
Page: 163 of 237
Thomas Virgona Doctoral Dissertation: Defense
this concept than the “business-day” concept discussed in this study. Financial technology on
Wall Street is heavily dependent on the concept of a business-day to perform business functions. Traditionally, the term “open for business” meant unlocking the front door and allowing customers to enter for business. Burns (in Horton and Lewis 1991) stressed the importance of team processing in technology. He recognized that the old industrial approach of segregating work and assigning it to specialists does not produce the best result when information work is involved. A carefully engineered collaborative approach is more innovative. As Flor wrote about software teams, software development can be a highly social activity involving frequent interaction between programmers and with their development tools in the performance of a task (Flor and Hutchins, 1992). The development and comprehension of a computer program is a function of how well the system performs as a whole. Other system-level variables include how well programmers communicate inside and outside the group and use of development tools.
On Wall Street, firms are heavily dependent on each other as business partners and simply can’t operate in a silo. Information, money, transactions, etc. are exchanged countless times a day between organizations. Computer applications are programmed to expect this information in an exact format on regular times. A single missed feed will impact many firms and cause reconcilement issues. A simple example stated during the focus group: Interest for certain collateral based loans is calculated by the worth of corporate stock. If the lending instruction does not receive a feed from the New York Stock Exchange (as is expected at the end of a normal business day), how will interest for that day be calculated? There is countless example of the inter-dependency of information, but the point is well made – the required information is dispersed throughout the environment. Cognitive processes are distributed across the members of a social group (in this case, technology teams) and involve
Page: 164 of 237
Thomas Virgona Doctoral Dissertation: Defense
coordination between internal and external groups. “As we build richer, more all- encompassing computational environments it becomes more important than ever to understand the ways human agents and their local environments are tightly coupled in the processing loops that result in intelligent action” (Hollan et al 2000).
As stated in the beginning of this study, single events have impacted information technology and the general society. The three examples cited from the many that were available were the introduction of the printing press, launching of Sputnik and the Internet innovation. Using Sputnik as a closing example: The changes to information technology and the general society since Sputnik can be demonstrated in two very un-scientific ways; watching football and the Honeymooners television show. The Honeymooners are heavily syndicated provides a glimpse into the state of technology in the years immediately prior to
Sputnik. The telephone was considered a luxury information technology, a stark contrast to the ubiquitous computing age we now enjoy. The year after the launch of Sputnik, the
National Football League championship played what many consider to be the greatest football game ever: The New York Giants versus the Baltimore Colts. What made the game unique is that it was the first National Football League game broadcast nationally - one game broadcast in black and white, with no remote control. It is now common to watch several games at once, use a laptop to track a fantasy league in real-time, use a cell-phone to make trades and program your television to show plays of your fantasy team players when they made a big play. And in color! Much has changed in the fifty years since the launch of Sputnik.
This research examined the changes to disaster recovery plans for financial firms located in the Wall Street area since the events of September 11, 2001. This dissertation investigated the role people played in the disaster recovery efforts and subsequent updates to disaster recovery plans to account for these roles and tasks. A safe conclusion to make after
Page: 165 of 237
Thomas Virgona Doctoral Dissertation: Defense
reviewing the data is that disaster recovery planning is no longer viewed as a dead end task and has value to a firm. Incremental strides have been made in the designing disaster recovery plans, but certainly not of the magnitude of changes introduced during the Cold War.
There was no evidence in this study that any firms has altered the Software Development Life
Cycle to enhance the disaster recovery design.
Where sparse or no plans existed before September 11, 2001, many (and one could argue too many) plans now exist. The issue is that DR plans are largely inadequate and rely heavily on subject matter experts. Reliance has these staff member poses two risks: (1) humans immediately think of family members during a crisis and (2) resources may not be available during the recovery period following a disaster. In what has been described as the era of the Internet, major information service providers (Wall Street firms) need to provide continuous information flow during disasters. Americans have enthusiastically embraced the
Internet and it has become part and parcel of everyday life, ubiquitous if you will. Lakos
(2204) goes as far as to say that web portals provide order during chaos. Or is Landauer correct in believing the behavior of human-computer systems is chaotic or worse, highly complex, dependent on many unpredictable variables, or just too hard to understand?
Page: 166 of 237
Thomas Virgona Doctoral Dissertation: Defense
6 Conclusions and Recommendations for Further Study
The study presented an intimate view of the complex nature of people during a crisis
of enormous magnitude. This research study has demonstrated the continued reliance and
dependency on humans to resolve disaster incidents. As the events of September 11, 2001
unfolded, their skills and dedication were critical factors in the recovery of Wall Street firms.
Many participants in the research made an incorrect assumption regarding the study,
believing that the goal of designing a disaster recovery plan should be absolutely no human
interaction, and that information systems should be “self-correcting.” The goal of the
research was to define the human aspects of the Wall Street recovery on September 11, 2001,
not to eliminate those roles (unrealistic). Quite to the contrary, this research demonstrate
clearly that due to the complexity of modern information systems, human intervention will
be required for the foreseeable future and needs to be accounted for in the design of
information systems. Preparedness is one of the key foundations in emergency management
and specifically, disaster recovery. This research should be ongoing. The research should be
broadened to study other business units and will provide an opportunity to compare other
subcultures and mitigating factors.
Ironically, there is only a modest amount of research on the subject (McEntire and
Myers 2004). Developing a recovery plan is critical but may appear to be a task best left in
the “too hard” basket. This has resulted in voluminous manuals which are not kept up to date
and which appear far too daunting for anybody to try to use (Edwards 1994). Research still
Page: 167 of 237
Thomas Virgona Doctoral Dissertation: Defense
struggles with the focus and improvement of disaster planning. While 90%62 of firms are
taking greater precautions in reviewing disaster-planning documentation, 80%63 of
companies lack information availability plans (Sikich 2003).
The review of disaster recover plans did indicate more attention and diligence in preparing for a disaster. The increased quality in the disaster recovery plans may be a results of the new federal guidelines published April 7, 2003 (http://www.sec.gov/news/studies/34-
47638.htm). The Federal Reserve Board, the Office of the Comptroller of the Currency
(OCC) and the Securities and Exchange Commission (SEC) published an Interagency White
Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System. The white paper identifies sound practices to ensure the resilience of the U.S. Financial System, which focus on minimizing the immediate systemic effects of a wide-scale disruption on critical financial market functions such as clearance and settlement. A suggestion for more research into the nature of the longer-term recovery process is still required, specifically the relationship between support needs, support providers, and people (Patton 1997).
History has demonstrated that information has been mismanaged somehow, somewhere, by someone, at some time, and often with disastrous consequences in terms of human misery, political misfortune or business failure (Horton and Lewis 1991). This study has uncovered several future research areas for designers of information systems. As the field of HCI treats the computer and its operator as equals (Verenikina and Gould 1998), much of the future research may be grounded in that growing discipline. Some questions for future research include:
• Do staff use disaster recovery plans during an outage or disaster? If not, why? Do the
deficiencies in disaster recovery planning mirror the problems (too rigid, does not
62 Page 43 of Sikich 2003. Page: 168 of 237
Thomas Virgona Doctoral Dissertation: Defense
account for real world development, etc.) with the System Development Life Cycle in
general? What is the gap between planned versus actual behaviors during a crisis?
• When preparing a disaster recovery plan, are religious and handicap/special needs
considered?
• Does “one-size-fits-all’ apply to disaster recovery planning? We saw in the literature
review the delineation of disaster recovery tasks by classification (Dynes in
Quarantelli 1998). Depending on the type of scenario, pre and post-crisis tasks take
on new meaning. While reviewing the disaster recovery artifacts, we saw the very
beginning of this type of framework. The Global Technology department document
the 4 levels of planning for a “loss of staff” scenario, but that was the only indication
that there was any granular definition unique types of a crisis. As was said in the
focus group, most of the planning is for a total disaster, with little or no thought into
the weekly problems (e.g., the power outage in Florida on February 26, 2008).
• When information system providers are using “brut-force” to re-establish service, are
information security concerns (such as emergency password management) placed on
hold to expedite the recovery?
• Appendix A (SDLC DELIVERABLES AND APPROVERS) of this document
indicates where the Contingency and Disaster Recovery plans are compiled within the
SDLC framework. The table also indicates the approvers of the plan – COB
Coordinator, Business Head and Project Manger. However, the follow up discussion
with an approver of a DR test plan signed off on the plan despite having the
knowledge the plan would not work during a disaster. The rationale was that signature
was simply for audit purposes. Are disaster recovery plans approved and put in place
63 Page 156 of Sikich 2003. Page: 169 of 237
Thomas Virgona Doctoral Dissertation: Defense
with the awareness that the plan itself will not work? Are approvers simply rubber
stamping the plans, and if so, does the approval process provide any value?
• Due to the intertwined nature of the financial services companies, should a centralized
organization be established to provide guidance during a catastrophe, such as stating if
a day is a “business-day” and if a company will not longer be operations (e.g., Cantor
Fitzgerald). Marchand and Weitzel (in Horton and Lewis 1991) call it an
“information ombudsman.: who dispassionately sits above the information battle and
judge when the technology becomes incapable of responding to the tide of human
events or the unfolding of natural phenomena.
• Can the informal information networks utilized be better defined so the information
flow during a crisis can tracked and utilized better by an organization?
• How critical to society is the information that Wall Street provides? Peter Drucker
(1992) said that the economy is being organized around the flow of information.
However, we cannot be sure whether it is useful or even possible to measure the value
of information in a meaningful way (Moody and Walsh 1999). There currently exists
no consensus on how to measure the value of information.
• Are several disaster recovery plans effective or even realistic? Do these plans
contradict each other, as was uncovered in study (where the building recovery plan
requires programmers to work from home but the application recovery plan requires
programmers to be on site to support the users)?
• The final recommendation is for research into disasters to focus more on qualitative
methods versus qualitative methods. Measurements have scientific value in measuring
earthquakes (The Richter scale) and hurricanes (Saffir-Simpson Hurricane Scale).
Trying to compare crisis via metrics or calculations provides no value – Would the Page: 170 of 237
Thomas Virgona Doctoral Dissertation: Defense
people of New Orleans feel better of Hurricane Katrina was rated a 5 crisis and
Chernobyl was rated a 6 crisis? Doubtful.
As a result of the exploratory nature of this study, the data and conclusions provides a rich ground upon which theorists can conduct further research. The recommendations in this section reflect the interests of this researcher and welcome the opportunity to conduct additional studies that may lead to a better understanding the behavior of people, specifically technologists, during a crisis.
Page: 171 of 237
Thomas Virgona Doctoral Dissertation: Defense
7 References
Ackerman, M. S. & Halverson, C.A. (1998). ‘Considering an Organization's Memory.’ In
Proceedings of the ACM Conference on Computer Supported Cooperative Work
(CSCW '98), New York: ACM, 39-48.
Agada, J. (1999). Inner-city gatekeepers: an exploratory survey of their information use
environment. Journal of the American Society for Information Science and
Technology, 50(1), 74-85.
Aguillo, Isidro. 2000. “A New Generation of tools for search, recovery and quality
evaluation of World Wide Web medical resources”. Journal of Management of
Medicine. Volume 14. Number 3 / 4. Pages: 240-248.
Alexander, David. 2002. From civil defense to civil protection – and back again. Disaster
Prevention and Management. Volume: 11. Issue 3. Available at:
http://www.emeraldinsight.com/Insight/ViewContentServlet?Filename=Published/Em
eraldFullTextArticle/Articles/0730110301.html
[Anonymous]. 1957. "Impact of Russian Satellite to Boost U.S. Research Effort," Aviation
Week, October 14, 1957, pp. 28–29.
[Anonymous]. 2005. “Only veteran medical records remain safe from New Orleans flood
waters“. Available at:
http://www.thedailycitizen.com/articles/2005/09/15/news/features/featuresrecords.prt.
Last visit: 11/24/2006
Auf der Heide, E. 1989. Disaster Response: Principles and Preparation and Coordination,
The C.V. Mosby Company, St. Louis, MO, .
Page: 172 of 237
Thomas Virgona Doctoral Dissertation: Defense
Baecker, Ronald, et al. 1995. “Readings in Human-Computer Interaction: Toward the Year
2000”. 2nd edition. (January 15, 1995).
Baeza-Yates, Ricardo and Berthier Ribeiro-Neto. 1999. Modern Information Retrieval. New
York: ACM Book Press.
Barsky, Lauren, Joseph Trainor, Manuel Torres. 2006. Miscellaneous Report #53: Disaster
realities in the aftermath of Hurricane Katrina; Revisiting the looting Myth.
University of Delaware. Disaster Research Center. Available at:
http://dspace.udel.edu:8080/dspace/bitstream/19716/2367/1/Misc+Report+53.pdf.
Last visit: 01/02/2007.
Ball, L.D. 2002. “CIO on center stage: September 11 changes everything “. Information
Systems Security. Volume 11. Number 2. Np: Auerbach Publications. (May-June
2002): Pages: 25-29.
Banipal. Kulwinder. 2006. Strategic approach to disaster management: lessons learned from
Hurricane Katrina. Disaster Prevention and Management. 2006. Volume: 15 Issue: 3
Page: 484 – 494.
Barker, Colin. 2007. The top 10 IT disasters of all time. ZDNet.co.uk. Published: 22 Nov
2007. Available at:
http://resources.zdnet.co.uk/articles/0,1000001991,39290976,00.htm. Last visit:
12/5/2007.
Barr, Jean. 2003. “A disaster plan in action: How a law firm in the World Trade Center
survived September 11, 2001 with vital records and employees intact.” Information
Management Journal. May/Jun 2003. Available at:
http://www.findarticles.com/p/articles/mi_qa3937/is_200305/ai_n9260326/pg_1.
Page: 173 of 237
Thomas Virgona Doctoral Dissertation: Defense
Boehm, Barry. 1988. “A Spiral Model of Software Development and Enhancement”.
Computer. Volume 21. Number 5. May 1988. Pages 61-72.
Bolin, R., Borton, P. (1986). Race, Religion and Ethnicity in Disaster Recovery, Institute of
Behavior Science. University of Colorado. Boulder, CO, Monograph Series No. 42.
Brake, Jeffrey D. 2003. “Terrorism and the Military’s Role in Domestic Crisis Management:
Background and Issues for Congress”. Report for Congress. Foreign Affairs,
Defense, and Trade Division. Available at:
http://www.terrorisminfo.mipt.org/pdf/crs_rl30938.pdf
Beacham, A.E. and McManus, D.J. 2004. “Recovery of financial services firms in the World
Trade Center, post September 11, 2001/01”. Information Systems Security. Volume
13. Number 2. (May-June 2004): Page: 46-55.
Beckett, Paul, and Jathon Sapsford. 2001. “Rebuilding Wall Street: How Wall Street's
Nervous System Caused Pain”. Wall Street Journal, September 21, 2001.
Beniger, James R. 1986.The Control Revolution: Technological and Economic Origins of the
Information Society. N.p.: Harvard University Press.
Bennet, George E. 1988. Librarians in Search of Science and Identity: the elusive profession.
Metuchen, N.J. & London: The Scarecrow Press, Inc. (1988).
Berg, Bruce L. 2001. Qualitative Research Methods For the Social Sciences. Needham
Heights, MA: Allyn and Bacon.
Berman, A. 2002. “Lessons learned: the aftermath of September 11. Information Systems
Security. Volume 11. Number 2. (May-June 2002).
Berners-Lee, Tim, Robert Caliliau, Jean-Francois Groff, and Bernd Pollerman. 1992.
“World-Wide Web: The Information Universe”. Electronic Networking, Volume 2,
Number 1. Spring 1992. Pages 52-58.
Page: 174 of 237
Thomas Virgona Doctoral Dissertation: Defense
Bertalanffy, L. V. 1968. General System Theory foundations development applications.
George Brazille Inc., New York.
Blanchard, B.W. 1984. American Civil Defense 1945-1984: The Evolution of Programs and
Policies, National Emergency Training Center, Federal Emergency Management
Agency, Emmitsburg, MD.
Blustein, Paul, and Kathleen Day. 2001. Industrialized Nations Act to Reassure World.
Washington Post. September 13, 2001.
Booms, Hans. 1997. “Society and the Formation of a Documentary Heritage: Issues in the
Appraisal of Archival Sources”. Archivaria. (Summer 1987).
*Note: Originally published 1972.
Borko, H. 1968. The foundations of information systems. In E.B. Montgomery (Ed.), The
foundations of access to knowledge: A symposium: Syracuse: Syracuse University
Press. In Hahn, Trudi and Michael Buckland. 1998. “Historical Studies in
Information Science”. Medford, NJ: Information Today, Inc.
Bowles, Mark D. 1998. “The Information Wars: Two Cultures and the Conflict in
Information Retrieval, 1945-1999”. in Bellardo Hahn, Trudi and Williams, Robert V.
– editors. “Proceedings of the 1998 Conference on the History and Heritage of
Science Information Systems”. ASIST.
Bozman Jean S. 1994. “Quake shakes IS, leaves networks in disarray”. Available at:
http://www.computerworld.com/news/1994/story/0,11280,16751,00.html. Last visit:
02/22/2006. (January 24, 1994).
*Note: “disarry” was misspelled in the URL.
Page: 175 of 237
Thomas Virgona Doctoral Dissertation: Defense
Britton, N. 1989. Anticipating the Unexpected: Is the Bureaucracy Able to Come to the
Dance?, Cumberland College of Health Sciences, Disaster Management Studies
Centre, Sydney, Working Paper No. 1, .
Brooks, John. 1969. Once in Golconda. New York: Harper & Row.
Brown, John and Paul Duguid. 2002. The Social Life of Information. N.p.: Harvard Business
School Press.
Buckland, Michael. 1998. “Overview of the History of Science Information Systems”. in
Bellardo Hahn, Trudi and Williams, Robert V. – editors. “Proceedings of the 1998
Conference on the History and Heritage of Science Information Systems”. ASIST.
Bush, Vannevar. 1945. “As we may think. The Atlantic Monthly. July 1945. Volume 176,
No. 1; pages 101-108.
Carafano, James Jay. 2006. "Beyond the Rainbow Plans: Military Industrial and Mobilization
Planning in an Uncertain Century," Heritage Foundation Backgrounder No. 1959,
August 10, 2006, Available at: at
www.heritage.org/Research/NationalSecurity/bg1959.cfm.
Carter, Guy D., C.P. Clare and D.C.J. Thorogood. 1987. Engineering project management
techniques and their application to computer projects. Software Engineering Journal.
Volume 2. Number 1. January 1987. Pages 15-20.
Case. 2002. in Johnson, C.A. "Choosing people: The role of social capital in information
seeking behaviour". Information Research. Volume 10. Issue 10. (2004).
*Note: Available at http://InformationR.net/ir/10-1/paper201.html.
Castells, Manuel. 2003. The Internet Galaxy: Reflections on the Internet, Business, and
Society. Oxford University Press. (April 2003).
Page: 176 of 237
Thomas Virgona Doctoral Dissertation: Defense
Centers for Disease Control and Prevention, Department of Health and Human services.
2006. “Surveillance for World Trade Center Disaster Health Effects Among
Survivors of Collapsed and Damaged Buildings”. Morbidity and Mortality Weekly
Report. April 7, 2006 / Vol. 55 / No. SS-2. Available at:
http://www.cdc.gov/MMWR/PDF/ss/ss5502.pdf. Last visit: December 27th, 2006.
Chandler, Alfred D. and James W. Cortada. 2000. A Nation Transformed by Information:
How Information Has Shaped the United States from Colonial Times to the Present.
N.p.: Oxford University Press. (2000).
Chen, Peter P. (1976). "The Entity-Relationship Model - Toward a Unified View of Data".
ACM Transactions on Database Systems. Volume 1. Number 1. Pages: 9-36.
Available at: http://csc.lsu.edu/news/erd.pdf. Last visit: 11/06/2007.
Childs, Merilyn, Michael Morris, Valerie Ingham. (2004). The rise and rise of clean, white-
collar (fire-fighting) work. Disaster Prevention and Management. Volume 13.
Number 5. 2004. Pages 409-414.
Choo, C.W. (1993). Environmental scanning: acquisition and use of information by chief
executive officers in the Canadian telecommunications industry. Unpublished Ph.D.
Thesis, University of Toronto, Toronto.
Clanchy, M.T. 1991. From Memory to Written Record: England, 1066-1307. 1979, rev. ed.
N.p.: Harvard University Press.
Commander, Naval Meteorology and Oceanography Command at Stennis Space Center
(CNMOC). 2005: Preliminary Model Hindcast of Hurricane Katrina Storm Surge,
Stennis Space Center, MS, Available at:
http://www.fnmoc.navy.mil/products/KATRINA/Notice.pdf. Last Visit: 01/02/2007.
Page: 177 of 237
Thomas Virgona Doctoral Dissertation: Defense
Cole. 1981. in Ravichandran, T. and Arun Rai. “Quality Management in Systems
Development: An organizational System Perspective”. MIS Quarterly. Volume 24.
Number 3. (September 2000): Pages 381-415.
Connell, Rory. Collective Behavior in the September 11,2001 Evacuation of the World Trade
Center. Disaster Research Center. University of Delaware. Newark, DE 19716.
Available at: http://dspace.udel.edu:8080/dspace/bitstream/19716/683/1/PP313.pdf.
Last visit: 12/28/2006.
Conway, Steve, Ian Combe , and David Crowther. 2003. Strategizing networks of power and
influence: the Internet and the struggle over contested space. Managerial Auditing
Journal. Volume 18. Number 3. pp. 254-262. Available at:
http://www.emeraldinsight.com/Insight/ViewContentServlet?Filename=Published/Em
eraldFullTextArticle/Articles/0510180308.html
Costa, Thomas F. 2001. Important Notices. September 12 and following. New York,
NY:Government Securities Clearing Corporation.
Cox, Richard J. 2000. Closing an Era: Historical Perspectives on Modern Archives and
Records Management. Greenwood Press. Westport, Connecticut.
Crichard, M. 2004. “The role of legislation in disaster recovery”. Computing. (Oct. 2004).
Page: 45.
Day, Brennan, Burnice Mckay Ruth, Michael Ishman, Ed Chung. (2004). The new normal:
lessons learned from SARS for corporations operating in emerging markets.
Management Decision. Jul 2004 Volume: 42 Issue: 6 Page: 794 – 806. Available at:
http://www.emeraldinsight.com/Insight/ViewContentServlet?Filename=Published/Em
eraldFullTextArticle/Articles/0010420606.html
Page: 178 of 237
Thomas Virgona Doctoral Dissertation: Defense
Danielyan, Artyon. 2001. Russian rouble slips, trade upset by U.S. woes. Reuters News,
September 13, 2001.
Demarest, Marc. 1997. “The Politics of Data Warehousing” Available at:
http://www.noumenal.com/marc/dwpoly.html. Last visit: February 21st, 2006. (June
1997).
Dervin, B. 2003. Human studies and user studies: a call for methodological interdisciplinary.
Information Research, 9(1).
Note: Also available at http://informationr.net/ir/9-1/paper166.html. Last visit:
October 10th, 2006.
Dentinger, Sue. 1998. “The public Electronic Library: Web-Munuing versus OPAC
Cataloging.” Library Hi Tech. Issue 63, Volume 16:3-4. pp 89-94.
Dory, Amanda J. Dory. (2003). American Civil Security: The U.S. Public and Homeland
Security. The Washington Quarterly. Volume 27:1. Pages 37–52.
Drabek, T.E. 1987. The Professional Emergency Manager, Institute for Behavioral Science,
Boulder, CO,.
Drucker, P. 1992. "The Economy's Power Shift". Wall Street Journal. September 24,
1992.
Dynes, R.R. 1994. "Community emergency planning: false and inappropriate analogies",
International Journal of Mass Emergencies and Disasters, Vol. 12 No.2, pp.141-58.
Edwards, Bruce. 1994. Developing a Successful Network Disaster Recovery Plan.
Information Management & Computer Security. Volume 2. Number 3. 1994. pp. 37-
42.
Eisenstein, Elizabeth L. 1979. The Printing Press as an Agent of Change. N.p.: Cambridge
University Press.
Page: 179 of 237
Thomas Virgona Doctoral Dissertation: Defense
Fagan, Amber L., Colleen M. More, Heather Warren. 2005. Conceptual Model of Emergency
Management in the 21st Century. 10th International Command and Control Research
and Technology Symposium. Available at:
http://www.dodccrp.org/events/2005/10th/CD/papers/147.pdf. Last Visit: 12/28/2006.
Ferre, Frederick. 1988. Philosophy of Technology. Engelwood Cliffs, New Jersey: Prentice
Hall.
Fidel, R. and A.M. Pejtersen. 2004. “From information behaviour research to the design of
information systems: the Cognitive Work Analysis framework”. Information
Research. Volume 10. Issue 1.
*Note: Available at http://InformationR.net/ir/10-1/paper210.html.
Fischer, David Hackett. 1970. Historians Fallacies: Toward a Logic of Historical Thought.
New York, New York: Harper & Row, Publishers.
Fisher, K. E. 2004. "Information behaviour of Migrant Hispanic Farm Workers and Their
Families in the Pacific Northwest". Information Research. Volume 10. Issue 1.
*Note: Available at http://InformationR.net/ir/10-1/paper199.html
Fitts, Paul M. 1954. The information capacity of the human motor system in controlling the
amplitude of movement. Journal of Experimental Psychology, volume 47, number 6,
June 1954, pp. 381-391.
Flor, N. V. and Hutchins, E. 1992. Analyzing Distributed Cognition in Software Teams: A
Case Study of Collaborative Programming During Adaptive Software Maintenance. In
J. Koenemann-Belliveau, T. Moher. and S. Robertson (eds.), Empirical Studies of
Programmers: Fourth Workshop. Ablex, Norwood, NJ, 36-64.
Fowler, Martin. 2004. UML Distilled: A Brief Guide to the Standard Object Modeling
Language. Third Edition. Boston, San Francisco, New York, Toronto, Montréal,
Page: 180 of 237
Thomas Virgona Doctoral Dissertation: Defense
London, Munich, Paris, Madrid, Capetown, Sydney, Tokyo, Singapore, Mexico City:
Addison-Wesley.
Garbis, C. and Wærn, Y. 1999. Team coordination and communication in a rescue command
staff: the role of public representations. La Traveail Humain, 62: 273-291.
Garvey, M.J. and McGee, M.K. 2002. “New priorities [disaster recovery planning] “.
InformationWEEK. Number 905. (September 2002): Pages: 36-40.
General Accounting Office. 2003. “Potential Terrorist Attacks: Additional Actions Needed to
Better Prepare Critical Financial Market Participants”. Washington, DC: U.S.
General Accounting Office. February 2003.
Glebocki, Jerry and David Lancaster. 1984. “In Search of the Wild Hypothesis, An
adventure in statistics for Non-statisticians”. TEC Building: by Anderson-Bell.
*Note: Copy available upon request. 11479 Pine Drive. Suite 400 W Parker, CO
80134. (303) 841-9755.
Gray, W.D. and E.M. Altmann. 2001. “Cognitive modeling and human-computer
interaction”. In W. Karwowski (ed.), International encyclopedia of ergonomics and
Human Factors. Volume 1. New York:Taylor and Francis, Ltd. Pages 387-391.
Grygo, Eugene. 2001. “U.S. recovery: IT heroes toil to restore trading.” Infoworld.
Available at:
http://www.infoworld.com/articles/hn/xml/01/09/21/010921hnheroes.html
Hahn, Trudi Bellardo. 1996. “Pioneers of the Online Age”. Information Processing &
Management. Volume 32. Number 1. Pages 33-48.
Note: Also in
Hahn, Trudi Bellardo. 1998. “Pioneers of the Online Age”. in Bellardo Hahn, Trudi
and Buckland, Michael – editors. “Historical Studies in Information Science”. ASIST. Page: 181 of 237
Thomas Virgona Doctoral Dissertation: Defense
Hayhoe, George F. 2002. Did 11 September Change Our Profession? Technical
Communication. Volume 49. Number 1. February 2002. pp. 15-16.
Herring, Ron. 2002. “Lessons learned from September 11”. Plant Engineering. Volume 56.
Number 3. (March 2002).
Heylighen, F., C. Joslyn, V. Turchin. 1999. What are Cybernetics and Systems Science?
Available at: http://pespmc1.vub.ac.be/CYBSWHAT.html. Last visit: October 5th,
2001.
Herbstmana, Julie B., Robert Frank, Margo Schwaba, D’Ann L. Williams, Jonathan M.
Sameta, Patrick N. Breysse and Alison S. Geyh. 2005. Respiratory effects of
inhalation exposure among workers during the clean-up effort at the World Trade
Center disaster site. Environmental Research. Volume 99. Issue 1. September 2005.
Pages 85-92. Available at:
http://www.sciencedirect.com/science?_ob=ArticleURL&_aset=V-WA-A-W-Z-
MsSAYWA-UUA-U-AACZDUZWEB-AACVWYDUEB-BAYVBDWV-Z-
U&_rdoc=1&_fmt=summary&_udi=B6WDS-4DTKH84-
1&_coverDate=09%2F30%2F2005&_cdi=6774&_orig=search&_st=13&_sort=d&vie
w=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=2fc710dd
3ae3b7488b1fb002d704017f.
Hick, W. E. 1952. On the rate of gain of information. Quarterly Journal of Experimental
Psychology, 4:11-26, 1952.
Hildreth, Charles R.1997. “The Use and Understanding of Keyword Searching in a
University Online Catalog”. Information Technology and Libraries. (June 1997):
Pages 52-62.
Page: 182 of 237
Thomas Virgona Doctoral Dissertation: Defense
Hlavacek, D.M., K.A. Madsen and R.M. Reimer. 2004. “A vendor and service provider
partnership for preparing to manage disaster recovery”. Bell Labs Technical Journal.
Volume 9. Number 2. (2004): Pages: 173-180.
Hogan, Michael J. 1987. The Marshall Plan: America, Britain and the reconstruction of
Western Europe, 1947-1952. New York, New York: Cambridge University Press.
Hollan, James, Edwin Hutchins and David Kirsh. 2000. "Distributed Cognition: Toward a
new Foundation for Human- Computer Interaction Research". ACM Transactions on
HCI. Volume 7. Issue 2. June 2000. Pages 174-196.
Horton, Forest Woody and Lewis, D. 1991. Great information disasters. London:Aslib.
Horton, Forest Woody, Jr. 1985. Information resource management; harnessing information
assets for productivity gains in the office, factory and laboratory. Prentice
Hall:Englewood Cliffs, New Jersey. 1985
Horton, Forest Woody, Jr. 1994a. Extending the Librarian's Domain: A Survey of Emerging
Occupation Opportunities for Librarians and Information Professionals. SLA
Occasional Papers Series, Number Four. Special Libraries Association:Washington,
DC 20009-2508. Publication Date: 1994-00-00.
Horton, Forest Woody, Jr. 1994b. Analyzing benefits and costs; a guide for information
managers. International Development Research Centre:Ottawa, Canada. June 1994.
Horton, Forest Woody, Jr. 1982. Understanding U.S. Information Policy. Library of
Congress Cataloging in Publication Data:Washington, D.C.. Volumes 1, 2, 3 and 4.
Note: Volume 1: Resources for the information economy.
Volume 2: The participants in the Information Marketplace.
Volume 3: Assets of the information Society.
Volume 4: The Information Policy Primer.
Page: 183 of 237
Thomas Virgona Doctoral Dissertation: Defense
Hseih-Yee, Ingrid (Proposed by). 2000. “A Delphi Study on Metadata: Curriculum
Implications and Research Priorities”. School of Library & Information Science.
Catholic University of America. Washington, D.C. 20064. (September 2000).
*Note: Physical copy available upon request. Author e-mail: [email protected] .
Hudson, Susan. 1988. How to Conduct Community Needs Assessment Surveys in Public Parks
and Recreation. Pub Horizons. (December 1, 1988). Chapter 6 – “How to ask
questions”.
Hughes, James W. and Marla K. Nelson. 2002. “The New York region's post-September 11
economic geography”. Transportation Quarterly. Volume 56. Number 4. Edward J.
Bloustein School of PPP Rutgers University, New Brunswick, NJ, United States. (Fall
2002): Pages: 27-42.
Hunter, Gregory S. 1997. Developing and Maintaining Practical Archives (How-to-Do-It
Manuals for Libraries, No. 122). New York: Neal-Schuman.
Hutchins, E., & Klausen, T. 1996. Distributed cognition in an airline cockpit. In Engeström,
Y., & Middleton, D. (Eds.), Cognition and communication at work, 15-34. Cambridge,
UK: Cambridge University Press.
Hutchins, E. & Palen, L. 1997. Constructing Meaning from Space, Gesture, and Speech. In L.
B. Resnick, R. Saljo, C. Pontecorvo, and B. Burge (Eds) Discourse, tools, and
reasoning: Essays on situated cognition. Heidelberg, Germany: Springer-Verlag. Pp.
23-40.
IBM. 1969. “Flowcharting Techniques”. Available at: http://www.fh-
jena.de/~kleine/history/software/IBM-FlowchartingTechniques-GC20-8152-1.pdf.
Last visit: 11/07/2007.
Page: 184 of 237
Thomas Virgona Doctoral Dissertation: Defense
Irwin, R.L. 1989. "The Incident Command System (ICS)". Disaster Responses: Principles of
Preparation and Coordination. Mosby, St Louis, MO.
Jackson, A.A. 1994. Recent developments in civil protection and the implications for disaster
management in the United Kingdom", International Journal of Mass Emergencies and
Disasters. Vol. 12. No.3. pp.345-55.
Jackson, C. 2002. “CSI checklist: how the September 11 attack should impact your continuity
planning”.
Computer Security Journal. Volume18. Number 1. (Winter 2002): Pages: 1-7.
Johnson, C.A. 2004. "Choosing people: The role of social capital in information seeking
behaviour". Information Research. Volume 10. Issue 1.
*Note: Available at http://InformationR.net/ir/10-1/paper201.html
Johnston. Barry, Oana M. Nedelescu. (2006). The impact of terrorism on financial markets.
Journal of Financial Crime. Volume: 13. Issue: 1; 2006. Research paper.
Johnson, Jackie. 2002. 11th September, 2001: will it make a difference to the global anti-
money laundering movement?. Journal of Money Laundering Control. 2002 Volume:
6. Issue: 1. Page: 9 – 16.
Jones, Karen Sparck and Peter Willett. 1997. “Readings in Information Retrieval”. Edited
by Karen Sparck Jones and Peter Willett. San Francisco, CA: Morgan Kaufmann
Publishers.
Jones, Peter H. “Information practices and cognitive artifacts in scientific research.”
Cognition, Technology & Work. Volume 7, Number 2 / July, 2005. Pages 88-100.
Kahan, S. 2005. “Plan for the worst [disaster recovery] “. Accounting Technology. Volume
21. Number 4. (May 2005).
Page: 185 of 237
Thomas Virgona Doctoral Dissertation: Defense
Kalinsky, L. 2002. "Duty calls: special event dynamics post-September 11". Stadia. Volume 14. (2002): Pages 15-16.
Karat J. and C. M. Karat. 2003. “The Evolution of user-centered focus in the human-computer
interaction field”. IBM Systems Journal. Volume 42. Number 4. Pages 532 – 541.
Karwowski, W. 2001. in Osorio, Nestor L. and Jitka Hurych. ““Literature of disasters from
the human factors point of view: a descriptive analysis”. Collection Building.
Volume: 23. Issue: 2. (2004).
*Note: Research paper.
Kasten, Joseph. 2001. “Knowledge Strategy Drivers: An Exploratory Study”. A Dissertation
Submitted to the Faculty of Long Island University by in partial fulfillment of the
requirements for the degree of Doctor of Philosophy. November 10, 2001.
Kaufmann, Morgan, et al. 1995. Readings in Human-Computer Interaction: Toward the Year
2000. 2nd edition. (January 15, 1995).
Kennedy, Peter, Charles Perrottet and Charles Thomas. 2003. “Scenario planning after
September 11, 2001: managing the impact of a catastrophic event”. Strategy &
Leadership. Volume 31. Number 1. (2003): Pages: 4-13.
Kent, Allen. 1998. “Pioneer’s Reminiscences”. in Bellardo Hahn, Trudi and Williams, Robert
V. – editors. “Proceedings of the 1998 Conference on the History and Heritage of
Science Information Systems”. ASIST.
Kippenberger, T. 1999. How did we get to a World Wide Web? The Antidote. Volume 4.
Issue 5. 1999. Research Paper.
Kerins, G., R. Madden and C. Fulton. 2004. "Information seeking and students studying for
professional careers: the cases of engineering and law students in Ireland."
Information Research. Volume 10. Number 1.
Page: 186 of 237
Thomas Virgona Doctoral Dissertation: Defense
*Note: Available at http://InformationR.net/ir/10-1/paper208.html.
Kincaid, J. Peter, Joseph Donovan, Beth Pettitt. 2003. “Simulation techniques for training
emergency response”. International Journal of Emergency Management. Volume 1.
Number 3.August 2003, pp. 238-246.
Koehler, Wallace. 1999. “An Analysis of Web Page and Web Site Constancy and
Permanence.” Journal of the American Society for Information Science. Volume 50.
(1999): Pages: 162-180.
Koenig, Michael. 1987. In Hahn, Trudi Bellardo. “Pioneers of the Online Age”. Information
Processing and Management. Volume 32. Number 1. (1996): Pages 33 – 48.
*Note: Interview.
Krebsbach, K. 2004. “Banks, the reluctant warriors [disaster recovery for terrorist threats]”.
US Banker. Volume 114. Number 9. (Sept. 2004): Page: 22.
Lacker, Jeffrey M. 2003. “Payment System Disruptions and the Federal Reserve Following
September 11, 2001.” Federal Reserve Bank of Richmond, Richmond, Virginia,
23219, USA. December 23, 2003.
Available at:
http://scholar.google.com/scholar?hl=en&lr=&q=cache:4MXjwHBHq8EJ:www.rich.f
rb.org/pubs/working_papers/pdfs/wp03-
16.pdf+system+outages+9+11+financial+services
Last visit: 03/07/2006
Lakos, Amos A. 2004. “Bulletin of the American Society for Information Science and
Technology; Special Section; Portals in Libraries”. Volume 31. Number 1. (October
/ November 2004).
Page: 187 of 237
Thomas Virgona Doctoral Dissertation: Defense
Landauer, T.K. (1991). Let's get real: a position paper on the role of cognitive psychology in
the design of humanly useful and usable systems. In: Carroll, J.M. (ed.) Designing
Interaction: Psychology at the Human-Computer Interface. Cambridge University
Press, Cambridge, UK.
Leckie, G.J., Pettigrew, K.E., & Sylvain, C. (1996). Modeling the information seeking of
professionals: a general model derived from research on engineers, health care
professionals, and lawyers. Library Quarterly, 66(2), 161-193.
Leazer, Gregory H. and Richard P. Smiraglia. 1999. “Bibliographic Families in the Library
Catalog: A Qualitative Analysis and Grounded Theory”. Library Resources and
Technical Services. Volume 43. Number 4.
Littlefield, James E., Yeqing Bao, Don L. Cook. 2000. Internet real estate information: are
home purchasers paying attention to it? Journal of Consumer Marketing. Volume 17.
Number 7.pp. 575-590.
Note: Available at
http://www.emeraldinsight.com/Insight/ViewContentServlet?Filename=Published/Em
eraldFullTextArticle/Articles/0770170702.html
Mackenzie, Donald A., and Judy Wajcman, eds., 1999. The Social Shaping of Technology.
Open University Press.
Marlin, S. and M.J. Garvey. 2004. “Disaster-recovery spending on the rise [financial
company]”. InformationWEEK. Number 1001. (August 2004): Page: 26.
Martin, Henri-Jean. 1994. The History and Power of Writing. Chicago: University of
Chicago Press.
McAndrews, James J., and Simon Potter. 2002. Liquidity Effects of the Events of September
11, 2001. FRBNY Economic Policy Review. Volume 8. Issue 1. Pages 59-79.
Page: 188 of 237
Thomas Virgona Doctoral Dissertation: Defense
McCrohan Kevin F. 2003. “Facing the threats to electronic commerce”. Journal of Business
& Industrial Marketing. Volume 18. Number 2. (2003): Pages 133-145.
McEntirem,David A., Amy Myers. 2004. Preparing communities for disasters: issues and
processes for government readiness. Disaster Prevention and Management. Volume
13. Number 2. 2004. pp. 140-152.
McHugh, Charles P. (1995). “Preparing public safety organizations for disaster response: a
study of Tucson, Arizona's response to flooding”. Disaster Prevention and
Management. Year Dec 1995. Volume 4. Issue 5. Page: 25 – 36.
McEntire, David A. 2004. “Development, disasters and vulnerability: a discussion of
divergent theories and the need for their integration”. Disaster Prevention and
Management. Jul 2004. Volume: 13. Issue: 3. Page: 193 – 198.
McNamara, Carter. 1999. “Thinking About Organizations as Systems”. Available at:
http://www.mapnp.org/library/org_thry/org_sytm.htm. Last visit: 02/21/2006. (1999).
Meyer, E. and E. Poniatowka. 1988. “Documenting the earthquake of 1985 in Mexico City”.
Oral History Review. Volume 16. Pages 1-31.
Middleton, P. 1999. “Managing information system development in bureaucracies”.
Information and Software Technology. Volume 41. (1999): Pages 473-482.
Miller, George A. 1956. “The Magical Number Seven, Plus or Minus Two: Some Limits on
Our Capacity for Processing Information”. The Psychological Review. Volume 63.
(1956): Pages: 81-97.
Mitcham, Carl. 1994. Thinking through Technology: The Path Between Engineering and
Philosophy. N.p.: University of Chicago Press.
Page: 189 of 237
Thomas Virgona Doctoral Dissertation: Defense
Moody, Daniel. Peter Walsh. 1999. Measuring The Value Of Information: An Asset Valuation
Approach. Paper presented at the Seventh European Conference on Information
Systems (ECIS'99), Copenhagen.
Moore, David Wayne, Karen Alicia Ranson, Lawrence Edward Sullivan, Donald Prentiss
Terry, Gregory Walter Vance, Vern Lee Watts. 2004. Database recovery to any point
in time in an online environment utilizing disaster recovery technology. Available at:
http://patft.uspto.gov/netacgi/nph-
Parser?Sect2=PTO1&Sect2=HITOFF&p=1&u=%2Fnetahtml%2FPTO%2Fsearch-
bool.html&r=1&f=G&l=50&d=PALL&RefSrch=yes&Query=PN%2F6732123
Morgan, Steve. 1995. Performance assessment in academic libraries. London and New York:
Mansell Pub.
*Note; Hard copy available upon request.
Moss, Mitchell L., Anthony M. Townsend. 2006. “Disaster Forensics: Leveraging Crisis
Information Systems for Social Science”. Available at:
http://urban.blogs.com/research/files/Moss-Townsend-ISCRAM2006-final.pdf. Last
Visit: 01/01/2007.
Muir, A., Shenton, S. (2002), "If the worst happens: the use and effectiveness of disaster plans
in libraries and archives", Library Management, Vol. 23 No.3, pp.115-23.
Nakagawa, Yuko and Rajib Shaw. 2004. Social Capital: A Missing Link to Disaster
Recovery. International Journal of Mass Emergencies and Disasters. March 2004.
Vol. 22. No. 1, pp. 5-34.
Nardi, Bonnie A. 1995. “Context and Consciousness: Activity Theory and Human-
Computer Interaction”. Available at:
Page: 190 of 237
Thomas Virgona Doctoral Dissertation: Defense
http://www.acm.org/pubs/interactions/vol2no4/depts/book.htm. Last visit: February
21st, 2006. (October 1995).
National Archives and Records Administration. 1996. Vital Records and Records Disaster
Mitigation and Recovery - An Instructional Guide. National Archives and Records
Administration. Office of Records Administration. College Park, MD. 1996.
Neal, R. 2003. “A project methodology for disaster recovery testing in a server based
environment”. Information Systems Security. Volume 12. Number 5. (Nov.- Dec.
2003): Pages: 52-67.
Nijboer, Jelke. 2004. Big Brother versus anonymity on the Internet: implications for Internet
service providers, libraries and individuals since September 11, 2001. New Library
World. Volume 105. Number 7/8. pp. 256-261. Available at:
http://www.emeraldinsight.com/Insight/ViewContentServlet?Filename=Published/Em
eraldFullTextArticle/Articles/0721050702.html
Noll, Michael. 2001. “Telecommunications in times of crisis: reflections on September 11”.
Info. Volume: 3. Issue: 6.
Norman, Donald A. 1998. The Design of Everyday Things. New York: Basic Books:
Osborne, Larry N. and Margaret Nakamura. 2000. Systems Analysis for Librarians and
Information Professionals. 2nd edition. Libraries Unlimited. (2000).
Otto, James R.; James H Cook and Q.B. Chung. 2001. Extensible markup language and
knowledge management”. Journal of Knowledge Management. Volume 5. Number
3. (2001): Pages 278 – 284.
Oz, Effy. 2003. The World Trade Center Disaster: A Study on Business Continuity Planning
at Organizations
Page: 191 of 237
Thomas Virgona Doctoral Dissertation: Defense
Directly Affected by the Sept. 11 Tragedy. Available at:
http://www.strohlsystems.com/MediaPR/_files/WTCReport.pdf
Paton, Douglas. 1997. “Post-event support for disaster workers: integrating recovery
resources and the recovery environment”. Disaster Prevention and Management.
1997. Vol. 6, Issue. 1; pg. 43.
Paton, Douglas. 2003. Stress in disaster response: a risk management approach. Disaster
Prevention and Management. Volume 12. Number 3. 2003. pp. 203-209.
Patterson, George T. 2002. “Predicting the effects of military service experience on stressful
occupational events in police officers”. Policing: An International Journal of Police
Strategies & Management. Volume 25. Issue: 3. September 2002. Page: 602 – 618.
Available at:
http://www.emeraldinsight.com/Insight/ViewContentServlet?Filename=/published/em
eraldfulltextarticle/pdf/1810250307.pdf
Payne, Rodger A. 1994. Public Opinion and Foreign Threats: Eisenhower's Response to
Sputnik. Armed Forces & Society. Volume 21. Number 1. Pages 89-111.
Pawlak, Zdzisaw. 2002. “Rough sets and intelligent data analysis”. Information Sciences.
Volume 147, Issues 1-4, November 2002, Pages 1-12. Available at:
http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V0C-464JTM7-
1&_user=10&_coverDate=11%2F30%2F2002&_rdoc=1&_fmt=&_orig=search&_sor
t=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=3
a1e85cc378bfca495f5f890fef0f5f8.
Perrault, Anna H. and Marjo Arseneau. 1995. “User Satisfaction and Interlibrary Loan
Services: A Study at Louisiana State University”. RQ. Volume 35. Number 1. (Fall
1995): Pages 90 – 100.
Page: 192 of 237
Thomas Virgona Doctoral Dissertation: Defense
Post, Gerald; Keim Albert Kagan, and T. Robert. 1998. “A comparative evaluation of CASE
Tools”. The Journal of Systems and Software. Volume 44. (1998): Pages 87-96.
Postman, N. R. (1992). Technopoly: The surrender of culture to technology. New York:
Vintage / Random House.
Powell, Ronald R. 1997. Data Collection Techniques in “Basic Research Methods for
Librarians”. Third Edition. Greenwich, Connecticut and London, England: Ablex
Publishing Corporation.
Preece, Jennifer; Yvonne Rogers and Helen Sharp. 2002. Interaction Design - Beyond
human-computer interaction. New York: John Wiley and Sons.
Price, Tom. 2004. “Post September 11 security measures hit science hard “. Optics and
Photonics News. Volume 15. Number 11. (November 2004).
Quarantelli, E.L. (1984), Organizational Behavior in Disasters and Implications for Disaster
Planning, Monograph Series, Vol. Vol. 1 No.2, pp.1-31.
Quaantelli, E.L. 2005. Catastrophes are Different from Disasters: Some Implications for
Crisis Planning and Managing Drawn from Katrina. Available at:
http://www.unitedsikhs.org/katrina/catastrophes_are_different_from_disasters.pdf.
Last Visit: 01/02/2007.
Quaantelli, E.L. (Editor). 1998. What is a Disaster? Perspectives on the Question.
Routledge. New York, New York. Notes: Article(s) contributed by: Claude Gilbert,
Wolf R. Dombrowsky, Gary A. Kreps, Boris N. Porfiriev, Kenneth Hewitt, Russell R.
Dynes, Robert A. Stallings, Uriel Rosenthal, Steve Kroll-Smith, Valerie J. Gunter,
Anthony Oliver-Smith, Ronald W. Perry, Russell R. Dynes, Anthony Oliver-Smith.
Page: 193 of 237
Thomas Virgona Doctoral Dissertation: Defense
Ravichandran T. and Arun Rai. 2000. “Quality Management in Systems Development: AN
organizational System Perspective”. MIS Quarterly. Volume 24. Number 3.
(September 2000): Pages 381-415.
Redsell S.A.; Cheater F.M. 2001. The Data Protection Act (1998): implications for health
researchers. Journal of Advanced Nursing. Volume 35, Number 4, August 2001, pp.
508-513(6). Abstract available at:
http://www.ingentaconnect.com/content/bsc/jan/2001/00000035/00000004/art01867.
Last Visit: 01/03/2007.
Redwine, S. T. 2002. “Virginia's Governmental Information Security: A Guide for
Commonwealth Executives after September 11, 2001 (Technical rept)”. Report
Number: CISC-TR-2002-002, Jun 2002, 20p x Dec 2002 to x (Dec 2002).
Richmond, Barry. 2000. The "Thinking" in Systems Thinking: Seven Essential Skills”. N.p.:
Pegasus Communications. (January 5, 2000).
Ritzenthaler, Mary Lynn. 2006. NARA Preservation Program Oversees Records Salvage in
Orleans Parish Post Hurricane Katrina. May 9th, 2006. Available:
http://www.archives.gov/records-mgmt/presentations/raco06-ritzenthaler.pdf. Last
visit: 12/30/2006.
Rogers, Everett M. 2003. Diffusion of Innovations, 5th Edition. NewYork, New York: Free
Press, A Division of Simon and Shuster, Inc.
Rogers, Y. 1994. Coordinating Computer-Mediated Work. Computer Supported Cooperative.
Journal of Information Technology. Volume 9. Number 2. Pages 119-128.
Rogers, Yvonne. 1997. “A Brief Introduction to Distributed Cognition.” Interact Lab, School
of Cognitive and Computing Sciences, University of Sussex, BRIGHTON, BN1 9QH,
UK. August 1997. Available at:
Page: 194 of 237
Thomas Virgona Doctoral Dissertation: Defense
http://www.cogs.susx.ac.uk/users/yvonner/papers/dcog-brief-intro.pdf. Last visit:
March 12th, 2006.
Rogers, Yvonne. 2004. “New theoretical approaches for HCI”. Annual Review of
Information Science and Technology, No. 38. 2004. Available at:
http://www.informatics.sussex.ac.uk/interact/papers/pdfs/ARIST-Rogers.pdf
Roukis, George S. 2004. The British East India Company 1600-1858: A model of transition
management for the modern global corporation. Journal of Management Development.
Year: Dec 2004 Volume: 23 Issue: 10 Page: 938 – 948.
Rowley, Jennifer and John Farrow. 2000. Organizing Knowledge: An Introduction to
Managing Access to Information. Third Edition. Hampshire, England and
Burlington, Vermont: Gower Publishing Limited. (2000).
Rush, M.A. and L. Paglia. 2002. “Balancing privacy, public safety, and network security
concerns after September 11”. Information Systems Security. Volume 11. Number 2.
Auerbach Publications. (May-June 2002): Pages: 15-24.
Russett, Bruce M. 1993. Grasping the democratic peace: principles for a post-Cold War
world. Princeton, N.J. Princeton University Press.
Saracevic, Tefko. 1999. Information Science. Journal of the American Society for
Information Science. Volume 50. Number 12. Pages 1051–1063, 1999.
Savage, Mick. 2002. “Business continuity planning.” Work Study. Volume 2002. Volume
51. Issue 5. Pages 254-261.
Available at: http://www.emeraldinsight.com/10.1108/00438020210437277
Schain, Martin. 2001. The Marshal Plan: Fifty Years later. Palgrave:New York, New York.
Scholl, Frederick. 2004. “New FDIC Ruling Highlights Data Disposal Issues”. Np: Monarch
Information Networks. (Q4/2004).
Page: 195 of 237
Thomas Virgona Doctoral Dissertation: Defense
Schutt, Russell K. 1999. Investigating the Social World. Thousand Oaks, California: Pine
Forge Press.
Seeger, Matthew W. et al 2005. “Post-crisis discourse and organizational change, failure and
renewal”. Journal of Organizational Change Management. Volume 18. Number 1.
(2005): Pages 78-95.
*Note: This article focuses on Cantor Fitzgerald and the events of September 11,
2001. The researcher lost two relatives that day that worked for the firm.
Seifert, J.W. 2002. “The effects of September 11, 2001, terrorist attacks on public and private
information infrastructures: a preliminary assessment of lessons learned”. Government
Information Quarterly. Volume 19. Number 3. (2002): Pages: 225-242.
Sellen, Abigail J. and Richard Harper. 2002. The Myth of the Paperless Office. Cambridge,
MA: MIT Press. (2002).
Sherry, Michael. 1977. Preparing For the Next War: American Plans for Postwar Defense,
1941-45. New Haven: Yale University Press.
Sikich. Geary W. 2003. Integrated Business Continuity: Maintaining Resilience in Uncertain
Times. Tulsa, Oklahoma:Penwell Corporation. 2003.
Sirkin, R. Mark. 1995. Statistics for the Social Sciences. Thousand Oaks, London, New
Delhi: Sage Publications -International Educational and Professional Publisher.
(1995).
Sproull, Natalie. 2002. Handbook of Research Methods. Lanham, Maryland: Scarecrow
Press, Inc.
Smiraglia, Richard P. (Editor). 2002. Works as Entities for Information Retrieval. New
York, London, Oxford: The Haworth International Press.
Page: 196 of 237
Thomas Virgona Doctoral Dissertation: Defense
*Note: Works as Entities for Information Retrieval has been co-published simultaneously as
Cataloging & Classification Quarterly, Volume 33, Numbers 3 / 4. 2002.
Srikantaiah, T. Kanti and Michael E.D. Koenig, eds. 2000. Knowledge Management for the
Information Professional. Medford, NJ: American Society for Information Science.
Strauss, Anselm Leonard. 1987. Qualitative Analysis for Social Scientists. Cambridge, New
York, Melbourne: Cambridge University Press.
Strauss, Anselm Leonard. Juliet M. Corbin. 1998. Basics of Qualitative Research:
Techniques and Procedures for Developing Grounded Theory. Thousand Oaks,
London, New Delphi: Sage Publications.
Tennant, Roy and Sarah Michalak. 2004. “Bulletin of the American Society for Information
Science and Technology; Special Section; Portals in Libraries”. Volume 31. Number
1. (October / November 2004).
The National Commission on Terrorist Attacks Upon the United States. 2004. The Complete
Investigation; The September 11, 2001 Report. New York, New York: St. Martins
Press.
Uden, Lorna and Neil Willis. [?]. “Designing User Interfaces using Activity Theory”. School
of Computing, Staffordshire University, Beaconside, Stafford, ST18 0AD, UK. N.d.:
[?]. *Note; Hardcopy available upon request. No date on class handout.
University of Chicago Press. (1993). The Chicago Manual of Style. 14th edition. The
University of Chicago Press, Chicago 60637.
*Notes: The following sections directly relate to the style composition of this
dissertation proposal.
15.15 – Bibliography is alphabetically listed.
15.69 – The reference list will be in Author-Date style.
Page: 197 of 237
Thomas Virgona Doctoral Dissertation: Defense
15.77 - Book information includes Authors full names, title, editor, edition, volume,
series title, facts of publication (city, publisher and date) and page numbers.
15.85 – Two authors use ‘and” in the bibliography.
15.86 – More than three authors, use et al.
15.90 – If the authorship of a work is not known, the name is given in brackets [].
15.159 – When the place of publication cannot be determined, n.p. (no place) may be
substituted for the missing information.
15.175 – When there is no ascertainable date of publication, n.d. no date, a question
mark is given in brackets.
15.204 – Periodical general requirements; Authors name, title of article, title of
periodical, issue information (volume, issue number, date) and page reference.
15.215 – When in reference to the article as a whole, inclusive pages are omitted.
16.3 – Author-Date Text Citations, specifically the “Basic Form”, will be used for this
dissertation proposal.
Verenikina, Irina and Edward Gould. 1998. “Tool Based Psychology as a Philosophy of
Technology”. Available at http://www.bauer.uh.edu/parks/fis/gould.htm. Last visit
February 21st, 2001. (November 4, 1998).
Volesko, M.M. 2002. “It wasn't raining when Noah built the ark: disaster preparedness for
hospitals and medical librarians post September 11”. Internet Reference Services
Quarterly. Volume 6. Number 3-4. (2002): Pages: 99-131.
Velluci, Sherry L. 1998. “Metadata”. Annual Review of Information and Technology
(ARIST). Volume 33. (1998).
*Note: Published for the American Society for Information Science (ASIS) by Information
Today, Inc. Medford, New Jersey.
Page: 198 of 237
Thomas Virgona Doctoral Dissertation: Defense
Wall Street Technology. 2002. “Talking BCP With the Federal Reserve Bank of N.Y.”.
Available at:
http://www.financetech.com/featured/showArticle.jhtml?articleID=14702676. Last
visit: 02/22/2006. (June 18, 2002).
Ward, Patricia Layzell. (2001). “Management and the management of information and library
services 2000. Library Management. Volume 22. Issue 3. Available at:
http://www.emeraldinsight.com/Insight/ViewContentServlet?Filename=Published/Em
eraldFullTextArticle/Articles/0150240303.html
Warner, E.S., Murray, A.D., & Palmour, V.E. (1973). Information needs of urban residents .
Baltimore, MD: Regional Planning Council. (Final Report Contract No. OEC-0-71-
4555)
Webster, Frank. 1995. “Theories of the Information Society”. 1995. Routledge. London.
Weibel, Stuart L. 1997. in Velluci, Sherry L. “Metadata”. Annual Review of Information and
Technology (ARIST). Volume 33. (1998).
*Note: Published for the American Society for Information Science (ASIS) by Information
Today, Inc. Medford, New Jersey.
Wechsler, Henry, et al. 2000. ”College Binge Drinking in the 1990’s: A Continuing Problem.
Results of the Harvard School of Public Health 1999 College Alcohol Study”.
College Health. Volume 48. (March 2000).
Whisenant, Warren A. 2003. “Using biometrics for sport venue management in a post 9-11
era”. Facilities. Volume: 21. Issue: 5/6. (May 2003): Pages: 134 – 141.
White, E.B. (Elwyn Brooks). 1949. Here Is New York. U.S. author, editor. Indianapolis,
IN:Holiday.
Page: 199 of 237
Thomas Virgona Doctoral Dissertation: Defense
Wikipedia. 2006. “Wikipedia, the free encyclopedia”. Available at: http://wikipedia.org/.
Last visit: 04/21/2006.
Wingfield, Joy. 2002. The Data Protection Act 1998. The Pharmaceutical Journal. Volume
265. Number 7106. p131. Available at:
http://www.pharmj.com/Editorial/20000722/articles/dataprotection.html. Last visit:
01/03/2007.
Winner, Langdon. 1986. The Whale and the Reactor: A Search for Limits in an Age of High
Technology. Univ. of Chicago Press.
Winner, Langdon. 1986. The Whale and the Reactor: A Search for Limits in an Age of High
Technology. N.p.: University of Chicago Press.
Woolnough, Paul. 2002. “Future of tall buildings after 11 September”. Civil Engineers
Australia. Volume 74. Number 3. (March 2002): Page 34.
Xing, Cong-cong and Boumediene Belkhouche. 2003. On Pseudo Object-Oriented
Programming Considered. Communications of the ACM. Volume 46. Issue 10.
(October 2003): Pages115-117.
Yiin, Lih-Ming; et al. 2004. “Comparisons of the Dust/Smoke Particulate that Settled Inside
the Surrounding Buildings and Outside on the Streets of Southern New York City
after the Collapse of the World Trade Center, September 11, 2001”. Journal of the Air
and Waste Management Association. Volume 54. Number 5. (May 2004): Pages 515-
528.
Yourdon, Edward and Larry L. Constantine. 1979. Structured Design: Fundamentals of a
Discipline of Computer Program and Systems Design. Engelwood Cliffs, NJ:
Yourdon Press.
Page: 200 of 237
Thomas Virgona Doctoral Dissertation: Defense
Yourdon, Edward. 1989. Modern Structured Analysis. Engelwood Cliffs, NJ: Yourdon
Press.
Zilper, Nadia. 2002. USA-Russia library materials exchanges: past, present and future.
Collection Building. Volume 21. Number 4. 2002.
Page: 201 of 237
Thomas Virgona Doctoral Dissertation: Defense
8 Glossary
Term Definition
BCP Business Continuity Planning
Community A community can be defined as some geographically delineated unit
within a larger society (Berg 2001).
Demographic data Background information, such as age, sex, marital status, religion,
ethnic background and so on.
Design Factors The architecture of the system is in some way deficient (Demarest
1997).
DR Disaster Recovery
Exabyte 1 billion gigabytes
External Validity Synonymous with Generalizability. Appropriate sampling procedure
have been chosen and carried out (Morgan 1995).
Grounded theory Build up inductively a systematic theory that is “grounded” in, or based
on, the observations. The observations are summarized into conceptual
categories, which are tested directly in the research setting with more
observations. Over time, the conceptual categories are refined and
linked, a theory evolves (Shutt 1999).
Page: 202 of 237
Thomas Virgona Doctoral Dissertation: Defense
Term Definition
Human Factors Human Factors is the scientific discipline concerned with the
fundamental understanding of interactions among human and other
elements of a system, and the application of appropriate methods and
theory to improve human well-being and overall system performance
(Karwowski, 2001). In this particular study, the terminology related to
the interaction of Wall Street technology resources interacting on
September 11, 2001 maintaining and restoring system usability. An
alternative term may be Sociotechnical Factors, where people and
politics aren’t considered explicitly within the project scope (Demarest
1997).
Induction Going from specific to general. Deductive: Going from general to
specific (Sirkin 1995).
Internal Validity Questions answered truthfully, respondent recall accurately, results a
valid representation of the collected data, is this a true picture (Morgan
1995)?
ISSS International Society for the Systems Sciences
IT Information Technology
Operationalization The process of specifying the operations that will indicate the value of
cases on a variable (Schutt 1999).
Population All members of a defined category of elements such as people, events
or objects (Sproull 2002).
Page: 203 of 237
Thomas Virgona Doctoral Dissertation: Defense
Term Definition
Portal A portal is a customized learning and transaction Web environment,
designed purposely to enable an individual end-user to ‘personalize’ the
content and look of the website for his/her individual preference. It is a
service environment and should be designed from the customer
perspective (Lakos 2004).
Reliability Use of stable, consistent and dependable methods, instruments,
questions and exercises. Could the results be repeated with full
confidence that similar results would be produced (Morgan 1995)?
Scientific Method State the problem, formulate the hypothesis, design the experiment or
survey, make observations, interpret the data, draw conclusions.
Spiral process 1. Identify the research questions
2. Gather the information to answer the questions
3. Analyze and interpret the information
4. Share the results with the participants (Berg 2001).
Technical Factors The technology (computer software, hardware and communication)
components selected for integration into an application.
Triangulation The use of multiple methods to study one research question (Shutt
1999).
Verification Conclusions must be verified (not merely the wishful thinking of the
researcher) and verification that all procedures used to arrive at the
eventual conclusions have been clearly articulated (Berg 2001).
Page: 204 of 237
Thomas Virgona Doctoral Dissertation: Defense
9 Appendices
9.1 Appendix A: SDLC deliverables and approvers
Initiation Deliverables: Deliverable Approver Business Case Project Sponsor Technology Area/Department Manager Feasibility Study Business Sponsor(s) Technology Management Level 0 Estimate Project Sponsor(s)
Definition Deliverables: Deliverable Approver Business Requirements Document Business Sponsor(s) (BRD) Stakeholder(s) Technology Management Level 1 Estimate (and Project Business Sponsor(s) Plan) Technology Management
Functional Requirements Business Sponsors (for business- related functionality Document (FRD) only such as screen design and report layout) Stakeholders Technology Management Function Point Count (for Eligible Function Point Coordinator Projects) Level 2 Estimate (and Project Business Sponsor(s) Plan) Technology Management Software Project Plan (SPP), as Business Sponsor(s) appropriate Stakeholders Technology Management, as appropriate Information Security Process GISO/BISO Standard Deliverables Strategy
Page: 205 of 237
Thomas Virgona Doctoral Dissertation: Defense
Technical System Design Deliverables: Deliverable Approver Technical System Design (TSD) Technology Management Document
Development Test Plan Technology Management
User Acceptance Test Plan Technology Management Business Sponsor (Client Manager) Stakeholders Relevant Departments: data center, network, business operations, etc. Contingency Approach/Disaster COB Coordinator Recovery Business Head Project Manager Peer Review – Technical Design, Architect as appropriate Level 3 Estimate (and updated Business Sponsor(s) project plan) Technology Management
Construction Deliverables: Deliverable Approver Development Testing Technology Management Release/Backout Plan Business Sponsors Stakeholders Technology Management User Acceptance Test Plan Project Sponsor(s) Stakeholders
Validation Deliverables: Deliverable Approver User Acceptance Test Project Sponsor(s) Stakeholders Parallel or Production Assurance Project Sponsor(s) Test (PAT), as appropriate Stakeholder Technology Management Information Security Review GISO/BISO Process (ISRP) Continuity of Business (COB) Senior Client Manager Contingency Test, as appropriate Senior Technology Management
Implementation Deliverables: Deliverable Approver
Page: 206 of 237
Thomas Virgona Doctoral Dissertation: Defense
Deliverable Approver User Sign-off / Implementation Business Sponsor(s) approval Stakeholders Technology Management
Change Request As required by Change Management Policy
Post-Project Review Deliverables: Deliverable Approver Post Project Review Document Business Sponsor(s) Stakeholders Technology Management
Page: 207 of 237
Thomas Virgona Doctoral Dissertation: Defense
9.2 Appendix B: Types of research design
(Sproull 2002)
Requires Minimum Manipulate Impose Type of Subject randomly two independent Measure Amount Research assigned to comparison variable On of design treatment groups (Experiment) Subjects Control Example True Possesses Characteristic Possesses Possesses Possesses Maximum 100 clerical workers are experimental Characteristic Characteristic Characteristic assigned to either the control group (n-50) or the experimental group (n=50). Pretests on attitudes toward using computers are given to all Ss. Ss in the experimental group are trained to use computers and use them for four weeks. At the end of the four-week period the attitude posttests are given to both the experimental and control groups Ss and a comparison of attitude between the two groups is made using pretest and posttest information. Quasi- May or may not possesses May or may not Possesses Possesses Less A researcher learns that the experimental characteristic possesses characteristic Characteristic XYZ corporation will install a characteristic Management Information System in 6 months and designs a study to measure the flow of information before and after the MIS is installed. A control group from the STR Corporation, which does not have MIS, will be used as a comparison group. Non- Does not possesses Does not Does not possesses Possesses Very Little Measuring the attitudes experimental characteristic possesses characteristic Characteristic towards the IRS of CPA’s and characteristic non-CPA managers to assess the relationship of attitude and job position. Historical Does not possesses Does not Does not possesses Does not None The relationship of last six characteristic possesses characteristic possesses month’s absenteeism and characteristic characteristic seniority of employees. Anytime the researcher has data which already exist, they are called historical data.
Page: 208 of 237
Thomas Virgona Doctoral Dissertation: Defense
9.3 Appendix C: Informed Consent Form LONG ISLAND UNIVERSITY
C.W. Post Campus
Palmer School of Library and Information Science
Informed Consent Form for Human Research Subjects
You are being asked to volunteer in a research study called September 11, 2001 - A Study of the Human Aspects in the Changes to Information Technology Disaster Recovery Efforts For Wall Street Financial Services Firms, conducted by Thomas James Virgona, Doctoral Student, Palmer School of Library and Information Science at Long Island University / C.W. Post Campus. The purpose of the research is to examine the impact of the September 11, 2001 events on Information Systems Technology. Additionally you will be asked to take part in an individual or group interview process to develop an understanding of information technology changes precipitated by the events of September 11, 2001. The investigator will provide explanation of and instructions for these tasks. These tasks will require approximately 2 hours. The day will be scheduled at your convenience at my home. There are no foreseeable risks or discomforts associated with your participation in this study. While there is no direct benefit to you for participation in the study, it is reasonable to expect that the results may provide information of value for the field of information systems technology. You will receive a paid lunch for your efforts.
Results of this study will be confidential and limited to academic (classroom presentation, presentation of results at scientific meetings and conferences) and research purposes. Your name will not be included in any forms, questionnaires, etc. This consent form is the only document identifying you as a participant in this study; it will be stored securely in investigator's home available only to the investigator. Additionally, you may drop out of the study at any time without penalty. Data collected may be destroyed at the end of a legally prescribed period or stored for further research. Results will be reported only in the aggregate. If you are interested in seeing them, you may send the principal investigator your name and address on a postcard.
If you have questions about the research, you may contact the investigator, Thomas James Virgona at (516) 599- 2890 or Dr. Charles Hildreth at (516) 299-2178. If you have questions concerning your rights as a subject, you may contact the Executive Secretary of the Institutional Review Board, Ms. Kathryn Rockett at (516) 299-2523.
Your signature indicates that you have fully read the above text (or had it read to you) and have had the opportunity to ask questions about the purposes and procedures of this study. Your signature acknowledges receipt of a copy of the consent form as well as your willingness to participate.
Typed/Printed Name of Participant Signature Date
Thomas James Virgona
Typed/Printed Name of Investigator Signature Date
Page: 209 of 237
Thomas Virgona Doctoral Dissertation: Defense
9.4 Appendix D: Institutional Review Board Approval
Page: 210 of 237
Thomas Virgona Doctoral Dissertation: Defense
Page: 211 of 237
Thomas Virgona Doctoral Dissertation: Defense
9.5 Appendix E: Interview Script
The initial contact script will be as follows:
“You are being asked to volunteer in a research study called September 11, 2001 - A
Study of the Human Aspects of Disaster Recovery Efforts For Wall Street Financial
Services Firms. The purpose of the research is to examine the impact of the
September 11, 2001 events on Information Systems Technology. You have been
asked to take part in an interview process to develop an understanding of information
technology changes precipitated by the events of September 11, 2001. This task will
require approximately 2 hours. The day will be scheduled at your convenience at my
home. There are no foreseeable risks or discomforts associated with your participation
in this study. “
Focus Group: Initial question:
No set script will be prepared. There was a question to start the conversations: On
September 11, 2001 a disaster was declared and many technology systems went into
disaster recovery mode. Who in your company made that declaration to cut over to
DR? How were you notified?
Interview: Initial set of questions and sub-questions:
Were systems on Wall Street negatively impacted by the events of September 11, 2001?
• Was your specific job function impacted by the terrorist attacks? If so, how?
• Were you able to recover the applications for your area? Page: 212 of 237
Thomas Virgona Doctoral Dissertation: Defense
• What were the major issues encountered during the days immediately following
September 11?
What happened to the systems that day and how did information systems technologists react?
• What was your initial reaction?
• Was there a unique set of circumstances that made your job even more difficult
during that period?
• Were you re-located to a backup site?
• Did you receive conflicting messages from management?
• Did you rely more on your personal relationships, documentation or company
direction during the days following September 11.
What changes to the SDLC (specifically humans role in Disaster recovery design planning have been implemented) since 9/11?
• Was a disaster declared in your firm?
• How were you kept abreast of current issues?
• Were the disaster recovery documents of value to you?
Lessons learned from September 11, 2001.
• What would you have done differently that day?
• What should your specific company do better during a disaster?
• What disaster recovery process(es) worked well that day?
Page: 213 of 237
Thomas Virgona Doctoral Dissertation: Defense
9.6 Appendix F: Defense Acceptance Form
Palmer School of Library and Information Science DOCTOR OF PHILOSOPHY IN INFORMATION STUDIES ______
DEFENSE ACCEPTANCE FORM ______
STUDENT NAME: Thomas James Virgona SS#: 115-60-1549
September 11, 2001 - A Study of the Human Aspects of Disaster Recovery Efforts For Wall Street Financial Services Firms
Defense: (Abstract Attached)
We move the acceptance of this dissertation; we certify that it satisfies the requirements for the conferral of the degree of Doctor of Philosophy in Information Studies.
COMMITTEE MEMBERS Name Signature Date
Dr. Charles Hildreth, C.W. Post (Co-Chair)
Dr. Gregory Hunter, C.W. Post (Co-Chair)
Dr. Michael E.D. Koenig, C.W. Post (Director)
Dr. Mary Westermann-Cicio, C.W. Post (dean)
Dr. Constance Knapp, Pace University (External)
Page: 214 of 237
Thomas Virgona Doctoral Dissertation: Defense
Dean: Dr. Mary L. Westermann-Cicio, Dean Pro Tem
Page: 215 of 237
Thomas Virgona Doctoral Dissertation: Defense
9.7 Appendix G: Individual Interview Transcripts
Senior Technology Manager (Gary)
Thomas Virgona: First, tell me a bit about your professional background.
Senior technology manager: I have over 20 years in application development and
management in both London and New York. On September 11, 2001 I was managing
6 global production applications and I have absolutely no college education!!!
Thomas Virgona: Describe the morning of September 11th, 2001.
Senior technology manager: On that morning, I was walking on Wall Street when the first
plane hit the World Trade Center. My first recollection after the crash was being hit in
the face with burning debris and paper. My initial reaction was to worry about the
safety of my staff that may have been in transit and my wife. I ran as fast as I could to
the building as people ran in the opposite direction towards the trade centers. Once
inside his building, I advised my staff not to leave the facility, as another strike might
be pending. My rationale for this fear is from my experiences in London and that IRA
attacks that came in pairs. I also contacted my wife that was commuting from upper
Manhattan and told her to return home.
Thomas Virgona: Once you settled into your office, what happened next?
Senior technology manager: By mid-morning, I was faced with a number of DR problems:
Should the production applications be cut over to the disaster recovery machines?
Should the Wall Street site contingency be invoked? Who was alive? Are we open
for business today? I quickly rejected a decision to use the backup machines for the
applications since the data center sustained no damage and the auxiliary power and
communications were functioning just fine. Our DR plans in place were a joke and
Page: 216 of 237
Thomas Virgona Doctoral Dissertation: Defense
had never been successfully tested and no plan existed for getting back to the
production machines from the DR machines.
Thomas Virgona: What other types of decision did you have to make that day?
Senior technology manager: The second decision involved the Wall Street site since it
became apparent that the immediate area would not be accessible for a long time. I
had to balance safety and enabling the business to perform. I made the obvious move,
stopping all programming and development tasks. Technology staff with better
“people and soft skills” were placed with the business users at the mid-town
Manhattan site to assist with any technical issues that may be encountered.
[3 sections deleted because of detailed information about disaster recovery locations]
Thomas Virgona: What about the rest of the staff?
Senior technology manager: The rest were asked to work from home and provide technical
expertise for the issues they could handle remotely, such as connections with the Bank
of New York. No plans existed for these issues, the brut force approach was used, and
in most cases trail and error. Luckily I have a friend in BoNY, so I was able to get our
jobs through first, since transmissions were prioritized by name of the institution in
alpha order.
[1 section deleted because of detailed information about batch processing]
Thomas Virgona: Any lessons learned from 9/11?
Senior technology manager: People, get the best people available with good people skills,
since that is when they are needed the most!!! You need to leverage existing
relationships.
Thomas Virgona: Was there a unique set of circumstances that made your job even more
difficult during that period?
Page: 217 of 237
Thomas Virgona Doctoral Dissertation: Defense
Senior technology manager: Ooo yeaaa… One huge issue in the days following the attacks
was a Jewish holiday. Half the staff was missing. I had to speak to Jacob (his
manager) as a one-way process, since he could to listen to the incoming phone
message but not return phone calls from the manager were not allowed for religious
reasons. Also, no other mangers were on site, and I became the de facto leader for
many. Very few people were allowed on Wall Street in the days following the attacks,
and I was amazed by the number of people who called asking him to water their office
plants! Communications that day were [expletive delete]. At one point, I called the
London office and asked them to send information by phone or e-mail. Thank god
there was the fitness center facilities for showers and the cafeteria for food. The
facilities were also used by emergency workers, as the lobby was transformed into a
triage unit.
Page: 218 of 237
Thomas Virgona Doctoral Dissertation: Defense
Help Desk manager (Phil)
Thomas Virgona: First, tell me a bit about your professional background.
Help Desk manager: I have over 20 years in development and management on financial
applications on the Street, and 2 kids that will need tuition in the next few years, so I
will be here for a while!!
Thomas Virgona: What were you doing on the morning of September 11th?
Help Desk manager: My main responsible was to provide on-site information technology
support to the fixed income trading (also known as Bonds) floor. Sorry, my main
responsibility was my family!!! Luckily I got in touch with my wife right away. From
a logistical perspective, I was located approximately 5 blocks north of the World
Trade Center. My primary responsibilities were to provide any technical assistance
required by bond traders on the trading floor itself. Tasks ranged from software to
hardware to phone support. Bond trading is very “relationship” based and phones are
as critical as any other technology to a bond trader.
Thomas Virgona: When did you hear about the attacks?
Help Desk manager: The bond trading market was open at the time of the first attack and
the (New York Stock) Exchange was not. The trading floor has about 30 televisions
and the trading floor first heard of the attacks by the television monitors which were
on CNBC. Senior managers [names removed] called the trading floor to say they felt
the rumble of the first plan flying over the building and did anyone else notice
anything – what a stupid question. Between the first and second attack, most of the
traders watched the television monitors. Taller buildings between this location and the
World Trade Center blocked the view. When the second plane hit the second World
Page: 219 of 237
Thomas Virgona Doctoral Dissertation: Defense
Trade Center tower, the head of the Fixed Income business screamed to the traders to
evacuate the building (No speaker system was used).
Thomas Virgona: Was any information provided at this time?
Help Desk manager: None, nothing by e-mail or phone.
Thomas Virgona: What did you do next??
Help Desk manager: Went outside the building – it was chaos. All the buildings were also
evacuated.
[1 section deleted because of detailed information about transportation service for a
specific firm]
Thomas Virgona: Were formal disaster plans were invoked.
Help Desk manager: No.
Thomas Virgona: What was the state of communication?
Help Desk manager: Phones were not functional but Blackberry’s performed well. Of
course, as a cost cutting measure, most programmers don’t have blackberries!!! When
you need them the most, they can’t be contacted. At that time, I sent messages to
colleagues in New Jersey to call his family and notify them that he was not injured. I
was also amazed at the number of people in the street calling 911 after the second
plane hit. From my vantage point, the impact zones were clear to see and the wind
was blowing the debris east towards Brooklyn. I remember being horrified by bodies
falling through the debris. I hope what I heard is true, that people were unconscious as
they descended. The screaming was “incredible” as the first building fell. Going
towards West Street, I ran into [name removed - managing director] who said to go to
the mid-town office. We walk up the West Side Highway, which took about 1 hour.
By the time I made it to the midtown office, nobody knew [expletive deleted].
Page: 220 of 237
Thomas Virgona Doctoral Dissertation: Defense
[2 sections deleted because of detailed information about business day status]
Thomas Virgona: What kind of shape were your applications?
Help Desk manager: [Name removed] The Chief Operations Officer for the firm was at the
midtown office trying to determine status, what was running, who was making
decisions, and so on. I was speaking to senior people but nobody knew anything. As
the day went on, little credible information spilled out. Simply commuting home to
Long Island was difficult as the railroad had shut down operations with no alternative
in place.
[1 section deleted because of detailed information about call trees]
Thomas Virgona: Was September 12th a business day?
Help Desk manager: I worked from home on the 12th with no interaction with business
group he supported, but I was told was told that the firm was invoking their disaster
recovery plans. He could not recall who told him DR was invoked, but it didn’t
matter, I already implemented by own DR plan – stay home. I never used disaster
recovery documentation for the entire September 11, 2001 period.
Thomas Virgona: Was there a unique set of circumstances that made your job even more
difficult during that period?
Help Desk manager: [location deleted] was temporarily unavailable as the lobby was being
used as a morgue, so I spent the rest of the week in New Jersey setting up desktops for
the business to use the following Monday when trading resumed. Traders were totally
off their game and distracted on the first trading day. Business was impossible
difficult to perform as many of the traders were unable to locate people in others firms
they normally contacted to conduct normal business.
Thomas Virgona: Any other items of note or lessons learned:
Page: 221 of 237
Thomas Virgona Doctoral Dissertation: Defense
Help Desk manager: I remember seeing two beams extending upwards for just a few
seconds as the second tower fell. I also remember the EMS staff vomiting and these
people see this everyday. Remember, before September 11, 2001, managing COB
meant that you had one foot out the door. Nobody took the job seriously.
Page: 222 of 237
Thomas Virgona Doctoral Dissertation: Defense
Application Manager (Sandy)
Thomas Virgona: First, tell me a bit about your professional background.
Application Manager: I manage the asset sales systems for [company name removed]. The
applications I support are not “heavy”’ volume systems but the transactions are for
large dollar amounts (averaging over $100,000). The business is driven by major
business acquisitions and not by financial market. The main group of users for this
system live in Delaware performing back office and accounting functions.
Thomas Virgona: Tell me about the morning of September 11.
Application Manager: On the morning of the terrorist attacks, I was at a meeting with
[government agency name removed]. I was notified of the first attack via e-mail. I
immediately walked to my office on Wall Street, it was simply bedlam!! When I got
to the office, I looked for my staff as I called my family. I was having no luck getting
through. I met some of my programmers that were confused – they had no idea if they
should allow the system to start up – will we be doing transactions today? The entire
team went down to Wall Street to get a better view – HUGE mistake!!! After a few
minutes, when we tried return to the building, they were told the facility was closed –
who they hell ordered THAT?????I then called the members of his team who had gone
into the building to retrieve wallets and medications for people who were not allowed
back in the building. For those 2 minutes, phones were working and I was relieved
when I got in touch with my wife, who was frantic at the time. At this point, there was
Page: 223 of 237
Thomas Virgona Doctoral Dissertation: Defense
no corporate direction or communications of any kind. One person was still
unaccounted for, as she usually walked up from the PATH train at about 9 a.m.
Thomas Virgona: Did you do anything in particular to try and find her?
Application Manager: No, from a logistic perspective, we could not get near the Path,
which is, sorry, WAS, below the Trade Center. The streets quickly filled with people,
the lack of credible information was unsettling to everyone. I first attempted to take
the subway to Penn Station to get on the Long Island Railroad, but the E, F and A
trains were not running. As the debris was blowing towards Brooklyn, attempting to
take a subway to Atlantic Avenue for a Long Island Railroad connection did not seam
feasible. My and my team had totally dispersed and I decided to walk to Penn Station.
While walking up Broadway, the first tower collapsed. He was able to get a cell
phone connection to his mother, she was beyond panic. Walking through the city at
that point became extremely difficulty as chaos erupted in the streets in the area of
NYU. After making it to Penn Station, he was notified that the Long Island Railroad
had invoked its “contingency” plan. To this day, I have no idea what that meant, but
essentially, there was no train service. At that point, I walked to a friend’s midtown
office. That company had increased security at the entrance to allow only employees
into the facility, but I was able to get through via intercom so they let me in. The
cafeteria was opened up and food was free, but people mainly looked for water after
walking from downtown. An e-mail was received that technology had set up a
temporary “war room” on the second floor, and all managers were to provide a status
update if they were in the building. I realized I then had to walk to [location removed]
to get a status update. The room had several senior managers and they were clearly
groping for information.
Page: 224 of 237
Thomas Virgona Doctoral Dissertation: Defense
[2 sections deleted because of detailed information about information security]
Thomas Virgona: Where the managers asking for or providing information?
Application Manager: They had nada, they were totally lost. The first question I was asked
was “Is your staff counted for?” I said one resource had not been located. I then was
asked to provide a status for his applications. I told them that his systems were up and
running, and since the systems had interfaces to faxing software and funds transfer
systems, my applications could be used by other business lines for these utilities.
Once I left the room, I wondered why he said that, since I had absolutely no
knowledge of what state the apps ns were in. My biggest worry was that he would be
called back into the room and be asked to provide those services. As the day went on,
I was on the phone with Delaware wondering what would transpire over the next few
days. The existing disaster recovery plans were of absolutely no value, as they mainly
addressed loss of power to a building – now the building were gone!!! I never used
any of the DR docs during the September 11, 2001 timeframe. Delaware Operations
was told that they would need to house additional staff from the City, but it was
unclear how the staff would get there or how long the site would be needed. I went
home that night still worried about the one missing person and how to support the
business the next few days. About 10pm the one missing staff member called his
home. I did not have his home number and needed to call directory service for it.
Thomas Virgona: What happened on September 12?
Application Manager: I was asked to provide one person for Delaware and one for the
midtown office. Delaware needed a technologist with good soft skills and
communication to assist with the staff driving down to Delaware and provide
application support and technology needs. The person needed for the midtown office
Page: 225 of 237
Thomas Virgona Doctoral Dissertation: Defense
needed deep networking skills, as laptops, printers and PC had to be added to the
network and engineering was short on people. Staff was easy to identify, but asking a
person to leave his or her family to work in Delaware for an extended period was
difficult. The staff person was told to stay for a few days and return on the weekend.
I noticed no senior managers were asked to leave THEIR families that week!!!
[2 sections deleted because of detailed information about encryption and business travel]
Thomas Virgona: What was your biggest issue getting the apps back up and running?
Application Manager: The main concern was the concept of a “business day.” Since no
deals were conducted on September 12th, 13th, or 14th, how would this impact the
accounting systems? Would the batch feeds to the general ledger work? Was
accounting out of balance? Nobody had these answers. The batch scheduler
continued to process feeds with headers and trailers but with no data. Another concern
for all application managers was the supply of diesel fuel. Deliveries were not
allowed below Canal Street and the data centers were already running on backup
power supplies. How long before fuel became an issue?
Thomas Virgona: What were the lessons learned?
Application Manager: The corporate books were out of synch, but were adjusted manually.
Emergency shipments of fuel were allowed. Looking back at he plans, they did not
address any of these issues – of course, the disaster was so diabolical, who the hell
ever thought New York would be shut down??? Ironically, I had trouble getting one
person back downtown, since they had left their wallet in the office - exiting the
subway required a corporate ID and the name had to be on a list held by the U. S.
Army. Why we went back so early was a dumb idea since phone lines were still not
operational. Most people still were still dispersed and difficult to locate. And most of
Page: 226 of 237
Thomas Virgona Doctoral Dissertation: Defense
all, the eerie silence of no traffic in the area was nerve wracking. In the spirit of
cooperation, the regulators were very understanding of our official answers when they
wanted to know the status a few weeks later – we simply did not know where certain
documents were.
[2 sections deleted because of detailed information about disaster recovery backups]
Thomas Virgona: Anything else of note or lesson learned?
Application Manager: During the blackout in the Northeast two years later in August 2003,
he recalled using UPS and FedEx drivers as sources of information, since their
communications were more reliable than other sources. And the concept of an
application is wrong for DR, it needs to be a system with all of the interfaces.
Page: 227 of 237
Thomas Virgona Doctoral Dissertation: Defense
Network Engineer (Mike)
Thomas Virgona: First, tell me a bit about your professional background.
Network manager: On 9/11 this person was a computer architect for a [company name
removed]. My role has changed to manager of technology control and risk. He has 30
years experience in the information technology field. Since this is for a college – I got
a GED and graduated from Baruch – it took 10 years!!!
Thomas Virgona: First, tell me a bit about your professional background.
Network manager: Due to a leg injury, I was working from home in September 11. I was
on a management conference call when the first plane hit the World Trade Center. A
person on the call announced what she saw on the call as she had a direct view of the
trade center. My wife and oldest child had just left for work and school, my first
reaction was to get my family back in the house. My daughter was already on her way
to school, but my wife came back into the house and described the loud explosion.
Minutes later the second plane hit. Within just a few minutes, I was contacted by his
manger and told to contact his staff to ensure everyone was accounted for.
[2 sections deleted because of detailed information about a person that died that day]
Network manager: Another staff member was in a state of shock as she witnessed bodies
falling from the buildings. By mid-day, the debris from the attacks was making the air
in Brooklyn difficult to breathe. I picked up my kids up from school and brought them
home. Later that day, I was told that the department’s COB plan had been invoked,
Page: 228 of 237
Thomas Virgona Doctoral Dissertation: Defense
and this essentially meant “go home and stay home.” I assume it meant that, since I
had never seen the plan.
Thomas Virgona: What was your role in the recovery?
Network manager: As an architect, I recalled thinking that none of the DR plans have ever
envisioned a disaster of this magnitude - where communications had stopped, travel
had ceased and there was no concept of making decisions based on an unknown status
(such as, is today a business day?). I realized that the brut force approach would be
used, aligning skilled people against the recovery tasks. I saw people testing
connections and “hoping”’ they would work. When connections did not work, it was
nearly impossible to determine why or the true root cause (Was the connection bad?
Were the systems on the other end up and running? Had the entitlements protocols
locked them out? Were the interfacing systems now on a new IP address for backup
and he was looking at the wrong system?) While “most” of his firms systems were
functioning in three days, they certainly were not operational since most of Wall Street
was still not working.
[1 section deleted because of detailed information about IP address and passwords]
Network manager: I also recalled seeing (Verizon Vice Chairman Larry) Babbio physically
handing out brooms to his workers in a Verizon building, as the debris had
accumulated in stairwells and made climbing the stairs difficult. Now in my role as
manager of risk and control, I now views COB and DR planning in a new light.
Documented plans during a disaster offer a comfort level for a “predictable” crisis, but
the unknown or unpredictable is the real risk. What DR plan accounts for a large
percentage of police officers resigning during a crisis?
Thomas Virgona: Any lessons learned from 9/11?
Page: 229 of 237
Thomas Virgona Doctoral Dissertation: Defense
Network manager: One negative impact of the increased focus on COB testing is the
resulting outages to production systems. Many times, when testing is conducted on
weekends, the production systems used by the business units are not put back in the
ready state for start of business on Mondays. Also, realize that disasters hare
unpredictable, so people will solve the issues. Make sure they are connected. Also,
from now on I go with my gut, if there is a disaster, I bolt the city right away.
Page: 230 of 237
Thomas Virgona Doctoral Dissertation: Defense
Business User (Anabella)
Thomas Virgona: First, tell me a bit about your professional background.
Business User: I am a trading analyst with over 20 years of experience. I have worked in two other Wall Street firms prior to accepting my current position. Currently, I am at NYU going for an MBA.
Thomas Virgona: Tell me about the events of September 11th.
Business User: I was in my office reviewing market data (Moody’s and Bloomberg) in preparation for the market opening. My office overlooked Wall Street and when I noticed debris in the air there were several news reports describing a plane that had crashed into the
World Trade Center. I was in the WTC during the first terrorist attack in 1993, and I knew this event was not an accident. Her first reaction was to check on her family. My husband was safe in his midtown office and she was confident her two children were already in school but I called to verify. My mother was in a nursing home on Long Island and she called to tell her she would be leaving immediately and not to worry (phone service was lost right after the call). I gathered my belongings and went onto Wall Street. Once I was on Wall Street, the second plane hit the WTC and wondered what possessed me to come out into a war zone. I heard the crash and the screaming, but actually seeing the impact was not possible due to the debris and the obstructed view to the WTC.
Thomas Virgona: What happened next?
Business User: By this time it was approaching 9:30, the normal start of the trading day.
The ship on normal day had sailed a few minutes ago. The streets were packed with workers from the New York Stock Exchange: I had heard that the market would not open and employees were told to evacuate the area. Did we really need to be told that? I went back the
Page: 231 of 237
Thomas Virgona Doctoral Dissertation: Defense
office to get my pager and blackberry and was struck by someone reading a disaster recovery plan. I immediately thought is was ridiculous that someone would read a 300 page manual while the immediate area was under attack.
[2 sections deleted because of detailed information about disaster recovery plans]
Thomas Virgona: Did you find the plans useful?
Business User: As I had never participated in a disaster recovery or continuity of business test, I did not know of any meeting places or escape routes. I had a car available in a nearby parking garage and had intended to pick up my husband in mid-town and drive home to Long
Island. The problem was that traffic was now at stand still and I could not contact my husband. I decided to get her car out of the garage and “grind-it-out” into midtown, in the hopes that she would eventually be able to contact my husband while in route. Luckily that plan worked.
Thomas Virgona: From a business perspective, what happened next?
Business User: Once I got home 4 hours later, I was instructed to follow the disaster recovery plan, which I did not have and had never read. I always thought the disaster recovery tests were too “staged”, with three months of preparation and they still failed. In my mind, all I needed was contact names and market data. On the morning of September 12, I realized that even those simple needs would not be met. Although I had the internal and external customers’ names and “normal” contacts, and I had no idea where these people were currently located. Using the corporate network from home was not allowed except for recovery reasons, so obtaining a true business picture was not possible.
Thomas Virgona: What tasks where you able to perform?
Page: 232 of 237
Thomas Virgona Doctoral Dissertation: Defense
Business User: By mid-day, I was able to get a copy of the DR plan on her Blackberry and I forward that to her personal e-mail to it could be printed out. The plan indicated that my department was to work from a DR site in New Jersey, which was not feasible given travel restrictions. My manager e-mailed me that every effort needed to be made between then and
Sunday to determine how the market would react when the NYSE re-opened that Monday.
This proved to be a difficult task without the ability to see how the foreign markets were reacting (except for news accounts) and no communications with my normal business contacts. As the week progressed and I pieced information from a variety of sources, it was established by that Sunday that the market would take a “sizeable” hit on the opening bell. I was told to report to the New Jersey office for the re-opening of the market, as connections and communications on Wall Street were still not fully operational.
[2 sections deleted because of detailed information about disaster recovery backup sites]
Thomas Virgona: Any other items strike you as unique?
Business User: First of all, I was disappointed in the large volume of selling that morning, as
I thought people would be more “patriotic.” Also, a few days later a departmental meeting was held to discuss improvements to the disaster recovery plans. My input was that everyone needed to participate, not just a selected few. Also, the plans themselves should be updated on regularly with current contact information and hard copies should be left at home. I also one colleagues input on the lessons learned. She recalled one manager stated that disaster recovery site, although handicap accessible, did not meet all of the business needs. Some users required specific ergonomic equipment (e.g., keyboards and monitors) for special needs
Page: 233 of 237
Thomas Virgona Doctoral Dissertation: Defense
and one user required voice recognition software. The disaster recovery plans and site did not address these requirements on September 11, 2001.
[2 sections deleted because of detailed information about vendor software]
Thomas Virgona: Any lessons learned?
Business User: Have real DR tests!! A DR test with 6 months notice and cutting over to the backup system is not a disaster scenario, 9/11 was a disaster!
Page: 234 of 237
Thomas Virgona Doctoral Dissertation: Defense
Database Administrator (Jensen)
Thomas Virgona: First, tell me a bit about your professional background.
Database Administrator: I specialized in Oracle, but also supported Sybase and DB2. I have over 10 years experience in the field and was working on Wall Street on September 11,
2001. I have changed firms since then but still work in the area.
Thomas Virgona: Tell me about the events of September 11th.
Database Administrator: I was at work by 7am. My normal “workload” was the care and maintenance of more than thirty production databases. Some of the tasks are: performance monitoring, load balancing, database version upgrades, table configuration, addition of columns and indexes. To monitor the status of the production databases, I use a “Dashboard
Monitor” which at 25 inches was much larger than a standard personal computer screen. The dashboard also displays status visually (Red, Yellow, Green) for the critical performance measurements.
When the first and second plane hit the WTC, there was absolutely no change of status to any of the production databases. I heard of the attacks from a colleague.
Thomas Virgona: What was your first reaction?
Database Administrator: I am divorced with no children, but was worried for my ex-wife’s safety, since she is trader on the floor (of the New York Stock Exchange). She was coming out of the #4 subway when the first plane hit and she immediately went into the stock exchange building to avoid the debris, using the Broad Street entrance. Ironically, security up until that day was much looser than it is today and entering the building only required a photo
ID at the time.
Thomas Virgona: Did your systems encounter any issues?
Page: 235 of 237
Thomas Virgona Doctoral Dissertation: Defense
Database Administrator: By 9:15 the issues had started to trickle in via e-mail. The New
York Stock Exchange announced that trading would be suspended that day. Application managers had started e-mailing asking me to call them to discuss disaster recovery plans. At this point I realized two critical issues:
1) As a group, the database administrators do not “own” disaster recovery or continuity of business plans. The database plans are embedded in the application plans, which makes us a largely dependency for other groups and a successful cut-over to DR/COB.
2) The underlying assumption is that one, or maybe two, applications would fail at a single time and the workload would be manageable. Nobody envisions a DBA supporting 30+ applications moving to the DR/COB environment all at the exact same time. We were in
WAY over our heads!!!! It was similar to Hurricane Katrina, where the DR plans never envisioned a total breakdown of all services.
[3 sections deleted because of detailed information about passwords and encryption]
Database Administrator: Once the first WTC tower collapsed, my dashboard immediately turned from green to “bleeding” red. The root cause was the loss of critical communication lines, which necessitated a move to the contingency databases. My initial reaction was that under normal DR scenarios, moving a database to a new environment can be a pain in the ass.
Moving over 30 databases would a joke.
[1 section deleted because of detailed information about database configuration]
Database Administrator: As a very general statement, having an application point to a contingency database is a very simple procedure and is tested on a regular basis. The issues that were encountered included transactions lost in transit during the cutover, insufficient access rights to a firewall or machine to perform configuration setups, applications software not recognizing the new database, etc.
Page: 236 of 237
Thomas Virgona Doctoral Dissertation: Defense
[1 section deleted because of detailed information about firewall configurations]
[1 section deleted because of detailed information about information security]
Database Administrator: The one phrase I recall was that “COB trumps IS.” The meaning behind that statement is that brut force was needed to get the applications up and running correctly, and the normal procedures simply had to be discarded. That is not to say there were not many arguments with management or colleagues. To perform the tasks required following September 11, 2001, the many processes simply were ignored.
Thomas Virgona: Where there other factors that made this even more challenging?
Database Administrator: Yes, “How do you get all of the databases back onto the production machines and synched up with the application and other system?” As I later found out, it was a manual process performed one at a time.
Thomas Virgona: Any lessons learned?
Database Administrator: I he never left the office; I slept about 12 hours during that week. I guess I made about 200 manual configuration errors during that period due to stress and lack of sleep. To this day, the underlying issue of DR plan ownership is still a problem, as DBAs do not own the DR/COB plans and do not have direct input into the DR documentation. And most of all, who is in charge – where do I find out if I start a system or we are still in a holding pattern?
Page: 237 of 237