T ECHNOLOGY Reliability in the Apollo Program A BalancedApproach Behind the Success

by Yasushi Sato cles and the Apollo spacecraft. In the dated the launch vehicles. Earlier beginning of the 1960s,NASA hadno con¬ missiles and rockets were also complex sistent philosophy on how to achieve high and expensive enough to call for acute Reliability assurance is a central reliability of those systems. Engineers at awareness of the importance of reliability.6 concern in the design and development of NASA headquarters, the Marshall Space The primary means for reliability assur¬ space systems. A minutest source of unreli¬ Flight Center (MSFC), and the Manned ance in those earlier programs was exten¬ ability in a component or a subsystem can Spacecraft Center (MSC) had diverse sive testing, not only at the component and cause the loss of an expensive system. assumptions on this question. Only after a subsystem levels but also at the level of the Reliability is all the more important in longperiod of trial-and-errors and negotia¬ whole vehicle. The test firing of many human spaceflight programs, where human tions did they attain workable approaches. flight models directly demonstrated the lives are at stake. Thus the disastrous fail¬ Some explanation on the word rate of success. Only after the reliability of ures of Space Shuttles Challenger and "reliability" is in order. Sometimes "relia¬ a vehicle was actually proven did it come Columbia have come under intensive bility assurance" and "quality assurance" into use for human spaceflight. Thirty- scrutiny not only from technical but from are understood as mutually exclusive in seven Redstones had been fired before the organizational viewpoints. On the other meaning. For example, an engineer at MSC missile was declared operational and then hand, reliability efforts in space* programs said: "In simple terms, reliability means was used for astronaut Alan Shepard's sub¬ ÿ that underwent no catastrophic failure have the thing is designed so that it will work; orbital flight. More than 100 Atlas mis¬ not attracted much scholarly attention. This quality means that it is built so that it will siles had been launched before the rocket does not mean that those successful pro¬ work. Inother cases, however, reliability carried astronaut John Glenn to an Earth 8 grams achieved reliability with ease. is a more general term subsuming quality orbit. ÿ Reliability assurance was an utmost issue assurance. This article adopts the latter In the beginning of the 1960s, in those programs also. Their reliability usage and uses the word "reliability" to Don R. Ostrander, director of the Office of problems are relatively invisible retrospec¬ mean generally the ability of achieving Launch Vehicle Programs at NASA head¬ tively only because of the lack of conspic¬ expected performance. quarters, still upheld this approach. uous tragedies. ÿ This article does not intend to Ostrander, an Air Force Major General Official histories of the Saturn describe the whole reliability program in temporarily on loan to NASA, had an over¬ launch vehicles and the Apollo spacecraft the Apollo program. Instead, it demon¬ all responsibility for all of NASA's rocket do note the importance of reliability. They strates the fact that divergent philosophies programs. He argued that NASA must also briefly review the techniques used, in the early 1960s interacted with one "create a fleet of standard vehicles with a including the failure mode and effect another until workable approaches minimum number of different designs and analysis, the closed failure reporting and emerged in the mid-1960s. Some advocat¬ configurations." Then, he continued, corrective action scheme, and the system¬ ed extensive use of statistical techniques NASA must "attain a high degree of relia¬ atic implementation of design reviews. based on test data, while others stressed the bility through repetitive use of these basic They also discuss conservative engineering inherent soundness of design. Striking the vehicles, much as the automotive industry practices of NASA and contractor engi¬ proper balance between the two gave rise has achieved reliable cars through the mil¬ neers, such as the use of proven parts and to the high reliability of the Apollo space¬ lions of miles of driving on each of their techniques, the pursuit of simplicity, and craft and the Saturn launch vehicles, which standardized vehicles." Here Ostrander the elaborate deployment of redundancies. in turn enabled the successful completion was talking about launch vehicles ingener¬ These studies, however, see the question of of the program. al but also had in mind those for human reliability in the Apollo program in a large¬ spaceflight programs. ly static manner. They do not tell how the This approach" of actually verify¬ reliability approach at NASA evolved over RELIABILITY OF THE SATURN ing the statistical probability of successful time.ÿ LAUNCH VEHICLES flights no longer worked for satellite and This article describes NASA's launch vehicle projects of the 1960s. They effort to establish effective approaches to The Needfor a New Approach became so expensive that reasonable finan¬ assure reliability of the Saturn launch vehi¬ The problem of reliability assurance pre- cial resources did not allow the building of

QUEST 13:1 2006 22 many flight models. Nor was there long The center's intensive reliability effort As MSFC aimed for a balanced approach enough time for testing all these models, extended to its contractors. Marshall engi¬ integrating statistical/analytical methods especially in the urgent circumstances of neers closely supervised their contractors' with engineers' unremitting efforts, NASA the Cold War. Thus the development of the operation and meticulously pointed out headquarters also came away from the Saturn launch vehicles required NASA sources of the unreliability. In 1962, von purely statistical approach. Golovin, the engineers to formulate a new scheme to Braun once told D. Brainerd Holmes, strongest proponent of the statistical estimate and improve their reliability. Director of the Office of Manned Space approach, left his position in the Office of Flight at NASA headquarters, that the pen¬ Manned Space Flight in late 1961. A less From Divergent Philosophies to an etration of contractors' operation by adamant statistician, Gephart, came to call IntegratedApproach Marshall engineers did "more for reliabili¬ for the integration of the two extreme In the early 1960s, officials in ty than all the statistical studies combined - views of von Braun and Golovin.ÿ He ÿ charge of reliability policy at NASA head¬ in my humble opinion, at least." His still considered it "essentially pointless" to quarters were engineers with background observation was justified when all ten discuss reliability without quantitative in statistics such as Nicholas E. Golovin Saturn Ilaunch vehicles achieved success¬ technique. He did not emphasize statisti¬ and Landis S. Gephart.10 Even those stat¬ ful flights from 1961 to 1965 despite pes¬ cal methods too much, however: isticians were aware that it was impractical simistic predictions by theoretically-ori¬ "Reliability engineering can be viewed as a to directly verify the success rate of launch ented reliability experts.ÿ mating of sound engineering disciplines vehicles through large numbers of test At the same time, however, with analytical techniques." 20 flights under actual operating conditions. Marshall engineers did not ignore analyti¬ NASA headquarters' reliability Instead, they argued for the indirect use of cal methods that engineers both within and philosophy changed even more when statistical techniques: first, components without NASA were refininginthose days. Joseph F. Shea, a senior systems engineer, and subsystems are tested under simulated One of such methods was a technique joined NASA in the beginning of 1962 as environments, such as vacuum, vibration, called the failure mode and effect analysis. Golovin's replacement. With his experi¬ and extreme temperature; second, func¬ It was a method to identify the most likely ence in military missile programs, he had a tional diagrams representing the relation¬ patterns of failures of a particular system practical view on the problem of reliability. ships between these components and sub¬ and estimate the effects of those failures on He considered the statistical demonstration systems are translated into statistical terms. the sound functioning of the system. Then, of reliability impractical not only for the These procedures then make it possible to those patterns of failures were eliminated entire vehicles but also for subsystems due integrate the reliability numbers of compo¬ one by one by either deploying redundancy to limitations in cost and time. Cautioning nents and subsystems and thus to calculate or sacrificing the systems' specifications. A against the tendency of engineers to be ÿ the reliability of the entire system. closely related method was the criticality attracted to the superficial rigor and preci¬ Von Braun and Marshall engi¬ analysis. Engineers assigned criticality sion of numbers,ÿ he argued that "The neers, on the other hand, tended to belittle numbers, which indicated the relative statistical confidence must be replaced NASA Headquarters' statistical approach, degree of criticality of components or sub¬ with 'engineering confidence.'" 22 The Von Braun admitted that statistical reliabil¬ systems for the success of the entire sys¬ key to attain engineering confidence, in his ity analysis was a powerful tool. Yet he tem, to all parts of the system. Then they view, was "the rigorous identification of believed that it was not an independent determined the optimum apportionment of the cause for all failures encountered dur¬ statisticians' group at NASA headquarters reliability requirements to those compo¬ ing all phases of developmental testing." that would guarantee the reliability of nents and subsystems, taking their critical¬ Shea's boss, Mueller, also supported Shea's launch vehicles. What was fundamental to ity into consideration.16 As Marshall engi¬ view, asserting that in the Apollo program reliability was, he argued, "an almost reli¬ neers came to recognize the usefulness of they were attempting to replace "statistical gious vigilance and attention to detail on those analytical methods, they sought to confidence" with "inherent confidence."23 the part of every member of a development incorporate them in their scheme of relia¬ Meanwhile, Golovin, who left team."12 This approach emphasized care¬ bility assurance. NASA inApril 1962, still advocated statis¬ fulness in work throughout the entire Coordinating the reliability tical approaches. He admitted the impor¬ developmental phases, constant and metic¬ efforts of various elements of the center tance of "engineering judgments," which ulous search for errors and defects in hard¬ was crucial, because a change in the relia¬ involved "the weighting of many elements ware through inspections and testing, and bility of one component or subsystem and only some of which can be usefully strict and thorough implementation of cor¬ affected that of others, and also because reduced to quantitative form." At the same rective actions. interfaces of components and subsystems time, however, Golovin saw such judg¬ Other Marshall engineers shared were themselves typical sources of unreli¬ ments as often "relatively subjective" and von Braun's view that reliability assurance ability. The principalplaces for the integra¬ criticized the "escape to 'engineering con¬ was the inherent responsibility of individ¬ tion effort were weekly and special meet¬ fidence.'" Thus he continued to insist on ual engineers. One of his aides believed ings. In those meetings, engineers in the "increased emphasis on quantitative that reliability belonged "in the first class" charge of reliability analysis brought tip methods." He believed that "the route of to the engineer himself and asserted that potential failure patterns, alternative greater reliance on qualitative approaches, "If the engineer designs one piece of hard¬ designs, and trade-off factors. They dis¬ such as those based on, so called, "engi¬ ware, he also has to look into the reliabili¬ cussed such issues with engineers from neering judgement [sic]," avoids rather ty."ÿ From their viewpoint, engineers laboratories and project offices, and sought than solves the problems which must be practicing hands-on work, not statisticians, consensus on the course of actions to be overcome."24 ÿ knew best how reliable the hardware was. taken. By contrast, von Braun's team at

QUEST 13:1 2006 23 MSFC habitually let their engineering sions. By such a rigorous testing process, live for the first time.32 judgment override statistical analysis. An which they called "limit testing" or the This cautious approach was example was their decision regarding the "testing to failure" philosophy, they could abandoned soon after George E. Mueller configuration of the first stage of . tell how robust the hardware was. joined NASA headquarters in September They were contemplating whether to Their conservatism and "testing 1963 as Associate Administrator for include the "engine-out capability," a to failure" philosophy had a remote origin. Manned Space Flight. Mueller proposed redundancy mechanism that would enable Using testing as a primary means for iden¬ the "all-up" concept for , which the rocket to continue its mission even tifying and solving problems was their tra¬ meant launching a complete vehicle with when one of the stage's eight engines mal¬ dition from, the Peenemiinde period. With its all three stages live for its first flight. functioned. In making a decision in the scientific theories in relevant fields sparse Mueller knew that the Air Force had affirmative, they weighed a wide range of and engineering experience still limited, already adopted this approach, for he had factors—a sophisticated mathematical failures in testing taught them more than previously been involved inAir Force mis¬ analysis on the net gain in reliability, the anything else. Failures were often more sile programs for a few years as a contrac¬ effect of the redundancy on requirements instructive than successes; they were "suc¬ tor's executive. Mueller decided that inthe vehicle's guidance scheme and struc¬ cessful failures" in historian Michael J. NASA should apply this approach to the tural strength, the possibility that it might Neufeld's words. Peenemiinders made Apollo program because there was not result in a more favorable abort condition progress by knowing what not to do rather enough time left for the step-by-step flight for a crew, and the probability that a simi¬ than what to do. They carried this testing testing to reach JFK's goal of "the end of lar mechanism might be needed for a later practice into the 1960s. the decade." ÿ booster. Marshall engineers made use of It might seem paradoxical at first Marshall engineers, who had statistical analysis, but their final resort sight that such an entrepreneurial group of taken their step-by-step testing philosophy was their overall engineering judgment. people who pioneered the field of rocketry for granted, reacted to Mueller's proposal harbored conservatism in engineering. But with "shock and incredulity."ÿ inlight of Conservatism vs. the All-Up Concept it was exactly because they hadto buildthe their experience to that day, the all-up test¬ In practicing either quantitative body of knowledge in rocketry from ing for an unprecedented launch vehicle or qualitative approaches to reliability, scratch that they acquired this solid like Saturn V appeared impossible. For testing was the most fundamental activity. approach. Their fundamental method in them, Mueller's idea violated the conven¬ At MSFC, approximately 50 percent of the pursuing the art of rocketry was to extend tional assumptions on how to assure the total man-hours for the development of the and clarify the boundary of their art and reliability of the complex system. Von Saturn launch vehicles were spent for test¬ knowledge step by step through "success¬ Braun, however, promptly decided to fol¬ ing.26 Engineers at the center tested the ful failures" and "limit testing." When they low Mueller's direction. He valued loy¬ hardware thoroughly and rigorously. Not actually built hardware, they depended on alty and order, and had always followed content with conducting routine tests, they what was within the boundary at that time, directions of his boss ever since the often deliberately let potential problems and then added safety margins that they Peenemiinde period. At the same time, he happen and made sure that they would not felt proper from their experience. This was was persuaded by Mueller's argument that lead to catastrophic failures. One famous how they built the high reliability of their the step-by-step approach actually did not example was their testing of the F-l hardware incrementally. increase the overall success rate of the engine, which was used for the first stage Marshall engineers' conservatism Saturn V program.ÿ ÿ of Saturn V. Inthe course of developing the sometimes had to be compromised with the Mueller's bold all-up approach F-l, they struggled for years with the nag¬ tight schedule of the Apollo program, how¬ worked. It turned out that all thirteen ging problem of combustion instability. In ever. In 1963, their cautious, step-by-step Saturn Vs completed their missions suc¬ testing combustion chambers, they inten¬ plans for the flight tests of Saturn V gave cessfully. Von Braun later reminisced that tionally caused instability by placing a way to NASA headquarters' direction to the all-up decision "sounded reckless, but small bomb inside and letting it explode. adopt a crash approach. Marshall's original George Mueller's reasoning was impecca¬ Then they saw if the instability would con¬ plan was to fly the first Saturn V inMarch ble."ÿ Others considered that it was verge and the combustion would return to 1966 with a live first stage and dummy Mueller's gamble. Nobody could prove at 97 normal. second and third stages, then the second that time that it would work; but nobody Marshall engineers were conser¬ vehicle in July with live first and second could prove that it would not. The decision vative in designing and developing hard¬ stages and a dummy third stage, and final¬ was proper because it worked; it would ÿ ware. They used proven parts and compo¬ ly a complete live vehicle if the preceding have been improper ifit had not worked. ÿ nents wherever possible, and set relatively two were successful. * Von Braun's team A little after the decision for the all-up test¬ high safety margins. According to an had always taken this incremental ing, Mueller's staff sponsored a study on industrial executive, they required 35 per¬ approach inthe past. It had an advantage of the reliability of the first all-up launch of cent safety margin in structural design offering them opportunities to cope with Saturn V. The contractor who did the study where the Air Force required 25 percent.28 technical difficulties, which they could not reported the number of 0.497 or 0.682, Their testing practices also reflected their foresee but nonetheless expected to be depending on the availability of the conservatism. They actually let compo¬ there. With Saturn I, they had actually engine-out capability.ÿ These numbers nents and subsystems break down in implemented this step-by-step approach. might have given Mueller reasonable con¬ ground tests by testing them in tougher The first three flight models of the two- fidence. But Marshall engineers must have environmental conditions and for longer staged Saturn Ihad only the first stages regarded them as invalid, still less with periods of time than those in actual mis¬ live. The fourth Saturn Ihad both stages their three-digit significant figures.

QUEST 13:1 2006 24 Aside from the validity of such a and its contractors had now arrived at a From the beginning, engineers at mathematical analysis, what ledMueller to rather clear concept on how to achieve the center knew that attaining perfect relia¬ his judgment for the all-up approach was high reliability. . .It is, in one sentence, the bility was impossible. Although human his logicalreasoning that there was no spe¬ application of sound and knowledgeable lives were at stake, they did not believe cific problem expected with the vehicle. engineering and engineeringjudgment - let that inexorable concern for crew safety On the other hand, Marshall engineers me repeat the word "engineering judg¬ should unduly preclude their developmen¬ believed from experience that such reason¬ ment" - based on long-range experience tal effort from moving forward. Walter C. ing was usually not good enough to antici¬ and supported by all the analytical tools ... Williams, who led the operations segment pate all problems in advance. They had by such as detailed analysis of each compo¬ of the center inthe early 1960s, once talked then gone through such a long period of nent and subsystem, logic diagrams, math¬ of "the trade-off between emphasis on constantly encountering failures that they ematical models, etc., and then most crew survival as such, and the end accom¬ assumed that they had to find out through important, an exhaustive test program, sys¬ plishment of the mission."ÿ While he said step-by-step testing what those unantici¬ tem tests program, and a quality assurance he had much respect for the value of life, pated problems were. Their disagreement programÿ" he considered it necessary to recognize that can thus be seen as one between Mueller's risks would remain. Joseph F. Shea, who confidence in his own reasoning and von joined MSC as manager of the Apollo Braun team's experientialjudgment. Inthis RELIABILITY OF THE APOLLO Spacecraft Program Office in 1963, had a instance, the advocate of reasoning had the SPACECRAFT similar view: decision-making authority, and the embod¬ iment of experience was ready to comply. Combination of Quantitative and The program ... must maintain As it turned out, this interaction of reason¬ Qualitative Techniques the proper balance between the safety of ing and experience resulted in a fortunate As Marshall engineers struggled the crew and the success of the mission. turn of events. to attain proper reliability of the Saturn Although we desire the probability of safe Engineers at MSFC and NASA launch vehicles, those at MSC faced the return of the crew to be approximately one- headquarters thus solved the problem of same problem with the Apollo spacecraft. hundred times greater than the probability reliability inherent in the development of The Mercury program had started before of mission success, the designs cannot be launch vehicles with the blendof statistical 1960, when the center was still called the implemented only to provide safety. In the treatment and vigilant attention, cold rea¬ Space Task Group. In the 1960s, they limit, the safest mission will be one in soning and long experience. A speech by moved on to develop the Gemini space¬ which the abort rules are set up so that the Eberhard Rees, the center's deputy director craft and the Apollo spacecraft. Although vehicle never leaves the pad - obviously who had closely followed the reliability those programs overlapped indevelopmen¬ reductio ad absurdum.ÿ problem of the Saturn launch vehicles in tal period, later programs benefited from the first half of the 1960s, summarizes well the experience of earlier ones. Reliability The relative reliability require¬ how they overcame the problem. Rees approaches at MSC evolved continuously ments for mission success and crew safety declared in 1965 that engineers at MSFC through these programs. would actually change over time.

FIGURE 1: Probabilistic Reliability Program

Prediction-assessment

Apportionments Mathematical Reliability goals Mathematical reliability

Subsystem Fabrication Ground test Flight Spacecraft Preliminary detail program lest status specifications design

Predictions Mathematical reliability model

QUEST 13:1 2006 25 fl

FIGURE 2: Qualitative Reliability Program ÿ

Failure mode and elfect failure mode Acceptance analysis and effect tests analysis

Quality control

Spacecraft Subsystem Ground Pre-flight Flight Operational Preliminary Fabrication test program specifications design detail design checkout test status

Design review Qualification status list

Ground test plans

Reliability end item engineers

Generally the probability of crew safety counterparts at MSFC, they defined the and enable appropriate corrective actions. was set an order of magnitude higher than overall numerical reliability and safety On the other hand, Morris of course appre¬ that of mission success. requirements and apportioned them to each ciated the role of qualitative techniques as Inachieving a high level of relia¬ subsystem and component. They did so not well. Particularly important among them bility, the management of MSC believed to strictly enforce such numerical require¬ were failure mode and effect analysis, that human-oriented approaches were fun¬ ments; they knew that was impractical. closed-loop failure reporting and correc¬ damental. Robert R. Gilruth, the director of Instead, their rational was to let those reli¬ tive action, and preflight checkout. (See MSC, once wrote: "Accomplishing true ability numbers "act as a guide and goal for Fig. 1and Fig. 2.) reliability will require people who will the individual designers."45 Thanks to Engineers at MSC also shared never overlook or ignore, but rather who those numbers, designers of subsystems conservatism with their counterparts at will recognize, the slightest sign of trouble and components could make decisions on MSFC. They used proven components and - people who will freely give the last bit of "the degree of redundancy, derating of simple designs even at the cost of some extra effort that so often spells the differ¬ parts, and other reliability improvement performance. For example, they restricted ence between success and failure."42 measures."46 Here ' "derating" means the use of integrated circuits, which still The center's deputy director, reducing stress on parts to levels below had relatively few years of history then and George M. Low, echoed Gilruth's view. their specified ratings or their proven capa¬ expected developmental problems. Only Low stated that what was needed to elimi¬ bilities and thereby enhancing their relia¬ where the potential gain in reliability and nate the minutest flaws and attain the near- bility. weight was significant enough did they use perfect reliability was "a dedication to get As was the case with Marshall integrated circuits.50 They also tested the job done well, by all people, at all lev¬ engineers, engineers at MSC took the posi¬ hardware in excess of the expected envi¬ els, on every element of Apollo."43 For tion that a combination of quantitative and ronmental conditions to identify failure Gilruth and Low, it was high morale and qualitative techniques would offer the best modes. Through the numerous cycles of uprightness of those involved that consti¬ strategy in assuring reliability of the taking corrective actions and retesting, the tuted the reliability of space systems. In Apollo spacecraft. Owen G. Morris, chief system achieved reliability. order to add a personal motivation in men of the Reliability and Quality Assurance Testing in various phases of inthe plants and shops, the management of Division in the Apollo Spacecraft Program design and development was cardinal for MSC often had astronauts meet with Office at the center, stated: "Although reli¬ reliability. Early in the design cycle, engi¬ ÿ them. ability cannot be rigorously demonstrated neers conducted developmental tests to While the management thus [through quantitative methods], unreliabil¬ obtain data useful for design. When proto¬ |j emphasized the human aspects of reliabili¬ ity can."4' In other words, proper use of type models came up, they conducted qual¬ ty effort, engineers at MSC did not ignore statistical methods would make any unreli¬ ification/certification tests, by which they analytical/statistical techniques. Like their able components and subsystems evident verified the soundness of design and the QUEST 13:1 2006 26 proper functioning of the hardware in all can cope with the unexpected," Low ity of spaceship. Their viewpoints moved environments, such as vacuum, vibration, said.ÿ3 back and forth between the extremes in the and extreme temperatures. Finally, in Thus the role assigned to astro¬ first years, but then stabilized toward the acceptance tests, engineers tested flight nauts had become large by 1963, when mid-1960s. items to make sure that there was no man¬ Shea joined MSC. In addition to the nor¬ ufacturing error. Those tests applied to all mal tasks of maneuvering the spacecraft, CONCLUSION levels of systems, subsystems, and compo¬ monitoring the systems, and communicat¬ nents. Low asserted: "The single most ing with the ground, astronauts were now When the Apollo program start¬ important factor leading to the high degree expected to carry out maintenance of the ed, NASA engineers had no clear prospect of reliability of the Apollo spacecraft was spacecraft during the flight. Inthis scheme, on the method and feasibility of lunar land¬ the tremendous depth and breadth of the the astronauts would carry tools and spare ing, still less with the reliability approach test activity. 1 components with them and replace the that they would employ. It was clear that Redundancy was employed malfunctioning components of the space¬ the conventional approach of statistically wherever practical. All subsystems, except craft as necessary. Thus, as Shea observed, demonstrating the rate of successful mis¬ the structure, heat shield, and certain por¬ "the pendulum [had] swung very far" into sions was impossible. Yet NASA engineers tions of the main propulsion systems, con¬ the other direction on this basic problem in needed to attain a near-perfect reliability in tained redundancies. There was a basic crewed spaceflight.ÿ the human spaceflight program. It took mission rule calling for the abort of a mis¬ By mid-1964, however, the con¬ several years for NASA headquarters, sion when one more failure in a critical cept of onboard maintenance died. MSFC, and MSC to establish workable component would cause loss of the crew. Astronauts were relieved from the task of approaches to assuring reliability of the Therefore, engineers at MSC often used maintenance during flight. This reversal of Saturn launch vehicles and the Apollo triple redundancies, which allowed one out trend resulted from both the operational spacecraft. Engineers at both MSFC and of three elements to fail without requiring and systems engineering viewpoints. First, MSC consistently emphasized the impor¬ abort. Along with conservative engi¬ it became apparent that onboard mainte¬ tance of nonanalytical aspects of reliability neering practices in component selection nance during lunar orbital maneuver was assurance-religious vigilance and attention and intensive testing, extensive and elabo¬ impossible. Christopher C. Kraft, who took to detail, "engineering judgment" and rate use of redundancies helped assure the charge of operational activities at the cen¬ "engineering confidence," and dedication reliability of the Apollo spacecraft. ter from 1963 on, argued that astronauts for work by all those involved. As they simply would not have time to do mainte¬ came to recognize the usefulness of statis¬ The Question of Onboard Maintenance nance during the busy maneuvering phase. tical techniques, they also incorporated A unique problem in the reliabil¬ Grumman, the contractor, agreed, main¬ them into their reliability efforts. ity assurance of crewed spacecraft was taining that onboard maintenance would This article has focused mainly whether astronauts onboard should per¬ end up with degraded reliability.ÿ on the evolution of reliability philosophies form maintenance activities. In the early Following the decision to cancel and approaches in the Apollo program, years of Project Mercury, the pilot was onboard maintenance for the Lunar while discussing the actual practices-test¬ hardly expected to maneuver, let alone Module, the concept was dropped for the ing, use of redundancy, failure reporting- conduct maintenance on, the spacecraft. Command and Service Module also. After only in a sketchy manner. More research in Systems were designed to function auto¬ weighing such factors as the weight this area would lead to a fuller understand¬ matically, as the capabilities of humans increase incurred by tools and spare com¬ ing of the reliability effort in the Apollo under alien conditions of weightlessness ponents, the decrease inreliability of spare program. It is certain that studying past and radiation, were unknown. One could electronic components due to humidity of failures and troubles is vital in reflecting expect that astronauts would have physio¬ the cabin, and the time-criticality of the on the reliability of space systems. But it is logical or psychological breakdowns and expected failures,' Shea in April 1964 con¬ also important to leam from the experience lose capacity for maneuver. cluded that NASA no longer favored of engineers who overcame difficult relia¬ However, once the experience of onboard maintenance. Instead, he took the bility problems in successful programs. Mercury revealed that astronauts could approach of preparing redundant, black- function well in outer space, the role of boxed components built into the system so astronauts was greatly augmented. Many that astronauts could switch to them in case ABOUT THE AUTHOR engineers at MSC, who had previously of the malfunction of main components.56 worked on aircraft research at NACA, Thus NASA corrected the pendulum that Yasushi Sato is a postdoctoral fellow at the were eager to give astronauts large room had gone too far and established a balanced Research Center for Advanced Science and for maneuver. Those NACA veterans position. Technology of the University of Tokyo. respected pilots and had propensity' to MSC approach to reliability He received a PhD in history and sociolo¬ emphasize human role in assuring reliabil¬ assurance emphasized the proper combina¬ gy of science from the University of ity. George M. Low, for example, asserted tion of human and analytical aspects of the' Pennsylvania in February 2005. His pri¬ in 1961 that "the reliability of any system problem. In this sense, engineers at the mary interest is the history of postwar is greatly enhanced by the integration of center shared the basic philosophy with American technology. He also does maninto the system." Low pointed out that MSFC engineers. They faced a problem research on the history of Japanese science many flights of the X-15 rocket airplane unique to crewed spacecraft, however; they and technology policy. would have failed without a pilot correct¬ had to define the extent to which astronauts ing malfunctions of the system. "Only man would be involved in assuring the reliabil¬

QUEST 13:1 2006 27 ENDNOTES Landis S. Gephart and William Wolman, "A 23 George E. Mueller, address before the 1966 Probabilistic Model for Reliability Estimation for Annual Symposium on Reliability,.San Francisco, 1 Diane Vaughan, The Challenger Launch Space System Analysis," in Bulletin de I'Institut California, January 26, 1966,p. 13, NASA/HO. (Paris, Decision:Risky Technology, Culture, andDeviance International de Statistique, 33e Session 1961), 73-83. at NASA (Chicago: University of Chicago Press, 24 N. E. Golovin, "Reliability Engineering and 1996). C. F. Larry Heimann, "Understanding the Success in Space Exploration," Western 12 Operations Challenger Disaster: Organizational Structure and , "What Is an Optimum Research Society of America, Annual Meeting, 10th, the Design of Reliable Systems," The American Program?,"Astronautics 5:11 (November 1960): 27. and International Meeting, 1st, Honolulu, Hawaii, PoliticalScience Review 87:2 (June 1993): 421-35. September 14-18, 1964, published in Industrial Michael Cabbage and William Harwood, Comm 11 "Interview with Dr. William Mrazek," inter¬ Quality Control 22 (March 1966): 457-464. Check...: The Final Flight of Shuttle Columbia viewed by Tom Ray on April 6, 1973, p. 3, (New York: Free Press, 2004). Joseph Lorenzo NASA/HO. 2ÿ John R. Levinson, "Reliability Prediction in Hall, "Columbia and Challenger. Organizational Design Decision," presented at the 10th National Failure at NASA," Space Policy 19 (2003): 239-47. 14 Letter from Wernher von Braun to D. Brainerd Symposium on Reliability and Quality Control, Holmes, May 3, 1962, January 7-9, 1964, Washington, D.C., p. 139-41, 2 MSFC/HO. Innoting the success of the Apollo program,how¬ ASC/UAH. ever, we of course should remember the tragedy of 11Eberhard Rees, "Marshall Space Flight Center the fatal fire incident in 1967. 2() Approach in Achieving High Reliability of the Bilstein, Stages to Saturn, 184. Saturn Class Vehicles," 4th Annual Reliability & 3 Roger E. Bilstein, Stages to Saturn: A Maintainability Conference, Statler-Hilton Hotel, 22 Bilstein,Stages to Saturn, 112-6. "Interview with Technological History the Apollo!Saturn Launch of Los Angeles, California, July 28-30, 1965, p. 4, J. R. Thompson, Interviewed by Stephen P. Waring Vehicles (Washington, D.C.: NASA, 1980), esp. NASA/HO. and Andrew J. Dunar, Orbital Services, Huntsville, 281-3. Courtney G. Brooks, James M. Grimwood, June 6, 1994,"p. 7-9, and Loyd S. Swenson, Jr., Chariots Apollo: A MSFC/HO. for m For discussions on these analytical methods, see History of MannedLunar Spacecraft (Washington, for example, "Approach in Achieving High 2® D.C.: NASA, 1979). Andrew J. Dunar and Stephen "Centaur Launch Vehicle Development Reliability for Saturn Class Vehicles with Particular P. Waring, Power to Explore:A History ofMarshall Program," Report of the Committee on Science and Emphasis on Their Navigation, Guidance and Space Flight Center, 1960-1990 (Washington,D.C.: Astronautics, House of Representatives, 87th Control System," by Eberhard Rees to NASA, 1999), esp. 43-46. presented Congress, 2nd Session, July 2, 1962, p. 12. AGARD Guidance & Control Panel Symposium on Reliability inAerospace Vehicle Guidance & Control 4 2® Stages to Saturn, William F. Rector, III, "LEM Lesson: Reliability Systems, Paris, France, March 6, 7 and 8, 1967, Bilstein, 148. Robert Lusser, "The Notorious Unreliability of as Never Before," Grumman Horizons 4:1 (1964): ASC/UAH; "Component Failure Effect on Systems: Complex Equipment," Astronautics 3:2 1958): 22. An Analytical Model," November 1963,prepared by (February 76. Dunar and Waring, Power to Explore, 43. R.L.Parkhill and J. Pauperas Jr. and presented to the 5 The Institute of Electrical and Electronics 4th Annual Seminar on Reliability For Space 30 Engineers (IEEE), for example, defines reliability Vehicles, Los Angeles, California, December 6, Neufeld, The Rocket and the Reich, 64-71. as "the ability of a system or component to perform 1963, Archives and Special Collections, University its required functions under stated conditions for a of Alabama, Huntsville (hereafter ASC/UAH). 31 Mitchell Sharpe, "Saturn and "Ail-Up" Flight- specified period of time." IEEE Standard Testing," MSFC/HO. Computer Dictionary: A Compilation of IEEE '2 John R. Levinson, "Reliability Prediction in Standard Computer Glossaries (New York, NY: Design Decision," presented at the 10th National 32 For a record of frights of the Saturn launch vehi¬ 1990). Symposium on Reliability and Quality Control, cles, see "Appendix C - Satum Flight History" in January 7-9, 1964, Washington, D. C., p. 139, Bilstein, Stages to Saturn, 413-9. 5 For a discussion of reliability assurance of missiles, ASC/UAH. see for example Robert Lusser, "The Notorious 33 Letter from R.B.Young to MitchellR. Sharpe, 11 Unreliability of Complex Equipment," Astronautics Landis S. Gephart, "Reliability - Propulsion & January 1974, MSFC/HO. 3:2 (February 1958): 26-27, 74-78. Launch Vehicles," American Rocket Society, Space Flight Report to the Nation/New York Coliseum, 34 Letter from Wernher von Braun to George E. 2 Bilstein, Stages to Saturn, 15. October 9-15, 1961, NASA/HO. Mueller, November 8, 1963,MSFC/HO. Key mem¬ bers of his team remained skeptical of the all-up ® ÿ GeorgeE. Mueller, address before the 1966Annual Landis S. Gephart, "Reliability and Quality," IRE decision, however. For example, Dieter Grau, Symposium on Reliability, San Francisco, Boston' Section Meeting, 20 April 1961, p. 2, Director of Quality and Reliability Assurance California, January 26, 1966, p. 12, NASA History NASA/HO. Laboratory at MSFC, doubtedthe decision even after Office (hereafter NASA/HO) ten years. Letter from Mr. Grau to Mr. Sharpe, 20 Landis S. Gephart, "System Reliability "Satum History," December 12, 1973, MSFC/HO. 9 Don R. Ostrander, "The U. S. Space Exploration Engineering," Presented at IRE Second Conference Program," address at Western Michigan University, on Reliability of Electronic Equipment, New York 35 "Interview of Dr. Wernher von Braun, Friday, Kalamazoo, Michigan, June 23, 1960, p. 6, City, October 20, 1961, p. 3, NASA/HO. August 28, 1970," Interviewed by Tom Ray, p. 34-5, NASA/HO. See also Don R. Ostrander, "Rocket NASA/HO. Power - Key to Space Supremacy," Astronautics 5:7 21 Joseph F. Shea, address at the 1963 National (July 1960): 22. Space Electronics Symposium, Hotel Fontainebleau, 35 Wernher von Braun, "Saturn the Giant," in Edgar Miami Beach, Florida, October 2, 1963, p. 5-7, M; Cortright (ed.), Apollo Expeditions to the Moon Swenson, Grimwood, and Alexander, This New NASA/HO. Shea even brought up a Roman saying (Washington: NASA, 1975), 50. Ocean, 265-8. Alvin Steinberg, "The Reliability "Qui Numerari incipit, enure incipit," translating it Picture at Marshall Space Flight Center - intwo ways: "He who begins to count, begins to en" 32 "Interview with Dr. William Mrazek," by Tom Philosophy, Staffing and Management," proposed and "Figures don't lie, but liars figure." Ray on April 6, 1973, p. 25-6, NASA/HO. Letter talk to Orlando Section of American Society for from Mr. Grau to Mr. Sharpe, "Satum History," Quality Control, November 15, 1962, Marshall 22 Joseph F. Shea, "Design Requirements for December 12, 1973,MSFC/HO. Space Flight Center History Office (hereafter Spacebome Digital Systems," Computer Design 2:4 MSFC/HO). (July-August 1963): 43. 33 t. T. Jackson, A. D. Tinkelenberg, D. Van Tijn, "Special Technical Report No. 13: The Reliability of 11For an exposition of this method, see for example QUEST 13:1 2006 28 the AU-Up Concept," 15 June 1964,ASC/UAH. Security Industrial Association New York Chapter, Operations in the Apollo Program," 45. October 28, 1964,p. 11, NASA/HO. 39 Rees, "Marshall Space Flight Center Approach in 33 Morris, "Apollo Reliability Analysis," 53-54. Achieving High Reliability of the Saturn Class 43 William F. Rector, HI, "LEM Lesson: Reliability Shea, "The Approach to Apollo," 27. Vehicles," p. 5. as Never Before," Grumman Horizons 4:1 (1964): 22. 33 George M. Low, "Manned Space Flight - When, 49 Walter C. Williams, address before the National Where and Why?" address presented to the National IAS-ARS Joint Meeting, Los Angeles, California, 4*> Robert R. Gilruth, "MSC Viewpoints on Aeronautics Association Convention, Westbury, New June 14, 1961, p. 10, NASA/HO. Reliability and Quality Control," address before the York, September 12, 1961, p. 5-6, NASA/HO. Low American Institute of Architects, Houston, Texas, p. was at still at NASA headquarters then, but would soon move to MSC. 4' Joseph F, Shea, "The Approach to Apollo," 9, NASA/HO. Astronautics andAeronautics 3:4 (April 1965): 26. 49 Owen G. Morris, "Apollo Reliability Analysis," 34 "Interview at Waltham, Massachusetts with Dr. 1965): F. 12, 43 Robert R. Gilruth, "Progress Through Astronautics andAeronautics 3:4 (April 55. Joseph Shea on January 1972, in conjunction Teamwork," Signal 17:10 (June 1963): 34. with J. Thomas Markley and conducted by Ivan D. 48 Ibid. Ertel on Apollo Oral History," p. 4, NASA JSC at UHCL (University of Houston 43 History Collection Low, "Introduction to a Series of Nine Articles Clear Lake). Covering Major Facets of Design, Development, and 49 Ibid. Operations in the Apollo Program," in "What Made 33 a Brooks, Grimwood, and Swenson, Chariots for Apollo Success?" Astronautics andAeronautics 8:3 39 Shea, "The Approach to Apollo," 28. (March 1970): 45. Apollo, 159.

Low, "Introduction to a Series of Nine Articles 3ÿ 44 Ibid., 135. Shea, "TheApproach toApollo," 27-28 George M. Low, "A Report on the Manned CoveringMajor Facets of Design, Development, and Spaceflight Program," presentation at National

"How Do You Go to the Bathroom in $0800?"

Whether it's a title like this for beginningcollectors or limited editions, Boggs SpaceBooks B#GGS has it. Ifwe don't we'll make it our mission to find it. SPACEBOOKS OUR MISSION ISYOUR SATISFACTION B0GGSSPACE.COM 765-649-3211

QUEST 13:1 2006 29