November 3, 2003 Greg R. Overbeck, Senior Vice President, Nuclear Arizona Public Service Company P. O. Box 52034 Phoenix, Arizona 85072-2034 SUBJECT: PALO VERDE NUCLEAR GENERATING STATION - NRC SPECIAL INSPECTION REPORT 05000528/2003011, 05000529/2003011, AND 05000530/2003011 Dear Mr. Overbeck: On September 19, 2003, the US Nuclear Regulatory Commission (NRC) completed a special inspection at your Palo Verde Nuclear Generating Station, Units 1, 2, and 3, facility. The enclosed report documents the inspection findings, which were discussed on September 19, 2003, with Mr. David Mauldin and other members of your staff. The inspection examined the response of all three units to the grid disturbance which occurred on July 28, 2003, the consequent Unit 3 reactor trip, and subsequent equipment problems. The inspection focused on the events leading up to the reactor trip, subsequent plant response, and equipment operation as they relate to safety and compliance with the Commission’s rules and regulations and with the conditions of your license. The inspectors reviewed selected procedures and records and interviewed personnel. On the basis of the results of this inspection, no findings of significance were identified. In accordance with 10 CFR 2.790 of the NRC’s "Rules of Practice," a copy of this letter, its enclosure, and your response (if any) will be made available electronically for public inspection in the NRC Public Document Room or from the Publicly Available Records (PARS) component of NRC’s document system (ADAMS). ADAMS is accessible from the NRC Web site at http://www.nrc.gov/reading-rm/adams.html (the Public Electronic Reading Room). Sincerely, /RA/ Mark A. Satorius, Deputy Director Division of Reactor Projects Arizona Public Service Company -2- Dockets: 50-528 50-529 50-530 Licenses: NPF-41 NPF-51 NPF-74 Enclosure: NRC Inspection Report 05000528/2003011, 05000529/2003011, and 05000530/2003011 w/Attachments: Supplemental Information Charter for the NRC Special Inspection Team at Palo Verde Nuclear cc w/enclosure: Steve Olea Arizona Corporation Commission 1200 W. Washington Street Phoenix, AZ 85007 Douglas K. Porter, Senior Counsel Southern California Edison Company Law Department, Generation Resources P.O. Box 800 Rosemead, CA 91770 Chairman Maricopa County Board of Supervisors 301 W. Jefferson, 10th Floor Phoenix, AZ 85003 Aubrey V. Godwin, Director Arizona Radiation Regulatory Agency 4814 South 40 Street Phoenix, AZ 85040 Craig K. Seaman, Director Regulatory Affairs/Nuclear Assurance Palo Verde Nuclear Generating Station Mail Station 7636 P.O. Box 52034 Phoenix, AZ 85072-2034 Hector R. Puente Vice President, Power Generation El Paso Electric Company 2702 N. Third Street, Suite 3040 Phoenix, AZ 85004 Arizona Public Service Company -3- Terry Bassham, Esq. General Counsel El Paso Electric Company 123 W. Mills El Paso, TX 79901 John W. Schumann Los Angeles Department of Water & Power Southern California Public Power Authority P.O. Box 51111, Room 1255-C Los Angeles, CA 90051-0100 John Taylor Public Service Company of New Mexico 2401 Aztec NE, MS Z110 Albuquerque, NM 87107-4224 Cheryl Adams Southern California Edison Company 5000 Pacific Coast Hwy. Bldg. DIN San Clemente, CA 92672 Robert Henry Salt River Project 6504 East Thomas Road Scottsdale, AZ 85251 Brian Almon Public Utility Commission William B. Travis Building P.O. Box 13326 1701 North Congress Avenue Austin, TX 78701-3326 Arizona Public Service Company -4- Electronic distribution by RIV: Regional Administrator (BSM1) DRP Director (ATH) DRS Director (DDC) Senior Resident Inspector (NLS) Branch Chief, DRP/D (LJS) Senior Project Engineer, DRP/D (JAC) Staff Chief, DRP/TSS (PHH) RITS Coordinator (NBH) PV Site Secretary (vacant) ANO Site Secretary (VLH) ADAMS: / Yes * No Initials: __LJS__ DMB (IE35) / Publicly Available * Non-Publicly Available * Sensitive / Non-Sensitive R:\_PV \2003\PV2003-11RP-NLS.wpd RIV:RI:DRP/D RI:DRS/EMB SRI:DRP/D C:DRP/D GGWarnick JLTaylor NLSagado LJSmith T-LJSmith T-LJSmith @PVNGS-LJS /RA/ 10/27/03 10/27/03 10/30/03 10/31/03 DD:DRP MASatorius /RA/ 11/03/03 OFFICIAL RECORD COPY T=Telephone E=E-mail F=Fax U.S. NUCLEAR REGULATORY COMMISSION REGION IV Dockets: 50-528, 50-529, 50-530 Licenses: NPF-41, NPF-51, NPF-74 Report No: 05000528/2003011, 05000529/2003011, and 05000530/2003011 Licensee: Arizona Public Service Company Facility: Palo Verde Nuclear Generating Station, Units 1, 2, and 3 Location: 5951 S. Wintersburg Tonopah, Arizona Dates: September 15-19, 2003 Inspectors: N. Salgado, Senior Resident Inspector, Project Branch D G. Warnick, Resident Inspector, Project Branch D J. Taylor, Reactor Inspector, Engineering and Maintenance Branch Approved By: Mark A. Satorius, Deputy Director Division of Reactor Projects Enclosure SUMMARY OF FINDINGS IR 05000529/2003011, 05000528/2003011; 05000530/2003011; 9/15/03 - 9/19/03; Palo Verde Nuclear Generating Station, Units 1, 2, and 3; Special Inspection Report. The report covered a 1-week special inspection by one senior resident inspector, one resident inspector, and one regional engineering inspector, who assessed the licensee and reactor plant response to an automatic reactor trip resulting from a three-phase ground on the offsite power grid. No findings of significance were identified. The NRC’s program for overseeing the safe operation of commercial nuclear power reactors is described in NUREG-1649, "Reactor Oversight Process," Revision 3, dated July 2000. A. NRC-Identified and Self-Revealing Findings No findings of significance were identified. Enclosure REPORT DETAILS 1. SPECIAL INSPECTION ACTIVITIES The NRC conducted this special inspection to better understand the response of all three units to the grid disturbance, consequent Unit 3 reactor trip, and subsequent equipment problems. The special inspection team evaluated the potential safety implications related to the cause of the reactor trip and the subsequent loss of some nonsafety-related equipment. The team used NRC Inspection Procedure 93812, "Special Inspection," to conduct the inspection. The team reviewed procedures, operator logs, corrective action documents, a posttrip review report, and design and maintenance records for equipment of concern. The team interviewed key station personnel regarding the reactor trip event and restoration of systems. The team performed a walkdown of the Unit 3 cabinets that contained the subsynchronous oscillation (SSO) relays and the undervoltage (UV) relays. Attachment 2 is the team charter which describes the inspection scope in greater detail. 2. DESCRIPTION OF EVENT AND CHRONOLOGY 2.1 System Description Palo Verde Nuclear Generating Station (PVNGS) Electrical Distribution System The PVNGS 525 kilovolt (kV) switchyard is the receiving location for offsite power from the Westwing 1 and 2, Rudd, Hassayampa 1, 2, and 3, and Devers lines, which are part of the offsite electrical grid system. The PVNGS switchyard also serves as the transmission system for electrical power generated by the site’s main generators, which exits the switchyard through the same lines when the plant is operating. The PVNGS switchyard consists of two buses, the 525 kV east bus and the 525 kV west bus. All breakers in the switchyard are electrically isolable by two disconnect switches on either side of the breakers. Power to PVNGS Class 1E electrical equipment is normally supplied from the PVNGS switchyard through startup Transformers X01, X02, and X03. This electrical equipment includes the engineered safety features (ESFs) equipment powered through electrical Buses PBA-S03 and PBB-S04 for each unit. If offsite power to these ESF buses is degraded or isolated, emergency diesel generators will start automatically and provide power to the ESF electrical equipment. Additionally, power is also available to the ESF buses from station blackout gas turbine generators. The SSO relays protect the main turbine generator from subsynchronous resonance phenomena that are associated with grid distribution resonances from long transmission lines and widely separated generation sources. The subsynchronous resonance phenomena can result in high torsion on the turbine generator shaft. The setpoints for the relays are staggered between the units, with Unit 3 being the most sensitive and Unit 1 being the least sensitive. The setpoint staggering allows for tripping of a single unit, which should provide enough detuning of the grid to avoid having the other units trip. Enclosure -2- 2.2 Event Summary On July 28, 2003, the Unit 3 reactor automatically tripped due to a low departure from nucleate boiling ratio signal caused by loss of reactor coolant pumps, which were powered from the nonsafety 13.8 kV buses that de-energized. The loss of power to these nonsafety buses was caused by a main turbine generator trip and failure to complete a fast bus transfer (FBT) from the normal auxiliary transformer supply to the alternate offsite startup transformer supply. The normal auxiliary transfer supply was unavailable because the main turbine generator tripped due to operation of SSO relays. The trip was caused by a grid disturbance. The FBT did not occur because the UV relay detected that voltage had not recovered from the voltage transient. This is a design feature which prevents a fast bust transfer to a dead bus or otherwise damaged power supply. The cause of the grid disturbance was a three-phase bolted ground on the offsite Hassayampa 1 525 kV line, which is approximately 1.5 miles from PVNGS. The ground was caused by a maintenance error in the Hassayampa switchyard. All three units sensed the grid disturbance. Unit 3 tripped, as designed, due to the staggered setpoints on the SSO relays. The transient’s duration was approximately 50-67 milliseconds, with the grid voltage being recovered in approximately 2 additional seconds. The 4.16 kV safety buses experienced a voltage drop, but the transient was so short that the bus UV relays did not de-energize the buses and emergency diesel generators did not start. The grid disturbance also caused a loss of power and lock-out of instrument air compressors and normal chilled water chillers on Units 1, 2, and 3.