NRC & Operator Integrity – Could This Happen to You? – And What is the Root Cause?
Article about an actual event written by Mark Sharp:
What Was I Thinking?
“… former senior reactor operator who mistakenly entered incorrect information into a plant computer during a maintenance operation on Nov. 8, 2006, and subsequently attempted to cover up the mistake by falsifying the record…” NRC news release, October 22, 2007
After 25 years of working in the nuclear power industry in both operations and operations training, you would think that I would have had a handle on operator fundamentals. Self-checking, placekeeping/logging, and error reduction techniques were a part of my every day activities. One of these fundamentals, the trustworthiness and honesty of the employees, became a stumbling block for me one night in the early hours of the morning. Before I can talk about that, I need to give you some background information, including some aspects of human performance/operator fundamentals that need to be reviewed and that could be applied to all nuclear plants.
The multi-unit site I was working at requires a few manual inputs into the plant computer(s) for maintaining the secondary calorimetric up to date. One of these inputs is to account for the Steam Generator Blowdown flowrate from each steam generator. Every time you change the flowpath, and thereby the flowrate, you have to update the computer with the Engineering supplied flowrate constant. This maintains the secondary calorimetric indicated power equal with actual power level. Optimum plant power levels can then be maintained without exceeding limits.
So, what happened that night shift back in November 2006? Following maintenance, our crew was tasked with restoring piping associated with the Steam Generator Blowdown heat exchanger. This would require realigning the blowdown flowpath. Following the required alignment, I made an error in pulling the Steam Generator blowdown constant number off of the operator aid (multiple column/multiple rowed 3” x 4” card) taped to the side of the computer screen and inputting this wrong number into the computers. I also logged this same incorrect number into the constant change log book. The person performing the independent verification (IV) of the action failed to identify my mistake.
What contributed to this event? Let’s identify some human performance error traps that the crew and I came across. Human performance error prevention tools that could have been used to prevent the event from occurring will also be discussed. What is an Error Trap? An Error Trap can be described as situations or conditions that are established that provide the opportunity for mistakes to occur. The following traps were there in the early morning hours on our shift in November 2006:
• First Shift/Late Shift – During the early hours of the morning, the mental alertness of those involved can be greatly diminished. This diminished thought process can lead to incorrect or inappropriate actions.
• Change/Off Normal – The crew that night was made up of operators and supervisors from three different crews due to personnel vacation and outage coverage for a sister unit at the same site. A newly promoted Control Room Supervisor (CRS), Senior Reactor Operators (SROs) doing Reactor Operator (RO) duties, and a Shift Manager (SM) from another unit all contributed to the crew makeup that night. People were not used to working with each other or the roles that each was to perform.
• Overconfidence – The least experienced SRO had 15 years of service at the facility. Between the four SRO licensed crew members that night, over 100 years of combined nuclear plant experience was present. Swapping of Steam Generator blowdown was a routine evolution that was performed weekly. All of us had years of doing this with success and knew that we could do it again, even while at minimum control room manning levels. This “Get ‘er Done” attitude may have contributed to the attentiveness level.
• Assumptions – The Independent Verifier assumed that I had performed the task correctly. His check only consisted of verifying that the log book matched the value inputted into the computer. He did not check either number against the standard, that being the number provided by Engineering. This assumption lead to the error going undetected for well over an hour. We need to trust the people with whom we work, but we also need to verify actions taken that can affect important plant systems or parameters. Accurate indication of plant power is essential to plant safety.
• Peer Pressure/Time Pressure – No one likes to “look bad” in front of your peers. At the time of the event, further advancements to the CRS position from the SRO ranks were being considered. The desire to perform well for others may have tainted attention to detail. Perceived time pressure was being applied by management. The SM wanted to have the task completed several hours before end of shift, so we had to get the job done and systems stable before 4:00 o’clock in the morning. His expectation was voiced to the CRS multiple times during the first part of the shift, and it was passed down to me each time. These pressures added a mental stress factor that contributed to the lack of Self-Checking on my part in identifying and using the correct blowdown constant.
While there were numerous error traps laid, there were also numerous human performance tools available to mitigate them. Several of these tools, had they been implemented more effectively by the crew, or by myself as an individual, would have added an extra layer of defense against the traps that existed. Correct and accurate implementation of the following fundamental operational practices could have prevented this error:
• Prejob Brief – While a prejob brief was held with all of the crew members involved, it failed to go into adequate detail. If we had discussed what exact flowpaths we would be using, we would have been more attentive to that aspect of the evolution. Also, better use of all crew members in the monitoring of the evolution would have had positive effects. Having the SM or the Shift Technical Advisor (STA) identified during the brief as the individuals providing the needed big-picture oversight would have ensured additional eyes and ears monitoring for success.
• Questioning Attitude – No one really questioned the outcome. We didn’t think about what constant we would be using. If we had questioned which one of the three different constants we would be using, or if one of us had questioned the flowpath, perhaps that would have prompted us to verify the constant used. I failed to question why the number I wrote in the constant change log was different than previously performed re-alignments. Having to answer that question could have identified the error.
• Verification Practices – An adequate and thorough independent verification would have caught this error immediately. The verification has to be one that is not tainted by assumptions as to the first person’s skill or past performance. It must not only verify that the number or parameter is correct, but must also check it against the identified standard for expectance.
• Self-Checking (STAR) – This is the tool that should have prevented the mistake from ever occurring. If I had stopped and thought about which one of the three blowdown constant numbers my finger was pointing at, I would have identified the missing piece of information – that of the flowpath I had just lined up back on the control board. If I had stopped long enough to get my thoughts re-focused on the job and not on the distractions, another error-free blowdown swap would have occurred. STAR isn’t just for field evolutions. It is not just for main control board manipulations. It is for all tasks that operators perform, even that of identifying and entering a correct number into a computer.
Inadequate implementation of human performance tools and making an error does not equal a trustworthiness and honesty issue. It is what I did when the error was realized that is the real issue that requires further discussion.
When I was performing the IV for subsequent blowdown realignments on the same Steam Generator later that shift, I noticed that the number I had used over an hour earlier didn’t line up with the number that the other operator had retrieved off of the same operator aid. One of us had made a mistake. It only took me seconds to realize that it was mine. But how do you fix something that happened over an hour ago and wasn’t even wrong anymore? What you don’t do is what I did – freeze up. I hesitated just long enough that I gave my night shift, sleep deprived brain time to start imagining all the negative possibilities and consequences that upper management could impose. So I simply didn’t tell the CRS what I had just discovered. I spent the next few hours trying to figure out what to do. At about 4:00 in the morning, the only idea I could come up with was to line out and initial the incorrect entry in the constant change log book and write in the correct number. When I was about half way home that morning, it dawned on me what I had done. I hadn’t fixed anything. I had made a stupid momentary lack of STAR into a major error. While I was home telling my family how I had just made a tremendous mistake, a possible career altering one, the Operating crew back at work were doing their job. One part of that job is a shiftly review of the computer’s demand-typer log that tracks all of the manual inputs. One of the dayshift crew members noticed that the Steam Generator Blowdown numbers didn’t look right and using that Questioning Attitude that we as an industry foster, investigated the constant log book. The discrepancies were elevated to supervision and management. I was scheduled to work overtime that night to support the sister unit’s refueling outage. Knowing that this log review would be done, and having a desire to own up to my actions, I went to work early that night to type up my personal statement as to what happened and why. While typing was in progress, the crew from the previous night came in to work at the other unit. Shortly thereafter, they called me.
So what did the momentary lack of trustworthiness and reliability cost me? The company’s investigation resulted in a recommendation to take me to the to the site disciple review board for action. Since they did not tell me what that would entail, me fearing the worst, resigned so I couldn’t be fired. I was told later that day that they would count it as a normal resignation. Shortly thereafter, I obtained employment at another nuclear facility. I naively thought I was Ok and was going to be able to work where ever I wanted. However, six weeks later, I’m told by that site’s security/access supervisor that my site access was being placed in an “Administrative Hold” condition pending the outcome of an NRC investigation at my previous plant. This was the first I knew that there was an investigation in progress. I remained on “Admin Hold” for about seven months. Being on “Admin Hold” prevents you from going in the protected area. During this time I had numerous phone call discussions with the NRC, and one in-person NRC Office of Investigation interview (arm raised to the square, with a court reporter type of interview). The investigation resulted in me getting an invitation to go to the Region’s office for a Pre-enforcement Conference. I choose to use the Alternative Dispute Resolution (ADR) process – which brings in a 3rd party mediator. As a result of that ADR meeting, an agreement was reached and a Confirmatory Order was written. Since it is a public record, I can share it with you. It required me to write a letter to the NRC re-committing to integrity in action and deed, which will be kept as part of my official file. I also was required to share my experience with others within the nuclear industry. This article and a separate Operating Events (OE) type report I submitted to INPO are my attempts to do just that. Once these three items are complete, I will be allowed to perform licensed operator duties again (if some company will allow me). The Confirmatory Order did not prohibit me from being involved in Operator Training; in fact, I’m required to mention in appropriate training settings, lessons learned from my actions for a period of one year – which the NRC can request documentation of at any time.
When the Confirmatory Order was finalized, the site I was working at had mixed reactions. The front line supervisors I had been working with through-out this ordeal in both Operations Training and Security/Access were happy to have it over with. Training Supervision starting making plans to keep me on site for another year to finish up with the license class I was teaching. Based on the details of the Order and on personal interviews, the Security/Access Supervisor recommended, and the Department Director agreed, that my site access would be and was re-instated.
However, the site’s executive leadership committee canceled my contract and sent me off-site due to possible “Optics” that they felt may have come their way. However, when you think about it, every nuclear plant has to take actions to protect this asset, that being the public’s trust.
The difficulty of being released was magnified by my history and the Personnel Access Database System (PADS). Because of a PADS note that my first station put in saying I had a trustworthiness issue, no other facility would pick me up, either in-house or contractor. I was lucky in that there was a change in management and philosophy at my first station during the time I was out-of-state working. Based on this occurring, and the positive input of my previous fellow workers and supervisors who spoke on my behalf, I was hired back into Operations Training. The fact that I self-disclosed the event to management before questioning the previous year was a factor in my favor. The fact that I wasn’t trying to claim that I was a victim helped in my six separate re-hire interviews, including one with the site Executive VP. I had made a serious mistake and was paying the cost of that mistake. I’m still trying to work through the PADS issue and get a site ACAD for unescorted access here. So things are still not over when dealing with this event. I consider myself lucky to be working. Cost? I lost 15 years of service to a great company, a year of my life with associated stress of dealing with the NRC, not being able to work where I want to and when I want to, the cost of travel back and forth across the country, and the intangible trust/respect of fellow workers. I even jeopardized a measure of the public’s trust in the nuclear power industry, this trust that you and I need to ensure so that our industry remains alive and well in this country. The unpleasant experience of sitting at a table across from the Region Administrator, the Region’s Legal Representative and one of the Region’s Enforcement Specialists to discuss your honesty and integrity is not something I’d wish on any operator.
Can you learn from my error and my response to it? I hope so. Here are a few learnings to consider:
• Use your company’s Employee’s Assistance Program – It is not just for substance abuse problems and issues. It can provide counseling and advice for stress management and emotional issues. It can provide guidance for employment options. Even experience operators who haven’t made a personal error in years may need this kind of assistance. Be proactive in looking for help, and don’t let over-confidence convince you that you are invincible to errors. There is no weakness in self-referral to a program that will help your performance. It is not vindictive for a supervisor to suggest this avenue to their employees. Not only is it good business, it can help maintain performance.
• Stress and pressures are a part of your job – There has been, currently is, and probably will be in the future, stress and pressures at work. Doing more with less while maintaining safety and production can cause chronic mental stress. There were extenuating circumstances surrounding my mistake; there may be some in your lives also. Do not let them become your masters. You need to recognize those behaviors talked about so much in Employee Behavioral Observation training – such as changes in attitudes and performance levels, and realize that these may be precursors to future events. Supervisors, who recognize issues with their employees, should take positive action to address them. Increased breaks or varying assignments, additional peer checking and/or concurrent verifications are tools that can be used to mitigate possible events.
• If you mess up, then fess up, don’t cover-up – It is extremely important that you inform supervision if you make an error. You need to start the process to make things right as soon as possible. The short term consequences of an error are small in comparison to the long term negative ramifications of falsification. A Human Performance Event Investigation may have the short term consequence of your removal from shift pending training or counseling. It may also have the benefit of correcting latent error traps in existing procedures, or providing opportunities for individual and crew assessments so that improvements can be made. If there are Station Management policies or practices that foster a “chilled work environment”, perhaps this investigation may promote needed change in that area as well. Long term negative ramifications include losing your individual NRC Operator License that you worked so hard to obtain and to maintain. Not being able to work in the nuclear industry, doing a job you like where you like it, or having employment at all can be another direct result. Falsification of records is a violation of 10CFR statues that can have fiscal and penal consequences. Regaining lost honor and respect from peers may be difficult, at best. Public trust in nuclear power maybe challenged which could result in slower expansion of our industry due to increased legal challenges or governmental investigations. Doing what is right, all the time not just when someone is watching, is a trait we each need to strengthen within ourselves and our crew.
Human performance error traps and human performance tools will be a part of our industry for as long as there is fabricated fuel. We must recognize that the traps exist so that we can implement the tools to mitigate them. Using the tools, as individuals and as crews, can prevent otherwise good, steady operators from making career altering decisions based on momentary personal weaknesses. We need to be attentive to those with whom we work so that we can support them in our common quest for excellence in operations. Error free performance is possible – hundreds of licensed operators do so everyday across the country. Think about it. I wish I had.
– – – – – – – – –
So what is amazing about this article?
Is it that someone could make a simple mistake? NO
Is it that even a nuclear operator could try to cover up a mistake? NO
Is it that someone in the nuclear industry could get so worked up about finding an error 30 minutes after it was made that they could jeopardize their entire career? NO – Especially if they believed they were in a blame oriented environment.
The amazing things are that:
1. The utility is having someone make manual entries that could so easily be made in error.
2. That a 3″X4″ card with data written on that has been taped to a computer is an “operator aid.”
3. That the nuclear industry thinks that the WEAK Safeguards that the author lists (Pre-Job Brief, Questioning Attitude, Verification Practices, and STAR) are sufficient when dealing with important data (entering data used to calculate the secondary calorimetric). Although this isn’t a “nuclear safety” issue, it reflects current thinking in the nuclear industry that if operators would just try hard enough and self-check enough, their performance would be 100% accurate.
4. That the root cause analysis of this error didn’t (and I’m assuming it didn’t) address the inherent Human Engineering root causes of this data entry problem – causes that are the true root causes of the initial error.
I’m not trying to give Mark Sharp a pass on his mistake. He should have immediately reported his error.
But perhaps if he had felt that a thorough root cause analysis with effective corrective actions would have been the outcome of reporting his mistake, MAYBE he would NOT have hesitated to report the mistake. And the utility could have changed the task so that future operators would have less chance of making the same mistake.