Root Cause Analysis Blog

James Wallace of the Seattle Post-Intelligencer reported that Boeing is having trouble meeting the production schedule for the 787 Dreamliner and will compress the testing schedule to a previously never accomplished time frame. If they miss this condensed schedule, they won’t meet their May 2008 delivery schedule and will owe the purchasers financial compensation (penalties).

An additional story with comments from an AP reporter mentioned that troubles at competitor Airbus has caused billions of losses due to “wiring problems.”

Billions lost and critical path delays … Sounds like an opportunity to apply advanced root cause analysis!

New of explosions reported by local press. See:

http://www.ncnewsonline.com/local/local_story_243235944.html

And the Root Cause is “Pilot Error” (or so says many of the accident investigations).

What could they learn by analyzing the root causes of these accidents with TapRooT®???

Or sometimes ground crew error!

In September of last year a German Maglev train crashed into a maintenance vehicle at 170 kph.  23 people were killed, and the system is still shutdown.

According to this link, a control room operator is being charged in the incident with manslaughter through culpable negligence.  In this brief article, it says that the operator had “simply forgotten” that the maintenance vehicle was still on the track.

Without access to the full report, you have to wonder what else came out of the investigation.  Any system as complex and potentially dangerous as this would be assumed to have multiple safeguards in place to prevent this type of accident. 

• Are there visible and audible warnings on the control panel?
• Since the system can see train positions on the tracks, are there automatic interlocks in place?
• How did the maintenance crew document their use of the track?
• How is a “maintenance vehicle” annotated on the panel?
• Who approves the opening and closing of track segments for maintenance?

If the major safeguard in place is “operator must remember that the track is not clear,” it is not surprising that the accident occurred.  Hopefully, the full results of the investigation will contain more than just “discipline the employee.”

I presented my paper at the IEEE/HPRCT Human Factors and Power Plants Conference. The paper for today is titled:

Improving and Existing Root Cause Analysis and Corrective Action Program

I had two attendee participation sections of the talk. These mini-Kaizen events required small teams to develop ideas based on Lean thinking to improve root cause analysis.

The first session looked at problems that cause waste in root cause analysis. The teams identified a waste type and develop a possible solution.

Heres what the teams said:

1. Waste: In an investigation, do things (interviews, evidence collection, …) over and over and waste time and effort because we didn’t plan enough before starting the investigation.

Solution: Better training for people so that they will be more efficient in performing the investigation, especially the pre-investigation planning.

2. Waste: People (including the facilitator/team leader) change in the middle of an investigation or multi-tasking (trying to get their regular work done) plus doing investigations just for political reasons.

Solution: Management needs to value investigations so that they assign adequate resources. Also, management need to stop requiring political investigations. How to get this accomplished? ????

3. Waste: Can’t get buy-in to corrective actions and waste time responding to push-back to effective corrective actions.

Solution: Involve management in corrective action development.

4. Waste: Takes to long to assign an investigator (evidence disappears, people’s memory isn’t fresh).

Solution: Pre-select and designate investigators and assignment is automatic.

5. Waste: CARB (Corrective Action Review Board) review and approval of an investigation is a waste of time (doesn’t add value).

Solution: Get good scope and charter from senior manager up-front. Give charter to CARB before review. May avoid the “You brought me the wrong rock” syndrome. Also, have a standard design for reports. May also eliminate CARB review of report (it is the team’s report) and only allow CARB to select corrective actions to implement.

6. Waste: Can’t get qualified people assigned OR not enough people assigned OR assign a single investigator when a team was needed. Insufficient or incorrect resources cause delays and waste.

Solution: Get adequate pool of people (cross functional) trained and assigned to investigations from a full-time investigator pool.

7. Waste: Interviewing does not get “fresh” information - all the stories are too much the same. Waste time collecting the decided upon story.

Solution: Develop a standardized list of questions focusing on the “What” on-line to be completed by the supervisor before the crew goes home.

8. Waste: Time wasted trying to gather information at the scene.

Solution: Provide supervisor with evidence collection form and training on evidence preservation/collection.

9. Waste: Waiting to decide to investigate an incident while management decides if the investigation is worth investigating.

Solution: Assign a single person to decide if an investigation is required.

The second attendee exercise was to look at there root cause analysis system and Corrective Action Program and make it more efficient by applying lean thinking. They could choose one of three topics to attack:

1. What does the customer want (Customer Focus)?
2. Process improvements (making your root cause process more efficient).
3. Apply 5S (Sort, Set Order, Standardize, Shine, & Sustain)) to improve your root cause analysis.

Here is what the teams reported were their ideas:

1. Customer Focus: Educate the customer. Head CAP guy ASKS senior management sponsor what they want and then discuss what root cause analysis can produce.

2. Apply 5-S:

Sort - Sort incidents into equipment or human performance

Set Order - Develop a checklist for the investigator.

Standardize - Adopt a standard report format.

Shine - Provide training to have proficient investigators/teams.

Sustain - Do evaluations of investigations an improve as required.

3. Process Improvement: Instead of department head assigning an investigator and the “letting go” of the investigation, the department manager should assign the investigator/team and then get daily/weekly feedback (updates) from the team.

4. 5-S:

Sort - Sort out tools required for investigation.

Set Order/Standardize - Organize tools into a standard toolbox and a standard list of questions to be asked. .

Shine/Sustain - Continue to use tools.

5. Customer Focus: Develop a written charter and scope and provide it to the CARB and (if applicable) get the DOE (Department of Energy) rep involved early. Provide training for managers and management sponsor so that they understand the process and the value added. Must determine what the customer wants.

6. Customer Focus: CARB  should want (and focus on) preventing incident recurrence. How do you sell root cause analysis is designed to prevent incident recurrence/prevention? Need to understand CARB’s mental model. Need high level by-in to CARB reviews to prevent CARB membership from becoming a revolving door.

7. 5-S of Prep for Investigation:

Sort - Sort out what is needed to perform investigation. Tools, projector/computer, interview sheets, measurement devices…

Set Order - Get a box or room and put all the “stuff” needed there. Pre-set-up..

Standardize - Checklist for replentishing supplies so that the room stays properly stocked. .

Shine/Sustain - Keep the room/box stocked so that it is always ready.

8. 5-S:

Sort - Develop a list of info requirements for an investigation including extent of condition and extent of cause. Also list should include who should be on a team for types of investigations.

Set Order - Develop a standard investigation order that includes checking operating experience.

Standardize - Develop a standard procedure for performing an investigation … a checklist!.

Shine - Continue to look at  priorty/order of steps in an investigation. .

Sustain - Conducting assessments of investigations to improve.

9. Process Improvement: Why do people not want to do investigations? Look at investigation requirements and plan/look at time required/what is required. Make investigation their primary duty. Give people the time needed to do a proper investigation. Have a senior level manager champion to make sure resources are available.

- - -

One more idea. After the presentation someone suggested:

Have “in-process peer reviews” rather than post-investigation peer reviews. Designate the reviewer when the team is assigned and have them perform continuous reviews as the investigation progresses. MUCH MORE effective than post investigation review. Also, people must perform investigations frequently enough so they don’t forget lessons learned about performing better investigations.

The recent Skype outage provided some interesting fodder for root cause analysis. See the incident description at this link:

http://heartbeat.skype.com/2007/08/the_microsoft_connection_explained.html

The best “network” root cause analyst I know is Gerald Starling. He is a TapRooT® User who learned to find real root causes (rather than symptoms) while working at BellSouth.

With network/IT reliability being such a big issue in the internet age, you would think that more IT folks would be applying advanced root cause analysis and get beyond troubleshooting symptoms. Unfortunately, most people think they have found a root cause when they find a piece of code that can be fixed that stops the problem. Thus they stop at the symptom (Causal Factor in TapRooT®ese) and never fix the true root causes.

Tailchi Ohno, the creator of the 5-Why technique, is quoted using the following example to demonstrate using 5-Why’s for root cause analysis:

1. “Why did the robot stop?”

The circuit has overloaded, causing a fuse to blow.

2. “Why is the circuit overloaded?”

There was insufficient lubrication on the bearings, so they locked up.

3. “Why was there insufficient lubrication on the bearings?”

The oil pump on the robot is not circulating sufficient oil.

4. “Why is the pump not circulating sufficient oil?”

The pump intake is clogged with metal shavings.

5. “Why is the intake clogged with metal shavings?”

Because there is no filter on the pump.

What do you think? Is “NO FILTER ON THE PUMP” a root cause?

Let me know your comments.

And if the inventor of 5-Why’s uses this as an example, should people call 5-Whys a root cause analysis technique?

Of course I’m saddened by the loss of life that occurred when an interstate bridge spanning the Mississippi River collapsed. But I can’t help thinking of the broader implications of this failure.

Click for video of fallen bridge.

Fox News, CNN, MSNBC, and even the Weather Channel were talking about the tragic collapse. The TV talking heads we already starting to either guess, or ask their guests to guess, what the causes of the collapse were. In all the coverage I only saw one true bridge expert, who seemed credible, explain how metal fatigue (fatigue cracking) could lead to this type of failure. Most of the other anchors, on-scene reports, and experts babbled endlessly to fill the airtime.

These people obviously needed instant root cause analysis.

Our 24 hour news cycle can’t wait for real answers at to what happened and how a bridge could fail so dramatically and tragically. And our political system can’t wait until they start pointing fingers and placing blame.

A good example of this is the post Katrina coverage of the levee failures. Only recently has the information become available as to the long history of failures that caused that disaster. But the press coverage of that history - that includes legislative failures, compromises of public safety due to environmental lawsuits, and issues of project management by the Corps of Engineers - has been almost nonexistent. They couldn’t wait for a detailed analysis of the facts. Instead, they jumped on the blame train.

What can you learn from this disaster at this early stage? First, if you are responsible for investigations of major accidents at your company, you had better be ready to deal with the press. They won’t be willing to wait for a detailed root cause analysis and they will get “experts” on the air to explain what “might have gone wrong.” You need to express concern and genuine sadness (which should be easy because you will be concerned and genuinely saddened) and then explain what you will be doing to find the real root causes of the accident without jumping to premature conclusions.

Another lesson learned from this accident could be that incident investigators need to be prepared to start an investigation and find out what happened and how it happened as quickly as possible to provide real information to the press rather than pure speculation. However, you should always caution those who will listen that WHAT HAPPENED and HOW IT HAPPENED still is not the ROOT CAUSES of WHY IT HAPPENED! What and how are just the information you need to draw your SnapCharT®. Further detailed, systematic analysis is required to find root causes.

What if you are senior management? You need to restrain your initial urge to jump to conclusions and start placing blame. This will be difficult because if you don’t find someone else to blame, you may become the target for blame. The example of how wrong this initial urge to place blame can go was demonstrated by the BP Texas City Refinery explosion aftermath. The entire senior management chain - from the Plant Manager to the CEO of a major corporation left the company under a cloud - either voluntarily or by being fired - after they tried to place the blame for the accident on the operator and the supervisor involved in the incident.

Perhaps the most important lesson learned at this point is that the best way to avoid the whole post-accident blame cycle is to avoid the accident entirely.

After the accident, the need for previous fixes is always apparent. However, to have truly excellent, zero accident performance requires management that carefully listens to the voice of the facility expressed by the root cause analyses of incidents, near-misses, audits, and proactive improvement initiatives. That’s why senior management needs a through understanding of advanced root cause analysis and performance improvement - something that is almost completely missing from the senior ranks of government as well as the senior ranks of many companies. Perhaps this is why disasters - natural or man-made - continue to create a constant supply of tragic headlines.

Some readers may be thinking - “HOLD ON, OUR MANAGEMENT UNDERSTANDS ROOT CAUSE ANALYSIS!” I’m sorry … 5-Whys, fishbone diagram, and brainstorming won’t do the trick. They aren’t advanced root cause analysis. They won’t produce the results needed to accurately and consistently find the read root causes of problems.

Government decision making and major corporate failures are notoriously hard to analyze. Government decision makers who started the chain of events or kept it progressing may be long gone. And the way major disasters are analyzed - with “blue ribbon commissions” - seldom produce the permanent change needed to change dysfunctional organizations.  The commissions are appointed by the powers-that-be and are often full of people either interested in ensuring that their political party or organization isn’t blamed (or the other party or organization is blamed) or have an agenda that skews their analysis.

This leaves us with the sad reality that disasters will probably continue to produce headlines because the knowledge and systematic processes needed to stop accidents probably won’t be learned by a large enough fraction of the population to demand change from our elected officials and corporate leaders.

Above is a pdf of an independent report sponsored by the Army Corps of Engineers. Very interesting reading.

Digging in to it made me want to draw a SnapCharT®, define the Causal Factors, and start through the Root Cause Tree®. The problem is that the information is so extensive and the sequence of events is so long that it would takes weeks - or maybe months - to do a good, thorough root cause analysis. Then we would be ready to analyze Generic Causes and start developing corrective actions.

I already have so much to do that I just can’t find the time to dig in, find the root causes, and answers to fix the problems. To tell the truth, I barley had time to read the report (ah - great vacation reading).

Plus, my guess is that the Management System causes (of which there seem to be many) would require the government - local and national - and even Congress to change. Sometime you shouldn’t start a root cause analysis if you aren’t ready (or able) to fix the problems you find because root cause analysis without the ability to implement corrective actions is a waste of time.

An AP Story at the CNN web site said the following:

Comair pilots’ failure to notice clues that they were heading to the wrong runway was the primary cause of last summer’s deadly Kentucky plane crash that killed 49 people, safety investigators concluded Thursday.

“That’s the frustration of this accident — no single cause, no single solution and no ‘aha’ moment,” Hersman said. “Rather than pointing to a mechanical or design flaw in the aircraft that could be fixed or a maintenance problem that could be corrected, this accident has led us into the briar patch of human behavior.”

Is “pilot error” really a root cause? Should there be only one “root cause” for a major accident?

More from the article:

NTSB staff member Joe Sedor identified one possible overriding factor — unnecessary chatter between pilot Jeffrey Clay and first officer James Polehinke as they prepared to taxi and take off. Comair has acknowledged some culpability as a result of the talk, which violated FAA rules calling for a “sterile cockpit.”

Did the 40 seconds of conversation really cause the accident?

Ken Turnbull, a TapRooT® Instructor and experienced investigator of accidents (but not aviation accidents) will present his analysis of the crash at the 2008 Summit. If you would like to see how TapRooT® can be used to find the real root causes of a major accident, attend Ken’s talk.

Here is the link to the NTSB’s press release:

http://www.ntsb.gov/Pressrel/2007/072607.htm

What can inadequate root cause analysis get you? A warning letter from the US FDA and a big headache.

Recent press reports detail the headaches of Srtyker Ireland, LTD and their troubles with the FDA. For more information see these links:

http://www.reuters.com/article/health-SP/idUSN1926664420070619

http://www.fdanews.com/newsletter/article?issueId=10468&articleId=96073

http://www.fda.gov/foi/warning_letters/s6387c.htm

http://www.ryortho.com/NEWSSHORTS/volume3/issue20/06-22-07-NS-FDA.htm

How big a problem is failure to perform adequate root cause analysis in industries regulated by the FDA? A quick search of the FDA’s reading room warning letter database yielded 26 letters in the past 12 months. That’s not an exhaustive study, but it is an indicator. Many medical device manufacturers and drug companies need to do a better job finding and fixing root causes.

Lesson Learned? Think ahead! Get an advanced root cause analysis program in place before the regulator arrives with a letter. Then make advanced root cause analysis a key part of your performance improvement and operational excellence program.

For more information about advanced root causes analysis courses around the world, see:

http://www.taproot.com/courses.php

The Cumberland Times-News reported on MSHA’s root cause analysis of a fatal highwall collapse at Tristar’s Caledonia Pit. For the story, click on this link:

http://www.times-news.com/local/local_story_191092144.html

For the complete MSHA investigation report, click on this link:

http://www.msha.gov/FATALS/2007/FTL07c0506.pdf

And once you’ve read the report, think of the management system violations (management shortcuts) that lead to this fatality.

Lean Root Cause Analysis

A two part series based on
a talk at the 2007 TapRooT® Summit
by Mark Paradies & Kevin McManus

Lean Intro

What is Lean? It starts with the Toyota Production System, a highly efficient, customer focused, streamlined manufacturing process that helped Toyota survive when faced with competition from GM and Ford after WW II. The Toyota Production System started with Sakichi Toyoda at his textile mills and at his son’s (Kiichiro Toyoda) company, Toyota Motor Corp. Much of the credit for developing Lean is given to a Toyota engineer, Taiichi Ohno, and a consultant, Shigeo Shingo.

The main goal of Lean is to drive waste (muda) out of the production system. The seven forms of “deadly waste” are:

• Overproduction
• Transportation
• Waiting
• Inventory
• Motion
• Over-Processing
• Defects

These types of waste should be reduced (or eliminated) through the application of systematic tools to improve efficiency and effectiveness. Some of the systematic tools that are commonly part of a Lean implementation are:

• Value Stream Mapping
• Poka-Yoke (mistake proofing)
• Kanban (pull production)
• Kaizen (change for the better)
• Just-in-Time (inventory reduction)
• Total Productive Maintenance
• Quick Changeover
• Cellular Manufacturing
• 5S (sort, set order, standardize, shine, and sustain)

These systematic manufacturing improvement tools are applied to provide the customer with what they want, when they want it. For details about the Toyota Production System, see Modern Approaches to Manufacturing Improvement: The Shingo System by Shigeo Shingo and Toyota Production System: Beyond Large-scale Production by Taiichi Ohno.

Who is the Customer?

Lean production gives the customer what they want, when they want it. This requires understanding of what adds value from the customer’s perspective. Waste comes from producing things that the customer doesn’t want. So the root cause analyst should ask:

• Who is the customer for my root cause analysis?
• What does the customer really want?

Perhaps you have already determined that your customer is management. And we all know what management wants:

• STOP THE PROBLEM.
• Don’t spend much time.
• Everything should continue as normal without disrupting any work and without causing delays to the tasks that the investigators were assigned prior to the investigation.
• Don’t point blame at management (especially senior management).
• Don’t provide evidence for lawsuits or for fines by government regulators.
• Find a few simple, inexpensive fixes that can be implemented easily.

This is NOT a joke. These are the desires of many managers who ask for a root cause analysis. And who wouldn’t want something like this? The desire is completely unreasonable, but understandable.

Therefore, as the leader of a lean root cause analysis, you must manage your customer’s (management’s) expectations. Make sure that your customer knows what they are buying. Management must have a reasonable expectation as to the cost for a good root cause analysis. And they must understand that root cause analysis IS NOT a magic bullet to solve all problems with no investment. Rather, advanced root cause analysis is a sophisticated performance improvement tool that when applied with diligence and thought, can lead to excellent (but perhaps not perfect) performance.

More Customers

Before you think this is all that you have to do, you may need to think more about who the customer is. First, “management” isn’t a good enough definition. What level of management? What manager (or managers) in particular?

The need to manage expectations and competing priorities from several managers and different levels of management can complicate delivery of a root cause analysis that is “just what the customer ordered.”

This should focus the analyst on their MAIN objective – stopping the problem. If the problem doesn’t stop (if an effective fix isn’t found for a serious safety, environmental, production, maintenance, or quality issue), then no level of management will be happy for long.

Next, you need to think about another set of unintended customers. These unintended customers include:

• The Government Regulator
• Workers
• The General Public
• Critics
• The Press
• The Company’s Shareholders
• Financial Analysts
• The Plaintiff’s Attorney

You may not be able to produce a report that satisfies all of the customers, but you should make rational decisions about the competing priorities.

Working to produce what the customer wants can produce radical changes in your root cause analysis and the reasonable desires of the customer.

Now that the first source of waste has been removed by producing a root cause analysis that is in line with reasonable customer expectations, the analyst can look for other ways to streamline the root cause analysis process to make it even leaner.

The first of those ways may seem very simple:

Only Investigate Incidents 
Worth Investigating

This sounds easy enough, but many companies have started by investigating only the biggest problems and then gone to the extreme of:

Investigating Everything!

Using root cause analysis to improve performance makes sense. But you have to pick targets with value to realize a return on your investment. You don’t need to investigate every burnt out bulb.

We will complete this discussion of applying Lean principles to root cause analysis in the September Root Cause Network™ Newsletter. If you would like to subscribe, CLICK HERE.

See the incident report at:

http://info.ogp.org.uk/safety/SafetyAlerts/alerts/Detail.asp?alert_id=186

RAIB reports released

The Rail Accident Investigation Branch (RAIB) has released its report into a runaway permanent way trolley incident at Notting Hill Gate on 24 May 2006. The RAIB has made nine recommendations. Full report here:

http://www.raib.gov.uk/publications/investigation_reports/reports_2007/report122007.cfm

The Rail Accident Investigation Branch (RAIB) has released its report into a locomotive runaway near East Didsbury on 27 August 2006. The RAIB has made eight recommendations. Full report here:

http://www.raib.gov.uk/publications/investigation_reports/reports_2007/report132007.cfm

The Rail Accident Investigation Branch (RAIB) has released its report into a fatal accident involving a train driver at Deal on 29 July 2006. The RAIB has made nine recommendations. Full report here:

http://www.raib.gov.uk/publications/investigation_reports/reports_2007/report142007.cfm

The Rail Accident Investigation Branch (RAIB) has released its report into a derailment at Starr Gate on the Blackpool Tramway on 30 May 2006. The RAIB has made two recommendations. Full report here:

http://www.raib.gov.uk/publications/investigation_reports/reports_2007/report152007.cfm

The Rail Accident Investigation Branch (RAIB) has released its report into two near misses at Crofton Old Station No. 1 level crossing near Wakefield on the 01 and 18 May 2006. The RAIB has made six recommendations. Full report here:

http://www.raib.gov.uk/publications/investigation_reports/reports_2007/report162007.cfm

The Rail Accident Investigation Branch (RAIB) has released its report into a tram collision at Soho Benson Road on Midland Metro on 19 December 2006. The RAIB has made three recommendations. Full report here:

http://www.raib.gov.uk/publications/investigation_reports/reports_2007/report172007.cfm

The Rail Accident Investigation Branch (RAIB) has released its report into the collision between a tram and a road vehicle at New Swan Lane level crossing on Midland Metro on 08 June 2006. The RAIB has made two recommendations. Full report here:

http://www.raib.gov.uk/publications/investigation_reports/reports_2007/report182007.cfm

RAIB investigation update

The RAIB is carrying out an investigation into a fatal accident at Ruscombe… see:

http://www.raib.gov.uk/publications/current_investigations_register/070429_ruscombe.cfm

The RAIB is carrying out an investigation into a collision at Pickering, North Yorkshire… see:

http://www.raib.gov.uk/publications/current_investigations_register/070505_pickering.cfm

The RAIB is carrying out an investigation into a derailment at King Edward’s Bridge, Newcastle upon Tyne… see:

http://www.raib.gov.uk/publications/current_investigations_register/070510_king_edwards_bridge.cfm

(more…)

For story on root cause analysis controversy, see:

http://www.earthtimes.org/articles/show/news_press_release,122558.shtmll

Watch the video and then vote by commenting on the root cause of this damage to an undersea pipeline protective coating.

click to play

System Improvements has been working since the mid-90’s to help healthcare facilities perform better root cause analysis and adopt advanced strategies to stop human error. Continuing that work, David Davis, Tommy Garnett and Ed Skompski presented and facilitated ideas for “Healthcare Best and Worst Practices for Root Cause Analysis” at the TapRooT® Summit. Below is David and Tommy’s handout for the presentation:

If you are at a healthcare facility and need to learn advanced root cause analysis or would like to learn best practices to stop medical errors, see our course info and our upcoming announcements about the 2008 TapRooT® Summit on this blog.

After a major flooding incident at a mine in Canada, Cameco published a report on their root cause analysis of the accident. The Regina Leader-Post published an article about the report and the reaction of some financial analysts that I thought was quite interesting.

First, the article said:

“The root-cause report into that flood concluded neither Cameco nor its contractor had identified risk scenarios, nor did they have necessary controls in place to prevent the flooding of the shaft.”

Later in the article it provided some quotes from analysts. One analyst was quoted as follows:

“William Vogel, an analyst with Harbor View Growth Equity Management in Connecticut, said Cameco appeared to have a ‘lax’ corporate culture. He said he would have expected the company would ‘have fired a lot of people,’ considering that lives were at stake in the mine. ‘I don’t think you have a standards problem. I think you have a people problem,’ Vogel said.”

What can you learn from this article?

Some people just aren’t happy until heads roll (discipline is taken by firing people).

This brings up the whole issue of the basis of performance improvement.

Do we BLAME incidents on people and fire them to improve performance OR do we find the system problems and fix them to ensure improved performance?

It seems that the analyst is in the blame camp. Without performing an investigation, he knew the answer … fire a lot of people!

Where does your corporate performance improvement philosophy fall? Is it oriented toward blame or system improvements? And what approach will yield the best long term results? This could be a major lesson learned!

Mark Paradies and Kevin McManus taught TapRooT® Summit attendees how to cut waste and gain efficiency in the root cause analysis process at the “Lean Root Cause Analysis” breakout. Ideas for lean root cause analysis discussed at the session include:

Building a team of experts by selecting those who know the subject matter.

Training people to use a documented investigation process.

Making sure to analyze good causal factors.

Preparing the initial SnapCharT® before meeting.

Avoiding excess wordsmithing (group around rule).

Moving meetings off shift - improving meeting scheduling.

Collecting all documents prior to meeting (using technology when possible).

Having a plan for collecting information prior to incident (and preserving evidence).

If you missed this informative session, here is a copy of the presentation:

Interesting article on brian research and distraction. See:

http://dsc.discovery.com/news/2007/03/29/attention_hea.html?category=health&guid=20070329151500

The research shows that you can’t “will” yourself not to be distracted. Willful attention and distraction are controlled by two different parts of the brain and distraction occurs even when you are trying to concentrate. Also, some people are naturally more easily distracted than others.

To see the report go to:

http://www.csb.gov/index.cfm?folder=completed_investigations&page=info&INV_ID=52

I’ll try to read and comment on it by the time I come home from England (next week).

For those who have been checking the CSB web site and are wondering …

“How come the BP report isn’t posted?”

The answer is that there were slight amendments made by the Board at the public meeting on Tuesday and those wording changes need to be made before the report is posted on the CSB web site.

When will the report be posted? Probably next week. So stay tuned.

Since I’m over in the UK teaching I can’t get to the public meeting on the BP Texas City explosion being held by the CSB today. But I’m trying to keep up on the news. In the UK, the controversy seems to be over the findings of cost cutting that, according to the press reports about what the CSB has found, are root causes of the explosion at the refinery.

For a sample of a press report in the UK, see:

http://observer.guardian.co.uk/business/story/0,,2031053,00.html
Here are a few of the key quotes from the story …

“The CSB says it has evidence in emails and other documents of budgetary considerations taking precedence over investment.”

“Merritt says that internal and external reports between 2002 and 2005 pointed to problems: ‘There was a complete failure to listen to the evidence that they were hearing: that this facility had been squeezed to the breaking point. That was received from their managers as well as from surveys by consultants.‘”

“CSB has emails indicating Texas City managers turned down requests for funding, claims Merritt. ‘We know that pleas from the plant managers were dealt with by instructions to continue cost cutting,’ she says. One such came from Walter Wundrow, a refinery investment manager, who refused an engineer’s request to install a flare, instructing him to ‘bank $150m savings’.”

“Merritt says there is an ‘iron-clad’ case of a causal link between cost savings and the accident. However, she emphasises that much had been done since the explosion. ‘There are huge changes going on in Texas City and a great deal of determination and effort to put things straight. But real culture change is very difficult.’”

I’m sure that by the time I wake up in the UK tomorrow, there will be a report posted on the CSB web site that I will need to read to see the totality of the evidence that the CSB has been collecting.

From the CSB Press Release about the root cause analysis of the BP Texas City Refinery explosion, I found the following comment:

By March 23, operators had been working 12-hour shifts for 29 or more consecutive days. “Fatigue causes cognitive fixation and impaired judgment and could lead operators to fixate on one operational parameter - such as the apparently declining liquid level - to the exclusion of other indicators,” Ms. MacKenzie said. Fatigue has been recognized as a cause of major accidents in the transportation sector. Fatigue prevention regulations have been developed for aviation and other transportation sectors, but there are no fatigue prevention guidelines that are widely used and accepted in the oil and chemical sector.

- - -

29 days on 12-hour shifts.

I remember the feeling. In my own experience at day 46 on 12-hour shifts we had an electrician almost get electrocuted (a near-miss). We didn’t conclude that fatigue was a factor (This incident occurred way before I invented TapRooT®). But now I know that it was related to fatigue.

When I hear that operators were on day 29 of 12-hour shifts … the “bad decisions” made based on faulty indicators just before the BP Texas City explosion start making much better sense. And firing the operators and supervisor after the accident make much less sense.

If you are interested in a way to judge if fatigue is a cause of an incident, you should attend the TapRooT® Summit to hear Bill Sirois talk about the FACT technique for assessing fatigue during an accident investigation. For a complete Summit schedule see:

http://www.taproot.com/summit.php?sched=1

The following message is from the U.S. Chemical Safety Board, Washington DC

U.S. Chemical Safety Board Investigators Conclude ‘Organizational and Safety Deficiencies at All Levels of the BP Corporation’ Caused March 2005 Texas City Disaster That Killed 15, Injured 180.

Full Board to Weigh Recommendations to OSHA, Oil Industry, BP, and Union to Improve U.S. Refinery Safety at Public Meeting Tonight

Houston, Texas, March 20, 2007 - In a 335-page final report released today, federal investigators from the U.S. Chemical Safety Board (CSB) conclude that ‘organizational and safety deficiencies at all levels of the BP Corporation’ caused the March 23, 2005, explosion at the BP Texas City refinery, the worst industrial accident in the United States since 1990.  The report calls on the U.S. Occupational Safety and Health Administration (OSHA) to increase inspection and enforcement at U.S. oil refineries and chemical plants, and to require these corporations to evaluate the safety impact of mergers, reorganizations, downsizing, and budget cuts.

CSB Chairman Carolyn W. Merritt said, ‘It is my sincere hope and belief that our report and the recent Baker report will establish a new standard of care for corporate boards of directors and CEO’s throughout the world.  Process safety programs to protect the lives of workers and the public deserve the same level of attention, investment, and scrutiny as companies now dedicate to maintaining their financial controls.  The boards of directors of oil and chemical companies should examine every detail of their process safety programs to ensure that no other terrible tragedy like the one at BP occurs.’

The CSB report calls on BP to appoint an additional member of the board of directors with expertise in process safety, and calls for BP senior executives to establish an improved incident reporting program and use new indicators to measure safety performance.

The independent Baker panel, formed and funded by BP in response to an urgent CSB safety recommendation, issued its final report in January 2007.  It found ‘material deficiencies’ in the safety of BP’s five U.S. refineries in Texas, California, Indiana, Ohio, and Washington.  The 11-member panel also issued ten safety recommendations, including calling on BP’s corporate board to closely monitor safety performance at its facilities.  The Baker panel was not charged with determining the root causes of the March 2005 explosion.

CSB Investigation Background

Chairman Merritt said, ‘Our investigation of BP was the largest and most complex undertaking in the agency’s nine-year history.  Under the leadership of Supervisory Investigator Don Holmstrom, the team interviewed 370 witnesses, reviewed more than 30,000 documents, and conducted a far-reaching program of equipment, instrumentation, and chemical testing.’  The final report is scheduled to be presented at a CSB public meeting beginning at 6 p.m. tonight at the Nessler Center, Wings of Heritage Room, located at 2010 5th Avenue North in Texas City.  The report and recommendations are subject to approval by the full Board at the public meeting.

BP cooperated with the investigation, furnished documents and interviews on a voluntary basis, and committed to widespread safety improvements and investments following the accident.  BP published its own report on the explosion in December 2005, pledged the total elimination of the kind of unsafe disposal equipment that led to the explosion, and developed a new siting policy to remove trailers from hazardous process areas.  All 15 fatalities occurred in or near trailers that were sited as close as 121 feet from a blowdown drum that vented flammable liquid and vapor directly to the atmosphere.

Safety Harmed by Cost-Cutting, Production Pressures, and Failure to Invest
(more…)

In an article from the Houston Chronicle, I found the following quotes from a speech at the National Petroleum Refiner’s Association made by Carolyn Merritt, Chairperson of the Chemical Safety Board:

The “ineffective or nonexistent” oversight of safety by the British oil company’s board of directors also played a direct role…

“Somebody has to be asking the question: ‘What is happening, and is this being done?’ “  … Yet those questions were rarely asked, she said.

With this information foreshadowing the report to be released today by CSB, BP’s management should be ready for a report that will be highly critical of their safety management efforts.

The following is quoted from a Toyota web site.

A Toyota Exec is talking about the Toyota Production System.

The article says:

- - - Start of quote - - -

When an error occurs, the first thing that needs to be done is fix the error. Minoura recalls that Ohno used to order them to ask the question “Why?” five times over because “that way you’ll find the root cause, and if you get rid of that it’ll never happen again.” However, Minoura emphasizes that on-the-spot observation rather than deduction is the only correct way to answer a “Why?” question. “I’m always struck that the five-why method doesn’t seem to be working as well as it should be because there’s been a lack of practical training. The reason is that they end up falling back on deduction. Yes, deduction. So when I ask them ‘Why?’ they reel off five causes as quick as a flash by deduction. Then I ask them five whys again for each of the causes they came up with. The result is that they start falling back on deduction again, and so many causes come back that you end up totally confused as to which of them is important.”

“Through real training,” Minoura says, “you’ll be able to discover dozens of problems and also get to their root causes. You’ll be able to make dozens of improvements. If you incorporate all the accumulated knowledge of root causes that you’ve got from always asking ‘Why? Why? Why? …’ into your equipment, you’re going to have something that no one else can come close to. I don’t think it’s got anything to do with nationality; it all has to do with whether or not you’ve received the proper training. I feel though that the tendency to give that kind of training and education forms the basis of Toyota’s approach to monozukuri.”

- - - End of quote from article - - -

Most people that talk to me about 5-Why’s, Cause-and-Effect, or Fault Trees stress the need for deductive reasoning to find root causes. Yet here is someone who worked under Ohno (the 5-Whys expert) who says that deduction is BAD - only direct observation is good.

I like the idea of direct observation. But I think people need more guidance than just asking “Why” 5 times to be able to solve difficult problems. Perhaps 5-Whys works for simple problem. But how do you know if a problem is simple before you solve it?

Maybe 5-Why’s would work for a dedicated master problem solver like Ohno. But it is difficult - or perhaps impossible - to train everyone needed to his level of skill.

That’s why in the TapRooT® System for Root Cause Analysis, we’ve built in an expert system to help people find the root causes of human performance and equipment problems. This helps everyone - from the novice problem solver to the expert - perform better root cause analysis.

And we teach people to use direct observation - the facts - to find root causes using a root cause analysis tool called the Root Cause Tree®.

And we’ve built this whole system into a patented Root Cause Analysis Software - the TapRooT® Software.

What is required to reap the benefits of the investment that we have made in developing TapRooT® into a world class problem solving system? Training for your people.

You will need to train your expert problem solvers in our 5-Day TapRooT® Advanced Root Cause Analysis Team Leader Course.The software is included in the course fee for this course.

You will need to train your everyday problem solvers in our 2-Day TapRooT® Incident Investigation and Root Cause Analysis Course.

And your maintenance and equipment problem solvers need the 3-Day TapRooT®/Equifactor® Equipment Troubleshooting and Root Cause Analysis Course.

What kind of benefits can you achieve after this investment? See the Success Stories on the About TapRooT® page.

Would you like to learn about TapRooT® and hear first hand about what TapRooT® users are doing? Then plan to attend the 2-Day TapRooT® Course prior to the TapRooT® Summit and the TapRooT® Summit.

Your investment in improvement is safe because we guarantee all of these events!

Course Guarantee (last paragraph)

Summit Guarantee (bottom right of page)

What can you and your executive team learn from this press release from Lieff Cabraser Heimann & Bernstein, LLP? Read the release and see…

$55 Million Verdict Imposed Against DaimlerChrysler Corporation For Failing To Fix Known Transmission “Park-to-Reverse” Defect That Killed Young Father At San Pedro/Long Beach Maritime Terminal

– Millions Of DaimlerChrysler Vehicles In Use With Similar Park-to-Reverse Defect

Robert J. Nelson, Scott P. Nealey, and Chuck Naylor, counsel for Adriana Mraz and her three children in a wrongful death action against DaimlerChrysler Corporation, announced that a California-state jury today returned a $50 million punitive damages award against DaimlerChrysler for knowing and intentional failure to cure a defect in millions of its vehicles. On March 2, 2007, the same jury found DaimlerChrysler liable for the death of Richard Mraz and returned a verdict of $5.2 million in compensatory damages for Mrs. Mraz and her children.

On April 13, 2004, Mr. Mraz suffered fatal head injuries when the 1992 Dodge Dakota pickup truck he had been driving at his work site, the San Pedro/Long Beach Maritime Terminal, ran him over after he exited the vehicle believing it was in park. The jury found that a defect in the Dodge Dakota’s automatic transmission, called a park-to-reverse defect, played a substantial factor in Mr. Mraz’s death, and that DaimlerChrysler was negligent in the design of the vehicle, for failing to warn of the defect, and then for failing to adequately recall or retrofit the vehicle.
(more…)

Board to Convene March 20 Public Meeting in Texas City, Texas, to Release and Vote upon Final Report on BP Refinery Disaster

March 7, 2007 - The U.S. Chemical Safety Board (CSB) announced today that it will convene a public meeting on the evening of Tuesday, March 20, 2007, at the Nessler Center in Texas City, Texas, to release its final investigation report on the explosion at the nearby BP refinery that took 15 lives and injured 180 on March 23, 2005.

The meeting will begin at 6 p.m. central time at the Nessler Center’s Wings of Heritage Room, 2010 5th Avenue North, Texas City, TX 77590, (409) 643-5990.  The Nessler Center is adjacent to the Doyle Convention Center, near city hall.
(more…)

… or Defending Categorization
(an excerpt from a draft of the revised TapRooT® Book
available later in 2007, Copyright © 2007)

Some cause-and-effect gurus object to use of the TapRooT® System’s Root Cause Tree® because they feel that any categorization restricts the thinking of an incident investigator. They maintain that the only way to ensure a complete, unbiased, unbounded root cause analysis is to attack each problem from the viewpoint of basic engineering and human performance principles and let the evid