Root Cause Analysis Blog

How much can a service outage cost? It’s costing Rackspace between $2.5 to $3.5 million because they guarantee reliability and provide credits if their service is out for more than a specified period of time.

With millions on the line, let’s hope they perform an adequate root cause analysis. The information in their preliminary release points to a series of electrical equipment failures. (They should have an Equifactor® Troubleshooting Chart for these.)

Here’s an article about the costs:

http://www.datacenterknowledge.com/archives/2009/07/02/rackspace-expects-credits-of-25-million/

Here’s information about the failure:

http://www.rackspace.com/downloads/pdfs/DFWIncidentReport6-29-2009.pdf

Share on Facebook

See:

http://www.roadtechs.com/nuke/wwwboard/getpost.php?rec_nbr=109957

Share on Facebook

See:

http://careers.northropgrumman.com/ExternalHorizonsWeb/getJobPostDetail.do?sequenceNumber=180651

Share on Facebook

See:

http://www.raib.gov.uk/publications/investigation_reports/reports_2009/report182009.cfm

Share on Facebook

Rackspace, a web hosting company, had a 45 minute outage on June 29th. An Article in Web Host Industry Reviews says that they are going to perform a root cause analysis of the event.

Analyzing the root causes of web outages can be very similar to other outage analyses that people do using TapRooT®.

For example, BellSouth used TapRooT® to review 911 outages, long distance network outages, and local service outages.

Another high reliability computer service provider, Tandem Computers who was later bought by HP, used tapRooT® to analyze network and computer reliability issues/outages.

It really is amazing how even with different technologies, the same proven techniques can be used to find the root causes of human error and equipment failure.

If you would like to learn advanced root cause analysis to analyze service problems, attend a TapRooT® Course. For more information, see:

http://www.taproot.com/courses.php

Share on Facebook

See:

http://www.raib.gov.uk/publications/bulletins/bulletins_2009/bulletin_07_2009.cfm

Share on Facebook

Did a series of computer failures lead to the recent Air France Crash in the Atlantic? here’s an interesting article from the Wall Street Journal:

http://online.wsj.com/article/SB124605948270463623.html

Share on Facebook

Download the PDF at:

http://www.raib.gov.uk/cms_resources/090622_R162009_Deptford%20Bridge.pdf

Share on Facebook

Here’s an article from Abu Dhabi about a second fatal accident involving a child and a school bus. It’s interesting because the writer discusses the hierarchy of Barriers (we call them Safeguards in the TapRooT® System) and how they can be used when developing corrective actions. You usually don’t see this level of understanding of root cause analysis in a newspaper article or letter to the editor.

To read the article, see:

http://www.khaleejtimes.com/DisplayArticleNew.asp?col=&section=opinion&xfile=data/opinion/2009/June/opinion_June135.xml

Share on Facebook

The short article from WTEN in Albany should make you think once again about the hazards involved in using a crane near a power line. See:

http://www.wten.com/Global/story.asp?S=10602635

Share on Facebook

Client located in the state of Pennsylvania has an immediate need for a Steam Turbine Specialist with 5+ years experience inspecting, repairing, maintaining and diagnosing steam turbines, gas turbines, diesel engines, and generators. Needs troubleshooting and root cause analysis skills. See:

http://seeker.dice.com/jobsearch/servlet/JobSearch?op=101&dockey=xml/a/7/a7d5e4b751556f18e97645095ef55699@endecaindex&c=1&source=20

Share on Facebook

A Global Investment Bank located in Jersey City, NJ has an immediate Full Time opportunity for a Voluntary Corporate Actions Specialist with root cause analysis skills. See:

http://www.jobg8.com/JobG8SearchFullView.aspx?aid=JOBG8&jbid=79&jid=746321

Share on Facebook

I’ll post the links here so that people can review the report.

The Executive Summary:

http://www.tva.gov/kingston/rca/Executive%20Summary-for%20VI-062409.pdf

The presentation slides:

http://www.tva.gov/kingston/rca/aecom.pdf

The independent review letter:

http://www.tva.gov/kingston/rca/letter.pdf

The complete report site:

http://www.tva.gov/kingston/rca/index.htm

I reviewed the slide show and the Executive Summary and I couldn’t find anything that I would call “root causes.”

I did see a good failure scenario that would make a good SnapCharT® and then could be used to identify Causal Factors (which are similar to the “Failure Conditions” in the presentation pdf). Their failure conditions were:

• Increased Loads Due to Higher Fill
• Hydraulically Placed Loose Wet Ash
• Fill Geometry & Setbacks
• Inusually Weak Slimes Foundation

But they didn’t analyze these factors to find the root causes behind them and they certainly didn’t look for Generic Causes.

They won’t be reopening this site so this accident won’t be repeated here. But I didn’t come away with lessons that TVA’s Management should be learning to improve their performance.

Am I missing something? Review the materials and see what you think.

Share on Facebook

Nine are now confirmed dead after the Washington Metro train collision.

Two stories now speculate about the failure of circuits to detect the train that was approaching the other train stopped at the station. To read more, see:

http://abcnews.go.com/News/story?id=7927243&page=1

http://www.chron.com/disp/story.mpl/ap/top/all/6498184.html

Share on Facebook

SAFETY ALERT NO. 212:

WORKERS INJURED USING DAMAGED LIFTING EQUIPMENT

Country: CANADA

Release Date: 22 June 2008

Type of Activity: Lifting, crane, rigging, deck operations

Type of Injury: Struck by

Two workers were injured when a lift sling dislodged from a winch hook that did not have a properly operating safety latch.

A service rig crew was in the process of laying down the working floor when they realized that the floor would not be able to come down all the way because a pressure switch was in the way. The operator began to raise the floor so the pressure switch could be turned. As the floor was raised to approximately 45 degrees, the winch hook came unlatched and the floor fell. To save time and speed up the operation, the floorhand and the derrick hand had stepped underneath the floor to pass a 36″ pipe wrench. This placed both individuals directly in the fall path of the floor. Both workers were contacted by the falling floor.

What Went Wrong?

The rig was equipped with two winch lines, the first of which had a damaged Kuplex hook. At the time of the incident the second winch line, which was not used, was fully operational.

The rig crew identified “overhead loads, swing paths, and pinch points” in a safety meeting.

The rig manager was aware that the safety latch on the winch hook was damaged and could not be properly secured. This was verbally communicated to all crew members and documented the hook in a hazard report.

The winch hook was taken out of service for repair after it had been identified as damaged.

A winch hook repair kit had been ordered and was on location at the time of the incident.

After the hook repair kit had been received, the rig manager had attempted to fix the winch hook but was unable to install the new latch locking mechanism.

Believing it would be “OK” for the task of positioning the work floor, the winch hook was returned to service.

When the rig operator began running the winch controls, the floor hand and derrick hand were standing out of the fall path of the rig floor.

At the time of the incident, both workers had moved below the fall zone of the rig floor before the rig floor was in position.

The winch hook was attached to the centre pad and an engineered sling was attached to the work floor. As the floor was raised, the weight of the floor transferred from the center pad sling to the work floor sling. Without the latch lock mechanism, the D-ring pushed out of the winch hook allowing the rig floor to fall.

Corrective actions and Recommendations:

Important hazard identification and control recommendations include:

Never use damaged equipment: Although all members of the rig crew were aware that the winch hook was damaged, the hook was returned to service and used to raise the rig floor.

Always give careful thought about your proper position for the task: In this case, both the derrick hand and the floor hand moved below the rig floor while it was being hoisted by the winch line and were standing directly below an overhead hazard.

Consider the need for design changes before an incident happens: In this case, the company initiated a review the work floor design after the incident and made changes to eliminate the necessity of having the winch hook on the work floor centre pad making lifting operations much safer.

A final thought: Efficiency is important and something we all take a lot of pride in. Always take an extra moment to consider your actions whenever attempting to save time or effort. Make sure your decision is a safe one.

DISCLAIMER:

This Safety Alert is designed to prevent similar incidents by communicating the information at the earliest possible opportunity. Accordingly, the information may change over time. It may be necessary to obtain updates from the source before relying upon the accuracy of the information contained herein. This material is presented for information purposes only. Managers and supervisors should evaluate this information to determine if it can be applied to their own situations and practices.

Source Contact:

This alert is being distributed via a partnership between the International Association of Oil and Gas Producers (http://www.ogp.org.uk/) and Enform (http://www.enform.ca/).

Share on Facebook

CNN reports that at least two people were killed and many were injured in a commuter train crash today.

For the CNN story see:

http://www.cnn.com/2009/US/06/22/washington.subway.crash/index.html#cnnSTCText

For a video report, see:

http://www.cnn.com/2009/US/06/22/washington.subway.crash/index.html#cnnSTCVideo

Share on Facebook

Read this story about the finding of a recent audit report about the TVA ash spill in Tennessee:

http://www.chron.com/disp/story.mpl/nation/6483719.html

In this cases, there was a “Causal Factors” after the “Incident” (circle) on the SnapCharT®.

These type of Causal Factors don’t “cause” the incident, they do “cause” the accident to have worse consequences.

If you would like to learn more about advanced root cause analysis and incident investigation techniques, here are three ideas to consider:

1. Attend a 5-Day TapRooT® Advanced Root Cause Analysis Team Leader Course. Pick from course in:

• Niagara Falls, Canada (July 6-10)
• Melbourne, Australia (July 6-10)
• San Antonio, USA (July 13-17)
• Perth, Australia (July 20-24)
• Seattle, USA (July 27-31)
• Newcastle, Australia (Aug 3-7)
• Brisbane, Australia (Aug 3-7)
• Hong Kong, China (August 31 - September 4)
• Edmonton, Canada (September 7-11)

For complete course information, schedule of courses, and registration, see:

http://www.taproot.com/courses.php?d=2

2. Already attended a TapRooT® Course but want to learn the latest improvements in the techniques? Attend the 2-Day TapRooT® Advanced Techniques Course that is being held on October 5-6 in Nashville, Tennessee. For more information and registration, see:

http://www.taproot.com/summit.php?t=pre-summit#taproot_advanced

3. If you want to learn best practices from around-the-world, attend the Investigation and Root Cause Analysis Track at the TapRooT® Summit on October 7-9 in Nashville, Tennessee. In addition to the great Keynote Speakers, you will also attend the following Best Practices sessions:

• Accident Analysis Presentations (Attack on the USS Stark, Crandall Canyon Mine, 230 Environmental Incidents)
• Success Stories from the Field (common cause analysis of multiple aviation accident investigations and proactive use of Safeguard Analysis for manufacturing quality improvement)
• DISCover How To Communicate After an Accident
• The Good, The Bad, and The Ugly: Which Describes Your Investigations and Reports and What Can You Do To Improve?
• TapRooT® User’s Best Practice Forum
• How To Prove That Fatigue Was the Cause of an Incident
• Advanced Ideas for Defining Causal Factors
• Lessons from Tennessee OSHA Fatality Investigations

For more Summit information or to register, see:

http://www.taproot.com/summit.php

Share on Facebook

See:

http://naombakazi.blogspot.com/2009/06/revenue-cost-assurance-analyst-job.html

Share on Facebook

For more info, see:

http://www.philly.com/philly/wires/ap/news/state/pennsylvania/20090619_ap_ohiomankilledinwesternpadrillingaccident.html

Share on Facebook

Another sad crane accident kills a worker in Quebec. The regulatory authority fines two companies involved a total of $31,080 Canadian Dollars.

A story in The Montreal Gazette provides the following quote from a company President about the fine:

“It’s unjust,” Poirier said. “An accident is an accident.”

You don’t penalize something that could not have been prevented, she added.

Hmmm…

If they can’t figure out how to stop killing people when disassembling a crane, maybe they shouldn’t be allowed to operate cranes?

Maybe it’s time the company President attended a TapRooT® Course? I’d suggest the 5-Day!

Share on Facebook

See:

http://www.computerjobs.com/job_display.aspx?jobid=2482381

Share on Facebook

See:

http://jobview.monster.com/GetJob.aspx?JobID=81572145&from=indeed

Share on Facebook

See:

http://www.hirediversity.com/jobseekers/jobs/view.asp?ID=2517750

Share on Facebook

Here’s an article that follows up on a previous post here at the Root Cause Analysis Blog.

The WSBTV story reports on the results of an OSHA investigation that does not include a root cause analysis.

Click here to see the OSHA report.

Share on Facebook

See:

https://www4.recruitingcenter.net/Clients/BNSF/PublicJobs/controller.cfm?jbaction=JobProfile&Job_Id=23185&esid=az

Share on Facebook

See:

http://www.hirediversity.com/jobseekers/jobs/view.asp?ID=2518605

Share on Facebook

Takes a problem situation and works to find the root causes. For more information, see:

https://sjobs.brassring.com/en/ASP/TG/cim_jobdetail.asp?SID=&jobId=417985&type=search&JobReqLang=1&recordstart=1&JobSiteId=5100&JobSiteInfo=417985_5100&GQId=0&partnerid=16015&siteid=5100&codes=IND

Share on Facebook

See:

http://jobview.monster.com/GetJob.aspx?JobID=81565867&from=indeed

Share on Facebook

ConocoPhillips, a TapRooT® User, has an opening for an Industrial Hygiene/Safety Rep in Bartlesville, OK. For more info, see:

http://www.hirediversity.com/jobseekers/jobs/view.asp?ID=2518032

Share on Facebook

See:

https://jobs-bannerhealth.icims.com/jobs/62970/job?sn=Indeed

Share on Facebook

See:

http://www.energyplacement.com/jobs/chemicals_petrochemical_jobs?id=25198

Share on Facebook

See:

https://tbe.taleo.net/NA1/ats/careers/requisition.jsp?org=CHAMPTECH&cws=1&rid=1044

Share on Facebook

An Article in the The Pratt Tribune says that cracks on wind farm turbine blades in Flat Ridge, Kansas, are due to a “manufacturing defect” and a “quality control defect in the manufacturing process.”

The problem was discovered during the root cause analysis of cracks at another wind farm. Since the same manufacturing process was used for the blades at the Flat Ridge site, the manufacturer has decided to replace the blades under warranty rather than waiting to see which ones start to crack.

First, if I am a manufacturer, a manufacturing defect is not a root cause. It is a causal factor that needs to be investigated. Since they have found out how the cracks occur (”During the manufacturing process numerous layers of laminate are used to make the fiberglass blades. During that process little folds develop that build up in a very defined location on the blades that will eventually lead to small cracks in the blades and cause pealing.”).

The “little folds developing” needs to be examined. It is probably the causal factor that needs to be examined to find it’s root causes.

Next, should these “little folds” have been caught in the design/manufacturing testing process?

Finally, how long have reports of these “little cracks” been coming in?

Because these cracks seem to occur across the manufacturing process (all the blades seem to have them), the root causes will be Generic Causes that apply to all blades manufactured. But are the problems with the design/manufacturing/testing process also present in other parts of the manufacturing process (not just the “little folds” on these blades)? That is a much bigger generic cause question.

What do you do to find the real fixable causes of manufacturing problems?

Do you look beyond the immediate causes to find root causes and then probe further to see if there are Generic Causes?

If you would like to learn a process that is used by industry leaders around the world, see:

http://www.taproot.com/about.php

Share on Facebook

From Yahoo’s space news section, “a 14-year old German boy was hit in the hand by a pea-sized meteorite that scared the bejeezus out of him and left a scar.”

“When it hit me it knocked me flying and then was still going fast enough to bury itself into the road,” the boy said.

Now as a TapRooT® Root Cause Analysis Instructor and incident investigation facilitator, I do not want to ever hear our clients say this incident was just bad luck. It was a one of a kind occurrence that never happened before. In reality it would be pretty hard to substantiate those types of findings.

In this case however it appears to be pretty much bad luck but it has happened a few times before. Now I don’t plan on issuing a “Chicken Little” the sky is falling report but I will look up every now and then.
(more…)

Share on Facebook

Washington, DC, June 9, 2009— The U.S. Chemical Safety Board (CSB) today sent an investigation team to the site of an explosion, fire and roof collapse at the ConAgra Foods plant in Garner, North Carolina.

According to media reports, at least 20 people have been transported to area hospitals and at least two were reported as unaccounted for following the explosion which occurred shortly before 11:30 a.m. today. The explosion caused the roof to collapse on one side of the building, knocked down walls, and blew debris about the area.

The CSB team will be headed by Investigations Supervisor Don Holmstrom. Board Member William Wark will accompany the team.

The CSB is an independent federal agency charged with investigating industrial chemical accidents. The agency’s board members are appointed by the president and confirmed by the Senate. CSB investigations look into all aspects of chemical accidents, including physical causes such as equipment failure as well as inadequacies in regulations, industry standards, and safety management systems.

The Board does not issue citations or fines but does make safety recommendations to plants, industry organizations, labor groups, and regulatory agencies such as OSHA and EPA. Visit our website, www.csb.gov.

For more information, contact Public Affairs Specialist Hillary Cohen (on location) cell 202.446.8094; or Director of Public Affairs Dr. Daniel Horowitz, 202-261-7613, cell 202-441-6074.

Share on Facebook

Some industrial projects are just amazing.

The picture above is the Saipem 7000. (Note that Saipem is a TapRooT® User.)

Watch the video at this link:

http://www.medgaz.com/medgaz/pages/archivos_multimedia-eng.htm

Share on Facebook

Entering Tuesday night’s season finale, the TapRooT® Softball team had won all 7 of their games, 5 of them by invoking the mercy rule.  They had scored no fewer than 15 runs in every game.    They had outscored their opponents by a margin of  150-41.  Yet all of these statistics were meaningless as they faced another undefeated team, to play for the regular season championship Tuesday.

In the end, a few costly errors and an atypical hitting performance cost TeamTapRooT® the game as they fell13-11 to a formidable opponent.  They ended the spring season with a 7-1 record, and an otherwise incredible season.  Team TapRooT® will try again for an undefeated season when the fall league starts ups in August.  Stay tuned!

Share on Facebook

The Associated Press came out with an article today about beefed up Inspections of Regional Airline Pilot Training. This increased activity is in REACTION to pilot errors listed following the New York Regional Jet Crash earlier this year. The good PROACTIVE note in this article was this comment, “Federal Aviation Administrator Randy Babbitt said in a statement they will also hold a meeting with the airline industry — both regional and major carriers — next week to seek better pilot training, cockpit discipline and other safety improvements.”

So what would you look at to determine the Training concerns:

1. “a series of critical errors by the captain and co-pilot preceded the crash of Continental Express Flight 3407 as it neared Buffalo Niagara International Airport on Feb. 12.”

2. “cockpit voice recorder showed the co-pilot describing her lack of experience flying in icy weather not long before the crash.” Did she miss training? Did the company decide not to train? Was she trained and testing needs improvement? Should there be continued training? Did practice and repetition need to be increased?

3. “captain may not have had hands-on training on a critical cockpit safety system. “Did he miss training? Did the company decide not to train? Was he trained and testing needs improvement? Should there be continued training? Was the task analyzed for this aircraft?

Then there was the Fatigue issue:

1. “co-pilot, Rebecca Shaw, lived near Seattle on the West coast with her parents and had commuted all-night to get to Newark Liberty International Airport in New Jersey on the East coast , where Flight 3407 originated”

2. “captain, Marvin Renslow, commuted to work from his home in Florida. It is not clear where either of them slept the night before the crash or how much sleep they received.”

3. “current rest rules “are less restrictive than truck drivers work under. Once you’ve been on duty for 13 hours, you are about 500 percent more likely to make an error, and once you’ve been on duty for 16 hours, you have the response rate of somebody who is legally drunk.”

Is the policy confusion or incomplete? Is the policy not strict enough? Does the communication of the Policy need improved? Is this a crew selection issue? A scheduling issue?

Now if you read the rest of the article linked below, you then have to ask about Oversight and Corrective Actions. There were “cracks” in the system? Don’t get me wrong, there are a lot of good aviation programs with good training….. maybe too many for the ratio of inspectors? What is the expectation of the controlling authority? How much is the push back from the private commercial sector when push comes to shove. After all, look at the discussion over the recent airbus accident and whether the pitot sensors needed to be replaced. According to reports, Air Bus highly “suggested” that they be replaced… and the air lines knowing that pitot sensor errors are only minor, instructed pilots to make sure they beefed their work around response for eradicate readings.

This is more than just a training issue. What questions do you have? Wonder where my questions come from? Come to one of our 5-day incident courses and walk through the early 1970 Florida aircraft crash.

5-Day TapRooT® Advanced Root Cause Analysis Team Leader Training: http://www.taproot.com/courses.php#c1

Or even better, come to our Summit and Pre-Summit and talk with aviation industry experts trained in our process from Rotorcraft to Alaska Airlines. http://www.taproot.com/summit.php
(more…)

Share on Facebook

Update: Missile was NOT LIVE….. does this now condone the actions taken? No, all weapons are considered live until proven otherwise. The questions below still stand.

Here are some questions following a live catch of a missile while fishing.

1. What kind of bait do you use for missiles … little fishing boats?

2. What do you do if you catch a LIVE MISSILE that is corroded and unstable?…. I know, keep it on your boat for ten days and then turn it in to the authorities.

3. What do you do if a live missile gets loose and runs away? …. I might need a little help on this one.

Wow, I don’t know the charge in the missile, but this could have turned out really bad. The question is what type of investigation would this need to keep it from happening again. After all, no property damage or injury… it might just get an “Apparent Cause” analysis. That works, right?
(more…)

Share on Facebook

See:

http://www.naukrigulf.com/ni/nijobsearch/080609000026::rss_source::::::::::

Share on Facebook

http://www.roadtechs.com/const/wwwboard/getpost.php?rec_nbr=108372

Share on Facebook

The TapRooT® softball team earned a win without playing an inning on Tuesday night as their opponents were unable to field a full roster.  The game was recorded as a forfeit, and a 7-0 win for team TapRooT®.  The 7-0 win ironically improves the teams record to the same ratio as they enter their 8th and final game of the spring season next week.  They will face the only other 7-0 team in the league for the league championship.

Share on Facebook

This title got my attention, “Driver ‘dozes off’ on India train.” Problem is that I did not expect to read these next lines from the article:

When some of the passengers managed to reach the engine, they apparently found the driver snoring away.

Now I understand the role of damage control and the need to investigate this incident. What I don’t agree with is how the Authority handled the public and media response. Either there was a medical condition, he was asleep, or this was inattentiveness. Regardless of the final investigation report, this should be considered a Near Miss. (more…)

Share on Facebook

Checking on the news in the town where my daughter lives I saw this article at www.kake.com, “Sno-Cone Mishap Leaves Sedgwick Co. Zoo Visitors Ill.” Turns out that the employees mistakingly used the dark colored degreaser instead of the dark colored blue sno-cone flavoring.

Here are the highlights from the article (see the link in read more to see the video):

Luckily, the bottle mix-up did not end up with any major injuries. Those involved did not ingest enough of the mixture to cause any harm. Still, they are not happy.

“They need to be more careful, especially serving food. Pay attention to what you’re grabbing because this could have turned out tragic,” the victim said.

The zoo says it will now stop serving blue sno-cones completely to make sure this kind of mix up never happens again.
Now do you think that the zoo officials have truly found the causal factors, identified the root causes, and found the failed safeguards? I am surprised they were not fined for having cleaners stored with food products.. that would be a failed safeguard and root cause of arrangement and placement… but everyone seemed to focus on the mistake made by the sno-cone machine attendants.

What about the other sno-cone machines and food service areas, this may be a generic issue. Why was the issue not caught with proactive audits? If you want to reduce the possibility of this type of incident in the food industry come to our TapRooT® Summit in Nashville this October 5-6 (for the Pre-Summit) and October 7-9 (for the Summit). Found out about our proactive risk assessments and industry best practice tracks at this link: http://www.taproot.com/summit.php
(more…)

Share on Facebook

Gettyburg, PA - RRI Energy - Plant Engineer

McGaw Park, IL - Baxter - Principal Engineer, Renal Medical Devices

Tukwila, WA - Boeing - Quality Production Specialist

Good luck with your job hunt!

Share on Facebook