Here at System Improvements, we practice what we preach. For example, we are using the Version 5 TapRooT® Web Enterprise Software to perform audits of our own internal processes: from Accounting, to Course Registration, to Information Technology.
But every once in a while we have to react to an actual incident, and perform our own investigation as well.
On August 13th we had to react to a situation that seemed unusual.
We have a very large group of people who use both the TapRooT® Process and Software, and we are fortunate to have a community that shares their best practices, as well as their problems, with us.
We received data that was outside what we would classify as ‘normal’ for our standard operations (and our upcoming trending release will allow you to determine this as well). We received several comments from customers that their computers were getting infected with a virus known as the ‘Live Security Platinum’ virus.
The Live Security Platinum virus has been around for years and is a particularly nasty virus that likes to pretend it is acting as an Anti-Virus software and protecting your computer, while in fact it is infecting it further with every action that you take. This virus is often observed when you try to click on an internet link or download a file. It presents a warning message that appears to caution you from exposing your computer to further risk, when in reality any selection you make is propagating the virus.
Our initial reaction when we received the first report of this on Friday August 10th, was to assume that it was an isolated incident, and one of our users had simply contracted the virus. However, between August 11th and August 13th, we received 11 reports of the incident. At this point we had to investigate further:
Were all the users from the same company? The answer was no. Several were from the same company, but not the whole group.
Were all the users using the same internet connection at a hotel? The answer was no. Eight of the 11 users were at the same hotel, attending a course.
Were all the users using the same Operating System? The answer was no. Some were using Windows 7, some XP, some Vista.
Were all the users using the same Anti-Virus Software? The answer was no, there were diverse selections.
Were all the users up to date on their Anti-Virus Software Definitions? The answer was no.
Had all the users accessed the TapRooT® website in the past few days? The answer (of course) was yes.
Through this interrogative process, we were able to determine that two key factors were highly correlated with the infection:
1.) The users who were affected were out-of-date with their virus definitions or had no Anti-Virus Software installed
2.) All the users had visited www.taproot.com in some way.
It would have been easy to conclude from this data that finding number 1 was the root cause of the issue. This is where the investigation stops for most.
As far as finding number 2, ALL people who attend TapRooT® courses have visited our website, either to download the software or to register for the course. So why would that be significant?
In fact 4 of the 5 people consulted (the same group who recommend Trident!) on this issue believed that finding number 1 was the explanation for the issue.
But TapRooT® has taught us that if one person says ‘yes’ and the others say ‘no’, you check ‘yes’ and proceed down that path.
And so we investigated further.
Step 1 – Virtual Machines were set up of all known operating systems (XP, Vista, Windows 7)
Step 2 – All Virtual Machines were fit with all known browsers (Internet Explorer, Firefox, Google Chrome)
Step 3 – All combinations of Operating System and browser were tested with the software downloads, and www.taproot.com links
This test reproduced no instances of virus or any issue whatsoever.
The investigation continued.
We performed a scan of all files on our website to determine if any of them had been updated. This test found no changes or vulnerabilities whatsoever.
And so we investigated further.
Having eliminated the machine/browser, as well as the website, we investigated the server. A scan of the server found no changes or vulnerabilities whatsoever.
At this point in the process is when the 11th report of the incident came in. We felt we had eliminated the Machine/Browser possibility since we could not replicate it. We also felt we had eliminated the website and the server. Yet at this point, we cannot ignore that we had just received our 11th report of an issue.
So despite all the evidence to the contrary, we took immediate corrective action and shut down our website. Even though this virus is running rampant on the internet and we can find no correlation to our website, we shut it down (and by doing so closed down our e-commerce store and lowered our potential website hits on our busiest day of the week).
Anyone who knows Mark Paradies knows that this caused him great pain, but he signed the order.
And so we investigated further.
We contacted our server host and asked for an additional virus software installation and scan. We signed a contract and they promised to expedite the process for us.
Here is where we found after nearly 8 hours of investigation:
Causal Factor 1: The scan had returned a result that a remote virtual directory of our website had been compromised.
Corrective Action 1: We quarantined and subsequently removed all the offending files in this directory. As of this writing, we are performing additional scans to detect any other vulnerability, to perform a validation of this Corrective Action.
Causal Factor 2: The remote virtual directory had different permissions than other folders in our website, as it existed long before our new website.
Corrective Action 2: We updated this directory’s permissions to prohibit further compromise.
So after all this, our website is back up and running.
I certainly do not want to appear to be celebrating the fact that our website was hit by a virus. Believe me, I do not. Yesterday was a very long day.
What I do celebrate is that we can work with the 11 affected individuals to make sure we restore them to complete functionality, and that this malicious virus did not affect the thousands who visit our website each day, especially on ‘Newsletter Tuesday’.
I know for a fact that would not be the case if I worked elsewhere. I’m glad we practice what we preach. It made this a small problem and not a larger disaster.
As you may or may not know, System Improvements has really ramped-up our software release schedule in the last 48 months. We are averaging two major releases a year during that time frame, and we have also deployed HotFix releases between our major releases. HotFix releases are not full installations, but minor updates that fix bugs or issues reported to us from our users.
In order for you to get the maximum benefit from our software, you will want to stay current with ALL our updates. For licensed companies, that means maintaining your Annual Maintenance agreement which entitles you to all upgrades and HotFixes. Our licensing department will notify you whenever new upgrades or HotFixes are released.
For people who utilize our Single User software on their laptops, you may already be entitled to FREE upgrades as part of a current Software License agreement, or you may be able to renew your Single User Software License agreement to recieve the upgrades at a discounted rate. Check out www.taproot.com/store.php for more information on the Single User Software License Renewal.
Our most recent software releases are:
TapRooT® Web Enterprise Software Version 184.108.40.206.4 (HotFix 4)
TapRooT® Single User Software Version 220.127.116.11.2 (HotFix 2)
Both allow the ability to upgrade data from the Version 4.0.6 Software!
Contact us at email@example.com for more information.
I have received several requests from our San Antonio Summit attendees for a copy of our ‘What’s New In The TapRooT® Software?” breakout session. So without further ado, here it is…Click Here
The TapRooT® softball team improved to 2-1 Tuesday night with a 13-2 victory. The victory came after the mercy rule was invoked in the 5th inning.
While team TapRooT® has yet to show the explosive offense that was their signature in the spring season, their defense continues to baffle opponents.
The infield played flawlessly on Tuesday turning two double plays to end innings and barely missing a third (Miguel Maria, Ron Verlinde and Jason Sturm turned the plays at shortstop and second base) .
Kori Manes (First Base) made several nice plays on tough throws to help out her fellow infielders.
Yet, the MVP this week had to be Pitcher Jason Hilliard who continued to find the strikezone all game long and refused to give away free base runners with walks.
Back on track, Team TapRooT® looks to start a winning streak next Tuesday.
It was an uncharacteristic night for Team TapRooT® Tuesday as they fell 15-5 by rule of mercy in the 5th inning. Without their regular pitchers and without their normal offensive production, Team TapRooT® had a hard time finding home plate on Defense or Offense.
Walks proved costly as their opponents were spotted at least 10 base runners by inexperienced pitchers. On most nights, the hits and runs would have been there to save Team TapRooT® but every line shot or hard hit ball seemed to find their way into an outfielders glove Tuesday.
It was a tough loss, but the team looks to bounce back next Tuesday with their normal lineup.
The TapRooT® Softball team kicked off their second season in a fashion that resembled their first, winning 17-4. That deficit resulted in the game being called midway through the 5th inning by rule of mercy. The deficit could have been larger if not for the invocation of a rule that is seldom applied.
Team TapRooT® had to replace a few players from the spring season, but this seasons ‘rookies’ provided ample fire power to an already potent lineup. In their debut, Miguel Maria, Graham Reynolds and Ashleigh Malone all contributed big hits, RBI’s and runs.
Miguel achieved the rare accomplishment of going 3 for 4, despite not being thrown or caught out while hitting. After leading the game off with a double, Miguel hit a home run over the right field fence in his second at bat, and hit an INSIDE-the-park home run in his third.
By rule of the league, a team can only record one OUT-of-the-park home run a game. So when Miguel blasted his second home run over the left field fence in his fourth at bat, he was called out by the home plate umpire.
It didn’t diminish his peformance to his teamates though. As one of them put it: “Miguel is apparently the only person who can get Miguel out”.
Team TapRooT® starts 1-0 in the fall season and improves to 8-1 in 2009. They will look to keep the momentum going next Tuesday.
Entering Tuesday night’s season finale, the TapRooT® Softball team had won all 7 of their games, 5 of them by invoking the mercy rule. They had scored no fewer than 15 runs in every game. They had outscored their opponents by a margin of 150-41. Yet all of these statistics were meaningless as they faced another undefeated team, to play for the regular season championship Tuesday.
In the end, a few costly errors and an atypical hitting performance cost TeamTapRooT® the game as they fell13-11 to a formidable opponent. They ended the spring season with a 7-1 record, and an otherwise incredible season. Team TapRooT® will try again for an undefeated season when the fall league starts ups in August. Stay tuned!
The TapRooT® softball team earned a win without playing an inning on Tuesday night as their opponents were unable to field a full roster. The game was recorded as a forfeit, and a 7-0 win for team TapRooT®. The 7-0 win ironically improves the teams record to the same ratio as they enter their 8th and final game of the spring season next week. They will face the only other 7-0 team in the league for the league championship.
The TapRooT® Softball team improved to 6-0 on the year last night, winning 20-5 in a four-inning mercy. The team has won 5 of their 6 games this year by mercy rule, and has scored at least 15 runs in every game played.
Every softball team has great hitters in their lineup; so what is the reason behind team TapRooT®’s run production? According to Ron V., who has played softball for over 30 years, what sets this team apart from the others in the league, is the team’s female contributors:
“Our women consistently hit the ball well which means there are no easy outs in our lineup. Our women also provide power that surprises most of our opponent’s outfielders when the ball sails over their head. Of all the Co-Ed teams I’ve ever played on, this is the best all-around group of women I’ve seen.”
That sentiment is seconded by Jason S., “Nobody can compete with our women. They stretch the outfielders away from the infield and keep them on their heels all game long.”
Lets take a look at this all-star class:
Bert S. (The Star on the Rise) - Bert has been the most improved player on the team this year, as evidenced by her 3 for 3 effort in Tuesday’s victory. In the bottom of the third inning she hit an important line drive to the outfield to help put the game out of reach. The fact that she adds a left-handed bat to the lineup also keeps opponents moving around in the field. Defensively, she acts as the team’s Diplomat to the Umpires when covering home plate at Catcher.
Jessica S. (The Scout) – Anyone who has played next to Jessica in the field (whether she’s playing Right Field or Third Base) knows that she is a true student of the game. Even from the outfield she can tell you what the current hitter did their last time up, and where they are looking to hit it this time. She also brings a live bat to the lineup, as evidenced by her one home run and countless other big hits this season.
Kori M. (The Newbie) – A midseason addition to the team, Kori has made an immediate impact. She has played two positions in her first two games (First Base and Left Field) and its doubtful there is a postion in the field she cannot play. Just ask the baserunner who eased back to first base after an outfield hit in Tuesday’s game. When she fired a cannon-shot to the second baseman, he realized quickly that a double was not in his future. And just for good measure, she added a three run homer to her performance.
Lisa C. (The Veteran) – One of the original members of this team (before it was Team TapRooT®), Lisa has not lost a step at third base, or at the plate. Hitting second or fourth in the lineup, she guarantees a high on base percentage and a lot of RBI’s, year in and year out. Hitting from the left side with a power swing, Lisa has been terrorizing Right Fielders all season long. And if you want to see her flash some leather, just watch a line drive hit at her at third base. Many a batter has sulked back to the dugout, shaking their heads after testing the third base line.
Sarah H. (The Run Producer) - Hitting second in the lineup, Sarah is arguably the best hitter in the league (male or female). After an at bat, she is rarely found anywhere but on base. That is unless she has hit one of her several home runs over a surprised Left Fielder’s head. And once she gets on base she is almost guaranteed to score. Whether it is by hussling for extra bases or tagging up on outfield flies, she always finds her way home. You’ll find her patrolling Left Field when she is not rounding the base paths.
Recent weeks have had the TapRooT® Softball team looking more like an emergency ward than a softball team. From nagging knee, heal and back injuries, to fresh sprains, the team has been struggling to keep their players healthy. The original ten-person lineup has not resembled its opening-day self since, well, opening day.
The team has survived though, thanks in large part to some lineup juggling, and some fortuitous weather. Rainouts have spared the team games without their starting pitcher and their number 3 hitter on seperate occasions. The team also activated Kori M. to their full time roster yesterday.
In earning yesterday’s 23-6 mercy in 5 innings, people contributed all over the field playing away from their normal positions. Jessica S. filled in nicely at third base. Kori M. showed the team what she could do at first base in her debut. Jason S. turned a double play from the second base position.
The hero of the night, however, was Lisa C., who played in the game despite wrestling with a back injury. When it looked like the team would be short a player for the second straight week, Lisa stepped in to hit at the number 10 position and promptly delivered an in-the-park home run to right field. A feat made even more impressive considering the walking pace she was forced to maneuver the bases with.
The team now has a 5-0 record on the year, with 3 games remaining. With a little health, some luck, and continued all-around unselfish play they might be able to win them all.
The System Improvements softball team faced its toughest challenge yesterday but escaped with a 17-14 victory. The game lasted 6 innings, 2 longer than Team TapRooT® had played in any game this year due to mercy rules. Despite playing with only 9 players (instead of the normal 10) , the team was able to post just enough offense and timely defense to earn the win. With the game on the line in the bottom of the 6th inning and time expiring, they were able to retire their opponents 1-2-3 and save their chances at a perfect season.
The TapRooT® softball team earned their third victory of the season Tuesday night, improving their record to 3 wins and 0 losses. The bats stayed hot for team TapRooT® as they won 28-8 after the mercy rule was invoked at the end of the third inning.
The key to their success in the previous two games has been consistent hitting from the top to the bottom of their lineup, allowing them to score runs in bunches. Tuesday night was no different as they batted through the order twice in the top of the first inning. The rally earned them a comfortable 16 run lead, before team TapRooT® had even stepped out onto the field to play defense.
Said veteran Ron V. about this year’s lineup, “One through ten, no other team can match-up with our batting order, which allows us to sustain big inning rallies. Most team’s rallies go for 4 or 5 runs, but with our batting order we are able to rally for 10 or more runs in any inning.”
Major contributions from the bottom half of the batting order this week included: Jason S. (hitting 7th) launched a 3 run homerun over the left field fence; and Jessica S. (hitting 8th) followed up her in-the-park home run last week with another strong hitting performance.
The System Improvements TapRooT® softball team improved their record to 2-0 with a win Tuesday night. The win came after the mercy-rule was invoked after the 4th inning with TapRooT® leading 40-8.
The team started off with a rally in the top half of the first inning, scoring 7 runs. Their opponents responded with 3 runs of their own in the bottom half of the inning. Both teams scored 2 runs each in the second inning and the game entered the third inning with TapRooT® leading by a score of 9-5.
Thats when the bats really came alive for team TapRooT® as the team posted 17 and 14 runs in the 3rd and 4th inning, respectively, earning them their second 4 inning win in as many games.
The System Improvements softball team kicked off its inaugural season Tuesday night with a 15-0 victory. The victory came after the mercy rule was invoked after 4 innings of play.
The performance was a collaborative team effort with notable highlights provided by Sarah H. and Andy C. hitting back to back home runs in the third inning, and Jason H. pitching a shutout (a rare accomplishment in slow-pitch softball).
The shutout was aided by solid defense, with no fielding errors to report. This of course made me VERY happy given that this is a TapRooT® sponsored team, and I am told that all errors may require a full incident investigation.
Am I filling out the roster or drawing a Spring SnapCharT®? You decide.
While that last part might be a joke, it does provoke some interesting questions. While I am sure many of you read this blog for the latest information on how the System Improvements softball team is performing, I am sure MANY MORE of you read it to stay informed and up-to-date on incident investigation and root cause analysis.
Investigators need to keep their brains trained on how to perform investigations even when (hopefully) there are no incidents to investigate! We need to be vigilant and proactive in sharpening our skills and polishing our abilities.
Does that mean that sometimes we view ordinary occurrences outside our workplace through Investigator’s goggles? Yes, it probably does. After all, we have conditioned ourselves to investigate, isolate and ultimately fix problems. Whether it is a car accident, a broken refrigerator, or an error in a softball game, we probably all are guilty of taking our work home with us.
So to illustrate the point, take a look at a hypothetical softball game error (double click on image to view larger)…
The interesting thing here is that we isolated TWO causal factors (problems) in this scenario. An official softball scorer would record this as a throwing error by the shortstop (E6). The official scorer would have been only partially right; the shortstop did commit an error. But how can we fix that? Discipline the shortstop (bench him)? Perform more training (practice)?
No shortstop, at any level of play, is error-proof. Just like any human error, we cannot eliminate it entirely no matter what we try. This is why there should have been a safeguard in place. The Right Center Fielder should have been backing up the throw to second, but was out of position causing the ball to roll to the fence.
Keep in mind that when we look for causal factors we aren’t just looking for direct causes, but also things that can significantly reduce the severity of the incident. In this case, if the Right Center Fielder was in position to back up the Shortstop’s throw, then the baserunner would not have scored (thus reducing the severity of the error and preventing a run from scoring).
Softball, like investigations, requires a team effort and every participant needs to be in position and doing their job on every play. Any good manager (or investigator) will tell you that these safeguards are what can make a good team, a great team.
To obtain more information on TapRooT® Web Enterprise Software Version 5.1.2 BETA used to create the above SnapCharT®, email us at firstname.lastname@example.org
Have a story on how you have used TapRooT® outside of work? Let us know in the comments.
To see how happy a team looks after winning their first game, see below