Fatal distraction: 7 IT mistakes that will get you fired
It’s hard to get a good job in IT these days, but it’s all too easy to lose one.
There are lots of reasons for instant termination. Failure to fulfill your obligation to protect your employer’s digital assets or abusing your vast powers for your own nefarious ends are two sure ways to end up on the unemployment line. You could be fired for opening your mouth at the wrong time or not opening your mouth at the right one. Spying on the boss, lying to your superiors, or being directly responsible for the loss of millions of dollars in downtime through your own negligence are all excellent ways to end up on the chopping block.
Everyone messes up at some point. But some screwups are almost always fatal — to jobs, if not entire careers.
Here are seven true tales of IT pros who screwed up big and got fired quick — even if some were terminated for the right reasons. The names have been withheld to protect the guilty. Don’t let their fatal mistakes become yours.
Fatal IT mistake No. 1: Slacking on backup
It was 10:30 on a Thursday night when Eric Schlissel’s phone rang. On the line was the chief operating officer of a midsize clothing manufacturer with whom Schlissel had never spoken before. The COO, who found his company’s phone number via Google, was frantic. His plant’s ERP system had been wiped out by a virus, and they had a major deadline in the morning.
Schlissel, CEO of managed service provider GeekTek IT Services, hopped in his car and headed down to the L.A. garment district to handle the situation personally.
“Within three minutes of logging in, I realized there was nothing on the server,” says Schlissel. “All the data files were gone, the database was gone, and the ERP software was nowhere to be found. I told him this was no virus. Someone had purged the system.”
It turned out a disgruntled IT contractor had enacted revenge by wiping the garment maker’s servers. But worse news was yet to come. The backups, which were supposed to run every night, hadn’t been working for a very long time. The most recent data Schlissel could find was a year old, making it virtually worthless.
The company only survived because someone in accounting, who did not trust technology, had kept paper copies of everything. It took Schlissel and his team six months to restore all the data by hand.
“It was a $10 or $12 million company, and they probably lost $2 million as a result of this,” he says. “It was the most catastrophic IT disaster I’ve ever seen.”
The factory’s general-purpose IT guy, who was responsible for ensuring backups were made, had simply forgotten about them. He was on the unemployment lines the next day.
Failure to maintain backups is an all too common screwup, and the mistake is often fatal to one’s job security, Schlissel says.
“The first thing we do when we visit a new client is to check the backups,” Schlissel says. “This is a classic IT horror story, one we often tell clients. We’re not trying to scare them, we just want to make sure their assets are protected.”
Moral of the story: A backup in hand is worth two bushels of paper.
Fatal IT mistake No. 2: Snooping on the boss
A few months ago, Oli Thordarson got a call from the CFO of a midsize health care provider in Southern California. As CEO of Alvaka, an advanced network management services company, Thordarson and his staff are often asked to act as a virtual CIO for small businesses and to perform forensic investigations.
The CFO told Thordarson he thought someone was secretly reading his email, and he had a pretty good idea who it was: the director of IT.
The CFO said that, over the past two years, this guy had made comments about things he had no business knowing, says Thordarson. “The running joke was that the director of IT knew more about what was going on inside the company than anybody else,” he says.
Thordarson had one of his techs modify a real-time network probe so that it would send a silent alert if anyone was reading emails they shouldn’t be accessing. Within a few days, Alvaka discovered that the director of IT was indeed reading the CFO’s email — as well as messages from the CEO, the chairman, and other top brass. The next day he was reading the want ads at Monster.com.
This problem is more common than you might think, Thordarson adds. In roughly two-thirds of the companies Alvaka advises, techs have the ability to read any employee’s email, including that of top executives.
“Did they do it to enable support and then forget to undo it, or did they do it because they wanted to snoop?” asks Thordarson. “We don’t really know.”
Moral of the story: A fool and his job are soon parted.
Fatal IT mistake No. 3: Covering up the crime
It was a mistake that could have happened to anyone. The IT staff at a major financial institution needed to replace a disk tray for an older storage array. A staffer called the vendor and had one shipped out. But the junior sales guy at the vendor made a mistake and shipped the wrong tray — one for a newer array that was incompatible with the old one.
The array then failed catastrophically, taking the entire bank’s system offline for nearly a week and costing millions of dollars in lost transactions. That’s when they called in Anthony R. Howard to troubleshoot.
There were three big screwups, says Howard, a best-selling author (“The Invisible Enemy: Black Fox”) and independent technology consultant for Fortune 50 companies and the U.S. military. One, of course, was that the vendor shipped the wrong unit. The second was that the bank’s IT staff tried to install the array itself without waiting for the vendor to send out a qualified technician to do it for them.
The third problem was the big one, though. Almost everyone involved in this screwup lied about it, says Howard. Only one staffer had the courage to admit what really happened.
“When the IT staff saw their jobs were in danger, they began to try to protect themselves and blaming the tech support staff of the vendor,” says Howard. “After the bank’s internal team was done with its investigation and found out that only one person told the truth, he was the only one who managed to keep his job.”
Moral of the story: If the crime doesn’t get you, the cover-up will.
Fatal IT mistake No. 4: The porn identity
Late one evening a couple years back, a network admin for a Fortune 100 firm was looking for an empty backup tape. He grabbed one from the desk drawer of a senior system administrator and popped it into the drive, but was surprised to find it was already full of data. What, he wondered, could be on it? So he looked at it.
You can guess what he found.
“It was filled with porn,” says Dave Amsler, president and CIO of Foreground Security, which was called in to handle the incident. “And so were dozens of other ‘blank’ tapes in the admin’s desk. There was nothing illegal on any of the tapes, thank goodness. Still, he was terminated on the spot.”
Yet that’s hardly the worst Amsler has seen in his 14 years with Foreground, which provides managed security services for major U.S. corporations and government agencies. Amsler says he’s been called in to deal with porn problems for at least 10 clients. Twice he found IT employees running adult sites on company servers. In those cases, the personnel suddenly found themselves with lots more spare time to pursue their hobbies.
Porn filters are useless against this kind of behavior because the IT guys know how to turn them off. Even when organizations have strict policies and filters in place, high-level admins often exempt themselves from these restrictions, says Amsler.
“Sometimes rightfully so,” he adds. “Often high-level admins need to get to sites that would normally be blocked in order to do their jobs. But that doesn’t mean they shouldn’t at least be monitored. Even good people end up doing things they normally wouldn’t when they think no one’s watching. If the admin knows he’s being watched, that would eliminate a significant portion of this behavior.”
Moral of the story: Some things are better done at home.
Fatal IT mistake No. 5: Keeping the wrong secrets
Until recently, Dana B. was a network engineer at a major U.S. Internet provider. One day, a former colleague was told to change the IP addresses on some production routers. Because these changes could impact Internet subscribers, taking them briefly offline, the ISP typically made such changes overnight.
But this engineer didn’t like to stay late, so he changed the addresses at the end of the day before he went home, then turned off his phone so that nobody would disturb him during his off-hours.
That was his first mistake. His bigger mistake was that he consistently refused to document anything he’d done, says Dana. That meant he had no idea which IP addresses he had already used in the past — and neither did anyone else.
After he left, the interfaces failed to come up because their IP addresses had already been used, leaving nearly 5,000 subscribers without Internet access. When other engineers tried to call him to figure out what went wrong, they couldn’t reach him.
“It took a team of five network engineers several hours to find the issue and correct the problems,” says Dana. “The next day he came in and was promptly walked out.”
Moral of the story: Some secrets are better left unkept.
Fatal IT mistake No. 6: Unmitigated disaster
They thought they were ready for anything. An organization in a heavily regulated industry had spent millions building out a comprehensive disaster-recovery plan, including a dedicated fail-over data center humming with hundreds of virtual hosts and a Gigabit Ethernet connection.
But when an unplanned network outage cut the connection to its primary data center, the money the organization spent on its DR solution was for naught.
“The CTO did not have the confidence to activate the disaster-recovery plan, because they had never tested it,” says Michael de la Torre, vice president of recovery services product management for SunGard Availability Services, which was called in by the organization later to shore up its DR strategy. “Instead, he stood by for more than a day hoping the circuit would be repaired. Everyone was offline that entire time. Employees had no access to email or data files, and the organization took a pretty big hit to its reputation.”
Shortly thereafter, the CTO’s career also suffered an unplanned outage.
More than half of all organizations with disaster-recovery plans in place fail to adequately test them, notes de la Torre. Even those that do test uncover an average of five critical errors in the people, process, and tools needed to make DR work.
Disaster recovery is neither glamorous nor easy, but it’s vital to the survival of your company, he adds.
“Protecting the business may never get you promoted. But failing to do so will almost always get you fired.”
Moral of the story: Test your umbrella before the **** hits the fan.
Fatal IT mistake No. 7: Speaking truth to power
Ten years ago, “Bob” was working for a payday loans franchise with more than 1,000 locations nationwide. (Bob asked that his real name be withheld from this story.) He had been hired to rearchitect the chain’s ASP-based system, which was running ancient code on servers in every store. But first he had to prove himself by converting the stores’ dozens of Web-based legal forms into a database.
One Friday afternoon, six months into the job and two weeks before his initial trial was over, the vice president of IT came into the weekly staff meeting to present his five-year vision for the company. The veep’s two-hour speech could be summed up in four bullet points, says Bob:
Stay the course
Fix bugs
Don’t rock the boat
No new technology
“I was floored,” says Bob. “I thought, ‘What about all the stuff they hired me for?’ They were spending millions of dollars a year maintaining creaky sites written by 50 different people.”
Later that afternoon, Bob went into the VP’s office and closed the door.
“He asked me, ‘So what do you think of my vision?'” Bob says. “I said, ‘Frankly, sir, you don’t have one. What you just described was a maintenance plan.'”
The VP thanked him for his candor and complimented him on his courage. The following Monday when Bob returned to his office his key no longer worked. He was gone.
“I drove home whistling,” he says. “I’ve never been so happy to be unemployed. I decided I would never have my career depend on an empty suit ever again. The next day I started my own business, which has kept me busy ever since.”
Moral of the story: Sometimes getting yourself fired is the right thing to do.
Best comptia A+ Training, Comptia A+ Certification at Certkingdom.com