2006, CMP Technology. Important Note: This PDF Is Provided Solely As a Reader Service
Total Page:16
File Type:pdf, Size:1020Kb
Copyright (c) 2006, CMP Technology. Important note: This PDF is provided solely as a reader service. It is not intended for reproduction or public distribu- tion. For more information on obtaining a Reprint, please contact a Reprint Services Rep at 951.698.1780 or visit www.cmpreprints.com/faxback.jhtml. Copyright (c) 2006, CMP Technology. Important note: This PDF is provided solely as a reader service. It is not intended for reproduction or public distribu- tion. For more information on obtaining a Reprint, please contact a Reprint Services Rep at 951.698.1780 or visit www.cmpreprints.com/faxback.jhtml. IT is complex, famously and infamous- ly so. Thousands of variables factor into the success or failure of a high-profile project. Will your career be defined by the data ware- house, delivered on time and under budget, that uncovered a thousand business opportun- ities? Or the ERP fiasco that forced the CFO to put one of those ugly footnotes about an unanticipated supply chain shortfall into the quarterly SEC filing? Here we recount eight tech blunders—costly mistakes for the people involved and lessons for the rest of us. They range from application upgrades gone awry to wholesale strategy shifts that landed in the muck. There’s the publicly traded company that prematurely un- plugged its accounting system, the federal agency whose failed upgrade opened the door to fraudsters, the utility company that plunged much of North America into darkness because of a failed server reboot, and more. What follows are business technology fias- y t 8 BLUNDERS t e G / n e h You Should Never Make c t i K s i n n e D By Paul McDougall informationweek.com Oct. 16, 2006 39 Copyright (c) 2006, CMP Technology. Important note: This PDF is provided solely as a reader service. It is not intended for reproduction or public distribu- tion. For more information on obtaining a Reprint, please contact a Reprint Services Rep at 951.698.1780 or visit www.cmpreprints.com/faxback.jhtml. IN DEPTH / IT FIASCOES coes, some lesser-known, that continue to serve as pated systemwide cost over several years was ex- shining examples of what not to do. pected to be in excess of $1 billion,” the filing says. Among all companies, the billion-dollar IT extrava- ganza has gone the way of the McDLT sandwich: It’s Rule DON’T BITE OFF TOO MUCH more than anyone wants to bite off. #1 cDonald’s Restaurants undertook a proj- Rule ect so grand in scale and scope that, well, #2 TROUBLESHOOT WITH CARE M it couldn’t be done. In 2001, the fast-food chain conceived a project to create an intranet con- ech disasters sometimes ripple far beyond the necting headquarters with far-flung restaurants that company that messed up. At First Energy, an would provide operational information in real time. T Ohio utility that distributes electricity to 4.5 mil- Under the plan, dubbed Innovate, a manager in the lion customers, a software bug contributed to a power company’s Oak Brook, Ill., headquarters would know failure that wiped out service for much of the northeast- instantly if sales were slowing at a franchise in Or- ern United States and parts of Canada three years ago. lando, Fla., or if the grill temperature at a London The blackout was blamed on cascading failures that restaurant wasn’t hot enough. McDonald’s always has occurred when falling trees in Ohio took out power been tight lipped about Innovate—the company didn’t lines and upset the balance of electrical inflows and return calls seeking comment for this story—but outflows in the Northeast grid. But the real failure oc- there’s no doubt about its far-reaching scope. Accord- curred in First Energy’s computer department, where ing to a white paper by Mpower, managers appear to have forgot- the consulting firm McDonald’s ten a fundamental rule of opera- hired for early planning and Nothing tions: Adhere to basic IT man- funny technology procurement, the about a agement protocols, such as those idea was to create “a global ERP $170 specified by the Information application that will eventually million Technology Infrastructure Li- write-off touch every one of McDonald’s brary, regardless of how much stores.” In other words, about tech automation is in place or 30,000 restaurants in more how experienced your staff is. than 120 countries. Piece of Mc- What’s not widely known is Cake, right? that the blackout could have According to Securities and been limited to a local problem if Exchange Commission docu- a software alarm system at First ments filed by McDonald’s, the Energy, designed to warn engi- company realized the project neers about unstable conditions, was over the top only after it hadn’t failed. That’s the thing spent $170 million on consult- about alarms: By the time you ants and initial implementation find out they’re not working, it’s planning. McDonald’s ultimately too late. “It didn’t cause the signaled Innovate’s demise in a power to go off, but it was a big paragraph buried within a 2003 contributor to the lack of re- SEC filing, which noted the write-off and “manage- sponse by First Energy,” says Stan Johnson, manager ment’s decision to terminate a long-term technology of infrastructure security at the North American Elec- project.” tric Reliability Council. The filing revealed what most experienced IT proj- At about 2 p.m. on Aug. 13, 2003, IT staff at the utility ect managers could have told McDonald’s from the discovered that the servers hosting its General Elec- start: An attempt to create a worldwide network deliv- tric XA/21 energy management system’s alarm mod- ering real-time information to thousands of stores, ule had crashed.The staff rebooted them in an effort to some in countries that lacked network infrastructure, fix the problem, but when the servers revived, the was destined to fail. “Although the terminated tech- alarm remained off-line. GE later said in a statement nology project was projected to deliver long-term that a software coding error caused the alarm applica- benefits, it was no longer viewed as the best use tion to go into an infinite loop rather than come back of capital in the current environment, as the antici- online. Control room operators at First Energy facili- 40 Oct. 16, 2006 informationweek.com Copyright (c) 2006, CMP Technology. Important note: This PDF is provided solely as a reader service. It is not intended for reproduction or public distribu- tion. For more information on obtaining a Reprint, please contact a Reprint Services Rep at 951.698.1780 or visit www.cmpreprints.com/faxback.jhtml. IN DEPTH / IT FIASCOES ties in Ohio were unaware that this line of defense was 2002 Earthcare filed kaput, because IT didn’t verify that all systems were for bankruptcy. Shin- back online after the reboot. Meanwhile, unstable skey left the com- power conditions were mounting rapidly. pany that June when Ignoring IT management best practices, staff didn’t Earthcare sold Inter- bother to check with end users to ensure that they had national Petroleum to regained full use of the critical system. Among other U.S. Filter Recovery things, ITIL recommends that IT departments create a Services. call list of affected users that administrators must con- That’s not the end tact following the reboot of a critical system. In its final of this story. After a report on the blackout, the North American Electric year at U.S. Filter, Shinskey: Reliability Council chastised First Energy’s IT depart- Shinskey and his ac- Accountant lost his ment for its lack of communications. counting team were accounting First Energy declined requests for an interview,but moved from Sage system Johnson says the company has complied with recom- Software’s MAS 200 mendations contained in the council’s final report on application to a system called Uptime from Uptime the blackout. Among other things, the council urged Software. “It was so bad we called it downtime,” Shin- First Energy to create precise, written protocols for skey says. The problem was that U.S. Filter was using managing the testing, deployment, and backup of key an outdated version of the software from the early hardware and software systems, including methods 1980s and didn’t want to spend money integrating for performing system upgrades, patch management, Shinskey’s unit into the company’s Oracle environ- rollbacks, and maintenance. It also said the company ment. “There were no drop-down menus, and if you needed to establish well-defined communications didn’t have a million cheat sheets tacked to your com- protocols in the event of a system failure. Sound ad- puter you couldn’t use it,” he says. It was so migraine- vice for all companies. inducing that accountants and bookkeepers left U.S. Filter “in droves” to escape it, Shinskey says. Repre- sentatives from U.S. Filter, now a unit of Siemens, didn’t Rule SWEAT THE DETAILS respond to our inquiries. #3 The lesson here, he says, is that companies must think magine working as an accountant in a company through all the operational implications when business with no accounting system.That’s what happened units are brought together in mergers. That includes I to Dale Shinskey, a finance professional who in having the right tools to support corporate governance. 2000 landed at oil recycling company Earthcare. The company had snapped up International Petroleum, Shinskey’s unit at his previous employer, World Fuel Rule DON’T MARRY FOR MONEY #4 Services, for $35 million.