Copyright (c) 2006, CMP Technology. Important note: This PDF is provided solely as a reader service. It is not intended for reproduction or public distribu- tion. For more information on obtaining a Reprint, please contact a Reprint Services Rep at 951.698.1780 or visit www.cmpreprints.com/faxback.jhtml. Copyright (c) 2006, CMP Technology. Important note: This PDF is provided solely as a reader service. It is not intended for reproduction or public distribu- tion. For more information on obtaining a Reprint, please contact a Reprint Services Rep at 951.698.1780 or visit www.cmpreprints.com/faxback.jhtml.

IT is complex, famously and infamous- ly so. Thousands of variables factor into the success or failure of a high-profile project. Will your career be defined by the data ware- house, delivered on time and under budget, that uncovered a thousand business opportun- ities? Or the ERP fiasco that forced the CFO to put one of those ugly footnotes about an unanticipated supply chain shortfall into the quarterly SEC filing? Here we recount eight tech blunders—costly mistakes for the people involved and lessons for the rest of us. They range from application upgrades gone awry to wholesale strategy shifts that landed in the muck. There’s the publicly traded company that prematurely un- plugged its accounting system, the federal agency whose failed upgrade opened the door to fraudsters, the utility company that plunged much of North America into darkness because of a failed server reboot, and more. What follows are business technology fias- y t 8 BLUNDERS t e G / n e

h You Should Never Make c t i K s i n n e

D By Paul McDougall

informationweek.com Oct. 16, 2006 39 Copyright (c) 2006, CMP Technology. Important note: This PDF is provided solely as a reader service. It is not intended for reproduction or public distribu- tion. For more information on obtaining a Reprint, please contact a Reprint Services Rep at 951.698.1780 or visit www.cmpreprints.com/faxback.jhtml. IN DEPTH / IT FIASCOES

coes, some lesser-known, that continue to serve as pated systemwide cost over several years was ex- shining examples of what not to do. pected to be in excess of $1 billion,” the filing says. Among all companies, the billion-dollar IT extrava- ganza has gone the way of the McDLT sandwich: It’s Rule DON’T BITE OFF TOO MUCH more than anyone wants to bite off. #1

cDonald’s Restaurants undertook a proj- Rule ect so grand in scale and scope that, well, #2 TROUBLESHOOT WITH CARE M it couldn’t be done. In 2001, the fast-food chain conceived a project to create an intranet con- ech disasters sometimes ripple far beyond the necting headquarters with far-flung restaurants that company that messed up. At First Energy, an would provide operational information in real time. T Ohio utility that distributes electricity to 4.5 mil- Under the plan, dubbed Innovate, a manager in the lion customers, a software bug contributed to a power company’s Oak Brook, Ill., headquarters would know failure that wiped out service for much of the northeast- instantly if sales were slowing at a franchise in Or- ern United States and parts of Canada three years ago. lando, Fla., or if the grill temperature at a London The blackout was blamed on cascading failures that restaurant wasn’t hot enough. McDonald’s always has occurred when falling trees in Ohio took out power been tight lipped about Innovate—the company didn’t lines and upset the balance of electrical inflows and return calls seeking comment for this story—but outflows in the Northeast grid. But the real failure oc- there’s no doubt about its far-reaching scope. Accord- curred in First Energy’s department, where ing to a white paper by Mpower, managers appear to have forgot- the consulting firm McDonald’s ten a fundamental rule of opera- hired for early planning and Nothing tions: Adhere to basic IT man- funny technology procurement, the about a agement protocols, such as those idea was to create “a global ERP $170 specified by the Information application that will eventually million Technology Infrastructure Li- write-off touch every one of McDonald’s brary, regardless of how much stores.” In other words, about tech automation is in place or 30,000 restaurants in more how experienced your staff is. than 120 countries. Piece of Mc- What’s not widely known is Cake, right? that the blackout could have According to Securities and been limited to a local problem if Exchange Commission docu- a software alarm system at First ments filed by McDonald’s, the Energy, designed to warn engi- company realized the project neers about unstable conditions, was over the top only after it hadn’t failed. That’s the thing spent $170 million on consult- about alarms: By the time you ants and initial implementation find out they’re not working, it’s planning. McDonald’s ultimately too late. “It didn’t cause the signaled Innovate’s demise in a power to go off, but it was a big paragraph buried within a 2003 contributor to the lack of re- SEC filing, which noted the write-off and “manage- sponse by First Energy,” says Stan Johnson, manager ment’s decision to terminate a long-term technology of infrastructure security at the North American Elec- project.” tric Reliability Council. The filing revealed what most experienced IT proj- At about 2 p.m. on Aug. 13, 2003, IT staff at the utility ect managers could have told McDonald’s from the discovered that the servers hosting its General Elec- start: An attempt to create a worldwide network deliv- tric XA/21 energy management system’s alarm mod- ering real-time information to thousands of stores, ule had crashed.The staff rebooted them in an effort to some in countries that lacked network infrastructure, fix the problem, but when the servers revived, the was destined to fail. “Although the terminated tech- alarm remained off-line. GE later said in a statement nology project was projected to deliver long-term that a software coding error caused the alarm applica- benefits, it was no longer viewed as the best use tion to go into an infinite loop rather than come back of capital in the current environment, as the antici- online. Control room operators at First Energy facili-

40 Oct. 16, 2006 informationweek.com Copyright (c) 2006, CMP Technology. Important note: This PDF is provided solely as a reader service. It is not intended for reproduction or public distribu- tion. For more information on obtaining a Reprint, please contact a Reprint Services Rep at 951.698.1780 or visit www.cmpreprints.com/faxback.jhtml. IN DEPTH / IT FIASCOES

ties in Ohio were unaware that this line of defense was 2002 Earthcare filed kaput, because IT didn’t verify that all systems were for bankruptcy. Shin- back online after the reboot. Meanwhile, unstable skey left the com- power conditions were mounting rapidly. pany that June when Ignoring IT management best practices, staff didn’t Earthcare sold Inter- bother to check with end users to ensure that they had national Petroleum to regained full use of the critical system. Among other U.S. Filter Recovery things, ITIL recommends that IT departments create a Services. call list of affected users that administrators must con- That’s not the end tact following the reboot of a critical system. In its final of this story. After a report on the blackout, the North American Electric year at U.S. Filter, Shinskey: Reliability Council chastised First Energy’s IT depart- Shinskey and his ac- Accountant lost his ment for its lack of communications. counting team were accounting First Energy declined requests for an interview,but moved from Sage system Johnson says the company has complied with recom- Software’s MAS 200 mendations contained in the council’s final report on application to a system called Uptime from Uptime the blackout. Among other things, the council urged Software. “It was so bad we called it downtime,” Shin- First Energy to create precise, written protocols for skey says. The problem was that U.S. Filter was using managing the testing, deployment, and backup of key an outdated version of the software from the early hardware and software systems, including methods 1980s and didn’t want to spend money integrating for performing system upgrades, patch management, Shinskey’s unit into the company’s Oracle environ- rollbacks, and maintenance. It also said the company ment. “There were no drop-down menus, and if you needed to establish well-defined communications didn’t have a million cheat sheets tacked to your com- protocols in the event of a system failure. Sound ad- puter you couldn’t use it,” he says. It was so migraine- vice for all companies. inducing that accountants and bookkeepers left U.S. Filter “in droves” to escape it, Shinskey says. Repre- sentatives from U.S. Filter, now a unit of Siemens, didn’t Rule SWEAT THE DETAILS respond to our inquiries. #3 The lesson here, he says, is that companies must think magine working as an accountant in a company through all the operational implications when business with no accounting system.That’s what happened units are brought together in mergers. That includes I to Dale Shinskey, a finance professional who in having the right tools to support corporate governance. 2000 landed at oil recycling company Earthcare. The company had snapped up International Petroleum, Shinskey’s unit at his previous employer, World Fuel Rule DON’T MARRY FOR MONEY #4 Services, for $35 million. Unfortunately,Shinskey says, “Earthcare was making acquisitions, but they weren’t ome try to avoid IT foul-ups by sending work paying attention to the details.” over the wall to an outsourcer. One major insur- Under the deal,World Fuel Services was to maintain S ance and financial services company became so the Oracle accounting system that Shinskey and his enamored of the concept that it bought its own out- colleagues used to keep the books for their unit. But a sourcing company.That decision, however, turned into problem surfaced: Shinskey maintains that Earthcare a huge case of buyer’s remorse. forgot to pay World Fuel Services for the system’s up- In the span of 15 months between 2001 and 2002, In- keep, and after a short time it was switched off. “For dianapolis-based Conseco bought India-based out- three months we didn’t have anything,” Shinskey says. sourcer ExlService.com—and then turned around and During that time, Earthcare simply estimated financial sold the company it had just acquired. ExlService was a performance for the group, ballpark figures that ulti- fledgling outfit that provided customer service func- mately were never justified. “They tried to true it all up, tions from low-cost centers in India. It was a tantalizing but they were never able to,” he says. morsel for Conseco’s CEO at the time, Gary Wendt, who Sloppy bookkeeping may have reflected larger prob- had risen through the ranks at pioneer outsourcing lems at Earthcare. In 2001 it was forced to pay $1.75 user General Electric. Under Wendt, Conseco acquired million to World Fuel to settle disputes related to the ExlService for $52.1 million. acquisition of International Petroleum, and in April Wendt thought ExlService would provide a way

42 Oct. 16, 2006 informationweek.com Copyright (c) 2006, CMP Technology. Important note: This PDF is provided solely as a reader service. It is not intended for reproduction or public distribu- tion. For more information on obtaining a Reprint, please contact a Reprint Services Rep at 951.698.1780 or visit www.cmpreprints.com/faxback.jhtml. IN DEPTH / IT FIASCOES

for Conseco to cut costs and improve customer ser- RuleRule SMORGASBORDS CAUSE INDIGESTION #5 vice in one swoop. He didn’t have much faith in the homegrown talent available in Indiana. “I’m he United Kingdom’s National Health Service convinced there’s better customer service in India. IT modernization program may be one of the It’s no good here,” he told the Indianapolis Star in T great IT disasters in the making. It’s two years response to a question about why he was bent behind schedule and $10 billion over budget, giving on moving the company’s customer service opera- Boston’s Big Dig a run for its money. One of the main tions abroad. contractors, health care applications maker iSoft, is on That’s one explanation, but Wendt also stood to gain the verge of going belly up because of losses incurred something more from the deal; he was a co-founder of deploying its software across the network.The company ExlService and owned a 20% stake in it at the time of can’t book revenue until its applications are actually de- the transaction. Wendt and his wife netted 692,567 ployed, so delays are killing its bottom line. The initia- Conseco shares from the deal, worth about $9.7 mil- tive has been plagued by technical problems resulting from attempts to weld incompatible systems, resistance from physicians who say they were never adequately consulted about program features, and squabbling among contractors about who’s responsible for what. IT executives and government ministers in charge of the program have been bumbling along for the past couple of years trying to get the project on track. But serious problems continue. A massive computer out- age this year disrupted operations for four days at 80 health care facilities in England’s North West and North Midlands regions. The fault originated in a server that stores patient records and other health care data on millions of Britons under the NHS’s care. For days, doctors in affected regions were unable to access appointment records, creating significant de- lays in patient care. In a statement, the NHS said pa- tient safety was not jeopardized. The broader problem, says one Cambridge Univer- sity professor who has studied the NHS overhaul, is that government officials divvied up A failed system alarm left more than just IT in the dark the modernization project among too many vendors that aren’t particularly good at cooperating with one lion, according to SEC documents.The shares were re- another. “It’s different software, different standards, stricted until Conseco realized a positive return on in- different everything,” says professor Ross Anderson, vestment from the transaction. Conseco didn’t return who leads the Foundation for Information Policy Re- calls requesting comment. search, a think tank that has been critical of the mod- Conseco shifted more than 2,000 customer service ernization plan. NHS officials have conceded that the jobs to ExlService in India after the purchase. In a reg- program has had problems, but they’ve insisted that ulatory filing, Conseco said it lost $20 million as a result improved patient care will be the ultimate result. of divesting ExlService in November 2002. Wendt’s In many cases, Anderson says, systems installed shares were voided. aren’t compatible. “This isn’t just a matter of wasting Some Conseco execs blamed the venture’s failure billions [of pounds]; it could cost lives,” he warns. The on bad timing, according to the Indianapolis Star. systems may become more compatible by default. One Conseco’s India-based call center went live on Sept. integrator working on the project, Accenture, last month 10, 2001. After the next day’s terrorist attacks, call dropped out, handing its share of the contract to Com- center agents with foreign accents had an immediate puter Sciences Corp. Accenture has set aside $450 mil- disadvantage. lion to cover losses from the project. What’s the takeaway? Beware the pitfalls of outsourc- The trend in big outsourcing projects is to split work ing? Choose your partners carefully and for the right among multiple vendors.The idea is to reduce risk and reasons? Never underestimate Indiana? All of the above. instill competition among the contractors. There are

44 Oct. 16, 2006 informationweek.com Copyright (c) 2006, CMP Technology. Important note: This PDF is provided solely as a reader service. It is not intended for reproduction or public distribu- tion. For more information on obtaining a Reprint, please contact a Reprint Services Rep at 951.698.1780 or visit www.cmpreprints.com/faxback.jhtml. IN DEPTH / IT FIASCOES

more than a dozen vendors working on the NHS modernization, including subcontrac- tors. It’s a technology Tower of Babel. Risk mitigation is important, but there are down- sides to multivendor arrangements, too.

Rule MURPHY’S LAW APPLIES #6 ot everyone hates IT miscues. Tax cheats, for instance, are thrilled N about them when the organization If anything could go at the center of the screw-ups is the Internal wrong with the IRS’s Revenue Service. That was the case earlier fraud-detection system, it did, Begg says this year when the IRS botched an upgrade to its fraud-detection software. “If anything could go wrong, it went wrong regarding this particular of the system were affected by the change. As a result, project,” says Margaret Begg, assistant inspector gen- the IRS’s System Acceptability Testing team would en- eral for audit with the Treasury Department. counter another problem in another place. This would The IRS planned to launch the system in January, in cause the SAT team to write another problem ticket, time for the 2006 tax season and one year after the which would go back to the contractor, requiring an- original implementation date. In anticipation, the tax other software fix, according to a report by Begg. In an agency’s IT staffers pulled the plug on the old system. interview, Begg says the IRS lacked the project man- The new version, which is being developed and imple- agement discipline to pull off an upgrade of such a crit- mented by Computer Sciences Corp. and is fed by a ical application. “They lacked testing rigor, and they that holds more information than any other didn’t have program management activities set up IRS system except its master database, didn’t work. Not where there’s a defined plan and that they could exe- good for a piece of software that’s been on the drawing cute against,” she says. board since 2001. Now, the Treasury Department’s Of- This, of course, isn’t the first big IT snafu for the IRS. fice of Inspector General estimates that the lack of a The agency has been trying to upgrade its overall com- functioning anti-fraud system cost $318 million in puting infrastructure for the past eight years, at a cost fraudulent refunds that didn’t get caught. of more than $8 billion. A Government Accountability A four-month investigation by the House Ways and Office review of that effort found multiple instances of Means Committee found “incompetence at all levels,” cost overruns, waste, and inefficiency.Part of the prob- committee chairman Bill Thomas, R-Calif., wrote in an lem is that IRS IT personnel are retiring in droves and August letter to Treasury Secretary Henry Paulson. qualified replacements are tough to find. “They’ve Among other things, IRS IT leaders erred when they lacked the stability and consistency you need to imple- classified the effort as a maintenance project rather ment certain activities,” says Begg, adding that IT man- than a major upgrade.That led to insufficient oversight agement turnover has been particularly high. The IRS and funding, Thomas says, “despite the critical impor- is in the process of booting up the old system in time tance of the system … in protecting federal revenue.” for the 2007 tax season, while plans for the Web-based Not to mention that the system is one of 19 IRS assets system have been shelved. Brace yourselves. the federal government has deemed part of the na- tion’s critical infrastructure that must be protected from terrorist attacks. Rule CHECK THE SHELF LIFE #7 In January,when the system was supposed to go live, users generated 534 problem tickets. One problem was n 2005, the Federal Bureau of Investigation that the electronic fraud-detection system project of- killed off its problem-plagued Virtual Case File fice wasn’t communicating with the Criminal Investi- I system, a custom-developed software applica- gation division of the IRS, the main user of the system. tion that was supposed to let agents search across In fact, communication was breaking down all over the multiple criminal in the hunt for perps. l a e D place. When one development team made a change, it That capability has been a Holy Grail for the bureau d i v a didn’t notify other development teams whose portions as far back as the 1980s, when software company In- D

46 Oct. 16, 2006 informationweek.com Copyright (c) 2006, CMP Technology. Important note: This PDF is provided solely as a reader service. It is not intended for reproduction or public distribu- tion. For more information on obtaining a Reprint, please contact a Reprint Services Rep at 951.698.1780 or visit www.cmpreprints.com/faxback.jhtml.

slaw alleged that the Department of of the original contractors on the Justice purloined Inslaw’s Promis Department of Justice’s efforts to case management software for use implement Promis. Let’s hope by the FBI. Inslaw ultimately lost things go better this time around. the case. The need for such a sys- tem took on new urgency after the 9/11 terrorist attacks. Rule BEWARE THE RUSH JOB The failure of the $170 million Vir- #8 tual Case File effort, part of the FBI’s n the mid-’90s, tech execs at Trilogy IT modernization program, Nielsen Media Research, the likely won’t result in any criminal I provider of viewer statistics to charges, but in launching the program the television industry, learned the the FBI did break one unspoken law hard way about the perils of rush- of IT planning: Don’t build a long- ing a critical project. The company term project on technology that will be outdated by the wanted a major rewrite of its core ratings system. It time the project is completed. Project planners must wasn’t a matter of altering some peripheral piece of think about what the overall tech environment will look back-office-ware, but of overhauling the main engine like, and what technology will be widely used, once an behind Nielsen’s billion-dollar ratings business. Un- initiative is completed. fortunately,IT execs accelerated that conversion. As a In planning the Virtual Case File in 2001, the FBI result, the coding was botched, millions of dollars were chose custom-developed software, although cheaper, wasted, a lawsuit was filed against the lead vendor, more flexible off-the-shelf apps were coming into fa- and now, a decade later, Nielsen is still trying to com- vor among many large organizations. “The pace of plete the project. “They wanted it done in a hurry, and technological innovation has overtaken our original they’re paying for that to this day,” says an insider on vision for VCF, and there are now existing products the project. that did not exist when Trilogy began,” the FBI con- The project entailed converting the ratings system ceded in a report last year on the demise of the proj- from its assembler code base, written for a mainframe, ect. to a client-server architecture, with a goal of improving The VCF project also suffered from having nine pro- user-friendliness and flexibility so Nielsen could cre- gram managers and five CIOs in its short, unhappy ate more precise data cuts to sell to its broadcast indus- four-year life. try customers. Given the complexity of the project, tech To its credit, the bureau appears to have learned its staffers at Nielsen calculated the data conversion would lesson. It’s ditched the VCF program in favor of an in- take three years to complete, wrapping up before the formation management system called Sentinel. Sentinel Y2K rollover. will be based on cutting-edge technology and should But that timetable didn’t sit well with upper man- have a long shelf life: It will be Web-based, and a serv- agement, who wanted the new system rolled out within ice-oriented architecture ensures that it will be able to a year, according to our source. The rush was on. connect to external law enforcement databases and Want a robot that can run your IT department? other information sources that incorporate Web services Somewhere out there is a vendor that will claim it has standards. one. So it’s not surprising Nielsen found an out- Sentinel’s first phase will provide FBI agents, ana- sourcer that swore it could complete the project lysts, and other personnel with a portal that will let within a year. In 1997, Nielsen signed a software de- them access the soon-to-be-replaced automated case- velopment contract with TenFold. “They made the de- support system and, later, data in the new case man- cision, and everybody had to go with it,” the source agement system. It also will include a case manage- says. TenFold and Nielsen both declined to comment. ment “workbox” that will summarize a user’s workload Staff-level technicians at Nielsen were skeptical from (the case files an agent or analyst is working on) and the beginning, and signs soon emerged that they were provide automatic indexing in case files according to right. An inspection a few months into the project re- person, place, or thing. vealed that TenFold was creating flat files for the ratings In March, the FBI awarded Lockheed Martin a $305 system, even though the contract called for object-ori- million contract to help with the implementation of ented code. Sentinel. Subcontractors include Accenture and The rationale: Surprise! TenFold didn’t have enough Computer Sciences Corp. CSC’s Dyncorp unit was one time to do the job properly.During one code review,an Copyright (c) 2006, CMP Technology. Important note: This PDF is provided solely as a reader service. It is not intended for reproduction or public distribu- tion. For more information on obtaining a Reprint, please contact a Reprint Services Rep at 951.698.1780 or visit www.cmpreprints.com/faxback.jhtml. IN DEPTH / IT FIASCOES

FIND MORE ON INFORMATIONWEEK.COM Despite the concerns, Nielsen’s senior IT executives told internal staff to back off TenFold. ”We were told not More errors that have cost time, money, and reputations at in- to upset them and cause a lot of grief,” the insider says. formationweek.com/1110/blunders_sb.htm Ultimately,however, Nielsen’s IT decision-makers got Have you worked at a company where a tech fiasco oc- fed up with TenFold’s lack of progress.They terminated curred? Take our poll at informationweek.com/1110/ poll_blunders.htm the project and, in June 2000, sued the company for $4.5 How to avoid joining our confederacy of IT dunces at infor- million. The sides reached an undisclosed settlement, mationweek.com/1110/blog_blunders.htm but Nielsen’s problems didn’t end there. It’s still trying to put an upgraded ratings system in place, almost 10 IT manager at Nielsen found that a TenFold staffer had years after the project began. written the following into his notes: “We have to do this crap because we don’t have time to do it right,” accord- Write to Paul McDougall ing to the source. at [email protected]