CASE STUDY

KMG International Accelerated Transformation of IT

1 Learning From Experience: Problems Created by Full Outsourcing

KMG International had been growing very fast in and as a result the IT infrastructure was completely redesigned between 2005 and 2009. Over that period, however, IT costs ballooned. At its peak, the IT team swelled to 140 people who covered everything from development and testing through to day-to-day operations.

Cost reduction in order to increase profitability of outsourcing company’s short-term revenues while the business became an imperative and so in 2010 exposing KMG International to high risks in the long the management team decided to restructure IT. term. For example, in 2011 there was a decision to stop Outsourcing the entire function seemed like a good paying license maintenance for critical applications proposal and so a contract was signed with an external — in order to bank a short term cost saving. However organization that had been set up to provide all without a valid maintenance contract in place it necessary IT services for KMG International. All the was impossible to escalate issues with software IT staff were transferred out and only one person providers and deploy upgrades. The risk of not having remained within KMG International as a contact point an escalation path for critical defects was added to for IT. The arrangements covered a flat fee plus change neglecting the software upgrade path for enterprise requests for any new projects. applications. In the long run, KMG International was compelled to renew the support and pay a However, the change requests were so numerous and hefty backlog of license maintenance, since critical complex that the overall IT costs for KMG International business applications cannot function without license actually went up rather than down. Furthermore the maintenance. outsourcing organization held more than 100 contracts with other technology providers in order to fulfill KMG KMG International Background International’s requests — adding further complexity and cost to the delivery process. KMG International N.V. (formerly known as The Group N.V.) is an international When a new management team took control of the oil company with operations in 12 countries business in 2012 they recognized that IT was a risk for throughout , Central Asia and North Africa. The group is active primarily in refining, the business both operationally and financially. They marketing and trading, with additional operations wanted to find a new solution for IT in order to reduce in exploration and production, and other oil costs while improving the quality and range of services industry services such as drilling, EPCM, and provided. transportation. As of 2010, the group is entirely owned by NC It was evident that during the period of the KazMunayGas JSC, the national oil and gas outsourcing agreement IT operations had run out of company of the Republic of Kazakhstan. KMG control. Service level agreements were lax and the International owns the largest and most modern contracts did not really incentivize any improvement. refinery in Romania and the Black Sea region — Petromidia Navodari — which boasts a processing capacity of over five million tons per year. Change requests coming from the business received responses late with price quotations much higher than The company’s retail network that operates under the market. Contracts with third party suppliers were the “Rompetrol” brand incorporates over 1,100 fuel distribution points in Romania, Georgia, not done through a transparent tender process and Bulgaria, Moldova, and under the Dyneff brand didn’t necessarily reflect the best interests of KMG in France and Spain. The company employs over International. 7,500 people worldwide, of which around 6,000 are in its home market, Romania. Poor decisions were made in order to maximize the

2 A New Model for IT Efficiency

KMG International management wanted to find a different solution. Having already experienced a full internal model followed by a fully outsourced model, KMG International moved to designing a hybrid model. Under this new design, certain functions mostly focused on strategy, governance and vendor management were held internally and most operational execution was outsourced.

The main objective was to reassert control for IT within KMG International so that it could direct proactively the level of service that was needed and also manage costs more effectively.

Today the Group IT function of 26 people covers the following areas:

• IT Strategy • Vendor Management • IT Budget Management • Systems Architecture • Demand Management • Contract Management • Information Security • Project Management

The team acts as a business partner to the business entities that form the group. They work with various business clients to define their requirements in terms of IT solutions and services, drive the approval and procurement processes, oversee the delivery of the projects executed by with external providers and coordinate the support activities. More recently the team also took on a business development role, proposing new solutions with strong business cases to companies and corporate functions in the group.

On the operations side KMG International has a contract with SkyConsult for helpdesk, infrastructure and application support — and in turn they have an agreement with IPsoft for infrastructure support. KMG International also holds other similar agreements with various vendors to perform support tasks and provide other services.

3 Planning a Transition without Full Data Access

Late in 2012, Marcel Chiriac engaged with KMG International’s senior management and acted as a consultant to SkyConsult in order to help shape the new organization model. He took over the CIO role formally in November 2012 moving from Vancouver back to his native Romania. At that point there were three people in Group IT.

His initial task was to reduce the risk in IT operations 1. Deliver a smooth transition with no negative for KMG International — re-establishing an IT function business impact. that could be the business partner to the business 2. Build and implement a new hybrid model — units. Group IT would now have direct responsibility establishing a new relationship with vendors to to top management and be accountable for control restore control for IT within KMG International. over all vendors. As Marcel Chirac began to plan 3. Deliver a more financially efficient IT service. the transition to the new model he bore three key objectives in mind:

The very first step in transitioning to the new model involved exploring the 100+ contracts that had been drawn up by the outsourcer in order to discover what they were delivering. New relationships needed to be put in place, contract terms needed to be renegotiated and in many instances contracts needed to be consolidated. For example, KMG International moved from having 38 telecoms contracts to just four.

The proposed change was received with complete hostility by the outsourcing firm involved who proceeded to deny any information access to the new KMG International IT management, let alone knowledge transfer. “The entire transition to the new model was planned and managed in the same way as you might expect in a hostile takeover,” explains Marcel Chiriac.

Most importantly the team worked to re-establish control of services. The layering of sub-contracts also meant that KMG International was unable to exert any authority over many of the sub-contractors. In one example, the new CIO was refused the administration passwords for the technology that was supplied to KMG International via the outsourcing company, as they had no direct relationship in place.

4 The Race to Cutover

After arriving in November 2012, Marcel Chiriac had to mobilize his partners at record pace in order to take on operational responsibility for running all IT services by the contract cutover date on January 1, 2013.

IPsoft played a critical role in supporting this transition With the clock ticking towards the cutover deadline, and the deployment of the IPcenter platform greatly the CIO’s combined team of 20 people made up of his facilitated information gathering of the IT operations skeleton internal team, the SkyConsult team and IPsoft, that were soon going to be transferred. IPsoft brought prepared themselves for the transfer of IT operations in a very experienced team of people to work on this supporting KMG International’s entire business. unique challenge. A core team came to supported online by IPsoft operations teams in other The hostility of the incumbent provider continued global locations. In a very short time the team had to cause problems right down to the wire as many learned and understood the new IT environment. Using administration passwords had not been handed over. IPmons they were able to find out what was present It seemed as though the team would have to shut and make preparations to secure the infrastructure in down all servers and restart them on January 1st. With time for the looming cutover date. only a few days to go, however, on December 28th Marcel Chiriac managed to secure the handover of key Despite significant obstacles to progress, such as information that would allow the service to transfer only having a partial view of the network rather than over smoothly. full administration rights, a solid picture of the IT landscape was put in place. It covered the underlying details of:

• 3 datacenters IPsoft was able to

• Hundreds of servers (physical and virtual) find out what was

• 3 virtualization environments present and make

• 4000 users preparations to secure

• 200 applications the infrastructure in

• Network spanning many countries. time for the looming

IPsoft carried out due diligence to manage all the cutover date underlying devices and integrated IPcenter with the other systems management tools that were in place.

3 100s 3 4000 200

datacenters of servers virtualization users applications (physical and environments virtual)

5 Cutover Success

The strategy the team had put in place proved to be solid. On January 1, 2013 the new team moved quickly in order to secure all the systems of which it took control. This was essential and in fact there were several attempts to attack the network in the first few weeks. Happily the team was able to defend against each of these.

KMG International had only one 4-hour outage on that first day based on the failure of a single application. Given the risk in the transfer and the scale and complexity of the IT environment this was a truly remarkable feat.

Within the first month more than 2000 configuration items, such as servers and network switches, were placed under IPsoft’s management using IPcenter, including all of KMG International’s 750 plus gas stations.

Technology Scale and Scope

Vblock is used to host virtual machines and all storage needs demanded by Vblock Technologies these systems. KMG International utilizes 3 Vblocks, one at each datacenter location.

15 Physical AIX servers hosting business applications such as Oracle EBS and AIX Web Message Broker

Linux Servers 100+ systems hosting applications such as Proxys, SAP, Maximo

Wintel Servers 150+ systems hosting applications such as Email/Time Control systems

1000+ devices based at gas stations in Romania, Georgia and Bulgaria Network Devices 400 critical devices located in datacenters and office buildings

Databases 150+ databases supporting applications such as Oracle EBS / CRM / Sharepoint

Fig. 1 — Close up view of KMG International’s ICT Environment Scope

Over the first six weeks the collective team deepened its knowledge of the environment in order to plan the next wave of improvements in the system. Once again IPsoft’s technology helped accelerate the information gathering. For example, a network topology diagram was created semi-automatically using IPcenter in a fraction of the time it would usually take to complete using manual methods.

6 Stabilizing the Service

Following the successful cutover, Group IT moved into the first phase of the operational plan. This was the stabilization stage, which lasted until May 2013. During this period the team focused on the following objectives:

• Ensure that all knowledge of the environment was completed in detail. • Establish clear SLAs across all parts of the IT landscape in relation to the business. • Define and prioritize what services needed to be addressed. For example there were servers that had not been rebooted for as long as six years, network nodes that had not been upgraded in years, databases that were never looked at. All such factors were a constant threat as they could provoke unnecessary outages so “cleaning them up” was a critical part of the stabilization plan.

The impact of the program on database performance provides an excellent example of how the service experienced by users improved. The number of problem tickets issued relating to KMG International’s two main business databases dropped by more than 50% within the first month of operations and then steadily fell further to average a more than 70% drop in tickets through the remaining months of the year.

Across all key areas Mean Time to Resolve has been cut extensively and in all instances performance has been consistently better than the new SLAs set (see tables below):

Applications: Mean Time to Resolve 25 Achieved MTTRSLA Targeted MTTR 23.7 23.8 23.9 24 24 24 24 20 SLA Targeted MTTRAchieved MTTR

15

10 5.3 4.4 3.4 2.8 2.9 3.4 2.6 5

0 Q3 2013 Q4 2013 Q1 2014 Q2 2014 Q3 2014 Q4 2014 Q1 2015 Time Period

Fig.2— Applications Mean Time to Resolve has fallen consistently and is well below SLA

Infrastructure: Mean Time to Resolve

120 Achieved MTTR

100 SLA Targeted MTTR

80 66.8 60 36.1 40 24.8 14.2 18.9 20 13.6 12.0 0 Q3 2013 Q4 2013 Q1 2014 Q2 2014 Q3 2014 Q4 2014 Q1 2015 Time Period

Fig.3 — Infrastructure Mean Time to Resolve has out-performed SLAs

7 Fixing problems fast became a mantra for the team. Many good practices had been neglected over the previous years and now was the time for change. Whenever the team spotted an issue they took immediate action to complete improvements such as cleaning up logs and cache in some machines, optimizing routing on network nodes.

IPsoft moved quickly to start building runbooks and standard operating procedures that could support KMG International’s ICT infrastructure fully. This involved documenting the entire environment and defining clear change management procedures that ensured that the user experience would be protected at all times.

As a result of all the changes service levels improved significantly for the business within the first six months. Not only had far better SLAs been defined but these were never breached. The business could now rely on a solid and consistent level of service.

Whenever the team spotted an issue they took immediate action to complete improvements such as cleaning up logs and cache in some machines, optimizing routing on network nodes

8 SLA Performance The team has consistently outperformed SLAs to deliver a consistently reliable service for users. In the 16 months following the start of the programme, there have been no unplanned outages that have impacted business critical applications. Improvements have climbed as automation rates have risen.

SLA (Minutes) for Achieved SLA Achieved SLA Priority mean time to respond Goals in 2013 Goals in 2014

P1 15 7.96 5.33

P2 15 6.02 3.7

P3 15 8.04 5.79

P4 15 7.49 4.17

Fig. 4 — The team outperformed its 2013 and 2014 SLAs for Mean Time to Respond

SLA (hour) for mean Achieved SLA Priority Achieved SLA Goals in 2014 time to resolution Goals in 2013

P1 4 1.9 1.18

P2 4 1.37 0.64

P3 24 1.45 1.2

P4 120 1.82 2.12

Fig. 5 — The team also outperformed its 2013 and 2014 SLAs for Mean Time to Resolution

9 Optimizing Performance

KMG International’s ambitions for IT services go much farther than stabilization. Chiriac is now overseeing a phased program of optimization in order to support the business in its future development.

The full understanding of business usage for Most importantly IPsoft has helped move KMG applications and services has already allowed the International from a reactive to a proactive approach team to tune the infrastructure in order to improve the to incident management. IPmon is continuously service to users. For example, a program of upgrading monitoring system thresholds and identifying operating systems, databases and applications was put symptoms of potential problems that could be in place, catching up with what had been neglected for addressed automatically, without human intervention. many years and ensuring the group keeps up with all new upgrades as they become available. As a result of the intelligence in the system provided by IPsoft, KMG International was able to identify Having consolidated telecom providers, KMG capacity issues with the virtualization systems and International has also ensured that network bandwidth take action before running out of capacity to host new is appropriately supplied to meet business needs at solutions. This could have been a major problem given all times. Improved processes using IPcenter also the number of new projects that KMG International mean that service is re-established much more quickly was planning to roll out over the following three to following any break in the system. When traffic is four years. IPsoft was one of the major participants broken and a site loses connection, an IPsoft ticket in the Vblock upgrade project that allows KMG immediately contacts the underlying vendor in order to International to scale its virtual environment capacity resolve the issue before it becomes a major problem. and integrate with public clouds seamlessly, thereby Previously when a network fault occurred it would take eliminating for good the risk of running out of capacity. a long time even to find out which telco provider was responsible. The automation that has been created to It was a large undertaking that involved replacing support this process has brought the response time for two virtual environments and relocating services to a third party teams down to five minutes whereas this new datacenter with full disaster recovery capability. previously could take a couple of hours. This impacts The execution began in April 2014 and was completed uptime directly. Given that the number of incidents in September 2014. IPsoft’s team managed the reported in a single month is around 3000, relating to application migration and optimized the capacity ongoing monitoring of more than 750 gas stations and usage for each application on the new servers. 30 sites, the time saving in manual labor is significant.

10 Across all applications, users have experienced the effective immediately. The mailboxes, email addresses benefit of faster response times both in main sites and associated passwords that provide an essential and for the many users who are traveling and need to channel of communication into the organization are set access their files remotely. Exchange was also upgraded up within ten minutes. Previously, this depended upon a and consolidated for the entire group onto a single manual process and could take several hours to complete. instance that offers a much larger mailbox size. The business case for the virtualization upgrade was further Significant time and effort is also being saved within improved by significant savings in software licensing. the IT infrastructure team through the use of another The migration involved a full review of all servers automata that is eliminating a common cause of false and databases, which allowed for consolidation and alerts – mistaking legitimate downtime for systems optimization of the licensing models. failure. This type of problem frequently occurs when emergency maintenance activities that require services to Marcel Chiriac developed a new strategy and be suspended are not appropriately identified throughout roadmap to 2018 in which he pushes forward with the IT environment. further optimization programs. SAP will be rolled out consistently across the business, implementing better In order to avoid this problem, automata implement the processes and controls. A major business intelligence precise downtime notification of hosts, monitoring and program is also being devised which will draw on the other systems without any manual intervention, after efficiencies of the underlying infrastructure. Finally, receiving an authorized request from the infrastructure or Group IT is fully enabled to take on a strategic business application teams. partner role, identifying and proposing to its business clients, technology solutions that will foster growth. These increased automation levels translate into service quality improvement that is tangible to KMG Continuous Service International’s staff. In 2013, the company’s internal IT customer satisfaction index returned a score of 4.2 out of Improvement Through 5 from users within the business. Previously IT had been Automation rated at less than 3.

Over the past two years, KMG International has gradually increased the number of automations Efficiency Impacts it runs Each quarter the number of automata that directly impact service has risen. At present, about Business 61% of tickets are resolved entirely without manual KMG International’s ambitious plans have intervention. In addition, automation speeds up other borne fruit already. In 2013 Group IT saved ticket resolutions so that a staggering 91.9% of all 27% in operating expense compared with tickets have been positively impacted by virtual the previous year. In 2014 the focus shifted engineers. to maintaining this more efficient level of The new automata being created impact a wide range spend while improving the range of services of business scenarios directly. For instance, automata delivered. have helped overcome the challenge of ensuring rapid response is provided to any of the company’s 700 gas “In just two years we have significantly reduced risk stations no matter what time of day or night. Within in our business by strengthening the reliability of our five minutes of an initial alert being received, relevant infrastructure and quality of our service,” summarizes issues are escalated to local third party vendor support Chiriac. “With the help of IPsoft we have transformed teams. As a result, all incidents are investigated and Group IT within KMG International from a reactive to resolved in a much shorter time frame. proactive organization. Our business now relies on us to innovate new methods that drive efficiency to new levels. In another example, automata are being used to In a market that has seen the price of oil drop markedly onboard new business partners so that they can be that translates into sustainable competitive advantage.”

11 IPsoft Background

IPsoft automates IT and business processes for enterprises across a wide range of industries. Through its portfolio of world leading autonomic and cognitive solutions it provides services that allow its clients to secure competitive advantage. Headquartered in New York City, IPsoft has offices in 13 countries across the world and serves more than 500 of the world’s leading brands directly as well as more than half of the world’s largest IT services providers.

UK: +44 (0)207 070 5700

US: +1 212-708-5500

Email: [email protected]

Web: www.ipsoft.com

Address: 17 State Street, 14th Floor, New York NY, 10004 USA

© Copyright 2015 IPsoft. All rights reserved. 12