Comic Relief: Multi-cloud, multi-foundation, 400 real-time donations per second, zero downtime, 100% success Over the past 30 years, Comic Relief [http://www.comicrelief.com/] has raised over £1 billion for good causes around the world and in the UK. Through their two campaigns Red Nose Day [www.rednoseday.com] and Sport Relief [http://www.sportrelief.com/], they used the money donated by the great British public to make their vision a reality: a just world, free from poverty. Founded in 1985 by comedy scriptwriter Richard Curtis and comedian Lenny Henry in response to famine in Ethiopia, every year Comic Relief has grown in size and influence. In its first telethon in 1998 it raised £76,610. In 2015, it raised £99,418,831. Every two years, Comic Relief’s Red Nose Day takes over primetime Friday night BBC TV for seven hours. The telethon is responsible for the majority of Comic Relief’s annual fundraising, attracting a large audience who donate money via the website or a network of 14,000 call centre operators across 120 call centres who donate their time for free. “With an increasing number of donations to Comic Relief being taken online, it is critical we have 100% confidence in our donations platform. Armakuni provides us with that confidence.” Derek Gannon, COO, Comic Relief The old donations platform The directors appreciated that the existing platform was nearing its end of life and a new platform was needed - when such a large percentage of your annual money is made in about 7 hours, you really want to know the thing is not going to fail. “Armakuni’s involvement gives us 100% confidence our donations platform can handle the considerable volume of traffic we experience during event night. Not only that, Armakuni has significantly reduced the cost and time associated with the build and operation of the platform.“ Zenon Hannick, CTO, Comic Relief Our pitch In 2012, Armakuni was approached by Comic Relief and asked to pitch for the ground-up platform rewrite. Armakuni was known to Comic Relief - the founders had worked as part of the team at Comic Relief before leaving to start Armakuni. We pitched an incremental, Agile approach. It was fairly daunting - at the time we were a small team, and we were competing with a number of larger, more established tech firms, several of which had been involved in standing up the donations platform in previous years. Comic Relief liked that our concept could be tried and tested before any critical long term decisions had to be made. We proposed to work on a time and materials basis, enabling the directors to disengage if things weren’t going the way they wanted. Part of the way we work (we call this the Armakuni Way) meant that we reassured the board they would see fortnightly demonstrations of the project. This allowed all involved to thrash through the issues, risks, unknowns and problems, miles in advance of the live night. Even though we were small fish in a big pond, our innovative approach, and risk-averse strategy won over the board. The brief The brief wasn’t for the feint hearted. ● Scalable, with 100% performance reliability ● Supporting 14,000 call centre operators and up to 100,000 concurrent web sessions ● Catering for up to 400 donations/second, and an estimated 800,000 donations in total ● PCI compliance ● Build and run costs need to be kept to a minimum Our approach We used a polyglot approach. This included multi-IaaS, multi-PaaS, multi-language and with multi payment-service providers. Infrastructure-wise we used PaaS to allow a single contract to the application development across multiple IaaS. After exploring several options and performing proof of concepts with OpenShift and Cloud Foundry, we settled on Cloud Foundry, deploying on both Amazon Web Services (AWS) and vSphere in a London data-centre. “From scratch, we can deploy the entire platform, infrastructure and applications, fully tested, on to three continents in about 90 minutes - and handle the live TV show!” Tim Savage, Director, Armakuni We now use AWS and Google Compute Engine GCE. The importance of being able to switch between clouds avoids the dangers and frustrations of being ‘locked-in’. We used an Agile approach, to disconnect applications from infrastructure, and to enable continuous deployment of both infrastructure and applications. We used continuous deployment through to production - enabled by a rigorous Test Driven Development (TDD) approach, and we integrated security and load tests into our pipelines. Architecture-wise we employed a microservices environment to allow us to decouple responsibility horizontally where required. We used stateless applications, and chose availability over consistency, adopting an ‘eventually consistent’ model (in the worst case scenario, we could lose reporting in favour of continuing to take donations). We did all this in nine months with a team of five. Overcoming problems Of course, we’ve had some problems in the past few years. For example, we initially considered OpenShift, but we found that, at the time, Cloud Foundry was the more mature of the two PaaS offerings. We have also spent many hours load testing with payment providers, finding out whether they could actually handle the level of traffic that they were claiming. “Armakuni’s use of Cloud Foundry PaaS enables us to deploy our donations platform on-demand to multiple infrastructure providers around the globe. And the cloud native approach and use of test- driven development enables us to deploy a change into production within 30 minutes. Even on event night when we’re processing 300 transactions per second!” Zenon Hannick, CTO, Comic Relief The result of our work The result: an overriding success. ● Performant to the tune of up to 400 donations per second. ● Highly scalable and resilient, and is deployed and scaled programmatically. ● Resilience across multiple points of failure; infrastructure, connectivity and payment provider. ● From scratch, the entire platform, infrastructure and applications, fully tested, can be deployed to the separate IaaS providers across three continents in about 90 minutes. ● Unlimited horizontally scalability “on demand”. ● Significant reduction in team size; a team of five builds, deploys and runs the donations platform - even on event night. ● No need for proprietary, high-spec infrastructure and licenses. ● Potential for human error removed; everything-as-code automates applications, services, environment provisioning and deployment. ● Fully tested deployment providing maximum confidence in the platform. “Our work has been so fundamental, that Gartner have highlighted us as a case study of multi-cloud usage”, Tim Savage, Director, Armakuni Continued support In March 2017, we will be supporting Comic Relief event night again. We are proud to work with such a prestigious and high-profile organisation, and for the Armakuni Way of working to benefit such an important charity. For details of how you can donate, please visit Red Nose Day [www.rednoseday.com]. .