Holistic Configuration Management at Facebook Chunqiang Tang, Thawan Kooburat, Pradeep Venkatachalam, Akshay Chander, Zhe Wen, Aravind Narayanan, Patrick Dowell, and Robert Karl Facebook Inc. ftang, thawan, pradvenkat, akshay, wenzhe, aravindn, pdowell,
[email protected] Abstract the many challenges. This paper presents Facebook’s holistic Facebook’s web site and mobile apps are very dynamic. configuration management solution. Facebook uses Chef [7] Every day, they undergo thousands of online configuration to manage OS settings and software deployment [11], which changes, and execute trillions of configuration checks to is not the focus of this paper. Instead, we focus on the home- personalize the product features experienced by hundreds grown tools for managing applications’ dynamic runtime of million of daily active users. For example, configuration configurations that may be updated live multiple times a changes help manage the rollouts of new product features, day, without application redeployment or restart. Examples perform A/B testing experiments on mobile devices to iden- include gating product rollouts, managing application-level tify the best echo-canceling parameters for VoIP, rebalance traffic, and running A/B testing experiments. the load across global regions, and deploy the latest machine Below, we outline the key challenges in configuration learning models to improve News Feed ranking. This paper management for an Internet service and our solutions. gives a comprehensive description of the use cases, design, Configuration sprawl. Facebook internally has a large implementation, and usage statistics of a suite of tools that number of systems, including frontend products, backend manage Facebook’s configuration end-to-end, including the services, mobile apps, data stores, etc.