S2CE: A Hybrid Cloud and Edge Orchestrator for Mining Exascale Distributed Streams Nicolas Kourtellis Herodotos Herodotou Maciej Grzenda
[email protected] [email protected] [email protected] Telefonica Research Cyprus University of Technology Warsaw University of Technology Barcelona, Spain Limassol, Cyprus Warsaw, Poland Piotr Wawrzyniak Albert Bifet
[email protected] [email protected] Lodz University of Technology LTCI, Telecom Paris, IP-Paris Lodz, Poland Paris, France ABSTRACT 1 INTRODUCTION The explosive increase in volume, velocity, variety, and veracity In the future Internet era, with hundreds of billions of devices, of data generated by distributed and heterogeneous nodes such as principle factors dominating the continuous utility of the Internet IoT and other devices, continuously challenge the state of art in big will be: 1) the massive population of devices and their intelligent data processing platforms and mining techniques. Consequently, it agents, 2) the big, fast, and diverse data produced from them and reveals an urgent need to address the ever-growing gap between their users, and 3) the need for large-scale, adaptive infrastructures this expected exascale data generation and the extraction of insights to process and extract knowledge from these exascale data in or- from these data. To address this need, this paper proposes Stream der to help make critical, data-driven decisions. Intelligent agents to Cloud & Edge (S2CE), a first of its kind, optimized, multi-cloud already exist in different forms, and are well embedded in various and edge orchestrator, easily configurable, scalable, and extensible. ways in our everyday lives, either as passive data collectors, or S2CE will enable machine and deep learning over voluminous and active producers.