A Tensor Compiler for Unified Machine Learning Prediction Serving Supun Nakandala, UC San Diego; Karla Saur, Microsoft; Gyeong-In Yu, Seoul National University; Konstantinos Karanasos, Carlo Curino, Markus Weimer, and Matteo Interlandi, Microsoft https://www.usenix.org/conference/osdi20/presentation/nakandala This paper is included in the Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation November 4–6, 2020 978-1-939133-19-9 Open access to the Proceedings of the 14th USENIX Symposium on Operating Systems Design and Implementation is sponsored by USENIX A Tensor Compiler for Unified Machine Learning Prediction Serving Supun Nakandalac,∗, Karla Saurm, Gyeong-In Yus,∗, Konstantinos Karanasosm, Carlo Curinom, Markus Weimerm, Matteo Interlandim mMicrosoft, cUC San Diego, sSeoul National University {<name>.<surname>}@microsoft.com,
[email protected],
[email protected] Abstract TensorFlow [13] combined, and growing faster than both. Machine Learning (ML) adoption in the enterprise requires Acknowledging this trend, traditional ML capabilities have simpler and more efficient software infrastructure—the be- been recently added to DNN frameworks, such as the ONNX- spoke solutions typical in large web companies are simply ML [4] flavor in ONNX [25] and TensorFlow’s TFX [39]. untenable. Model scoring, the process of obtaining predic- When it comes to owning and operating ML solutions, en- tions from a trained model over new data, is a primary con- terprises differ from early adopters in their focus on long-term tributor to infrastructure complexity and cost as models are costs of ownership and amortized return on investments [68]. trained once but used many times. In this paper we propose As such, enterprises are highly sensitive to: (1) complexity, HUMMINGBIRD, a novel approach to model scoring, which (2) performance, and (3) overall operational efficiency of their compiles featurization operators and traditional ML models software infrastructure [14].