IOTG Russia | Intel CV Academy 10 Mar 2021 Federated Learning About speaker
Alexey Gruzdev ▪ Federated Learning Team Lead @ Intel ▪ MS in Computer Science ▪ HSE invited lecturer for ML/DL ▪ https://www.linkedin.com/in/alexe y-gruzdev-454399128 ▪ [email protected]
Intel CV Academy 2020-21 2 AGENDA
1. Federated Learning : concepts & basic idea 2. Federated Learning : research status 3. OpenFL introduction 4. OpenFL demo 5. FL Tools comparison 6. Q & A
Intel CV Academy 2020-21 3 Intel CV Academy 2020-21 4 Federated Learning Federated Learning -Google, 2017
https://ai.googleblog.com/2017/04/federated-learning-collaborative.html
Intel CV Academy 2020-21 6 Federated Learning -Google, 2017
ai.googleblog.com/2017/04/federated-learning-collaborative.html w w
푛푖 푤푖 w 푁 w
Intel CV Academy 2020-21 7 Federated Learning a a t t+1, a
b b t t+1, b Parameter t+1 Server c Aggregator t c t+1, c
Intel CV Academy 2020-21 8 https://www.nature.com/articles/s41598-020-69250-1
Intel CV Academy 2020-21 9 Collaborative Federated Learning For Healthcare: Multi-Modal COVID-19 Diagnosis at the Edge (2021)
https://arxiv.org/pdf/2101.07511.pdf
Intel CV Academy 2020-21 10 Federated Object Detection: Optimizing Object Detection Model with Federated Learning
https://dl.acm.org/doi/pdf/10.1145/3387168.3387181
Intel CV Academy 2020-21 11 Turn Signal Prediction: A Federated Learning Case Study (2020)
https://arxiv.org/pdf/2012.12401.pdf
Intel CV Academy 2020-21 12 Privacy-Preserving News Recommendation Model Learning (2020)
https://arxiv.org/pdf/2003.09592.pdf
Intel CV Academy 2020-21 13 How to backdoor FL? (2019)
https://arxiv.org/pdf/1807.00459.pdf
Intel CV Academy 2020-21 14 Inverting Gradients - How easy is it to break privacy in federated learning? (2020)
https://arxiv.org/pdf/2003.14053.pdf
Intel CV Academy 2020-21 15 Split learning for health: Distributed deep learning without sharing raw patient data (2018)
https://arxiv.org/pdf/1812.00564.pdf
Intel CV Academy 2020-21 16 Federated Learning Open Research/Engineering Questions
• Vertical Federated Learning (sample is distributed over multiple machines)
• Non-IID/Heterogeneous data
• Communication efficiency at scale (what if we have 1k participants)
• Collaborative Learning with variable number of participants
• Collaborative Learning with different model architectures for participants
• Weakly/Unsupervised setting (data is born without labels)
• Can we develop better algorithm rather than just taking average? ☺
Intel CV Academy 2020-21 17 OpenFL introduction OpenFL Software Status v1.0
• Initial Open Source Release (Apache v2, github.com/intel/openfl) • TensorFlow 2 / PyTorch support • TaskRunner – Multiple workloads on different clients • Docker / Singularity containers • Examples: MNIST, Brain Tumor, Histology, Unet Polyp Segmentation, Adversarial training, Fine-tuning for TinyImageNet
Docs: openfl.readthedocs.io/en/latest/index.html
Intel CV Academy 2020-21 19 OpenFL Interfaces: Python API fx CLI Experimentation, Single Node Production, Multi-node
Intel CV Academy 2020-21 20 OpenFL Architecture – 100k feet view
Aggregator Collaborator CollaboratorCollaborator Tensor DB Tensor Codec Tensor DB Tensor Codec
GLOBAL LOCAL
gRPC mTLS Task Assigner Task Runner Private Data
Intel CV Academy 2020-21 21 OpenFL Components: Plan ▪ Plan object responsibilities: • Parsing of FL Plan: • Plan.yaml • Defaults files for all components • List of collaborators • Network configuration • Hash configuration to enforce participants are using same plan • Serialization / deserialization of plan • Initialization of main OpenFL components
Intel CV Academy 2020-21 22 FL Plan Aggregator Plan
Collaborator Plan
Data Loader
Task Runner
Intel CV Academy 2020-21 23 OpenFL Components: Aggregator ▪ What does the aggregator do? Aggregator
• Acts as parameter server Tensor DB Tensor Codec • Aggregates model and metrics GLOBAL • After model aggregation, applies compression / delta computation to tensors Task Assigner • Performs end of round check
Intel CV Academy 2020-21 24 OpenFL Components: Collaborator ▪ What does the collaborator do? Collaborator
• Fetches tasks from Aggregator Tensor DB Tensor Codec • Calls functions on TaskRunner (could LOCAL be training, validation, etc.) • Local training on private data.
Task Runner Private Data
Intel CV Academy 2020-21 25 OpenFL demo Existing Federated Learning (FL) Solutions TensorFlow PySyft PaddleFL Flower Federated Concept integrated integrated static static Based on ML-Framework TensorFlow PyTorch PaddlePaddle Any (TensorFlow, PyTorch, JAX, ...) Remote Data Loading No No Yes Yes Supported Systems Local simulation and Local simulation and Docker OS and K8s (Docker Container), OS and K8s (Docker Docker container container distribution mobile, embedded and internet Container), mobile, distribution of things devices embedded and internet of things devices Currently supports Remote No Yes Yes Yes Worker GPU Support No Yes Yes Yes Differential Privacy Yes Not Now Yes Not Now Secure Aggregation (SMPC) No Yes Yes Not Now Resources for Code Examples Just a few very basic Tutorial & examples in Few examples in the repository Tutorials / examples in examples in the documentation and repository documentation, boilerplate documentation and + blog posts and community project examples in the repository projects repository
Documentation & community Only developer Good documentation and Documentation in English. Most Good documentation and support documentation and good community support of community, issues and their good community support, stack overflow website are in Chinese e.g. through Slack community support Stars of Framework 1.4 k TensorFlow 6.8 k PySyft & 193 PaddleFL & 247 Flower & plenty of stars Federated & from your favourite ML- Framework https://www.inovex.de/blog/federated-learning-frameworks-part-2/ Intel CV Academy 2020-21 27 Existing Federated Learning (FL) Solutions
PySyft OpenFL Concept integrated static Based on ML-Framework PyTorch Agnostic (PyTorch, TensorFlow)
Remote Data Loading No Yes Supported Systems Local simulation and Docker container OS and Docker Container distribution
Currently supports Remote Worker Yes ??? Yes
GPU Support Yes Yes Differential Privacy Not Now Future Secure Aggregation (SMPC) Yes Yes, manually via Intel® SGX Resources for Code Examples Tutorial & examples in documentation and Fully-functional demos for PyTorch and repository + blog posts and community TensorFlow. projects
Documentation & community support Good documentation and good community Documentation. Slack channel. GitHub issues. support
Stars of Framework 6.8 k PySyft & 70
Intel CV Academy 2020-21 28 OpenFL vs PySyft
OpenFL PySyft
▪ Concepts of Collaborator/Aggregator/ FL ▪ Concepts of VirtualWorker/Pointer to PyTorch Tensor/Duet/PyGrid plan/Workspace/TaskRunner ▪ Low-level integration in PyTorch Tensor object ▪ Own training engine with TaskRunner – you through pointer to Tensor entity must bring training code to OpenFL, not OpenFL to the training code (WIP under ▪ Much cleaner Python API from Data Scientist improvements here) point of view – more natural fit for existing codebase
▪ Harder to integrate with existing training ▪ Looks like an awesome API, however lack of code (due to more entities for real multi- support for multi-node real experiments. node secured scenarios ) • https://github.com/OpenMined/PySyft/issues/3101 • https://github.com/OpenMined/PySyft/issues/3885
Intel CV Academy 2020-21 29 OpenFL: open-source library for Federated Learning
Python: Keras, TensorFlow, PyTorch
GitHub github.com/intel/openfl
Slack bit.ly/2MKAyAv
Intel CV Academy 2020-21 30 We are hiring☺
▪Deep Learning Engineer intel.ly/3r9LbvQ ▪Deep Learning R&D Intern bit.ly/3rgk8yX
Intel CV Academy 2020-21 31 32