![]() Towards federated learning at scale: System design. Keith Bonawitz, Hubert Eichner, Wolfgang Grieskamp, Dzmitry Huba, Alex Ingerman, Vladimir Ivanov, Chloe Kiddon, Jakub Konečny, Stefano Mazzocchi, H Brendan McMahan, et al.International Conference on Machine Learning (2018), 560-569. signSGD: Compressed Optimisation for Non-Convex Problems. Jeremy Bernstein, Yu-Xiang Wang, Kamyar Azizzadenesheli, and Animashree Anandkumar.In Advances in Neural Information Processing Systems. Qsparse-local-SGD: Distributed SGD with quantization, sparsification and local computations. Debraj Basu, Deepesh Data, Can Karakus, and Suhas Diggavi.Federated Learning with Personalization Layers. Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary.A public domain dataset for human activity recognition using smartphones. Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge Luis Reyes-Ortiz.QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding. Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic.Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 440-445. Sparse Communication for Distributed Gradient Descent. FedMask also achieves 1.56X inference speedup and reduces the energy consumption by 1.78X. Our experiments show that compared with status quo approaches, FedMask improves the inference accuracy by 28.47% and reduces the communication cost and the computation cost by 34.48X and 2.44X. Instead of learning a shared global model in classic FL, each device obtains a personalized and structured sparse model that is composed by applying the learned binary mask to the fixed parameters of the local model. To achieve this, each device learns a sparse binary mask (i.e., 1 bit per network parameter) while keeping the parameters of each local model unchanged only these binary masks will be communicated between the server and the devices. By applying FedMask, each device can learn a personalized and structured sparse DNN, which can run efficiently on devices. In this paper, we present FedMask - a communication and computation efficient FL framework. In addition, considering mobile devices usually have limited computational resources, improving computation efficiency of training and running DNNs is critical to developing on-device deep learning applications. Such statistical heterogeneity and communication bandwidth limit are two major bottlenecks that hinder applying FL in practice. However, the data residing across devices is intrinsically statistically heterogeneous (i.e., non-IID data distribution) and mobile devices usually have limited communication bandwidth to transfer local updates. ![]() Hence, FL becomes a natural choice for deploying on-device deep learning applications. Federated learning (FL) is a distributed machine learning paradigm which allows for model training on decentralized data residing on devices without breaching data privacy. However, it is technically challenging to locally train a DNN model due to limited data on devices like mobile phones. Recent advancements in deep neural networks (DNN) enabled various mobile deep learning applications.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |