Adaptive Partial Training for Model-Heterogeneous Federated Learning

open_access
Open Access
Author:
Swope, Jason
Area of Honors:
Computer Science
Degree:
Bachelor of Science
Document Type:
Thesis
Thesis Supervisors:
Mehrdad Mahdavi, Thesis Supervisor
Mohamed Khaled Almekkawy, Thesis Honors Advisor
Keywords:
Machine Learning
Distributed Machine Learning
Federated Learning
Model-Heterogeneous Federated Learning
Partial Training
Model Hetegeneity
Data Heterogeneity
Abstract:
Federated Learning (FL) has increasingly become an area of interest within Machine Learning (ML) recently for its ability to combine the performance of multiple devices. Model-Heterogeneous FL in particular allows for the clients to train a larger model than each individual device could train individually by dropping out specific neurons from the global model. This allows even low-performance devices to contribute to training even when the device would otherwise would not be able to contribute under traditional Model-Homogeneous FL. The state of the art method for sub-model extraction is FedRolex, which systematically steps through the available neurons. In addition to model-heterogeneity, another major factor in the performance of FL is the level of data-heterogeneity between the devices. This study investigates the performance of Model-Heterogeneous methods FedRolex and FedDropout at differing levels of dropout, data-heterogeneity, and synchronization, and compares their performance with the Model-Homogeneous method FedAvg. In addition, three new methods are proposed to tackle the problem: FedStack, FedCover, and FedMinOccurances. The performance of FedDropout falls below the performance of any of the other methods, and FedMinOccurances shows inferior performance with high model heterogeneity.