Machine Learning Distributed Machine Learning Federated Learning Model-Heterogeneous Federated Learning Partial Training Model Hetegeneity Data Heterogeneity
Abstract:
Federated Learning (FL) has increasingly become an area of interest within Machine Learning (ML) recently for its ability to combine the performance of multiple devices. Model-Heterogeneous FL in particular allows for the clients to train a larger model than each individual device could train individually by dropping out specific neurons from the global model. This allows even low-performance devices to contribute to training even when the device would otherwise would not be able to contribute under traditional Model-Homogeneous FL. The state of the art method for sub-model extraction is FedRolex, which systematically steps through the available neurons. In addition to model-heterogeneity, another major factor in the performance of FL is the level of data-heterogeneity between the devices. This study investigates the performance of Model-Heterogeneous methods FedRolex and FedDropout at differing levels of dropout, data-heterogeneity, and synchronization, and compares their performance with the Model-Homogeneous method FedAvg. In addition, three new methods are proposed to tackle the problem: FedStack, FedCover, and FedMinOccurances. The performance of FedDropout falls below the performance of any of the other methods, and FedMinOccurances shows inferior performance with high model heterogeneity.
