Byzantine-tolerant distributed learning of finite mixture models

Abstract Traditional statistical methods need to be updated to work with modern distributed data storage paradigms. The split-and-conquer framework that learns models on local machines and averaging their parameter estimates is common. However, this does not work for the important problem of learning finite mixture models, because subpopulation indices on each local machine may be arbitrarily permuted (the ‘label switching problem’). Earlier work proposed mixture reduction (MR) to address this issue, offering an effective and efficient solution for aligning and aggregating local mixture components. Building upon this foundation, this paper considers the additional challenge of Byzantine failure, where a fraction of local machines may transmit arbitrarily erroneous information. We introduce distance-filtered mixture reduction (DFMR), a Byzantine-tolerant framework that enhances MR by adding a distance-based filtering mechanism to identify and exclude corrupted local estimates before the MR aggregation. This integration allows DFMR to maintain MR’s efficiency while achieving strong robustness against Byzantine failure. We provide theoretical justification for DFMR, proving its optimal convergence rate and asymptotic equivalence to the global maximum likelihood estimate under standard assumptions. Numerical experiments on simulated and real-world data validate the effectiveness of DFMR in achieving robust and accurate aggregation in the presence of Byzantine failure.