We gratefully acknowledge support from
the Simons Foundation and member institutions.
Full-text links:


Current browse context:


Change to browse by:

References & Citations


(what is this?)
CiteULike logo BibSonomy logo Mendeley logo del.icio.us logo Digg logo Reddit logo ScienceWISE logo

Statistics > Machine Learning

Title: Ten Steps of EM Suffice for Mixtures of Two Gaussians

Abstract: The Expectation-Maximization (EM) algorithm is a widely used method for maximum likelihood estimation in models with latent variables. For estimating mixtures of Gaussians, its iteration can be viewed as a soft version of the k-means clustering algorithm. Despite its wide use and applications, there are essentially no known convergence guarantees for this method. We provide global convergence guarantees for mixtures of two Gaussians with known covariance matrices. We show that the population version of EM, where the algorithm is given access to infinitely many samples from the mixture, converges geometrically to the correct mean vectors, and provide simple, closed-form expressions for the convergence rate. As a simple illustration, we show that, in one dimension, ten steps of the EM algorithm initialized at infinity result in less than 1\% error estimation of the means. In the finite sample regime, we show that, under a random initialization, $\tilde{O}(d/\epsilon^2)$ samples suffice to compute the unknown vectors to within $\epsilon$ in Mahalanobis distance, where $d$ is the dimension. In particular, the error rate of the EM based estimator is $\tilde{O}\left(\sqrt{d \over n}\right)$ where $n$ is the number of samples, which is optimal up to logarithmic factors.
Comments: Accepted for presentation at Conference on Learning Theory (COLT) 2017
Subjects: Machine Learning (stat.ML); Data Structures and Algorithms (cs.DS); Statistics Theory (math.ST)
Cite as: arXiv:1609.00368 [stat.ML]
  (or arXiv:1609.00368v5 [stat.ML] for this version)

Submission history

From: Emmanouil Zampetakis [view email]
[v1] Thu, 1 Sep 2016 19:57:26 GMT (453kb,D)
[v2] Thu, 15 Sep 2016 19:55:11 GMT (1608kb,D)
[v3] Wed, 12 Apr 2017 17:59:27 GMT (1064kb,D)
[v4] Thu, 13 Apr 2017 00:55:32 GMT (1065kb,D)
[v5] Mon, 5 Jun 2017 07:53:53 GMT (1063kb,D)

Link back to: arXiv, form interface, contact.