625.664 Computational Statistics
Problem Set 3
Associated Reading: Chapter 2: 2.2.2-2.2.2.2
Chapter 4: Introduction - Section 4.2
Complete the problems either by hand or using the computer and upload your final document
to the Blackboard course site. All final submittals are to be in PDF form. Please document any
code used to solve the problems and include it with your submission.
1. Problem 2.5. Only do parts (a), (b), (c), and (e).
2. Consider the following mixture of two normal densities:
p(x; θ) = πϕ(x; µ1, σ21) + (1 − π)ϕ(x; µ2, σ22)
where θ = (π, µ1, µ2, σ21, σ22) is the collection of all the parameters driving the model, with
π denoting the the mixing proportion (and not the usual number), and ϕ is the standard
density for a normal random variable. Direct use of maximum likelihood is made difficult
here because of the expression of the likelihood function. So, we wish to simplify this
problem by use of the EM algorithm.
Let Zi be the random variable that denotes the class membership of of an observed xi
in
a given sample. Clearly, zi
is the missing data, with Zi ∈ {0, 1} and Zi ∼ Bernoulli(π).
The complete-data density for Y = (X, Z) is given by
p(xi, zi|θ) = [πϕ(x; µ1, σ21)]zi
[(1 − π)ϕ(x; µ2, σ22)](1−zi)
(a) Show that the complete-data log-likelihood function is
l(θ; x, z) = log(π)∑ni=1zi +
∑ni=1zilog(ϕ(xi; µ1, σ21)) + log(1 − π)(n −∑ni=1zi)+
∑ni=1(1 − zi) log(ϕ(xi; µ2, σ22)).
For the Expectation step of the EM algorithm, we need Q(θ|θ
(k)) = E[l(θ; y)|x, θ(k)].
(b) Find Q(θ|θ
(k)) in terms of E[Zi|xi, θ(k)], again it is best to leave your answer as the
sum of four main terms. Also, if is fine to leave your answer in terms of the function
ϕ(xi; µ(k)1,(σ21)(k)).
(c) Show that, given θ
(k) and xi,E[Zi|xi, θ(k)] = P[Zi = 1|xi, θ(k)] = π
(k)ϕ(xi; µ(k)1,(σ21)(k))π(k)ϕ(xi; µ(k)1,(σ21)(k)) + (1 − π(k))ϕ(xi; µ(k)2,(σ22)(k))
For the Maximization step of the EM algorithm, we need to find the new set of parameters
θ(k+1) that maximizes Q(θ|θ(k)). This can be done in a straightforward manner by solving
δQ(θ|θ(k))δθ = 0
for each of the parameters in θ = (π, µ1, σ21, µ2, σ22). To simplify your notation let η
(k)i =E[Zi|xi, θ(k)] and η
(k) =∑ni=1 η(k)i.
(d) First show that
(e) Now, use the fact that
(f) Suppose that you have observed the sample x1, x2, . . . , xn from the above distribution.
Write the psuedocode required to estimate θ using the EM algorithm. Be sure to include
some form of convergence criteria.