Bishop PRML - Ch2. Probability Distributions (2)

Multinomial Variables

one hot encoding
- $x = (0, 0, 1, 0, 0, 0)^{T}$
- $\sum_{k = 1}^{K} x_{k} = 1$
- multinomial label을 표현하는 방법
categorical distribution
- $p (x ∣ μ) = \prod_{k = 1}^{K} μ_{k}^{x_{k}}$ $p (x ∣ μ) = \prod_{k = 1}^{K} μ_{k}^{x_{k}}$
  - $p (x_{k} = 1) = μ_{k}$
  - $μ = (μ_{1}, \dots, μ_{K})^{T}$
- $\sum_{x} p (x ∣ μ) = \sum_{k = 1}^{K} μ_{k} = 1$
- $E [x ∣] μ x = μ$
- likelihood
  - $p (D ∣ μ) = \prod_{n = 1}^{N} \prod_{k = 1}^{K} μ_{k}^{x_{n k}} = \prod_{k = 1}^{K} μ_{k}^{\sum_{n} x_{n k}} = \prod_{k = 1}^{K} μ_{k}^{m_{k}}$ $p (D ∣ μ) = \prod_{n = 1}^{N} \prod_{k = 1}^{K} μ_{k}^{x_{nk}} = \prod_{k = 1}^{K} μ_{k}^{\sum_{n} x_{nk}} = \prod_{k = 1}^{K} μ_{k}^{m_{k}}$
    - $m_{k} = \sum_{n} x_{n k}$ : sufficient statistic ( $x_{k} = 1$ 인 관측값의 수)
- MLE
  - Lagrange multiplier
    - $L = \sum_{k = 1}^{K} m_{k} \ln μ_{k} + λ (\sum_{k = 1}^{K} μ_{k} - 1)$
  - find extrema
    - $μ_{k} = - m_{k} / λ$
  - substituting into constraint
    - $λ = - N$
  - $μ_{k}^{M L} = \frac{m_{k}}{N}$ (N개의 관측값 중 $x_{k} = 1$ 인 경우의 비율과 동일)
multinomial distribution
- $Mult (m_{1}, m_{2}, \dots, m_{K} ∣ μ, N) = (\frac{N}{m_{1}, \dots, m_{K}}) \prod_{k = 1}^{K} μ_{k}^{m_{k}}$ $Mult (m_{1}, m_{2}, \dots, m_{K} ∣ μ, N) = (m _{1} , \dots , m _{K} N) \prod_{k = 1}^{K} μ_{k}^{m_{k}}$
  - $(\frac{N}{m_{1}, \dots, m_{K}}) = \frac{N!}{m_{1}! \dots m_{K}!}$
  - $\sum_{k = 1}^{K} m_{k} = N$
The Dirichlet distribution
- $Dir (μ ∣ α) = \frac{Γ (α_{0})}{Γ (α_{1}) \dots Γ (α_{K})} \prod_{k = 1}^{K} μ_{k}^{α_{k} - 1}$ $Dir (μ ∣ α) = \frac{Γ ( α _{0} )}{Γ ( α _{1} ) \dots Γ ( α _{K} )} \prod_{k = 1}^{K} μ_{k}^{α_{k} - 1}$
  - $α_{0} = \sum_{k = 1}^{K} α_{k}$
  - $0 \leq μ_{k} \leq 1$
  - $\sum_{k} μ_{k} = 1$
  - 잘 보면, 이 Dirichlet distribution도 multinomial의 실수영역에서의 확장임
posterior of multinomial likelihood & Dirichlet prior
- $p (μ ∣ D, α) \propto p (D ∣ μ) p (μ ∣ α) \propto \prod_{k = 1}^{K} μ_{k}^{α_{k} + m_{k} - 1}$
- $p (μ ∣ D, α) = Dir (μ ∣ α + m) = \frac{Γ (α_{0} + N)}{Γ (α_{1} + m_{1}) \dots Γ (α_{K} + m_{K})} \prod_{k = 1}^{K} μ_{k}^{α_{k} + m_{k} - 1}$ $p (μ ∣ D, α) = Dir (μ ∣ α + m) = \frac{Γ ( α _{0} + N )}{Γ ( α _{1} + m _{1} ) \dots Γ ( α _{K} + m _{K} )} \prod_{k = 1}^{K} μ_{k}^{α_{k} + m_{k} - 1}$
  - $m = (m_{1}, \dots, m_{K})^{T}$
- posterior가 Dirichlet임을 확인할 수 있고, Dirichlet prior가 multinomial likelihood의 conjugate prior임을 확인할 수 있음