Linear Models for Classification - Probabilistic Generative Models
4.2 Probabilistic Generative Models
In the probabilistic generative modelling approach, we model the class-conditional densities \(p(X|C_k)\), as well as the class priors \(p(C_k)\), and then use these to compute posterior probabilities \(p(C_k|X)\) through Bayes’ theorem. The posterior probability for class \(C_1\) is
\[\begin{align} p(C_1|X) = \frac{p(X|C_1)p(C_1)}{p(X)} = \frac{p(X|C_1)p(C_1)}{p(X|C_1)p(C_1) + p(X|C_2)p(C_2)} \end{align}\]\[\begin{align} = \frac{1}{1+exp(-a)} = \sigma(a) \end{align}\]where
\[\begin{align} a = \ln \frac{p(X|C_1)p(C_1)}{p(X|C_2)p(C_2)} \end{align}\]and \(\sigma(a)\) is called as the logistic sigmoid function. It is a S-shaped function and also called as squashing function as it maps the entire real axis into a finite interval. It satisfies: \(\sigma(-a) = 1 - \sigma(a)\). The inverse of the function is given as
\[\begin{align} a = \ln \bigg(\frac{\sigma}{1 - \sigma}\bigg) = \ln \frac{p(C_1|X)}{p(C_2|X)} \end{align}\]and is called as logit function or log odds. For a \(K>2\) class case, the posterior distribution is given as
\[\begin{align} p(C_k|X) = \frac{p(X|C_k)p(C_k)}{\sum_{j}p(X|C_j)p(C_j)} = \frac{exp(a_k)}{\sum_{j}exp(a_j)} \end{align}\]and is called as normalized exponential or softwax function. The quantity \(a_k\) is defined as
\[\begin{align} a_k = \ln p(X|C_k)p(C_k) \end{align}\]4.2.1 Continuous Inputs
For the continuous input case, let us assume that the class conditional densities are Gaussian where all classes share the same covariance matrix. Then the density for class \(C_k\) is given as
\[\begin{align} p(X|C_k) = \frac{1}{(2\pi)^{D/2}} \frac{1}{|\Sigma|^{1/2}} exp \bigg[-\frac{1}{2}(X-\mu_k)^T \Sigma^{-1} (X - \mu_k)\bigg] \end{align}\]For a two class case, the posterior distribution is given as
\[\begin{align} p(C_1|X) = \sigma(a) = \frac{1}{1+exp(-a)} \end{align}\]where
\[\begin{align} a = \ln \frac{p(X|C_1)p(C_1)}{p(X|C_2)p(C_2)} \end{align}\]Evaluating \(a\), we have
\[\begin{align} a = \ln \bigg(\frac{p(X|C_1)}{p(X|C_2)} \times \frac{p(C_1)}{p(C_2)}\bigg) = \ln \frac{p(X|C_1)}{p(X|C_2)} + \ln \frac{p(C_1)}{p(C_2)} \end{align}\]\[\begin{align} = \ln \frac{exp \bigg[-\frac{1}{2}(X-\mu_1)^T \Sigma^{-1} (X - \mu_1)\bigg]}{exp \bigg[-\frac{1}{2}(X-\mu_2)^T \Sigma^{-1} (X - \mu_2)\bigg]} + \ln \frac{p(C_1)}{p(C_2)} \end{align}\]\[\begin{align} = (\mu_1^T - \mu_2^T)\Sigma^{-1}X - \frac{1}{2}\mu_1^T\Sigma^{-1}\mu_1 + \frac{1}{2}\mu_2^T\Sigma^{-1}\mu_2 + \ln \frac{p(C_1)}{p(C_2)} \end{align}\]\[\begin{align} = W^TX + W_0 \end{align}\]Hence, the posterior distribution is given as
\[\begin{align} p(C_1|X) = \sigma(W^TX + W_0) \end{align}\]where
\[\begin{align} W = \Sigma^{-1}(\mu_1 - \mu_2) \end{align}\]\[\begin{align} W_0 = - \frac{1}{2}\mu_1^T\Sigma^{-1}\mu_1 + \frac{1}{2}\mu_2^T\Sigma^{-1}\mu_2 + \ln \frac{p(C_1)}{p(C_2)} \end{align}\]The prior probabilities only affect the bias parameter \(W_0\) and hence change in priors have the effect of making parallel shifts of thd decision boundary. For a \(K>2\) class case,
\[\begin{align} p(C_k|X) = \frac{exp(a_k)}{\sum_{j}exp(a_j)} \end{align}\]where
\[\begin{align} a_k(X) = W_k^TX + W_{k0} \end{align}\]\[\begin{align} W_k = \Sigma^{-1}\mu_k \end{align}\]\[\begin{align} W_{k0} = - \frac{1}{2}\mu_k^T\Sigma^{-1}\mu_k + \ln p(C_k) \end{align}\]If the shared covariance matrix assumption is not true, we will obtain quadratic functions of \(X\), giving rise to quadratic discriminant.