Pattern Recognition on Amit Rajan

Mixture Models and Expectation Maximization - The EM Algorithm in General

Fri, 18 Nov 2022 23:07:28 +0100

9.4 The EM Algorithm in General

The expectation maximization (EM) algorithm is a general technique for finding maximum likelihood solutions for probabilistic models having latent variables. Consider a probabilistic model in which observed variable is denoted by \(X\) and the hidden variable by \(Z\). The joint distribution \(p(X,Z|\theta)\) is governed by a set of parameters deonoted by \(\theta\). Our goal is to maximize the likelihood function given by

\[\begin{align} p(X|\theta) = \sum_{Z} p(X,Z|\theta) \end{align}\]

For a continuous \(Z\), summation is replaced by integration. Direct optimization of \(p(X|\theta)\) is difficult but the optimization of comlete-data likelihood function \(p(X,Z|\theta)\) is somewhat easier. A distribution \(q(Z)\) over the latent variable is introduced and we observe that for any choice of \(q(Z)\), following decomposition holds true.

Mixture Models and Expectation Maximization - An Alternative View of EM

Fri, 04 Nov 2022 23:07:28 +0100

9.3 An Alternative View of EM

The goal of the EM algorithm is to find the maximum likelihood solutions for the models having latent variables. The set of all observed data is denoted by \(X\) where \(n^{th}\) row represents \(x_n^T\) and the latent variables by \(Z\) where the \(n^{th}\) row represets \(z_n^T\). Let the set of all model parameters be denoted by \(\theta\). Then, we have

\[\begin{align} \ln p(X|\theta) = \ln \bigg[ \sum_Z p(X,Z|\theta) \bigg] \end{align}\]

The expression will be valid for continuous variable with summation replaced with integration. The presence of summation inside the logarithm prevents the logarithm to directly act on the joint distribution and resulting in complicated expression for maximum likelihood solution. For each observation in \(X\), we have a correspondoing value of latent variable \(Z\). The data set \(\{X,Z\}\) is called as the complete data set and maximization of log likelihood function for the complete data set \(\ln p(X,Z|\theta)\) is straightforward.

Mixture Models and Expectation Maximization - Mixtures of Gaussians

Tue, 18 Oct 2022 23:07:28 +0100

9.2 Mixtures of Gaussians

This section describes the formulation of Gaussian mixtures in terms of discrete latent variables. The gaussian mixture distribution can be written as a linear superposition of Gaussian in the form

\[\begin{align} p(X) = \sum_{k=1}^{K} \pi_k N(X|\mu_k, \Sigma_k) \end{align}\]

Let us introduce a \(K-\) dimensional binary random variable \(z\) having a \(1-of-K\) representation in which a particular element \(z_k\) satisfies \(z_k \in \{0,1\}\) and \(\sum_k z_k = 1\). The joint distribution \(p(X,z)\) can then be represented in terms of the marginal distribution \(p(z)\) and conditional distribution \(p(X|z)\). The marginal distribution is given as

Mixture Models and Expectation Maximization - K-means Clustering

Tue, 11 Oct 2022 23:07:28 +0100

9.1 K-means Clustering

Let we have a data set \(\{X_1,X_2,...,X_N\}\) consisting of \(N\) observations of random \(D-\) dimensional Euclidean variable \(X\). Our goal is to partition the data set into some number \(K\) of clusters. Let the cluster \(k\) is represented by a \(D-\) dimensional vector \(\mu_k\) where \(k=1,2,...,K\). Our goal is then the assignment of data points to clusters and find the set of vectors \(\{\mu_k\}\) such that the sum of the squares of the distanaces of each data point to its closets vector \(\mu_k\) (or from the center of the assigned cluster), is a minimum.

Graphical Models - The Sum-product Algorithm, The Max-Sum Algorithm

Tue, 04 Oct 2022 23:07:28 +0100

8.4.4 The Sum-product Algorithm

Let us assume that all the variables in a model are discrete and hence marginalization corresponds to performing sums. To use the same algorithm for all kind of graphs, we first convert the original graph into a factor graph so that we can deal with both directed and undirected model using the same framework. To find the marginal distribution \(p(x)\) for a particular variable node \(x\), we need to sum the joint distribution over all variables except \(x\) so that

Graphical Models - Inference in Graphical Models

Tue, 20 Sep 2022 23:07:28 +0100

8.4 Inference in Graphical Models

The problem of inference in graphical models, in which some of the nodes in a graph are clamped to the observed values, aims to compute the posterior distribution of one or more subset of nodes. The graphical structure can be exploited to find the efficient and transparent algorithms for inference. These algorithms usually can be expressed in terms of the propagation of local messages around the graph. Let us consider the graphical representation of Bayes theorem shown in the below figure.

Graphical Models - Markov Random Fields

Thu, 15 Sep 2022 23:07:28 +0100

8.3 Markov Random Fields

A Markov random field, also known as a Markov network or an undirected graphical model has a set of nodes each of which corresponds to a variable or grpup of variables and the undirected links each of which connects a pair of nodes.

8.3.1 Conditional Independence Properties

In a directed graph, d-separataion can be used to check the conditional independence property. By removing the directed links from the graph, the assymetry between parent and child nodes is removed and hence the logic of d-separation can’t be applied directly.

Graphical Models - Conditional Independence

Sun, 11 Sep 2022 23:07:28 +0100

8.2 Conditional Independence

Consider three variables \(a,b,c\) and suppose that the conditional distribution of \(a\) given \(b\) and \(c\) is such that it does not depend on value of \(b\), i.e.

\[\begin{align} p(a|b,c) = p(a|c) \end{align}\]

We say that \(a\) is conditionally independent of \(b\) given \(c\). This can be expressed in a different way if we consider the joint distribution of \(a\) and \(b\) conditioned on \(c\), which can be written as

Graphical Models - Bayesian Networks

Mon, 05 Sep 2022 23:07:28 +0100

Probabilistic models can also be analyzed using diagrammatic representations of probability distributions, called probabilistic graphical models. A graph comprises nodes (also called vertices) connected by links (also known as edges or arcs). Each node represents a random variable (or group of random variables), and the links express probabilistic relationships between these variables. Graph then captures the way in which the joint distribution over all of the random variables can be decomposed into a product of factors each depending only on a subset of the variables.

Sparse Kernel Methods - Maximum Margin Classifiers: Relation to Logistic Regression, Multiclass SVMs, SVMs for Regression

Sun, 28 Aug 2022 23:07:28 +0100

7.1.2 Relation to Logistic Regression

For nonseparable case of SVM, for the data points that are on the correct side of margin, we have \(\xi_n=0\) and hence \(t_ny_n \geq 1\). For the remaining points, we have \(\xi_n = 1 - t_ny_n\). The below objective function can then be written as

\[\begin{align} \frac{1}{2}||W||^2 + C\sum_{n=1}^{N}\xi_n = \lambda||W||^2 + \sum_{n=1}^{N}E_{SV}(t_ny_n) \end{align}\]

where \(\lambda = (2C)^{-1}\) and \(E_{SV}(.)\) is the hinge error function defined as

Sparse Kernel Methods - Maximum Margin Classifiers: Overlapping Class Distributions

Tue, 23 Aug 2022 23:07:28 +0100

7.1.1 Overlapping Class Distributions

In practice, the class-conditional distributions may overlap, in which case exact separation of the training data can lead to poor generalization. In maximum margin classifier for separable classes, we implicitly used an error function that gave infinite error if a data point was misclassified and zero error if it was classified correctly, and then optimized the model parameters to maximize the margin. We now modify this approach so that data points are allowed to be on the wrong side of the margin boundary, but with a penalty that increases with the distance from that boundary. It is convenient to make this penalty a linear function of this distance. To do this, we introduce slack variables \(\xi_n \geq 0\) for all \(n=1,2,...,N\), with one slack variable for each training data point. With a slack, some of the data points are allowed to be on the other side of margin and some are even allowed to be misclassified. The value of slack variables for different data points are as follows:

Sparse Kernel Methods - Maximum Margin Classifiers

Thu, 18 Aug 2022 23:07:28 +0100

7.1 Maximum Margin Classifiers

Let us consider a two-class classification problem using linear models of the form

\[\begin{align} y(X) = W^T\phi(X) + b \end{align}\]

where \(\phi(X)\) denotes a fixed feature-space transformation. Let the training data set comprises \(N\) input vectors \(X_1,X_2,...,X_N\), with corresponding target values \(t_1,t_2,...,t_N\) where \(t_n \in \{-1,1\}\), and new data points \(X\) are classified according to the sign of \(y(X)\).

We shall assume for the moment that the training data set is linearly separable in feature space, so that by definition there exists at least one choice of the parameters \(W\) and \(b\) such that the linear function satisfies \(y(X_n) > 0\) for points having \(t_n = 1\) and \(y(X_n) < 0\) for points having \(t_n = -1\), so that \(t_ny(X_n) > 0\) for all training data points.

Sparse Kernel Methods - Lagrange Multipliers

Sun, 14 Aug 2022 23:07:28 +0100

One of the significant limitations of learning algorithms based on non-linear kernels is that the kernel function \(k(X_n, X_m)\) must be evaluated for all possible pairs \(X_n\) and \(X_m\) of training points. For us, it will be favourable that we shall look at kernel-based algorithms that have sparse solutions, so that predictions for new inputs depend only on the kernel function evaluated at a subset of the training data points. One of the most used sparse solution is support vector machine (SVM). An important property of support vector machines is that the determination of the model parameters corresponds to a convex optimization problem, and so any local solution is also a global optimum. It should be noted that the SVM is a decision machine and so does not provide posterior probabilities.

Kernel Methods - Gaussian Process

Thu, 04 Aug 2022 23:07:28 +0100

6.4 Gaussian Process

For a linear regression models of the form \(y(X,W) = W^T\phi(X)\) in which \(W\) is a vector of parameters and \(\phi(X)\) is a vector of fixed nonlinear basis functions that depend on the input vector \(X\), we showed that a prior distribution over \(W\) induced a corresponding prior distribution over functions \(y(X,W)\). Given a training data set, we then evaluated the posterior distribution over \(W\) and thereby obtained the corresponding posterior distribution over regression functions, which in turn (with the addition of noise) implies a predictive distribution \(p(t|X)\) for new input vectors \(X\).

Kernel Methods - Constructing Kernels & Radial Basis Function Networks

Fri, 29 Jul 2022 23:07:28 +0100

6.2 Constructing Kernels

One way to construct a kernel is to choose a feature space mapping \(\phi(X)\) and then use this to find the corresponding kernel. Let \(x\) be a one dimensional input. The kernel function is then given as

\[\begin{align} k(x,x^{'}) = \phi(x)^T\phi(x^{'}) = \sum_{i=1}^{M} \phi_i(x)\phi_i(x^{'}) \end{align}\]

where \(\phi_i(x)\) are the basis functions.

An alternative approach is to construct kernel functions directly. In this case, we must ensure that the function we choose is a valid kernel, in other words that it corresponds to a scalar product in some (perhaps infinite dimensional) feature space. For example, consider a simple kernel function given as

Kernel Methods - Dual Representations

Mon, 25 Jul 2022 23:07:28 +0100

There is a calss of pattern recognition techniques, where the training data points or a subset of them are kept and used during the prediction phase. For example, in a simple technique for classification called as nearest neighbours, each new test sample is assigned the same label as the closest example form the training set. These type of prediction algorithms typically require a metric to be defined that measures the similarity of any two vectors in input space, and are generally fast to train but slow at making predictions for test data points.

Neural Networks - Mixture Density Networks & Bayesian Neural Networks

Tue, 19 Jul 2022 23:07:28 +0100

5.6 Mixture Density Networks

The goal of supervised learning is to model a conditional distribution \(p(t|X)\) which for many simple regression problems is chosen to be Gaussian. However, practical machine learning problems can often have significantly non-Gaussian distributions. The main problem arises when we have to solve the inverse problem.

Forward problems often corresponds to causality in a physical system and generally have a unique solution. For instance, a specific pattern of symptoms in the human body may be caused by the presence of a particular disease. In pattern recognition, however, we typically have to solve an inverse problem, such as trying to predict the presence of a disease given a set of symptoms. If the forward problem involves a many-to-one mapping, then the inverse problem will have multiple solutions. For instance, several different diseases may result in the same symptoms. When the underlying distribution is multimodal, the inverse problem is more difficult to solve making the solution less obvious. The effect is shown in the below figure.

Neural Networks - Regularization in Neural Networks

Sun, 17 Jul 2022 23:07:28 +0100

5.5 Regularization in Neural Networks

The number of input and outputs units in a neural network is generally determined by the dimensionality of the data set, whereas the number \(M\) of hidden units is a free parameter that can be adjusted to give the best predictive performance. Note that \(M\) controls the number of parameters (weights and biases) in the network, and so we might expect that in a maximum likelihood setting there will be an optimum value of \(M\) that gives the best generalization performance. One approach is to choose a relatively large value for \(M\) and then to control complexity by the addition of a regularization term to then error function. A quadratic regularizer is given as

Neural Networks - The Hessian Matrix

Fri, 15 Jul 2022 23:07:28 +0100

5.4 The Hessian Matrix

Backpropagation can also be used to evaluate the second derivatives of the error, given by

\[\begin{align} H = \frac{\partial^2 E}{\partial W_{ji}\partial W_{lk}} \end{align}\]

An important consideration for many applications of the Hessian is the efficienc with which it can be evaluated. If there are \(W\) parameters (weights and biases) in the network, then the Hessian matrix has dimensions \(W \times W\) and so the computational effort needed to evaluate the Hessian will scale like \(O(W^2)\) for each pattern in the data set. As we shall see, there are efficient methods for evaluating the Hessian whose scaling is indeed \(O(W^2)\).

Neural Networks - Error Backpropagation

Wed, 13 Jul 2022 23:07:28 +0100

5.3 Error Backpropagation

Our goal in this section is to find an efficient technique for evaluating the gradient of an error function \(E(W)\) for a feed-forward neural network. We shall see that this can be achieved using a local message passing scheme in which information is sent alternately forwards and backwards through the network and is known as error backpropagation, or sometimes simply as backprop.

Most training algorithms involve an iterative procedure for minimization of an error function, with adjustments to the weights being made in a sequence of steps. At each such step, we can distinguish between two distinct stages.

Neural Networks - Network Training

Mon, 11 Jul 2022 23:07:28 +0100

5.2 Network Training

A simple approach to the problem of determining the network parameters is to make an analogy with the discussion of polynomial curve fitting, and therefore to minimize a sum-of-squares error function. Given a training set comprising a set of input vectors \(\{X_n\}\), where \(n=1,2,...,N\) with the corresponding output set as \(\{t_n\}\), we minimize the error function

\[\begin{align} E(W) = \frac{1}{2}\sum_{n=1}^{N}||y_n(X_n,W) - t_n||^2 \end{align}\]

Let us consider a regression case with single target variable \(t\) that can take any real value. We assume that \(t\) has a Gaussian distribution with an \(X\)-dependent mean, which is given by the output of the neural network as

Neural Networks - Feed-forward Network Functions

Sat, 09 Jul 2022 23:07:28 +0100

One of the major limitations of the modeling techniques discussed so far is the way fixed basis functions are used to defind the transformation of data points. This approach leads to a much sparser models. An alternative approach is to fix the number of basis functions in advance but allow them to be adaptive, in other words to use parametric forms for the basis functions in which the parameter values are adapted during training. The most successful model of this type in the context of pattern recognition is the feed-forward neural network, also known as the multilayer perceptron. The resulting model is significantly more compact and hence faster to evaluate. The price to be paid for this compactness is that the likelihood function, which forms the basis for network training, is no longer a convex function of the model parameters.

Linear Models for Classification - The Laplace Approximation & Bayesian Logistic Regression

Wed, 06 Jul 2022 23:07:28 +0100

4.4 The Laplace Approximation

In the Bayesian treatment of logistic regression, we can not integrate exactly over the parameter \(W\) as the posterior distribution is no longer Gaussian. Laplace approximation aims to find a Gaussian approximation to a probability density defined over a set of continuous variables. Let for a single continuous variable \(z\), the distribution is defined as

\[\begin{align} p(z) = \frac{1}{Z}f(z) \end{align}\]

where \(Z = \int f(z)dz\) is the normalizing coefficient. In the Laplace method, the goal is to find a Gaussian approximation \(q(z)\) of \(p(z)\) which is centred on the mode of the distribution \(p(z)\). The first step is to find the model of \(p(z)\), i.e. the point \(z_0\) such that \(p^{'}(z_0) = 0\), or

Linear Models for Classification - Probabilistic Discriminative Models

Tue, 05 Jul 2022 23:07:28 +0100

4.3 Probabilistic Discriminative Models

For two-class classification problem, the posterior probability of classes are the logistic sigmoid transformation of a linear function of input \(X\). For multi-class classification problem, they are given by the softmax transformation of a linear function of input \(X\). In maximum likelihood solution, we chose the class-conditional densities and then maximized the log likelihood to obtain posterior densities. However, an alternative approach is to use the functional form of the generalized linear model explicitly instead and to determine its parameters directly by using maximum likelihood. In this approach, we are maximizing a likelihood function defined through the conditional distribution \(p(C_k|X)\), which represents a form of discriminative function. One advantage of the discriminative approach is that there will typically be fewer adaptive parameters to be determined. Unlike generative modeling, it can not be used to generate synthetic data as we won’t have the distributions \(p(X)\) and \(p(X|C_k)\).

Linear Models for Classification - Probabilistic Generative Models (Maximum Likelihood Solution)

Mon, 04 Jul 2022 23:07:28 +0100

4.2.2 Maximum Likelihood Solution

Once the class-conditional densities \(p(X|C_k)\) are expressed in a parametric form, the value of the parameter can be determinied using a maximum likelihood approach. This requires a data set having input \(X\) together with their class labels. Suppose we have a data set \(\{X_n,t_n\}\) where \(n=1,2,...,N\) and \(t_n=1\) for class \(C_1\) and \(t_n=0\) for class \(C_2\). Let the prior class probablities be \(p(C_1) = \pi\) and \(p(C_2) = 1-\pi\). The joint densities are given as

Linear Models for Classification - Probabilistic Generative Models

Sun, 03 Jul 2022 23:07:28 +0100

4.2 Probabilistic Generative Models

In the probabilistic generative modelling approach, we model the class-conditional densities \(p(X|C_k)\), as well as the class priors \(p(C_k)\), and then use these to compute posterior probabilities \(p(C_k|X)\) through Bayes’ theorem. The posterior probability for class \(C_1\) is

\[\begin{align} p(C_1|X) = \frac{p(X|C_1)p(C_1)}{p(X)} = \frac{p(X|C_1)p(C_1)}{p(X|C_1)p(C_1) + p(X|C_2)p(C_2)} \end{align}\]\[\begin{align} = \frac{1}{1+exp(-a)} = \sigma(a) \end{align}\]

where

\[\begin{align} a = \ln \frac{p(X|C_1)p(C_1)}{p(X|C_2)p(C_2)} \end{align}\]

and \(\sigma(a)\) is called as the logistic sigmoid function. It is a S-shaped function and also called as squashing function as it maps the entire real axis into a finite interval. It satisfies: \(\sigma(-a) = 1 - \sigma(a)\). The inverse of the function is given as

Linear Models for Classification - The Perceptron Algorithm

Sat, 02 Jul 2022 23:07:28 +0100

4.1.7 The Perceptron Algorithm

In a perceptron algorithm, input vector \(X\) is first transformed using a fixed nonlinear transformation to get a feature vector \(\phi(X)\), and this is then used to construct a generalized linear model of the form

\[\begin{align} y(X) = f(W^T\phi(X)) \end{align}\]

where the nonlinear activation function is given as

\[\begin{align} f(a) = \begin{cases} +1, & a \geq 0\\ -1, & a < 0 \end{cases} \end{align}\]

Here we will use a target coding scheme of \(\{-1,+1\}\), where \(t_n=+1\) for \(C_1\) and \(t_n=-1\) for \(C_2\). A natural choice of error function would be the total number of misclassified patterns. But this error function is piecewise constant funtion of \(W\) and hence the gradient with respect to \(W\) will be \(0\) almost everywhere. Insted, we can use an alternative error function known as perceptron criterian. Here, we are looking for a weight vector \(W\) which, for class \(C_1\) satisfies \(W^T\phi(X_n) > 0\) and for class \(C_2\) satisfies \(W^T\phi(X_n) < 0\). Hence, when combined together, it should satisfy \(W^T\phi(X_n)t_n < 0\). The perceptron criterian associates \(0\) error with the inputs that are correctly classified. For misclassified inputs, it tries to minimize \(-W^T\phi(X_n)t_n\). The perceptron criterian is hence given as

Linear Models for Classification - Fisher’s Linear Discriminant

Fri, 01 Jul 2022 23:07:28 +0100

4.1.4 Fisher’s Linear Discriminant

Linear classification model can be viewed as projecting the \(D\)-dimensional data onto a one-dimensional space. The equation \(y=W^TX\) projects the \(D\)-dimensional input vector on a one dimensional space. Projection onto one dimension leads to a considerable loss of information and classes that are well-separated in the \(D\)-dimensional space may become overlapping in the one dimensional space. The goal of the classification problem is to adjust the weight \(W\) so that we can have the projection that maximizes the separation.

Linear Models for Classification - Least Squares for Classification

Thu, 30 Jun 2022 23:07:28 +0100

4.1.3 Least Squares for Classification

For the regression problem, minimizing the sum of squares error function led to a simple closed form solution for the parameters. We can check whether the same can be applied to the classification problem in hand. Consider a general classification problem with \(K\) classes with a \(1-of-K\) binary coding scheme for the target variable \(t\). Each class \(C_k\) is described by its own linear model given as

Linear Models for Classification - Discriminant Functions (Part 2)

Wed, 29 Jun 2022 23:07:28 +0100

4.1 Discriminant Functions

A discriminant is a function which takes the input vector \(X\) and assigns it to one of the classes \(C_k\). When the decision surfaces are hyperplanes, we call them as linear discriminants.

4.1.1 Two Classes

The simplest representation of linear discriminant function is

\[\begin{align} y(X) = W^TX + W_0 \end{align}\]

where \(W\) is called the weight vector and \(W_0\) is bias. The negative of the bias is called threshold. The input vector will be assigned to the class \(C_1\) if \(y(X) \geq 0\) and \(C_2\) otherwise. The decision boundary will be defined by the relation \(y(X) = 0\). Let \(X_A,X_B\) are the two points lying on the decision surface, then \(y(X_A) = y(X_B) = 0\) and hence \(W^T(X_A - X_B) = 0\). This means that the weight vector \(W\) is orthogonal to every vector in the decision surface, i.e. \(W\) determines the direction of the decision surface. Let \(X\) be a point on the decsion surface which is closest to the origin. Thie means that this point can be represented as \(X = \alpha W\) as the perpendicular weight vector \(W\) will be the direction in which we will have the smallest vector passing through origin and the hyperplace. As \(X\) lies on the hyperplace, we have \(y(X) = W^TX + W_0 = 0\), i.e. \(W^TX + W_0 = \alpha W^TW + W_0 = 0\) and hence \(\alpha = -W_0/||W||^2\). The distance of the point \(X\) from the origin is \(||X|| = ||\alpha W|| = \frac{-W_0}{||W||^2}||W|| = \frac{-W_0}{||W||}\). Hence, the normal distance of the hyperplane from the origin is given by \(\frac{-W_0}{||W||}\). This means that the bias parameter \(W_0\) decides the location of the decision surface.

Linear Models for Classification - Discriminant Functions

Tue, 28 Jun 2022 23:07:28 +0100

The goal of a calssification problem is to take the input vector \(X\) and assign it to \(K\) discrete classes \(C_k\) where \(k=1,2,3,...,K\). The input space is divided into decision regions whose boundaries are called as decision boundaries or decision surfaces. For linear models for classification, the decision surfaces are linear functions of the input vector \(X\). Hence, for a \(D\) -dimensional input space, decision surface will be a \((D-1)\) -dimensional hyperplane. Data sets whose classes can be separated exactly by linear decision surfaces are said to be linearlly separable.

Linear Models for Regression - Evidence Approximation & Limitations of Fixed Basis Function

Mon, 27 Jun 2022 23:07:28 +0100

3.5 The Evidence Approximation

In a fully Bayesian treatment of the linear basis function model, we would introduce prior distributions over the hyperparameters \(\alpha\) and \(\beta\) and make predictions by marginalizing with respect to these hyperparameters as well as with respect to the parameters \(W\). However, although we can integrate analytically over either \(W\) or over the hyperparameters, the complete marginalization over all of these variables is analytically intractable. Our goal is to find the predictive distribution for each of the models \(p(t|X,M_i)\). If we introduce hyperpriors(priors over hyperparameters) over \(\alpha,\beta\), the predictive distribution is obtained by marzinalizing over \(W,\alpha,\beta\) and is given as

Linear Models for Regression - Bayesian Model Comparison

Sun, 26 Jun 2022 23:07:28 +0100

3.4 Bayesian Model Comparison

Here we consider the problem of model selection from a Bayesian perspective. The Bayesian view of model comparison simply involves the use of probabilities to represent uncertainty in the choice of model. Suppose we wish to compare a set of \(L\) models \(\{M_i\}\) where \(i = 1,...,L\). Here a model refers to a probability distribution over the observed data \(D\). We shall suppose that the data is generated from one of these models but we are uncertain which one. Our uncertainty is expressed through a prior probability distribution \(p(M_i)\). Given a training set \(D\), we then wish to evaluate the posterior distribution

Linear Models for Regression - Bayesian Linear Regression

Sat, 25 Jun 2022 23:07:28 +0100

3.3 Bayesian Linear Regression

One of the problems with frequentist approach and using maximum likelihood estimator is the issue of deciding the appropriate model complexity for the particular problem, which cannot be decided simply by maximizing the likelihood function, because this always leads to excessively complex models and over-fitting. We therefore turn to a Bayesian treatment of linear regression, which will avoid the over-fitting problem of maximum likelihood, and which will also lead to automatic methods of determining model complexity using the training data alone.

Linear Models for Regression - Bias-Variance Decomposition

Thu, 23 Jun 2022 23:07:28 +0100

3.2 Bias-Variance Decomposition

We have seen that the use of maximum likelihood, or equivalently least squares, can lead to severe over-fitting if complex models are trained using data sets of limited size. However, limiting the number of basis functions in order to avoid over-fitting has the side effect of limiting the flexibility of the model to capture interesting and important trends in the data. The phenomenon of over-fitting is really an unfortunate property of maximum likelihood and does not arise when we marginalize over parameters in a Bayesian setting. From a frequentist point of view, the phenomena of overfitting can be analyzed using bias-variance trade-off.

Linear Models for Regression - Linear Basis Function Models : Part 2

Tue, 21 Jun 2022 23:07:28 +0100

3.1.2 Geometry of Least Squares

The least square problem can be geometrically interpreted as follows. Let the target veactor \(t\) (having output for \(N\) data points) spans a \(N\)-dimensional space. Each basis vector \(\phi_j(X_n)\) will also be present in the same space. If the number \(M\) of basis functions is smaller than the number of data points \(N\), then the \(M\) basis vectors will span a subspace \(S\) of dimensionality \(M\). The prediction \(y(X,W)\) is the linear combination of basis vectors, it will lie in the subspace \(S\). The least square solution corrsponds to the choice of \(W\) for which \(y(X,W)\) is closest to \(t\). The interpretation is shown in the below figure.

Linear Models for Regression - Linear Basis Function Models : Part 1

Mon, 20 Jun 2022 22:07:28 +0100

Linear regression model has the property of being linear functions of adjustable parameters. We can add more complexity in the linear regression models by taking linear combinations of a fixed set of nonlinear functions of the input variables, known as basis functions. In the modeling process, giveb \(x\), we have to predict \(t\) which can be predicted as \(y(x)\). Form a probabilistic prespective, we aim to model the predictive distribution \(p(t|x)\) as this expresses the uncertainty about the value of \(t\) for each value of \(x\). Linear models have significant limitations as practical techniques for pattern recognition, particularly for problems involving input spaces of high dimensionality.

Probability Distributions - Nonparametric Methods

Sun, 19 Jun 2022 20:07:28 +0100

2.5 Nonparametric Methods

Till now to model the data, we have choosen distributions which are governed by some fixed small number of parameters. This is called as parametric approach to density modeling. The chosen density might be a poor model of the distribution that generates the data, which can result in poor predictive performance. In nonparametric approach, we make fewer assumptions about the form of distribution.

One common nonparametric approach is histogram density models. In this approach, we partition \(x\) into bins of width \(\Delta_i\) and then count the number of observations falling in the bins (denoted as \(n_i\)). To have the normalized probability distribution, we can calculate the probability for eac bin as

Probability Distributions - The Exponential Family

Sat, 18 Jun 2022 19:07:28 +0100

2.4 The Exponential Family

Exponential family of distributions over \(X\) with parameters \(\eta\) is defined as

\[\begin{align} p(X|\eta) = h(X)g(\eta)exp[\eta^Tu(X)] \end{align}\]

Here \(\eta\) is the natural parameter of the distribution and \(u(X)\) is some function of \(X\). \(g(\eta)\) ensures that the distribution is normalized and satisfies

\[\begin{align} g(\eta) \int h(X)exp[\eta^Tu(X)] dX = 1 \end{align}\]

Bernoulli distribution is a common exponential distribution. It is given as

\[\begin{align} p(x|\mu) = Bern(x|\mu) = \mu^x(1-\mu)^{1-x} = exp[x\ln \mu + (1-x)\ln(1-\mu)] \end{align}\]\[\begin{align} = exp\bigg[x\ln \bigg(\frac{\mu}{1-\mu}\bigg) + \ln(1-\mu)\bigg] = (1-\mu) exp\bigg[x\ln \bigg(\frac{\mu}{1-\mu}\bigg)\bigg] \end{align}\]

Comparing this with the expression for the exponetial family of distribution, we get

Probability Distributions - The Gaussian Distribution: Part 5

Fri, 17 Jun 2022 19:07:28 +0100

2.3.8 Periodic Variables

Periodic variables usually repeats its behaviour after a set amount of time. Let us assume a periodic variable represented as an angle from the x-axis by \(D=\{\theta_1, \theta_2, ..., \theta_N\}\), where \(\theta\) is measured in radians. The mean and variance of these data points will depend on the choice of origin and the axis. To find the invariant measure of the mean, we denote these observations as the points on the unit circle and can be described as a two-dimensional unit vectors \(X_1,X_2,...,X_N\) where \(||X_n|| = 1\) for \(n=1,2,...,N\) as shown in the following figure.

Probability Distributions - The Gaussian Distribution: Part 4

Thu, 16 Jun 2022 19:07:28 +0100

2.3.6 Bayesian Inference for the Gaussian

We can use Bayesian treatment to derive the point estimates for the mean and variance of the Gaussian by introducing prior distributions over these parameters. For a single ranodm variable, let us suppose that the variance \(\sigma^2\) is known and we have to determine the mean \(\mu\) given \(N\) data points \(X=\{x_1,x_2,...,x_N\}\). Under the assumption of independence, the likelihood function is give as

\[\begin{align} p(X|\mu) = \prod_{n=1}^{N} p(x_n|\mu) = \frac{1}{(2\pi\sigma^2)^{N/2}} exp \bigg(-\frac{1}{2\sigma^2}\sum_{n=1}^{N}(x_n - \mu)^2\bigg) \end{align}\]

As the likelihood function takes the form of the exponential of a quadratic form in \(\mu\), we can choose a conjugate prior \(p(\mu)\) given by the Gaussian as

Probability Distributions - The Gaussian Distribution: Part 3

Wed, 15 Jun 2022 19:07:28 +0100

2.3.3 Bayes’ Theorem for Gaussian Variables

The applicatio of Bayes’ theorem in the Gaussian setting is when we have Gaussian marginal distribution \(p(X)\) and a Gaussian conditional distribution \(p(Y|X)\) and we wish to find the marginal and consitional distribution \(p(Y)\) and \(p(X|Y)\). It should be noted that the mean and covariace of the marginal distribution \(p(X)\) are constant with respect to \(X\). The covarince of the conditional distribution \(p(Y|X)\) is constant with respect to \(X\) but its mean is a linear function of \(X\). Hence, the marginal can consitional distribution can be represented as

Probability Distributions - The Gaussian Distribution: Part 2

Tue, 14 Jun 2022 19:07:28 +0100

2.3.1 Conditional Gaussian Distribution

If two sets of variables are jointly Gaussian, then the conditional distribution of one set conditioned on the othre is Gaussian. The marginal distribution of either set is also Gaussian. Let \(X\) is a \(D\)-dimensional vector with Gaussian distribution \(N(X|\mu,\Sigma)\). \(X\) is partitioned into two disjoint subsets \(X_a,X_b\). Without loss of generality we can assume thet \(X_a\) forms the first \(M\) components of \(X\) and \(X_b\) the remaining \(D-M\), such that

Probability Distributions - The Gaussian Distribution: Part 1

Mon, 13 Jun 2022 19:07:28 +0100

2.3 The Gaussian Distribution

Gaussian Distribution is a widely used model for the distribution of continous variables. For a single variable \(x\), the gaussian distribution is given as

\[\begin{align} N(x|\mu,\sigma^2) = \frac{1}{(2\pi\sigma^2)^{1/2}} exp \bigg[\frac{-1}{2\sigma^2} (x-\mu)^2\bigg] \end{align}\]

where \(\mu\) and \(\sigma^2\) are mean and variance respectively. For a \(D\) dimensional vector \(X\), the multivariate gaussian distribution takes the form

\[\begin{align} N(X|\mu,\Sigma) = \frac{1}{(2\pi)^{D/2}}\frac{1}{|\Sigma|^{1/2}} exp \bigg[\frac{-1}{2} (X-\mu)^T\Sigma^{-1}(X-\mu)\bigg] \end{align}\]

where \(\mu\) is a \(D\) dimensional mean vector and \(\Sigma\) is a \(D\times D\) dimensional covariance matrix with \(|\Sigma|\) being its determinant.

Probability Distributions - Multinomial Variables

Sun, 12 Jun 2022 17:07:28 +0100

2.2 Multinomial Variables

A multinomila variable can be in any of \(K\) states instead of just \(2\) (in case of a binary variable). For example, a multinomial variable having \(K=5\) states can be represented as \(x=(0,0,1,0,0)^T\) where it is in a state where \(x_3=1\). This vector will satisfy \(\sum_{k=1}^K = 1\). If we denote the probability of \(x_k=1\) by \(\mu_k\), the distribution of \(x\) is given as:

\[\begin{align} p(x|\mu) = \prod_{k=1}^{K} \mu_k^{x_k} \end{align}\]

where \(\mu = (\mu_1, \mu_2, ..., \mu_k)^T\) and \(\mu_k \geq 0\) and \(\sum_{k}\mu_k = 1\). This distribution can be considered as the generalization of bernoulli distribution for more than \(2\) outcomes. This distribution is normalized as (only one of the \(x_k=1\) for all \(x\)):

Probability Distributions - Binary Variables

Sat, 11 Jun 2022 14:07:28 +0100

One goal of pattern recognition is to model the probability distribution \(p(x)\) of a random variable \(x\) given the finite set of points \(x_1, x_2, ..., x_N\). This problem is known as density estimation. For simplicity, we can assume that the points are independent and identically distributed. There can be infinitely many distributions that can give rise to the given data points with any distribution that is non-zero at the points \(x_1, x_2, ..., x_N\) as a potential candidate.

Introduction - Information Theory

Fri, 10 Jun 2022 14:07:28 +0100

1.6 Information Theory

Some amount of information is recieved when we observe the value of a discrete random variable \(x\). If a highly improbable event has occured, the amount of information received is higher. For an event which was certain to happen, no amount of information is received. Hence, the measure of information \(h(x)\) will therefore depend on the probability distribution \(p(x)\) and will be a monotnic function of \(p(x)\). For two unrelated events \(x\) and \(y\), the information gain from observing both of them \(h(x,y) = h(x) + h(y)\) is sum of the information gained from each of them separately. From probability prespective, the joint probability of unrelated event is given as \(p(x,y) = p(x)p(y)\). Hence, the information gain will be a logarithmic function of \(p(x)\) and can be denoted as:

Introduction - Decision Theory

Sat, 04 Jun 2022 14:07:28 +0100

1.5 Decision Theory

The idea behind decision theorey is to convert the model probabilities (mainly for classification problem) to a decision. Given an input vector \(x\) with the corresponding output \(t\), the joint probability distribution \(p(x,t)\) will provide the complete summary of uncertainity associated with these variables. Determination of \(p(x,t)\) from the training data is a difficult problem and hence in a practical setting, we are more intersted in taking decisions based on probable value of \(t\) for a given \(x\). This intution sets the premise for the decision theory.

Introduction - Model Selection & Curse of Dimensionality

Sun, 29 May 2022 14:07:28 +0100

1.3 Model Selection

One of the commonly used technique for model selection is cross-validation. In a \(S-fold\) cross-validation, training set is divided into \(S\) equal subests where one of the subset is used for model vaidation and the remaining \(S-1\) sets will be used for model training. This means that a total fraction of \(\frac{S-1}{S}\) of dataset is used for model training. For a scarce dataset, it is common practice to set \(S=N\), which is called as leave-one-out cross-validation technique. One of the major drawback of cross-validation is that we need to run \(S\) training runs for \(S\) subsets. The ideal model validation technique should rely only on training data and allow the comparison of multiple hyperparameters and model types in a single training run.

Introduction - Probability Theory

Mon, 23 May 2022 14:07:28 +0100

1.2 Probability Theory

Consider two random variables \(X,Y\) where \(X\) can take any of the values \(x_i\) where \(i=1,2,...,M\) and \(Y\) can take the values \(y_j\) where \(j=1,2,...,L\). In a total of \(N\) trials, both \(X,Y\) are sampled and the number of trials for which \(X=x_i,Y=y_j\) is \(n_{ij}\). The number of trials in which \(X=x_i\) is \(c_i\) and \(Y=y_j\) is \(r_j\) respectively. Then, the joint probability of \(X\) taking the value \(x_i\) and \(Y\) taking the value \(y_j\) is given as:

Introduction - Polynomial Curve Fitting

Fri, 20 May 2022 14:07:28 +0100

Any pattren recognition or machine learning task can be primarily divided into two categories: Supervised and Unsupervised Learning. In a supervised machine learning problem, we have the input and corresponding desired output. For any supervised learning problem, the aim of the pattern recognition algorithm is to come up with an algorithm or model which can predict the output given the input. Based on the output, the supervised learning problem can be divided into two categories: Classification (when we have a finite number of discrete output) and Regression (if the desired output consists of one or more continuous variables). For an unsupervised learning problem, we don’t have the desired output for the input variables. The goal of pattern recognition is: to discover groups of similar examples within the data (clustering), to determine the distribution of data within the input space (density estimation), to project the data from a high-dimensional space to low-dimensional space etc..