Projection of a Matrix
12.1 Projections in one-dimensional Space
Given two vectors \(a,b\) in a plane, the projection of \(b\) onto \(a\) is shown below. The projection \(p\) will be a multiple of \(a\) and the error vector \(e\) will be orthogonal to \(a\). Error vector can be denoted as: \(e = b - p\). As \(e \perp a\), we can say that: \(a^Te =0 \implies a^T(b - p) = 0 \implies a^T(b - xa) = 0 \implies xa^Ta = a^Tb \implies x = \frac{a^Tb}{a^Ta}\). Hence, \(p = ax = a\frac{a^Tb}{a^Ta}\).
Some of the observations about the projection \(p\) are as follows:
- If \(b\) is doubled, projection \(p\) is doubled
- Change in magnitude of \(a\) doesn’t affect projection \(p\)
- Projection \(p\) can be obtained by multiplying \(b\) by a projection matrix \(P\), where \(P = \frac{aa^T}{a^Ta}\). For a vector \(a\), \(a^Ta\) is a number and \(aa^T\) is a matrix.
- Column Space of \(P, C(P)\) is the line through \(a\)
- \(Rank(P) = 1\)
- Projection Matrix \(P\) is symmetric as \(P^T = (\frac{aa^T}{a^Ta})^T = \frac{(aa^T)^T}{a^Ta} = \frac{aa^T}{a^Ta} = P\)
- For projection matrix \(P\), \(P^2 = P\) as taking the projection of the projection will land us on the same vector \(p\).
12.1.1 Why take Projections?
For any \(m \times n\) matrix \(A\) such that \(m > n\), our goal is to solve \(Ax=b\) when it has no solution. The reason for \(Ax=b\) to have no solution is the fact that \(Ax\) will always be in the column space of \(A\) but \(b\) may not. In these cases, we can project \(b\) on the column space of \(A\) and solve for \(Ax=p\) instead where \(p\) is the projection of \(b\) on \(C(A)\).
12.2 Projections in multi-dimensional Space
Let \(A\) be a matrix representing a two-dimensional space with the basis as vectors \(a_1\) and \(a_2\), i.e. \(A = \begin{bmatrix}a_1 & a_2 \end{bmatrix}\). Let \(b\) be the vector whose projection needs to be taken on \(A\). The projection \(p\) will be in the plane and can be denoted as \(p=a_1\hat{x_1} + a_2\hat{x_2}\) or \(p=A\hat{x}\). The error \(e\), which is \(e = b-p = b-A\hat{x}\) is perpendicular to the plane.
Or, we can say that the error vector \(e\) is perpendicular to both the basis vectors \(a_1\) and \(a_2\), i.e. \(a_1^T(b-A\hat{x}) = 0\) and \(a_2^T(b-A\hat{x}) = 0\). In the matrix form, \(\begin{bmatrix}a_1^T \\ a_2^T\end{bmatrix}(b - A\hat{x}) = \begin{bmatrix}0 \\ 0\end{bmatrix}\) or \(A^T(b-A\hat{x}) = 0\).
This equation tells us that the error vector \(e = b-A\hat{x}\) is in the null space of \(A^T\). We know that the null space of \(A^T\) is perpendicular to the column space of \(A\). This means that the above derived equation is sound. The above equation can be rewriten as: \(A^TA\hat{x} = A^Tb\). Hence,
\[\begin{align} \hat{x} = (A^TA)^{-1}A^Tb \end{align}\]\[\begin{align} p = A\hat{x} = A(A^TA)^{-1}A^Tb \end{align}\]\[\begin{align} P = A(A^TA)^{-1}A^T \end{align}\]The projection matrix \(P\) holds the properties: \(P^T=P\) and \(P^2=P\).