11.1 Orthogonal Vectors
Orthogonal means perpendicular. For two vectors $x,y$, they are orthogonal if and only if $x^Ty=0$. As per Pythagoras Theorem, the test for orthogonality is: $\lVert x \rVert^2 + \lVert y \rVert^2 = \lVert x+y \rVert^2$.
We can conclude the dot product condition from Pythoagoras Theorem as follows:
$$\begin{align} \lVert x \rVert^2 + \lVert y \rVert^2 = \lVert x+y \rVert^2 \\ x^Tx + y^Ty = (x+y)^T(x+y) \\ x^Tx + y^Ty = x^Tx + y^Ty + x^Ty + y^Tx \\ x^Ty + y^Tx = 0 \\ 2x^Ty = 0 \\ x^Ty = 0 \end{align}$$
One importnt thing to note is: zero vector is orthogonal to all the other vectors.
11.2 Orthogonal Subspaces
Subspace $S$ is orthogonal to Subspace $T$ if and only if: Every vector in subspace $S$ is orthogonal to every vector in $T$. Some of the examples of orthogonal subspaces are: Row Space $\perp$ Null Space, Column Space $\perp$ Null Space($A^T$).
11.1.1 Row Space $\perp$ Null Space:
Null Space consists of the solution vectors $x$ of the equation $Ax=0$. If we expand this equation, we get the following:
$$\begin{align} \begin{bmatrix} row1 \\ row2 \\ … \\ rowN \end{bmatrix} \begin{bmatrix} x \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \\ … \\ 0 \end{bmatrix} \end{align}$$
Expanding this equation using row times column multiplication method, we get: $[row1]^T[x]=0; [row2]^T[x]=0; … ;[rowN]^T[x]=0$. Row Space of the matrix will have these row vectors or linear combination of these row vectors ($c_1[row1] + c_2[row2] + … + c_N[rowN]$) in it. Combining these equations together we get: $(c_1[row1] + c_2[row2] + … + c_N[rowN])^T[x] = 0$. This means any vector in row space is perpendicular to $x$ as it’s dot product with $x$ is $0$.
Using similar arguments, we can prove that Column Space $\perp$ Null Space($A^T$). It should be noted that Row Space with Null Space carves the entire $n-dimensional$ space. Similarly, Column Space with Null Space of $A^T$ carves the entire $m-dimensional$ space.
Hence, to conclude:
- Null Space and Row Space are Orthogonal Complements in $\mathbb{R}^n$
- Null Space of $A^T$ and Column Space are Orthogonal Complements in $\mathbb{R}^m$
11.3 Solve $Ax=b$ when we have No Solution
Let $A$ be a $m \times n$ matrix such that $m > n$. Our goal is to solve $Ax=b$ when it has no solution. By solving this equation, we need to find the best solution in the given constraint.
Consider the matrix $A^TA$. One of the important property of $A^TA$ is that this matrix is a $n \times n$ symmetric matrix. If we nultiply the above equation by $A^T$, we get $A^TA \hat{x} = A^Tb$.
One of the examples of unsolvalbe $Ax=b$ is as follows:
$$\begin{align} \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 5 \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \end{bmatrix}= \begin{bmatrix} b_1 \\ b_2 \\ b_3 \end{bmatrix} \end{align}$$
Transforming this equation into the form $A^TA\hat{x} = A^Tb$, we get
$$\begin{align} \begin{bmatrix} 3 & 8 \\ 8 & 30 \end{bmatrix} \begin{bmatrix} \hat{x_1} \\ \hat{x_2} \end{bmatrix}= \begin{bmatrix} 1 & 1 & 1 \\ 1 & 2 & 5 \end{bmatrix} \begin{bmatrix} b_1 \\ b_2 \\ b_3 \end{bmatrix} \end{align}$$
If $A^TA$ is invertible, we can get the solution for $A^TA\hat{x} = A^Tb$. Two important properties of $A^TA$ are as follows:
- $N(A^TA) = N(A)$
- $Rank(A^TA) = Rank(A)$
- Using these properties, we can conclude that $A^TA$ is invertible if and only if $N(A)$ only has zero vectors in it, i.e. $A$ has independent columns.