25.1 Similar Matrices
One of the major generator of a positive definite matrix is when we square one. In a nutsheel, the matrix $A^TA$ is always positive definite. To prove this, let $A$ be a $m \times n$ rectangular matrix. Then $A^TA$ is a square symmetric matrix. If we evaluate the expression $x^T(A^TA)x$, we get $x^T(A^TA)x = (x^TA^T)(Ax) = (Ax)^T(Ax) = |Ax|^2 > 0$ if $Ax$ is non-zero. Another important thing to note is if matrix $A, B$ are positive definite, $A+B$ is positive definite.
Matrices $A$ and $B$ are similar if for some invertible matrix $M$, $B=M^{-1}AM$. One of the examples is the eigen value matrix $\Lambda$. As we know that for any matrix $A$, $S^{-1}AS = \Lambda$. This means $A$ is similar to $\Lambda$ where $M=S$. Let’s take an example, where
$$\begin{align} A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix};\Lambda = \begin{bmatrix} 3 & 0 \\ 0 & 1 \end{bmatrix} \end{align}$$
We can find a lot of values of $M$ and hence $B$ and one of it is as follows:
$$\begin{align} \begin{bmatrix} 1 & -4 \\ 0 & 1 \end{bmatrix}\begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}\begin{bmatrix} 1 & 4 \\ 0 & 1 \end{bmatrix}=\begin{bmatrix} -2 & -15 \\ 1 & 6 \end{bmatrix} = B \end{align}$$
The most important fact about similar matrices is that they have the same eigenvalues.
25.2 Why Similar Matrices have same Eigenvalues?
Let $A$ and $B$ are similar matrices and $\lambda$ is one of the eigen values of $A$. Then, $Ax = \lambda x$ and $B = M^{-1}AM$. We can write $Ax$ as $Ax = AIx = AMM^{-1}x = \lambda x$. Multiplying by $M^{-1}$ on the both side, we get $M^{-1}AMM^{-1}x = \lambda M^{-1}x \implies (M^{-1}AM)M^{-1}x = \lambda M^{-1}x \implies BM^{-1}x = \lambda M^{-1}x \implies B(M^{-1}x) = \lambda (M^{-1}x)$. Hence $\lambda$ is alse an eigenvalue of $B$ with $M^{-1}x$ being it’s eigenvector. Hence, Similar Matrices have the same eigenvalues with the eigenvectors moved around.
25.3 Non-diagonalizable Matrix and Jordan Form
Let the eigenvalues of $A$ are same. For example, let us say $\lambda_1 = \lambda_2 = 4$. In this case, one family of similar matrices is just the matrix $\begin{bmatrix} 4 & 0 \\ 0 & 4 \end{bmatrix}$ as for all $M$, $M^{-1}\begin{bmatrix} 4 & 0 \\ 0 & 4 \end{bmatrix}M = \begin{bmatrix} 4 & 0 \\ 0 & 4 \end{bmatrix}$.
Another family of similar matrices will have matrices in the form of $\begin{bmatrix} 4 & 1 \\ 0 & 4 \end{bmatrix}, \begin{bmatrix} 4 & 0 \\ 17 & 4 \end{bmatrix}, \begin{bmatrix} 5 & 1 \\ -1 & 3 \end{bmatrix}$, where the first matrix representing the simplest (the best we can get to a diagonal matrix) matrix in the family. This matrix is called in the Jordan Form. A matrix in the Jordan Form can be divided into Jordan Blocks and each Jordan Block has one eigenvalue in it. In a nutshell, every square matrix $A$ is similar to a Jordan Matrix $J$ with $n$ Jordan Blocks where $n$ is the number of eigenvalues of $A$, as we have one eigenvalue per Jordan Block. When the matrix $A$ is diagonalizable, i.e. it has distinct eigenvalues, then $J = \Lambda$.