## Projection onto Vector To project vector $v$ onto vector $w$, we have: $\text{proj}_{w}v=\frac{ww^{\mathrm{T}}}{w^{\mathrm{T}}w} v$ We can write this as the projection matrix $P = \frac{ww^{\mathrm{T}}}{w^{\mathrm{T}}w}$ ### Projection onto Subspace To find the orthogonal projection of vector $w$ onto subspace $\mathcal{V}$, i.e. find $v \in \mathcal{V}$ such that $(w-v)\perp u$ for all $u \in \mathcal{V}$, we have: $v = \text{proj}_{\mathcal{V}}w =\left( \sum_{i=1}^{k}v_{i}v_{i}^{\mathrm{T}} \right)w$ where $\{v_{1}, \dots v_{k}\}$ is the orthonormal basis of $\mathcal{V}$. This transform can be as a symmetric orthogonal projection matrix $P = \sum_{i=1}^{k}v_{i}v_{i}^{\mathrm{T}}$. ### Least Squares (Overdetermined) When solving $Ax = b$ when $A$ is overdetermined, i.e. more equations than unknowns, we can project $b$ onto the column space of $A$ to get the closest solution: ${} Ax = \text{proj}_{C(A)}b$ The solution to this is: ${} x = (A^{\mathrm{T}}A)^{-1}A^{\mathrm{T}}b$ ### Minimal Norm (Underdetermined) $x^{\star} = A^{\mathrm{T}}(AA^{\mathrm{T}})^{-1}b$ Regularization: $x^{\star}=(A^{\mathrm{T}}A+\lambda I)^{-1}A^{\mathrm{T}}b$ $\lambda$ gives us a tradeoff between the error and the norm of the solution ### Finding Orthonormal Bases (Gram-Schmidt) The orthonormal basis $\mathcal{W}$ of the subspace spanned by a set of linearly independent vectors $\mathcal{V} = \{v_{1}, \dots v_{k}\}$ can be derived by: $w_{1} = \lvert v_{1} \rvert$ $w_{2} = \lvert v_{2}-\text{proj}_{w_{1}}v_{2} \rvert$ $w_{3} = \lvert v_{3} - \text{proj}_{w_{1}}v_{3}-\text{proj}_{w_{2}}v_{3} \rvert$ … $w_{k}=\left\lvert v_{k}-\sum_{i=1}^{k-1}\text{proj}_{w_{i}}v_{k} \right\rvert$