Example of Singular Value Decomposition
This is a similar example to the one we did in class. Unfortunately the example we did in class did not save (the notebooks should save automatically but something went wrong). I do not remember the exact values I used in class, but I will try to make my matrix close to the one from class.
Start with a matrix or rank 2. The third row is simply the sum of the first and second:
Calculate the s.v.d. and save the results for future use:
The vector S
contains the singular values:
The third singular value is very close to 0, it is surely a roundin/approximation error, and it should be 0. We can even fix it:
To create the matrix, we can do for example this:
Verify that this really worked:
They seem to be the same. Note that they are actually not exactly the same, due to rounding errors and numerical approximations. For example, while the first entry of the matrix shows as 2.0, it is actually a tiny bit off:
Because if that, we cannot actually compare the matrices:
We can see that few of the entries are exactly the same, but most of them a little bit different.
Fundamental spaces
The columns of and form bases of the four fundamental spaces of .
From the number of non-zero singular values we see that the rank of is 2.
Kernel
The last columns of form an orthonormal basis of the kernel of :
Multiplying by and should give us zero vectors:
Again, we see very tiny values that should be zeros, but are slightly off due to rounding and numerical approximations.
Range
The first two columns of form an orthonormal basis of the range of (a.k.a. the columns space of ):
Any vector in the range of can be written as a linear combination of and . In fact, since they form an orthonormal basis, we can find the coefficients of the linear combination simply using a dot product:
If we try this with a vector that is not in the range, it will not work:
This time is not , it is instead the orthogonal projection of onto the range of .
The other two spaces are the kernel of , spanned by the last column of , and the row space of , a.k.a. the range of , spanned by the first two columns of :
Pseudoinverse
Using the s.v.d. we can find the pseudoinverse of . It will be :
Faster way to get this is using the pinv
function:
Solving systems
We will use the pseudoinverse to find "solutions" of some systems. Start with
In matrix form this is
We already know that the vector on the right hand side is in the range of (it is the vector above), so this system will have a solution. Since the dimension of the kernel of is 2, this system will have infinitely many solutions, in fact, a two-parametric family of solutions.
To check that this is a solution of the system, all we need to do is multiply it by :
Out of all the solutions that the system has, is the one with the lowest norm, and also the unique one that is in the row space of . To see that it really is in the row space of , we will try to write it as a linear combination of and which form an orthonormal basis of the rowspace. Again, since the basis is orthonormal, we can get the coefficients using dot products:
We see that , so is in the row space of .
To get other solutions, we can add any vector from the kernel of to . Since the kernel of is spanned by and , we can get any solution of the system as
for any choice of the scalars and . For example,
will also be a solution:
Let's now look at a system that does not have a solution:
In matrix form this is
We already know that the vector on the right hand side is not in the range of (it is the vector above), so this system will not have a solution. Instead, we will find the "least squares solution", which in some sense is the best approximation to a solution.
We can see that this is not a solution of the system , since we do not get but only something close to . In fact, when comparing this to the result we obtained above when we introduced in the first place, we can see that is in fact the orthogonal projection of onto the range of . In other words, the closest vector in the range of to .
Again, this is the least square solution with the smallest norm, or the one that is in the row space of . We can find other least squares solutions of this system by adding linear combinations of and .
Systems with no solutions:
If we wanted to find other right hand sides for the system so that the system either has solutions or has no solutions, we can do this:
If we want the system to have solutions, we need to take the right hand side from the range of . The range is generated by the first two columns of , which we called and . So for example with the right hand side
the system has infinitely many solutions.
On the other hand, if we make the right hand side a linear combination of the columns of in which the third column, which we called , has a non-zero coefficient, the resulting system will have no solutions: