Numerical methods to solve systems of equations in Python

Contents
Introduction: ................................................................................................................................................. 1
Conjugate gradient method .......................................................................................................................... 2
Definitions and methodology ................................................................................................................... 2
Python code .............................................................................................................................................. 2
Jacobi method ............................................................................................................................................... 3
Methodology............................................................................................................................................. 3
Python code .............................................................................................................................................. 4
Jacobi method for diagonal dominant matrix .............................................................................................. 5
Definitions and methodology ................................................................................................................... 5
Python code .............................................................................................................................................. 5
Gauss Seidel Method .................................................................................................................................... 7
Methodology............................................................................................................................................. 7
Python code .............................................................................................................................................. 7
Jacobi relaxation method............................................................................................................................ 11
Methodology........................................................................................................................................... 11
Python code ............................................................................................................................................ 11

We will make frequent use of the libraries numpy.linalg and numpy (for using norm, dot operator, inv,
transpose).

Introduction:
All the 5 methods presented below are iterative methods.

When for the system 𝐴𝑥 = 𝑏, the matrix 𝐴 is not symmetric and pos. definite we can turn this system
into 𝐴𝑡 𝐴𝑥 = 𝐴𝑡 𝑏 and it becomes one.

Hence most of the algorithms presented below are for symmetric matrices (except for Jacobi relaxed
method).

Also the norm I will be working with in considering the error terms will be the 2-norm, induced by the
classical dot product between two vectors, or between a matrix and a vector.
Conjugate gradient method
Definitions and methodology
1. We say that two directions 𝑢, 𝑣 are conjugate over A if < 𝑢, 𝐴𝑣 > = 0 where < 𝑢, 𝑣 > is the scalar
product of two vectors.

2. Problem:

Suppose we want to solve the system 𝐴𝑥 = 𝑏 where 𝐴 is positive definite,real and symmetric.

The bilinear form < 𝑢, 𝑣 >𝐴 =< 𝑢, 𝐴𝑣 > defines a scalar product over 𝑅 𝑛 and if 𝑃 = {𝑝1 , … , 𝑝𝑛 } forms a
basis over 𝑅 𝑛 then we may express the solution 𝑥 ∗ = ∑𝑛𝑖=1 𝛼𝑖 𝑝𝑖 .

Based on this expansion we calculate:

𝑛

𝐴𝑥 = ∑ 𝛼𝑖 𝐴𝑝𝑖 , 𝑝𝑘𝑇 𝐴𝑥∗

∗

𝑖=1
𝑛

= ∑ 𝛼𝑖 𝑝𝑘𝑇 𝐴𝑝𝑖 ⇒ 𝑝𝑘𝑇 𝑏

𝑖=1
𝑛
< 𝑝𝑘 , 𝑏 >
= ∑ 𝛼𝑖 < 𝑝𝑘 , 𝑝𝑖 >𝐴 ⇒< 𝑝𝑘 , 𝑏 >= 𝛼𝑘 < 𝑝𝑘 , 𝑝𝑘 >𝐴 ⇒ 𝛼𝑘 = .
< 𝑝𝑘 , 𝑝𝑘 >𝐴
𝑖=1

This gives the following method for solving the equation Ax = b: find a sequence of n conjugate
directions, and then compute the coefficients 𝛼𝑘 .

The corresponding algorithm:

S1) We read 𝐴 ∈ 𝑀𝑚 (𝑅),symmetric and positive definite matrix, 𝑏 ∈ 𝑅 𝑚 , 𝑥 0 ∈ 𝑅 𝑚 .

S2) We compute 𝑟 (0) = 𝑏 − 𝐴𝑥 (0) , 𝑝(0) = 𝑟 (0) .

S3) For 𝑖 ≥ 0,we compute 𝑥 (𝑖+1) , 𝑟 (𝑖+1) , 𝑝(𝑖+1) ∈ 𝑅 𝑚 , 𝛼𝑖 , 𝛽𝑖 ∈ 𝑅 such that:

<𝑝(𝑖) ,𝑟 (𝑖) > <𝑟 (𝑖+1) ,𝐴𝑝(𝑖) >
𝛼𝑖 = , 𝑥 (𝑖+1) = 𝑥 (𝑖) + 𝛼𝑝(𝑖) , 𝑟 (𝑖+1) = 𝑟 (𝑖) − 𝛼𝑖 𝐴𝑝(𝑖) , 𝛽𝑖 = − , 𝑝 (𝑖+1) = 𝑟 (𝑖+1) +
<𝑝(𝑖) ,𝐴𝑝(𝑖) > <𝑝(𝑖) ,𝐴𝑝(𝑖) >
𝛽𝑖 𝑝(𝑖) until 𝛼𝑖 = 0 or 𝑝(𝑖+1) = 0.

S4) Display the solution 𝑥 (𝑖+1) .

Python code
First I am testing the functionality of the aforementioned libraries.

def tests_norms():
def test1():
A = np.array([[1,2],[2,3]])
print(np.linalg.norm(A))
"Computes Frobenius norm"
print(np.sqrt(1**2+2**2+3**2+4**2))
print(np.linalg.norm(A,2))
"Computes 2-norm"
 test1()
def test2():
"For inverse of a matrix one can use directly the linalg package "
"inv member"
A = np.array([[1,2],[2,3]])
print(np.linalg.inv(A))
Now we get on with our algorithms and tests.

def tests_CGM():
def test(A,b,x0):
"I test the first step of the Conjugate Gradient Method"
r0 = b-np.dot(A,x0)
p0 = r0
alpha0 = np.dot(p0,r0)/np.dot(p0,np.dot(A,p0))
x= x0
x = x + alpha0*p0
print(x)
def ttest():
A = np.array([[2,1],[1,4]])
b = np.array([1,2])
x0 = np.array([1,1])
test(A,b,x0)
ttest()
def CGM(A,b,x0,tol):
err,y = 1,x0
r = b-np.dot(A,x0)
p=r
i=0
while (err>tol):
alpha = np.dot(p,r)/np.dot(p,np.dot(A,p))
z = y + alpha * p
r = r-alpha * np.dot(A,p)
beta = -np.dot(r,np.dot(A,p))/np.dot(p,np.dot(A,p))
p = r+beta * p
i=i+1
err = np.dot(z-y,z-y) #squared 2-norm of z-y
y=z
return y,err,i
def ttest3():
A = np.array([[10,7,8,7],[7,5,6,5],[8,6,10,9],[7,5,9,10]])
b = np.array([32,23,33,31])
x0 = np.zeros(4)
print(CGM(A,b,x0,10**(-10)))
Remark:
The errors are computed under the 2-norms.

Jacobi method
Methodology
We again seek to solve 𝐴𝑥 = 𝑏, 𝐴 ∈ 𝑀𝑚 (𝐶), 𝑏 ∈ 𝐶 𝑚 .
We consider 𝐵 = 𝐼 − 𝐴 so the system becomes (𝐼 − 𝐵)𝑥 = 𝑏 ⇒ 𝑥 = 𝐵𝑥 + 𝑏.

For any 𝑥 (0) ∈ 𝐶 𝑚 we define the sequence (𝑥 (𝑛) )𝑛∈𝑁 through the relation 𝑥 (𝑛+1) = 𝐵𝑥 (𝑛) + 𝑏, ∀𝑛 ∈ 𝑁.

The sequence 𝑥 (𝑛) is convergent iff 𝜌(𝐵) < 1.

Remark: We denote 𝜌(𝐵) = spectral radius = max{|𝜆1 |, |𝜆2 |, … , |𝜆𝑚 |}.

Python code
def jacobi_methods(num):
def spectral_radius(A):
eig = np.linalg.eig(A)[0]
abs_eig = np.abs(eig)
return np.max(abs_eig)
def spectral_radius2(A):
"spectral radius of I-A"
B = np.identity(np.shape(A)[0])-A
return spectral_radius(B)
def jacobi_method(A,b,x0,tol):
"A = square matrix,b=the free term col.vector, x0 = initial guess "
sz = np.shape(A)
if sz[0]!=len(b):
raise IndexError('the number of rows of A and length of b are not\
compatible')
x,i,err,B = x0,0,1,np.identity(np.shape(A)[0])-A
if spectral_radius(B)>=1:
print('The method is not convergent')
return False
else:
while (err>=tol and i<30):
z = b+np.dot(B,x)
i = i+1
err = np.dot(z-x,z-x)
x=z
return x,err,i
def test_jacobi():
A = np.array([[0.2,0.3],[0.1,0.2]])
b = np.array([0.2,0.3])
x0 = np.array([0.1,0.2])
tol = 0.01 * np.exp(-5)
print(jacobi_method(A,b,x0,tol))
def test_matrix():
A = np.array([[0.2,0.3],[0.1,0.2]])
B = np.identity(np.shape(A)[0])-A
print(spectral_radius(B))
#test_matrix() -- Uncomment these to be able to test the functions
#test_jacobi() -- from outside
dic = {1:spectral_radius,2:spectral_radius2,3:jacobi_method}
return dic[num]
Jacobi method for diagonal dominant matrix
Definitions and methodology
Definition1:

We say that the matrix 𝐴 = (𝑎𝑖𝑗 )1≤𝑖,𝑗≤𝑚 is diagonal dominant on lines if and only if |𝑎𝑖𝑖 | >
∑𝑚
𝑗=1,𝑗≠𝑖|𝑎𝑖𝑗 |, ∀1 ≤ 𝑖 ≤ 𝑚.

Similarly, 𝐴 = (𝑎𝑖𝑗 )1≤𝑖,𝑗≤𝑚 is diagonal dominant on columns if and only if |𝑎𝑗𝑗 | > ∑𝑚
𝑖=1,𝑖≠𝑗|𝑎𝑖𝑗 |, ∀1 ≤ 𝑗 ≤
𝑚.
Method of solving the system:

1.We consider 𝐷 = 𝑑𝑖𝑎𝑔(𝐴) ∈ 𝐺𝐿𝑚 (𝑅), 𝐵 = 𝐼 − 𝐷 −1 𝐴, 𝑏 = 𝐷 −1 𝑎.

2. The system becomes 𝐴𝑥 = 𝑎 ⇔ 𝐷 −1 𝐴𝑥 = 𝐷 −1 𝑎 ⇔ (𝐼 − 𝐵)𝑥 = 𝑏 ⇔ 𝑥 = 𝐵𝑥 + 𝑏.

Python code
Python code for testing the dominance of a matrix.

"Jacobi for diagonal dominant matrices"

import numpy as np
def tests(num):
def dominance_rows(A):
"I test if a matrix A is diagonal dominant on rows"
sz = np.shape(A)[0]
for i in range(sz):
if(np.abs(A[i,i])<=1/2*np.sum(np.abs(A[i]))):
return 0
return 1
def dominance_cols(A):
"I test if a matrix A is diagonal dominant on columns"
sz = np.shape(A)[1]
for i in range(sz):
if(np.abs(A[i,i]<=1/2 *np.sum(np.abs(np.transpose(A)[i])))):
return 0
return 1
def diagonal(A):
"Create a diagonal matrix out of A"
return np.diag(np.diag(A))
dic = {1:dominance_rows,2:dominance_cols,3:diagonal}
return dic[num]

def test_dominance():
A = np.array([[1,0,0,0.5],[1,2,0.3,0.6],[0,1,3,1.5],[0,1,1,-3]])
print(A)
print(tests(1)(A))
print(tests(2)(A))
print(1/2*np.sum(np.abs(A[3])))
print(np.abs(A[3,3]))
 print(np.abs(A[3,3])<=1/2*np.sum(np.abs(A[3])))
Now I get to the core algorithm.

def jacobi_functions():
def jacobi_matrix1(A,b):
"creates the matrix B = I - D^(-1)*A, and the new vector b=D^(-1)*A"
"for diagonal dominant matrix on rows"
D = np.diag(np.diag(A))
return np.identity(np.shape(A)[0]) - np.dot(np.linalg.inv(D),A),\
np.dot(np.linalg.inv(D),b)
def jacobi_matrix2(A):
"creates the matrix B = I - A*D^(-1)"
"in case of diagonal dominant matrix on columns"
D = np.diag(np.diag(A))
B = np.identity(np.shape(A)[0]) - np.dot(A,np.linalg.inv(D))
return B
def test_jacobi_matrix():
A = np.array([[1,2,3],[1,2,0],[0,1,1]])
b = np.array([1,1,1])
print(jacobi_matrix1(A,b))
print(jacobi_matrix2(A))
#test_jacobi_matrix()
def test_jacobi_matrix2():
h2 = tests(1)
A = np.array([[1,0,0,0],[0,2,0,0],[0,0,3,0],[0,0,0,4]])
B = np.diag(([1,2,3,4]))
print(A)
print(B)
print(h2(A)==1)
#test_jacobi_matrix2()
def jacobi_method(A,x0,a,tol):
sz = np.shape(A)[0]
if sz!=len(x0):
raise IndexError('The dimensions of A and x0 are not compatible')
h2,h3 = tests(1),tests(2)
if h2(A)==1:
"If A is dominant on rows"
B,b = jacobi_matrix1(A,a)
x,err,i = x0,1,0
while(err>=tol and i<30):
z = np.dot(B,x) + b
err = np.dot(z - x,z-x)
i = i+1
x=z
return x,err,i
elif h3(A)==1:
"If A is dominant on cols"
B = jacobi_matrix2(A)
x,err,i = x0,1,0
y = np.diag(np.diag(A))*x
 while(err>=tol and i<30):
z = np.dot(B,y) + a
err = np.dot(z-y,z-y)
i = i+1
y=z
return x,err,i
else:
raise ValueError('You better try another method for solving the system')
def test_jacobi_method():
A = np.diag([2,3,2,4])
b = np.array([1,2,3,4])
x0 = np.ones(4)
print(jacobi_method(A,x0,b,10**(-4)))
test_jacobi_method()

Gauss Seidel Method

Methodology
We note 𝐵 = 𝐼 − 𝐴, 𝐵 = 𝐿 + 𝑅, 𝐿 = (𝑙𝑖𝑗 )1≤𝑖,𝑗≤𝑚 where 𝑙𝑖𝑗 = 𝑏𝑖𝑗 , ∀1 ≤ 𝑗 < 𝑖 ≤ 𝑚 and 𝑙𝑖𝑗 = 0,otherwise.

Then det(𝐼 − 𝐿) = 1 so (𝐼 − 𝐿)−1 exists.

The system 𝐴𝑥 = 𝑏 ⇔ (𝐼 − 𝐵)𝑥 = 𝑏 ⇔ (𝐼 − 𝐿 − 𝑅)𝑥 = 𝑏 ⇔ (𝐼 − 𝐿)−1 (𝐼 − 𝐿 − 𝑅)𝑥 = (𝐼 − 𝐿)−1 𝑏 ⇔

(𝐼 − (𝐼 − 𝐿)−1 𝑅)𝑥 = (𝐼 − 𝐿)−1 𝑏 ⇔ 𝑥 − (𝐼 − 𝐿)−1 𝑅𝑥 = (𝐼 − 𝐿)−1 𝑏.

If we denote 𝐶 = (𝐼 − 𝐿)−1 𝑅 and 𝑐 = (𝐼 − 𝐿)−1 𝑏 then 𝐴𝑥 = 𝑏 ⇔ 𝑥 = 𝐶𝑥 + 𝑐.

Hence we can define, for 𝑥 (0) ∈ 𝐶 𝑚 ,the seq. 𝑥 (𝑛+1) = 𝐶𝑥 (𝑛) + 𝑐 ⇔ 𝑥 (𝑛+1) = 𝐿𝑥 (𝑛+1) + 𝑅𝑥 (𝑛) +
𝑏, ∀𝑛 ∈ 𝑁.

Remark on convergence of the sequence 𝒙(𝒏)

The sequence 𝑥 (𝑛) is convergent if and only if 𝜌(𝐶) < 1.

Python code
"Gauss Seidel method"
import numpy as np
def tests():
def test1():
l = list([[(i,j) for j in range(4)] for i in range(4)])
print(l)
m = list([x if x[0]>x[1] else 0 for x in l])
print(m)
def test2():
A = np.random.normal(0,1,(4,4))
B,C = np.zeros((4,4)),np.zeros((4,4))
for i in range(1,4):
for j in range(i):
B[i,j] = A[i,j]
 for i in range(0,4):
for k in range(i,4):
C[i,k] = A[i,k]
print(A)
print(B)
print(C)
print(np.zeros((4,4)))
test2()

def decomposition(A):
"A = L + R, where L contains the matrix A under its main diagonal"
sz = np.shape(A)[0]
L,R = np.zeros((sz,sz)),np.zeros((sz,sz))
for i in range(1,sz):
for j in range(i):
L[i,j] = A[i,j]
for i in range(0,sz):
for k in range(i,sz):
R[i,k]=A[i,k]
return L,R

def test_decomposition():
A = np.random.normal(0,1,(5,5))
print(A)
print(decomposition(A)[0])
print(decomposition(A)[1])

def gauss_seidel_method(A,b,x0,tol):
if np.linalg.det(A)==0:
raise Exception("The matrix is singular")
else:
sz = np.shape(A)[0]
B = np.identity(sz) - A
L,R = decomposition(B)
K = np.identity(sz) - L
C = np.dot(np.linalg.inv(K),R)
c = np.dot(np.linalg.inv(K),b)
x,err,i= x0,1,0
while (err>=tol and i<30):
z = np.dot(C,x)+c
err = np.dot(z-x,z-x)
i=i+1
x=z
return x,err,i

def test_gauss_seidel_method():
A1,A2 = np.diag([1,2,3,4]),np.diag([1,2,3,4])
b = np.ones(4)
x0 = np.zeros(4)
 A1[3,0],A1[0,3] = 2,2
print(gauss_seidel_method(A2,b,x0,0.000001))
#print(gauss_seidel_method(A1,b,x0,0.000001))
"We see that for diagonal matrices the method is not always converging"

"I will need to import from Jacobi the jacobi_methods(num) function containing"
"spectral radius function"

from Jacobi import jacobi_methods

def gs_spectral_radius(A):
B=np.identity(np.shape(A)[0])-A
L,R = decomposition(B)
K = np.identity(np.shape(A)[0])-L
C = np.dot(np.linalg.inv(K),R)
return jacobi_methods(1)(C)

def test_spectral_radius():
A = np.array([[1,2,-2],[1,1,1],[2,2,1]])
print(gs_spectral_radius(A))
B = np.diag([1,2,3,4])
print(gs_spectral_radius(B))

def gauss_seidel_method2(A,b,x0,tol):
"An improved version, which abandonds the task if the spectral radius "
" of the matrix C previously defined is greater or equal than 1"
sz = np.shape(A)[0]
B = np.identity(sz)-A
L,R = decomposition(B)
K = np.identity(sz) - L
C = np.dot(np.linalg.inv(K),R)
if np.linalg.det(A)==0:
raise Exception("The matrix is singular")
elif gs_spectral_radius(C)>=1:
raise Exception("The method will not converge, the spectral radius is >=1")
else:
c = np.dot(np.linalg.inv(K),b)
x,err,i = x0,1,0
while (err>=tol and i<30):
z=np.dot(C,x)+c
err = np.dot(z-x,z-x)
i = i+1
x=z
return x,err,i

def test_gauss_seidel_method2():
A1,b,x0 = np.diag([1,2,3,4]), np.ones(4),np.zeros(4)
print(gauss_seidel_method2(A1,b,x0,0.00001))
def norm_inf_gauss_tests(num):
def modified_ones(q,n):
"q is a vector of length that must be inferior to n"
"the result would be [q1,q2,...,qk,1,1,..,1]"
z = np.ones(n)
for i in range(len(q)):
z[i] = q[i]
return z
def test1():
q = [2.1,2.2,2.3]
x = modified_ones(q,5)
print(x)
def norm_inf(B):
"Cumulative scalar product"
"q1 = sum of elements of B[0],q2 = <(q1,1,1...,1),(B[1,0],...,B[1,m])>"
"q3 = <(q1,q2,1,1,...,1),(B[2,0],B[2,1],...,B[2,m])>"
sz = np.shape(B)[0]
q=[]
q.append(np.dot(np.ones(sz),B[0]))
for i in range(1,sz):
aux = q[0:i]
q.append(np.dot(modified_ones(aux,np.shape(B)[0]),B[i]))
return q
def test2():
B = np.array([[1,2],[3,-1]])
print(norm_inf(B))
"should return 3.0 and 8.0 if it is correctly implemented"
#test2()
def norm_inf_gauss(A):
"the return of the function will specify if Gauss-Seidel is applicable"
"If the maximum of the q's is <1 then Gauss-Seidel converges otherwise no"
B = np.identity(np.shape(A)[0]) - A
q = norm_inf(np.abs(B))
return q
dic = {1:modified_ones,2:norm_inf,3:norm_inf_gauss}
return dic[num]
Remark (Alternative sufficient condition for convergence):

Regarding the last function norm_inf_gauss I have to take into consideration the following:

Considering 𝑞1 = ∑𝑚 𝑖−1 𝑚
𝑗=1|𝑏1𝑗 |, 𝑞𝑖 = ∑𝑗=1|𝑏𝑖𝑗 |𝑞𝑗 + ∑𝑗=𝑖|𝑏𝑖𝑗 |, 2 ≤ 𝑖 ≤ 𝑚, 𝑞 = max 𝑞𝑖 , if 𝑞 < 1 then we
1≤𝑞≤𝑚
(𝑛) 𝑞 𝑞𝑛
have the following evaluations: ||𝑥 − 𝑥|| ≤ 1−𝑞
||𝑥 (𝑛) − 𝑥((𝑛 − 1))|| ≤ 1−𝑞 ||𝑥 (1) − 𝑥 (0) || .
∞ ∞ ∞

So to test if a system is convergent or not we can see if 𝑞 < 1.

Jacobi relaxation method
Methodology
Suppose we have a system A symmetric and positive definite.

If 𝐷 = 𝑑𝑖𝑎𝑔(𝐴). Instead of considering 𝐶 = 𝐼 − 𝐷 −1 𝐴, 𝑐 = 𝐷 −1 𝑎, we take for iterative purpose, 𝐶𝜎 =

𝐼 − 𝜎𝐷 −1 𝐴. 𝑐𝜎 = 𝜎𝐷 −1 𝐴. The system of equations𝐴𝑥 = 𝑎 becomes 𝜎𝐷 −1 𝐴𝑥 = 𝜎𝐷 −1 𝑎 ⇔ (𝐼 − 𝐶𝜎 )𝑥 =
𝑐𝜎 ⇒ 𝑥 = 𝐶𝜎 𝑥 + 𝑐𝜎 .
The optimal choice of 𝜎 is made as follows:
2
Let 𝜆1 ≥ 𝜆2 ≥ ⋯ ≤ 𝜆𝑚 eigenvalues of 𝐷 (−1) 𝐴. For 𝜎 = 𝜆 +𝜆
we can be sure of a convergent array.
1 𝑚

Python code
"Simultaneous Relaxation method"
import numpy as np
def relaxed_method():
def issymetric(A):
if np.all(np.transpose(A)==A):
return 1
else:
return 0
def is_pos_def(A):
if np.all(np.linalg.eig(A)[0]>0) and issymetric(A)==1:
return 1
else:
return 0
def opt_relax_param(A):
"Optimal relaxation parameter.Interest on positive-definite and "
"symmetric matrices"
eig = np.linalg.eig(A)[0]
m,M = np.min(eig),np.max(eig)
return 2/(m+M)
def chg_var(A,a):
"change of variables in relaxed method"
sz = np.shape(A)[0]
sigma = opt_relax_param(A)
D = np.diag(np.diag(A))
C = np.identity(sz) - sigma * np.dot(np.linalg.inv(D),A)
c = sigma*np.dot(np.linalg.inv(D),a)
return C,c
def test():
A = np.array([[5,3,2],[3,6,3],[2,3,5]])
a= np.array([1,2,3])
print(chg_var(A,a))
#test()
def solution1(A,a,x0,tol,n):
"n is the number of iterations"
"I assume matrix A fullfils all necessary conditions"
 x,err,i=x0,1,0
C,c = chg_var(A,a)
while (err>=tol and i<n):
z = np.dot(C,x) + c
err = np.dot(z-x,z-x)
i=i+1
x=z
return x,err,i
def solution(A,a,x0,tol,n):
"The matrix might not be symmetric so I transform the system into "
" A^tAx=A^ta"
if issymetric(A)!=1 or is_pos_def(A)!=1:
return solution1(np.dot(np.transpose(A),A),np.dot(np.transpose(A),a),x0,tol,n)
else:
return solution1(A,a,x0,tol,n)
def test_sol():
A = np.array([[2,-1],[1,3]])
a = np.array([1,2])
x0,tol,n = np.array([1,1]),0.0000000001,50
print(solution(A,a,x0,tol,n))
test_sol()