ARTIFICIAL NEURAL NETWORKS-moduleIII

ARTIFICIAL NEURAL
NETWORKS
MODULE-3
Module-3.
Counter Propagation Networks: Kohonen layer - Training
the Kohonen layer - Pre initializing the weight vectors statistical properties Training the Grossberg layer - Full
counter propagation network - Application
INTRODUCTION
Perceptron Training.
Back Propagation Networks.
Self-organizing Maps & Counter Propagation.
Unsupervised Training.
Self-organized
clustering may be defined as a
mapping through which N-dimensional pattern space

can be mapped to into a smaller number of points in an
output space.
The
matching is achieved autonomously without
supervision. i.e., clustering is done in a self-organized

manner
The term self-organizing refers to the ability to learn

and organize information without being given the
correct answer.
Self-organized network perform unsupervised

learning.
Competitive Networks
When more than one neuron in the out put layer

fires, an additional structure is included in the
network so that the net is forced to trigger only one
neuron.
This mechanism is termed as competition. When one
competition is completed only one neuron in the
competing group will have a non zero out put.
The competition is based on the winner take all
policy
Counter Propagation Networks.

Counter Propagation is a combination of two well
known algorithm :
Kohenen s Self organizing maps.
Gross bergs Outstar.
Counter Propagation is a network with high
representational power compared to single layer
perceptron.
It is having high speed of training.
KOHONEN SELF ORGANISING MAPS.

Kohonen self organizing maps assume a topological structure
among clustering unit. Kohonen network aims at using
kohonen learning to adjust weights and finally results in a
pattern.
Structure of Kohonen
There are m clustering units, arranged in a linear or two
dimensional array.
The input are arranged as n-tuples.
All input are given to all the neuron.
The weights of clustering unit will serve as an exempler of
input patteren.
Structure of Kohonen
There are
m clustering units, arranged in a linear or two
dimensional array.
The input are arranged as n-tuples.
All input are given to all the neuron.
The weights of clustering unit will serve as an exemplar of
input pattern.
Kohonen Network also follows Winner Takes All policy.

The network cluster unit ,whose weight vector matches
more closely with the input pattern is considered as
Winner.
The winning is usually decided based on the Euclidean distance

Euclidean distance, D=
( xi wij ) 2
Kohonens Squire grid clustering unit structure
*
*
*
* *
*
*
*
*
Kohonen Training Algorithm
Step 1: Initialize weights ,set learning rate and neighborhood

parameters
Step 2: While stopping condition is false do the following
steps(3 to 7) .
Step 3:For each input vector calculate the Euclidean distance.
D(j)=(wij-xi)2
Step 4: Locate the winner.
Step 5: Adjust the weights
wij(new)=wij(old)+(xi-wij(old))
Example (Refer Problem-1)

Consider a Kohonen net with two cluster units and five
input units. The weight vector for the cluster units are
w1=[1 0.8 0.6 0.4 0.2 ],W2=[1 0.5 1 0.5 1 ].Use the square of
the Euclidean distance to find the winning clustering unit for
input pattern X =[0.5 1 0.5 0.0 0.5 ].Find the new weight
for the winning neuron.assume learning rate =0.2.
Answer: Unit 1 D(1)=0.55,
Unit 2 D(2)=1.25
New weight for Unit 1(Winner)=[0.9 0.84 0.58 0.32 0.26]
Counter Propagation Network Structure

Counter propagation networks consists TWO layers
Kohonen Layer
Grossberg layer
w11
1
W21
Grossberg
Layer
Kohononen
Layer
K1`
V11
k2
3
w33
K3
G1
G2
V32
V33
G3
NORMAL OPERATION OF KOHONEN LAYER
NET=xi*wij
NET=X*W
The Kohonen Neuron with largest value of net is considered
as winner
NORMAL OPERATION OF GROSSBERG LAYER
If k1,K2be the kohonen layer output, then the Grossberg
layer net output is weighted kohonen layer output.
NETj=kivij
Y=KV
Where V=Grossberg Layer weight Matrix
K = Kohonen Layer weight Matrix
Y=Grossberg Layer output Vector.
Kohonen Training (Physical Interpretation)
Preprocessing of the input vectors

Need for preprocessing
Xi'=Xi/(X12+X22++Xn2)
Example: X1=[3,4 ]
X1=[3/5 4/5 ]
3/5
4/5
Representation of input vectors before and after normalization
Two Dimensional Unit Vectors on The Unit Circle
Training process of Kohonen Layer weights

(X-Wold)
X-Wold
Wold
Wnew
Xi
Pre initialization of weight vectors
All the weight vectors are to be set to initial values

before starts training.
Initial values are randomly selected and small values

arte selected.
For
Kohonen the initial training vectors should be
normalized.
The weights vectors must end up equal to normalized

input vectors
Pre normalization will shorten the training process.
Problems with randomizing Kohonen layer weights
It will uniformly distribute weight vectors around the

hypersphere.
Most
of the input vectors are grouped and
concentrated at relatively in small area.
Hence most of the Kohonen neuron may be wasted

due to zero output.
The remaining weights may be too few in number

to categorize the input into groups.
The most desirable solution is to distribute the weight

vectors according to density of input vectors that must be
separated. This is impractical to implement.
Method-I (convex combination method)

1
Set all weight to the same value
,where n is the number
n
of components of input vectors (hence weight vectors). All the
input xiare given a value equal to the X [1 ][ 1 ]
i
n
,where n is the number of inputs.
Assignment :No.2(A)
Explain the need of initialization of weight matrix in
Kohonen layer?
What are the different methods used?
Statistical property of The trained Network
Kohonen network has a useful and interesting ability to
extract statistical properties of the input data set.
It is shown by Kohonen that probability of randomly selected
input vectors closest to any given weight vector is 1/k,where
k is the number of Kohonen neuron.
Training
layer
trainingofisGrossberg
a supervised
training.
Grossberg
An input vector is applied ,from the output of Kohonen
Layer.
Output
from Grossberg layer is calculated and is
compared with the desired output.
The
amount of weight adjustment proportional to this
difference.
vij ,new vi j ,old ( y i vi j ,old )k i

ki=output from Kohonen Layer.
yj=desired output component.
The unsupervised Kohonen layer produces outputs at

indeterminate position .
These
are mapped into the desired outputs by the
Grossberg layer.
CPN
Full Counter Propagation
Forward Counter Propagation
x1
x2
y1
Y-input Layer
X-input Layer
FULL COUNTER PROPAGATION NETWORK
y2
z1
x3
yn
X1*
X2*
X3*
zn
y 1*
T
Y2*
Yn*
Y-output Layer
X-output Layer
zj
The Major aim of a full counter propagation network is to

provide an efficient means of representing a large number of
vector pairs X:Y by adaptively constructing a look up table.
It Produces an approximation X:Y

With X alone.
With Y alone
With X:Y
It uses winner Take All policy

Vectors are normalized
Learning Algorithms for Kohonen Layer are
vij ,new vi j ,old ( xi vi j ,old )

wij ,new wi j ,old ( xi wi j ,old )
Learning Algorithm for Grossberg are
t ij ,new t i j ,old a( y i t i j ,old )k i

u ij ,new u i j ,old b( y i u i j ,old )k i
A full counter propagation network to compliment the

function y=1/x
Applications of CPN
Vector Mapping
Data Compression
Image Compression
A CPN can be used to compress data before transmission.

The image to be transmitted is first divided into sub images.
Each sub image is further classified into pixels.
Each pixel represent either one (Light) or zero (dark).
If there are n pixels, n pixels are required to transmit this.
If some distortion can be tolerated, a fewer bits are
sufficient for transmission.
A CPN can perform vector quantization.
Only one neuron of the Kohonen layer output become 1.
The Grossberg layer will generate a code for that neuron,
and is transmitted.
At the receiving end an identical CPN accept the binary
code and produces the inverse function .

ARTIFICIAL NEURAL NETWORKS-moduleIII

Uploaded by

Copyright:

Available Formats

ARTIFICIAL NEURAL NETWORKS-moduleIII

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ARTIFICIAL NEURAL NETWORKS-moduleIII

Uploaded by

Copyright:

Available Formats

ARTIFICIAL NEURAL

clustering may be defined as a

mapping through which N-dimensional pattern space

matching is achieved autonomously without

supervision. i.e., clustering is done in a self-organized

The term self-organizing refers to the ability to learn

Self-organized network perform unsupervised

When more than one neuron in the out put layer

Counter Propagation Networks.

KOHONEN SELF ORGANISING MAPS.

m clustering units, arranged in a linear or two

Kohonen Network also follows Winner Takes All policy.

The winning is usually decided based on the Euclidean distance

Kohonens Squire grid clustering unit structure

Kohonen Training Algorithm

Step 1: Initialize weights ,set learning rate and neighborhood

Example (Refer Problem-1)

Counter Propagation Network Structure

NORMAL OPERATION OF KOHONEN LAYER

Kohonen Training (Physical Interpretation)

Preprocessing of the input vectors

Representation of input vectors before and after normalization

Two Dimensional Unit Vectors on The Unit Circle

Training process of Kohonen Layer weights

Pre initialization of weight vectors

All the weight vectors are to be set to initial values

Initial values are randomly selected and small values

Kohonen the initial training vectors should be

The weights vectors must end up equal to normalized

Pre normalization will shorten the training process.

Problems with randomizing Kohonen layer weights

It will uniformly distribute weight vectors around the

of the input vectors are grouped and

concentrated at relatively in small area.

Hence most of the Kohonen neuron may be wasted

The remaining weights may be too few in number

The most desirable solution is to distribute the weight

Method-I (convex combination method)

from Grossberg layer is calculated and is

compared with the desired output.

amount of weight adjustment proportional to this

vij ,new vi j ,old ( y i vi j ,old )k i

The unsupervised Kohonen layer produces outputs at

are mapped into the desired outputs by the

Full Counter Propagation

Forward Counter Propagation

FULL COUNTER PROPAGATION NETWORK

The Major aim of a full counter propagation network is to

It Produces an approximation X:Y

It uses winner Take All policy

vij ,new vi j ,old ( xi vi j ,old )

t ij ,new t i j ,old a( y i t i j ,old )k i

A full counter propagation network to compliment the

A CPN can be used to compress data before transmission.

You might also like