BAHAGIAN B

Part B

QUESTION 1

a) Give 3 applications of deep learning and for each application state the suitable
algorithms. [3 marks]

3 applications of deep learning are Handwrite Digit Recognition, Natural Language

b) What is the difference between Multilayer Perceptron NN (ANN) and Recurrent NN

Difference between
Artificial Neural Network Recurrent Neural Network
(ANN) (RNN)
Architecture also known as a feedforward have connections that form
neural network, an ANN cycles, allowing them to
consists of an input layer, one maintain internal state and
or more hidden layers, and an process sequential data. They
output layer. The information have an internal memory that
flows only in one direction, allows them to remember and
from the input layer through process information from
the hidden layers to the output previous time steps, making
layer. There are no cycles or them suitable for sequential
loops in the network data processing tasks.
architecture.
Sequential Data Processing primarily designed for static specialized for sequential data
input data where each input is processing tasks due to their
independent of others. They are ability to maintain hidden
not explicitly designed to states that capture temporal
handle sequential data, such as dependencies. They are
time series, text, or speech. commonly used in natural
language processing, speech
recognition, time series
analysis, and other tasks where
order matters.
Information Flow Each input is processed Process inputs sequentially one
independently, and there is no by one, and they maintain
inherent memory or temporal hidden states that carry
context. The output is solely information from previous time
based on the current input. steps. This allows them to
capture temporal dependencies
and context, making them well-
suited for tasks involving
sequences.
 Training and Backpropagation Training ANNs typically Training RNNs can be more
involves feedforward and complex than ANNs due to the
backpropagation algorithms. presence of recurrent
The backpropagation algorithm connections and the
adjusts the network’s weights vanishing/exploding gradient
to minimize the error between problem. Various RNN
the predicted output and the variants like LSTM (Long
target output. Short-Term Memory) and
GRU (Gated Recurrent Unit)
have been introduced to
address these issues and
improve training.

In summary, ANN is suitable for static data where input samples are independent of each other,
while RNN is designed for sequential data processing, where the order and temporal context
are essential. Each type of network has its strengths and weaknesses, and the choice between
them depends on the nature of the data and the specific problem being addressed.

1
c) Assume that all internal nodes in Figure 1 compute the sigmoid function, ⱷⱷ = .
1+𝑒 −𝑡
Write an explicit expression to how back propagation (applied to minimize the least
squares error function) changes the values of w1, w2, w3, w4 and w5 when the algorithm
is given the example x1 = 0, x2 = 1, with the desired response y = 0 (notice that x0 = 1 is
the bias term). Assume that the learning rate is α and that the current values of the
weights are: w1 = 3, w2 = 2, w3 = 2, w4 = 3 and w5 = 2. Let yj be the output of the hidden
units and output units indexed by j.

i. Write equations for y1, y2 and y3 in terms of the given weights and example, for
forward propagation.

[3 marks]

ii. Write equations for δ1, δ2 and δ3 in terms of the given weights, for backward
propagation.

[5 marks]
 iii. Give an explicit expression for the new weights.

[6 marks]

W1 δ2
X0
2 W4
O1
X1 W2 1

3 W5 δ1
X2 W3
δ3

Figure 1: Artificial neural network with two inputs, one hidden layer and one output layer.

QUESTION 2

Answer the following questions regarding, RNN, LSTM and CNN.

a) Explain in the layman the concept of how the LSTM network improve the Basic
RNN. [5 marks]
b) Explain the important features of LSTM that improve RNN. [3 marks]

c) Explain how CNN relates to MLP (ANN) [3 marks]

d) CNN is divided in two big steps, feature learning and classification. Explain what
happens in each step? [4 marks]
QUESTION 3

a) CNN is divided in two main steps, feature learning and classification. Explain what
happens in each step?
(4 mark)

b) In this question, you need to design a convolutional network to detect vertical boundaries
in an image. The architecture of the image network is as shown in Figure 1.

The ReLU activation function is applied to the first convolution layer. The output layer uses
the linear activation function. Assume either the standard definition of convolution (which
flips and filters) or the version used in conv nets (which skips the filtering step). Use white to
denote 0 and darker values to denote larger (more positive) values.

Figure 1: Architecture of CNN Network for Image Classification

(a) Design two convolution kernels(filter) for the first layer, of size 3 × 3. One of them
should detect dark/light boundaries, and the other should detect light/dark boundaries.
(3 marks)

(b) Design convolution kernels of size 3×3 (filter) for the output layer, which computes the
desired output.
(3 marks)