CERTIFICATE

It is hereby certified that the work incorporated in the dissertation entitled Development
of Automated Surveillance And Warning System, has been completed by Aayush
Goyal , a student of Bachelor of Technology, under our supervision. He has accom-
plished the tasks as per my guidance and We are satisfied with his performance.

Place: Bikaner Dr. Sudhakar Modem

Project Supervisor
Date: 17th Dec 2020 Assistant Professor
Dept. of Electrical Engineering
IIT Jammu, 181221

Dr. Sahil Kalra

Project Supervisor
Assistant Professor
Dept. of Mechanical Engineering
IIT Jammu, 181221
 ACKNOWLEDGEMENTS

I express my sincere gratitude towards my project Supervisors Dr.Sudhakar Modem

and Dr.Sahil Kalra for their help, inspiration and encouragement throughout the project
work. Without their invaluable guidance, this work would never have been a successful
one.

At last I am also thankful to my Late father and my mother for their never ending
encouragement and support, by virtue of which I am able to present this dissertation

Place: Bikaner Aayush Goyal

Date: 17th Dec 2020

i
 ABSTRACT

IoT based automated surveillance and warning systems are used differently by de-
fence organizations. The harsh climatic as well as hostile conditions in which defence
personnels operate are a big challenge for these organizations. It is the valor of the per-
sonnels that the risk their lives to operate in these locations. So developing automated
surveillance and warning systems using existing contemporary technologies can be a
great sigh of relief for the armed forces.

This can increase the efficiency of surveillance as machines don’t have any con-
straint of fatigue. Moreover, risk to the lives of personnel can be reduced. Although
this can be done in multiple ways, I have chosen to make this project IoT based with in-
tegration of readily available robust and efficient AI facial recognition techniques.This
project is to deliver a fully automated surveillance and warning system with the idea to
detect trespassing in the area under surveillance. Computer-based surveillance systems
are much more efficient than humans. They can be deployed 24x7 without any interrup-
tion in areas where human reach is difficult. In comparison with human surveillance,
this system is very cost-efficient and more robust.

ii
 TABLE OF CONTENTS

Page

ACKNOWLEDGEMENTS. . . . . . . . . . . . . . . . . . . . . . . . . . . i

ABSTRACT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii

LIST OF TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

LIST OF FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi

ABBREVIATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

CHAPTER 1: INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . 1

1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Project Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

CHAPTER 2: PROJECT FEATURES . . . . . . . . . . . . . . . . . . . 5

2.1 Live Stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2.2 Low light operation . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 Trespassing Detection . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.4 Object tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

CHAPTER 3: EXECUTION . . . . . . . . . . . . . . . . . . . . . . . . . 7

3.1 Components used and details . . . . . . . . . . . . . . . . . . . . . 7

3.1.1 Raspberry Pi 4 Model B . . . . . . . . . . . . . . . . . . . 7
3.1.2 Camera Module Pi NoIR Camera V2 . . . . . . . . . . . . 8
3.1.3 Raspberry Pi 4 Case . . . . . . . . . . . . . . . . . . . . . 8
3.1.4 Raspberry Pi 15.3W USB-C Power Supply . . . . . . . . . 8
3.1.5 Micro servo motor . . . . . . . . . . . . . . . . . . . . . . 10
3.1.6 Camera mounting case . . . . . . . . . . . . . . . . . . . . 10
3.2 Capturing video from Camera . . . . . . . . . . . . . . . . . . . . 12
3.3 Analysing the video feed using algorithm . . . . . . . . . . . . . . 12

iii
 Table of Contents (continued) Page
3.4 Streaming the video feed over Local Area Network . . . . . . . . . 13

3.5 Configuring Simple Mail Transfer Protocol . . . . . . . . . . . . . 14

CHAPTER 4: RESULTS AND DISCUSSION . . . . . . . . . . . . . . . 15

4.1 Different detection program results . . . . . . . . . . . . . . . . . . 15

4.2 Streaming on multiple devices and controlling the camera view . . 17
4.3 Detection and Notification . . . . . . . . . . . . . . . . . . . . . . 20

CHAPTER 5: CONCLUSION . . . . . . . . . . . . . . . . . . . . . . . . 22

CHAPTER 6: RESULTS AND DISCUSSION . . . . . . . . . . . . . . . 23

REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
 LIST OF TABLES

Table Title Page

v
 LIST OF FIGURES

Figure Title Page

1.1 Schematic of clustering for the surveillance system . . . . . . . . . 3

1.2 Flowchart of surveillance system working . . . . . . . . . . . . . . 4

3.1 Raspberry Pi 4 Model B [1] . . . . . . . . . . . . . . . . . . . . . . 8

3.2 Camera Module Pi NoIR Camera V2 [2] . . . . . . . . . . . . . . . 9
3.3 Raspberry Pi 4 Case [3] . . . . . . . . . . . . . . . . . . . . . . . . 9
3.4 Raspberry Pi 15.3W USB-C Power Supply [4] . . . . . . . . . . . . 10
3.5 Micro servo motor [5] . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.6 Camera mounting case . . . . . . . . . . . . . . . . . . . . . . . . 11

4.1 Result of the face detection program . . . . . . . . . . . . . . . . . 15

4.2 Result of the full-body detection program . . . . . . . . . . . . . . 16
4.3 Result of the upper body detection program . . . . . . . . . . . . . 16
4.4 Authentication window to login into the live stream . . . . . . . . . 17
4.5 Streaming live on Raspberry interface . . . . . . . . . . . . . . . . 18
4.6 Streaming live on mobile . . . . . . . . . . . . . . . . . . . . . . . 18
4.7 Streaming live on a windows machine . . . . . . . . . . . . . . . . 19
4.8 Control of camera . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.9 E-mail notification with the detected face of a trespasser . . . . . . 20
4.10 Continuous tracking and notification . . . . . . . . . . . . . . . . . 21

vi
 ABBREVIATIONS

RGB Red-Green-Blue
AI Artificial Intelligence
RPi Raspberry Pi
FPS Frame per Second
SMTP Simple Mail Transfer Protocol
DIP Digital Image Processing
NoIR No infrared
UAV unmanned aerial vehicle
DC Direct Current
FOV Field of Vision

vii
 CHAPTER 1

INTRODUCTION

1.1 Motivation

The motivation for the project not only came from the fact that automated Surveillance
has become significant during recent times, but also that our country is yet to achieve the
idea of self-reliance in the defense sector. So this is a little effort by us as an engineering
undergraduate for realizing the idea of “Atmanirbhar Bharat” in the defense sector. At
the end of the day, it is the life of a military person that is at stake during surveillance,
this further motivates us to come up with technical solutions that can ease the life of
the soldiers. Being an engineering graduate has its limitation that we cannot help them
hand to hand directly in combat, but things that can be changed are in the ambit of
technologies studied by us during our stay at IIT. Nothing more can be more satisfying
than working for the defense personnel.

Also support from the institute and the Faculty supervisor Dr.Sahil Kalra has been
a key factor for the successful completion of the project with the desired results. The
faculties are motivating and are always pushing us to do better than what has already
been done. Financial support is a big aspect in any project, and particularly for this
project, we are thankful to faculty supervisors for promptly working on providing de-
sired financial support so that we can start the project practically as soon as possible.

The motivation for developing an in-house surveillance system is that it gives con-
fidentiality regarding the technology used as the programs and algorithms are always
secured with the developers. Since there are restricted 3rd party components used limits
the information flow. The cost of these types of surveillance systems is on a slightly
higher-end but when security is concerned cost is not an issue. Also, in-house devel-
opment gives more flexibility and faster development in the system so overall it is a
beneficial deal. Every path in the information flow is highly secured by using an au-
thentication window to make the data secure.

1.2 Project Outline

The project is to make a surveillance system that is to be deployed in an area that is

out of reach for humans. So in this project, a package module is developed which
can be deployed at the location and has the ability to perform communication com-
pletely wireless.The automated surveillance system is developed with the use of vari-
ous components and programming with object detection algorithms. This surveillance
is done by capturing the video and streaming live on the central command center. Mod-
ule deployed on the location of surveillance is based on the powerful microcomputer
Raspberry Pi 4 Model B integrated with camera and motors for pan vision. The video
captured by the camera module is processed with computer vision algorithms for the
detection of transpassing objects/humans.

When object detection is done by the security module a notification with details
(image and time) is sent on the electronic mail to the concerned authority using SMTP
protocol. This surveillance system has many features like secured live streaming on
multiple devices, changing the surveillance area with camera control, and giving audio
warnings from the command center. This surveillance module is very power efficient
which requires only a 5V DC supply to perform all the work. Surveillance at night is
also an important feature that is possible with the use of an IR camera module. This
project has enormous potential to make the future of surveillance more robust. The next
phase of this project is to integrate the data of the module with UAV and robots for a
fast approach to the location of transpassing to get the real-time situation of the area.

Figure 1.1 show the plan to cover all the area with multiple interconnected surveil-
lance modules and these modules are connected with the central command center. The
central command center will control the camera position and select the program which

2
 Fig. 1.1: Schematic of clustering for the surveillance system

needs to be run on each surveillance module. From the central command center, an
audio warning can be sent to the location of transpassing. Since all the surveillance
modules are interconnected and wireless it is very easy to track the trespasser by chang-
ing the camera direction of other surveillance modules even if the trespasser gets out of
view. All this gives the power to look over a huge area from a single point located at
the central command center.

In Figure 1.2 a flow chart is shown which indicates the working process of the
surveillance system. After deployment at the location of surveillance. Video is cap-
tured with the help of a camera module. Every frame of the video is processed with
computer vision algorithms for the detection of any object/human. If any trespassing is
detected then the frame (snapshot) is sent on dedicated email to the authorized security
person. This notification is helping in deciding the action after trespassing is detected.
All the information flow is completely wireless which gives freedom from wires and re-
strictions on the location of deployment. Several lines of sight communication modules
can also be integrated to make the surveillance of kilometers of range.

3
Fig. 1.2: Flowchart of surveillance system working

4
 CHAPTER 2

PROJECT FEATURES

2.1 Live Stream

Command centre to which the PiModule is connected over a network will have access
to the 24X7 live stream of the location in which the surveillance device is deployed.This
particular feature will be achieved by sending preprocessed data received from the cam-
era attached to the Pi module. The FPS and texture quality of this live stream will de-
pend on three factors. Firstly on the specifications of the specifications of the camera
used. In the present case the camera which is being used can provide stream quality
upto 720P.

But the latter two factors limit its operation at full capacity. The second factor is
the computation power of the module which we are using for DIP techniques for the
purpose of object detection. It requires more computational power for DIP of a good
resolution image. But have computational constraints in the RaPi Module.

Thirdly the bandwidth of the channel which we are using for communication pur-
poses also plays a major role. We can afford inconsistencies in that, so keeping that in
mind our camera is operated to maintain a balance.

2.2 Low light operation

The operability of the surveillance system in low light is highly dependent on the spec-
ifications of the camera. The NoIR camera module used in the project enables it to
extract features of the image even at low lights. Feature extraction forms an essential
part for expecting good outcomes after DIP algorithms. So this feature enhances the
capability of our prototype by operating at low lights also.It is important to be noted
here that the infrared sensor is not used in current prototypes.

2.3 Trespassing Detection

The digital data captured by the camera module will be processed in the RPi module
with the help of pre-trained models based on DIP algorithms. This processing for a pos-
itive match will yield whether there is a positive object detection or not. The efficiency
of object detection depends on the accuracy of the model which is 95% in our case.
This feature further depends on the computation capacity also. Even if a more efficient
algorithm exists, we have a constraint over computational capacity.So the present model
used is best suited with given computational power.

2.4 Object tracking

Object tracking feature can be realised with the help of servo motors to change FOV
of the camera to live track the object The servo motors attached to the module can be
controlled by the command centre over a two way communication channel. Further
other object tracking devices adjacent to which the object is detected can also be used
for the purpose of object tracking if object is getting out of the range of FOV even after
the use of servo mechanism

6
 CHAPTER 3

EXECUTION

3.1 Components used and details

This section is about the components used in the development of this automated surveil-
lance system. List of components used are as follows:

(a) Raspberry Pi 4 Model B

(b) Camera Module Pi NoIR Camera V2

(c) Raspberry Pi 4 Case

(d) Raspberry Pi 15.3W USB-C Power Supply

(e) Micro servo motor

(f) Camera mounting case

3.1.1 Raspberry Pi 4 Model B

Figure 3.1 shows Raspberry Pi 4 Model B which is a microcomputer capable of do-

ing on-board computation of various programs. This energy-efficient microcomputer
has 8GB RAM onboard for faster processing of video data generated from the cam-
era module. Raspberry Pi 4 also provides an onboard WiFi module which is used for
communication to the surveillance module from the command center. This module of-
fers an onboard camera port to attach the camera module direct and hassle-free camera
integration.
 Fig. 3.1: Raspberry Pi 4 Model B [1]

3.1.2 Camera Module Pi NoIR Camera V2

Figure 3.2 shows the camera module with a Sony IMX219 8 MegaPixel sensor. This
module delivers high-resolution video and is also capable of performing low light cap-
turing with infrared lighting.

3.1.3 Raspberry Pi 4 Case

Figure 3.3 shows the Raspberry Pi 4 case which protects the module from dust and
water at the location of deployment. This case keeps every component intact and safe.

3.1.4 Raspberry Pi 15.3W USB-C Power Supply

Figure 3.4 shows the Raspberry Pi USB-C power supply adaptor to power Raspberry
Pi 4 Model B boards. It gives 5.1V / 3.0A DC output with a 96-264Vac operating input
range for working in the high range of AC voltage supply. This power supply unit has
various important features like SC, overcurrent, and over-temperature protection.

8
Fig. 3.2: Camera Module Pi NoIR Camera V2 [2]

Fig. 3.3: Raspberry Pi 4 Case [3]

9
 Fig. 3.4: Raspberry Pi 15.3W USB-C Power Supply [4]

3.1.5 Micro servo motor

Figure 3.5 shows the servo motor used in this project. It is a lightweight, energy-
efficient, and precise servo motor. On this motor, the camera is mounted with the help
of a case designed to place the Raspberry camera on the servo motor. The motor is
controlled by a web module which is programmed with a slider to rotate the camera
according to the need of changing the field of view of the area of surveillance.

3.1.6 Camera mounting case

Figure 3.6 shows the model of a personalized case for the camera. This case helps in
mounting the camera on the servo motor. All the appropriate slots are made according
to the camera dimension and mounting head of the servo motor.

10
Fig. 3.5: Micro servo motor [5]

Fig. 3.6: Camera mounting case

11
3.2 Capturing video from Camera

Capturing of video will be done primarily with the help of NoIR 8 megapixel camera.
Using Imutils library we perform basic operations on the captured video. The FPS and
resolution of the captured video as dissed will depend on four factors.

(a) Specifications of camera

(b) Computational resources for DIP
(c) Bandwidth of Communication
(d) Minimal features required for DIP

Considering the above factors like computational resource constraints, we can say
that even if bandwidth and specifications of camera can be enough for higher FPS and
resolution. We have to settle with 24 FPS at 640x480 pixels for higher efficiency of
object detection algorithm. Although we can directly stream a high quality DIP if it
needs not to be done at the module itself.

But the idea of DIP being done at module itself is supported by fact that if there is
a failure at central unit than whole system will collapse, but in this case even if some
module do down , then we can easily use servo controls on adjacent modules to cover
FOV of that module meanwhile technical team is arriving.

3.3 Analysing the video feed using algorithm

The algorithms of object detection and different backend processes are provided in de-
tail. This includes the library used for programming of object detection, processing of
live stream video, and hosting a local server for streaming on multiple machines to-
gether. Rapid Object Detection technique using a Boosted Cascade of Simple Features
by Paul Viola and Michael Jones[7]. The Object detection is done using a haar feature
based cascade classifier.

AdaBoost Algorithm:It is used to narrow down the number of features extracted from
image to only a few useful ones required for object detection.

12
Weak classifier:perform at least better than random

Combining weak classifiers in a weighted sum to form a strong classifier: The Cascade

Classifier:

3.4 Streaming the video feed over Local Area Network

A live stream and authentication window before access to live stream is programmed
using Flask-SocketIO[8] which provides application development capabilities with low

13
latency bi-directional communications between the clients and the server. With Flask-
SocketIO a local server is hosted at the same address as the IP assigned to Raspberry Pi
on which video is streamed and a camera control window is also available.

3.5 Configuring Simple Mail Transfer Protocol

SMTP (Simple Mail Transfer Protocol) is used to send [10] email on the designated
email address. SMTP automatically authenticates the account from which email is to
be sent with the details provided like username and password accessing the account and
then sends the email. This process is very long from the theoretical aspect but all the
procedure from the detection of the object till receiving of an email notification is done
within a second in an efficient manner.

14
 CHAPTER 4

RESULTS AND DISCUSSION

This section contains various results of the surveillance module. All the images in the
result sections are generated from the Raspberry Pi deployed in a room. No, image is
taken from any source.

4.1 Different detection program results

The surveillance is integrated with multiple options for the detection of an object. Three
programs are developed with detection of a face, detection of full-body, and detection
of the upper body of the transpassing object/human. This program selection can be con-
trolled at a central command center by picking the program to be executed at a particular
location. Every individual surveillance unit can work on different programs according
to the need.

Fig. 4.1: Result of the face detection program

Fig. 4.2: Result of the full-body detection program

Fig. 4.3: Result of the upper body detection program

16
Figure 4.1-4.3 shows the results obtained with different programs with correct detection
and classification of the human in the frame of the camera. The rate of detection is very
good and the efficiency of object detection is approx 95%[7]. Even if the object crosses
the camera frame at a fast speed the detection is possible due to high frame rate capture
and processing in the Raspberry Pi.

4.2 Streaming on multiple devices and controlling the

camera view

The surveillance module gives the facility to stream the live feed on any device on the
same network. For restriction on unauthorized access to the module and live feed, an
extra layer of security is added which asks for authentication details to enter into the
live stream. This makes the flow of information completely secured and prevents any
type of cyber intrusion also.

Fig. 4.4: Authentication window to login into the live stream

Figure 4.4 shows the login window on which the authentication details such as username
and password are mandatory for signing in

Figure 4.5-4.7 shows the streaming of live feed on multiple devices at the same time.
This gives the advantage to watch the surveillance are on any machine and from any
location.

17
Fig. 4.5: Streaming live on Raspberry interface

Fig. 4.6: Streaming live on mobile

18
Fig. 4.7: Streaming live on a windows machine

Fig. 4.8: Control of camera

19
Figure 4.8 shows the control window to provide camera motion from the command
center only. Slider 1 decides the angle by sliding left to right. An additional submit
button is added for the prevention of any undesired motion of the camera which can
change the field of view.

4.3 Detection and Notification

The video is processed frame by frame in the program and if the object is detected the
next step is to notify the command center and send an email to record the trespassing
image with time and location. For this purpose, the surveillance module uses SMTP
(Simple Mail Transfer Protocol) on Gmail or any other email service. The email can be
sent to any email address and from any email address. These email addresses are fixed
before the deployment at the location. Also, these emails can be changed at any time
with the requirement.

Fig. 4.9: E-mail notification with the detected face of a trespasser

Figure 4.9 shows the email received from the mail address registered in the surveillance
module. The subject of this notification is Urgent::Person Detected as this is a priority
mail. In this mail image with a boundary on the detected object is attached. The time

20
taken in the process id depends on the network speed with which the surveillance mod-
ule is connected as the computer vision process is happening in a very fast and efficient
way.

Fig. 4.10: Continuous tracking and notification

Figure 4.10 is showing the consecutive emails with the face detected image. This hap-
pens when the person remains in the frame for a longer time. The time limit between
two emails can be controlled in the programming. A low time gap can choke the net-
work by sending multiple emails in a very short time. Also, this makes it a risk to make
the email as spam on the receiver end because bulk mails are received.

These are the results generated from the working surveillance module. A fully auto-
mated and powerful surveillance system is developed in this project.

21
 CHAPTER 5

CONCLUSION

The project started with the motivation for developing an automated surveillance and
warning system. With the use of available components and resources surveillance mod-
ules are developed. With various functionalities and different detection programs, the
model is built with the idea to make the system robust. At the completion of the project,
the results are promising with a complete working and are at executable stage. The
module is ready for deployment for the surveillance over an area just connecting with a
WiFi network.
 CHAPTER 6

RESULTS AND DISCUSSION

This project is currently working at its half capability and still has many possibili-
ties to make use of complete capabilities by developing training for Machine Learning
models on the images which are captured with the trespasser. This will recognize the
person if the trespasser is repeated the intrusion. Integration of GPS in the project will
allow the process of the location details which can be used in post-application like giv-
ing the location coordinates to a UAV or robot so that it can reach the exact location
automatically. Several other programs can also be deployed like the classification of
different objects like airplanes, animals, or humans. In scenarios like surveillance at
entry gates, features like counting of entry and exit can also be integrated in this. In
the current pandemic situation, this project can be deployed to check social distancing
by calculating the distance between two frames in which humans are detected. Also,
check for masks on a human face. In the future, a more robust and faster automated
surveillance system with vast features will be developed by inspirations and ideas from
the faculty supervisor Dr.Sudhakar Modem and Dr. Sahil Kalra.