Sensors 23 01315 v2

sensors
Article
Development of a Machine-Learning Intrusion Detection
System and Testing of Its Performance Using a Generative
Adversarial Network
Andrei-Grigore Mari, Daniel Zinca * and Virgil Dobrota
Communications Department, Technical University of Cluj-Napoca, 400114 Cluj-Napoca, Romania

* Correspondence: [email protected]
Abstract: Intrusion detection and prevention are two of the most important issues to solve in network
security infrastructure. Intrusion detection systems (IDSs) protect networks by using patterns to
detect malicious traffic. As attackers have tried to dissimulate traffic in order to evade the rules
applied, several machine learning-based IDSs have been developed. In this study, we focused on one
such model involving several algorithms and used the NSL-KDD dataset as a benchmark to train and
evaluate its performance. We demonstrate a way to create adversarial instances of network traffic
that can be used to evade detection by a machine learning-based IDS. Moreover, this traffic can be
used for training in order to improve performance in the case of new attacks. Thus, a generative
adversarial network (GAN)—i.e., an architecture based on a deep-learning algorithm capable of
creating generative models—was implemented. Furthermore, we tested the IDS performance using
the generated adversarial traffic. The results showed that, even in the case of the GAN-generated
traffic (which could successfully evade IDS detection), by using the adversarial traffic in the testing
process, we could improve the machine learning-based IDS performance.
Keywords: generative adversarial network; intrusion detection system; intrusion evasion; machine
learning; NSL-KDD dataset; Python
Citation: Mari, A.-G.; Zinca, D.;
Dobrota, V. Development of a
Machine-Learning Intrusion
Detection System and Testing of Its
Performance Using a Generative 1. Introduction
Adversarial Network. Sensors 2023, Network security is one of the fastest growing industries due to the increasing number
23, 1315. https://doi.org/10.3390/ and sophistication of attacks. Data are considered to be among the most valuable assets
s23031315 of a company. Cyber-attacks are usually launched against a user’s network and consist of
Academic Editors: Hisham Kholidy, exploiting a vulnerability to enable unauthorized use of electronic data. Current networks
Mohammad Rahman, Sherif Saad encounter proliferating and increasingly sophisticated attacks. Therefore, network protec-
and Pratik Satam tion against cyber-attacks is of utmost importance. Intrusion detection systems (IDSs) are
deployed in defense-in-depth scenarios in order to protect networks from various types of
Received: 28 December 2022
attacks. Depending on the placement of the device and whether it tries to prevent attacks,
Revised: 16 January 2023
the system can act also as an intrusion prevention system (IPS). In general, an IDS monitors
Accepted: 19 January 2023
network traffic to search for signatures of malicious activity or violations of rules created
Published: 24 January 2023
based on security policies. Signature-based and anomaly-based methods are the two main
approaches used to build such systems. A signature represents an indicator of compromise
created based on previously identified (known) threats. In the case of unknown threats or
Copyright: © 2023 by the authors. zero-day attacks, anomaly-based detection systems are more appropriate. The drawbacks
Licensee MDPI, Basel, Switzerland. of anomaly-based IDSs include higher rates of false positives (classifying normal traffic as
This article is an open access article attacks) and false negatives (allowing malicious traffic to propagate). According to [1], vari-
distributed under the terms and ous techniques can be used in order to build an anomaly-based detection system: threshold
conditions of the Creative Commons detection, either static or heuristic; statistical measures; rule-based measures; and machine
Attribution (CC BY) license (https:// learning (ML).
creativecommons.org/licenses/by/
4.0/).
Sensors 2023, 23, 1315. https://doi.org/10.3390/s23031315 https://www.mdpi.com/journal/sensors

Sensors 2023, 23, 1315 2 of 32
1.1. Objectives
Although signature-based IDSs are more common, recent developments in anomaly-
based IDSs use machine-learning algorithms. A model uses one or more algorithms to
learn to recognize malicious activity. Each model is built using a certain set of features that
are available for a specific dataset. The accuracy of machine learning IDSs with test datasets
can be higher than 90%, as demonstrated in Section 2.2. Unfortunately, an attacker can
evade detection by machine learning IDSs by generating malicious traffic that retains the
functional features of attack traffic. In Section 2.3, we present a GAN architecture that can
be used in order to test and improve IDS performance in the presence of adversarial traffic.
1.2. Dataset Used

ML algorithms [2–6] are currently being used to successfully develop IDSs. Note that
the quality and the relevance of the training dataset [7] (e.g., NSL-KDD, UNSWNB15) are
of great importance for the final accuracy. A review of the available datasets that have been
used for machine-learning applications is presented in [8].
For the purposes of this paper, we used the NSL-KDD dataset, which is a refined
version of its predecessor KDD’99, a well-known benchmark in the research on intrusion
detection techniques [6,9–13]. This labeled dataset is split into training and testing files and
can be downloaded from [12]. In addition to the normal traffic, there are 23 attack types
documented (for example, neptune, ipsweep, portsweep), which are grouped into four
main categories: denial of service (DoS), probing (probe), user to root (U2R), and remote
to user (R2L) [6]. The DoS category refers to attacks that consume the server’s available
resources, making it difficult for the attacked system to fulfill legitimate user requests. A
probing attack usually precedes an access attack and consists of collecting information
about the victim, usually by scanning IP addresses and ports. A user to root attack attempts
to obtain unauthorized root access in a system where the attacker already has user access.
In a remote to user case, an unauthorized connection from a remote system under the
attacker’s control is attempted in order to gain local access. For each network traffic record,
there are 41 types of features, and these are classified as attack or normal types; 9 of them
are discrete values and the remaining 32 values are continuous. Discrete data employ
values that can only take on specific values and cannot be subdivided. Continuous data
employ values that can take on any numerical value and can be subdivided into smaller
parts. These features can be classified into four categories: basic, content, time-related
traffic, and host-based traffic [6,13].
1.3. Machine-Learning Algorithms Implemented

Several machine-learning algorithms were used in this study to implement the IDS.
K-Nearest Neighbors
The K-nearest neighbors (KNN) algorithm is a data classification method for estimating
the likelihood that a data point will become a member of one group or another based on
which groups the data points nearest to it belong to. This non-parametric algorithm is used
in the case of classification problems. We implemented the K-nearest neighbors algorithm
by using the KNeighborsClassifier class from the sklearn library.
Decision Tree
Decision trees are supervised learning techniques that can be used in the case of
classification problems. They are tree-structured classifiers, where internal nodes represent
the features of a dataset, branches represent the decision rules, and each leaf node represents
an outcome. The decisions are made on the basis of the features of the given dataset.
We implemented the decision tree by using the DecisionTreeClassifier class from the
sklearn library.
Sensors 2023, 23, 1315 3 of 32
Random Forest
The random forest algorithm is a supervised machine-learning algorithm that is used
widely in classification and regression problems. It builds decision trees with different sam-
ples and calculates the majority class for classification and average prediction for regression.
We implemented the random forest algorithm by using the RandomForestClassifier class
from the sklearn library.
Support Vector Machine
A support vector machine (SVM) is a very powerful and versatile machine-learning
model capable of performing linear or nonlinear classification, regression, and outlier
detection. We implemented it using the SVC class from the sklearn library.
Neural Networks
Neural networks, also referred to as artificial neural networks (ANNs) or simulated
neural networks (SNNs), are a subgroup of machine-learning models and are at the core of
deep-learning algorithms. Even though they are a subset of machine-learning models, neu-
ral networks are handled separately as they had the greatest impact in the implementation
we realized.
We implemented an artificial neural network using the Keras API with Relu and
LeakyRelu as the activation functions.
Generative Adversarial Networks
Generative adversarial networks (GANs) represent a recent development in machine
learning and are a powerful class of neural networks that are utilized in unsupervised
learning. They are generative models in the sense that they create new data instances that
resemble the original training data. New data are created based on learning of the patterns
in the original data. Some of the first examples used images as training data. The initial
paper on GANs [14] described their components; for instance, GANs can be understood
as two participants in a game where each player tries to minimize their cost based on the
parameters they control and the other player’s output. In this context, each player will
reach a point called the local Nash equilibrium (from game theory) with the minimum cost,
which can be computed based on the parameters. There are two players, the generator
and the discriminator. The generator can take an input from a source of randomness (for
example, a uniform distribution) and should be able to generate outputs that are similar
to the training data used by the other player. These outputs are useful when the local
equilibrium is reached. The second player is the discriminator and uses inputs from the
original training set and from the output of the generator. The discriminator predicts the
probability that the input from the generator is real or fake. The discriminator cost is
minimized when it correctly classifies the original and the fake data. On the other hand,
the generator cost is minimized when it successfully generates adversarial data that are
incorrectly classified as real.
There are several applications of GANs in network security. One approach is to use the
output from the generator as test data and to consider a black-box IDS as a discriminator; its
performance is then tested against adversarial data. In [13], a framework called IDSGAN is
described in which a generator transforms original malicious traffic into adversarial traffic
examples that are later sent to an IDS. The framework proposed inspired us to create and
test an alternative solution, which is detailed in this paper. This solution not only makes
it possible to assess the performance of various IDS implementations against adversarial
traffic but also, more important, allows for the improvement of IDS detection by including
generated adversarial traffic in the training phase of the IDS. Other research papers are
discussing different approaches to use GAN de detect network intrusions [15–21]. The
principle of the GAN is shown in Figure 1.
Sensors
Sensors 2023,
2023, 23,
23, 1315
1315 4 4of
of 32
32
Figure 1.
Figure Generative adversarial
1. Generative adversarial network.
network.
2. Materials and Methods

2. Materials and Methods
This section presents the platforms and components used for the development environ-
ment,This sectionbypresents
followed a step-by the platforms
step description and of
components
the developmentused for the machine-learning
of the development en-
vironment, followed by a step-by step description of the
IDS and the process of testing and tuning the performance of the IDS using a GAN. development of the ma-
chine-learning IDS and the process of testing and tuning the performance of the IDS us-
ing
2.1. aPlatforms
GAN. and Components Used for the Development Environment
2.1.1. TensorFlow
2.1. Platforms and Components Used for the Development Environment
TensorFlow is one of the most powerful platforms for creating and maintaining deep-
2.1.1.
learningTensorFlow
applications. It was launched by Google as an end-to-end machine-learning
platform [22],
TensorFlow andisit was
one initially
of the most used powerful
for projectsplatforms
related to for
areas such asand
creating recommendation
maintaining
engines and even translation.
deep-learning applications. It is an open-source platform created
was launched by Google as an end-to-end for the automation of
ma-
machine-learning
chine-learning applications.
platform [22], and It itcontains tools, used
was initially libraries, and resources
for projects relatedthat allowsuch
to areas devel-
as
opers to build andengines
recommendation deploy andmachine-learning
even translation. applications for the entire
It is an open-source productcreated
platform lifecycle.
for
TensorFlow of
the automation canmachine-learning
run on both graphics processingItunits
applications. (GPUs)
contains and central
tools, libraries,processing
and re-
units (CPUs). Some components use NVidia’s CUDA language to
sources that allow developers to build and deploy machine-learning applications for theincrease performance in
environments where
entire product lifecycle. GPUs are used. However, most of the time, TensorFlow applications
are developed
TensorFlow oncan
desktops
run onwith bothCPUs only.processing
graphics The training of the
units model
(GPUs) andtakes placepro-
central on
these devices and, after it is completed, the model can be
cessing units (CPUs). Some components use NVidia’s CUDA language to increase per- deployed on the cloud, on
various desktop
formance platforms where
in environments running GPUscurrent operating
are used. systems,
However, mostor of
even
the on mobile
time, devices
TensorFlow
or IoT devices.
applications are developed on desktops with CPUs only. The training of the model takes
place on these devices and, after it is completed, the model can be deployed on the cloud,
2.1.2. Keras API
on various desktop platforms running current operating systems, or even on mobile de-
vicesKeras
or IoTisdevices.
a high-level neural network API written in Python that uses machine-learning
computation platforms as its backend, such as TensorFlow (the default one), CNTK, or
Theano
2.1.2. [23].API
Keras TensorFlow adopted Keras as the default high-level API in Python for its
2.0 release (the tf.keras package). It allows for fast prototyping, supporting both convolu-
Keras is a high-level neural network API written in Python that uses ma-
tional and recurrent networks (and combinations of the two), and can be used in both CPU
chine-learning computation platforms as its backend, such as TensorFlow (the default
and GPU environments.
one),ToCNTK,
performor Theano
low-level [23]. TensorFlow
operations, adopted
Keras must useKeras as the default
a backend engine,high-level
the defaultAPI in
being
Python for its 2.0 release (the tf.keras package). It allows for fast prototyping,
TensorFlow, as a mutual support between them. The backend can be changed by modifying supporting
both convolutional
a configuration file. and recurrent
Simpler networks
applications can(and combinations
use the Sequential of themodel
class two), available
and can be in
used in both CPU and GPU environments.
Keras. For more complex applications requiring a different arrangement of layers, the
Model Toclass
perform
with low-level
the functionaloperations, Keras must
API is available use a In
in Keras. backend
the caseengine, the default
of the Model class, be-
the
ing TensorFlow, as a mutual support between them. The backend can
layers are defined and then used to create the model. The learning and predicting processesbe changed by
modifying a configuration file. Simpler applications can use the Sequential
are the same for both model types. Keras can be deployed across a vast range of platforms, class model
available
includingin Keras.Cloud
Google For more complex
(by using applicationsServing
the TensorFlow requiring a different arrangement of
library).
layers, the Model class with the functional API is available in Keras. In the case of the
2.1.3. Development
Model class, the layersEnvironment
are defined and then used to create the model. The learning and
predicting
Google processes
Colab was arechosen
the same for development
as the both model types. Keras can
environment be deployed
because across a
of its simplicity,
vast
ease of use, and the fact that it provides computational resources when the code is run onli-a
range of platforms, including Google Cloud (by using the TensorFlow Serving
brary).
virtual machine. In addition, it supports most of the machine-learning libraries available
Sensors 2023, 23, 1315 5 of 32
ease of use, and the fact that it provides computational resources when the code is run on
a virtual machine. In addition, it supports most of the machine-learning libraries availa-
ble2.1.3.
on the market. The
Development environment provides most of needed libraries, such as numpy,
Environment
pandas, tensorflow, sklearn, keras, etc., and it can be checked by running the command
Google Colab was chosen as the development environment because of its simplicity,
Sensors 2023, 23, 1315 !pipeasefreeze. Additional
of use, and libraries
the fact that can be
it provides installed byresources
computational using the command
when the code !pip
is runinstall
5 of 32 .
on
In order
a virtual to maintain
machine. version
In addition, control over
it supports mostthe entire
of the project, the GitHub
machine-learning librariesplatform
availa- was
used. In the
ble on order to synchronize
market. The environment our local
providesrepository with the
most of needed remotesuch
libraries, one,as we used the
numpy,
Google
pandas,Colab “Save
tensorflow, a copy
sklearn, in Git
keras, option”,
etc., and which
it can be allowed
checked us
by to actively
running
on the market. The environment provides most of needed libraries, such as numpy, pandas, the commit
command to an
!pip repository.
existing freeze.
tensorflow, Additional
sklearn, Akeras, libraries
screen
etc., andcan
capture bethe
of
it can installed
be Google
checkedby Colab
using
by the command
interface
running !pip install
is presented
the command !pipin .
Figure
freeze.
In order
2.Additional to maintain
libraries can beversion
installedcontrol over the
by using the entire
command project,
!piptheinstall.
GitHub platform was
used. In order
In order to synchronize
to maintain versionour local over
control repository withproject,
the entire the remote one, weplatform
the GitHub used thewas
Google
used. Colabto“Save
In order a copy in
synchronize ourGit option”,
local whichwith
repository allowed us to actively
the remote one, we commit
used thetoGoogle
an
existing
Colab repository.
“Save a copy A inscreen capture which
Git option”, of the Google
allowed Colab
us tointerface
activelyiscommit
presented to in
anFigure
existing
2.
repository. A screen capture of the Google Colab interface is presented in Figure 2.
Figure 2. Screen capture of the Google Colab interface.

Figure2.2.Screen
Figure Screencapture
capture of
of the
the Google
Google Colab
Colabinterface.
interface.
2.2. Development of a Machine-Learning IDS
2.2.
2.2.Development
Development
The ofofaa Machine-Learning
steps involved Machine-Learning IDS
IDS
in the development were data preprocessing, creation of the
model,TheThe steps
stepsinvolved
training involved
and inin
testing the development
ofthe
thedevelopment were
IDS, creation data
were
of preprocessing,
data
the preprocessing,
model, creation
and then of the
creation
training ofofmodel,
the
the IDS
training
model, and testing
training and of the
testing IDS,
of thecreation
IDS, of
creationthe
ofmodel,
the and
model,
and testing of its performance using the GAN architecture. Figure 3 presents the then
and training
then of
training the
of IDS
the IDS and
pro-
and
testingtesting
of its of its
performanceperformance
using using
the GANthe GAN architecture.
architecture. Figure Figure
3 3 presents
presents
posed flow for the IDS using the machine-learning algorithms described in the previous the the
proposedpro-flow
posed
for flowusing
the The
IDS for the theIDS using the machine-learning
machine-learning algorithms algorithms
described described
in in thesection.
the previous previousThe
section. building blocks are described in Section 2.2.
section. blocks
building The building blocks are
are described in described in Section 2.2.
Section 2.2.
Figure 3. Proposed flow for the machine-learning IDS.
In order to train the model, the NSL-KDD Train supervised dataset was used. The NSL-
KDD Test dataset was used to test the model and evaluate its performance. These datasets
are described in Section 1.2. The data preprocessing step is described in Section 2.2.1.
The model was trained one algorithm at a time, and the entire process is described in
Section 2.2.2. Then, in the testing phase, the model received inputs processed from the
2.2.1. The model was trained one algorithm at a time, and the entire process is described
in Section 2.2.2. Then, in the testing phase, the model received inputs processed from the
NSL-KDD Test dataset. The output of the model was a classification in five classes: nor-
mal traffic, DoS, probe, U2R, and R2L.
Sensors 2023, 23, 1315 6 of 32
2.2.1. Data Preprocessing

NSL-KDD Test
The first implementation dataset.
step in theThedevelopment
output of the model wasmachine-learning
of the a classification in fiveIDS
classes:
was normal
traffic, DoS, probe, U2R, and R2L.
data preprocessing. This step involves the improvement of the dataset quality, which will
affect the IDS performance. The data preprocessing step included dataset splitting and
2.2.1. Data Preprocessing
loading, correlation, removal
The firstof the highly correlated
implementation features, and
step in the development data
of the entry normali-
machine-learning IDS was
zation. The dataset was split into two files, the training dataset kdd_train.csv and the will
data preprocessing. This step involves the improvement of the dataset quality, which
affect the IDS In
testing dataset kdd_test.csv. performance.
addition The to data
the preprocessing
NSL-KDD step included
dataset, thedataset splitting and
additional
loading, correlation, removal of the highly correlated features, and data entry normalization.
metadata files nslkdd.names and training_attack_types were added, which provided
The dataset was split into two files, the training dataset kdd_train.csv and the testing
information about the attack
dataset types inInthe
kdd_test.csv. dataset.
addition to theThese
NSL-KDD filesdataset,
were thekept in the metadata
additional Google files
Drive directory thatnslkdd.names
was created andfor the development
training_attack_types wereofadded,
the system presented
which provided in thisabout
information
paper. In order to access these files, it was necessary to grant access to the Google Drive that
the attack types in the dataset. These files were kept in the Google Drive directory
instance where the was
filescreated
were for the development of the system presented in this paper. In order to access
saved. After access was granted, the files could be loaded
these files, it was necessary to grant access to the Google Drive instance where the files
using the mount command.
were saved.Figure 4 shows
After access the commands
was granted, that
the files could can beusing
be loaded usedthetomount
performcommand.
the mounting task. Figure 4 shows the commands that can be used to perform the mounting task.
Figure 4. Mounting the drive to access the dataset.

Figure 4. Mounting the drive to access the dataset.
At runtime, authorization was requested to allow notebook access to the Drive location.
At runtime, authorization
The content ofwas requested
the file was loaded to using
allowthenotebook
DataFrameaccess
propertyto of
thetheDrive
pandas lo-library.
Then, we mapped the attack entries to match the major
cation. The content of the file was loaded using the DataFrame property of the pandasattack categories probing (probe),
denial of service (DoS), user to root (U2R), and remote to local (R2L). In the NSL-KDD
library. Then, we mapped the attack entries to match the major attack categories probing
files, the attack type was specified for each record. Based on the training_attack_types
(probe), denial of service (DoS), user
file, we performed to root (U2R),
the conversion andDoS,
to normal, remote
probe,toU2R,
local
and(R2L). In the
R2L. Next, the data
NSL-KDD files, thecorrelation
attack step
typewaswas specified for each record. Based on the train-
performed. Correlation is a statistical measure that expresses the linear
relationship
ing_attack_types file, we performedbetweenthetwoconversion
variables. Fortothis, we used
normal, DoS,the probe,
corr() method
U2R, of andtheR2L.
dataframe
object. It is recommended that highly correlated features be removed from a dataset in
Next, the data correlation step was performed. Correlation is a statistical measure that
order to improve the performance of machine-learning algorithms. Features with a high
expresses the linearcorrelation
relationship
have abetween
value closetwo variables.
to 1.0. For example,Forthethis, we dst_host_srv_rerror_rate
features used the corr()
method of the dataframe object. It have
and srv_rerror_rate is recommended
a correlation valuethat highly
of 0.97. correlated
The heat map of thefeatures
correlatedbefeatures
removed from a dataset in order to improve the performance of machine-learning algo-
is presented in Figure 5.
rithms. Features with aThe highhighly correlated have
correlation features
a were
valueremoved
close tofrom theFor
1.0. dataset in order the
example, to improve
fea- the
performance of the machine-learning algorithm. Once this step was completed, we were
tures dst_host_srv_rerror_rate and srv_rerror_rate have a correlation value of
left with four categorical features (protocol_type, flag, service, and attack_type) that had to
0.97. The heat map of
be the correlated
mapped features
to numerical is presented
values. in Figure
The service feature was 5.dropped as it did not influence
the learning process. We did not notice any performance improvement when the service
parameter was present in the training and testing datasets. An alternative way to reduce
the features is to use a BAT evolutionary algorithm [10].
Next, the dataset was split in two to prevent overfitting of the models. The first half
was used in the IDS model training, while the second was used for the GAN model training.
Figure 6 presents the modality used to split the dataset in two: the first half for the IDS
model training and the second for the GAN model training.
Sensors 2023, 23, 1315 7 7of
of 32
Figure 5. Heat map of the correlated features.
The highly correlated features were removed from the dataset in order to
the performance of the machine-learning algorithm. Once this step was comple
were left with four categorical features (protocol_type, flag, service, and attack_ty
had to be mapped to numerical values. The service feature was dropped as it did
fluence the learning process. We did not notice any performance improvement w
service parameter was present in the training and testing datasets. An alternative
reduce the features is to use a BAT evolutionary algorithm [10].
Next, the dataset was split in two to prevent overfitting of the models. The
was used in the IDS model training, while the second was used for the GAN
training. Figure 6 presents the modality used to split the dataset in two: the first
the IDS
Figure
Figure 5. model
5. Heat
Heat map training
map of
of the andfeatures.
the correlated
correlated the second for the GAN model training.
features.
The highly correlated features were removed from the dataset in order to improve
the performance of the machine-learning algorithm. Once this step was completed, we
were left with four categorical features (protocol_type, flag, service, and attack_type) that
had to be mapped to numerical values. The service feature was dropped as it did not in-
fluence the learning process. We did not notice any performance improvement when the
service parameter was present in the training and testing datasets. An alternative way to
reduce the features is to use a BAT evolutionary algorithm [10].
Next, the dataset was split in two to prevent overfitting of the models. The first half
was used in the IDS model training, while the second was used for the GAN model
training. Figure 6 presents the modality used to split the dataset in two: the first half for
the IDS model training and the second for the GAN model training.
Figure
Figure Dataset
6. 6. splitsplit
Dataset in two
intotwo
trainto
the IDS and
train the GAN models.
IDS and GAN models.
The training and testing of the GAN was undertaken using data from one attack type
at a time. Therefore, the training data had to be further split into dedicated attack-type
data frames.
The last step in the data preprocessing step was the normalization process for the data
entries. The reason for the normalization was the fact that variables that are measured at
different scales do not contribute equally to model training and are prone to create bias. In
order to deal with this potential problem, feature-wise normalization, such as MinMaxScal-
ing, was used before fitting the model. By doing so, all features were transformed into the
Figure 6. Dataset
[0, 1] range. split7in
Figure two to train
presents the IDScode
the source and GAN models.
that used to apply MinMax normalization to
the training dataset.
The last
The last step
step inin the
the data
data preprocessing
preprocessing step
step was
was the
the normalization
normalization process
process for
for the
the
data entries. The reason for the normalization was the fact that variables
data entries. The reason for the normalization was the fact that variables that are meas-that are meas-
ured at
ured at different
different scales
scales dodo not
not contribute
contribute equally
equally to
to model
model training
training and
and are
are prone
prone toto cre-
cre-
ate bias. In order to deal with this potential problem, feature-wise normalization,
ate bias. In order to deal with this potential problem, feature-wise normalization, such as such as
MinMaxScaling, was used before fitting the model. By doing so, all features
MinMaxScaling, was used before fitting the model. By doing so, all features were trans- were trans-
Sensors 2023, 23, 1315 8 of 32
formed into
formed into the
the [0,
[0, 1]
1] range.
range. Figure
Figure 77 presents
presents the
the source
source code
code that
that used
used to
to apply
apply MinMax
MinMax
normalization to
normalization to the
the training
training dataset.
dataset.
Figure7.7.
Figure
Figure 7.Training
Trainingdataset
Training datasetnormalization
dataset normalizationusing
normalization usingMinMax.
using MinMax.
MinMax.
At
Atthe
At theend
the endof
end ofthe
of thepreprocessing
the preprocessingstep,
preprocessing step,the
step, thedataset
the datasetcontained
dataset contained31
contained 31normalized
31 normalizednumerical
normalized numerical
numerical
features
features separated into dedicated attack-type training and test datasets. Figure 8 presents
features separated
separated into
into dedicated
dedicated attack-type
attack-type training
training and
andtest
testdatasets.
datasets.Figure
Figure88 presents
presents
the
thetest
testentry
entryfor
forone
one of
ofthe
theattacks
attacks(the
(theDoS
DoS attack).
attack).
the test entry for one of the attacks (the DoS attack).
Figure 8. Record from the testing dataset for the DoS attack.
Figure 8.
Figure 8. Record
Record from
from the
the testing
testing dataset
dataset for
for the
the DoS
DoS attack.
attack.
2.2.2. Implementation of the machine-learning IDS
2.2.2.The
2.2.2. Implementation of the
the machine-learning
IDS model creation
Implementation of machine-learning IDS
step was realized IDS
in the IDS.ipynb notebook based on both
classical
Themachine-learning
The IDS model
IDS model creation algorithms
creation step wasand
step was neuralin
realized
realized network
in models.notebook
the IDS.ipynb
the IDS.ipynb The first step
notebook based
based consisted
on both
on both
of loading
classical the preprocessed
machine-learning dataset
algorithmsfrom the
and previous
neural step.
network Once the
models.
classical machine-learning algorithms and neural network models. The first step con- dataset
The was
first loaded,
step con-
we started
sisted of training
loading our
the IDS model.
preprocessed Five different
dataset from algorithms
the previous were
step.
sisted of loading the preprocessed dataset from the previous step. Once the dataset wasused:
Once the dataset was
loaded,
•loaded, we
K-nearest started training
neighbors
we started our
(using
training IDS model. Five different
the KNeighborsClassifier
our IDS model. Five different class algorithms
from the
algorithms were used:
sklearn
were used:library);
••• Decision
K-nearest tree (using
neighbors the DecisionTreeClassifier
(using the class
KNeighborsClassifier from the
class sklearn
from
K-nearest neighbors (using the KNeighborsClassifier class from the sklearn library); the library);
sklearn library);
••• Random
Decision forest
tree (using
(using the
the RandomForestClassifier
DecisionTreeClassifier class
class from
from thethe
Decision tree (using the DecisionTreeClassifier class from the sklearn library); sklearn
sklearn library);
library);
••• Support
Randomvector
Random forestmachine
forest (using the
(using
(using
the the SVC class from the
RandomForestClassifier
RandomForestClassifier
sklearn
class
class from the
from
library);
the sklearn library);
sklearn library);
•• Artificial
Support neural
vector network
machine (using
(using the
the Keras
SVC API
class with
from Relu
the and
sklearnLeakyRelu
library); as the activa-
• Support vector machine (using the SVC class from the sklearn library);
••
tion functions).
Artificial neural network
network (using
(using thethe Keras
Keras API
API with
with Relu
Relu and
and LeakyRelu
LeakyRelu as as the
the ac-
ac-
Artificial neural
tivation
The first functions).
algorithm
tivation functions). used was K-nearest neighbors. From the sklearn library, we
imported the KNeighborsClassifier
The first
first algorithm usedused was class. Theneighbors.
was K-nearest
K-nearest optimal K Fromvaluetheprimarily
sklearnchosen
library,waswe theim-
The algorithm neighbors. From the sklearn library, we im-
square
ported root
the of N, where N was the
KNeighborsClassifier totalThe
class. number
optimalof data
K points.
value Figurechosen
primarily 9 presentswas the
the
Sensors 2023, 23, 1315 ported the KNeighborsClassifier class. The optimal K value primarily chosen was 9 ofthe
32
source code
square root for
root of
of N,the K-nearest
N, where
where N N wasneighbors
was the
the total algorithm
total number
number of and the
of data performance
data points.
points. Figure for the training
Figure 99 presents
presents the the
square
and testing
source codedataset.
for the
the K-nearest
K-nearest neighbors
neighbors algorithm
algorithm andand the
the performance
performance for for the
the training
training
source code for
and testing dataset.
and testing dataset.
Figure 9. Source
Figure9. Source code
code for
for the
theIDS
IDSwith
withthe
theK-nearest
K-nearestneighbors
neighborsalgorithm.
algorithm.
For
For the
the decision
decision tree
tree algorithm,
algorithm, the
the DecisionTreeClassifier
DecisionTreeClassifier class
class was
was imported
imported from
from
the
the sklearn library. The max_depth value represented the height of the binary tree.
sklearn library. The max_depth value represented the height of the binary tree. The
The
value for max_depth was selected after numerous trials. Figure 10 presents part of the
source code for the decision tree algorithm.
Figure 9. Source code for the IDS with the K-nearest neighbors algorithm.
Figure 9. Source code for the IDS with the K-nearest neighbors algorithm.
Sensors 2023, 23, 1315 9 of 32
For the decision tree algorithm, the DecisionTreeClassifier class was imported from
For the library.
the sklearn decisionThe
treemax_depth
algorithm, the DecisionTreeClassifier
value class
represented the height ofwas
the imported from
binary tree. The
the sklearn library. The max_depth value represented the height of the binary
value for max_depth was selected after numerous trials. Figure 10 presents part of tree. Thethe
value
value for max_depth
max_depth was was selected after numerous
numerous trials.
trials. Figure
Figure10
10presents
presentspart
partofofthe
the
sourcefor
code for the decision tree algorithm.
source
sourcecode
code for
for the
the decision
decision tree algorithm.
Figure 10.Source
Source codefor
for theIDS
IDSwith
withthe
thedecision
decisiontree
treealgorithm.
algorithm.
Figure 10. Source code
Figure10. code for the
the IDS with the decision tree algorithm.
The
The third
The third algorithm
third was
algorithm was
algorithm the
was the random
therandom forest
randomforest algorithm,
forestalgorithm,
algorithm,forfor which
forwhich we
whichwe used
weused
usedthethe Ran-
theRan-
Ran-
domForestClassifier
domForestClassifier class
class imported
imported from
from the
the sklearn
sklearn library.
library. The
The number
number
domForestClassifier class imported from the sklearn library. The number of estimatorsofofestimators
estimators
was
was selected
was selected after
selected after numerous
after numerous trials.
numerous Figure
trials. Figure
trials. Figure1111 presents
11presents the
presentsthe source
thesource code
sourcecode for
codefor the
forthe random
therandom
random
forest
forestalgorithm.
algorithm.
forest algorithm.
Figure 11. Source

Figure11. Sourcecode
Source codefor
code forthe
theIDS
IDSwith
withthe random
therandom forest
randomforest algorithm.
forestalgorithm.
algorithm.
The
The fourth
fourthalgorithm
fourth algorithmwas
algorithm was the
wasthe support
thesupport vector
supportvector machine
vectormachine algorithm,
machinealgorithm, for
algorithm,for which
forwhich we
whichwe used
weused
used
Sensors 2023, 23, 1315 10 of 32
the SVC class
the SVC class from
class from the
from the sklearn
the sklearn library.
sklearn library. The
library.The source
Thesource code
sourcecode for
codefor the
forthe IDS
theIDS obtained
IDSobtained with
obtainedwith this
withthis
this
algorithm is
algorithm is presented
is presented in
presented in Figure
inFigure 12.
Figure12.
12.
Figure 12.Source
Figure12. Sourcecode
codefor
forthe
theIDS
IDSwith
withthe
theSVM
SVMalgorithm.
algorithm.
For the last model, the artificial neural network, the Keras API was involved. The
model used the Keras Sequential class, which was a convenient way of adding a linear
stack of layers. The architecture of the model was designed in such a way as to limit the
number of hidden layers. The activation functions for the input layer and the hidden
layers were the following: Relu and LeakyRelu. At the output layer, the Softmax activation
layers were the following: Relu and LeakyRelu. At the output layer, the Softmax activa-
function was chosen, predicting a multinomial probability distribution. Figure 13 presents
tion function was chosen, predicting a multinomial probability distribution. Figure 13
the source code for the artificial neural network model.
presents the source code for the artificial neural network model.
layers were the following: Relu and LeakyRelu. At the output layer, the Softmax activa-
Sensors 2023, 23, 1315 10 of 32
tion function was chosen, predicting a multinomial probability distribution. Figure 13
presents the source code for the artificial neural network model.
Figure 13. createModel method for the ANN IDS.

Figure 13. createModel method for the ANN IDS.
Next,we
Next, we used
used the
theKerasClassifier
KerasClassifierclass from
class thethe
from scikeras library,
scikeras which
library, provided
which us
provided
Sensors 2023, 23, 1315 with a wrapper for the deep-learning models to be used for classification. Figure
us with a wrapper for the deep-learning models to be used for classification. Figure 14 pre-
11 of 14
32
sents the source code for the training of the ANN model.
presents the source code for the training of the ANN model.
Figure 14. Source code for the training of the ANN IDS.
2.2.3. Performance Analysis of the Machine-Learning IDS

In the last step, all the IDS models were compared, and the first three classifiers were
saved for use in the performance evaluation and training of the GAN. The models all
performed very well when using the training dataset, exhibiting accuracy close to or over
0.9. However, only the ANN, random forest, and K-nearest neighbor classifiers were
Sensors 2023, 23, 1315 11 of 32
2.2.3. Performance Analysis of the Machine-Learning IDS

In the last step, all the IDS models were compared, and the first three classifiers
were saved for use in the performance evaluation and training of the GAN. The models
all performed very well when using the training dataset, exhibiting accuracy close to
or over 0.9. However, only the ANN, random forest, and K-nearest neighbor classifiers
were used in the next step. For the performance analysis [24], we computed the precision
(or detection rate), recall, and F1 score (which combines both precision and recall) using
Equations (1)–(3). In addition, we also show the confusion matrix.
TP
Precision = (1)
TP + FP
TP
Recall = (2)
TP + FN
Precision ∗ Recall
F1 = 2 ∗ (3)
Precision + Recall
where TP (true positive) represents the number of records correctly matched as attack
traffic, TN (true negative) represents the number of records correctly matched as normal
traffic, FP (false positive) represents the number of normal records incorrectly labeled as
attack traffic, and FN (false negative) represents the number of attack records incorrectly
labeled as normal traffic.
Table 1 presents the performance analysis and Table 2 presents the confusion matrix
for the kNN IDS.
Table 1. Performance analysis of the kNN IDS.
Predicted Attack Category Precision Recall F1 Score

Normal 0.92 0.98 0.95
DoS 0.95 0.93 0.94
Probe 0.92 0.67 0.77
R2L 0.52 0.41 0.46
U2R 0 0 0
Table 2. Confusion matrix for the kNN IDS.
Predicted Attack Category Normal DoS Probe R2L U2R

Normal 97.91% (32,751) 0.94% (317) 0.76% (257) 0.36% (122) 0
DoS 6.30% (1456) 93.26% (21,553) 0.37% (87) 0.06% (14) 0
Probe 18.72% (1100) 13.58% (798) 66.58% (3912) 1.10% (65) 0
R2L 57.57% (304) 0 1.70% (9) 40.72% (215) 0
U2R 96.23% (26) 3.7% (1) 0 0 0
for the decision tree IDS.
for the random forest IDS.
Sensors 2023, 23, 1315 12 of 32
Table 3. Performance analysis of the decision tree IDS.

Normal 0.98 0.99 0.99
DoS 1.00 0.99 1.00
Probe 0.96 0.95 0.95
R2L 0.94 0.78 0.85
U2R 1.00 0.22 0.36
Table 4. Confusion matrix for the decision tree IDS.

Normal 99.12% (33,152) 0.14% (48) 0.66% (221) 0.07% (26) 0
DoS 0.49% (113) 99.4% (22,972) 0.1% (25) 0 0
Probe 4.87% (286) 0.03% (2) 95.08% (5586) 0.01% (1) 0
R2L 22.35% (118) 0 0 77.65% (528) 0
U2R 77.78% (21) 3.7% (1) 0 0 22.22% (6)
Table 5. Performance analysis of the random forest IDS.

Normal 1.00 1.00 1.00
DoS 1.00 1.00 1.00
Probe 1.00 1.00 1.00
R2L 1.00 0.99 0.99
U2R 1.00 0.89 0.94
Table 6. Confusion matrix for the random forest IDS.

Normal 99.99% (33,443) 0.002% (1) 0.008% (3) 0 0
DoS 0.02% (6) 99.97% (23,103) 0.004% (1) 0 0
Probe 0.10% (6) 0 99.89% (5869) 0 0
R2L 1.13% (6) 0 0 98.86% (522) 0
U2R 11.11%(3) 0 0 0 88.88% (24)
for the SVM IDS.
for the ANN IDS.
The values in Tables 1–10 were computed with the testing dataset. It was found
that the machine-learning IDSs performed very well, especially for the normal, DoS, and
probe classes with high numbers of records in the dataset. The U2R attack had a very low
number of records in the training dataset and, with one exception (random forest), was
not recognized correctly by the algorithms, raising a high number of false negatives. The
high value for the precision parameter in the case of U2R was not a good indicator, as
the total number of false positives was 0 and it did not take into consideration the high
number of false negatives. Another point is that the usage of artificial neural networks
seems promising, since even with a simple ANN the results were similar to the other
Sensors 2023, 23, 1315 13 of 32
algorithms tested. Compared with the results presented in [9], we obtained better results
with the testing dataset for classes that were better represented in terms of the number
of samples in the dataset. However, the authors of [9] obtained better performance using
PPGO, a bio-inspired optimization technique, for the U2R class, which had few samples in
the dataset.
Table 7. Performance analysis of the SVM IDS.

Normal 0.86 0.99 0.92
DoS 0.96 0.92 0.94
Probe 0.93 0.37 0.53
R2L 0.00 0.00 0.00
U2R 0.00 0.00 0.00
Table 8. Confusion matrix for the SVM IDS.

Normal 99.29% (33,210) 0.52% (175) 0.18% (62) 0 0
DoS 7.75% (1791) 91.8% (21,220) 0.43% (99) 0 0
Probe 49.58% (2913) 13.65% (802) 36.77% (2160) 0 0
R2L 99.24% (524) 0 0.76% (4) 0 0
U2R 96.3% (26) 3.7% (1) 0 0 0
Table 9. Performance analysis of the ANN IDS.

Normal 0.99 0.99 0.99
DoS 1.00 1.00 1.00
Probe 0.98 0.96 0.97
R2L 0.81 0.79 0.80
U2R 1.00 0.56 0.71
Table 10. Confusion matrix for the ANN IDS.

Normal 99.17% (33,169) 0.19% (63) 0.36% (122) 0.28% (93) 0
DoS 0.06% (13) 99.92% (23,093) 0.01% (3) 0.004% (1) 0
Probe 3.52% (207) 0.1% (6) 96.36% (5661) 0.02% (1) 0
R2L 20.83% (110) 0 0 79.17% (418) 0
U2R 25.92% (7) 3.7% (1) 3.7% (1) 11.11% (3) 55.56% (27)
2.3. Development of a GAN for Testing and Tuning of the IDS

In order to develop the GAN, the generator and the discriminator had to be imple-
mented. To make testing easier, a configuration file was created. Then, the IDS performance
had to be tested (using the models that were developed in the previous step).
In order to develop the GAN, the generator and the discriminator
implemented. To make testing easier, a configuration file was created. Th
performance had to be tested (using the models that were developed in t
Sensors 2023, 23, 1315 step). 14 of 32
2.3.1. The Configuration File

2.3.1. The Configuration File
To change the model’s parameters when testing the GAN against
To change the model’s parameters when testing the GAN against the IDS, a con-
configuration
figuration file wasfile wasincreated
created the JSONinformat.
the JSON
Figureformat. Figure
con
15 presents 15 presents
the contents on the the
configuration
configuration file the
file with with the model’s
model’s parameters.
parameters.
Sensors 2023, 23, 1315 15 of 32

Figure 15. Contents of the configuration file.
The
Theconfiguration
configuration file
file data had to
data had to be
beloaded
loadedbefore
beforebeing
beingused
usedforfor training
training or or testing
testing
purposes.
purposes.The
The creation
creation and
and training of the
training of theGAN
GANwas wasundertaken
undertakeninin thethe IDSGAN.ipynb
IDSGAN.ipynb
notebook.
notebook.The
Thedataset
datasetdedicated
dedicatedto tothe
theGAN
GANwas wasloaded
loadedfrom
fromthe
theGoogle
Google Drive
Drive directory.
direc-
The commands
tory. used to
The commands load
used tothe
loadcontent of theofdataset
the content are presented
the dataset in Figure
are presented 16. 16.
in Figure
Figure 16. Loading of the dataset used for the GAN.

Afterthat,
After that, we
we loaded
loaded the
the IDS
IDS models
models that
that were
werecreated
createdininSection
Section2.2.
2.2.Figure 17 17
Figure
presents the source code for loading of the models used for GAN.
The parameters of the model were loaded from the configuration file as is presented
in Figure 18.
Figure 17. Loading of the models used for the GAN.

After that, we loaded the IDS models that were created in Section 2.2. Figure 17
presents
Figure 16.the source
Loading code
of the for loading
dataset used for of
thethe models used for GAN.
GAN.
Sensors 2023, 23, 1315 15 of 32
After that, we loaded the IDS models that were created in Section 2.2. Figure 17
The
Figure 17.parameters ofmodels
Loading of the the model were
used for theloaded
GAN. from the configuration file as is presented
in Figure 18.
The parameters of the model were loaded from the configuration file as is presented
in Figure 18.
Sensors 2023, 23, 1315 16 of 32

Figure18.
Figure 18.Loading
Loadingthe
theparameters
parametersfrom
fromthe
theconfiguration
configurationfile.
file.
Figure 18. Loading the parameters from the configuration file.
The training of the model was undertaken according to the configuration file data as
The training of the model was undertaken according to the configuration file data as
isispresented
presentedininFigure
Figure19.
19.
Figure 19. Training of the GAN according to the configuration file.

Thetraining
The training of
of the
the model
model was
was undertaken
undertakenaccording
accordingtotothe configuration
the filefile
configuration data.
data.
First, the components of the IDSGAN architecture were initiated. This was undertaken
First, the components of the IDSGAN architecture were initiated. This was undertaken with
with the help of the createIDSGAN<attack_type>Components() method, which returned
the help of the createIDSGAN<attack_type>Components() method, which returned object
object instances of the GAN model, generator, and discriminator as is described in Figure
instances of the GAN model, generator, and discriminator as is described in Figure 20.
20.
The training of the model was undertaken according to the configuration file data.
First, the components of the IDSGAN architecture were initiated. This was undertaken
Sensors 2023, 23, 1315 with the help of the createIDSGAN<attack_type>Components() method, which returned 16 of 32
object instances of the GAN model, generator, and discriminator as is described in Figure
20.
Figure20.
Figure 20.createIDSGANDOSComponents()
createIDSGANDOSComponents() method
methodexample
examplefor
forthe
theDoS
DoSattack type.
attack type.
Thismethod
This method involved
involved thethe build_generator()and
build_generator()and build_discriminator()
build_discriminator() methods,
methods, which
which created and returned the model instances for the generator and discriminator.
created and returned the model instances for the generator and discriminator.
2.3.2.The
2.3.2. TheGenerator
Generator Component
Component of
of the
the GAN
GAN
Sensors 2023, 23, 1315
Thebuild_generator()
The build_generator()method
method created
created a sequential
a sequential model
model withwith
threethree layers.
layers. Its
Its content
content is presented in
is presented in Figure 21.Figure 21.
Figure
Figure 21.21. build_generator()
build_generator() method.
method.
The model.summary() method offered a graphical representation of the model coupled

The model.summary() method offered a graphical representation of the mo
with the trainable and non-trainable parameters. Figure 22 presents the summary of the
pled with
generator the where
model, trainable and non-trainable
the Param parameters.
# represents the number Figure 22
of the parameters forpresents
each layer.the sum
the generator model, where the Param # represents the number of the param
2.3.3. The Discriminator Component of the GAN
each layer.
The build_discriminator() method created a sequential model with the same structure
as the one from the ANN IDS discussed in Section 2.2.2. The model.summary() method for
the discriminator is presented below in Figure 23. We can see the difference in the trainable
parameters, as the discriminator model had an input shape of “(None, 31)”, whereas the
generator model had an input shape of “(None, 40)”. This was due to the fact that the
generator received as input the modified adversarial data entry, whereas the discriminator
handled data with the original format.
Figure 21. build_generator() method.
The model.summary() method offered a graphical representation of the model cou-

pled with the trainable and non-trainable parameters. Figure 22 presents the summary of
Sensors 2023, 23, 1315 the generator model, where the Param # represents the number of the parameters
17 of 32 for
each layer.
3, 1315 18 of 32
22.Summary
Figure 22.
Figure Summaryofof
thethe
generator model.
generator model.
2.3.3. The Discriminator Component of the GAN

The build_discriminator() method created a sequential model with the same struc-
ture as the one from the ANN IDS discussed in Section 2.2.2. The model.summary()
method for the discriminator is presented below in Figure 23. We can see the difference in
the trainable parameters, as the discriminator model had an input shape of “(None, 31)”
whereas the generator model had an input shape of “(None, 40)”. This was due to the fact
that the generator received as input the modified adversarial data entry, whereas the
discriminator handled data with the original format.
Figure 23. The discriminator model

Figure 23. The method. model method.
discriminator
2.3.4. The Training Algorithm

2.3.4. for the
The Training GAN for the GAN
Algorithm
Next,with
Next, in accordance in accordance with the configuration
the configuration data, the train<attack_type>IDSGAN
data, the train<attack_type>IDSGAN
method was called. This method is intermediary, and it doesitnot
method was called. This method is intermediary, and doesinclude
not include
thethe actual training
actual
training algorithm but instead performs additional logical operations before calling thethe actual
algorithm but instead performs additional logical operations before calling
training method. Figure 24 presents the intermediary training method.
actual training method. Figure 24 presents the intermediary training method.
Figure 23. The discriminator model method.
Figure 23. The discriminator model method.
2.3.4. The Training Algorithm for the GAN

2.3.4. The Training Algorithm for the GAN
Next, in accordance with the configuration data, the train<attack_type>IDSGAN
Next, in accordance with the configuration data, the train<attack_type>IDSGAN
method was called. This method is intermediary, and it does not include the actual
method was called. This method is intermediary, and it does not include the actual
Sensors 2023, 23, 1315 18 the
training algorithm but instead performs additional logical operations before calling of 32
training algorithm but instead performs additional logical operations before calling the
Figure 24. The intermediary training method.

The trainIDSGAN method handled the training of the GAN model. Due to the fact
The trainIDSGANmethod methodhandled
handledthe thetraining
training of the GAN model. Due to the
factfact
that The
the trainIDSGAN
GAN model is a relatively new discovery ofin
thetheGAN model. Due
machine-learning to world,
the that
the
thatGAN
the the GAN
model model
is a is a relatively
relatively new new discovery
discovery in the in the machine-learning
machine-learning world, world,
the the
algorithm
algorithm for the GAN model had to be written manually. Additional helper methods
algorithm
for theused
GAN formodel
the GAN had model
to of had to manually.
bebatches
written be written Additional
manually. Additional helperwere
helper methods methods
were for the fetching of data, creation of the adversarial sample, andused re-
were
for used for
the fetching the fetching of
of batchesfeatures. batches
of data, The of data,
creation creation
of the of the adversarial sample, andofre-
taining of the functional creation of adversarial
an adversarial sample, andinretaining
sample the generator the
taining of features.
functional the functional
The features.
creation ofThe
an creation of sample
adversarial an adversarial
in the sample inwas
generator theintended
generatorto
was intended to keep the functional features specific to the attack and to change some of
was intended
keep to keep the functional features specific to the attack and to change some of
the functional
the other features. features specific to the attack and to change some of the other features.
the other
The features.
ThegetBatch()
getBatch() method
method presented
presented inin Figure
Figure 2525 was
was responsible
responsiblefor
forreturning
returningaabatch
batch
size The
for getBatch()
the data used method
in the presented
training of in Figure
the model. 25 was training
Batch responsible
wasfor returningtoarequire
considered batch
size for the data used in the training of the model. Batch training was considered to re-
sizememory
less for the data
and used wasin
it and the training
to helptoof the model. Batch training was considered to re-
quire less memory itfound
was found the model
help train
the model faster.
train faster.
quire less memory and it was found to help the model train faster.
Sensors 2023, 23, 1315 19 of 32
Figure 25. getBatch()

Figure25. getBatch() method.
method.
The
The adjustBinaryFeatures() method described
adjustBinaryFeatures() method describedin inFigure
Figure2626was
wasused
usedtotoadjust
adjustthe
the
binary
binary features from the dataset. Generated binary features were set to the value 0 if theif
features from the dataset. Generated binary features were set to the value 0
the value
value waswas below
below a predefined
a predefined threshold
threshold or toorthe
to value
the value 1 if value
1 if the the value was above
was above the
the threshold.
threshold.
Figure 26. adjustBinaryFeatures () method.

Figure 26. adjustBinaryFeatures() method.
The adjustContinuousFeatures()
The adjustContinuousFeatures() method methoddescribed
describedininFigure
Figure2727was
wasused
usedtotoadjust
adjust
the continuous features from the dataset. Generated continuous features wereset
the continuous features from the dataset. Generated continuous features were settotothe
the
value 00 ifif the
value the generated
generated values
values were
were negative
negative and
and to
to the
thevalue
value11ififthe
thegenerated
generatedvalues
values
were above
were above 1. 1. If
If they
they were
were within
within the
the [0,
[0, 1]
1] interval,
interval, then
thenthe
thegenerated
generatedvalues
valueswere
werenotnot
adjusted. This was required due to the fact that these continuous features were
adjusted. This was required due to the fact that these continuous features were normalized normal-
ized during
during the data the processing
data processing
step. step.
Figure 26. adjustBinaryFeatures () method.
The adjustContinuousFeatures() method described in Figure 27 was used to adjust
The adjustContinuousFeatures()
the continuous method
features from the dataset. described
Generated in Figure features
continuous 27 was used
weretoset
adjust
to the
the continuous
value features from
0 if the generated the
values dataset.
were Generated
negative and tocontinuous
the value 1features were set tovalues
if the generated the
valueabove
were 0 if the
1. generated
If they werevalues were
within thenegative and to the
[0, 1] interval, thenvalue 1 if the generated
the generated values
values were not
Sensors 2023, 23, 1315 19 of 32
were above
adjusted. This1. was
If they were within
required due tothe
the[0,fact
1] interval, then
that these the generated
continuous values
features were
were not
normal-
adjusted.
ized during This
thewasdatarequired duestep.
processing to the fact that these continuous features were normal-
ized during the data processing step.
Figure27.
27. adjustContinuousFeatures () method.
method.
Figure 27.adjustContinuousFeatures()
Figure adjustContinuousFeatures () method.
The adjustDiscreteFeatures()
The adjustDiscreteFeatures() method described
methoddescribed in
describedin Figure
inFigure
Figure 28 was used to adjust the
The adjustDiscreteFeatures() method 2828 was
was usedusedto to adjust
adjust thethe
discretefeatures
discrete featuresfrom
fromthe
thedataset.
dataset.Generated
Generateddiscrete
discretefeatures
featureswere
weresetsettoto
thethe value
value 0 0 the
if if
discrete features from the dataset. Generated discrete features were set to the value 0 if
the generated
generated valuesvalues
were were negative
negative and and
to tonearest
the the nearest
positivepositive integer
integer if the if the generated
generated values
the generated values were negative and to the nearest positive integer if the generated
values
were were
were1.above
above
values above 1.
1.
Figure
Figure28.
28.adjustDiscreteFeatures() method.
adjustDiscreteFeatures() method.
Figure 28. adjustDiscreteFeatures() method.
TheretainFunctionalFeatures()
The
The retainFunctionalFeatures() method
retainFunctionalFeatures() method
methodwas was responsible
wasresponsible
responsible for
forfor preserving
preserving
preserving the
the thefunc-
func-
functional
Sensors 2023, 23, 1315 20 of 32
tional characteristics
tional characteristics of the malicious
malicious traffic.
traffic. ItIt contained
contained three
three cases:
cases:DoS,
DoS, probe,
probe, and
and
characteristics of the malicious traffic. It contained three cases: DoS, probe, and traffic
traffic(U2R
traffic (U2R and
and R2L).
R2L). If any of the
the functional
functional features
features were
were changed,
changed, the traffic
the trafficwas
wasnono
(U2R and R2L). If any of the functional features were changed, the traffic was no longer
considered to be malicious.
longer considered For For
to be malicious. eacheachattack
attack type,
type,a acertain
certain set offunctional
set of functionalfeatures
features
was intended to remain unchanged in the discriminator process. Figure
was intended to remain unchanged in the discriminator process. Figure 29 presents the 29 presents the
retainFunctionalFeatures() method.
Figure29.
Figure 29. retainFunctionalFeatures()
method.
In the
In thegetAdversarialSample()
getAdversarialSample()method,
method,we weappended
appendeda arandom
randomuniformly
uniformlydistributed
distrib-
uted sample taking values between 0 and 1 and having a shape (9,). This resulted in an
sample taking values between 0 and 1 and having a shape (9,). This resulted in an adversar-
adversarial entry that had the shape (40,0). Figure 30 presents the getAdversarialSam-
ial entry that had the shape (40,0). Figure 30 presents the getAdversarialSample() method.
ple() method.
Figure 30. getAdversarialSample() method.

In the getAdversarialSample() method, we appended a random uniformly distrib-
uted sample taking values between 0 and 1 and having a shape (9,). This resulted in an
adversarial
Sensors 2023, 23, 1315 entry that had the shape (40,0). Figure 30 presents the getAdversarialSam-
20 of 32
ple() method.

The training of the IDSGAN took place in epochs, with the value being taken from the
The training of the IDSGAN
configuration filetook place
described in epochs,
in Section with
2.3.1. In orderthe value being
to complete taken
a full epoch, thefrom
model
the configuration filewasdescribed in Section
trained in steps 2.3.1. Intoorder
that corresponded to complete
the number of batches itatook
fulltoepoch,
parse all the
data.
The algorithm was performed with each available batch of data.
model was trained in steps that corresponded to the number of batches it took to parse all The steps included in the
training are presented below.
data. The algorithm was performed
Required items:with each available batch of data. The steps included
in the training are presented
• below.
Normal and malicious traffic records;
Required items: • IDS model;
• Helper methods (discussed previously);
• Normal and malicious
• trafficGAN
Initialized records;
components and parameters defined;
• IDS model; • Initial state;
• Helper methods •(discussed
Discriminator trainable property set to false.
previously);
Steps:
• Initialized GAN components and parameters defined;
1. Retrieve batch of attack data corresponding to current step;
• Initial state; 2. Construct adversarial sample;
• Discriminator trainable
3. property
Generate set to
adversarial false. record for discriminator;
malicious
4. Adjust generated features respective to their data type;
Steps: 5. Retain functional features of original traffic;
1. 6. Retrieve
Retrieve batch of attack databatch of training data to
corresponding corresponding
current step;to current step;
7. Classify instances of real and adversarial data using the IDS;
2. Construct adversarial
8.
sample;
Mark discriminator as trainable;
3. Generate adversarial
9. malicious
Train record based
the discriminator for discriminator;
on the results of the IDS;
4. Adjust generated features respective to their data type;
10. Update discriminator parameters;
5. Retain functional11. Mark discriminator as not trainable;
features of original traffic;
12. Generate adversarial malicious record for generator;
6. Retrieve batch of13.training data corresponding
Train generator through the GANto current step;
model;
7. Classify instances
14.of Update
real and adversarial
generator data using the IDS;
parameters.
8. Mark discriminator as Thetrainable;
training proceeded in alternating periods. The generator was kept constant during
the discriminator training phase and, correspondingly, the discriminator was kept constant
9. Train the discriminator based on the results of the IDS;
during the generator training phase. Figure 31 presents the GAN training algorithm. The
10. Update discriminator parameters;
discriminator loss d_loss is computed as the median of the two matrices, d_loss_real and
d_loss_fake.
2.3.5. Testing and Tuning the IDS by Launching the Adversarial Attack
The testing of the GAN adversarial attack was undertaken in the GAN Adversarial
Attack.ipynb notebook. This step was dedicated to testing the attacking capabilities of
our implementation. The purpose of this attack was to make the IDS perceive generated
adversarial malicious traffic as normal traffic and, therefore, to access the system. In this
way, we could both test the performance of the IDS and also, by changing the saved model
of the IDS, tune that component.
11. Mark discriminator as not trainable;
12. Generate adversarial malicious record for generator;
13. Train generator through the GAN model;
Sensors 2023, 23, 1315 14. Update generator parameters. 21 of 32
The training proceeded in alternating periods. The generator was kept constant
during the discriminator training phase and, correspondingly, the discriminator was
keptThis step also
constant made
during the use of the training
generator configuration
phase. file to specify
Figure the scenario
31 presents the GANwe wanted
training
toalgorithm.
verify. For
Thethis, we loadedloss
discriminator thed_loss
testing is dataset
computedandas selected theofIDS
the median theaccording to the
two matrices,
configuration file.d_loss_fake.
d_loss_real and The source code that was used is presented in Figure 32.
Sensors 2023, 23, 1315 22 of 32

Figure 31. GAN training algorithm.
Figure 31. GAN training algorithm.
2.3.5. Testing and Tuning the IDS by Launching the Adversarial Attack
The testing of the GAN adversarial attack was undertaken in the GAN Adversarial
Attack.ipynb notebook. This step was dedicated to testing the attacking capabilities of
our implementation. The purpose of this attack was to make the IDS perceive generated
adversarial malicious traffic as normal traffic and, therefore, to access the system. In this
way, we could both test the performance of the IDS and also, by changing the saved
model of the IDS, tune that component.
This step also made use of the configuration file to specify the scenario we wanted to
verify.32.
Figure For this, we
Loading loaded
of the the
IDS to betesting dataset
tested and tunedand selected
by the GAN. the IDS according to the con-
Figure 32. Loading
figuration of source
file. The the IDScode
to bethat
tested and
was tuned
used by the GAN.
is presented in Figure 32.
After this, the testing dataset was loaded as presentedin
After this, the testing dataset was loaded as presented inFigure
Figure33.
33.On
Onthe
thefirst
firstline,
line, #
# represents a comment in the source code.
represents a comment in the source code.
Next, the generator models were loaded. They were created and trained as described
in Section 2.2.2. In order to estimate the GAN performance and the IDS performance in
the presence of the GAN, we verified the IDS by predicting the real attack data, and the
results are described in Section 2.2.3. Then, the same predictions were computed using the
adversarial dataset. This helped us obtain a clear view of how the IDS was affected by the
adversarial traffic.
Sensors 2023, 23, 1315 22 of 32
Figure 32. Loading of the IDS to be tested and tuned by the GAN.
The percentage
After of adversarial
this, the testing entries
dataset was perceived
loaded as normal
as presented traffic
in Figure was
33. On thecomputed
first line, in
order to calculate
# represents the effectiveness
a comment of the
in the source adversarial attack and is presented in Figure 34.
code.
Figure 33. Loading of the testing dataset.
Next, the generator models were loaded. They were created and trained as de-
scribed in Section 2.2.2. In order to estimate the GAN performance and the IDS perfor-
mance in the presence of the GAN, we verified the IDS by predicting the real attack data,
and the results are described in Section 2.2.3. Then, the same predictions were computed
using the adversarial dataset. This helped us obtain a clear view of how the IDS was af-
fected by the adversarial traffic.
The percentage of adversarial entries perceived as normal traffic was computed in
Figure 33. Loading of the testing dataset.
Loading of
order to33.calculate
Figure theeffectiveness
the testing dataset. of the adversarial attack and is presented in Figure 34.
Next, the generator models were loaded. They were created and trained as de-
scribed in Section 2.2.2. In order to estimate the GAN performance and the IDS perfor-
mance in the presence of the GAN, we verified the IDS by predicting the real attack data,
and the results are described in Section 2.2.3. Then, the same predictions were computed
using the adversarial dataset. This helped us obtain a clear view of how the IDS was af-
fected by the adversarial traffic.
The percentage of adversarial entries perceived as normal traffic was computed in
order
Figure toIDS
Figure34.
34. calculate the effectiveness
IDS prediction
prediction ofentries.
ofofadversarial
adversarial the adversarial attack and is presented in Figure 34.
entries.
Thesource
The sourcecode
codeused
usedtotocompute
computethe theresults
resultsofofthe
theadversarial
adversarialattack
attackclassification
classificationisispre-
presented in Figure
sented in Figure 35. 35.
In addition to these measurements, the detection rate and the evasion increase rate
were calculated. The original detection rate and the adversarial detection rate were the
detection rate compared to the original malicious traffic records and to the adversarial
malicious
Figure 34. traffic records,
IDS prediction respectively.
of adversarial In addition, the evasion increase rate (EIR) was
entries.
the rate of the increase in undetected adversarial malicious traffic by the IDS, measuring
the attack evasion
The source codeefficiency of thethe
used to compute GAN [13].
results Equations
of the adversarial(4) andclassification
attack (5) presentis these
pre- met-
sented
rics. in Figure
Lower 35. for the EIR indicate better performance for the IDS in the presence of
values
adversarial traffic.
Number o f attacks detected

DR = (4)
Total number o f attacks
Detection rate in the case o f adversarial attacks
EIR = 1 − (5)
Original detection rate
The computation of the DR and EIR parameters is presented in Figure 36.
Finally, we displayed the overall results of the constructed generative models using
the matplotlib.pyplot plotting library using the source code presented in Figure 37.
Sensors 2023,
Sensors 23, 23,
2023, 13151315 23 of2332of 32
Figure 35. Computing the results of the adversarial attack classification.
detection rate compared to the original malicious traffic records and to the adversarial
malicious traffic records, respectively. In addition, the evasion increase rate (EIR) was the
rate of the increase in undetected adversarial malicious traffic by the IDS, measuring the
attack evasion efficiency of the GAN [13]. Equations (4) and (5) present these metrics.
Lower values for the EIR indicate better performance for the IDS in the presence of ad-
versarial traffic.
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑡𝑡𝑎𝑐𝑘𝑠 𝑑𝑒𝑡𝑒𝑐𝑡𝑒𝑑
DR = (4)
𝑇𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑡𝑡𝑎𝑐𝑘𝑠
𝐷𝑒𝑡𝑒𝑐𝑡𝑖𝑜𝑛 𝑟𝑎𝑡𝑒 𝑖𝑛 𝑡ℎ𝑒 𝑐𝑎𝑠𝑒 𝑜𝑓 𝑎𝑑𝑣𝑒𝑟𝑠𝑎𝑟𝑖𝑎𝑙 𝑎𝑡𝑡𝑎𝑐𝑘𝑠

EIR =1- (5)
𝑂𝑟𝑖𝑔𝑖𝑛𝑎𝑙 𝑑𝑒𝑡𝑒𝑐𝑡𝑖𝑜𝑛 𝑟𝑎𝑡𝑒
Figure 35. Computing the results of the adversarial attack classification.
35. Computing
FigureThe theofresults
computation the DRof and
the adversarial attackisclassification.
EIR parameters presented in Figure 36.
Sensors 2023, 23, 1315 detection rate compared to the original malicious traffic records and to the adversarial 24 of 32
malicious traffic records, respectively. In addition, the evasion increase rate (EIR) was the
rate of the increase in undetected adversarial malicious traffic by the IDS, measuring the
attackSensors 2023,efficiency
evasion 23, 1315 of the GAN [13]. Equations (4) and (5) present these metrics.
Lower values for the EIR indicate better performance for the IDS in the presence of ad-
versarial traffic.
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑡𝑡𝑎𝑐𝑘𝑠 𝑑𝑒𝑡𝑒𝑐𝑡𝑒𝑑
Figure 36. DR and EIR computation.
DR = (4)
𝑇𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑎𝑡𝑡𝑎𝑐𝑘𝑠
Finally, we displayed the 𝐷𝑒𝑡𝑒𝑐𝑡𝑖𝑜𝑛

overall 𝑟𝑎𝑡𝑒
results
𝑖𝑛 𝑡ℎ𝑒of the
𝑐𝑎𝑠𝑒 constructed
𝑜𝑓 𝑎𝑑𝑣𝑒𝑟𝑠𝑎𝑟𝑖𝑎𝑙 generative models using
𝑎𝑡𝑡𝑎𝑐𝑘𝑠
Figure
EIR =1-
36. DR and EIR computation. (5)
the matplotlib.pyplot Figure
plotting library 36.the
using DR
𝑂𝑟𝑖𝑔𝑖𝑛𝑎𝑙 and EIR
source
𝑑𝑒𝑡𝑒𝑐𝑡𝑖𝑜𝑛 computation.
code
𝑟𝑎𝑡𝑒 presented in Figure 37.
The computation of the DR and EIR parameters is presented in Figure 36.
Finally, we displayed the overall results of the constructe
the matplotlib.pyplot plotting library using the source code pr
Figure37.
Figure 37.Plotting
Plottingof
ofgenerative
generativemodel
modelresults.
results.
3. Results
Figure
First, the performance of the IDS 37. Plotting
models with of
thegenerative
NSL-KDD model results.
dataset was assessed.
Then, the performance of the ID models was evaluated in the presence of the GAN traffic.
Figure 38 shows that the IDS models 3. performed
Results very well, all exhibiting accuracy close to
or over 0.9. However, only the ANN, randomFirst, the performance
forest, of the neighbor
and K-nearest IDS models with the NSL-
classifiers
Sensors 2023, 23, 1315 24 of 32
Figure 37. Plotting of generative model results.
3.3.Results
Results
First, the
First, the performance
performance of of the
the IDS
IDS models
models with
with the
the NSL-KDD
NSL-KDD dataset
dataset was
was assessed.
assessed.
Then, the performance of the ID models was evaluated in the
Then, the performance of the ID models was evaluated in the presence ofpresence of the GAN traffic.
Figure 38 shows
Figure 38 shows that the IDS models performed very well, all exhibiting accuracy close
models performed very well, all exhibiting accuracy close to
oror
to over
over0.9.
0.9.However,
However,onlyonlythe
theANN,
ANN, random
random forest,
forest, and K-nearest neighbor
neighbor classifiers
classifiers
wereused
were usedin inthe
thetesting
testingand
andtuning
tuningwith
withthe
theGAN
GANsections.
sections.
Figure38.
Figure 38.Performance
Performanceof
ofthe
theIDS
IDSmodels.
models.
For
Foreach
each model,
model, the
the confusion
confusion matrix
matrix was
was created
created and
and the
the precision,
precision, recall,
recall, and
and F1
F1
factor were computed.
factor were computed.
Next,
Next, the
the performance
performance of of the
the tested
tested models
models with
with the
the GAN-generated
GAN-generated traffictraffic is
is pre-
pre-
sented. We tested
sented. We tested the
the algorithm
algorithmperformance
performancewith withadversarial
adversarial traffic
traffic in in two
two situations
situations for
for each attack category and compared it with the initial performance with
each attack category and compared it with the initial performance with the original the original
NSL-KDD
NSL-KDDdataset.
dataset.
3.1. Asessing the Performance in the Case of Probe Attack Traffic
3.1. Asessing the Performance in the Case of Probe Attack Traffic
3.1.1. ANN IDS Performance in the Case of Generated Adversarial Probe Traffic
3.1.1. ANN IDS Performance in the Case of Generated Adversarial Probe Traffic
For the case of probe traffic detection, we tested and compared the performance with
the original NSL-KDD database (which contained 2157 probe records) against two configu-
rations of the ANN IDS that received generated adversarial traffic. Both the configurations
that were tested in this case for the discriminator used the ANN that was previously
implemented as the IDS with the NL-KDD dataset as input.
Table 11 presents the simulation parameters used with the ANN for detecting
probe attacks.
Table 11. Simulation parameters used with the ANN with generated adversarial probe traffic.
Parameter Value
Algorithm ANN
Epochs 32
Batch size 32
Retain functional parameters Yes
Optimizer ADAM or SGD
Table 12 presents the probe attack detection in the case of the ANN. The first column
specifies the predicted attack category. In an ideal situation, only probe traffic should be
detected. The second column specifies the results for the initial ANN IDS implementation
Sensors 2023, 23, 1315 25 of 32
that was trained using the original NSL-KDD dataset. The third and the fourth columns
specify the results obtained with the GAN setup when the discriminator implemented
with the ANN was tested with the generated adversarial traffic. The results in column
three were obtained using the ADAM optimizer, whereas the results in column four were
obtained using the stochastic gradient descent (SGD) optimizer.
Table 12. Probe attack detection with the ANN used for the discriminator.
Predicted Attack Category Initial IDS Testing ANN, ADAM ANN, SGD
Normal 265 2041 1591
DoS 84 22 258
Probe 1808 0 280
R2L 0 94 28
U2R 0 0 0
As the total number of probe records was 2157, the detection rate (DR) for the initial
IDS test with the original NSL-KDD dataset was 83.8%. For the ANN using the ADAM
discriminator, the percentage of adversarial entries classified as normal data was 94.6%.
The detection rate for the adversarial probe data was 0. The evasion increase rate was 1.0.
For the ANN that used SGD, the percentage of adversarial entries classified as normal data
was 73.75%. These results are presented in Table 13.
Table 13. Performance of the ANN.
Parameters Initial IDS Testing ANN, ADAM ANN, SGD

Detection rate (DR) 0.838 0 0.1298
Evasion increase rate (EIR) N/A 1.0 0.845
Percentage of adversarial entries
N/A 94.6 73.75
classified as normal data (%)
Tables 12 and 13 show the advantages of using adversarial training in the development
of IDS systems. Even if the IDS performance decreased significantly after careful training
of the GAN generator with the IDS, the detection performance could be improved and
offers benefits over training using information from the original dataset.
3.1.2. Random Forest IDS Performance in the Case of Generated Adversarial Probe Traffic
Table 14 presents the results obtained with the use of the random forest model for the IDS.
Table 14. Probe attack detection with the use of the random forest model for the discriminator.
Predicted Attack Category Initial IDS Testing RF, ADAM RF, SGD
Normal 285 717 1591
DoS 45 1 258
Probe 1827 1430 280
R2L 0 9 28
U2R 0 0 0
Sensors 2023, 23, 1315 26 of 32
Table 15 presents the performance of the random forest model.
Table 15. Performance of the random forest model.
Parameters Initial IDS Testing RF, ADAM RF, SGD

Detection rate (DR) 84.6 0.66 0.56
N/A 33.2 42
Tables 14 and 15 highlight the better performance of the random forest algorithm
compared to the previous ANN algorithm.
3.1.3. kNN IDS Performance in the Case of Generated Adversarial Probe Traffic
Tables 16 and 17 show the performance of the kNN algorithm in the case of adversarial
traffic. Table 17 presents the performance of the kNN algorithm.
Table 16. Probe attack detection with kNN used for the discriminator.
Predicted Attack Category Initial IDS Testing kNN, ADAM kNN, SGD
Normal 399 722 1427
DoS 156 15 10
Probe 1600 1402 713
R2L 2 18 7
U2R 0 0 0
Table 17. Performance of kNN algorithm.
Parameters Initial IDS Testing kNN, ADAM kNN, SGD

N/A 33.5 66.2
As shown in Table 17, in the case of kNN, the ADAM optimizer performed better
than SGD.
3.2. Asessing the Performance in the Case of DoS Attack Traffic

The results with DoS traffic are displayed for the ANN, random forest, and kNN IDSs.
3.2.1. ANN IDS Performance in the Case of Generated Adversarial DoS Traffic
Table 18 presents the DoS attack detection in the case of ANN as discriminator.
Table 18. DoS attack detection with the use of the ANN for the discriminator.
Predicted Attack Category Initial IDS Testing ANN, ADAM ANN, SGD
Normal 500 1690 8095
DoS 7557 884 0
Probe 38 0 0
R2L 0 5419 0
U2R 0 102 0
Sensors 2023, 23, 1315 27 of 32
Table 19 presents the performance of the ANN.
Table 19. Performance with DoS when using the ANN IDS.
Parameters Initial IDS Testing ANN, ADAM ANN, SGD

N/A 99.9 100
As shown in Table 19, with the ANN, both the ADAM and the SGD optimizers
performed poorly.
3.2.2. Random Forest IDS Performance in the Case of Generated Adversarial DoS Traffic
Table 20 presents the values measured in the case of the Random Forest discriminator
in the presence of generated adversarial DoS traffic.
Table 20. DoS attack detection with the use of the random forest algorithm for the discriminator.
Predicted Attack Category Initial IDS Testing RF, ADAM RF, SGD
Normal 558 5638 3420
DoS 7536 591 4540
Probe 1 1886 0
R2L 0 0 135
U2R 0 0 0
Table 21 presents the performance of the Random Forest algorithm.
Table 21. Performance with DoS when using the random forest IDS.
Parameters Initial IDS Testing RF, ADAM RF, SGD

N/A 66.9 42.2
As shown in Table 21, with the random forest algorithm, the SGD optimizer performed
better than the ADAM optimizer.
3.2.3. kNN IDS Performance in the Case of Generated Adversarial DoS Traffic
Table 22 presents the DoS attack detection in the case of kNN used for the discriminator.
Table 22. DoS attack detection with the use of kNN for the discriminator.
Predicted Attack Category Initial IDS Testing kNN, ADAM kNN, SGD
Normal 887 7726 4685
DoS 7177 324 2765
Probe 30 42 504
R2L 1 3 141
U2R 0 0 0
Sensors 2023, 23, 1315 28 of 32
Table 23 presents the performance of the kNN algorithm.
Table 23. Performance with DoS when using the kNN IDS.
Parameters Initial IDS Testing kNN, ADAM kNN, SGD

N/A 95.4 57.9
As shown in Table 23, with kNN, the SGD optimizer performed significantly better
than the ADAM optimizer.
4. Discussion
In this paper, we investigated the effect of adversarial data generated using a GAN on
an IDS implementing machine-learning algorithms. First, we implemented the IDS and
demonstrated its performance with each algorithm. Then, we implemented a GAN where
the discriminator used one of the previously tested algorithms and the generator used
the same algorithm and tried to maximize the loss of the discriminator. We compared the
original performance (when tested with the NSL-KDD dataset) with the performance in
the presence of the adversarial traffic. The results described in Section 3 are summarized
in Section 4.1 for the case of probe traffic detection and Section 4.2 for the case of DoS
traffic detection.
4.1. Comparison between ML Methods in the Case of Probe Traffic

In Table 24, we display the results we obtained using the ANN, random forest, and
kNN classifiers for the case of KDD traffic compared to the results obtained in the presence
of the generated adversarial probe traffic.
Table 24. IDS rates for KDD and generated adversarial probe traffic.
ML Method Used DR, KDD Traffic ADAM DR ADAM EIR SGD DR SGD EIR
ANN 0.838 0.0 1.0 0.1298 0.845
Random forest 0.846 0.66 0.21 0.56 0.32
kNN 0.741 0.64 0.123 0.33 0.55
From Table 24, it can be seen that the generative model performed very well in evasion
when the black-box IDS used was based on an artificial neural network, but the IDS was not
capable of correctly classifying the adversarial traffic. When the random forest algorithm
was used as a classifier for the IDS, the detection and evasion rates were worse than
the neural network, but the traffic still succeeded, to a small extent, in fooling the IDS.
Therefore, the random forest algorithm seemed to be a more appropriate solution for the
implementation of an IDS in the presence of a GAN-generated traffic attack. Figure 39
presents a comparison of the original (using KDD training data) detection rate and the
adversarial detection rate in the case of probe traffic.
4.2. Comparison between ML Methods in the Case of DoS Traffic

It the case of DoS traffic Table 25 presents the comparative performances of DR and
EIR for the three ML method studied (ANN, Random Forest and kNN) in the presence of
the original NSL-KDD traffic and the adversarial traffic.
Sensors 2023, 23, 1315 29 of 32
29 of 32
Table 25. IDS rates for KDD and generated adversarial DoS traffic.
ML Method Used DR, KDD Traffic ADAM DR ADAM EIR SGD DR SGD EIR
the implementation
ANN
of an IDS in the0.0presence of a0.99
0.93
GAN-generated 0.0
traffic attack.1.0Figure 39
presents a comparison
Random Forest 0.92
of the original
0.07
(using KDD0.92
training data)
0.56
detection rate
0.39
and the
adversarial
kNN detection
0.84rate in the case
0.04 of probe traffic.
0.95 0.34 0.61
Figure 39. Comparison between probe original and adversarial detection rates using different
Figure 39. Comparison IDS
between
models.
probe original and adversarial detection rates using different IDS
models.
It can be seen that the generative model performed very well in the process of evasion
when the IDS used was based on an artificial neural network but, similarly to the probe
4.2. Comparison between MLtheMethods
traffic, in capable
IDS was not the Case of DoS classifying
of correctly Traffic the adversarial traffic.
When the random forest algorithm was used as a classifier for the IDS, the detection
and evasion
It the case of DoS traffic Table rates
25were worse the
presents thancomparative
the neural network but still veryof
performances good.
DR This
andcould be
EIR for
attributable to the fact that the DoS dataset was larger than the probe one, and the model
the three ML method studied (ANN, Random Forest and kNN) in the presence of the original
succeeded in learning the pattern in the traffic records better than in the previous case.
NSL-KDD traffic and the adversarial
Figure 40 presentstraffic.
a comparison between the different IDS models in the case of DoS traffic.
It is clear that, in the case of DoS, the performance of the IDS must be improved because
the adversarial
Table 25. IDS rates for KDD detectionadversarial
and generated rate was veryDoSlow for each method. One possible approach to
traffic.
increase the detection rate in the case of adversarial traffic is to include adversarial samples
DR, KDD Traffic ADAM in theDR training dataset
ADAM or toEIR
use additional datasets.
SGD A DRthird option wouldSGDbe to implement
EIR
a honeypot [7] with feature extraction capabilities. This would collect new attack traffic
0.93 0.0
with the same functional 0.99features as in the dataset.
0.0The new records would 1.0 be beneficial
0.92 for
0.07 training the models for
0.92 U2R and R2L. 0.56 0.39
0.84 0.04 0.95 0.34 0.61
It can be seen that the generative model performed very well in the process of eva-
sion when the IDS used was based on an artificial neural network but, similarly to the
probe traffic, the IDS was not capable of correctly classifying the adversarial traffic.
When the random forest algorithm was used as a classifier for the IDS, the detection
and evasion rates were worse than the neural network but still very good. This could be
30 of 32
Sensors 2023, 23, 1315 30 of 32
Figure 40. Comparison of DoS original and adversarial detection rates using different IDS models.
Figure 40. Comparison of DoS original and adversarial detection rates using different IDS models.
5. Conclusions and Future Work
In thisWork
5. Conclusions and Future paper, we implemented an end-to-end flow for creating adversarial generated
network traffic and testing its malicious potential using the generative adversarial network
In this paper, we implemented
framework. The purpose anof end-to-end flow for
the generated network trafficcreating
was to evade adversarial gener-
machine-learning
intrusion detection systems while keeping the functional features of its respective attacking
ated network traffic and testing its malicious potential using the generative adversarial
types intact. In order to achieve this, we proceeded through various steps: acquisition of
network framework.theThe purpose
NSL-KDD dataset,of the generated
preprocessing network
of the training dataset,traffic
creationwas to evade
of multiple intrusionma-
detection systems using diverse machine-learning algorithms,
chine-learning intrusion detection systems while keeping the functional features of its and implementation of
the generative adversarial network model. Finally, by connecting all of these building
respective attackingblocks,
typesweintact. In order to achieve this, we proceeded through various
generated adversarial malicious traffic and successfully tested its attacking and
steps: acquisition of evading
the NSL-KDD
capabilities.dataset, preprocessing of the training dataset, creation
For
of multiple intrusion detection the technical
systems implementation
using diverse of thesemachine-learning
steps, we used the Google Colab Jupyter
algorithms, and
notebook service, in which we made use of data science tools, such as the numpy and pan-
implementation of the generative
das libraries, adversarial network
and machine-learning-oriented model. such
frameworks, Finally, by connecting
as TensorFlow and Keras, all
of these building blocks, weactual
to create the generated adversarial
machine-learning models formalicious traffic
the IDS and the GAN.and successfully
The programming
language used for coding all of these items was Python, which is the perfect tool for both
tested its attacking and evading capabilities.
simple and complex data manipulation.
For the technical implementation
In order to train andof thesethe
generate steps,
GAN we used
model, the Google
we devised multipleColab Jupyter
IDS machine-
notebook service, inlearning
whichmodelswe madeusing diverse
use of machine-learning
data sciencealgorithms,
tools, such such asas the
therandom
numpy forestand
algorithm and an artificial neural network, and we managed to obtain stable classifiers
pandas libraries, and thatmachine-learning-oriented
we used as a foundation for the training frameworks,
process withsuch the GANas TensorFlow
model. Out of alland
Keras, to create the the
actual machine-learning
attacking types, DoS resultedmodelsin the bestfor the IDS inand
performance termstheof GAN.
detectionThe pro-
by the
IDS, which was expected, as it had the most numerous
gramming language used for coding all of these items was Python, which is the perfect traffic entries with the exception
of the normal traffic type. In order to devise a configurable method of training the GAN
tool for both simple model,
and complex
we used a data
JSON manipulation.
configuration file with which we managed to easily tweak the
In order to train and generate
hyperparameters the GAN
of the GAN training. model, wewedevised
Furthermore, managed multiple IDS ma-
to write a functional
algorithm representing the training requirements of the GAN model architecture and
chine-learning models using diverse machine-learning algorithms, such as the random
obtain functional generative models, which were used later to generate the adversarial
forest algorithm andnetwork
an artificial
traffic. neural network, and we managed to obtain stable clas-
sifiers that we used as a foundation for the training process with the GAN model. Out of
all the attacking types, DoS resulted in the best performance in terms of detection by the
IDS, which was expected, as it had the most numerous traffic entries with the exception
of the normal traffic type. In order to devise a configurable method of training the GAN
Sensors 2023, 23, 1315 31 of 32
The generated adversarial malicious traffic was used to test the detection capabilities
of the IDS models. For this, we used three attack types: DoS, probe, and traffic (R2U
and U2R). Among these three types, the DoS and probe types were very successful in
evading the detection capabilities of the IDS, the DoS-generated instances even being
close to evading them completely, while retaining their functional features. It can be
concluded that our generative model succeeded in creating network traffic instances that
were successful in evading the intrusion detection system while retaining the functional
features of their attacking types. We also demonstrated that the discriminator and the
generator components from a GAN architecture can be used in order to improve the
performance of a machine-learning IDS by feeding the discriminator with the output from
the generator.
For three attack categories (probe, U2R, and R2L), the detection performance was very
low due to the low number of records available in the training dataset. For future work, we
intend to improve the detection performance for these classes by using the generator to
generate new records for the training dataset while retaining the functional features.
Another direction for future work will be to test the system with real traffic by ex-
tracting relevant features from live packets and feeding them to the machine-learning IDS
system. This approach will enable measurement of the classification delay, which is critical
for IDS systems.
Author Contributions: Conceptualization, D.Z.; software, A.-G.M.; writing—original draft prepara-

tion, D.Z. and A.-G.M.; writing—review and editing, D.Z., A.-G.M. and V.D.; supervision, D.Z. and
V.D. All authors have read and agreed to the published version of the manuscript.
Funding: This research received no external funding. The APC was funded by the Technical Univer-
sity of Cluj-Napoca.
Institutional Review Board Statement: Not applicable.
Informed Consent Statement: Not applicable.
Data Availability Statement: The data that support the findings of this study are available from the
corresponding author (D.Z.) upon reasonable request.
Conflicts of Interest: The authors declare no conflict of interest.
References
1. Bace, R.; Mell, P. Intrusion Detection Systems; NIST Special Publication on Intrusion Detection Systems; NIST: Gaithersburg, MD,
USA, 2001.
2. Mbona, I.; Eloff, J.H.P. Detecting Zero-Day Intrusion Attacks Using Semi-Supervised Machine Learning Approaches. IEEE Access
2022, 10, 69822–69838. [CrossRef]
3. Buczak, A.L.; Guven, E. A Survey of Data Mining and Machine Learning Methods for Cyber Security Intrusion Detection. IEEE
Commun. Surv. Tutor. 2016, 18, 1153–1176. [CrossRef]
4. Mishra, P.; Varadharajan, V.; Tupakula, U.; Pilli, E.S. A Detailed Investigation and Analysis of Using Machine Learning Techniques
for Intrusion Detection. IEEE Commun. Surv. Tutor. 2019, 21, 686–728. [CrossRef]
5. Aldweesh, A.; Derhab, A.; Emam, A.Z. Deep learning approaches for anomaly-based intrusion detection systems: A survey,
taxonomy, and open issues. Knowl.-Based Syst. 2020, 189, 105124. [CrossRef]
6. Dhanabal, L.; Shantharajah, S.P. A Study on NSL-KDD Dataset for Intrusion Detection system Based on Classification Algorithms.
Int. J. Adv. Res. Comput. Commun. Eng. 2015, 4, 446–452.
7. Olouhal, O.U.; Yange, T.S.; Okerekel, G.E.; Bakpol, F.S. Cutting Edge Trends in Deception Based Intrusion Detection Systems-A
Survey. J. Inf. Secur. 2021, 12, 250–269. [CrossRef]
8. Asharf, J.; Moustafa, N.; Khurshid, H.; Debie, E.; Haider, W.; Wahab, A. A Review of Intrusion Detection Systems Using Machine
and Deep Learning in Internet of Things: Challenges, Solutions and Future Directions. Electronics 2020, 9, 1177. [CrossRef]
9. Shitharth, S.; Kshirsagar, P.R.; Balachandran, P.K.; Alyoubi, K.H.; Khadidos, A.O. An Innovative Perceptual Pigeon Galvanized
Optimization (PPGO) Based Likelihood Naïve Bayes (LNB) Classification Approach for Network Intrusion Detection System.
IEEE Access 2022, 10, 46424–46441. [CrossRef]
10. Prashanth, S.K.; Shitharth, S.; Praveen Kumar, B.; Subedha, V.; Sangeetha, K. Optimal Feature Selection Based on Evolutionary
Algorithm for Intrusion Detection. SN Comput. Sci. 2022, 3, 439. [CrossRef]
11. Sheikh, M.S.; Peng, Y. Procedures, Criteria, and Machine Learning Techniques for Network Traffic Classification: A Survey. IEEE
Access 2022, 10, 61135–61158. [CrossRef]
Sensors 2023, 23, 1315 32 of 32
12. NSL-KDD Dataset. Available online: https://www.unb.ca/cic/datasets/nsl.html (accessed on 27 December 2022).

13. Lin, Z.; Shi, Y.; Xue, Z. IDSGAN: Generative adversarial networks for attack generation against intrusion detection. arXiv 2021,
arXiv:1809.02077.
14. Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial
networks. Commun. ACM 2020, 63, 11, 139–144. [CrossRef]
15. Ring, M.; Schlör, D.; Landes, D.; Hotho, A. Flow-based network traffic generation using Generative Adversarial Networks.
Comput. Secur. 2019, 82, 156–172. [CrossRef]
16. Abdelmoumin, G.; Whitaker, J.; Rawat, D.B.; Rahman, A. A Survey on Data-Driven Learning for Intelligent Network Intrusion
Detection Systems. Electronics 2022, 11, 213. [CrossRef]
17. Zhu, Y.; Cui, L.; Ding, Z.; Li, L.; Liu, Y.; Hao, Z. Black box attack and network intrusion detection using machine learning for
malicious traffic. Comput. Secur. 2022, 123, 102922. [CrossRef]
18. Balyan, A.K.; Ahuja, S.; Lilhore, U.K.; Sharma, S.K.; Manoharan, P.; Algarni, A.D.; Elmannai, H.; Raahemifar, K. A Hybrid
Intrusion Detection Model Using EGA-PSO and Improved Random Forest Method. Sensors 2022, 22, 5986. [CrossRef] [PubMed]
19. Shahriar, M.D.; Haque, N.I.; Rahman, M.A.; Alonso, M. G-IDS: Generative Adversarial Networks Assisted Intrusion Detection
System. In Proceedings of the 2020 IEEE 44th Annual Computers, Software and Applications Conference (COMPSAC), Madrid,
Spain, 13–17 July 2020. [CrossRef]
20. Vaccari, I.; Carlevaro, A.; Narteni, S.; Cambiaso, E.; Mongelli, M. eXplainable and Reliable Against Adversarial Machine Learning
in Data Analytics. IEEE Access 2022, 10, 83949–83970. [CrossRef]
21. Fasci, L.S.; Fisichella, G.L.; Qian, C. Disarming visualization-based approaches in malware detection systems. Comput. Secur.
2023, 126, 103062. [CrossRef]
22. TensorFlow. 2022. Available online: https://www.tensorflow.org (accessed on 27 December 2022).
23. Keras. 2022. Available online: https://keras.io/about/ (accessed on 27 December 2022).
24. Shitharth, S.; Prince Winston, D. An enhanced optimization based algorithm for intrusion detection in SCADA network. Comput.
Secur. 2017, 70, 16–26. [CrossRef]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual
author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to
people or property resulting from any ideas, methods, instructions or products referred to in the content.

Sensors 23 01315 v2

Uploaded by

Copyright:

Available Formats

Sensors 23 01315 v2

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Sensors 23 01315 v2

Uploaded by

Copyright:

Available Formats

sensors

Communications Department, Technical University of Cluj-Napoca, 400114 Cluj-Napoca, Romania

Sensors 2023, 23, 1315. https://doi.org/10.3390/s23031315 https://www.mdpi.com/journal/sensors

1.2. Dataset Used

1.3. Machine-Learning Algorithms Implemented

2. Materials and Methods

Figure 2. Screen capture of the Google Colab interface.

Figure 3. Proposed flow for the machine-learning IDS.

2.2.1. Data Preprocessing

Figure 4. Mounting the drive to access the dataset.

Figure 5. Heat map of the correlated features.

Figure 11. Source

Figure 13. createModel method for the ANN IDS.

2.2.3. Performance Analysis of the Machine-Learning IDS

2.2.3. Performance Analysis of the Machine-Learning IDS

Table 1. Performance analysis of the kNN IDS.

Predicted Attack Category Precision Recall F1 Score

Table 2. Confusion matrix for the kNN IDS.

Predicted Attack Category Normal DoS Probe R2L U2R

Table 3. Performance analysis of the decision tree IDS.

Predicted Attack Category Precision Recall F1 Score

Table 4. Confusion matrix for the decision tree IDS.

Predicted Attack Category Normal DoS Probe R2L U2R

Table 5. Performance analysis of the random forest IDS.

Predicted Attack Category Precision Recall F1 Score

Table 6. Confusion matrix for the random forest IDS.

Predicted Attack Category Normal DoS Probe R2L U2R

Table 7. Performance analysis of the SVM IDS.

Predicted Attack Category Precision Recall F1 Score

Table 8. Confusion matrix for the SVM IDS.

Predicted Attack Category Normal DoS Probe R2L U2R

Table 9. Performance analysis of the ANN IDS.

Predicted Attack Category Precision Recall F1 Score

Table 10. Confusion matrix for the ANN IDS.

Predicted Attack Category Normal DoS Probe R2L U2R

2.3. Development of a GAN for Testing and Tuning of the IDS

2.3.1. The Configuration File

Sensors 2023, 23, 1315 15 of 32

Figure 16. Loading of the dataset used for the GAN.

Figure 17. Loading of the models used for the GAN.

Figure 17. Loading of the models used for the GAN.

Sensors 2023, 23, 1315 16 of 32

Figure 19. Training of the GAN according to the configuration file.

The model.summary() method offered a graphical representation of the model coupled

The model.summary() method offered a graphical representation of the model cou-

2.3.3. The Discriminator Component of the GAN

Figure 23. The discriminator model

2.3.4. The Training Algorithm

2.3.4. The Training Algorithm for the GAN

Figure 24. The intermediary training method.

Sensors 2023, 23, 1315 19 of 32

Figure 25. getBatch()

Figure 26. adjustBinaryFeatures () method.

Figure 30. getAdversarialSample() method.

Figure 30. getAdversarialSample() method.

Sensors 2023, 23, 1315 22 of 32

Figure 33. Loading of the testing dataset.

Number o f attacks detected

Figure 35. Computing the results of the adversarial attack classification.