IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 15, NO.

2, FEBRUARY 2019 901

Fragmentation-Based Distributed Control

System for Software-Defined Wireless
Sensor Networks
Hlabishi Isaac Kobo , Adnan M. Abu-Mahfouz , Senior Member, IEEE,
and Gerhard Petrus Hancke , Senior Member, IEEE

Abstract—Software-defined wireless sensor networks to industrial Internet of things (IIoT) [1]. The IIoT is envisioned
(WSNs) are a new and emerging network paradigm that to inspire great economic growth and rapid production in various
seeks to address the impending issues in WSNs. It is industrial production systems. This digital-oriented industrial-
formed by applying software-defined networking to WSNs
whose basic tenet is the centralization of control intelli- ization is termed as “the fourth industrial revolution or 4IR.”
gence of the network. The centralization of the controller The 4IR is described as the fourth disruptive and major in-
rouses many challenges such as security, reliability, dustrialization trend, an epoch marked with rapid growth and
scalability, and performance. A distributed control system development as a result of automation and data technologies in
is proposed in this paper to address issues arising from
various disciplines. Some of the major technologies behind this
and pertaining to the centralized controller. Fragmentation
is proposed as a method of distribution, which entails a trend include but no limited to IoT, IIoT, artificial intelligence,
two-level control structure consisting of local controllers virtual realities, cyber-physical systems, cloud, and cognitive
closer to the infrastructure elements and a global controller, computing. Most of the devices and elements, which will par-
which has a global view of the entire network. A distributed take in these technologies especially IIoT, will be equipped
controller system brings several advantages and the
with sensors and actuators: some wireless and some wired. The
experiments carried out show that it performs better than a
central controller. Furthermore, the results also show that networking of these sensor nodes augments the scope and pur-
fragmentation improves the performance and thus have a pose of wireless sensor networks (WSNs), whose involvement
potential to have major impact in the Internet of things. has always been confined to monitoring [2] in the main. These
Index Terms—Industrial Internet of things (IIoT), indus- sensor nodes are small, inexpensive, and intelligent due to the
trial wireless networks, software-defined wireless sensor advancement of microelectrical mechanical systems. The major
networks (SDWSNs). challenges facing industrial systems include the management of
the various systems with different proprietary protocols as well
I. INTRODUCTION as their sensitivity to time delay, failure, and security.
HE Internet of things (IoT) is at the center of the future The software-defined networking (SDN) is a new emerging
T Internet. The IoT framework is an interconnection of many
devices, systems, and applications to the Internet. Accordingly
networking and computing paradigm earmarked as a potential
resolve of most of the above-mentioned challenges. SDN advo-
to Cisco, an estimated 50 billion devices will be connected to the cates for a common standardized protocol to avoid the challenge
Internet by the year 2020. The IoT paradigm is envisaged to per- of vendor locking [3]. The SDN model separates the control and
meate into the industrial manufacturing and production, leading data forwarding on the networking elements; thus, it removes
the control logic from the network devices and centralizes it on
a controller [4]. The adoption of SDN has gained traction in
Manuscript received February 21, 2018; accepted March 26, 2018.
Date of publication April 2, 2018; date of current version February 1, both the industry and the academia. Most of the 4IR systems
2019. Paper no. TII-18-0485. (Corresponding author: Hlabishi Isaac are already applying SDN including IIoT and WSN such as in
Kobo.) [5]–[7].
H. I. Kobo is Department of Electrical, Electronic and Computer En-
gineering, University of Pretoria, Pretoria 0084, South Africa (email: Software-defined wireless sensor networks (SDWSNs) are an
[email protected]). emerging model formed by applying the SDN model in WSNs.
A. M. Abu-Mahfouz is with the Department of Electrical, Electronic The emergence of SDWSN as a pivot, in the stead of WSNs,
and Computer Engineering, University of Pretoria, Pretoria 0084, South
Africa, and also with the Meraka Institute, Council for Scientific and for the highly anticipated and imminent IoT and IIoT paradigms
Industrial Research, Pretoria 0001, South Africa (e-mail: aabumahfouz@ has ignited much interest and research focus. WSNs are envis-
csir.co.za). aged to play a vital role in IoT as a major building block [8].
G. P. Hancke is with the Department of Computer Science, City Uni-
versity of Hong Kong, Hong Kong, and also with the Department of However, WSNs have always been riddled with challenges ema-
Electrical, Electronic and Computer Engineering, University of Pretoria, nating from their inherent nature of resource constraints thereby
Pretoria 0084, South Africa (e-mail: [email protected]). hindering their progress, efficiency, and applicability [9].
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org. The application of SDN in WSNs is also receiving much
Digital Object Identifier 10.1109/TII.2018.2821129 attention especially on the basis of its imminent role in IoT
1551-3203 © 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications standards/publications/rights/index.html for more information.
902 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 15, NO. 2, FEBRUARY 2019

[10]. SDWSN is regarded as a potential to overcome some of simple, evolutionary, etc. SDN decouples the control plane from
the challenges besetting WSNs while meeting the demands of the data forwarding plane. The decoupling leaves network ele-
IoT. Most of the energy intensive functions are moved from the ments or devices as dump devices, which only forwards packets.
sensor node to the controller. With SDWSN, the sensor nodes The control intelligence is centralized into a controller. SDN
would be sheer devices with only the forwarding capabilities, comprises of three layers or planes in data/infrastructure plane,
whereas the control intelligence is centralized. control plane, and an application plane. The data plane is made
A distributed control system in SDWSN seeks to address the of the sheer infrastructure devices, which only understand in-
challenges of a centralized controller in order to achieve relia- structions from the controller. The control plane hosts the con-
bility, scalability, and efficient performance. There are different troller. The application plane hosts various applications that
ways and forms of distribution mainly determined by the na- offer functionality and services to the network. The applica-
ture of the network or the data concerned. This paper presents tion and adoption of SDN to various computing and networking
an efficient distribution technique suitable for SDWSN con- platforms is rapidly growing [15]. The fusion of SDN and WSN
trol system. It takes into account all considerations pertaining begets SDWSN, which has a potential to resolve some of the
to SDWSN challenges such as the inherent ills of low band- WSN’s inherent challenges. The SDWSN is envisioned to play
width, energy, memory, and processing, as well as the amount a significant role in the IoT revolution.
of data exchange or update expected especially from the IoT
perspective. The proposed method uses the concept of fragmen- B. Controller
tation. Thus, each cluster segment of the network has its own
A controller plays a very vital role in SDWSN. It is the
controller, which is lean and very close to the infrastructure ele-
central point of control where decisions are made. This SDN
ments. There also exists a global controller, which has a view of
abstraction brings lots of benefits especially in management
the whole network. This two-level control architecture allows a
and configuration [16]. It also allows new policies and other
faster response between the sensor nodes and the control.
changes to be implemented with ease. However, this model is
Central to distributed systems is the consistency models,
not immune to drawbacks. The critical shortcomings emanate
which determine the data convergence on the distributed par-
from its centralization of the control intelligence, thereby
ticipants (nodes). There are two major consistency models, i.e.,
invoking issues such as reliability, security, performance, and
eventual consistency and strong consistency. The former uses
scalability [17]. A central controller implies a central point of
gossip protocols such as antientropy and rumor mongering,
failure and also a potential target for adversaries [4], [18]–[20].
whereas the latter uses consensus algorithms such as RAFT
Also, as the whole network relies on it for functionality, i.e.,
[11] and/or Paxos [12]. The choice of the consistency model
flow rule setup, potential performance degradation can be
depends on the need of the application and the type of data. This
encountered as the network grows. Furthermore, SDWSN is
paper investigates the applicability of these consistency mod-
envisioned to handle lots of data especially with the advent of
els and their algorithms in the distributed control systems for
IoT. Unlike enterprise networks, SDWSN will deal with lot of
SDWSNs. The contributions of this paper are as follows:
sensory data, which are delay sensitive. Therefore, a central
1) to investigate the feasibility of distributed controllers for
controller for SDWSN will not be practical [15]; thus, there is
SDWSN;
need for a distributed control system. Distributed controllers
2) to propose fragmentation as a method of distribution to
do exist for mainstream SDN enterprise networks but not so in
achieve efficient SDWSN control;
the SDWSN space. Some of the popular distributed controllers
3) to propose alternative algorithms to best effort and antien-
in the SDN community are OpenDaylight [21], ONOS [22],
tropy algorithms to achieve a suitable consistency data
Hyperflow [23], Kandoo [24], and Elasticon [25].
model for fragmentation; and
4) to show through the evaluation that fragmentation does
bring efficiency to SDWSN control. C. Related Work
The rest of this paper is organized as follows. We highlight SDN-WISE [26] is a comprehensive SDWSN framework
a brief overview of SDWSN in Section II, which includes con- based on state automata. The SDN-WISE software stack gives
troller and related work. Section III highlights the distributed detailed description of the SDN sensor nodes, sink nodes, and
control system for SDWSN with the proposed model. Gossip or the emulated nodes. The stateful structure includes the flow
epidemic protocols are discussed in Section IV. Sections V and table composition in the follow of OpenFlow. It also details
VI deal with the experimental setup and results, respectively, the protocol architecture dealing with the packet handling and
whereas Section VII is a discussion. Section VIII concludes the processing, and topology discovery techniques. The stack is
paper. adaptable to various SDN controllers and as such an adapta-
tion with ONOS was implemented by the authors and tested for
II. SOFTWARE-DEFINED WSNS interoperability with the enterprise network [27].
The combination of SDN-WISE and ONOS is progressively
A. Software-Defined Networking and Wireless
in the right direction toward the realization of an effective and
Sensor Network
efficient SDWSN for IoT. In [28], this solution was tested us-
SDN is an emergent computing and networking model, which ing multiple controllers in a distributed fashion. The authors of
has brought convenient disruption in both the academia and the SDN-WISE also implemented a custom java controller to test
industry at large [4], [13], [14]. It is regarded as innovative, this framework, which used Dijkstra’s algorithm. Dijkstra’s al-
KOBO et al.: FRAGMENTATION-BASED DISTRIBUTED CONTROL SYSTEM FOR SOFTWARE-DEFINED WIRELESS 903

gorithm finds the shortest paths between nodes in a connected

graph. In [29], collection tree protocol is used to find a con-
troller in a multiple controller solution called TinySDN. The
controller is attached to the sink node through a serial connec-
tion; a hierarchical extension of this solution was implemented
in [30].

III. DISTRIBUTED SDWSN CONTROL SYSTEM

A. Background
Distributed systems found more prevalence in database the-
ory. E. Brewer profoundly stated at the 2000 Symposium of
Distributed Computing that “in any highly distributed data sys-
tem there are three common desirable properties: consistency,
availability, and partition tolerance. However, it is impossible
for a system to provide all three properties at the same time.”
[31], [32]. This became known as the consistency, atomicity,
Fig. 1. Distributed control system with fragmentation.
partition tolerance (CAP) theorem. Coronel and Morris [32] de-
fine these three properties as follows: consistency—all nodes
should see the same data at the same time, which means that the ensures that each local controller node independently controls
replicas should be immediately updated; availability—a request its segment (cluster) of the network and work concurrently with
is always fulfilled by the system; and partition tolerance—the the other local controller nodes. These nodes would operate si-
system continues to operate even in the event of a node failure. multaneously, thus guaranteeing parallelism. This would allow
In the context of this paper, consistency would mean all frag- a faster response between the sensor nodes and the controller.
ments (local controllers) have the same network state all the Most of the data would be upstream and only infrequently would
times, i.e., symmetric; availability—all nodes available; and par- the controller send control instruction to the devices.
tition tolerance—network continues to function after a node fail-
ure. Another phenomenon common in database systems is ACID
(atomicity, consistency, isolation, and durability), regarded as B. Fragmentation
the four properties of transactional database. ACID ensures that We propose fragmentation for our distributed control solu-
all transactions result in a consistent database state. Coronel and tion. Fragmentation entails dedicating part of the control sys-
Morris note that this is well suited for a centralized and small tem to different segments (fragments) of the sensor network. In
distributed database system [32]. Otherwise, latency becomes addition to fragmenting the control of the network, this could
an issue as the system scales. They further state that it is for this be extended to abstracting different sensor traits together, e.g.,
reason that many systems sacrifice consistency and isolation for temperature nodes. Fig. 1 depicts the distributed control system
availability, leading to another phenomenon BASE (basically with fragmentation. This architecture is akin to Kandoo [24],
available, soft state, eventually consistent). In BASE, data ex- which also has local controllers as well as a global controller
changes are not immediate but propagate slowly until all nodes overseeing the whole network. However, Kandoo does not im-
are eventually consistent [31]. plement fragmentation as defined in this paper. Also like other
A distributed SDN control system draws reference from the distributed SDN controllers such as ODL, Hyperflow, etc., they
above, particularly the database systems. However, the funda- are dedicated to traditional SDN networks and are not applicable
mental roles of a SDN controller and a database system are to SDWSN.
distinguishable. The primacy of a database is to store data and Fragmentation takes distribution further by taking the control
enable the CRUD (create, read, update, and delete) operation. logic closer to the infrastructure devices. The reduced distance
The SDN controller on the other hand is an engine of the net- between the controller and the sink node saves much trans-
work, more pointedly to control the infrastructure devices by mission power and also reduces pressure from its low data rates
defining data propagation rules. Furthermore, SDWSN brings capacity thereby improving throughput. Fragmentation involves
another dimension, different from the traditional SDN by putting a two-level control structure where there is a global or central
the sensor nodes at the periphery of the network. This also tilts controller overseeing the whole network and a local controller
the paradigm of the controller’s role. Our system follows the in charge of only a portion or segment of the network. The local
BASE as a consistency model thereby preferring availability controllers only have the knowledge of the portion of the net-
over consistency. work they are controlling. The characteristics of each controller
This paper undertakes the eventually consistency model and as well as the sink node are as follows.
customizes it for SDWSN control. This process takes cog- 1) Global Controller:
nizance of the perpetual SDWSN challenges such as limited en- 1) Global view/knowledge of the network.
ergy, memory, bandwidth, etc. This method applies concurrency 2) Failure mechanism.
and parallelism on the local controller nodes. This effectively 3) Load balancing.
904 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 15, NO. 2, FEBRUARY 2019

TABLE I
CONSTRUCTOR

Fig. 2 depicts the high-level design of the research mecha-

nism technique; each controller, herein referred as a fragment
controlling a cluster of nodes. This diagram shows all concepts
considered in this paper by order of application.

IV. EPIDEMIC/GOSSIP PROTOCOLS

Fig. 2. Complete research structure.
An epidemic is the spread of disease (infectious) to a large
number of people in a population within a short period of time
[33], whereas gossip is the spread of rumors in an informal
4) SDN functionalities of a controller. way in the social circles. These two concepts, with the common
5) Failure mechanism. denominator of “spread,” inform the basis of gossip/epidemic
a) Failure of the global controller does not affect the protocols. Gossip protocols disseminate information across a
operation of the network, only a temporal disruption distributed system using a gossiplike method. A participant ran-
of display and other functions that requires global domly pairs with a peer and exchange update information be-
knowledge. tween them and after a period of time full consistency is reached.
b) Replication method used to create redundancy. The evolution gossip protocols was initially introduced by
2) Local Controller: Demer’s 1987 paper titled “Epidemic algorithms for replicated
1) Take charge/control of a cluster. database maintenance” [34]. There are different gossip protocols
2) Updates the global controller. in best effort also called direct mail, antientropy, and Rumor
3) Local cluster knowledge. mongering. This paper focuses on best effort and antientropy.
4) Lightweight for cost effectiveness. Best effort ensures that every new event or update is sent to
5) Failure mechanism. all other nodes immediately. Antientropy compares the repli-
a) Another local controller takes over. cas/nodes and reconciles the differences, thus updating each
b) Learns cluster state from global controller. copy to the newest [35]. Rumor mongering floods the network
c) Sink connects to the closest controller. with updates for a period of time sufficient enough to have all
3) Sink Node: nodes updated. These method suites networks with a moderate
1)
Connects to the local controller (the closest). latency tolerance, however, would potentially lead to latency
2)
Communicates with the sensor nodes. challenges under high update load [35]. The SDWSN is en-
3)
Relays/convey information to the local controller. visaged to be a high update network with little to none latency
4)
Uses RF to communicate with the other sensor nodes and tolerance and given the low capacity bandwidth; these two meth-
Internet to connect to the local controller. ods could be problematic.
5) Failure mechanism. The best effort or direct mail is event based. An update to
a) Sensor nodes find another closest sink node. other nodes is triggered by an occurrence of an event. Upon
There are three fundamental things that distribution seeks receiving an event, the node will broadcast that event to all
to achieve: reliability, scalability, and efficiency. However, a other nodes in the cluster. An event could be any occurrence of
special consideration has to be accorded in SDWSN due to the an update such as a new node, node failure, etc. The receiving
inherent low capacity of WSNs. nodes in the cluster evaluate the recency of the event. If recent,
Reasons for fragmentation the matching entry will be updated accordingly. Otherwise, the
1) Dedicated controller to a cluster. sender will have to update its state. The following tables show
2) Avoid having global knowledge on all controllers, reduce the pseudocodes of the algorithms and their time efficiency.
overhead cost by updates. Table I shows the constructor, which forms the base of all the
3) Improve responsiveness. algorithms discussed. Table II shows the pseudocode of the best
4) Allow proper isolation of sensed data types. effort algorithm.
5) Reduce redundancy on the local nodes. The antientropy algorithm is almost similar to best effort in
6) Updates to the global controller cannot affect the opera- content but different. Antientropy is periodic; thus, the synchro-
tion of the network, i.e., delay and congestion. nization occurs after every set period of time. A peer periodically
7) Low latency—controller close to the sensor nodes. chooses a random partner from the list of peers (nodes) and starts
KOBO et al.: FRAGMENTATION-BASED DISTRIBUTED CONTROL SYSTEM FOR SOFTWARE-DEFINED WIRELESS 905

TABLE II TABLE IV
BEST EFFORT ALGORITHM [34] BEST EFFORT ALGORITHM WITH FRAGMENTATION

TABLE III
ANTIENTROPY ALGORITHM [34], [35]

Fig. 3. Flowchart of the best effort algorithm with fragmentation.

to exchange state information. Thus, peer p sends its state to q,

and q applies it to its own state; this is called push method. The best effort algorithm is redesigned to ensure that it only
Otherwise in pull method, p sends its state to q, which only con- sends updates to the global controller; thus, all updates trig-
sists of keys and timestamps, then q responds with appropriate gered by events are sent to the global controller. Upon receiving
matching updates to p. Pull–push method is the combination of an event, the node sends it to the global controller. This step
both methods: while q sends updates to p as in the pull method, reduces the computation of the algorithm from O(n) to O(1).
it also sends its outdated values to p. This is the most used and The global controller then checks if the event is recent before
most efficient method. Thus, using the pull–push method, a node updating its state. If the global controller does not have that
sends its state to a peer node. The peer node checks the received entry, it is created; however, if an entry exists in the global con-
state for recency and if recent, it updates its state accordingly. troller and not in the local controller, the local controller ignores
Otherwise, it sends a message to the sender node with a set of the entry. This is to ensure that each local controller has a view
updates. Upon receiving the reply, the original sender (now the and control of its cluster of the network. The best effort algo-
receiver) applies the changes, first by checking the recency, then rithm is modeled as described by the pseudocode in Table IV
updating if recent. If an entry is missing, it is requested. The for the fragmentation model. The steps described in Table IV
pseudocode of the algorithm is described in Table III. are further depicted by the flowchart in Fig. 3.
The ONOS architecture is based on the eventual consistency The antientropy algorithm is also redesigned from updating
data model; however, applications that require stronger data its peers to only update the main global controller. The antien-
guarantees can use the strong consistency model as an alter- tropy handling method is enhanced to be able to distinctively
native. The strong consistency model is backed by RAFT [11] handle updates from either the global controller or the local
algorithm. The eventual consistency uses best effort to update controller. The local controller does the main updates to the
all other peers when an event occurs and antientropy to resolve global controller. If the local controller has new devices, the
the differences among nodes. The first intervention toward frag- global controller will be updated during the data exchange. How-
mentation is to change the behavior of the gossip algorithms. ever, the local controller cannot accept any new devices from
906 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 15, NO. 2, FEBRUARY 2019

TABLE V
ANTIENTROPY ALGORITHM WITH FRAGMENTATION

Fig. 5. Experiment A, single central controller.

the advanced processing capabilities of the modern systems, it is

still vital to the SDWSN. The time complexities of the above are
listed alongside the algorithms. The best effort algorithm, which
sends updates to all peers upon a new event runs at O(n); after
the change, which sends all updates to the global controller,
the time complexity reduces to O(1). On the other hand, the
antientropy algorithm changes from O(nlogn) to O(n).

V. EXPERIMENTAL SETUP
The experimental evaluation seeks to determine the viability
of the fragmentation model for the SDWSN. Hypothetically,
the two algorithms used in the formation of the fragmentation
model have an efficient time complexity as shown through the
Fig. 4. Flowchart of the antientropy algorithm with fragmentation. big O notation. The big O notation is a model used to measure
the performance or complexity of an algorithm. This section
the global controller (might be from other clusters). The local describes the experiments used for this evaluation.
controller is envisaged to get updates only from the main global The experimental setup used SDN-WISE solution guidelines.
controller if it is coming alive due to having been down or when Thus, a Cooja simulator to establish SDN enabled sensor nodes
taking over another down node. In case a node goes down, the making the SDWSN as well as the SDN-WISE ONOS con-
sink node will connect to the next available controller node. To troller. SDN-WISE 1, which uses ONOS 1.0.2, is used. ONOS
get the updated state, the new controller will do an antientropy provides a mastership service where there is a master and slave
synchronization with the global controller; however, this would controllers. In a case of failure, one slave is elected to the master.
be rare given the nature of updates in SDWSN. The sensory data In this context, each local controller becomes a master, whereas
are mostly upstream and therefore most updates will be from the others in the cluster become slaves. The algorithms were
the local controller to the global controller. implemented in the ONOS controller. Three tests experiments
The antientropy protocol between the local controller and the were conducted comparatively: a single central controller, a
global controller is depicted in Table V. The steps of the algo- distributed controller, and a distributed controller with fragmen-
rithm are depicted on the flowchart in Fig. 4. As stated above, tation. The first two experiments were previously explored in
there are similarities between these two algorithms. The main [28], where a determination was made that a distributed control
difference is on the initiation stages; best effort algorithm is trig- system for SDWSN is indeed possible and a necessity.
gered by an event, whereas the antientropy algorithm is periodic. All the experiments were run in independent virtual machines
The similarities are on the information exchange between the (VM). All SDWSN simulations were conducted from a VM with
nodes. As shown in Figs. 3 and 4, the contents of the algorithms 2-GHz CPU and 2 GB RAM; the same VM is also used for the
are similar. global controller. All other controller variations are conducted
from different VMs with 2-GHz CPU and 1 GB RAM specifi-
cations. Experiment A, as depicted in Fig. 5, uses a single cen-
A. Time Complexity tral controller. Experiment B uses three distributed controller
Time complexity is the amount of time or steps an algorithm in a form of a cluster ran in three different VMs, as shown
takes to run as a function given the length of the input. The more in Fig. 6. This experiment used the symmetric ONOS, herein
complex an algorithm is, the longer it takes to run. Although referred as ONOS original where all the controller instances
many scholars consider it insignificant or negligible because of eventually converge to an equal state. In experiment C, as de-
KOBO et al.: FRAGMENTATION-BASED DISTRIBUTED CONTROL SYSTEM FOR SOFTWARE-DEFINED WIRELESS 907

Fig. 8. Average RTT on 39 nodes.

Fig. 6. Experiment B, distributed controllers.

Fig. 9. RTT versus number of nodes.

distributed controllers exhibit a better setup time. The fragmen-

tation model took longer than the other distributed experiment;
however, the difference is not huge.
Fig. 7. Experiment C, distribution with fragmentation. The central controller exhibited a higher RTT than the dis-
tributed versions. Fig. 8 depicts the RTT of scenario 39 (3 sink
picted in Fig. 7, three distributed controllers are used for the and 39 sensor nodes); other scenarios are not shown but they are
fragmentation as local controllers, each independently control- relatively similar. Fig. 9 shows the average RTT for all scenar-
ling their own clusters as well as a global controller. ios. The distributed versions differ slightly, with the fragmenta-
The experiments were evaluated through three main eval- tion model slightly lower than the original version, as shown in
uation metrics in round-trip time (RTT) or round-trip delay, Figs. 8 and 9. The 30 nodes scenario exhibited a slightly higher
standard deviation, and packet error rate. The RTT measures the RTT on average on all the scenarios. The results also show that
time it takes for a packet to traverse from the simulation to the the addition of nodes has a slight impact on the average RTT. In
controller and back. The standard deviation measures the varia- the first experiment, scenario 15 has the very least on the min-
tion of the time (RTT) or the delay across all packets recorded imum RTTs compared to the rest. This is because the first few
over the evaluation period of 300 s. This assists in determin- packets enjoy the free reign before the other sinks join in and
ing the relational proportion of consistency or lack thereof. The can be largely attributed to the manual method used to connect
packet error rate looks at the rate of packet loss. to the controller. In contrast, the very same scenario (central)
exhibits the highest maximum RTTs across all experiments and
scenarios. The maximum RTTs are also in proportion of the
VI. RESULTS
increase in nodes. The fragmentation model exhibits the lowest
This section provides the detailed results obtained from the minimum and maximum RTTs on the distributed versions.
described experiments. All the results are listed in Table VI. The contrast between the minimum RTTs and the highest
Each experiment was run using 3 sink nodes plus 15, 30, 39 RTTs on the central controller experiment has a major impact
sensor nodes, respectively; herein referred as scenario 15, sce- on the standard deviation. Thus, the variation of time delay
nario 30, and scenario 39. is inconsistent with high levels of fluctuations. The distributed
The RTT of the SYNC packet indicates the time it takes by experiments show a lower variation with the fragmentation ex-
the controller to process the first packet. This also highlights hibiting the lowest, as depicted in Fig. 10.
the controller setup time. The first sink nodes to establish a The packet error rate is very minimal however not negligible.
connection to the controller always have the lowest connection This can be attributed to the SDN-WISE inherent architectural
time. The results show that the average setup time is high on maturity, as described in [26]. The packet error rate is relative
the central controller in all scenarios. They also show that the across all experiments. It is also high when the numbers of nodes
908 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 15, NO. 2, FEBRUARY 2019

TABLE VI
RESULTS

Fig. 10. Standard deviation. Fig. 12. Number of packets produced during the test.

fragmentation model suffers some additional delay compared

to the original version because of the two-level architecture.
However, this only affects the first packet of handshaking. The
subsequent packets are handled and processed by the local con-
trollers. The RTT is a very important measure to gauge the
interaction between the SDWSN simulation and the controller.
Although some research works, such as Erickson [36] and Dixit
et al. [37], assert the controller can handle millions of packets
per second, it is important to qualify this on the perspective
Fig. 11. Packet error rate.
of SDWSN. The average RTT is high on the central controller
as compared to the distributed versions. The central controller
performs well in the initial stages and stagger as more packets
are few. As shown in Fig. 11, the total packet loss is around and
come through. This can also be observed as the number of nodes
less than 2% of the total packets across all experiments. The total
increases. Although limited in the scaling of the nodes, it can
number of packets increases in direct proportion of the number
be deducted that scalability has direct effect on RTT. This can
of nodes. The fragmentation model produced more packets than
also be affirmed by the distributed version. The two distributed
the other experiments, as shown in Fig. 12.
versions of the experiments exhibit consistent RTTs across. The
fragmentation is slightly lower, which is promising improve-
VII. DISCUSSION ment. The slight difference in the average RTT is because the
The purpose of the proposed system is to investigate the vi- fragmentation model produced more packets than the other dis-
ability of using fragmentation as a method of distribution for tributed experiment and thus the average is out of more samples.
SDWSN controller. The controller setup time shows that the The real extend of the difference can be observed from the vari-
KOBO et al.: FRAGMENTATION-BASED DISTRIBUTED CONTROL SYSTEM FOR SOFTWARE-DEFINED WIRELESS 909

ation of the RTTs in the standard deviation; the quantity does investigate the scalability issue in detail to adequately gauge
not affect the average. The improvement can also be qualified impact the proposed system. The efficiency of the SDWSN
by looking at the minimum and maximum RTTs. The minimum framework has an immense contribution to the digital industrial
average RTT is relative but the maximum RTT on the frag- revolution.
mentation is lower across all scenarios, and also, the variation
against the increase in nodes is minimal compared to the other
two experiments. However, the real extend of the improvement REFERENCES
can only be conclusively ascertained with an extended degree [1] P. Hu, “A system architecture for software-defined industrial internet of
of scalability, which at this stage could not be reached due to things,” in Proc. 2015 IEEE Int. Conf. Ubiquitous Wireless Broadband,
2015, pp. 1–5.
the limited capacity of the simulation tool. [2] B. Cheng, L. Cui, W. Jia, W. Zhao, and P. H. Gerhard, “Multiple region of
The variation trajectory in proportion of the scenarios is interest coverage in camera sensor networks for tele-intensive care units,”
clearly observed in the standard variation. The central con- IEEE Trans. Ind. Informat., vol. 12, no. 6, pp. 2331–2341, Dec. 2016.
[3] J. W. Guck, M. Reisslein, and W. Kellerer, “Function split between delay-
troller exhibited the worst with an average of more than 1000 ns, constrained routing and resource allocation for centrally managed QoS in
whereas distributed versions in experiments B and C are better industrial networks,” IEEE Trans. Ind. Informat., vol. 12, no. 6, pp. 2050–
at more than 200 and 100 ns, respectively. This shows among 2061, Dec. 2016.
[4] D. Kreutz, F. M. V. Ramos, P. Esteves Verissimo, C. Esteve Rothenberg,
others, that the fragmentation model does exhibit a consistent S. Azodolmolky, and S. Uhlig, “Software-defined networking: A compre-
variation. The packet error rate or loss is very low; this is inherent hensive survey,” Proc. IEEE, vol. 103, no. 1, pp. 14–76, Jan. 2015.
and thus the distribution had no impact. However, this needs a [5] J. Li, L. Huang, Y. Zhou, S. He, and Z. Ming, “Computation partitioning
for mobile cloud computing in a big data environment,” IEEE Trans. Ind.
further evaluation on a large network for absolute certainty. The Informat., vol. 13, no. 4, pp. 2009–2018, Aug. 2017.
three experiments produced different number of packets. The [6] D. A. Chekired, L. Khoukhi, and H. T. Mouftah, “Decentralized cloud-
packets are mainly determined by the number of nodes, the time SDN architecture in smart grid: A dynamic pricing model,” IEEE Trans.
Ind. Informat., vol. 14, no. 3, pp. 1220–1231, Mar. 2018.
of the test, and the efficiency of the controller(s). The first two [7] G. S. Aujla, N. Kumar, A. Y. Zomaya, and R. Rajan, “Optimal decision
are apparent and common logic; the third is intrinsically related making for big data processing at edge-cloud environment: An SDN per-
to the response time, which is determined by the effectiveness of spective,” IEEE Trans. Ind. Informat., vol. 14, no. 2, pp. 778–789, Feb.
2018.
the protocol. This is because the distributed controllers operate [8] M. Jacobsson and C. Orfanidis, “Using software-defined networking prin-
independently and closer to the infrastructure elements and also ciples for wireless sensor networks,” in Proc. 11th Swedish Nat. Comput.
because of the improved timed complexity of the algorithms. Netw. Workshop, Karlstad, Sweden, May 28–29, 2015, pp. 1–5.
[9] G. A. Akpakwu, B. J. Silva, G. P. Hancke, and A. M. Abu-Mahfouz,
The only caveat is on the first handshaking packet. On the orig- “A survey on 5G networks for the internet of things: Communication
inal ONOS, the lateral exchange of data among the controllers technologies and challenges,” IEEE Access, vol. 5, pp. 3619–3647, 2017.
increase the delay, the fragmentation model averts this. [10] K. M. Modieginyane, B. B. Letswamotse, R. Malekian, and A. M.
Abu-Mahfouz, “Software defined wireless sensor networks application
opportunities for efficient network management: A survey,” Comput. Elect.
Eng., vol. 66, no. 2, pp. 274–287, 2018.
VIII. CONCLUSION [11] D. Ongaro and J. Ousterhout, “In search of an understandable consensus
algorithm,” in Proc. 2014 USENIX Annu. Tech. Conf., 2014, pp. 305–320.
The SDWSN model is a new networking paradigm that arises [12] L. Lamport, “Paxos made simple,” ACM SIGACT News, vol. 32, no. 4,
as a result of applying SDN into WSN. The controller, as in other pp. 51–58, 2001.
SDN-based systems, holds the intelligence of the entire network. [13] B. A. A. Nunes, M. Mendonca, X.-N. Nguyen, K. Obraczka, and T.
Turletti, “A survey of software-defined networking: Past, present, and
This paper proposes an efficient distribution mechanism for the future of programmable networks,” IEEE Commun. Surveys Tuts., vol. 16,
SDWSN controller using fragmentation. no. 3, pp. 1617–1634, Jul.–Sep. 2014.
Fragmentation entails distributing the control logic into dif- [14] W. Xia, Y. Wen, C. H. Foh, D. Niyato, and H. Xie, “A survey on software-
defined networking,” IEEE Commun. Surveys Tuts., vol. 17, no. 1,
ferent segments (fragments), each responsible for a particular pp. 27–51, Jan.–Mar. 2015.
cluster. This method consists of small and lean local controllers [15] H. I. Kobo, A. M. Abu-Mahfouz, and G. P. Hancke, “A survey on software-
closer to the infrastructure elements. These local controllers op- defined wireless sensor networks: Challenges and design requirements,”
IEEE Access, vol. 5, pp. 1872–1899, 2017.
erate independently and only occasionally with a global con- [16] M. Ndiaye, G. P. Hancke, and A. M. Abu-Mahfouz, “Software defined
troller. The proposed system uses eventual consistency data networking for improved wireless sensor network management: A survey,”
model, which consists of best effort and antientropy algorithms. Sensors, vol. 17, no. 5, 2017, Art. no. E1031.
[17] G. M. Omolemo and A. M. Abu-Mouhfaz, “Utilising artificial intelligence
These algorithms are restructured and reused to ensure fragmen- in software defined wireless sensor network,” in Proc. 43rd IEEE Conf.
tation. Ind. Electron. Soc., 2017, pp. 6131–6136.
The results shows that distributed controllers perform better [18] I. Ahmad, S. Namal, M. Ylianttila, and A. Gurtov, “Security in software
defined networks: A survey,” IEEE Commun. Surveys Tuts., vol. 17, no. 4,
than a central controller. The results showed a slight improve- pp. 2317–2346, Oct.–Dec. 2015.
ment in performance for the fragmentation as compared to the [19] D. Kreutz, F. M. V. Ramos, and P. Verissimo, “Towards secure and depend-
original implementation. Therefore, it can be deducted that dis- able software-defined networks,” in Proc. 2nd ACM SIGCOMM Workshop
Hot Topics Softw. Defined Netw., 2013, pp. 55–60.
tribution is indeed essential and that fragmentation does bring [20] S. W. Pritchard, G. P. Hancke, and A. M. Abu-Mahfouz, “Security in
efficiency to SDWSN control system. The improvement in time software-defined wireless sensor networks: Threats, challenges and po-
is particularly important because many IIoT systems and appli- tential solutions,” in Proc. 15th IEEE Int. Conf. Ind. Informat., 2017,
pp. 168–173.
cation, which the SDWSN is envisaged to play a critical role, [21] J. Medved, R. Varga, A. Tkacik, and K. Gray, “OpenDaylight: Towards
are time sensitive. The proportional difference can relatively a model-driven SDN controller architecture,” in Proc. IEEE Int. Symp.
improve as the network scales. The future work thus needs to World Wireless, Mobile Multimedia Netw., 2014, pp. 1–6.
910 IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS, VOL. 15, NO. 2, FEBRUARY 2019

[22] P. Berde et al., “ONOS: Towards an open, distributed SDN OS,” in Proc. Adnan M. Abu-Mahfouz received his M.Eng.
3rd Workshop Hot Topics Softw. Defined Netw., 2014, pp. 1–6. and P.hD. degrees in computer engineering from
[23] A. Tootoonchian and Y. Ganjali, “HyperFlow: A distributed control plane the University of Pretoria. He is currently Princi-
for OpenFlow,” in Proc. 2010 Internet Netw. Manage. Conf. Res. Enter- pal Research Engineer at the Council for Scien-
prise Netw., 2010. tific and Industrial Research (CSIR), Research
[24] S. H. Yeganeh and Y. Ganjali, “Kandoo: A framework for efficient and and Innovation Associate at Tshwane Univer-
scalable offloading of control applications,” in Proc. 1st Workshop Hot sity of Technology and Extraordinary faculty
Topics Softw. Defined Netw., 2012, pp. 19–24. member at University of Pretoria. His research
[25] A. A. Dixit, F. Hao, S. Mukherjee, T. V. Lakshman, and R. interests are wireless sensor and actuator net-
Kompella, “ElastiCon: An elastic distributed SDN controller,” in Proc. work, low power wide area networks, software
10th ACM/IEEE Symp. Archit. Netw. Commun. Syst., 2014, pp. 17–28. defined wireless sensor network, cognitive ra-
[26] L. Galluccio, S. Milardo, G. Morabito, and S. Palazzo, “SDN-WISE: dio, network security, network management, sensor/actuator node de-
Design, prototyping and experimentation of a stateful SDN solution for velopment, smart grid and smart water systems. Dr Abu-Mahfouz is an
wireless sensor networks,” in Proc. 2015 IEEE Conf. Comput. Commun., associate editor at IEEE Access, Senior Member of the IEEE and Mem-
2015, pp. 513–521. ber of many IEEE Technical Communities. Dr Abu-Mahfouz participated
[27] SDN-WISE, “Controlling heterogeneous networks using SDN-WISE in the formulation of many large and multidisciplinary R&D successful
and ONOS.” [Online]. Available: http://sdn-wise.dieei.unict.it/docs/ proposals (as Principal Investigator or main author/contributor). He is
guides/GetStartedONOS.html. Accessed on: Apr. 26, 2017. currently the principal investigator of a large multidisciplinary collabora-
[28] H. I. Kobo, G. P. Hancke, and A. M. Abu-Mahfouz, “Towards a distributed tive project entitled “Smart Water Management System.” Dr Abu-Mahfouz
control system for software defined wireless sensor networks,” in Proc. is the founder of the Smart Networks collaboration initiative that aims to
43rd IEEE Ind. Electron. Soc., 2017, pp. 6125–6130. develop efficient and secure networks for the future smart systems, such
[29] B. T. de Oliveira, C. B. Margi, and L. B. Gabriel, “TinySDN: Enabling as smart cities, smart grid and smart water grid.
multiple controllers for software-defined wireless sensor networks,” in
Proc. 2014 IEEE Latin-Amer. Conf. Commun., 2014, pp. 1–6.
[30] B. T. de Oliveira and C. B. Margi, “Distributed control plane architecture
for software-defined wireless sensor networks,” in Proc. 2016 IEEE Int.
Symp. Consum. Electron., 2016, pp. 85–86.
[31] P. Bailis and A. Ghodsi, “Eventual consistency today: Limitations, exten-
sions, and beyond,” Queue, vol. 11, no. 3, pp. 20–32, Mar. 2013.
[32] C. Coronel and S. Morris, Database Systems: Design, Implementation,
and Management, 11th ed. Boston, MA, USA: Cengage, 2015.
[33] J. Holliday, R. Steinke, D. Agrawal, and A. El Abbadi, “Epidemic algo-
rithms for replicated databases,” IEEE Trans. Knowl. Data Eng., vol. 15,
no. 5, pp. 1218–1238, Sep. 2003.
[34] A. Demers et al., “Epidemic algorithms for replicated database mainte-
nance,” in Proc. 6th Annu. ACM Symp. Principles Distrib. Comput., 1987,
pp. 1–12.
[35] R. van Renesse, D. Dumitriu, V. Gough, and C. Thomas, “Efficient recon-
ciliation and flow control for anti-entropy protocols,” in Proc. 2nd Work-
shop Large-Scale Distrib. Syst. Middlew., 2008, pp. 1–6.
[36] D. Erickson, “The beacon openflow controller,” in Proc. 2nd ACM SIG-
COMM Workshop Hot Topics Softw. Defined Netw., 2013, pp. 13–18.
[37] A. Dixit, F. Hao, S. Mukherjee, T. V. Lakshman, and R. Kompella, “To-
wards an elastic distributed SDN controller,” SIGCOMM Comput. Com-
mun. Rev., vol. 43, no. 4, pp. 7–12, 2013.

Hlabishi Isaac Kobo received the Bachelor and

Masters of Science degrees in computer science Gerhard Petrus Hancke (S’99–M’07–SM’11)
from the University of the Western Cape, Cape received the B.Eng. and M.Eng. degrees in
Town, South Africa, in 2010 and 2012, respec- computer engineering from the University of
tively. He is currently working toward the Ph.D. Pretoria, Pretoria, South Africa, in 2002 and
degree in an efficient distributed control system 2003, respectively, and the Ph.D. degree in com-
for software-defined wireless sensor networks puter science from the University of Cambridge,
from the University of Pretoria, Pretoria, South Cambridge, U.K., in 2008.
Africa. He is currently an Assistant Professor with
He was a Technology Architect with the the City University of Hong Kong, Hong Kong.
Telkom SA. He is currently a Ph.D. researcher He is also an extraordinary Senior Lecturer with
with the Council for Scientific and Industrial Research, Pretoria, South the University of Pretoria. His research interests
Africa. His research interests include software-defined networking, include system security, embedded platforms, and distributed sensing
software-defined wireless sensor networks, and software architecture. applications.