Hivemq 10 Mio Benchmark PDF

10,000,000 MQTT Clients
HiveMQ Cluster Benchmark Paper
10/2017
Tables of Contents
Introduction ....................................................................................................................3
About dc-square ..........................................................................................................4
About HiveMQ ............................................................................................................. 5
Executive Summary....................................................................................................... 6
Benchmark Explained ...................................................................................................7
Purpose of this benchmark ......................................................................................... 7
Goal of this benchmark ...............................................................................................7
Testscenario ..................................................................................................................8
Technical Setup.............................................................................................................12
Overview ......................................................................................................................12
HiveMQ ........................................................................................................................ 12
10 Million MQTT Clients ............................................................................................14
Publisher....................................................................................................................... 17
Infrastructure ................................................................................................................19
Monitoring of the Benchmark ...................................................................................19
Test Execution and Results ......................................................................................... 21
Connect & Subscribe ..................................................................................................22
10 million MQTT Clients using QoS 0 .....................................................................26
10 million MQTT Clients with QoS 1 ........................................................................30
Key Take Aways ............................................................................................................34
Adaption for your MQTT use case .............................................................................35
Appendix....................................................................................................................... 36
Further Resources ...................................................................................................... 36
Files and Configs.........................................................................................................36
© dc-square GmbH 2017. All rights reserved. hivemq.com | 2

Introduction
MQTT has come a long way since its official OASIS release in 2014. Nowadays MQTT is widely
used in all kinds of IoT use cases across all industries, including automotive, logistics and smart
home. Nowadays the tiny protocol is also used outside of the classic IoT domains, e.g. for light-
weight inter-server communication.
As predictions from various research facilities show, the number of devices connected over the in-
ternet will grow to a high 8 digit number over the next couple of years. Our day to day experience
with Enterprise customers shows that this definitely seems in reach.
More and more customers and future customers ask us, if HiveMQ is able to scale with their own
growth: To millions of devices according to their business needs in the near future.
That’s why, over a year ago, we have set an ambitious goal for ourselves, which we considered
enormous at that moment:
Connect 10,000,000 real MQTT clients to HiveMQ and demonstrate that HiveMQ can handle
them smoothly even under high throughput.
Now, almost 7 months after we successfully achieved the goal, the numbers in terms of messages
per second, and messages in total and bandwidth, are still mind blowing.
We hope that you enjoy reading this paper and going on a journey with us to try to understand
what it means to connect 10 million MQTT devices on cloud infrastructure. Getting data-driven in-
sights and an understanding for the numbers involved when building high scale IoT systems.
Exciting times are ahead!
In case you have any questions about this paper or any other related MQTT topic, we are eager to
get in a discussion with you. Feel free to drop me an email to [email protected].
Best,
Head of Customer Success

dc-square GmbH - the company behind HiveMQ

About dc-square
dc-square - the MQTT company, is a software company from Southern Germany focused on the
Internet of Things (IoT) protocol ‘MQTT’’. dc-square delivers everything from its world class MQTT
broker HiveMQ to individual consulting for its customers and any company interested in MQTT.
dc-square has a long history in MQTT, working with the de-facto standard IoT protocol since 2012
and being part of its standardization at OASIS. dc-square's mission is based on the following three
pillars:
Push MQTT as the standard protocol in IoT by providing (free) resources about MQTT through
different channels (blog posts, conference talks, webinars, etc.) and educating customers on
how to use MQTT in Enterprise scenarios.
Building the best MQTT broker on the market, by breaking fresh ground with foundational re-
search on scaling MQTT and how to build highly scalable distributed systems and applications
with extreme performance. HiveMQ is built for Enterprise MQTT deployments with outstand-
ing demands.
Supporting our customers in their MQTT journey from the very early start toward a production-
ready deployment and for the whole time the system is in operation, which is typically way
more than a decade. Our portfolio contains everything from MQTT evaluation support, MQTT
use case consulting, integration of HiveMQ into the customer's IT landscape, designing archi-
tectures on a green field or into an already sophisticated backend system up to customized
development of HiveMQ extensions, consulting on which MQTT client libraries to use and pro-
viding 24/7 HiveMQ support for business critical deployments.
dc-square in one sentence 

We are the MQTT company. With our experts in MQTT and distributed systems, we help you to de-
sign, grow and operate your Enterprise MQTT deployment with millions of concurrent connections.

About HiveMQ
HiveMQ is the most scalable Enterprise MQTT broker, bringing together all major features needed
for an Enterprise MQTT deployment
Best-in-class performance through its event-driven and asynchronous nature, fitted perfectly in
accordance with the MQTT characteristics
Elastic Clustering allows scaling out in both dimensions: horizontal as well as vertical with zero
downtime even during upgrades
Built with security in mind from the beginning, providing essential security features in the
HiveMQ core and the endless possibilities of adding custom security mechanisms via its plugin
system, HiveMQ is predestinated for building secure systems that keep sensitive data safe.
Resilience and robustness, in the age of cyber attacks it is paramount to have a battle-tested
system, especially an MQTT broker, as it is residing at the edge between mobile devices and
enterprise systems.
These and additional Enterprise features can be found in detail on our website .
Besides the feature set HiveMQ is know as one of the most matured MQTT broker available, it is
on the market since 2013 and is used in more than 60 mission-critical deployments all over the
world. From the USA over the UK to Europe and Asia, as well as spanning different industries from
Automotive, Smart Home, Fleet Management and many more. Currently our customers connect
between 100 and around 3 million devices concurrently. This will radically increase over the next
few years. That's why we constantly improve HiveMQ to handle more concurrent connections while
also improving message throughput.
HiveMQ 3.3.0 is out!
The newest version of HiveMQ is out and comes with an awesome new feature set including a
brand-new Web UI. You can try out our free evaluation version with 25 concurrent connections. For
a more extensive evaluation, please let us know, so we can support you along the way.

Executive Summary
The 10 million MQTT client benchmark was an ambitious goal we accomplished in early 2017 in
order to demonstrate the enormous horizontal scalability of HiveMQ and its capability to handle
web-scale throughput use cases.
The sole purpose of this benchmark is to achieve 10,000,000 concurrently connected MQTT
clients and do throughput tests with MQTT Quality of Service (QoS) levels 0 and 1.
As the test scenario we picked a fan-out scenario, where the 10 million devices were subscribed to
1000 topics, meaning that one message to an individual topic would yield messages to 10,000
clients. In two different scenarios we published QoS 0 (4 msg/sec) and QoS 1 (2 msg/sec) with 40
MQTT publishers equally distributed over the topics to generate 1.6 mio msg/sec and 800k msg/
sec.
All tests were conducted on commodity hardware on AWS. No dedicated or reserved instances
were used. Doing the tests on bare metal would produce far better results, due to the fact that
AWS gives only guarantees on capacity not performance. We deployed a 40 node HiveMQ cluster,
40 instances of MQTT client instances running a Benchmark Tool and one monitoring instance
with InfluxDB and Grafana with the help of Terraform.
We have successfully reached the 10 million concurrent connections and during the QoS 0 test a
massive outgoing throughput of more than 1.7 million messages per second has been shown.
The outgoing message throughput of HiveMQ is 2.5 times greater than the amount of total mes-
sages WhatsApp processed in a single day in 2014.
In numbers HiveMQ can handle 6.26 billion outgoing messages per hour and around 150 billion
outgoing messages on a single day. During the test the average CPU usage in the HiveMQ cluster
was ~64%.
In the second test with QoS 1 we showed a throughput of 780 thousand messages per second
outgoing and ingoing at the same time. The total number of outgoing messages per hour were
2.49 billion (2,487,848,486 messages) and the incoming messages were 2.81 billion
(2,806,475,764 messages). This yield a bandwidth of 102.41 billion messages (102,408,992,000
messages) outgoing and 7.73 billion messages (7,267,899,154 messages) incoming.
These numbers clearly express that HiveMQ is capable of handling huge web-scale deployments
with an outstanding throughput.
© dc-square GmbH 2017. All rights reserved. hivemq.com | 6 5

Benchmark Explained
Let's jump right into the benchmark by first going over our motivation to carry the benchmark
conduction out and make it public, as well as the test scenario.
Purpose of this benchmark
The purpose of this benchmark is to showcase HiveMQ’s current capabilities when it comes to
concurrently connected devices to a broker cluster. In the IoT space, the number of how many
concurrently connected device we will have in the future predicted by research is constantly get-
ting higher over time. While a few years ago 1 million was a huge number, nowadays more than
the tenfold is in reach for Enterprise systems. The fact that HiveMQ was built exactly for this kind of
Enterprise MQTT deployment has pushed us as dc-square to think big and in completely new di-
mensions. We have cracked our internally set goal from 2016 of benchmarking 10 million concur-
rent connections at the beginning of 2017 already. The purpose of this paper is to show every
HiveMQ customer and MQTT enthusiast what HiveMQ and MQTT is capable of. And all of this in a
cloud environment that is leveraged by a lot of HiveMQ customers and can easily be reconstruct-
ed. It’s noteworthy that no special configuration was used for the benchmark and all MQTT fea-
tures are supported by HiveMQ in the scalability demonstrated, including retained messages, QoS
1 and 2, Queued Messages and persistent sessions.
Goal of this benchmark
The goal is very simple for this particular benchmark:
Demonstrate that HiveMQ can handle 10,000,000 (million)  

concurrent connections with a very high throughput.
That's it!
Note: We picked 10 million as a line in the sand to work towards, which sounded massive for us,
when we picked it almost over a year ago. We knew that our customers will be moving toward this
number very quickly. We are confident that a single HiveMQ cluster can scale beyond 10 million
concurrent connections.

Testscenario
In order to reach our goal we created the test scenario described in this chapter. The next chapter
will go into the technical details and the concrete implementation.
Description
The goal of reaching 10 million connections with HiveMQ could be achieved with a multitude of
possible test scenario designs. From a simple connection-only test to exhausting tests demonstrat-
ing different MQTT use cases. We have picked a Fan-Out test case for this benchmark, which is a
common use case for our customers. The focus is on reaching 10 million MQTT connections and
showing a huge throughput with different MQTT Quality of Service (QoS) levels.

The execution of the described benchmark scenario requires the following steps:
1. Set up a monitoring instance for all HiveMQ nodes and benchmark tool instances
2. Set up enough HiveMQ nodes to form a cluster and handle 10 million clients and con-
nect to monitoring instance
3. Set up enough Benchmark Tool instances to establish 10 million client connections,

connect to the monitoring instance and subscribe to the defined topics (see number 1
and 2 in the graphic above)
4. Starting the publisher clients and letting them run for the predetermined time with
QoS 0 (see number 3 and 4 in the graphic above)
5. Wait for a few minutes
6. Starting the publisher clients and let them run for the predetermined time with QoS 1
(see number 3 and 4 in the graphic above)
7. Wait for a few minutes
8. Collect all results
9. Shut down the whole test setup and clean up
Deciding on the variables
After having defined the general test scenario the more interesting part is discovering the variables
for the test:
Number of topics: We decided to use 1,000 topics, having a distribution of 10,000 subscribers
per topic with 10 million clients.
Payload: As the payload we used a random string with a 32 byte size
Number of HiveMQ nodes: From prior benchmarks we know that HiveMQ can handle up to 1
million connections per instance without issues, however our recommendation for production
HiveMQ installations is to have between 100k and 500k concurrently connected MQTT clients
per HiveMQ instance. Due to this test being extremely high in terms of concurrently connected
clients we decided to go ahead with 250k per HiveMQ instance, which resulted in 40 HiveMQ
nodes.

Number of Benchmark Tool instances: After previous tests with our benchmark tool we have
decided to also use 40 instances, each spawning 250,000 MQTT clients. This is only possible
due to the async and event-driven nature of the tool, built on similar principles as HiveMQ.
Run time for the test: We decided to use 30 minutes, because this seemed like a good amount
for showing the stability of the 10 million connections and gave us the option of doing multi-
ple tests in a short time period. As this was the first test setup in this scale, we will do longer
running tests in the future as we are now very confident in executing tests in this scale.
Message rate for the publisher: This was the most interesting question for us in the beginning,
because it was hard to predict. After a couple of tests we have decided to use the following
values:
Message Rate for Publisher

Test Msg/sec
QoS 0 4.00
QoS 1 2.00
Summary of the scenario
Having 10 million clients subscribing to 1,000 topics results in 10,000 subscribers per topic. The
publishing clients will send messages in a round robin fashion to each topic, each on one broker.
That means in numbers we have
40 publishing clients (referenced as “Publisher” from now on) connected to the 40 HiveMQ
nodes, publishing approx. 4 messages per second to 10,000 subscribers each, that means (40
times 4 times 10,000) around 1,600,000 outgoing QoS 0 messages per second.
40 publishing clients (referenced as “Publisher” from now on) connected to the 40 HiveMQ
nodes, publishing approx. 2 messages per second to 10,000 subscribers each, that means (40
times 2 times 10,000) around 800,000 outgoing QoS 1 messages per second.
Deciding on the target platform
After having the scenario mapped out we decided to use Amazon Web Services (AWS) to run the
benchmark on. For us this was the obvious choice, because a large portion of our customers lever-

age AWS as their target platform for running a HiveMQ cluster. Also we are very familiar with it,
having done infrastructure setups and benchmarks on AWS for quite a long time. Of course AWS
comes with a downside in regard to benchmarking, due to them only guaranteeing capacity and
not performance . A benchmark on bare metal would definitely yield a better result.
As the purpose of this benchmark is to represent a use case as common as possible, AWS is the
ideal choice.
Let's have a look how we set up and configured all the components on AWS in the next chapter.
HiveMQ is a product of dc-square GmbH. © dc-square GmbH 2016. All rights reserved.
This document is current as of the initial date of publication and may be changed by dc-square at any time.
Technical Setup
This chapter explains the benchmark setup in detail and lists the tools we used.
Overview
This graphic provides a high level overview of the architecture used to conduct the benchmark. We
will walk you through each of the components in this chapter.
HiveMQ
Overview
The main component and subject under test (SUT) was the newest version of HiveMQ 3.3.0 to
showcase its capabilities of connecting and handling 10 million concurrent connections in a single
HiveMQ cluster.

Hardware and Operating System used
Name Value
Instance Type c4.2xlarge
RAM 15GiB (~16GB)
vCPU 8
Physical Processor Intel Xeon E5-2666 v3
Clock Speed (GHz) 2.9
Storage EBS-only
Dedicated EBS Bandwidth 1,000

(Mbps)
Operating System Vanilla Amazon Linux

ami-b73b63a0
Setup Steps
The following steps have been executed on each HiveMQ broker machine in order to prepare
HiveMQ for the 10 million benchmark.
1. Install telegraf for monitoring the Linux system metrics (like CPU, memory, bandwidth, etc.)
2. Prepare the Linux system by increasing open files and tuning the linux kernel. We have used
the same settings as described in the HiveMQ documentation. The commands can also be
found in the appendix of this document.
3. Install JDK 8: We used JDK 8u131
4. Install HiveMQ as a Linux service: We automated the installation of HiveMQ as a service ac-
cording to our documentation and chose the start script init-script/hivemq, as explained under
the headline "For all other unix systems" in the documentation.
5. Increase the heap of the Java Virtual Maschine (JVM) running HiveMQ to 10GB by adjusting
the run.sh file (adding -Xmx10g)
6. Configuring HiveMQ for clustering by adding the following to the config.xml  
... 
 
<cluster> 
<enabled>true</enabled> 
<transport> 
<tcp> 
<bind-port>7800</bind-port> 

<bind-address>*LOCAL IP ADDRESS*</bind-address> 
</tcp> 
</transport> 
<discovery> 
<plugin> 
<reload-interval>10</reload-interval> 
</plugin> 
</discovery> 
</cluster>
7. Add necessary HiveMQ Plugins
a) S3 Discovery plugin (Link) to leverage the S3 database on AWS for the discovery of all
nodes of the HiveMQ cluster. Configuration is attached in the appendix.
b) InfluxDB Plugin (Link) for publishing HiveMQ metrics to Influx DB for analyzing the bench-
mark results afterwards. We also attached the configuration in the appendix.
8. Add license file  

 
Adding a license file that allows up to 10mio connections is essential, as the HiveMQ evalua-
tion version allows only up to 25 connections. If you want to evaluate HiveMQ for more con-
nections, please let us know.
9. Start HiveMQ and telegraf 
service telegraf start 

 
service hivemq start
That’s it.
10 Million MQTT Clients
Overview
After setting up HiveMQ we need to setup 10 million MQTT clients, which will connect to HiveMQ
and act as subscribers for the described fan-out scenario. We have used our own benchmarking
tool in order to achieve that. As far as we know, there is no other tool that can create such a huge
amount of connections (250,000 on a single server), while being deployed and controlled over a
REST API. The tool is based on similar asynchronous and event-driven principles as HiveMQ itself
and therefore is able to handle a massive amount of throughput and connections. The benchmark

tool is currently not publicly available. We have the intention of releasing it at a later point of time.
If you are already a HiveMQ customer, you can get access to the current version. Feel free to con-
tact us [email protected].
Hardware and Operating System used
Name Value
Instance Type c4.2xlarge
RAM 15GiB (~16GB)
vCPU 8
Physical Processor Intel Xeon E5-2666 v3
Clock Speed (GHz) 2.9
Storage EBS-only
Dedicated EBS Bandwidth 1000

(Mbps)
Operating System Vanilla Amazon Linux

ami-b73b63a0
Setup Steps
1. Prepare the Linux system by increasing open files and tuning the linux kernel: We used the
same settings as described in our documentation.
2. Additionally to the linux kernel optimization for HiveMQ, it is necessary to increase the available
ports via the following command:  
echo "net.ipv4.ip_local_port_range = 1024 65535" >> /etc/sysctl.conf
3. The benchmark tool is a Java application that is deployed to all client instances as a jar file,
which will be executed via the command line. Our deployment just copies the file and exe-
cutes it with:  
sudo nohup java -Xmx10g -jar hivemq-benchmark-tool.jar rest -ba 0.0.0.0 -bp 8080 &
4. This command starts the benchmark tool in the REST API mode on all network interfaces
and port 8080, as well as increasing the heap for the JVM to 10 GB.
And we are good to go!

Setting up the subscribers
The benchmark tool can be remotely controlled via a REST API. In the following we explain how
this works and the exact parameters used in the benchmark.
For the test case we have written a small Java application, which assembles the HTTP request and
its body which will be sent to all client instances. The URL determines which mode the benchmark
tool will operate in. We have 3 modes: Publish, Subscribe, Roundtrip.
The Publish Mode simulates a certain amount of MQTT clients that publish messages in a cer-
tain rate.
The Subscribe Mode simulates a certain amount of MQTT clients that subscribe to one topic
and wait for messages to be sent to the topic. When messages arrive, the clients will record
metrics about the messages.
The Roundtrip Mode combines the two and simulates MQTT clients, which send messages to
themselves. Meanwhile it calculates the latency between sending and receiving in different
percentiles.
In this case we used the Subscribe Mode to establish 10 million MQTT clients subscribing to 1,000
topics.
The trigger URL for the Subscribe Mode looks like this: http://<Client IP>:8080/client/subscribe
All the relevant data will be handed over to the tool as part of the POST body.
This table is an excerpt of the most important parameters of the benchmark tool, relevant for this
benchmark case:
Benchmark Tool Parameter (excerpt)

Parameter Name Used Value Comment
c amount of 250.000 this specifies how many connections each

connections tool should open
cc connections per 500 this specifies how fast the MQTT clients
second should connect to the broker
h broker ip(s) all 40 IPs   this option allows specifying one or more
(1.2.3.4, 2.3.4.5, 3.4.5.6, …)
broker ips
idh IP Address for <IP Influx DB> this option allows naming a influx db
Influx DB instance into which the recorded metrics
should be reported

Parameter Name Used Value Comment
idt Influx DB tag clientX the tag will group all reported metrics by the
(X is no of client instance
client tool instance
1-40)
k keep alive (sec) 60 this specifies the MQTT keep alive interval in
which the MQTT client will send a PING to
the broker
pe persistent session FALSE this parameter specifies if the client

establishes a persistent or clean MQTT
session
q Quality of Service 2 this allows specifying the MQTT QoS level

Level for the
Subscription
to topic name topic1, topic2, …, this option allows specifying one or more
topic 1,000 topics
Why QoS 2? QoS 2 is our default setting, because the highest QoS will always be the one that is
sent by the publisher. It is not possible in MQTT to upgrade the QoS for a certain client higher than
the original Publisher QoS.
The used script to send the HTTP request to the REST APIs can be found in the appendix.
After sending the REST request to each client simultaneously, each client instance will start on their
own to connect the specified number of clients. After finishing the connection process, it will start
to send the subscriptions to the broker. For connecting and subscribing it will use the provided
broker IP addresses and topic list, selecting IP addresses and topics in a round robin fashion. After
finishing all subscriptions the tool will just wait for messages to come for recording them to In-
fluxDB. No further proactive action will be done by the client instances
Publisher
After the client instances are up and running, we already have 10 million devices connected and
subscribed!

Concurrently Connected MQTT Clients
10M
8M
Connections
5M
3M
0
00:00 05:00 10:00
Time passed (min)
Yes, Mission 10 million clients accomplished!
But wait, now the throughput test has to prove the capability of HiveMQ also stably handling a
huge number of messages, flowing through the 40 node cluster to the 10 million connected
MQTT clients
Therefore we built a very simple MQTT publisher tool with Eclipse Paho for Java to send messages
to each of the broker nodes. We start 40 instances of MQTT clients acting as publisher, each send-
ing a single digit number of messages per second, depending on each test.
Publishing Tool Parameter

Parameter Name
Client ID Publisher-<BrokerIP>
Payload random 32 bytes
Clean Session TRUE
QoS 0, 1
The code of the publishing tool can be found in the appendix.

Infrastructure
After describing the test scenarios and different tools used to execute the benchmark, this chapter
elaborates more on the underlying infrastructure and how we provisioned it.
Provider
As mentioned above all benchmarks were executed on Amazon Web Services (AWS).
AWS allows to deploy servers on a shared environment and is often used for cloud services. AWS is
the most popular cloud provider of HiveMQ customers, so using AWS for the benchmark was a
natural choice.
Please note that no dedicated hardware servers and no dedicated AWS instances were used. We
opted for common AWS instances. Be aware that these test results would be far better, if we have
used bare metal hardware, where better performance guarantees are provided. If you use this
benchmark as scalability comparison for your own deployment, please keep this in mind.
Provisioning
For making the benchmark setup as smooth and easy as possible we have to provision 80 in-
stances (40 HiveMQ instances and 40 client instances) and a S3 bucket for the cluster discovery
and for providing all the tools like HiveMQ, Benchmark Tool, etc. for being automatically installed.
We decided to use Terraform for the provisioning of the whole benchmark infrastructure. Terraform
is made to create, change and improve production infrastructure across different cloud providers.
It follows the infrastructure as code paradigm and is built and open sourced by HashiCorp.
It is the ideal tool for quickly setting up this kind of benchmark without hassle.
The terraform scripts we used are attached in the appendix.
Monitoring of the Benchmark
Successfully executing a benchmark is the first important step, having accurate and trustworthy
numbers is the second and more important one.
We have experimented a lot in the last years, what the best method for live monitoring and post
analysis of benchmarks is. Our favourite setup is using InfluxDB together with Grafana, which we

also recommend for HiveMQ monitoring in general (more on that in this blog post). The combina-
tion is ideal for monitoring during the test and allows the extraction of all values as CSV files.
InfluxDB is an open source time series database, which makes it easy to store time driven metrics
and to query them via an SQL-like syntax. All metrics HiveMQ exposes about itself and the JVM
were inserted via the HiveMQ Influx DB plugin (@TODO https://www.hivemq.com/plugin/influxdb-
plugin/). We also used Telegraf for this benchmark. Telegraf is an agent, which reports Linux system
metrics to Influx DB. The MQTT benchmark tool also reports metrics directly to InfluxDB.
Grafana is responsible for displaying the persisted data series and allows building cumulative
charts of our huge amount of nodes.
This setup allows us to capture and easily analyze all relevant data from all 80 instances.

Test Execution and Results
This chapter summarizes the test execution, the gained results and a discussion about the tests.
For each test we are going to look at the performance of HiveMQ from two main angles:
throughput in terms of messages per second and bandwidth
load in terms of CPU and memory
The first gives a clear indication of HiveMQ’s performance in handling MQTT messages. The sec-
ond takes a closer look, on how many resources HiveMQ needs for achieving the performance
and if there is still breathing room left for occasional peaks and to handle the failure of individual
nodes.
According to the described test scenario, we gathered results to 3 concrete tests:
Connecting and subscribing 10 million clients
Publishing messages in a fan-out case with QoS 0 to 10 million clients
Publishing messages in a fan-out case with QoS 1 to 10 million clients
Let's get into the details.
How to read and interpret the charts?
During the test we have collected metrics from every individual HiveMQ node out of the 40 nodes
used. For better readability and comprehension we have accumulated the metrics of the different
nodes and calculated the average (e.g. CPU usage) or sum (e.g. connections), depending on the
metric.

Connect & Subscribe
Description
Before starting the load tests, the 10 million subscribers have to be connected and subscribed to
the defined topics. These results show the phase during which the clients connected and sub-
scribed. Before starting the phase we opened an ssh session to double check if all HiveMQ in-
stances successfully formed one cluster in the HiveMQ log:
2017-10-13 23:50:21,463 INFO - Cluster size = 40, members : [u7BAI, Key8h, u3j53, Qo4HW,
4RlNy, 8yW9p, InAb7, xYS2G, s1Bxv, hH08h, Vbks9, GXx6K, BOVaP, ahq4U, 26CsN, ca
JP0, mvLse, YeHrp, 2IjsD, iqEBL, FdeHr, ccvP0, V83nP, AX5ZO, 3f2oC, ZLlMR, xiPuz,
0FN9p, MPD0R, vRnAH, YRL0U, wL0cG, RwXZj, 6QH5M, yJIBo, bA99c, epLWi, UNiIZ,
Fdl5v, xtEAF].
 
Quick Results: Connect & Subscribe Test - 10,000,000 MQTT clients
Connect Duration: 8 min 47 sec

Connect Speed (avg): 19,011 connects/sec
CPU 30 %
Memory 5.07 GB
Subscribe Duration: 4 min 31 sec

Subscribe Speed (avg): 36,765 subscribes/sec
CPU 38 %
Memory 6.97 GB
HiveMQ 3.3.0
40 node Cluster

Results
Here are the results of the establishment of 10,000,000 MQTT connections and the successful
subscribe of the same.
MQTT connections MQTT subscriptions
10M 10M
Subscriptions total
Connections total
8M 8M
5M 5M
3M 3M
0 0
00:00 05:00 10:00 00:00 05:00 10:00
Time passed (min) Time passed (min)
It can be observed from the charts above that the connections are established perfectly linear.
Each single benchmark tool operates independent and if you have a close look the end of the
connecting period and beginning of the subscription period, you can see that the time is overlap-
ping a few seconds.
MQTT connections MQTT subscriptions
40K 50K
Subscriptions / sec
30K 38K
Connects / sec
20K 25K
10K 13K
0 0
00:00 05:00 10:00 00:00 05:00 10:00
As expected the MQTT CONNECTs averages at around 19,000 per second. As we have config-
ured 500 connects per second for each of the 40 benchmark tool instances.

CPU Usage Memory Usage
100 8
75 6
Memory (GB)
CPU (%)
50 4
25 2
0 0
00:00 05:00 10:00 00:00 05:00 10:00
When looking at the load of the HiveMQ instances, the CPU stabilises at 30% when connecting
and 38% when subscribing, dropping back to 15% afterwards. The memory grows during the con-
nection period, while slightly rising as the clients subscribe. Staying at around 6,97 GB after the
subscription period is completed.
Take aways from the Connect & Subscribe
The charts above show the successful connection and subscription of 10 million MQTT clients. It
can be seen that the connections as well as subscriptions happen very smoothly. The whole
process took around 12 minutes in total. We haven't focused on getting the best possible connec-
tion/subscription rate in this test. When looking at the CPU usage, there is definitely lots of room to
achieve even higher rates.
When looking at the CPU usage, it can be observed that it grows slowly while connecting from un-
der 25% up to approx. 38% and stays their during subscribing. Afterwards it drops back down to
15%.
Everything is stable and behaves as expected. 
Summary:
The HiveMQ Cluster successfully handles the CONNECT and SUBSCRIBE of 10 million MQTT
clients.
It took the HiveMQ cluster 8 min 47 sec to connect 10 million clients with a rate of 500 con-
nects/second per client instance (20.000 per second in total)
The MQTT clients connected on average with a rate of 19,011 MQTT connections per second.
It took the Benchmark Tool MQTT clients 4 min 31 sec to send 10 million subscriptions.

The MQTT clients subscribed on average with a rate of 36,765 MQTT subscriptions per sec-
ond.
Higher connection and subscription rates are definitely possible, as the average CPU usage is
below 38%

10 million MQTT Clients using QoS 0
Description
After we have successfully connected 10 million clients we want to see how HiveMQ behaves when
it gets under load. Therefore we started the Publishing tool with the following parameters:
Message payload: random 32 bytes

Publishing rate: 4 messages per second
QoS: 0
This results in around 1,600,000 outgoing messages per sec, because we have 40 brokers, getting
4 messages per second from the individual publishing tool (remember one publishing tool per
node) and the broker forwards each message to 10,000 subscribing MQTT clients, which in total
equals 1.6 million outgoing messages per sec.
 
Quick Results: 10,000,000 MQTT clients 30min Throughput Test using QoS 0
Messages/sec outgoing (avg) 1.74 mio/sec

Message outgoing total 3.13 bn msg
Bandwidth outgoing (avg) 389,78 MB/sec
Bandwidth outgoing total 685,16 GB
CPU (avg) 64,72 %

Memory (avg) 11,33 GB
Messages/sec incoming (avg) 167 k/sec

Message incoming total 300 mio msg
Bandwidth incoming (avg) 327,84 MB/sec
HiveMQ 3.3.0 Bandwidth incoming total 576,28 GB
40 node Cluster

Results
MQTT messages/sec, outgoing MQTT messages/sec, incoming
3.0M 300K
2.3M 225K
Messages / sec
Messages / sec
1.5M 150K
750K 75K
0 0
00:00 05:00 10:00 15:00 20:00 25:00 30:00 00:00 05:00 10:00 15:00 20:00 25:00 30:00
Time passed Time passed
The test shows that HiveMQ can handle an average of 1.74 million messages per second
(1,739,876 msg/sec) outgoing and 167 thousand messages per second (166,925 msg/sec) in-
coming. In total that is 3.13 billion messages (3,131,776,800 msg) outgoing and 300 million msg
(300,465,888 msg) incoming for the test period of 30min.
Bandwidth outgoing Bandwidth incoming
600M 600M
Bandwidth (Bytes/sec)
450M 450M
300M 300M
150M 150M
0 0
00:00 05:00 10:00 15:00 20:00 25:00 30:00 00:00 05:00 10:00 15:00 20:00 25:00 30:00
When looking at the system bandwidth HiveMQ handled an average of 389.78 Megabytes per
second (408,711,565 Bytes/sec) outgoing and 327.84 MB/sec (343,762,614 Bytes/sec) incoming.

100 12.0
10.0
75
Memory (GB)
8.0
CPU (%)
50 6.0
4.0
25
2.0
0 0.0
00:00 05:00 10:00 15:00 20:00 25:00 30:00 00:00 05:00 10:00 15:00 20:00 25:00 30:00
During the execution of the test the CPU usage averaged at 64.72 % and the memory at 11.33 GB
(12,164,655,331 Bytes).
MQTT PINGRESP outgoing MQTT PUBLISH incoming

MQTT PUBLISH outgoing MQTT PINGREQ incoming
4M 600K
3M 450K
Messages / sec
Messages / sec
2M 300K
1M 150K
0 0K
00:00 05:00 10:00 15:00 20:00 25:00 30:00 00:00 05:00 10:00 15:00 20:00 25:00 30:00
Time Passed (min) Time Passed (min)
The two charts above show the composition of the outgoing and incoming MQTT messages.
Outgoing includes MQTT PUBLISH messages sent to the 10 million clients and MQTT PINGRESP
the broker sends in response to the PINGREQ of the 10 million clients. The outgoing PINGRESP
messages were on average 167 thousand messages per second ( 167,043 msg/sec) and the PUB-
LISH messages 1.57 million messages per second (1,574,370 msg/sec). Additionally to the PIN-
GREQ messages the incoming traffic includes the PUBLISH messages from the 40 Publisher. The
incoming PING REQ messages were on average 166 thousand messages per second (166,467
msg/sec) and the PUBLISH messages incoming were on average 106 messages per second
(106.47 msg/sec). Therefore, the PUBLISH messages can be barely seen in the chart.

Note: The MQTT 3.1.1 specification allows a client to send a PINGREQ at any time (section 3.1.2.10).
Most MQTT clients only sent PINGREQ in the absence of other messages, as it is mandatory by the
specification. The MQTT Benchmark Tool used in this benchmark sends PINGREQ independent of
other MQTT messages.
Take aways from the QoS 0 Test
The QoS 0 Test shows the massive throughput capabilities of HiveMQ in a large cluster to fully play
out its strength in scaling horizontally.
HiveMQ with 10 million concurrently connected clients achieved a constant throughput of 1.7
million outgoing messages/sec with a CPU load of not more than 64 % on average.
In order to make the number easier to grasp, let’s compare them with some huge internet ap-
plications, one good example is WhatsApp. In 2014 WhatsApp tweeted that they have reached 64
billion messages (20 billion incoming, 44 billion outgoing) in one day.
When comparing the total of 64 billion messages to the numbers we have achieved in the test, we
have 3,432,242,688 messages when adding incoming and outgoing traffic per 30min. This adds
up to 164,75 billion messages (164,747,649,024 messages) on a single day.
A HiveMQ cluster can easily handle 2.5 times the amount of messages sent through the whole
WhatsApp infrastructure each day!
When looking at the performance, as well as the load metrics, it is clear that HiveMQ is not only
delivering a high throughput, but is also staying in a controlled and stable state with a usage of
64% CPU and 73% of the memory.

10 million MQTT Clients with QoS 1
Description
After we have successfully finished the QoS 0 test, we wanted to see how HiveMQ behaves with
QoS 1. Therefore we started the Publishing tool with the following parameters:
Message payload: random 32 bytes

Publishing rate: 2 messages per second
QoS: 1
This results in around 800,000 outgoing messages per sec, as we have 40 brokers, getting 2 mes-
sages per second from the individual publishing tool (remember one publishing tool per node)
and the broker forwards each message to 10,000 subscribing MQTT clients, which in total equals
800,000 outgoing messages per sec.
 
Quick Results: 10,000,000 MQTT clients 30min Throughput Test using QoS 1
Messages/sec outgoing (avg) 780 k/sec

Message outgoing total 1.40 bn msg
Bandwidth outgoing (avg) 279,44 MB/sec
Bandwidth outgoing total 491,20 GB
CPU (avg) 59,31 %

Memory (avg) 11,44 GB
Messages/sec incoming (avg) 780 k/sec

Message incoming total 1,40 bn msg
Bandwidth incoming (avg) 220,24 MB/sec
HiveMQ 3.3.0 Bandwidth incoming total 387,14 GB
40 node Cluster

Results
MQTT messages outgoing MQTT messages incoming
2.0M 2.0M
1.5M 1.5M
Messages / sec
Messages / sec
1.0M 1.0M
500K 500K
0 0
00:00 05:00 10:00 15:00 20:00 25:00 30:00 00:00 05:00 10:00 15:00 20:00 25:00 30:00
The test shows that HiveMQ can handle an average of 780 thousand messages per second
(779,576 msg/sec) outgoing and 780 thousand messages per second (779.577 msg/sec) incom-
ing. In total that is 1.40 billion messages (1,403,237,449 msg) outgoing and 1.40 million msg
(1,403,237,882 msg) incoming for the test period of 30min.
Bandwidth outgoing Bandwidth incoming
600M 600M
450M 450M
300M 300M
150M 150M
0 0
00:00 05:00 10:00 15:00 20:00 25:00 30:00 00:00 05:00 10:00 15:00 20:00 25:00 30:00
When looking at the system bandwidth HiveMQ handled an average of 279.44 Megabytes per
second (293,010,785 Bytes/sec) outgoing and 220.24 MB/sec (230,935,379 Bytes/sec) incoming.

100% 12
10
75%
Memory (GB)
8
CPU (%)
50% 6
4
25%
2
0% 0
00:00 05:00 10:00 15:00 20:00 25:00 30:00 00:00 05:00 10:00 15:00 20:00 25:00 30:00
Time Passed (min) Time Passed (min)
During the execution of the test the CPU usage averaged at 59.31 % and the memory at 11.44 GB
(12,287,767,674 Bytes).
MQTT PUBACK, outgoing MQTT PUBLISH, incoming

MQTT PINGRESP, outgoing MQTT PINGREQ, incoming
MQTT PUBLISH, outgoing MQTT PUBACK, incoming
1.5M 1.5M
1.1M 1.1M
Messages / sec
Messages / sec
750K 750K
375K 375K
0 0
00:00 05:00 10:00 15:00 20:00 25:00 30:00 00:00 05:00 10:00 15:00 20:00 25:00 30:00
Times passed (min) Times passed (min)
The two charts above show the composition of the outgoing and incoming MQTT messages.
Outgoing includes MQTT PUBLISH messages sent to the 10 million clients, MQTT PINGRESP the
broker sends in response to the PINGREQ of the 10 million clients and PUBACK for the QoS 1
messages. The outgoing PUBLISH messages were on average 634 thousand messages per sec-
ond (634,199 msg/sec), the outgoing PINGRESP were on average 167 thousand messages per

second (167,419 msg/sec) and the outgoing PUBACK messages were on average 62 messages
per second (61,54 msg/sec). Additionally to the incoming PINGREQ messages the incoming traf-
fic includes the PUBACK messages from the 10 million clients and the PUBLISH messages from the
40 Publishers. The incoming PUBLISH messages were on average 41 messages per second (41.39
msg/sec), the incoming MQTT PINGREQ messages were on average 167 thousand messages
per second (167,420 msg/sec) and the incoming PUBACK messages were on average 637 thou-
sand messages per second (637,314 msg/sec). Therefore, the amount of PUBLISH messages is
hardly recognizable.
Therefore, the PUBLISH messages can be barely seen in the chart.
Note: The MQTT 3.1.1 specification allows a client to send a PINGREQ at any time (section 3.1.2.10).
Most MQTT clients only sent PINGREQ in the absence of other messages, as it is mandatory by the
specification. The MQTT Benchmark Tool used in this benchmark sends PINGREQ independent of
other MQTT messages.
Take aways from the QoS 1 Test
The QoS 1 Test shows the massive throughput capabilities of HiveMQ even in case of QoS 1
messages, where its large clusters fully play out its strength in scaling horizontally.
HiveMQ with 10 million concurrently connected clients achieved a constant throughput of 780
thousand outgoing messages/sec and 780 thousand incoming messages/sec with a CPU load of
not more than 59 % on average.
In order to make the number easier to grasp, let’s compare them with some huge internet ap-
plications, one good example is WhatsApp. In 2014 WhatsApp tweeted that they have reached 64
billion messages (20 billion incoming, 44 billion outgoing) in one day.
When comparing the total of 64 billion messages to the numbers we have achieved in the test,
we have 2,806,475,331 messages when adding incoming and outgoing traffic per 30min. This
adds up to 134.71 billion messages (134,710,815,888 messages) on a single day.
A HiveMQ cluster can easily handle 2.1 times the amount of messages sent through WhatsApp
each day!
When looking at the performance, as well as the load metrics, it is clear that HiveMQ is not only
delivering a high throughput, but is also staying in a controlled and stable state with a usage of
59% CPU and 76% of the memory.

Key Take Aways
HiveMQ can handle 10.000.000 connections on commodity hardware with an outstanding

and rock stable throughput of 1.7 million outgoing messages (QoS 0 Test) and 780 msg/sec
simultaneously in both directions incoming and outgoing (QoS 1 Test).
HiveMQ is able to scale horizontally up to a mid double digit number of cluster nodes with
ease
HiveMQ is able to handle large fan-out cases with high amplification factors
The throughput was constant and rock stable on the time scale under test
The message rates scales with the cluster size and amount of CPUs
HiveMQ is ideally suited for both small and massive deployments due to its ability to scale hor-
izontally. It is the most scalable MQTT broker with outstanding performance even with the de-
livery of QoS 1 guarantees

Adaption for your MQTT use case
This benchmark was designed to show the impressive capabilities of HiveMQ, which the numbers
clearly do. Please keep the following factors in mind when you draw conclusions for your own
MQTT use case.
Transport security - TLS: We have used MQTT over TCP without encrypting the connection with
TLS. TLS introduces more network and CPU overhead than plain MQTT. We strongly recom-
mend to offload TLS to a load balancer, ideally with a cryptographic hardware module for faster
processing.
Latency: We didn't focus on nor did we measure the end-to-end latency of the messages sent
in the scenario. This benchmark was aiming to demonstrate the performance and scalability of
the broker. In order to achieve that, the MQTT client instances and the HiveMQ cluster nodes
were both deployed in the same cloud environment on purpose. This was necessary to create
the desired load and amount of connections, without introducing too much external and un-
controllable factors like network latency.
Other MQTT use cases: Of course there are other MQTT use cases as well. As demonstrated in
an earlier benchmark - the fan-out scenario is only one of 3 very common MQTT scenarios.
Test duration: The conducted tests showed a stable performance for 30 minutes. Enterprise
MQTT brokers are made to be in continuous operation 24/7/365. So for an even better signifi-
cance a long running test for multiple hours or days would be more appropriate.
Load balancer, firewalls, etc.: We didn't use any third party component in this benchmark, be-
cause the focus was on HiveMQ. Nevertheless in most deployments there are other network
components in place. If you want to know how HiveMQ would behave in your specific scenario,
we can execute a similar test in your specific environment.
If you have any questions regarding the adaptability of this benchmark to your use case, please let
us know.

Appendix
Further Resources
Blog Post “Clustering MQTT”
HiveMQ Cluster Paper
Files and Configs

terraform-test.tf
provider "aws" {
access_key = "${var.access_key}"
secret_key = "${var.secret_key}"
region = "${var.region}"
allowed_account_ids = [
"474125479812"]
}
resource "aws_instance" "hivemq" {
count = "${var.broker_count}"
ami = "${lookup(var.amis, var.region)}"

instance_type = "${var.broker_instance_type}"
key_name = "${aws_key_pair.deployer.key_name}"
security_groups = [
"${aws_security_group.mqtt.name}"]
associate_public_ip_address = "true"
iam_instance_profile = "${var.instance_profile}"
tags {
Name = "HiveMQ Server ${count.index+1}"
}
root_block_device {
volume_size = 50
}
depends_on = [
"aws_s3_bucket_object.hivemq_zip_file",
"aws_s3_bucket_object.hivemq_license_file",
"aws_security_group.mqtt",
"aws_s3_bucket_object.influxdb_plugin",
"aws_s3_bucket_object.influxdb_properties_file",
"aws_instance.influxdb",
"aws_s3_bucket_object.telegraf_config"]
provisioner "file" {
source = "scripts/install-hivemq.sh"
destination = "/tmp/install-hivemq.sh"
}
HiveMQ is a product of dc-square GmbH. © dc-square GmbH 2017. All rights reserved.
provisioner "remote-exec" {
inline = [
"chmod +x /tmp/install-hivemq.sh",
"sudo /tmp/install-hivemq.sh ${aws_instance.influxdb.private_ip} ${count.index+1} $
{var.s3_bucket} ${var.region} ${var.broker_heap}"
]
}
connection {
user = "ec2-user"
private_key = "${file("ssh/insecure-deployer")}"
}
resource "aws_instance" "influxdb" {

instance_type = "${var.influxdb_instance_type}"
security_groups = [
tags {
Name = "Influx DB"
}
root_block_device {
volume_size = 200
}
depends_on = [
"aws_s3_bucket_object.grafana_json_file"]
source = "scripts/install-influxdb.sh"
destination = "/tmp/install-influxdb.sh"
}
inline = [
"chmod +x /tmp/install-influxdb.sh",
"sudo /tmp/install-influxdb.sh ${var.s3_bucket} ${var.region}"
]
}
connection {
user = "ec2-user"
}
resource "aws_instance" "client" {
count = "${var.client_count}"

instance_type = "${var.client_instance_type}"
security_groups = [

tags {
Name = "Benchmark Client ${count.index+1}"
}
depends_on = [
"aws_s3_bucket_object.benchmark_tool_jar"]
source = "scripts/install-client.sh"
destination = "/tmp/install-client.sh"
}
inline = [
"chmod +x /tmp/install-client.sh",
"sudo /tmp/install-client.sh ${var.s3_bucket} ${var.region} ${var.client_heap}"
]
}
connection {
user = "ec2-user"
}
output "hivemq" {
value = "${format("\n%v", join("\n",formatlist("%v | %v | ssh -i insecure-deployer ec2-
user@%v | %v", aws_instance.hivemq.*.tags.Name, aws_instance.hivemq.*.public_ip, aws_in-
stance.hivemq.*.public_ip, aws_instance.hivemq.*.private_ip)))}"
}
output "client" {
value = "${format("\n%v", join("\n",formatlist("%v | %v | ssh -i insecure-deployer ec2-
user@%v | %v", aws_instance.client.*.tags.Name, aws_instance.client.*.public_ip, aws_in-
stance.client.*.public_ip, aws_instance.client.*.private_ip)))}"
}
output "influxdb" {
value = "${format("\n%v | %v | ssh -i insecure-deployer ec2-user@%v | %v", aws_in-
stance.influxdb.tags.Name, aws_instance.influxdb.public_ip, aws_instance.influxdb.pub-
lic_ip, aws_instance.influxdb.private_ip)}"
}
output "client_ips_public" {
value = "${format("\n%v", join("\n",formatlist("%v",
aws_instance.client.*.public_ip)))}"
}
output "broker_ips_public" {
aws_instance.hivemq.*.public_ip)))}"
}
output "broker_ips_private" {
aws_instance.hivemq.*.private_ip)))}"
}

resource "aws_s3_bucket_object" "hivemq_zip_file" {
bucket = "${var.s3_bucket}"
key = "hivemq.zip"
source = "${var.hivemq_zip}"
etag = "${md5(file("${var.hivemq_zip}"))}"
}
resource "aws_s3_bucket_object" "hivemq_license_file" {

key = "hivemq.lic"
source = "${var.hivemq_license}"
etag = "${md5(file("${var.hivemq_license}"))}"
}
resource "aws_s3_bucket_object" "influxdb_plugin" {

key = "influxdb-monitoring-plugin.jar"
source = "${var.influxdb_plugin}"
etag = "${md5(file("${var.influxdb_plugin}"))}"
}
resource "aws_s3_bucket_object" "influxdb_properties_file" {

key = "influxdb.properties"
source = "${var.influxdb_properties}"
etag = "${md5(file("${var.influxdb_properties}"))}"
}
resource "aws_s3_bucket_object" "grafana_json_file" {

key = "grafana-dashboard-view.json"
source = "${var.grafana_json_file}"
etag = "${md5(file("${var.grafana_json_file}"))}"
}
resource "aws_s3_bucket_object" "benchmark_tool_jar" {

key = "hivemq-benchmark-tool.jar"
source = "${var.benchmark_tool_jar}"
etag = "${md5(file("${var.benchmark_tool_jar}"))}"
}
resource "aws_s3_bucket_object" "telegraf_config" {

key = "telegraf.conf"
source = "${var.telegraf_config}"
etag = "${md5(file("${var.telegraf_config}"))}"
}
keypairs.tf
resource "aws_key_pair" "deployer" {

key_name = "deployer-key"
public_key = "${file("ssh/insecure-deployer.pub")}"
}

variables.tf
variable "access_key" {
default = "<AWS Access Key>"
}
variable "secret_key" {
default = “<AWS Secret Key>"
}
variable "region" {
default = "us-east-1"
}
variable "amis" {
type = "map"
default = {
eu-central-1 = "ami-d3c022bc"
eu-west-1 = "ami-b0ac25c3"
us-east-1 = "ami-b73b63a0"
}
}
variable "office_ip1" {
default = “<IP allowed to access Grafana and instances>"
}
variable "office_ip2" {
default = “<IP allowed to access Grafana and instances>"
}
variable "broker_count" {
default = 4
}
variable "client_count" {
default = 4
}
variable "broker_instance_type" {
default = "c4.2xlarge"
}
variable "client_instance_type" {
}
variable "influxdb_instance_type" {
}
variable "broker_heap" {
default = "10g"
}
variable "client_heap" {
default = "10g"
}
variable "s3_bucket" {
default = "terraform-hivemq-aws-bucket-us"
}
variable "hivemq_zip" {

default = "files/hivemq.zip"
}
variable "hivemq_license" {
default = "files/hivemq.lic"
}
variable "influxdb_plugin" {
default = "files/influxdb-monitoring-plugin.jar"
}
variable "influxdb_properties" {
default = "files/influxdb.properties"
}
variable "grafana_json_file" {
default = "files/grafana-dashboard-view.json"
}
variable "benchmark_tool_jar" {
default = "files/hivemq-benchmark-tool.jar"
}
variable "telegraf_config" {
default = "files/telegraf.conf"
}
variable "instance_profile" {
default = "s3-download"
}
install-hivemq.sh
#!/bin/bash -ex
HIVEMQ_LOCATION=/opt
HIVEMQ_ZIP_NAME=hivemq.zip
IP=$(curl -ss http://169.254.169.254/latest/meta-data/local-ipv4)
PUBLICIP=$(curl -ss curl http://169.254.169.254/latest/meta-data/public-ipv4)
S3_PLUGIN_ZIP_NAME=s3-cluster-discovery-plugin-1.0.zip
S3_PLUGIN_NAME=s3-cluster-discovery-plugin-1.0
INFLUXDB_PLUGIN_NAME=influxdb-monitoring-plugin.jar
INFLUXDB_PROPERTIES_NAME=influxdb.properties
TELEGRAF_CONFIG=telegraf.conf
INFLUXDB_IP=$1
NODE_INDEX=$2
S3_LOCATION=$3
S3_REGION=$4
BROKER_HEAP=$5
# Install needed software

yum -y update
yum -y install htop ncdu iptraf
yum -y install https://dl.influxdata.com/telegraf/releases/telegraf-1.0.0_rc1.x86_64.rpm
# Increase open file limit

echo "#For HiveMQ Load Testing" >> /etc/sysctl.conf
echo "net.ipv4.tcp_fin_timeout = 30" >> /etc/sysctl.conf
echo "fs.file-max = 5097152" >> /etc/sysctl.conf
echo "net.ipv4.tcp_tw_recycle = 1" >> /etc/sysctl.conf

echo "net.ipv4.tcp_tw_reuse = 1" >> /etc/sysctl.conf
echo "net.core.rmem_default = 524288" >> /etc/sysctl.conf
echo "net.core.wmem_default = 524288" >> /etc/sysctl.conf
echo "net.core.rmem_max = 67108864" >> /etc/sysctl.conf
echo "net.core.wmem_max = 67108864" >> /etc/sysctl.conf
echo "net.ipv4.tcp_rmem = 4096 87380 16777216" >> /etc/sysctl.conf
echo "net.ipv4.tcp_wmem = 4096 65536 16777216" >> /etc/sysctl.conf
echo "* hard nofile 1000000" >> /etc/security/limits.conf

echo "* soft nofile 1000000" >> /etc/security/limits.conf
echo "root hard nofile 1000000" >> /etc/security/limits.conf
echo "root soft nofile 1000000" >> /etc/security/limits.conf
sysctl -p
# Install Oracle JDK 8

wget --quiet -c --header "Cookie: oraclelicense=accept-securebackup-cookie" http://down-
load.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-
linux-x64.rpm
rpm -i jdk-8u131-linux-x64.rpm
/usr/sbin/alternatives --install /usr/bin/java java /usr/java/jdk1.8.0_131/bin/java 20000
/usr/sbin/alternatives --auto java
# Copy hivemq
aws s3 --region $S3_REGION cp s3://$S3_LOCATION/$HIVEMQ_ZIP_NAME $HIVEMQ_LOCATION/
$HIVEMQ_ZIP_NAME
# Install hivemq as service

unzip /opt/$HIVEMQ_ZIP_NAME -d /opt/
rm -f /opt/$HIVEMQ_ZIP_NAME
ln -s /opt/$(ls /opt | grep hivemq-) /opt/hivemq
useradd -s /bin/bash -d /opt/hivemq hivemq
# Remove Systopic plugin

rm -rf /opt/hivemq/plugins/hivemq-sys-topic-plugin-3.0.0.jar
# Download s3 discovery plugin

wget --quiet --content-disposition http://www.hivemq.com/wp-content/uploads/$S3_PLUG-
IN_ZIP_NAME -P /tmp
# Unzip s3 discovery plugin
mkdir /tmp/$S3_PLUGIN_NAME
unzip /tmp/$S3_PLUGIN_ZIP_NAME -d /tmp/$S3_PLUGIN_NAME
rm -f /tmp/$S3_PLUGIN_ZIP_NAME
# Copy s3 discovery plugin

cp /tmp/$S3_PLUGIN_NAME/plugins/$S3_PLUGIN_NAME.jar /opt/hivemq/plugins/.
cp /tmp/$S3_PLUGIN_NAME/conf/s3discovery.properties /opt/hivemq/conf/.
# Copy influx db plugin

aws s3 --region $S3_REGION cp s3://$S3_LOCATION/$INFLUXDB_PLUGIN_NAME /opt/hivemq/plug-
ins/$INFLUXDB_PLUGIN_NAME
# Copy influx db properties

aws s3 --region $S3_REGION cp s3://$S3_LOCATION/$INFLUXDB_PROPERTIES_NAME /opt/hivemq/
conf/$INFLUXDB_PROPERTIES_NAME
# Copy telegraf config

aws s3 --region $S3_REGION cp s3://$S3_LOCATION/$TELEGRAF_CONFIG /etc/telegraf/
$TELEGRAF_CONFIG
# Add influx db ip to properies

sed -i -e "s|host:--INFLUX-DB-IP--|host:$INFLUXDB_IP|" /opt/hivemq/conf/$INFLUXDB_PROPER-
TIES_NAME

sed -i -e "s|tags:host=--NODE-NAME--|tags:host=node$NODE_INDEX|" /opt/hivemq/conf/$IN-
FLUXDB_PROPERTIES_NAME
# Add telegraf db ip to properies

sed -i -e "s|url = \"http://--INFLUX-DB-IP--:8086\"|url = \"http://$INFLUXDB_IP:
8086\"|" /etc/telegraf/$TELEGRAF_CONFIG
sed -i -e "s|node = \"--NODE-NAME--\"|node = \"node$NODE_INDEX\"|" /etc/telegraf/
$TELEGRAF_CONFIG
# Enable cluster in hivemq config

sed -i -e $'s|</hivemq>|<cluster>\\\n<enabled>true</enabled>\\\n<transport>\\\n<tcp>\\
\n<bind-port>7800</bind-port>\\\n<bind-address>--LOCAL_IP--</bind-address>\\\n</tcp>\\
\n</transport>\\\n<discovery>\\\n<plugin>\\\n<reload-interval>10</reload-interval>\\\n</
plugin>\\\n</discovery>\\\n</cluster>\\\n</hivemq>|' /opt/hivemq/conf/config.xml
sed -i -e "s|<bind-address>--LOCAL_IP--</bind-address>|<bind-address>$IP</bind-address>|"
/opt/hivemq/conf/config.xml
# Add aws credentials to cluster config

sed -i -e "s|credentials-type:default|credentials-type:instance_profile_credentials|" /
opt/hivemq/conf/s3discovery.properties
sed -i -e "s|s3-bucket-name:hivemq|s3-bucket-name:$S3_LOCATION|" /opt/hivemq/conf/s3dis-

covery.properties
sed -i -e "s|s3-bucket-region:us-east-1|s3-bucket-region:$S3_REGION|" /opt/hivemq/conf/
s3discovery.properties
# insert license file

aws s3 --region $S3_REGION cp s3://$S3_LOCATION/hivemq.lic /opt/hivemq/license/
# Continue HiveMQ Installation

chown -R hivemq:hivemq /opt/$(ls /opt | grep hivemq-)
chown -R hivemq:hivemq /opt/hivemq
chmod +x /opt/hivemq/bin/run.sh
cp /opt/hivemq/bin/init-script/hivemq /etc/init.d/hivemq
chmod +x /etc/init.d/hivemq
sed -i -e "s|-Djava.net.preferIPv4Stack=true|-Djava.net.preferIPv4Stack=true -Xmx$BRO-

KER_HEAP|" /opt/hivemq/bin/run.sh
service telegraf start

service hivemq start
chkconfig hivemq on
Install-influxdb.sh
#!/bin/bash -ex
S3_LOCATION=$1
S3_REGION=$2
wget --quiet https://dl.influxdata.com/influxdb/releases/influxdb-0.13.0.x86_64.rpm

yum localinstall -y influxdb-0.13.0.x86_64.rpm
#Allow Graphite reporting

#cat <<EOF >> /etc/influxdb/influxdb.conf
#[[graphite]]
# enabled = true
# bind-address = ":2003"
# protocol = "tcp"
#

#EOF
service influxdb restart
sleep 10
curl -i -XPOST http://localhost:8086/query --data-urlencode "q=CREATE DATABASE hivemq"
echo "InfluxDB installed. Graphite Listener is running on TCP port 2003"
echo "Installing Grafana ..."
yum install -y https://grafanarel.s3.amazonaws.com/builds/

grafana-3.1.0-1468321182.x86_64.rpm
service grafana-server restart
echo "Grafana installed. Available on port 3000."

echo "User: admin"
echo "Pass: admin"
chkconfig --add influxdb

chkconfig --add grafana-server
echo "Waiting for Graphana API to become available"
sleep 10
echo "Provisioning Grafana ..."
cat <<EOF > datasource-influxdb.json

{
"name":"InfluxDB",
"type":"influxdb",
"access":"proxy",
"url":"http://localhost:8086",
"password":"",
"user":"",
"database":"hivemq",
"basicAuth":false,
"basicAuthUser":"",
"basicAuthPassword":"",
"withCredentials":false,
"isDefault":true
}
EOF
curl -vX POST http://admin:admin@localhost:3000/api/datasources -d @datasource-influxd-

b.json --header "Content-Type: application/json"
aws s3 --region $S3_REGION cp s3://$S3_LOCATION/grafana-dashboard-view.json /tmp/grafana-

dashboard-view.json
curl -vX POST http://admin:admin@localhost:3000/api/dashboards/db -d @/tmp/grafana-dash-

board-view.json --header "Content-Type: application/json"
install-client.sh
#!/usr/bin/env bash
S3_LOCATION=$1

S3_REGION=$2
HEAP_SIZE=$3
BENCHMARK_JAR=hivemq-benchmark-tool.jar
# Install needed software

yum -y update
yum -y install htop ncdu iptraf
# Copy benchmark tool

aws s3 --region $S3_REGION cp s3://$S3_LOCATION/$BENCHMARK_JAR /home/ec2-user/$BENCH-
MARK_JAR
# Increase open file limit

echo "#For HiveMQ Load Testing" >> /etc/sysctl.conf
echo "net.ipv4.tcp_fin_timeout = 30" >> /etc/sysctl.conf
echo "fs.file-max = 5097152" >> /etc/sysctl.conf
echo "net.ipv4.tcp_tw_recycle = 1" >> /etc/sysctl.conf
echo "net.ipv4.tcp_tw_reuse = 1" >> /etc/sysctl.conf
echo "net.core.rmem_default = 524288" >> /etc/sysctl.conf
echo "net.core.wmem_default = 524288" >> /etc/sysctl.conf
echo "net.core.rmem_max = 67108864" >> /etc/sysctl.conf
echo "net.core.wmem_max = 67108864" >> /etc/sysctl.conf
echo "net.ipv4.tcp_rmem = 4096 87380 16777216" >> /etc/sysctl.conf
echo "net.ipv4.tcp_wmem = 4096 65536 16777216" >> /etc/sysctl.conf
echo "* hard nofile 1000000" >> /etc/security/limits.conf

echo "* soft nofile 1000000" >> /etc/security/limits.conf
echo "root hard nofile 1000000" >> /etc/security/limits.conf
echo "root soft nofile 1000000" >> /etc/security/limits.conf
sysctl -p
sleep 5s
sudo nohup java -Xmx$HEAP_SIZE -jar hivemq-benchmark-tool.jar rest -ba 0.0.0.0 -bp 8080 &
sleep 5s
s3discovery.properties
############################################################
# AWS Credentials #
############################################################ 
#
# Use environment variables to specify your AWS credentials
# the following variables need to be set:
# AWS_ACCESS_KEY_ID
# AWS_SECRET_ACCESS_KEY
#
#credentials-type:environment_variables
#
# Use Java system properties to specify your AWS credentials
# the following variables need to be set:
# aws.accessKeyId
# aws.secretKey
#
#credentials-type:java_system_properties

#
# Uses the credentials file wich can be created by calling 'aws configure' (AWS CLI)
# usually this file is located at ~/.aws/credentials (platform dependent)
# The location of the file can be configured by setting the environment variable
# AWS_CREDENTIAL_PROFILE_FILE to the location of your file
#
#credentials-type:user_credentials_file
#
# Uses the IAM Profile assigned to the EC2 instance running HiveMQ to access S3
# Notice: This only works if HiveMQ is running on an EC2 instance !
#
#credentials-type:instance_profile_credentials
#
# Tries to access S3 via the default mechanisms in the following order
# 1) Environment variables
# 2) Java system properties
# 3) User credentials file
# 4) IAM profiles assigned to EC2 instance
#
credentials-type:default
#
# Uses the credentials specified in this file.
# The variables you must provide are:
# credentials-access-key-id
# credentials-secret-access-key
#
#credentials-type:access_key
#credentials-access-key-id:{access_key_id}
#credentials-secret-access-key:{secret_access_key}
#
# Uses the credentials specified in this file to authenticate with a temporary session
# The variables you must provide are:
# credentials-access-key-id
# credentials-secret-access-key
# credentials-session-token
#
#credentials-type:temporary_session
#credentials-access-key-id:{access_key_id}
#credentials-secret-access-key:{secret_access_key}
#credentials-session-token:{session_token}
############################################################
# S3 Bucket #
############################################################
#
# Region for the S3 bucket used by hivemq
# see http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region for a list of
regions for S3
# example: us-west-2
#
s3-bucket-region:us-east-1
#
# Name of the bucket used by HiveMQ
#
s3-bucket-name:hivemq

#
# Prefix for the filename of every node's file (optional)
#
file-prefix:hivemq/cluster/nodes/
#
# Expiration timeout (in minutes).
# Files with a timestamp older than (timestamp + expiration) will be automatically delet-
ed
# Set to 0 if you do not want the plugin to handle expiration.
#
file-expiration:360
#
# Interval (in minutes) in which the own information in S3 is updated.
# Set to 0 if you do not want the plugin to update its own information.
# If you disable this you also might want to disable expiration.
#
update-interval:180
influxdb.properties
mode:http
host:--INFLUX-DB-IP--
port:8086
protocol:http
auth:
prefix:
database:hivemq
reportingInterval:1
connectTimeout:5000
tags:host=—NODE-NAME--
ControlBenchmarkTool.java
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.nio.charset.StandardCharsets;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
/**
* @author Lukas Brandl
*/
public class ControlBenchmarkTool {
public static void main(String[] args) throws InterruptedException, IOException {
final int connections = 250000;

final int connectsPerSecond = 500;
final int amountOfTopics = 1000;
final DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss");

final String[] brokerPrivateIps = {
“1.2.3.4”,
“2.3.4.5”,
…
40 broker IPs
};
final String[] clientPublicIps = {

“9.8.7.6”,
“8.7.6.5”,
…
40 client IPs
};
final String influxDbIp = “6.7.8.9”; // private ip
if (influxDbIp.isEmpty()) {
throw new IllegalArgumentException(“InfluxDB IP is missing");
}
String topicsString = "";

for (int i = 1; i <= amountOfTopics; i++) {
topicsString += ("&to=topic" + i);
}
String hostString = "";
for (int i = 0; i < brokerPrivateIps.length; i++) {
hostString += ("&h=" + brokerPrivateIps[i]);
}
for (int i = 0; i < clientPublicIps.length; i++) {
String path = "http://" + clientPublicIps[i] +

“:8080/client/subscribe";
String parameter = "cc=" + connectsPerSecond +

"&c=" + connections +
hostString +
topicsString +
"&idh=" + influxDbIp +
"&idt=client" + i +
"&q=2";
byte[] postData = parameter.getBytes(StandardCharsets.UTF_8);

int postDataLength = postData.length;
System.out.println("Sending Request to "

+ clientPublicIps[i] + " "+dateFormat.format(new Date()));
final URL url = new URL(path);

final HttpURLConnection connection = (HttpURLConnection) url.openConnection();
connection.setDoOutput(true);
connection.setInstanceFollowRedirects(false);
connection.setRequestMethod("POST");
connection.setRequestProperty("Content-Type",
"application/x-www-form-urlencoded");
connection.setRequestProperty("charset", "utf-8");
connection.setRequestProperty("Content-Length",
Integer.toString(postDataLength));
connection.setUseCaches(false);
connection.getOutputStream().write(postData);

final BufferedReader reader = new BufferedReader( 
new InputStreamReader(connection.getInputStream()));
reader.close();
}
}
}
BenchmarkPublishTool.java
import org.apache.commons.cli.Options;
import org.apache.commons.lang3.RandomStringUtils;
import org.eclipse.paho.client.mqttv3.MqttClient;
import org.eclipse.paho.client.mqttv3.MqttException;
import org.eclipse.paho.client.mqttv3.persist.MemoryPersistence;
import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.concurrent.CountDownLatch;
/**
* @author Lukas Brandl
*/
public class BenchmarkPublishTool {
private final String[] brokerIps = { //public IP's

“5.6.3.4”,
“8.7.4.5”,
…
40 broker IPs
};
private final double messagesPerSecond = 2;

private final int duration = 1800; // in seconds
private final int amountOfTopics = 1000;
private final int qos = 1;
private final DateFormat dateFormat = new SimpleDateFormat("yyyy/MM/dd HH:mm:ss");
public static void main(final String[] args)  

throws MqttException, InterruptedException
{
new BenchmarkPublishTool().start();
}
private void start() throws MqttException, InterruptedException

{
System.out.println("Start Publishing "+dateFormat.format(new Date()));

final CountDownLatch countDownLatch = new CountDownLatch(brokerIps.length);
for (String brokerIp : brokerIps) {
new Thread(() -> {
try {
final byte[] randomPayload =
RandomStringUtils.random(32, true, true).getBytes();
final MqttClient client = new MqttClient(
"tcp://" + brokerIp + ":" + 1883, "publisher-" + brokerIp,
new MemoryPersistence());
client.connect();
int counter = 0;

final long startTime = System.currentTimeMillis();
while (System.currentTimeMillis() - startTime < duration * 1000)
{
client.publish("topic" + (counter % amountOfTopics + 1),
randomPayload, qos, false);
counter++;
Thread.sleep((long) (1000 / messagesPerSecond));
}
System.out.println("Finished publishing to " + brokerIp+" “
+dateFormat.format(new Date()));
countDownLatch.countDown();
} catch (final Exception e) {
e.printStackTrace();
}
}).start();
}
countDownLatch.await();
}
}

Additional Resources More Information
HiveMQ You can find more information about the
HiveMQ Documentation Enterprise MQTT Broker HiveMQ, plugin
HiveMQ off-the-shelf plugins development and commercial services at
HiveMQ Download http://www.hivemq.com.
White papers Not sure if HiveMQ is the perfect fit for

Clustering HiveMQ your use case? Talk to one of our experts:
[email protected]
HiveMQ Benchmarks
HiveMQ TLS Benchmarks

Hivemq 10 Mio Benchmark PDF

Uploaded by

Copyright:

Available Formats

Hivemq 10 Mio Benchmark PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Hivemq 10 Mio Benchmark PDF

Uploaded by

Copyright:

Available Formats

10,000,000 MQTT Clients

HiveMQ Cluster Benchmark Paper

© dc-square GmbH 2017. All rights reserved. hivemq.com | 2

Exciting times are ahead!

Head of Customer Success

© dc-square GmbH 2017. All rights reserved. hivemq.com | 3

dc-square in one sentence

© dc-square GmbH 2017. All rights reserved. hivemq.com | 4

HiveMQ 3.3.0 is out!

© dc-square GmbH 2017. All rights reserved. hivemq.com | 5

© dc-square GmbH 2017. All rights reserved. hivemq.com | 6 5

Purpose of this benchmark

Goal of this benchmark

The goal is very simple for this particular benchmark:

Demonstrate that HiveMQ can handle 10,000,000 (million)

© dc-square GmbH 2017. All rights reserved. hivemq.com | 7

© dc-square GmbH 2017. All rights reserved. hivemq.com | 8

3. Set up enough Benchmark Tool instances to establish 10 million client connections,

5. Wait for a few minutes

7. Wait for a few minutes

8. Collect all results

9. Shut down the whole test setup and clean up

Deciding on the variables

Payload: As the payload we used a random string with a 32 byte size

© dc-square GmbH 2017. All rights reserved. hivemq.com | 9

Message Rate for Publisher

Summary of the scenario

That means in numbers we have

Deciding on the target platform

© dc-square GmbH 2017. All rights reserved. hivemq.com | 10

© dc-square GmbH 2017. All rights reserved. hivemq.com | 12

Instance Type c4.2xlarge

RAM 15GiB (~16GB)

Physical Processor Intel Xeon E5-2666 v3

Clock Speed (GHz) 2.9

Dedicated EBS Bandwidth 1,000

Operating System Vanilla Amazon Linux

3. Install JDK 8: We used JDK 8u131

6. Configuring HiveMQ for clustering by adding the following to the config.xml

© dc-square GmbH 2017. All rights reserved. hivemq.com | 13

7. Add necessary HiveMQ Plugins

8. Add license file

9. Start HiveMQ and telegraf

service telegraf start

10 Million MQTT Clients

© dc-square GmbH 2017. All rights reserved. hivemq.com | 14

Hardware and Operating System used

Instance Type c4.2xlarge

RAM 15GiB (~16GB)

Physical Processor Intel Xeon E5-2666 v3

Clock Speed (GHz) 2.9

Dedicated EBS Bandwidth 1000

Operating System Vanilla Amazon Linux

echo "net.ipv4.ip_local_port_range = 1024 65535" >> /etc/sysctl.conf

And we are good to go!

© dc-square GmbH 2017. All rights reserved. hivemq.com | 15

Benchmark Tool Parameter (excerpt)

c amount of 250.000 this speciﬁes how many connections each

© dc-square GmbH 2017. All rights reserved. hivemq.com | 16

dc-square in one sentence 

Demonstrate that HiveMQ can handle 10,000,000 (million)  

6. Configuring HiveMQ for clustering by adding the following to the config.xml  

8. Add license file  

9. Start HiveMQ and telegraf 

service telegraf start 

Everything is stable and behaves as expected. 