SNS COLLEGE OF ENGINEERING

Kurumbapalayam (Po), Coimbatore – 641 107

AN AUTONOMOUS INSTITUTION
Accredited by NBA – AICTE and Accredited by NAAC – UGC with ‘A’ Grade
Approved by AICTE, New Delhi & Affiliated to Anna University, Chennai
DEPARTMENT OF INFORMATION TECHNOLOGY

BIG DATA ANALYTICS

INTRODUCTION
What is big data?

Big data is high-volume, high-velocity and high-variety information assets that demand cost-
effective, innovative forms of information processing for enhanced insight and decision making.

Big data is a term applied to data sets whose size or type is beyond the ability of traditional
relational databases to capture, manage and process the data with low latency.

Big data is larger, more complex data sets, especially from new data sources. These data sets are
so voluminous that traditional data processing software just can’t manage them.

SOURCES OF BIG DATA

This data comes from a wide variety of sources: sensors used to gather climate information, posts
to social media sites, digital pictures and videos, purchase transaction records and cell phone
GPS signals, to name a few.

Artificial intelligence (AI), Mobile, Social Media and the Internet of Things (IoT) are driving
data complexity through new forms and sources of data.

For example, big data comes from Sensors, Devices, Video/Audio, Networks, Log files,
Transactional applications, Web, and Social media — much of it generated in real time and at a
very large scale.

APPLICATIONS OF BIG DATA

Healthcare
– Predictive Analysis
– Tele-medicine
– Fitness wearables
– Remote Monitoring
• Banking
– ATM’s
– E-Wallets
• Stock Market
• Education / Academics
• Manufacturing
• E-Commerce
• IT
• Government
• Social Media

TYPES OF BIG DATA

1.Structured Data
It owns a dedicated data model. It also has a well defined structure, it follows a consistent order
and it is designed in such a way that it can be easily accessed and used by person or a computer.
Structured data is usually stored in well defined columns and databases.
– Structured Schema
– Tables with rows and columns of data
– Example : DBMS,RDBMS

2.Semi-Structured Data
It is considered as another form of structured data. It inherits few properties of structured data,
but major parts of this kind of data failures to have a definitive structure and also it does not obey
the formal structure of data models such as RDBMS.
– Schema is not defined properly
– JSON, XML, CSV,RSS
– Ex: Transactional history file, Log file

3.Unstructured Data
Unstructured data is completely different of which neither has a structure nor obeys to follow
formal structural rules of data models. It does not even have a consistent format and it found to
be varying all the time. But rarely it has information related to data and time.
– Heterogeneous Data
– Text file, Images, Videos, Audio.

BIG DATA CHARACTERISTICS

Big data characteristics are mere word that explain the remarkable potential of big data. In early
stages development of big data and related terms there were only 3 V’s (Volume, Variety,
Velocity) considered as potential characteristics.
But ever growing technology and tools and variety of sources where information being received
has potentially increased these 3 V’s into 5 V’s and still evolving.
The 5 V’s are
 Volume
 Variety
 Velocity
 Veracity
 Value

I.Volume
 Volume refers to the unimaginable amounts of information generated every second. This
information comes from variety of sources like social media, cell phones, sensors,
financial records, stock market etc.

 Enterprises are awash with ever- growing data of all types, easily amassing terabytes and
even peta bytes of information. Data has grown exponentially over the past few years
than in the past few decades. Social media, web portals and real time data using sensors
have increased the amount of data

 We are currently using distributed systems, to store data in several locations and brought
together by a software Framework like Hadoop.

 For example, Facebook alone can generate about billion messages, 4.5 billion times that
the “like” button is recorded, and over 350 million new posts are uploaded each day.
Such a huge amount of data can only be handled by Big Data Technologies.

II.Variety
 Variety refers to the many types of data that are available. A reason for rapid growth of
data volume is that the data is coming from different sources in various formats.

 Big data extends beyond structured data to include unstructured data of all varieties: text,
sensor data, audio, video, click streams, log files and more.

The variety of data is categorized as follows:

o Structured – RDBMS
o Semi Structured – XML, HTML, RDF, JSON
o Unstructured- Text, audio, video, logs, images
III.Velocity
 Velocity is the fast rate at which data is received and (perhaps) acted on. In other words
it is the speed at which the data is generated and processed to meet the demands and
challenges that lie in the path of growth and development.

 For time-sensitive processes such as catching fraud, big data must be analyzed as it
streams into the enterprise to maximize its business value.

 Climate control, Health care, Stock exchange predictions, Banking transactions and IoT
generates billions of data every second.

IV.Veracity
 Data veracity, in general, is how accurate or truthful a data set may be. More specifically,
when it comes to the accuracy of big data, it’s not just the quality of the data itself but
how trustworthy the data source, type, and processing of it is.

 The data quality of captured data can vary greatly, affecting the accurate analysis.

V.Value
 Value is the major issue that we need to concentrate on. It is not just the amount of data
that we store or process. It is actually the amount of valuable, reliable and trustworthy
data that needs to be stored, processed, analyzed to find insights.

 Mine the data, i.e., a process to turn raw data into useful data. Value represents benefits
of data to your business such as in finding out insights, results, etc. which were not
possible earlier.

VALIDATING BIG DATA

 Validation is factors considered before Adopting Big Data Technology. Big data may
helpful to an organization but without proper validation it may go wrong for organization
benefits.

 In general just because big data is feasible within the organization, it does not necessarily
mean that it is reasonable.

 Therefore clear value proposition for business improvements needs to be identified.

Unless there are clear processes for determining the value proposition, there is a risk that
it will remain a fad until it hits the disappointment phase of the hype cycle.

 As a way to properly ground any initiatives around big data, one initial task would be to
evaluate the organization’s fitness such as feasibility, reasonability, value, integrability,
and sustainability.
  Validation done with some framework that determines a score for each of these factors
rangingfrom 0 (lowest level) to 4 (highest level).

 The resulting scores can be reviewed. Each of these variables is, for the most part,
somewhat subjective, but there are ways of introducing a degree of objectivity, especially
when considering the value of big data.

THE PROMOTION OF THE VALUE OF BIG DATA

A thoughtful approach must differentiate between hype and reality, and one way to do this is to
review the difference between what is being said about big data and what is being done with big
data.

A scan of existing content on the “value of big data” sheds interesting light on what is being
promoted as the expected result ofbig data analytics and, more interestingly, how familiar those
expectations sound.

A good example is provided within an economic study on the value of big data undertaken and
published by the Center for Economics and Business Research (CEBR) that speaks to the
cumulative value of:

 optimized consumer spending as a result of improved targeted customer marketing;

 improvements to research and analytics within the manufacturing sectors to lead to new
product development;

 improvements in strategizing and business planning leading to innovation and new start-
up companies;

 predictive analytics for improving supply chain management to optimize stock

management, replenishment, and forecasting;

 improving the scope and accuracy of fraud detection.

Curiously, these are exactly the same types of benefits promoted by business intelligence and
data warehouse tools vendors and system integrators for the past 15 to 20 years, namely:
 Better targeted customer marketing
 Improved product analytics
 Improved business planning
 Improved supply chain management
 Improved analysis for fraud, waste, and abuse
BIG DATA USECASES
Big data techniques can be used to leverage the business benefits and by increasing the value of
an organization. Big data has beneficial in many applications and in general the following are the
common categories. It is derived from The Apache Software Foundation’s Powered By Hadoop
Web site.

 Business intelligence, querying, reporting, searching, including many implementation of

searching, filtering, indexing, speeding up aggregation for reporting and for report
generation, trend analysis, search optimization, and general information retrieval.

 Improved performance for common data management operations, with the majority
focusing on log storage, data storage and archiving, followed by sorting, running joins,
extraction/transformation/ loading (ETL) processing, other types of data conversions, as
well as duplicate analysis and elimination.

 Non-database applications, such as image processing, text processing in preparation for

publishing, genome sequencing, protein sequencing and structure prediction, web
crawling, and monitoring workflow processes.

 Data mining and analytical applications, including social network analysis, facial
recognition, profile matching, other types of text analytics, web mining, machine
learning, information extraction, personalization and recommendation analysis, ad
optimization, and behavior analysis.

In turn, the core capabilities that are implemented using the big data application can be further
abstracted into more fundamental categories:

 Counting functions applied to large bodies of data that can be segmented and distributed
among a pool of computing and storage resources, such as document indexing, concept
filtering, and aggregation (counts and sums).

 Scanning functions that can be broken up into parallel threads, such as sorting, data
transformations, semantic text analysis, pattern recognition, and searching.

 Modeling capabilities for analysis and prediction.

 Storing large datasets while providing relatively rapid access.

Generally, Processing applications can combine these core capabilities in different ways.
CHARACTERISTICS OF BIG DATA APPLICATIONS
The big data approach is mostly suited to addressing or solving business problems that are
subject to one or more of the following criteria:

1) Data throttling: The business challenge has an existing solutions, but on traditional
hardware, the performance of a solution is throttled as a result of data accessibility, data
latency, data availability, or limits on bandwidth in relation to the size of inputs.

2) Computation-restricted throttling: There are existing algorithms, but they are heuristic
and have not been implemented because the expected computational performance has not
been met with conventional systems.

3) Large data volumes: The analytical application combines a multitude of existing large
datasets and data streams with high rates of data creation and delivery.

4) Significant data variety: The 11data in the different sources vary in structure and content,
and some (or much) of the data is unstructured.

5) Benefits from data parallelization: Because of the reduced data dependencies, the
application’s runtime can be improved through task or thread-level parallelization applied
to independent data segments.

These criteria can be used to assess the degree to which business problems are suited to big data
technology.

PERCEPTION AND QUANTIFICATION OF VALUE

So far we have looked at two facets of the appropriateness of big data, with the first being
organizational fitness and the second being suitability of the business challenge.

The third facet must also be folded into the equation, and that is big data’s contribution to the
organization.

In essence, these facets drill down into the question of value and whether using big data
significantly contributes to adding value to the organization by:

A. Increasing revenues: As an example, an expectation of using a recommendation engine

would be to increase same-customer sales by adding more items into the market basket.

B. Lowering costs: As an example, using a big data platform built on commodity hardware
for ETL would reduce or eliminate the need for more specialized servers used for data
staging, thereby reducing the storage footprint and reducing operating costs.

C. Increasing productivity: Increasing the speed for the pattern analysis and matching done
for fraud analysis helps to identify more instances of suspicious behavior faster, allowing
 for actions to be taken more quickly and transform the organization from being focused
on recovery of funds to proactive prevention of fraud.

D. Reducing risk: Using a big data platform or collecting many thousands of streams of
automated sensor data can provide full visibility into the current state of a power grid, in
which unusual events could be rapidly investigated to determine if a risk of an imminent
outage can be reduced.

UNDERSTANDING BIG DATA STORAGE

TWO APPROACHES CONSIDERED FOR BIG DATA

Some algorithms expect that massive amounts of data are immediately available quickly,
necessitating large amounts of core memory.

Other applications may need numerous iterative exchanges of data between different computing
nodes, which would require high-speed networks.

STORAGE CONSIDERATIONS
The need to acquire and manage massive amounts of data suggests a need for specialty storage
systems to accommodate the big data applications.

When evaluating specialty storage offerings, some variables to consider include:

 Scalability- Whether storage subsystem can support massive data volumes of increasing
size.

 Extensibility - how flexible the storage system’s architecture is in allowing the system to
be grown without the constraint of artificial limits.

 Fault tolerance, which imbues the storage environment with the capability to recover
from intermittent failures.

 Integratability, which measures how well the storage environment can be integrated into
the production environment.

 Accessibility - limitations or constraints in providing simultaneous access to an

expanding user community without compromising performance

 High-speed I/O capacity, which measures whether the input/output channels can satisfy
the demanding timing requirements for absorbing, storing, and sharing large data
volumes.

All big data applications achieve their performance and scalability through deployment on a
collection of storage and computing resources.
The ability to design, develop, and implement a big data application is directly dependent on an
awareness of the architecture of the underlying computing platform, It includes the following
four resources,

1. Processing capability, often referred to as a CPU, processor, or node. Modern processing

nodes often incorporate multiple cores that are individual CPUs that share the node’s
memory and are managed and scheduled together, allowing multiple tasks to be run
simultaneously; this is known as multithreading.

2. Memory, which holds the data that the processing node is currently working on. Most
single node machines have a limit to the amount of memory.

3. Storage, providing persistence of data—the place where datasets are loaded, and from
which the data is loaded into memory to be processed.

4. Network, which provides the “pipes” through which datasets are exchanged between
different processing and storage nodes.

Because single-node computers are limited in their capacity, they cannot easily accommodate
massive amounts of data.

That is why the high-performance platforms are composed of collections of computers in which
the massive amounts of data and requirements for processing can be distributed among a pool of
resources.

A GENERAL OVERVIEW OF HIGH-PERFORMANCE ARCHITECTURE

Most high-performance platforms are created by connecting multiple nodes together via a variety
of network topologies.

The general architecture distinguishes the management of computing resources (and

corresponding allocation of tasks) and the management of the data across the network of storage
nodes. It is depicted in following image. It is generally called as master / slave architecture.

In this configuration, a master job manager oversees the pool of processing nodes, assigns tasks,
and monitors the activity.

At the same time, a storage manager oversees the data storage pool and distributes datasets
across the collection of storage resources
While there is no apriori requirement that there be any colocation of data and processing tasks, it
is beneficial from a performance perspective to ensure that the threads process data that is local,
or close to minimize the costs of data access latency.

HDFS
Hadoop Distributed File System (HDFS): A distributed file system that provides high throughput
access to application data.

 The Hadoop Distributed File System (HDFS) is based on the Google File System (GFS)
and provides a distributed file system that is designed to run on large clusters (thousands
of computers) of small computer machines in a reliable, fault-tolerant manner.

 Unlike other distributed systems, HDFS is highly fault tolerant and designed using low
cost hardware.

 HDFS holds very large amount of data and provides easier access. To store such huge
data, the files split into blocks are stored across multiple machines.

 These files are stored in redundant fashion to rescue the system from possible data losses
in case of failure.
HDFS Architecture
HDFS uses a master/slave architecture where master consists of a single Name Node that
manages the file system metadata and one or more slave. Data Nodes that store the actual data.

Name Node
The name node is the commodity hardware that contains the GNU/Linux operating system and
the name node software. It is a software that can be run on commodity hardware. The system
having the name node acts as the master server and it does the following tasks:
 Manages the file system namespace.
 Stores metadata for the files, like the directory structure of a typical File System.
 Regulates client’s access to files.
 It also executes file system operations such as renaming, closing, and opening files and
directories. It also determines the mapping of blocks to Data Nodes.

Data Node:
The data node is a commodity hardware having the GNU/Linux operating system and data node
software. These nodes manage the data storage of their system.
 A file in an HDFS namespace is split into several blocks and those blocks are stored in a
set of Data Nodes.
 Data nodes store and retrieve blocks when they are requested by client or name node.
 They report back to name node periodically, with list of blocks that they are storing.
 The data node also perform operations such as block creation, deletion and replication as
stated by the name node.
Block
 Generally the user data is stored in the files of HDFS. The file in a file system will be
divided into one or more segments and/or stored in individual data nodes.
 These file segments are called as blocks. In other words, the minimum amount of data
that HDFS can read or write is called a Block.
 The default block size is 64MB, but it can be increased as per the need to change in
HDFS configuration.

Goals of HDFS
 Fault detection and recovery: Since HDFS includes a large number of commodity
hardware, failure of components is frequent. Therefore HDFS should have mechanisms
for quick and automatic fault detection and recovery.
 Huge datasets: HDFS should have hundreds of nodes per cluster to manage the
applications having huge datasets.
 Hardware at data: A requested task can be done efficiently, when the computation takes
place near the data. Especially where huge datasets are involved, it reduces the network
traffic and increases the throughput.

Features of HDFS
 It is suitable for the distributed storage and processing.
 Hadoop provides a command interface to interact with HDFS.
 The built-in servers of namenode and datanode help users to easily check the status of
cluster.
 Streaming access to file system data.
 HDFS provides file permissions and authentication.

Fault Tolerance
HDFS provides performance through distribution of data and fault tolerance through replication.

Failure management can be implemented in HDFS through

 Monitoring: There is a continuous “heartbeat” communication between the data nodes to
the name node. If a data node’s heartbeat isnot heard by the name node, the data node is
considered to have failed and is no longer available. In this case, a replica is employed to
replace the failed node, and a change is made to the replication scheme.

 Rebalancing: This is a process of automatically migrating blocks of data from one data
node to another when there is free space, when there is an increased demand for the data
and moving it may improve performance (such as moving from a traditional disk drive to
a solid-state drive that is much faster or can accommodate increased numbers of
 simultaneous accesses), or an increased need to replication in reaction to more frequent
node failures.

 Managing integrity: HDFS uses checksums, which are effectively “digital signatures”
associated with the actual data stored in a file(often calculated as a numerical function of
the values within the bits of the files) that can be used to verify that the data stored
corresponds to the data shared or received. When the checksum calculated for a retrieved
block does not equal the stored checksum of that block, it is considered an integrity error.
In that case, the requested block will need to be retrieved from a replica instead.

 Metadata replication: The metadata files are also subject to failure, and HDFS can be
configured to maintain replicas of the corresponding metadata files to protect against
corruption.

 Snapshots: This is incremental copying of data to establish a point in time to which the
system can be rolled back.

MAPREDUCE
 MapReduce is a programming model for writing applications that can process Big Data
in parallel on multiple nodes. MapReduce provides analytical capabilities for
analyzinghuge volumes of complex data.
 MapReduce is a processing technique and a program model for distributed computing
based on java.
 The MapReduce algorithm contains two important tasks, namely Map and Reduce.
 The Map task takes a set of data and converts it into another set of data, where individual
elements are broken down into tuples (key-value pairs).
 The Reduce task takes the output from the Map as an input and combines those data
tuples (key-value pairs) into a smaller set of tuples.
 The major advantage of MapReduce is that it is easy to scale data processing over
multiple computing nodes. Under the MapReduce model, the data processing primitives
are called mappers and reducers.

The Algorithm
 MapReduce program executes in three stages, namely map stage, shuffle stage, and
reduce stage.
 Map stage : The map or mapper’s job is to process the input data. Generally the input
data is in the form of file or directory and is stored in the Hadoop file system (HDFS).
The input file is passed to the mapper function line by line. The mapper processes the
data and creates several small chunks of data.
  Reduce stage: This stage is the combination of the Shuffle stage and the Reduce stage.
The Reducer’s job is to process the data that comes from the mapper. After processing, it
produces a new set of output, which will be stored in the HDFS.

Phases
 Input Phase − Here we have a Record Reader that translates each record in an input file
and sends the parsed data to the mapper in the form of key-value pairs.
 Map − Map is a user-defined function, which takes a series of key-value pairs and
processes each one of them to generate zero or more key-value pairs.
 Intermediate Keys − They key-value pairs generated by the mapper are known as
intermediate keys.
 Combiner − A combiner is a type of local Reducer that groups similar data from the map
phase into identifiable sets. It takes the intermediate keys from the mapper as input and
applies a user-defined code to aggregate the values in a small scope of one mapper. It is
not a part of the main MapReduce algorithm; it is optional.
 Shuffle and Sort − The Reducer task starts with the Shuffle and Sort step. It downloads
the grouped key-value pairs onto the local machine, where the Reducer is running. The
individual key-value pairs are sorted by key into a larger data list. The data list groups the
equivalent keys together so that their values can be iterated easily in the Reducer task.
 Reducer − The Reducer takes the grouped key-value paired data as input and runs a
Reducer function on each one of them. Here, the data can be aggregated, filtered, and
combined in a number of ways, and it requires a wide range of processing. Once the
execution is over, it gives zero or more key-value pairs to the final step.
 Output Phase − In the output phase, we have an output formatter that translates the final
key-value pairs from the Reducer function and writes them onto a file using a record
writer.
The MapReduce framework consists of a single master JobTracker and one slave TaskTracker
per cluster-node.
 The master Job Tracker is responsible for resource management, tracking resource
consumption/availability and scheduling the jobs component tasks on the slaves,
monitoring them and re-executing the failed tasks.
 The slaves TaskTracker execute the tasks as directed by the master and provide task-
status information to the master periodically.
There are three limitations of MapReduce
1.Network Latency
2.Does not support all applications
3.Processor Underutilization

YARN

 Hadoop Yet Another Resource Negotiator (YARN): This is a framework for job
scheduling and cluster resource management. A resource management framework for
scheduling and handling resource requests from distributed applications.

 Issues in MapReduce are addressed in future versions of Hadoop through the segregation of
duties within a revision called YARN.

 In this approach, overall resource management has been centralized while management of
resources at each node is now performed by a local Node Manager.

 In addition, there is the concept of an Application Master that is associated with each application
that directly negotiates with the central Resource Manager for resources while taking over the
responsibility for monitoring progress and tracking status.

 Pushing this responsibility to the application environment allows greater flexibility in the
assignment of resources as well as be more effective in scheduling to improve node utilization.

 Last, the YARN approach allows applications to be better aware of the data allocation across the
topology of the resources within a cluster.

 This awareness allows for improved colocation of compute and data resources, reducing data
motion, and consequently, reducing delays associated with data access latencies. The result
should be increased scalability and performance.
THE MAPREDUCE PROGRAMMING MODEL

 Application development in MapReduce is a combination of the familiar

procedural/imperative approaches used by Java or C++ programmers embedded within
what is effectively a functional language programming model such as the one used within
languages like Lisp and APL.
 A MapReduce application is envisioned as a series of basic operations applied in a sequence to
small sets of many (millions, billions, or even more) data items.
 These data items are logically organized in a way that enables the MapReduce execution model to
allocate tasks that can be executed in parallel.
 The data items are indexed using a defined key into ,key, value. pairs, in which the key represents
some grouping criterion associated with a computed value.
 Combining both data and computational independence means that both the data and the
computations can be distributed across multiple storage and processing units and automatically
parallelized.