DEFINITION OF BIG DATA

• Big data is high-velocity and high-variety information assets that

demand cost effective, innovative forms of information processing for
enhanced insight and decision making.
• Big data refers to datasets whose size is typically beyond the storage
capacity of and alsocomplex for traditional database software tools
• Big data is anything beyond the human & technical infrastructure
needed to supportstorage, processing and analysis.
• It is data that is big in volume, velocity and variety. Refer to figure
1.3

Variety: Data can be structured data, semi-structured data and unstructured

data. Data stored in a database is an example of structured data.HTML
data, XML data, email data, CSV files are the examples of semi-structured
data. Power point presentation, images, videos, researches, white papers,
body of email etc are the examples of unstructured data.

Velocity: Velocity essentially refers to the speed at which data is being

created in real- time. We have moved from simple desktop applications
like payroll application to real-time processing applications.

Volume: Volume can be in Terabytes or Petabytes or Zettabytes.

Part I of the definition: "Big data is high-volume, high-velocity, and
high-variety information assets" talks about voluminous data (humongous
data) that may have great variety (a good mix of structured, semi-
structured. and unstructured data) and will require a good speed/pacefo
storage, preparation, processing and analysis.

Part II of the definition: "cost effective, innovative forms of information

processing" talks about embracing new techniques and technologies to
capture (ingest), store, process, persist, integrate and visualize the high-
volume, high-velocity, and high-variety data.
Part III of the definition: "enhanced insight and decision making" talks
about deriving deeper, richer and meaningful insights and then using
these insights to make faster and better decisions to gain business value
and thus a competitive edge.
Data —> Information —> Actionable intelligence —> Better decisions —>Enhanced business
value
CHALLENGES OF BIG DATA

There are mainly seven challenges of big data: scale, security,

schema, Continuous availability, Consistency, Partition tolerant and data
quality.

Scale: Storage (RDBMS (Relational Database Management System) or

NoSQL (Not only SQL)) is one major concern that needs to be addressed
to handle the need for scaling rapidlyand elastically. The need of the hour
is a storage that can best withstand the attack of large volume, velocity
and variety of big data. Should you scale vertically or should you scale
horizontally?

Security: Most of the NoSQL big data platforms have poor security
mechanisms (lack of proper authentication and authorization
mechanisms) when it comes to safeguarding big data. A spot that cannot
be ignored given that big data carries credit card information, personal
information and other sensitive data.
schema: Rigid schemas have no place. We want the technology to be
able to fit our big data and not the other way around. The need of the
hour is dynamic schema. Static (pre-defined schemas) are obsolete.
Continuous availability: The big question here is how to provide 24/7
support because almostall RDBMS and NoSQL big data platforms have a
certain amount of downtime built in.
Consistency: Should one opt for consistency or eventual consistency?
Partition tolerant: How to build partition tolerant systems that can take
care of both hardwareand software failures?
Data quality: How to maintain data quality-data accuracy, completeness,
timeliness, etc.? Dowe have appropriate metadata in place?

Predictive Analytics and Data Mining (Data Science)

Typical Techniques and  Optimization. predictive
Data Types modelling, forecasting. statisticalanalysis
 Structured/unstructured data, many types of
sources, verylarge datasets
Common Questions  What if ... ?
 What's the optimal scenario for our business?
 What will happen next? What if these trends
continue? Whyis this happening?
 Business Intelligence
Typical Techniques and  Standard and ad hoc reporting, dashboards,
Data Types alerts, queries,details on demand
 Structured data. traditional
sources. manageable datasets
Common Questions  What happened last quarter?
 How many units sold?
 Where is the problem? Hey in
which situation?
Table 1.4: BI

KEY ROLES FOR THE NEW BIG DATA ECOSYSTEM

1. Deep Analytical Talent is technically savvy, with strong analytical

skills. Members possess a combination of skills to handle raw,
unstructured data and to apply complex analytical techniques at
massive scales. This group has advanced training in quantitative
disciplines, such as mathematics, statistics, and machine learning. To
do their jobs, members need access to a robust analytic sandbox or
workspace where they can perform large-scale analytical
dataexperiments.

Examples of current professions fitting into this group include

statisticians, economists,mathematicians, and the new role of the Data
Scientist.
2. Data Savvy Professionals-has less technical depth but has a basic
knowledge of statistics or machine learning and can define key
questions that can be answered using advanced analytics.
These people tend to have a base knowledge of working with data, or
an appreciation for some of the work being performed by data
scientists and others with deep analytical talent.
Examples of data savvy professionals include financial analysts,
market research analysts, life scientists, operations managers, and
business and functional managers.

3. Technology and Data Enablers- This group represents people

providing technical expertise to support analytical projects, such as
provisioning and administrating analytical sandboxes, and managing
large-scale data architectures that enable widespread analytics within
companies and other organizations. This role requires skills related to
computer engineering, programming, and databaseadministration.

These three groups must work together closely to solve complex Big
Data challenges.
 Most organizations are familiar with people in the latter two groups
mentioned, but thefirst group, Deep Analytical Talent, tends to be the
newest role for most and the least understood.
Phases of Data Analytics Lifecycle
The seven roles follow:
Business User: Someone who understands the domain area and usually
benefits from the results. This person can consult and advise the project
team on the context of the project, the value of the results, and how the
outputs will be operationalized. Usually a business analyst, line manager,
or deep subject matter expert in the project domain fulfills this role.
Project Sponsor: Responsible for the genesis of the project. Provides the
impetus and requirements for the project and defines the core business
problem. Generally provides the funding and gauges the degree of value
from the final outputs of the working team. This person sets the priorities
for the project and clarifies the desired outputs.
Project Manager: Ensures that key milestones and objectives are met on
time and at the expected quality.
Business Intelligence Analyst: Provides business domain expertise based
on a deep understanding of the data, key performance indicators (KPIs),
key metrics, and business intelligence from a reporting perspective.
Business Intelligence Analysts generally create dashboards and reports and
have knowledge of the data feeds and sources.
Database Administrator (DBA): Provisions and configures the database
environment to support the analytics needs of the working team. These
responsibilities may include providing access to key databases or tables
and ensuring the appropriate security levels are in place related to the data
repositories.
Data Engineer: Leverages deep technical skills to assist with tuning SQL
queries for data management and data extraction, and provides support for
data ingestion into the analytic sandbox, which was discussed in Chapter 1,
“Introduction to Big Data Analytics.” Whereas the DBAsets up and
configures the databases to be used, the data engineer executes the actual
data extractions and performs substantial data manipulation to facilitate the
analytics. The data engineer works closely with the data scientist to help
shape data in the right ways for analyses.
Data Scientist: Provides subject matter expertise for analytical techniques,
data modeling, and applying valid analytical techniques to given business
problems. Ensures overall analytics objectives are met. Designs and
 executes analytical methods and approaches with the data available to the
project.
Although most of these roles are not new, the last two roles—data engineer and
data scientist—have become popular and in high demand [2] as interest in Big
Data has grown.

DIAGNOSTICS OF CLASSIFIERS(confusion matrix)

A confusion matrix is a specific table layout that allows
visualization of the performance of a classifier. Table 7-6 shows the
confusion matrix for a two-class classifier. True positives (TP) are the
number of positive instances the classifier correctly identified as positive.
False positives (FP) are the number of instances in which the classifier
identified as positive but in reality are negative. True negatives (TN) are
the number of negative instances the classifier correctly identified as
negative, False negatives (FN) are the number of instances classified as
negative but in reality are positive. In a two-class classification, a preset
threshold may be used to separate positives from negatives. TP and TN are
the correct guesses. A good classifier should have large TP and TN and
small (ideally zero) numbers for FP and FN.

The accuracy (or the overall success rate) is a metric defining the
rate at which a model has classified the records correctly. It is defined as
the sum of TP and TN divided by the total number of instances, as shown
in Equation 7-18.
A good model should have a high accuracy score, but having a
high accuracy score alone does not guarantee the model is well
established. The true positive rate (TPR) shows what percent of positive
instances the classifier correctly identified. It's also illustrated in Equation
7-19.
 A well-performed model should have a high TPR that is ideally 1
and a low FPR and FNR that are ideally 0. In some cases, a model with a
TPR of 0.95 and an FPR of 0.3 is more acceptable than a model with a
TPR of 0.9 and an FPR of 0.1 even if the second model is more accurate
overall. Precision is the percentage of instances marked positive that really
are positive, as shown in Equation 7-22.

ROC curve is a common tool to evaluate classifiers. The

abbreviation stands for receiver operat ing characteristic, a term used in
signal detection to characterize the trade-off between hit rate and false-
alarm rate over a noisy channel. A ROC curve evaluates the performance
of a classifier based on the TP and FP, regardless of other factors such as
class distribution and error costs.
Related to the ROC curve is the area under the curve (AUC). The
AUC is calculated by measuring the area under the ROC curve. Higher
AUC scores mean the classifier performs better. The score can range from
0.5 (for the diagonal line TPR=FPR) to 1.0 (with ROC passing through the
top-left corner).
4.0 LINEAR REGRESSION

Linear regression is an analytical technique used to model the

relationship between several input variables and a continuous outcome
variable. A key assumption is that the relationship between an input
variable and the outcome variable is linear.
A linear regression model is a probabilistic one that accounts for
the randomness that can affect any particular outcome. Based on known
input values, a linear regression model provides the expected value of the
outcome variable based on the values of the input variables, but some
uncertainty may remain in predicting any particular outcome.

4.0.1 Use Cases:

Linear regression is often used in business, government, and other
scenarios. Some common practical applications of linear regression in the
real world include the following:

 Real estate: A simple linear regression analysis can be used to

model residential home prices as a function of the home's living area. Such
a model helps set or evaluate the list price of a home on the market. The
model could be further improved by including other input variables such
as number of bathrooms, number of bedrooms, lot size, school district
rankings, crime statistics, and property taxes

 Demand forecasting: Businesses and governments can use

linear regression models to predict demand for goods and services. For
example, restaurant chains can appropriately prepare for the predicted type
and quantity of food that customers will consume based upon the weather,
the day of the week, whether an item is offered as a special, the time of
day, and the reservation volume. Similar models can be built to predict
retail sales, emergency room visits, and ambulance dispatches.

 Medical: A linear regression model can be used to analyze the

effect of a proposed radiation treatment on reducing tumor sizes. Input
variables might include duration of a single radiation treatment,
frequency of radiation treatment, and patient attributes such as age or
weight.

Linear Regression Model (Ordinary Least Squares):

In the linear model, the β2s represent the unknown p parameters.

The estimates for these unknown parameters are chosen so that, on
average, the model provides a reasonable estimate of a person's income
7
based on age and education. In other words, the fitted model should
minimize the overall error between the linear model and the actual
observations. Ordinary Least Squares (OLS) is a common technique to
estimate the parameters.

To illustrate how OLS works, suppose there is only one input

variable, x, for an outcome variable y. Furthermore, n observations of (x,
y) are obtained and plotted in below Figure.

Figure 4.1 Scatterplot of y versus x

The goal is to find the line that best approximates the relationship
between the outcome variable and the input variables. With OLS, the
objective is to find the line through these points that minimizes the sum of
the squares of the difference between each point and the line in the vertical
direction. In other words, find the values of β0 and β1, such that the
summation shown in Equation is minimized.

n
∑ y    x 

i 1  i 0 1 i 

The n individual distances to be squared and then summed are

illustrated in below figure. The vertical lines represent the distance
between each observed y value and the line y = β0 + β1x1

2
 Figure 4.2: Scatterplot of y versus x with vertical distance from the
observed points to a fitted line

Linear Regression Model (with Normally Distributed Errors):

In the normal model description, there were no assumptions made
about the error term; no additional assumptions were necessary for OLS to
provide estimates of the model parameters. However, in most linear
regression analyses, it is common to assume that the error term is a
normally distributed random variable with mean equal to zero and
constant variance. Thus, the linear regression model is expressed as shown
in Equation.
Y = β0 + β1x1 + β2 x 2.......................................+ βp-1 x p-1 + ↋

where:
y is the outcome variable
xj are the input variables, for j=1,2,...,p-1
β0 is the value of y when each xj equals zero
βj is the change in y based on a unit change in xj for j=1,2,...,p-1
↋ ~N(0,σ ) and the s are independent of each other
↋

Thus, for a given (x1, x2,... xp-1), y is normally distributed with

mean β0 + β1x1 + β2 x 2………. + βp-1 x p-1 and variance σ2. For a
regression model with just one input variable, below figure illustrates the
9

2
normality assumption on the error terms and the effect on the outcome
variable, y, for a given value of x.

Figure 4.3: Normal distribution about y for a given value of x

4.1 LOGISTIC REGRESSION

4.1.1 Use Cases:

The logistic regression model is applied to a variety of situations in
both the public and the private sector.Some common ways that the logistic
regression model is used include the following:

 Medical: Develop a model to determine the likelihood of a patient's

successful response to a specific medical treatment or procedure. Input
variables could include age, weight, blood pressure, and cholesterol levels.

 Finance: Using a loan applicant's credit history and the details

on the loan, determine the probability that an applicant will default on the
loan. Based on the prediction, the loan can be approved or denied, or the
terms can be modified.

 Marketing: Determine a wireless customer's probability of

switching carriers (known as churning) based on age, number of family
members on the plan, months remaining on the existing contract, and
social network
contacts. With such insight, target the high-probability customers with
appropriate offers to prevent churn.

10
 Engineering: Based on operating conditions and various diagnostic
measurements, determine the probability of a mechanical part
experiencing a malfunction or failure. With this, probability estimate,
schedule the appropriate preventive maintenance activity.

11
12