What is Apache Pig?

Apache Pig is an abstraction over MapReduce. It is a tool/platform which is used to analyze

larger sets of data representing them as data flows. Pig is generally used with Hadoop; we can
perform all the data manipulation operations in Hadoop using Apache Pig.

To write data analysis programs, Pig provides a high-level language known as Pig Latin. This
language provides various operators using which programmers can develop their own functions
for reading, writing, and processing data.

To analyze data using Apache Pig, programmers need to write scripts using Pig Latin language.
All these scripts are internally converted to Map and Reduce tasks. Apache Pig has a component
known as Pig Engine that accepts the Pig Latin scripts as input and converts those scripts into
MapReduce jobs.

Why Do We Need Apache Pig?

Programmers who are not so good at Java normally used to struggle working with Hadoop,
especially while performing any MapReduce tasks. Apache Pig is a boon for all such
programmers.

 Using Pig Latin, programmers can perform MapReduce tasks easily without having to
type complex codes in Java.
 Apache Pig uses multi-query approach, thereby reducing the length of codes. For
example, an operation that would require you to type 200 lines of code (LoC) in Java can
be easily done by typing as less as just 10 LoC in Apache Pig. Ultimately Apache Pig
reduces the development time by almost 16 times.
 Pig Latin is SQL-like language and it is easy to learn Apache Pig when you are familiar
with SQL.
 Apache Pig provides many built-in operators to support data operations like joins, filters,
ordering, etc. In addition, it also provides nested data types like tuples, bags, and maps that
are missing from MapReduce.
Features of Pig
Apache Pig comes with the following features −

 Rich set of operators − It provides many operators to perform operations like join, sort,
filer, etc.
 Ease of programming − Pig Latin is similar to SQL and it is easy to write a Pig script if
you are good at SQL.
 Optimization opportunities − The tasks in Apache Pig optimize their execution
automatically, so the programmers need to focus only on semantics of the language.
 Extensibility − Using the existing operators, users can develop their own functions to read,
process, and write data.
 UDF’s − Pig provides the facility to create User-defined Functions in other programming
languages such as Java and invoke or embed them in Pig Scripts.
 Handles all kinds of data − Apache Pig analyzes all kinds of data, both structured as well
as unstructured. It stores the results in HDFS.
Apache Pig Vs MapReduce

Listed below are the major differences between Apache Pig and MapReduce.

Apache Pig MapReduce

Apache Pig is a data flow language. MapReduce is a data processing

paradigm.

It is a high level language. MapReduce is low level and rigid.

Performing a Join operation in Apache Pig It is quite difficult in MapReduce

is pretty simple. to perform a Join operation
between datasets.

Any novice programmer with a basic Exposure to Java is must to work

knowledge of SQL can work conveniently with MapReduce.
with Apache Pig.

Apache Pig uses multi-query approach, MapReduce will require almost 20

thereby reducing the length of the codes to times more the number of lines to
 a great extent. perform the same task.

There is no need for compilation. On MapReduce jobs have a long

execution, every Apache Pig operator is compilation process.
converted internally into a MapReduce job.

Apache Pig Vs SQL

Listed below are the major differences between Apache Pig and SQL.

Pig SQL

Pig Latin is a procedural language. SQL is a declarative language.

In Apache Pig, schema is optional. We can Schema is mandatory in SQL.

store data without designing a schema (values
are stored as $01, $02 etc.)

The data model in Apache Pig is nested The data model used in SQL is
relational. flat relational.

Apache Pig provides limited opportunity There is more opportunity for

for Query optimization. query optimization in SQL.

In addition to above differences, Apache Pig Latin −

 Allows splits in the pipeline.

 Allows developers to store data anywhere in the pipeline.
 Declares execution plans.
 Provides operators to perform ETL (Extract, Transform, and Load) functions.
Apache Pig Vs Hive

Both Apache Pig and Hive are used to create MapReduce jobs. And in some cases, Hive operates
on HDFS in a similar way Apache Pig does. In the following table, we have listed a few
significant points that set Apache Pig apart from Hive.
 Apache Pig Hive

Apache Pig uses a language called Pig Hive uses a language called HiveQL.
Latin. It was originally created It was originally created at Facebook.
at Yahoo.

Pig Latin is a data flow language. HiveQL is a query processing

language.

Pig Latin is a procedural language and HiveQL is a declarative language.

it fits in pipeline paradigm.

Apache Pig can handle structured, Hive is mostly for structured data.
unstructured, and semi-structured data.

Applications of Apache Pig

Apache Pig is generally used by data scientists for performing tasks involving ad-hoc processing
and quick prototyping. Apache Pig is used −

 To process huge data sources such as web logs.

 To perform data processing for search platforms.
 To process time sensitive data loads.

Apache Pig – History

In 2006, Apache Pig was developed as a research project at Yahoo, especially to create and
execute MapReduce jobs on every dataset. In 2007, Apache Pig was open sourced via Apache
incubator. In 2008, the first release of Apache Pig came out. In 2010, Apache Pig graduated as
an Apache top-level project.

What is Hive

Hive is a data warehouse infrastructure tool to process structured data in Hadoop. It resides on
top of Hadoop to summarize Big Data, and makes querying and analyzing easy.
Initially Hive was developed by Facebook, later the Apache Software Foundation took it up and
developed it further as an open source under the name Apache Hive. It is used by different
companies. For example, Amazon uses it in Amazon Elastic MapReduce.

Hive is not

A relational database

A design for OnLine Transaction Processing (OLTP)

A language for real-time queries and row-level updates

Features of Hive

It stores schema in a database and processed data into HDFS.

It is designed for OLAP.

It provides SQL type language for querying called HiveQL or HQL.

It is familiar, fast, scalable, and extensible.

Architecture of Hive

The following component diagram depicts the architecture of Hive:

 HBase - Overview
What is HBase?

HBase is a distributed column-oriented database built on top of the Hadoop file system. It is an
open-source project and is horizontally scalable.

HBase is a data model that is similar to Google’s big table designed to provide quick random
access to huge amounts of structured data. It leverages the fault tolerance provided by the
Hadoop File System (HDFS).

It is a part of the Hadoop ecosystem that provides random real-time read/write access to data in
the Hadoop File System.

One can store the data in HDFS either directly or through HBase. Data consumer reads/accesses
the data in HDFS randomly using HBase. HBase sits on top of the Hadoop File System and
provides read and write access.

HBase and HDFS

HDFS HBase

HDFS is a distributed file system HBase is a database built on top of the HDFS.
suitable for storing large files.

HDFS does not support fast individual HBase provides fast lookups for larger tables.
record lookups.
It provides high latency batch It provides low latency access to single rows from billions of
processing; no concept of batch records (Random access).
processing.

It provides only sequential access of HBase internally uses Hash tables and provides random access,
data. and it stores the data in indexed HDFS files for faster lookups.

What is Apache Cassandra?

Apache Cassandra is an open source, distributed and decentralized/distributed storage system

(database), for managing very large amounts of structured data spread out across the world. It
provides highly available service with no single point of failure.

Listed below are some of the notable points of Apache Cassandra −

 It is scalable, fault-tolerant, and consistent.

 It is a column-oriented database.
 Its distribution design is based on Amazon’s Dynamo and its data model on Google’s
Bigtable.
 Created at Facebook, it differs sharply from relational database management systems.
 Cassandra implements a Dynamo-style replication model with no single point of failure,
but adds a more powerful “column family” data model.
 Cassandra is being used by some of the biggest companies such as Facebook, Twitter,
Cisco, Rackspace, ebay, Twitter, Netflix, and more.
Features of Cassandra

Cassandra has become so popular because of its outstanding technical features. Given below are
some of the features of Cassandra:

 Elastic scalability − Cassandra is highly scalable; it allows to add more hardware to

accommodate more customers and more data as per requirement.
 Always on architecture − Cassandra has no single point of failure and it is continuously
available for business-critical applications that cannot afford a failure.
 Fast linear-scale performance − Cassandra is linearly scalable, i.e., it increases your
throughput as you increase the number of nodes in the cluster. Therefore it maintains a
quick response time.
 Flexible data storage − Cassandra accommodates all possible data formats including:
structured, semi-structured, and unstructured. It can dynamically accommodate changes to
your data structures according to your need.
 Easy data distribution − Cassandra provides the flexibility to distribute data where you
need by replicating data across multiple data centers.
 Transaction support − Cassandra supports properties like Atomicity, Consistency,
Isolation, and Durability (ACID).
 Fast writes − Cassandra was designed to run on cheap commodity hardware. It performs
blazingly fast writes and can store hundreds of terabytes of data, without sacrificing the
read efficiency.