Modern big data processing with Hadoop : expert techniques for architecting end-to-end big data solutions to get valuable insights / V. Naresh Kumar, Prashant Shindgikar.

Format:

Book

Author/Creator:

Kumar, V. Naresh, author.

Shindgikar, Prashant, author.

Language:

English

Subjects (All):

Apache Hadoop.

Electronic data processing--Distributed processing.

Electronic data processing.

Physical Description:

1 online resource (394 pages)

Edition:

1st edition

Place of Publication:

Birmingham ; Mumbai : Packt Publishing, 2018.

System Details:

text file

Summary:

A comprehensive guide to design, build and execute effective Big Data strategies using Hadoop About This Book Get an in-depth view of the Apache Hadoop ecosystem and an overview of the architectural patterns pertaining to the popular Big Data platform Conquer different data processing and analytics challenges using a multitude of tools such as Apache Spark, Elasticsearch, Tableau and more A comprehensive, step-by-step guide that will teach you everything you need to know, to be an expert Hadoop Architect Who This Book Is For This book is for Big Data professionals who want to fast-track their career in the Hadoop industry and become an expert Big Data architect. Project managers and mainframe professionals looking forward to build a career in Big Data Hadoop will also find this book to be useful. Some understanding of Hadoop is required to get the best out of this book. What You Will Learn Build an efficient enterprise Big Data strategy centered around Apache Hadoop Gain a thorough understanding of using Hadoop with various Big Data frameworks such as Apache Spark, Elasticsearch and more Set up and deploy your Big Data environment on premises or on the cloud with Apache Ambari Design effective streaming data pipelines and build your own enterprise search solutions Utilize the historical data to build your analytics solutions and visualize them using popular tools such as Apache Superset Plan, set up and administer your Hadoop cluster efficiently In Detail The complex structure of data these days requires sophisticated solutions for data transformation, to make the information more accessible to the users.This book empowers you to build such solutions with relative ease with the help of Apache Hadoop, along with a host of other Big Data tools. This book will give you a complete understanding of the data lifecycle management with Hadoop, followed by modeling of structured and unstructured data in Hadoop. It will also show you how to design real-time streaming pipelines by leveraging tools such as Apache Spark, and build efficient enterprise search solutions using Elasticsearch. You will learn to build enterprise-grade analytics solutions on Hadoop, and how to visualize your data using tools such as Apache Superset. This book also covers techniques for deploying your Big Data solutions on the cloud Apache Ambari, as well as expert techniques for managing and administering your Hadoop cluster. By the end of this book, you will have all the knowled...

Contents:

Cover

Title Page

Copyright and Credits

Packt Upsell

Contributors

Table of Contents

Preface

Chapter 1: Enterprise Data Architecture Principles

Data architecture principles

Volume

Velocity

Variety

Veracity

The importance of metadata

Data governance

Fundamentals of data governance

Data security

Application security

Input data

Big data security

RDBMS security

BI security

Physical security

Data encryption

Secure key management

Data as a Service

Evolution data architecture with Hadoop

Hierarchical database architecture

Network database architecture

Relational database architecture

Employees

Devices

Department

Department and employee mapping table

Hadoop data architecture

Data layer

Data management layer

Job execution layer

Summary

Chapter 2: Hadoop Life Cycle Management

Data wrangling

Data acquisition

Data structure analysis

Information extraction

Unwanted data removal

Data transformation

Data standardization

Data masking

Substitution

Static

Dynamic

Encryption

Hashing

Hiding

Erasing

Truncation

Variance

Shuffling

What is Apache Ranger?

Apache Ranger installation using Ambari

Ambari admin UI

Add service

Service placement

Service client placement

Database creation on master

Ranger database configuration

Configuration changes

Configuration review

Deployment progress

Application restart

Apache Ranger user guide

Login to UI

Access manager

Service details

Policy definition and auditing for HDFS

Chapter 3: Hadoop Design Consideration

Understanding data structure principles

Installing Hadoop cluster

Configuring Hadoop on NameNode

Format NameNode

Start all services

Exploring HDFS architecture.

Defining NameNode

Secondary NameNode

NameNode safe mode

DataNode

Data replication

Rack awareness

HDFS WebUI

Introducing YARN

YARN architecture

Resource manager

Node manager

Configuration of YARN

Configuring HDFS high availability

During Hadoop 1.x

During Hadoop 2.x and onwards

HDFS HA cluster using NFS

Important architecture points

Configuration of HA NameNodes with shared storage

HDFS HA cluster using the quorum journal manager

Configuration of HA NameNodes with QJM

Automatic failover

Configuring automatic failover

Hadoop cluster composition

Typical Hadoop cluster

Best practices Hadoop deployment

Hadoop file formats

Text/CSV file

JSON

Sequence file

Avro

Parquet

ORC

Which file format is better?

Chapter 4: Data Movement Techniques

Batch processing versus real-time processing

Batch processing

Real-time processing

Apache Sqoop

Sqoop Import

Import into HDFS

Import a MySQL table into an HBase table

Sqoop export

Flume

Apache Flume architecture

Data flow using Flume

Flume complex data flow architecture

Flume setup

Log aggregation use case

Apache NiFi

Main concepts of Apache NiFi

Apache NiFi architecture

Key features

Real-time log capture dataflow

Kafka Connect

Kafka Connect - a brief history

Why Kafka Connect?

Kafka Connect features

Kafka Connect architecture

Kafka Connect workers modes

Standalone mode

Distributed mode

Kafka Connect cluster distributed architecture

Example 1

Example 2

Chapter 5: Data Modeling in Hadoop

Apache Hive

Apache Hive and RDBMS

Supported datatypes

How Hive works

Hive architecture

Hive data model management

Hive tables

Managed tables.

External tables

Hive table partition

Hive static partitions and dynamic partitions

Hive partition bucketing

How Hive bucketing works

Creating buckets in a non-partitioned table

Creating buckets in a partitioned table

Hive views

Syntax of a view

Hive indexes

Compact index

Bitmap index

JSON documents using Hive

Example 1 - Accessing simple JSON documents with Hive (Hive 0.14 and later versions)

Example 2 - Accessing nested JSON documents with Hive (Hive 0.14 and later versions)

Example 3 - Schema evolution with Hive and Avro (Hive 0.14 and later versions)

Apache HBase

Differences between HDFS and HBase

Differences between Hive and HBase

Key features of HBase

HBase data model

Difference between RDBMS table and column - oriented data store

HBase architecture

HBase architecture in a nutshell

HBase rowkey design

Example 4 - loading data from MySQL table to HBase table

Example 5 - incrementally loading data from MySQL table to HBase table

Example 6 - Load the MySQL customer changed data into the HBase table

Example 7 - Hive HBase integration

Chapter 6: Designing Real-Time Streaming Data Pipelines

Real-time streaming concepts

Data stream

Batch processing versus real-time data processing

Complex event processing

Continuous availability

Low latency

Scalable processing frameworks

Horizontal scalability

Storage

Real-time streaming components

Message queue

So what is Kafka?

Kafka features

Kafka architecture

Kafka architecture components

Kafka Connect deep dive

Kafka Connect workers standalone versus distributed mode

Install Kafka

Create topics

Generate messages to verify the producer and consumer

Kafka Connect using file Source and Sink.

Kafka Connect using JDBC and file Sink Connectors

Apache Storm

Features of Apache Storm

Storm topology

Storm topology components

Installing Storm on a single node cluster

Developing a real-time streaming pipeline with Storm

Streaming a pipeline from Kafka to Storm to MySQL

Streaming a pipeline with Kafka to Storm to HDFS

Other popular real-time data streaming frameworks

Kafka Streams API

Spark Streaming

Apache Flink

Apache Flink versus Spark

Apache Spark versus Storm

Chapter 7: Large-Scale Data Processing Frameworks

MapReduce

Hadoop MapReduce

Streaming MapReduce

Java MapReduce

Apache Spark 2

Installing Spark using Ambari

Service selection in Ambari Admin

Add Service Wizard

Server placement

Clients and Slaves selection

Service customization

Software deployment

Spark installation progress

Service restarts and cleanup

Apache Spark data structures

RDDs, DataFrames and datasets

Apache Spark programming

Sample data for analysis

Interactive data analysis with pyspark

Standalone application with Spark

Spark streaming application

Spark SQL application

Chapter 8: Building Enterprise Search Platform

The data search concept

The need for an enterprise search engine

Tools for building an enterprise search engine

Elasticsearch

Why Elasticsearch?

Elasticsearch components

Index

Document

Mapping

Cluster

Type

How to index documents in Elasticsearch?

Elasticsearch installation

Installation of Elasticsearch

Create index

Primary shard

Replica shard

Ingest documents into index

Bulk Insert

Document search

Meta fields

Static mapping

Dynamic mapping

Elasticsearch-supported data types

Mapping example

Analyzer

Elasticsearch stack components.

Beats

Logstash

Kibana

Use case

Chapter 9: Designing Data Visualization Solutions

Data visualization

Bar/column chart

Line/area chart

Pie chart

Radar chart

Scatter/bubble chart

Other charts

Practical data visualization in Hadoop

Apache Druid

Druid components

Other required components

Apache Druid installation

Select Druid and Superset

Service placement on servers

Choose Slaves and Clients

Service configurations

Service installation

Installation summary

Sample data ingestion into Druid

MySQL database

Sample database

Download the sample dataset

Copy the data to MySQL

Verify integrity of the tables

Single Normalized Table

Apache Superset

Accessing the Superset application

Superset dashboards

Understanding Wikipedia edits data

Create Superset Slices using Wikipedia data

Unique users count

Word Cloud for top US regions

Sunburst chart - top 10 cities

Top 50 channels and namespaces via directed force layout

Top 25 countries/channels distribution

Creating wikipedia edits dashboard from Slices

Apache Superset with RDBMS

Supported databases

Understanding employee database

Employees table

Departments table

Department manager table

Department Employees Table

Titles table

Salaries table

Normalized employees table

Superset Slices for employees database

Register MySQL database/table

Slices and Dashboard creation

Department salary breakup

Salary Diversity

Salary Change Per Role Per Year

Dashboard creation

Chapter 10: Developing Applications Using the Cloud

What is the Cloud?

Available technologies in the Cloud

Planning the Cloud infrastructure

Dedicated servers versus shared servers

Dedicated servers

Shared servers

High availability.

Business continuity planning.

Notes:

Description based on print version record.

OCLC:

1031344421