Data Engineering with AWS : Acquire the Skills to Design and Build AWS-Based Data Transformation Pipelines Like a Pro / Gareth Eagar.

Format:

Book

Author/Creator:

Eagar, Gareth, author.

Series:

Expert insight.

Expert insight

Language:

English

Subjects (All):

Amazon Web Services (Firm).

Cloud computing.

Big data.

Physical Description:

1 online resource (637 pages)

Edition:

Second edition.

Place of Publication:

Birmingham, England : Packt Publishing Ltd., [2023]

Summary:

This book, authored by a seasoned Senior Data Architect with 25 years of experience, aims to help you achieve proficiency in using the AWS ecosystem for data engineering. This revised edition provides updates in every chapter to cover the latest AWS services and features, takes a refreshed look at data governance, and includes a brand-new section on building modern data platforms which covers; implementing a data mesh approach, open-table formats (such as Apache Iceberg), and using DataOps for automation and observability. You'll begin by reviewing the key concepts and essential AWS tools in a data engineer's toolkit and getting acquainted with modern data management approaches. You'll then architect a data pipeline, review raw data sources, transform the data, and learn how that transformed data is used by various data consumers. You’ll learn how to ensure strong data governance, and about populating data marts and data warehouses along with how a data lakehouse fits into the picture. After that, you'll be introduced to AWS tools for analyzing data, including those for ad-hoc SQL queries and creating visualizations. Then, you'll explore how the power of machine learning and artificial intelligence can be used to draw new insights from data. In the final chapters, you'll discover transactional data lakes, data meshes, and how to build a cutting-edge data platform on AWS. By the end of this AWS book, you'll be able to execute data engineering tasks and implement a data pipeline on AWS like a pro!

Contents:

Cover

Copyright

Contributors

Table of Contents

Preface

Section 1: AWS Data Engineering Concepts and Trends

Chapter 1: An Introduction to Data Engineering

Technical requirements

The rise of big data as a corporate asset

The challenges of ever-growing datasets

The role of the data engineer as a big data enabler

Understanding the role of the data engineer

Understanding the role of the data scientist

Understanding the role of the data analyst

Understanding other common data-related roles

The benefits of the cloud when building big data analytic solutions

Hands-on - creating and accessing your AWS account

Creating a new AWS account

Accessing your AWS account

Summary

Chapter 2: Data Management Architectures for Analytics

The evolution of data management for analytics

Databases and data warehouses

Dealing with big, unstructured data

Cloud-based solutions for big data analytics

A deeper dive into data warehouse concepts and architecture

Dimensional modeling in data warehouses

Understanding the role of data marts

Distributed storage and massively parallel processing

Columnar data storage and efficient data compression

Feeding data into the warehouse - ETL and ELT pipelines

An overview of data lake architecture and concepts

Data lake logical architecture

The storage layer and storage zones

Catalog and search layers

Ingestion layer

The processing layer

The consumption layer

Data lake architecture summary

Bringing together the best of data warehouses and data lakes

The data lake house approach

New data lake table formats

Federated queries across database engines

Hands-on - using the AWS Command Line Interface (CLI) to create Simple Storage Service (S3) buckets

Accessing the AWS CLI.

Using AWS CloudShell to access the CLI

Creating new Amazon S3 buckets

Chapter 3: The AWS Data Engineer's Toolkit

An overview of AWS services for ingesting data

Amazon Database Migration Service (DMS)

Amazon Kinesis for streaming data ingestion

Amazon Kinesis Agent

Amazon Kinesis Firehose

Amazon Kinesis Data Streams

Amazon Kinesis Data Analytics

Amazon Kinesis Video Streams

Amazon MSK for streaming data ingestion

Amazon AppFlow for ingesting data from SaaS services

AWS Transfer Family for ingestion using FTP/SFTP protocols

AWS DataSync for ingesting from on premises and multicloud storage services

The AWS Snow family of devices for large data transfers

AWS Glue for data ingestion

An overview of AWS services for transforming data

AWS Lambda for light transformations

AWS Glue for serverless data processing

Serverless ETL processing

AWS Glue DataBrew

AWS Glue Data Catalog

AWS Glue crawlers

Amazon EMR for Hadoop ecosystem processing

An overview of AWS services for orchestrating big data pipelines

AWS Glue workflows for orchestrating Glue components

AWS Step Functions for complex workflows

Amazon Managed Workflows for Apache Airflow (MWAA)

An overview of AWS services for consuming data

Amazon Athena for SQL queries in the data lake

Amazon Redshift and Redshift Spectrum for data warehousing and data lakehouse architectures

Overview of Amazon QuickSight for visualizing data

Hands-on - triggering an AWS Lambda function when a new file arrives in an S3 bucket

Creating a Lambda layer containing the AWS SDK for pandas library

Creating an IAM policy and role for your Lambda function

Creating a Lambda function

Configuring our Lambda function to be triggered by an S3 upload

Summary.

Chapter 4: Data Governance, Security, and Cataloging

The many different aspects of data governance

Data security, access, and privacy

Common data regulatory requirements

Core data protection concepts

Personally identifiable information (PII)

Personal data

Encryption

Anonymized data

Pseudonymized data/tokenization

Authentication

Authorization

Putting these concepts together

Data quality, data profiling, and data lineage

Data quality

Data profiling

Data lineage

Business and technical data catalogs

Implementing a data catalog to avoid creating a data swamp

Business data catalogs

Technical data catalogs

AWS services that help with data governance

The AWS Glue/Lake Formation technical data catalog

AWS Glue DataBrew for profiling datasets

AWS Glue Data Quality

AWS Key Management Service (KMS) for data encryption

Amazon Macie for detecting PII data in Amazon S3 objects

The AWS Glue Studio Detect PII transform for detecting PII data in datasets

Amazon GuardDuty for detecting threats in an AWS account

AWS Identity and Access Management (IAM) service

Using AWS Lake Formation to manage data lake access

Permissions management before Lake Formation

Permissions management using AWS Lake Formation

Hands-on - configuring Lake Formation permissions

Creating a new user with IAM permissions

Transitioning to managing fine-grained permissions with AWS Lake Formation

Activating Lake Formation permissions for a database and table

Granting Lake Formation permissions

Section 2: Architecting and Implementing Data Engineering Pipelines and Transformations

Chapter 5: Architecting Data Engineering Pipelines

Approaching the data pipeline architecture

Architecting houses and pipelines.

Whiteboarding as an information-gathering tool

Conducting a whiteboarding session

Identifying data consumers and understanding their requirements

Identifying data sources and ingesting data

Identifying data transformations and optimizations

File format optimizations

Data standardization

Data quality checks

Data partitioning

Data denormalization

Data cataloging

Whiteboarding data transformation

Loading data into data marts

Wrapping up the whiteboarding session

Hands-on - architecting a sample pipeline

Detailed notes from the project "Bright Light" whiteboarding meeting of GP Widgets, Inc

Meeting notes

Chapter 6: Ingesting Batch and Streaming Data

Understanding data sources

Data variety

Structured data

Semi-structured data

Unstructured data

Data volume

Data velocity

Data veracity

Data value

Questions to ask

Ingesting data from a relational database

AWS DMS

AWS Glue

Full one-off loads from one or more tables

Initial full loads from a table, and subsequent loads of new records

Creating AWS Glue jobs with AWS Lake Formation

Other ways to ingest data from a database

Deciding on the best approach to ingesting from a database

The size of the database

Database load

Data ingestion frequency

Technical requirements and compatibility

Ingesting streaming data

Amazon Kinesis versus Amazon Managed Streaming for Kafka (MSK)

Serverless services versus managed services

Open-source flexibility versus proprietary software with strong AWS integration

At-least-once messaging versus exactly once messaging

A single processing engine versus niche tools

Deciding on a streaming ingestion tool

Hands-on - ingesting data with AWS DMS

Deploying MySQL and an EC2 data loader via CloudFormation.

Creating an IAM policy and role for DMS

Configuring DMS settings and performing a full load from MySQL to S3

Querying data with Amazon Athena

Hands-on - ingesting streaming data

Configuring Kinesis Data Firehose for streaming delivery to Amazon S3

Configuring Amazon Kinesis Data Generator (KDG)

Adding newly ingested data to the Glue Data Catalog

Querying the data with Amazon Athena

Chapter 7: Transforming Data to Optimize for Analytics

Overview of how transformations can create value

Cooking, baking, and data transformations

Transformations as part of a pipeline

Types of data transformation tools

Apache Spark

Hadoop and MapReduce

SQL

GUI-based tools

Common data preparation transformations

Protecting PII data

Optimizing the file format

Optimizing with data partitioning

Data cleansing

Common business use case transformations

Enriching data

Pre-aggregating data

Extracting metadata from unstructured data

Working with Change Data Capture (CDC) data

Traditional approaches - data upserts and SQL views

Modern approaches - Open Table Formats (OTFs)

Apache Iceberg

Apache Hudi

Databricks Delta Lake

Hands-on - joining datasets with AWS Glue Studio

Creating a new data lake zone - the curated zone

Creating a new IAM role for the Glue job

Configuring a denormalization transform using AWS Glue Studio

Finalizing the denormalization transform job to write to S3

Create a transform job to join streaming and film data using AWS Glue Studio

Chapter 8: Identifying and Enabling Data Consumers

Understanding the impact of data democratization

A growing variety of data consumers

How a data mesh helps data consumers.

Meeting the needs of business users with data visualization.

Notes:

Includes index.

Description based on print version record.

ISBN:

9781804613139

1804613134

OCLC:

1407633144