Data Engineering with Databricks Cookbook : Build Effective Data and AI Solutions Using Apache Spark, Databricks, and Delta Lake / Pulkit Chadha.

Format:

Book

Author/Creator:

Chadha, Pulkit, author.

Language:

English

Subjects (All):

Spark (Electronic resource : Apache Software Foundation).

Data mining.

Electronic data processing.

Databases.

Physical Description:

1 online resource (438 pages)

Edition:

First edition.

Place of Publication:

Birmingham, England : Packt Publishing, [2024]

System Details:

Mode of access: World Wide Web.

Biography/History:

Chadha Pulkit: Pulkit Chadha is a Sr. Solutions Architect at Databricks. He has over 12 years of experience working in Data Engineering. With his expertise in building and optimizing data pipelines using various tools and technologies Pulkit has worked with enterprises in various industries, including healthcare, Media and Entertainment, Hi-Tech, and Manufacturing providing data engineering solutions to meet enterprises' unique business needs. His work history includes the likes of Dell Services, Adobe, and Databricks. Pulkit holds a Masters's Degree in Management Information Systems from Eller College of Management at the University Of Arizona and has several cloud certifications in data analytics.

Summary:

Data Engineering with Databricks Cookbook will guide you through recipes to effectively use Apache Spark, Delta Lake, and Databricks for data engineering, beginning with an introduction to data ingestion and loading with Apache Spark. As you progress, you’ll be introduced to various data manipulation and data transformation solutions that can be applied to data. You'll find out how to manage and optimize Delta tables, as well as how to ingest and process streaming data. The book will also show you how to improve the performance problems of Apache Spark apps and Delta Lake. Later chapters will show you how to use Databricks to implement DataOps and DevOps practices and teach you how to orchestrate and schedule data pipelines using Databricks Workflows. Finally, you’ll understand how to set up and configure Unity Catalog for data governance. By the end of this book, you’ll be well-versed in building reliable and scalable data pipelines using modern data engineering technologies.

Contents:

Intro

Title Page

Copyright and Credits

Dedication

Contributors

Table of Contents

Preface

Part 1 - Working with Apache Spark and Delta Lake

Chapter 1: Data Ingestion and Data Extraction with Apache Spark

Technical requirements

Reading CSV data with Apache Spark

How to do it...

There's more…

See also

Reading JSON data with Apache Spark

Reading Parquet data with Apache Spark

Parsing XML data with Apache Spark

How to do it…

Working with nested data structures in Apache Spark

Processing text data in Apache Spark

Writing data with Apache Spark

Chapter 2: Data Transformation and Data Manipulation with Apache Spark

Applying basic transformations to data with Apache Spark

Filtering data with Apache Spark

Performing joins with Apache Spark

Performing aggregations with Apache Spark

Using window functions with Apache Spark

Writing custom UDFs in Apache Spark

Handling null values with Apache Spark

Chapter 3: Data Management with Delta Lake

Creating a Delta Lake table

Reading a Delta Lake table

There's more...

Updating data in a Delta Lake table

How to do it.

See also

Merging data into Delta tables

Change data capture in Delta Lake

Optimizing Delta Lake tables

Versioning and time travel for Delta Lake tables

Managing Delta Lake tables

Chapter 4: Ingesting Streaming Data

Configuring Spark Structured Streaming for real-time data processing

Getting ready

How it works…

Reading data from real-time sources, such as Apache Kafka, with Apache Spark Structured Streaming

Defining transformations and filters on a Streaming DataFrame

Configuring checkpoints for Structured Streaming in Apache Spark

Configuring triggers for Structured Streaming in Apache Spark

Applying window aggregations to streaming data with Apache Spark Structured Streaming

Handling out-of-order and late-arriving events with watermarking in Apache Spark Structured Streaming

Chapter 5: Processing Streaming Data

Writing the output of Apache Spark Structured Streaming to a sink such as Delta Lake

Idempotent stream writing with Delta Lake and Apache Spark Structured Streaming

See also.

Merging or applying Change Data Capture on Apache Spark Structured Streaming and Delta Lake

Joining streaming data with static data in Apache Spark Structured Streaming and Delta Lake

Joining streaming data with streaming data in Apache Spark Structured Streaming and Delta Lake

Monitoring real-time data processing with Apache Spark Structured Streaming

Chapter 6: Performance Tuning with Apache Spark

Monitoring Spark jobs in the Spark UI

Using broadcast variables

Optimizing Spark jobs by minimizing data shuffling

Avoiding data skew

Caching and persistence

Partitioning and repartitioning

Optimizing join strategies

Chapter 7: Performance Tuning in Delta Lake

Optimizing Delta Lake table partitioning for query performance

Organizing data with Z-ordering for efficient query execution

Skipping data for faster query execution

Reducing Delta Lake table size and I/O cost with compression

Part 2 - Data Engineering Capabilities within Databricks

Chapter 8: Orchestration and Scheduling Data Pipeline with Databricks Workflows

Building Databricks workflows

Running and managing Databricks Workflows

Passing task and job parameters within a Databricks Workflow

Conditional branching in Databricks Workflows

Triggering jobs based on file arrival

Setting up workflow alerts and notifications

Troubleshooting and repairing failures in Databricks Workflows

Chapter 9: Building Data Pipelines with Delta Live Tables

Creating a multi-hop medallion architecture data pipeline with Delta Live Tables in Databricks

Building a data pipeline with Delta Live Tables on Databricks

Implementing data quality and validation rules with Delta Live Tables in Databricks

Quarantining bad data with Delta Live Tables in Databricks

Monitoring Delta Live Tables pipelines

Deploying Delta Live Tables pipelines with Databricks Asset Bundles

Applying changes (CDC) to Delta tables with Delta Live Tables

Chapter 10: Data Governance with Unity Catalog

Connecting to cloud object storage using Unity Catalog

Creating and managing catalogs, schemas, volumes, and tables using Unity Catalog

Defining and applying fine-grained access control policies using Unity Catalog

Tagging, commenting, and capturing metadata about data and AI assets using Databricks Unity Catalog

Filtering sensitive data with Unity Catalog

Using Unity Catalogs lineage data for debugging, root cause analysis, and impact assessment

Accessing and querying system tables using Unity Catalog

Chapter 11: Implementing DataOps and DevOps on Databricks

Using Databricks Repos to store code in Git

Automating tasks by using the Databricks CLI

Using the Databricks VSCode extension for local development and testing

Using Databricks Asset Bundles (DABs)

Leveraging GitHub Actions with Databricks Asset Bundles (DABs)

Index

About Packt

Other Books You May Enjoy.

Notes:

Includes bibliographical references and index.

Description based on publisher supplied metadata and other sources.

Description based on print version record.

ISBN:

9781837632060

1837632065

OCLC:

1434094957