Building Trusted Data Platforms with Azure Databricks and GenAI : A Hands-On Guide to Creating Governed Data Products in a Lakehouse.

Format:

Book

Author/Creator:

Kukreja, Manoj.

Language:

English

Subjects (All):

Microsoft Azure (Computing platform).

Artificial intelligence.

Physical Description:

1 online resource (774 pages)

Edition:

1st ed.

Place of Publication:

Birmingham : Packt Publishing, Limited, 2026.

Summary:

A practical guide to building a modern, GenAI-powered data platform with a Lakehouse foundation, covering MDM, data mesh, AI enablement, streaming pipelines, observability, and cloud-driven architectures for trusted analytics.

Contents:

Intro

Building Trusted Data Platforms with Azure Databricks and GenAI

Second Edition

A Hands-On Guide to Creating Governed Data Products in a Lakehouse

Foreword

Contributors

About the author

About the technical implementation expert

About the reviewers

Table of Contents

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Get in touch

Share your thoughts

Free benefits with your book

How to unlock

1

The Story of Data Engineering and Analytics

The journey of data

Exploring the evolution of data analytics

Core capabilities of storage and compute resources

Availability of varying datasets

The paradigm shift to distributed computing

Adoption of cloud computing

Data storytelling

Compliance

Widespread AI Adoption

The monetary power of data

Organic growth

Customer retention

Fraud prevention

Problem detection

Data monetization

Summary

Get this book's PDF version and more

2

Discovering Data Platforms

Introducing data platforms

Exploring the benefits of a data platform

Accommodating varying data formats

Storing data in zones

Accommodating varying data characteristics

Adhering to compliance frameworks

Segregating storage and compute in a data platform

Storage

Block storage

Object storage

Compute

Containers

Storage is cheap

compute is expensive

Discovering data platform architectures

The CAP theorem

The trade-offs

Traditional data lake architecture

Lambda architecture

Batch layer

Serving layer

Speed layer

Kappa architecture

Streaming layer

Lakehouse architecture.

The give-and-take struggle

3

Data Engineering on Microsoft Azure

Introducing data engineering in Azure

Performing data engineering in Microsoft Azure

Self-managed data engineering services (IaaS)

IaaS use case

Hadoop/Spark/Kafka distributions

Infrastructure as Code

Containerization

Azure-managed data engineering services (PaaS)

PaaS use case

Data storage services in Microsoft Azure

Azure Data Lake Storage Gen2

Azure Event Hubs

Azure Cosmos DB

Data processing services in Microsoft Azure

Azure HDInsight

Azure Databricks

Data engineering

Data warehousing

Machine learning and AI

Data engineering as a service (SaaS)

SaaS use case

Azure Data Factory

Azure Stream Analytics

Azure Synapse Analytics

Data cataloging and sharing services in Microsoft Azure

Microsoft Purview

Unity Catalog

Azure Data Share

Delta Sharing

Opening a free account with Microsoft Azure

4

Designing Future Data Platforms

Hallmarks of a future data platform

Cloud-driven infrastructure

Resilience of cloud services

Faster ramp-up time

Cost Control

Data security and governance

Cutting-edge innovation

Global outreach

Lakehouse architecture

Master data management

Data mesh

Convergence in data delivery

Data-as-a-product

Decentralized ownership

Domain experts

Data experts

Data contract

Contract validation

Data consumers

Self-Serve Analytics

Data product creation process

Data product distribution process

Data product consumption process

Federated Governance

Role of the central authority

Real-time processing

Automation and data observability

Automation

Infrastructure as code.

Continuous integration and deployment (CI/CD)

Data product toolkits

Data observability

AI enablement

Exploring data using natural language

Code generation

Domain specific chatbots

Embedding in SQL

5

Understanding Databricks

Introduction to Databricks

Unified data platform

Harmonized data platform

Unified governance

Collaborative development

Integrated analytics workloads

Databricks architecture

Operating model

Control plane

Compute plane

Deployment options

Workspaces

Workspace overview

Workspace access

Workspace repositories

Domain-driven Workspace deployment

Databricks One

Interfaces

Authentication and authorization

Entitlements

Databricks capabilities

Lakeflow Connect

Procedural processing

Notebooks

Lakeflow Spark Declarative Pipelines

Lakeflow Jobs

Databricks SQL Warehouse

Serverless SQL Warehouse

Pro SQL Warehouse

Classic SQL Warehouse

Machine learning and artificial intelligence

Business intelligence

Classic compute clusters

All-purpose compute clusters

Job compute clusters

Pipeline compute clusters

Serverless compute

Serverless compute for notebooks

Serverless compute for Jobs

Serverless pipelines

SQL Warehouse compute

Metastore

Object model

Catalogs

Schemas

Data objects

Tables

Views

Volumes

Functions

Models

Non-data objects

Lakehouse Federation

Privilege management

Governed tags

Data lineage

Data sharing

Data Marketplace

Clean rooms

Security

Data protection

Secrets manager

Process Isolation

Billing

Databricks platform costs

Cloud infrastructure costs

Summary.

Get this book's PDF version and more

6

Understanding Medallion Architecture and Delta Lake

Medallion architecture

Bronze layer

Silver layer

Gold layer

Understanding Spark

In-memory computations

Partitioning

Immutable

Persistence

Location Stickiness

Lazy evaluation

DataFrames

Spark SQL

Understanding how Delta Lake enables the Lakehouse

The Single Source of Truth (SSOT)

Understanding Delta Lake

Preparing Azure resources

Provision Azure Databricks

Provision Azure Data Lake Storage (ADLS)

Creating storage credentials and an external location in Databricks

Running the notebook

Prepare code repository

Prepare Spark compute cluster

Create your first Unity Catalog and schema

Import PySpark libraries and DataFrame functions

Create a Pyspark Dataframe

Create a Delta table and apply partitioning to optimize query performance

Check the Delta table data and details

Check Delta history

Create a Delta table with a generated column

Perform DML operations in the Delta table

Perform DML operations in the Delta table - INSERT

Perform DML operations in the Delta table - UPDATE

Perform DML operations in the Delta table - DELETE

Perform time-travel in the Delta table

Perform the CDC merge in the Delta table

Perform CDC merge in the Delta table with schema evolution

Explicitly modify the schema of the Delta table

Use the replaceWhere feature in Delta Lake

Set up constraints in the Delta Table

Set up a NOT NULL constraint in the Delta Table

Set up a CHECK constraint in the Delta Table

Set up a foreign key constraint in the Delta table

Improve Delta table performance

Partitioning the Delta table

Optimize the Delta table

Optimize using Z-ordering in the Delta table

Optimize writes.

Run Analyzer

Optimize using Liquid Clustering in the Delta table

Regularly clean old Delta files

Isolation levels in the Delta table

Understanding concurrency control in the Delta table

Understanding deletion vectors in the Delta table

7

Understanding Modern Data Pipelines

Exploring Data Pipelines

Components of a data pipeline

Data source

Data ingestion

Watermark technique

Change Data Capture (CDC)

Log-based CDC

Change Tracking

Data transformation

Data destination

Data orchestration

Workflows

Workflow actions

Input and output actions

Transformation Actions

Control Actions

Workflow Scheduling

Workflow Monitoring

Data pipeline types

Imperative pipelines

Declarative pipelines

Understanding the data pipeline creation process

Discovery Phase

Identify the product owner

Shortlist use cases

Conduct Discovery Sessions

Design Phase

Shortlist Common Datasets

Design Pipeline using Parent-Child Approach Common Datasets

Design Cloud Architecture

Design Automation and Monitoring

Costing

Development Phase

Unit testing phase

Deployment Phase

Monitoring phase

Documentation phase

Electroniz data platform

Data platform requirements

Data platform blueprint

Reference architecture

Data storage design

Data compute design

Data pipeline design

Batch ingestion pipeline

Streaming ingestion pipeline

Bronze pipeline

Silver pipeline

Gold pipeline

Data product creation pipeline

Data product strategy

Governance

Deployment Strategy

Observability

8

Data Ingestion Stage: Building the Landing Zone

Technical requirements

Need for a landing zone

High-level plan to build the landing zone.

Batch ingestion pipeline.

Notes:

Description based on publisher supplied metadata and other sources.

Part of the metadata in this record was created by AI, based on the text of the resource.

ISBN:

9781806679768

OCLC:

1605653591