Engineering Data Mesh in Azure Cloud : Implement Data Mesh Using Microsoft Azure's Cloud Adoption Framework / Aniruddha Deswandikar.

Format:

Book

Author/Creator:

Deswandikar, Aniruddha, author.

Language:

English

Subjects (All):

Management information systems.

Microsoft Azure (Computing platform).

Cloud computing.

Business--Data processing.

Business.

Big data.

Physical Description:

1 online resource (314 pages)

Edition:

First edition.

Place of Publication:

Birmingham, England : Packt Publishing, [2024]

Biography/History:

Deswandikar Aniruddha: Aniruddha Deswandikar holds a Bachelor's degree in Computer Engineering and is a seasoned Solutions Architect with over 30 years of industry experience as a developer, architect and technology strategist. His experience spans from start-ups to dotcoms to large enterprises. He has spent 18 years at Microsoft helping Microsoft customers build their next generation Applications and Data Analytics platforms. His experience across Application, Data and AI has helped him provide holistic guidance to companies large and small. Currently he is helping global enterprises set up their Enterprise-scale Analytical system using the Data Mesh Architecture. He is a Subject Matter Expert on Data Mesh in Microsoft and is currently helping multiple Microsoft Global Customers implement the Data Mesh architecture.

Summary:

Overcome data mesh adoption challenges using the cloud-scale analytics framework and make your data analytics landscape agile and efficient by using standard architecture patterns for diverse analytical workloads Key Features Delve into core data mesh concepts and apply them to real-world situations Safely reassess and redesign your framework for seamless data mesh integration Conquer practical challenges, from domain organization to building data contracts Purchase of the print or Kindle book includes a free PDF eBook Book Description Decentralizing data and centralizing governance are practical, scalable, and modern approaches to data analytics. However, implementing a data mesh can feel like changing the engine of a moving car. Most organizations struggle to start and get caught up in the concept of data domains, spending months trying to organize domains. This is where Engineering Data Mesh in Azure Cloud can help. The book starts by assessing your existing framework before helping you architect a practical design. As you progress, you'll focus on the Microsoft Cloud Adoption Framework for Azure and the cloud-scale analytics framework, which will help you quickly set up a landing zone for your data mesh in the cloud. The book also resolves common challenges related to the adoption and implementation of a data mesh faced by real customers. It touches on the concepts of data contracts and helps you build practical data contracts that work for your organization. The last part of the book covers some common architecture patterns used for modern analytics frameworks such as artificial intelligence (AI). By the end of this book, you'll be able to transform existing analytics frameworks into a streamlined data mesh using Microsoft Azure, thereby navigating challenges and implementing advanced architecture patterns for modern analytics workloads. What you will learn Build a strategy to implement a data mesh in Azure Cloud Plan your data mesh journey to build a collaborative analytics platform Address challenges in designing, building, and managing data contracts Get to grips with monitoring and governing a data mesh Understand how to build a self-service portal for analytics Design and implement a secure data mesh architecture Resolve practical challenges related to data mesh adoption Who this book is for This book is for chief data officers and data architects of large and medium-size organizations who are struggling to maintain silos of data and analytics projects. Data architects and data engineers looking to understand data mesh and how it can help their organizations democratize data and analytics will also benefit from this book. Prior knowledge of managing centralized analytical systems, as well as experience with building data lakes, data warehouses, data pipelines, data integrations, and transformations is needed to get the most out of this book.

Contents:

Cover

Title Page

Copyright

Dedication

Contributors

Table of Contents

Preface

Part 1: Rolling Out the Data Mesh in the Azure Cloud

Chapter 1: Introducing Data Meshes

Exploring the evolution of modern data analytics

Discovering the challenges of modern-day enterprises

DaaP

Data domains

The data mesh solution

Summary

Chapter 2: Building a Data Mesh Strategy

Is a data mesh for everybody?

Aligning your analytics strategy with your business strategy

Understanding data maturity models

Stage 1

Stage 2

Stage 3

Stage 4

Building the technology stack

The analytics team

Data governance

Approaches to building your data mesh

Chapter 3: Deploying a Data Mesh Using the Azure Cloud-Scale Analytics Framework

Introduction to Azure CSA

Understanding landing zones

Organizing resources

Designing a cloud management structure

Hierarchical policies

Diving deeper into landing zones in CSA

Data management landing zone

Data landing zone

Automating landing zone deployment

IaC

Organizing resources in a landing zone

Networking topologies

Security and access control

Streamlining deployment through DevOps

Chapter 4: Building the Data Mesh Governance Framework Using Microsoft Azure Services

Data mesh governance requirements

Data catalog

Collecting and managing metadata

Step 1 - ensure accuracy and completeness

Step 2 - verify data classification

Step 3 - add a business glossary

Step 4 - add lineage information

Monitoring and managing data quality

Implementing data observability

Chapter 5: Security Architecture for Data Meshes

Understanding the security requirements of data mesh architecture

Understanding authentication and authorization in Azure

Managing data access

SQL Database.

Data lakes

Data lake structure

Managing data privacy

Data masking

Data retention

Chapter 6: Automating Deployment through Azure Resource Manager and Azure DevOps

Azure Resource Manager templates for landing zones

Understanding the ARM template structure

Source code control for ARM templates

Azure DevOps pipelines for deploying infrastructure

Base data product templates

T-shirt sizing

Landing zone requests

Landing zone approval

Landing zone deployment

Self-service portal

Customized templates

Chapter 7: Building a Self-Service Portal for Common Data Mesh Operations

Why do we need a self-service portal?

Gathering requirements for the self-service portal

Requesting a data product zone

Browse and reuse pipeline

Data discovery

Access management

Requesting landing zones or data products

Hosting common data pipeline templates

Azure Data Factory

Azure Data Factory instance

Integration runtime

Creating linked services

Create a sequence of activities

Parameterize the pipeline

Continuous integration/continuous development

Data mesh portal integration

Other common features of a self-service portal

Architecting the self-service portal

Active Directory and Domain Name System (DNS)

Application Gateway

Azure App Service

Azure Cosmos DB

Git Repo and Azure DevOps pipelines

Network and security

Azure Cache for Redis (optional)

Azure SQL DB (optional)

Part 2: Practical Challenges of Implementing a Data Mesh

Chapter 8: How to Design, Build, and Manage Data Contracts

What are data contracts?

What are the contents of a data contract?

Who creates and owns a data contract?

Who consumes the data contract?

How do we store data and access contracts?.

How do we link data contracts to data consumption or pipelines?

Catalog and contract document design

Set up Cosmos DB

Write the integration code

Searching contracts and data assets

Put the pieces together

Chapter 9: Data Quality Management

Why is data quality important?

How is data quality defined?

How to manage data quality

Accuracy

Completeness

Consistency

Timeliness

Validity

Uniqueness

Reliability

Data quality management systems

Completely decentralized

Completely centralized

The hybrid approach

Build versus buy

Popular data quality frameworks and tools

Chapter 10: Master Data Management

Single source of truth

What causes discrepancies in master data?

MDM design patterns

MDM architecture for a data mesh

Popular MDM tools

Chapter 11: Monitoring and Data Observability

Piecing it all together - the importance of data mesh monitoring and data observability

How data mesh monitoring differs

Baking diagnostic logging into the landing zone templates

Azure Platform Metrics

Azure platform logs

Enabling diagnostic settings in an ARM template

Designing a data mesh operations center

Step 1 - collection

Step 2 - rank the critical metrics and events

Step 3 - build a threshold logic for each service in a data product

Step 4 - build a monitoring view for each resource

Step 5 - build a threshold logic for each data product

Step 6 - build a threshold logic for each data landing zone

Step 7 - set up alerts for critical metrics

Step 8 - host the dashboards in one location

Tooling for the DMOC

Azure Monitor

Log Analytics

Azure Data Explorer

Grafana

Power BI

Data observability

Setting up alerts

Piecing it all together

Summary.

Chapter 12: Monitoring Data Mesh Costs and Building a Cross-Charging Model

Components of data mesh costs

Cost models in a data mesh

Overview of cost management in Azure

Allocating costs to different data product groups and domains

How to determine the cost of shared resources

Chapter 13: Understanding Data-Sharing Topologies in a Data Mesh

What is in-place sharing?

Understanding data-sharing challenges in a data mesh

Latency

Data formats and protocols

Exploring different methods available for sharing data

In-place access

Data pipelines

Data APIs

Data Share

Picking the right data-sharing topologies

In-place sharing

Data sharing

Part 3: Popular Data Product Architectures

Chapter 14: Advanced Analytics Using Azure Machine Learning, Databricks, and the Lakehouse Architecture

Requirements

Architecture

Components

Source data

Azure Data Lake Storage Gen2

Azure Databricks

Azure Machine Learning

Azure Kubernetes Service (AKS)

Azure Data Share

Data flow

Scenarios

Chapter 15: Big Data Analytics Using Azure Synapse Analytics

Azure Synapse pipelines

Azure Synapse

Azure AI Search

Chapter 16: Event-Driven Analytics Using Azure Event Hubs, Azure Stream Analytics, and Azure Machine Learning

Azure Event Hubs

Azure IoT Hub

Azure Stream Analytics

Data flow.

Combining architectures for real-time and big data analytics

Chapter 17: AI Using Azure Cognitive Services and Azure OpenAI

Azure Translator

Azure AI Document Intelligence

Azure OpenAI embedding models

Azure Redis Cache

Semantic Kernel

Azure OpenAI

Bing search

Content filtering and security

Data flow/interactions

Index

About PACKT

Other Books You May Enjoy.

Notes:

Description based upon print version of record.

Parameterize the pipeline

Description based on publisher supplied metadata and other sources.

Description based on print version record.

ISBN:

9781805128946

1805128949

OCLC:

1427667024