Azure data factory cookbook : build and manage ETL and ELT pipelines with Microsoft Azure's serverless data integration service / Dmitry Anoshin [and three others].

Format:

Book

Author/Creator:

Anoshin, Dmitry, author.

Language:

English

Subjects (All):

Data warehousing.

Physical Description:

1 online resource (382 pages)

Edition:

1st ed.

Place of Publication:

Birmingham, England ; Mumbai : Packt, [2020]

Summary:

With the help of well-structured and practical recipes, this book will teach you how to integrate data from the cloud and on-premise. You'll learn how to transform, clean, and consolidate data into a single data platform and get to grips with using ADF as the main ETL and orchestration tool for your data warehouse or data platform project.

Contents:

Cover

Title Page

Copyright and Credits

About Packt

Contributors

Table of Contents

Preface

Chapter 1: Getting Started with ADF

Introduction to the Azure data platform

Getting ready

How to do it...

How it works...

Creating and executing our first job in ADF

There's more...

See also

Creating an ADF pipeline by using the Copy Data tool

Creating an ADF pipeline using Python

Creating a data factory using PowerShell

Using templates to create ADF pipelines

Chapter 2: Orchestration and Control Flow

Technical requirements

Using parameters and built-in functions

Using Metadata and Stored Procedure activities

Using the ForEach and Filter activities

Chaining and branching activities within a pipeline

Using the Lookup, Web, and Execute Pipeline activities

Creating event-based triggers

Chapter 3: Setting Up a Cloud Data Warehouse

Connecting to Azure Synapse Analytics

Loading data to Azure Synapse Analytics using SSMS.

Getting ready

Loading data to Azure Synapse Analytics using Azure Data Factory

Pausing/resuming an Azure SQL pool from Azure Data Factory

Creating an Azure Synapse workspace

Loading data to Azure Synapse Analytics using bulk load

Copying data in Azure Synapse Orchestrate

Using SQL on-demand

Chapter 4: Working with Azure Data Lake

Setting up Azure Data Lake Storage Gen2

Connecting Azure Data Lake to Azure Data Factory and loading data

Creating big data pipelines using Azure Data Lake and Azure Data Factory

How it works

Chapter 5: Working with Big Data - HDInsight and Databricks

Setting up an HDInsight cluster

Processing data from Azure Data Lake with HDInsight and Hive

Processing big data with Apache Spark

Building a machine learning app with Databricks and Azure Data Lake Storage

Chapter 6: Integration with MS SSIS

Creating a SQL Server database

Building an SSIS package

Running SSIS packages from ADF.

Chapter 7: Data Migration - Azure Data Factory and Other Cloud Services

Copying data from Amazon S3 to Azure Blob storage

Copying large datasets from S3 to ADLS

Copying data from Google Cloud Storage to Azure Data Lake

Copying data from Google BigQuery to Azure Data Lake Store

Migrating data from Google BigQuery to Azure Synapse

Moving data to Dropbox

Chapter 8: Working with Azure Services Integration

Triggering your data processing with Logic Apps

Using the web activity to call an Azure logic app

Adding flexibility to your pipelines with Azure Functions

Getting ready...

Automatically building ML models with speed and scale

Transforming and preparing your data via Azure Databricks

Chapter 9: Managing Deployment Processes with Azure DevOps

Setting up Azure DevOps

Publishing changes to ADF

Deploying your features into the master branch

How it works.

Getting ready for the CI/CD of ADF

Creating an Azure pipeline for CD

Chapter 10: Monitoring and Troubleshooting Data Pipelines

Monitoring pipeline runs and integration runtimes

Investigating failures - running in debug mode

Rerunning activities

Configuring alerts for your Data Factory runs

Other Books You May Enjoy

Index.

Notes:

Description based on print version record.

Description based on publisher supplied metadata and other sources.

OCLC:

1232721432