Stream Analytics with Microsoft Azure : real-time processing for quick insights using Azure Stream Analytics / Anindita Basak [and three others].

Format:

Book

Author/Creator:

Basak, Anindita, author.

Language:

English

Subjects (All):

Windows Azure.

Real-time data processing.

Physical Description:

1 online resource (1 volume) : illustrations

Edition:

1st edition

Place of Publication:

Birmingham, England ; Mumbai, [India] : Packt, 2017.

System Details:

text file

Summary:

Develop and manage effective real-time streaming solutions by leveraging the power of Microsoft Azure About This Book Analyze your data from various sources using Microsoft Azure Stream Analytics Develop, manage and automate your stream analytics solution with Microsoft Azure A practical guide to real-time event processing and performing analytics on the cloud Who This Book Is For If you are looking for a resource that teaches you how to process continuous streams of data in real-time, this book is what you need. A basic understanding of the concepts in analytics is all you need to get started with this book What You Will Learn Perform real-time event processing with Azure Stream Analysis Incorporate the features of Big Data Lambda architecture pattern in real-time data processing Design a streaming pipeline for storage and batch analysis Implement data transformation and computation activities over stream of events Automate your streaming pipeline using Powershell and the .NET SDK Integrate your streaming pipeline with popular Machine Learning and Predictive Analytics modelling algorithms Monitor and troubleshoot your Azure Streaming jobs effectively In Detail Microsoft Azure is a very popular cloud computing service used by many organizations around the world. Its latest analytics offering, Stream Analytics, allows you to process and get actionable insights from different kinds of data in real-time. This book is your guide to understanding the basics of how Azure Stream Analytics works, and building your own analytics solution using its capabilities. You will start with understanding what Stream Analytics is, and why it is a popular choice for getting real-time insights from data. Then, you will be introduced to Azure Stream Analytics, and see how you can use the tools and functions in Azure to develop your own Streaming Analytics. Over the course of the book, you will be given comparative analytic guidance on using Azure Streaming with other Microsoft Data Platform resources such as Big Data Lambda Architecture integration for real time data analysis and differences of scenarios for architecture designing with Azure HDInsight Hadoop clusters with Storm or Stream Analytics. The book also shows you how you can manage, monitor, and scale your solution for optimal performance. By the end of this book, you will be well-versed in using Azure Stream Analytics to develop an efficient analytics solution that can work with any type of data. Style and...

Contents:

Cover

Copyright

Credits

About the Authors

About the Reviewers

www.PacktPub.com

Customer Feedback

Table of Contents

Preface

Chapter 1: Introducing Stream Processing and Real-Time Insights

Understanding stream processing

Understanding queues, Pub/Sub, and events

Queues

Publish and Subscribe model

Real-world implementations of the Publish/Subscribe model

Azure implementation of queues and Publish/Subscribe models

What is an event?

Event streaming

Event correlation

Azure implementation of event processing

Architectural components of Event Hubs

Simple event processing

Event stream processing

Complex event processing

Summary

Chapter 2: Introducing Azure Stream Analytics and Key Advantages

Services offered by Microsoft

Introduction to Azure Stream Analytics

Configuration of Azure Stream Analytics

Key advantages of Azure Stream Analytics

Security

Programmer productivity

Declarative SQL constructs

Built-in temporal semantics

Lowest total cost of ownership

Mission-critical and enterprise-less scalability and availability

Global compliance

Microsoft Cortana Intelligence suite integration

Azure IoT integration

Chapter 3: Designing Real-Time Streaming Pipelines

Differencing stream processing and batch processing

Logical flow of processing

Out of order and late arrival of data

Session grouping and windowing challenges

Message consistency

Fault tolerance, recovery, and storage

Source

Communication and collection

Ingest, queue, and transform

Hot path

Cold path

Data retention

Presentation and action

Canonical Azure architecture

Chapter 4: Developing Real-Time Event Processing with Azure Streaming

Stream Analytics tools for Visual Studio.

Prerequisites for the installation of Stream Analytics tools

Development of a Stream Analytics job using Visual Studio

Defining a Stream Analytics query for Vehicle Telemetry job analysis using Stream Analytics tools

Query to define Vehicle Telemetry (Connected Car) engine health status and pollution index over cities

Testing Stream Analytics queries locally or in the cloud

Stream Analytics job configuration parameter settings in Visual Studio

Implementation of an Azure Stream Analytics job using the Azure portal

Provisioning for an Azure Stream Analytics job using the Azure Resource Manager template

Azure ARM Template - Infrastructure as code

Getting started with provisioning Azure Stream Analytics job using the ARM template

Deployment and validation of the Stream Analytics ARM template to Azure Resource Group

Configuration of the Azure Streaming job with different input data sources and output data sinks

Data input types-data stream and reference data

Data Stream inputs

Reference data

Job topology output data sinks of Stream Analytics

Chapter 5: Building Using Stream Analytics Query Language

Built-in functions

Scalar functions

Aggregate and analytic functions

Array functions

Other functions

Data types and formats

Complex types

Query language elements

Windowing

Tumbling windows

Hopping windows

Sliding windows

Time management and event delivery guarantees

Chapter 6: How to achieve Seamless Scalability with Automation

Understanding parts of a Stream Analytics job definition (input, output, reference data, and job)

Deployment of Azure Stream Analytics using ARM template

Configuring input

Configuring output

Building the sample test code

How to scale queries using Streaming units and partitions

Application and Arrival Time.

Partitions

Input source

Output source

Embarrassingly parallel jobs and Not embarrassingly parallel jobs

Sample use case

Configuring SU using Azure portal

Out of order and late-arriving events

Chapter 7: Integration of Microsoft Business Intelligence and Big Data

What is Big Data Lambda Architecture?

Concepts of batch processing and stream processing in data analytics

Specifications for slow/cold path of data - batch data processing

Moving to the streaming-based data solution pattern

Evolution of Kappa Architecture and benefits

Comparison between Azure Stream Analytics and Azure HDInsight Storm

Designing data processing pipeline of an interactive visual dashboard through Stream Analytics and Power BI

Integrating Power BI as an output job connector for Stream Analytics

Chapter 8: Designing and Managing Stream Analytics Jobs

Reference data streams with Azure Stream Analytics

Configuration of Reference data for Azure Stream Analytics jobs

Integrating a reference data stream as job topology input for an Azure Stream Analytics job

Stream Analytics query configuration for Reference Data join

Refresh schedule of a reference data stream

Configuration of output data sinks for Azure Stream Analytics with Azure Data Lake Store

Configuring Azure Data Lake Store as an output data sink of Stream Analytics

Configuring Azure Data Lake Store as an output sink of Stream Analytics jobs

Configuring Azure Cosmos DB as an output data sink for Azure Stream Analytics

Features of Azure Cosmos DB for configuring output sinks of Azure Stream Analytics

Configuring Azure Cosmos DB integrated with Azure Stream Analytics as an output sink

Stream Analytics job output to Azure Function Apps as Serverless Architecture

Provisioning steps to an Azure Function.

Configuring an Azure function as a serverless architecture model integrated with Stream Analytics job output

Chapter 9: Optimizing Intelligence in Azure Streaming

Integration of JavaScript user-defined functions using Azure Stream Analytics

Adding JavaScript UDF with a Stream Analytics job

Stream Analytics and JavaScript data type conversions

Integrating intelligent Azure machine learning algorithms with Stream Analytics function

Data pipeline Streaming application building concepts using Azure .NET Management SDK

Implementation steps of Azure Stream Analytics jobs using .NET management SDK

Chapter 10: Understanding Stream Analytics Job Monitoring

Troubleshooting with job metrics

Visual monitoring of job diagram

Logging of diagnostics logs

Enabling diagnostics logs

Exploring the logs sent to the storage account

Configuring job alerts

Viewing resource health information with Azure resource health

Exploring different monitoring experiences

Building a monitoring dashboard

Chapter 11: Use Cases for Real-World Data Streaming Architectures

Solution architecture design and Proof-of-Concept implementation of social media sentiment analytics using Twitter and a sentiment analytics dashboard

Definition of sentiment analytics

Prerequisites required for the implementation of Twitter sentiment analytics PoC

Steps for implementation of Twitter sentiment analytics

Remote monitoring analytics using Azure IoT Suite

Provisioning of remote device monitoring analytics using Azure IoT Suite

Implementation of a connected factory use case using Azure IoT Suite

Connected factory solution with Azure IoT Suite

Real-world telecom fraud detection analytics using Azure Stream Analytics and Cortana Intelligence Gallery with interactive visuals from Microsoft Power BI.

Implementation steps of fraud detection analytics using Azure Stream Analytics

Steps for building the fraud detection analytics solution

Index.

Notes:

Description based on online resource; title from PDF title page (EBC, viewed January 4, 2018).

ISBN:

9781788390620

1788390628

OCLC:

1019129068