PySpark cookbook : over 60 recipes for implementing big data processing and analytics using Apache Spark and Python / Denny Lee, Tomasz Drabas.

Format:

Book

Author/Creator:

Lee, Denny, author.

Drabas, Tomasz, author.

Language:

English

Subjects (All):

SPARK (Computer program language).

Python (Computer program language).

Application software--Development.

Application software.

Physical Description:

1 online resource (321 pages) : illustrations

Edition:

1st edition

Place of Publication:

Birmingham, UK ; Mumbai : Packt, 2018.

System Details:

text file

Biography/History:

Lee Denny: Denny Lee is a Principal Program Manager at Microsoft for the Azure DocumentDB teamMicrosoft's blazing fast, planet-scale managed document store service. He is a hands-on distributed systems and data science engineer with more than 18 years of experience developing Internet-scale infrastructure, data platforms, and predictive analytics systems for both on-premise and cloud environments. He has extensive experience of building greenfield teams as well as turnaround/ change catalyst. Prior to joining the Azure DocumentDB team, Denny worked as a Technology Evangelist at Databricks; he has been working with Apache Spark since 0. 5. He was also the Senior Director of Data Sciences Engineering at Concur, and was on the incubation team that built Microsoft's Hadoop on Windows and Azure service (currently known as HDInsight). Denny also has a Masters in Biomedical Informatics from Oregon Health and Sciences University and has architected and implemented powerful data solutions for enterprise healthcare customers for the last 15 years. Drabas Tomasz: Tomasz Drabas is a Data Scientist working for Microsoft and currently residing in the Seattle area. He has over 12 years' international experience in data analytics and data science in numerous fields: advanced technology, airlines, telecommunications, finance, and consulting. Tomasz started his career in 2003 with LOT Polish Airlines in Warsaw, Poland while finishing his Master's degree in strategy management. In 2007, he moved to Sydney to pursue a doctoral degree in operations research at the University of New South Wales, School of Aviation; his research crossed boundaries between discrete choice modeling and airline operations research. During his time in Sydney, he worked as a Data Analyst for Beyond Analysis Australia and as a Senior Data Analyst/Data Scientist for Vodafone Hutchison Australia among others. He has also published scientific papers, attended international conferences, and served as a reviewer for scientific journals. In 2015 he relocated to Seattle to begin his work for Microsoft. While there, he has worked on numerous projects involving solving problems in high-dimensional feature space.

Summary:

Combine the power of Apache Spark and Python to build effective big data applications About This Book Perform effective data processing, machine learning, and analytics using PySpark Overcome challenges in developing and deploying Spark solutions using Python Explore recipes for efficiently combining Python and Apache Spark to process data Who This Book Is For The PySpark Cookbook is for you if you are a Python developer looking for hands-on recipes for using the Apache Spark 2.x ecosystem in the best possible way. A thorough understanding of Python (and some familiarity with Spark) will help you get the best out of the book. What You Will Learn Configure a local instance of PySpark in a virtual environment Install and configure Jupyter in local and multi-node environments Create DataFrames from JSON and a dictionary using pyspark.sql Explore regression and clustering models available in the ML module Use DataFrames to transform data used for modeling Connect to PubNub and perform aggregations on streams In Detail Apache Spark is an open source framework for efficient cluster computing with a strong interface for data parallelism and fault tolerance. The PySpark Cookbook presents effective and time-saving recipes for leveraging the power of Python and putting it to use in the Spark ecosystem. You'll start by learning the Apache Spark architecture and how to set up a Python environment for Spark. You'll then get familiar with the modules available in PySpark and start using them effortlessly. In addition to this, you'll discover how to abstract data with RDDs and DataFrames, and understand the streaming capabilities of PySpark. You'll then move on to using ML and MLlib in order to solve any problems related to the machine learning capabilities of PySpark and use GraphFrames to solve graph-processing problems. Finally, you will explore how to deploy your applications to the cloud using the spark-submit command. By the end of this book, you will be able to use the Python API for Apache Spark to solve any problems associated with building data-intensive applications. Style and approach This book is a rich collection of recipes that will come in handy when you are working with PySpark Addressing your common and not-so-common pain points, this is a book that you must have on the shelf.

Contents:

Cover

Title Page

Copyright and Credits

Packt Upsell

Contributors

Table of Contents

Preface

Chapter 1: Installing and Configuring Spark

Introduction

Installing Spark requirements

Getting ready

How to do it...

How it works...

There's more...

Installing Java

Installing Python

Installing R

Installing Scala

Installing Maven

Updating PATH

Installing Spark from sources

See also

Installing Spark from binaries

Configuring a local instance of Spark

Configuring a multi-node instance of Spark

Installing Jupyter

Configuring a session in Jupyter

Working with Cloudera Spark images

Chapter 2: Abstracting Data with RDDs

Creating RDDs

Spark context parallelize method

.take(...) method

Reading data from files

.textFile(...) method

.map(...) method

Partitions and performance

Overview of RDD transformations

.map(...) transformation

.filter(...) transformation

.flatMap(...) transformation

.distinct() transformation

.sample(...) transformation

.join(...) transformation

.repartition(...) transformation

.zipWithIndex() transformation

.reduceByKey(...) transformation.

.sortByKey(...) transformation

.union(...) transformation

.mapPartitionsWithIndex(...) transformation

Overview of RDD actions

.take(...) action

.collect() action

.reduce(...) action

.count() action

.saveAsTextFile(...) action

Pitfalls of using RDDs

Chapter 3: Abstracting Data with DataFrames

Creating DataFrames

From JSON

From CSV

Accessing underlying RDDs

Performance optimizations

Inferring the schema using reflection

Specifying the schema programmatically

Creating a temporary table

Using SQL to interact with DataFrames

Overview of DataFrame transformations

The .select(...) transformation

The .filter(...) transformation

The .groupBy(...) transformation

The .orderBy(...) transformation

The .withColumn(...) transformation

The .join(...) transformation

The .unionAll(...) transformation

The .distinct(...) transformation

The .repartition(...) transformation

The .fillna(...) transformation

The .dropna(...) transformation

The .dropDuplicates(...) transformation

The .summary() and .describe() transformations

The .freqItems(...) transformation

See also.

Overview of DataFrame actions

The .show(...) action

The .collect() action

The .take(...) action

The .toPandas() action

Chapter 4: Preparing Data for Modeling

Handling duplicates

Only IDs differ

ID collisions

Handling missing observations

Missing observations per row

Missing observations per column

Handling outliers

Exploring descriptive statistics

Descriptive statistics for aggregated columns

Computing correlations

Drawing histograms

Visualizing interactions between features

Chapter 5: Machine Learning with MLlib

Loading the data

Exploring the data

Numerical features

Categorical features

Testing the data

See also...

Transforming the data

Standardizing the data

Creating an RDD for training

Classification

Regression

Predicting hours of work for census respondents

Forecasting the income levels of census respondents

Building a clustering models

Computing performance statistics

Regression metrics

Classification metrics

Chapter 6: Machine Learning with the ML Module

Introducing Transformers

Introducing Estimators

Introducing Pipelines

Selecting the most predictable features

Predicting forest coverage types

Estimating forest elevation

Clustering forest cover types

Tuning hyperparameters

Extracting features from text

Discretizing continuous variables

Standardizing continuous variables

Topic mining

Chapter 7: Structured Streaming with PySpark

Understanding Spark Streaming.

Understanding DStreams

Terminal 1 - Netcat window

Terminal 2 - Spark Streaming window

Understanding global aggregations

Continuous aggregation with structured streaming

Chapter 8: GraphFrames - Graph Theory with PySpark

Installing GraphFrames

Preparing the data

Building the graph

Running queries against the graph

Understanding the graph

Using PageRank to determine airport ranking

Finding the fewest number of connections

Visualizing the graph

Index.

Notes:

Description based on print version record.

ISBN:

9781788834254

1788834259

OCLC:

1045023903