Python web scraping cookbook : over 90 proven recipes to get you scraping with Python, microservices, Docker, and AWS / Michael Heydt.

Format:

Book

Author/Creator:

Heydt, Michael, author.

Language:

English

Subjects (All):

Python (Computer program language).

Web usage mining.

Cloud computing.

Physical Description:

1 online resource (364 pages)

Edition:

1st edition

Other Title:

Over 90 proven recipes to get you scraping with Python, microservices, Docker, and AWS

Place of Publication:

Birmingham, England ; Mumbai, [India] : Packt, 2018.

System Details:

text file

Summary:

Untangle your web scraping complexities and access web data with ease using Python scripts About This Book Hands-on recipes for advancing your web scraping skills to expert level. One-Stop Solution Guide to address complex and challenging web scraping tasks using Python. Understand the web page structure and collect meaningful data from the website with ease Who This Book Is For This book is ideal for Python programmers, web administrators, security professionals or someone who wants to perform web analytics would find this book relevant and useful. Familiarity with Python and basic understanding of web scraping would be useful to take full advantage of this book. What You Will Learn Use a wide variety of tools to scrape any website and data—including BeautifulSoup, Scrapy, Selenium, and many more Master expression languages such as XPath, CSS, and regular expressions to extract web data Deal with scraping traps such as hidden form fields, throttling, pagination, and different status codes Build robust scraping pipelines with SQS and RabbitMQ Scrape assets such as images media and know what to do when Scraper fails to run Explore ETL techniques of build a customized crawler, parser, and convert structured and unstructured data from websites Deploy and run your scraper-as-aservice in AWS Elastic Container Service In Detail Python Web Scraping Cookbook is a solution-focused book that will teach you techniques to develop high-performance scrapers and deal with crawlers, sitemaps, forms automation, Ajax-based sites, caches, and more.You'll explore a number of real-world scenarios where every part of the development/product life cycle will be fully covered. You will not only develop the skills to design and develop reliable, performance data flows, but also deploy your codebase to an AWS. If you are involved in software engineering, product development, or data mining (or are interested in building data-driven products), you will find this book useful as each recipe has a clear purpose and objective. Right from extracting data from the websites to writing a sophisticated web crawler, the book's independent recipes will be a godsend on the job. This book covers Python libraries, requests, and BeautifulSoup. You will learn about crawling, web spidering, working with AJAX websites, paginated items, and more. You will also learn to tackle problems such as 403 errors, working with proxy, scraping images, LXML, and more. By the end of this book, you will...

Contents:

Cover

Copyright and Credits

Contributors

Packt Upsell

Table of Contents

Preface

Chapter 1: Getting Started with Scraping

Introduction

Setting up a Python development environment

Getting ready

How to do it...

Scraping Python.org with Requests and Beautiful Soup

Getting ready...

How it works...

Scraping Python.org in urllib3 and Beautiful Soup

How it works

There's more...

Scraping Python.org with Scrapy

Scraping Python.org with Selenium and PhantomJS

Chapter 2: Data Acquisition and Extraction

How to parse websites and navigate the DOM using BeautifulSoup

Searching the DOM with Beautiful Soup's find methods

Querying the DOM with XPath and lxml

Querying data with XPath and CSS selectors

Using Scrapy selectors

Loading data in unicode / UTF-8

Chapter 3: Processing Data

Working with CSV and JSON data

How to do it

Storing data using AWS S3

Storing data using MySQL

Storing data using PostgreSQL

There's more.

Storing data in Elasticsearch

How to build robust ETL pipelines with AWS SQS

How to do it - posting messages to an AWS queue

How to do it - reading and processing messages

Chapter 4: Working with Images, Audio, and other Assets

Downloading media content from the web

Parsing a URL with urllib to get the filename

Determining the type of content for a URL

Determining the file extension from a content type

Downloading and saving images to the local file system

Downloading and saving images to S3

Generating thumbnails for images

Taking a screenshot of a website

Taking a screenshot of a website with an external service

Performing OCR on an image with pytesseract

Creating a Video Thumbnail

There's more..

Ripping an MP4 video to an MP3

Chapter 5: Scraping - Code of Conduct

Scraping legality and scraping politely

Respecting robots.txt

How it works.

There's more...

Crawling using the sitemap

Crawling with delays

Using identifiable user agents

Setting the number of concurrent requests per domain

Using auto throttling

Using an HTTP cache for development

Chapter 6: Scraping Challenges and Solutions

Retrying failed page downloads

Supporting page redirects

Waiting for content to be available in Selenium

Limiting crawling to a single domain

Processing infinitely scrolling pages

Controlling the depth of a crawl

Controlling the length of a crawl

Handling paginated websites

Handling forms and forms-based authorization

Handling basic authorization

Preventing bans by scraping via proxies

Randomizing user agents

Caching responses

Chapter 7: Text Wrangling and Analysis

Installing NLTK

Performing sentence splitting

Performing tokenization

Performing stemming

Performing lemmatization.

How to do it

Determining and removing stop words

Calculating the frequency distributions of words

Identifying and removing rare words

Removing punctuation marks

Piecing together n-grams

Scraping a job listing from StackOverflow

Reading and cleaning the description in the job listing

Chapter 8: Searching, Mining and Visualizing Data

Geocoding an IP address

How to collect IP addresses of Wikipedia edits

Visualizing contributor location frequency on Wikipedia

Creating a word cloud from a StackOverflow job listing

Crawling links on Wikipedia

Theres more...

Visualizing page relationships on Wikipedia

Calculating degrees of separation

Chapter 9: Creating a Simple Data API

Creating a REST API with Flask-RESTful

Integrating the REST API with scraping code

Adding an API to find the skills for a job listing

Storing data in Elasticsearch as the result of a scraping request

Checking Elasticsearch for a listing before scraping

Chapter 10: Creating Scraper Microservices with Docker

Installing Docker

Installing a RabbitMQ container from Docker Hub

Running a Docker container (RabbitMQ)

Creating and running an Elasticsearch container

Stopping/restarting a container and removing the image

Creating a generic microservice with Nameko

Creating a scraping microservice

Creating a scraper container

Creating an API container

Composing and running the scraper locally with docker-compose

Chapter 11: Making the Scraper as a Service Real

Creating and configuring an Elastic Cloud trial account

Accessing the Elastic Cloud cluster with curl

Connecting to the Elastic Cloud cluster with Python

Performing an Elasticsearch query with the Python API

Using Elasticsearch to query for jobs with specific skills

Modifying the API to search for jobs by skill

Storing configuration in the environment

Creating an AWS IAM user and a key pair for ECS

Configuring Docker to authenticate with ECR

Pushing containers into ECR

Creating an ECS cluster

How to do it.

Creating a task to run our containers.

Notes:

Includes bibliographical references and index.

Description based on online resource; title from PDF title page (EBC, viewed March 8, 2018).

OCLC:

1024273826