Learning concurrency in Python : speed up your Python code with clean, readable, and advanced concurrency techniques / Elliot Forbes.

Format:

Book

Author/Creator:

Forbes, Elliot, author.

Language:

English

Subjects (All):

Python (Computer program language).

Physical Description:

1 online resource (352 pages) : illustrations

Edition:

1st edition

Place of Publication:

Birmingham : Packt, 2017.

System Details:

text file

Biography/History:

Forbes Elliot: Elliot Forbes has worked as a full-time software engineer at a leading financial firm for the last two years. He graduated from the University of Strathclyde in Scotland in the spring of 2015 and worked as a freelancer developing web solutions while studying there. He has worked on numerous different technologies such as Golang, Node. js, and plain old Java, and he has spent years working on concurrent enterprise systems. Elliot has even worked at Barclays Investment Bank for a summer internship in London and has maintained a couple of software development websites for the last three years.

Summary:

Practically and deeply understand concurrency in Python to write efficient programs About This Book Build highly efficient, robust, and concurrent applications Work through practical examples that will help you address the challenges of writing concurrent code Improve the overall speed of execution in multiprocessor and multicore systems and keep them highly available Who This Book Is For This book is for Python developers who would like to get started with concurrent programming. Readers are expected to have a working knowledge of the Python language, as this book will build on these fundamentals concepts. What You Will Learn Explore the concept of threading and multiprocessing in Python Understand concurrency with threads Manage exceptions in child threads Handle the hardest part in a concurrent system - shared resources Build concurrent systems with Communicating Sequential Processes (CSP) Maintain all concurrent systems and master them Apply reactive programming to build concurrent systems Use GPU to solve specific problems In Detail Python is a very high level, general purpose language that is utilized heavily in fields such as data science and research, as well as being one of the top choices for general purpose programming for programmers around the world. It features a wide number of powerful, high and low-level libraries and frameworks that complement its delightful syntax and enable Python programmers to create. This book introduces some of the most popular libraries and frameworks and goes in-depth into how you can leverage these libraries for your own high-concurrent, highly-performant Python programs. We'll cover the fundamental concepts of concurrency needed to be able to write your own concurrent and parallel software systems in Python. The book will guide you down the path to mastering Python concurrency, giving you all the necessary hardware and theoretical knowledge. We'll cover concepts such as debugging and exception handling as well as some of the most popular libraries and frameworks that allow you to create event-driven and reactive systems. By the end of the book, you'll have learned the techniques to write incredibly efficient concurrent systems that follow best practices. Style and approach This easy-to-follow guide teaches you new practices and techniques to optimize your code, and then moves toward more advanced ways to effectively write efficient Python code. Small and simple practical examples will help you test...

Contents:

Cover

Copyright

Credits

About the Author

About the Reviewer

www.PacktPub.com

Customer Feedback

Table of Contents

Preface

Chapter 1: Speed It Up!

History of concurrency

Threads and multithreading

What is a thread?

Types of threads

What is multithreading?

Processes

Properties of processes

Multiprocessing

Event-driven programming

Turtle

Breaking it down

Reactive programming

ReactiveX - RxPy

GPU programming

PyCUDA

OpenCL

Theano

The limitations of Python

Jython

IronPython

Why should we use Python?

Concurrent image download

Sequential download

Concurrent download

Improving number crunching with multiprocessing

Sequential prime factorization

Concurrent prime factorization

Summary

Chapter 2: Parallelize It

Understanding concurrency

Properties of concurrent systems

I/O bottlenecks

Understanding parallelism

CPU-bound bottlenecks

How do they work on a CPU?

Single-core CPUs

Clock rate

Martelli model of scalability

Time-sharing - the task scheduler

Multi-core processors

System architecture styles

SISD

SIMD

MISD

MIMD

Computer memory architecture styles

UMA

NUMA

Chapter 3: Life of a Thread

Threads in Python

Thread state

State flow chart

Python example of thread state

Different types of threads

POSIX threads

Windows threads

The ways to start a thread

Starting a thread

Inheriting from the thread class

Forking

Example

Daemonizing a thread

Handling threads in Python

Starting loads of threads

Breaking it down.

Slowing down programs using threads

Getting the total number of active threads

Getting the current thread

Main thread

Enumerating all threads

Identifying threads

Breakdown

Ending a thread

Best practice in stopping threads

Output

Orphan processes

How does the operating system handle threads

Creating processes versus threads

Multithreading models

One-to-one thread mapping

Many-to-one

Many-to-many

Chapter 4: Synchronization between Threads

Synchronization between threads

The Dining Philosophers

Race conditions

Process execution sequence

The solution

Critical sections

Filesystem

Life-critical systems

Shared resources and data races

The join method

Putting it together

Locks

RLocks

RLocks versus regular locks

Condition

Definition

Our publisher

Our subscriber

Kicking it off

The results

Semaphores

Class definition

The TicketSeller class

Thread race

Bounded semaphores

Events

Barriers

Chapter 5: Communication between Threads

Standard data structures

Sets

Extending the class

Exercise - extending other primitives

Decorator

Class decorator

Lists

Queues

FIFO queues

LIFO queues

PriorityQueue

Queue objects

Full/empty queues

Example.

Output

The join() function

Deque objects

Appending elements

Popping elements

Inserting elements

Rotation

Defining your own thread-safe communication structures

A web Crawler example

Requirements

Design

Our Crawler class

Our starting point

Extending the queue object

Future enhancements

Conclusion

Exercise - testing your skills

Chapter 6: Debug and Benchmark

Testing strategies

Why do we test?

Testing concurrent software systems

What should we test?

Unit tests

PyUnit

Expanding our test suite

Unit testing concurrent code

Integration tests

Debugging

Make it work as a single thread

Pdb

An interactive example

Catching exceptions in child threads

Benchmarking

The timeit module

Timeit versus time

Command-line example

Importing timeit into your code

Utilizing decorators

Timing context manager

Profiling

cProfile

Simple profile example

The line_profiler tool

Kernprof

Memory profiling

Memory profile graphs

Chapter 7: Executors and Pools

Concurrent futures

Executor objects

Creating a ThreadPoolExecutor

Context manager

Maps

Shutdown of executor objects

Future objects

Methods in future objects

The result() method

The add_done_callback() method

The .running() method

The cancel() method

The .exception() method

The .done() method

Unit testing future objects.

The set_running_or_notify_cancel() method

The set_result() method

The set_exception() method

Cancelling callable

Getting the result

Using as_completed

Setting callbacks

Chaining callbacks

Exception classes

ProcessPoolExecutor

Creating a ProcessPoolExecutor

Context Manager

Exercise

Getting started

Improving the speed of computationally bound problems

Full code sample

Improving our crawler

The plan

New improvements

Refactoring our code

Storing the results in a CSV file

Exercise - capture more info from each page crawl

concurrent.futures in Python 2.7

Chapter 8: Multiprocessing

Working around the GIL

Utilizing sub-processes

The life of a process

Starting a process using fork

Spawning a process

Forkserver

Daemon processes

Identifying processes using PIDs

Terminating a process

Getting the current process

Subclassing processes

Multiprocessing pools

The difference between concurrent.futures.ProcessPoolExecutor and Pool

Submitting tasks to a process pool

Apply

Apply_async

Map

Map_async

Imap

Imap_unordered

Starmap

Starmap_async

Maxtasksperchild

Communication between processes

Pipes

Anonymous pipes

Named pipes

Working with pipes

Handling Exceptions

Using pipes

Multiprocessing managers

Namespaces

Listeners and clients

The Listener class

The Client class

Logging

Communicating sequential processes

PyCSP

Processes in PyCSP

Chapter 9: Event-Driven Programming

The event loop

Asyncio

Event loops

The run_forever() method

The run_until_complete() method

The stop() method

The is_closed() method

The close() function

Tasks

The all_tasks(loop=None) method

The current_tasks() function

The cancel() function

Task functions

The as_completed(fs, *, loop=

The ensure_future(coro_or_future, *, loop=

The wrap_future(future, *, loop=

The gather(*coroes_or_futures, loop=

The wait() function

Futures

Coroutines

Chaining coroutines

Transports

Protocols

Synchronization between coroutines

Events and conditions

Semaphores and BoundedSemaphores

Sub-processes

Debugging asyncio programs

Debug mode

Twisted

A simple web server example

Gevent

Greenlets

Simple example-hostnames

Monkey patching

Chapter 10: Reactive Programming

Basic reactive programming

Maintaining purity

ReactiveX, or RX

Installing RxPY

Observables

Creating observers

Example 2

Lambda functions

On_next, on_completed, and on_error in lambda form

Operators and chaining

Filter example

Chained operators

The different operators

Creating observables

Transforming observables

Filtering observables

Error-handling observables

Hot and cold observables

Emitting events

Multicasting

Combining observables

Zip() example

The merge_all() operator

Concurrency.

Example.

Notes:

Includes index.

Description based on online resource; title from PDF title page (ebrary, viewed August 31, 2017).

OCLC:

1004395221