Natural language processing with TensorFlow : teach language to machines using Python's deep learning library / Thushan Ganegedara.

Format:

Book

Author/Creator:

Ganegedara, Thushan, author.

Language:

English

Subjects (All):

Python (Computer program language).

Machine learning.

Artificial intelligence.

Physical Description:

1 online resource (472 pages)

Edition:

1st ed.

Place of Publication:

Birmingham : Packt, 2018.

Summary:

TensorFlow is the leading framework for deep learning algorithms critical to artificial intelligence, and natural language processing (NLP) makes much of the data used by deep learning applications accessible to them. This book brings the two together and teaches deep learning developers how to work with today's vast amount of unstructured data.

Contents:

Cover

Copyright

Packt Upsell

Contributors

Table of Contents

Preface

Chapter 1: Introduction to Natural Language Processing

What is Natural Language Processing?

Tasks of Natural Language Processing

The traditional approach to Natural Language Processing

Understanding the traditional approach

Example - generating football game summaries

Drawbacks of the traditional approach

The deep learning approach to Natural Language Processing

History of deep learning

The current state of deep learning and NLP

Understanding a simple deep model - a Fully Connected Neural Network

The roadmap - beyond this chapter

Introduction to the technical tools

Description of the tools

Installing Python and scikit-learn

Installing Jupyter Notebook

Installing TensorFlow

Summary

Chapter 2: Understanding TensorFlow

What is TensorFlow?

Getting started with TensorFlow

TensorFlow client in detail

TensorFlow architecture - what happens when you execute the client?

Cafe Le TensorFlow - understanding TensorFlow with an analogy

Inputs, variables, outputs, and operations

Defining inputs in TensorFlow

Feeding data with Python code

Preloading and storing data as tensors

Building an input pipeline

Defining variables in TensorFlow

Defining TensorFlow outputs

Defining TensorFlow operations

Comparison operations

Mathematical operations

Scatter and gather operations

Neural network-related operations

Reusing variables with scoping

Implementing our first neural network

Preparing the data

Defining the TensorFlow graph

Running the neural network

Chapter 3: Word2vec - Learning Word Embeddings

What is a word representation or meaning?

Classical approaches to learning word representation.

WordNet - using an external lexical knowledge base for learning word representations

Tour of WordNet

Problems with WordNet

One-hot encoded representation

The TF-IDF method

Co-occurrence matrix

Word2vec - a neural network-based approach to learning word representation

Exercise: is queen = king - he + she?

Designing a loss function for learning word embeddings

The skip-gram algorithm

From raw text to structured data

Learning the word embeddings with a neural network

Formulating a practical loss function

Efficiently approximating the loss function

Implementing skip-gram with TensorFlow

The Continuous Bag-of-Words algorithm

Implementing CBOW in TensorFlow

Chapter 4: Advanced Word2vec

The original skip-gram algorithm

Implementing the original skip-gram algorithm

Comparing the original skip-gram with the improved skip-gram

Comparing skip-gram with CBOW

Performance comparison

Which is the winner, skip-gram or CBOW?

Extensions to the word embeddings algorithms

Using the unigram distribution for negative sampling

Implementing unigram-based negative sampling

Subsampling - probabilistically ignoring the common words

Implementing subsampling

Comparing the CBOW and its extensions

More recent algorithms extending skip-gram and CBOW

A limitation of the skip-gram algorithm

The structured skip-gram algorithm

The loss function

The continuous window model

GloVe - Global Vectors representation

Understanding GloVe

Implementing GloVe

Document classification with Word2vec

Dataset

Classifying documents with word embeddings

Implementation - learning word embeddings

Implementation - word embeddings to document embeddings

Document clustering and t-SNE visualization of embedded documents

Inspecting several outliers.

Implementation - clustering/classification of documents with K-means

Chapter 5: Sentence Classification with Convolutional Neural Networks

Introducing Convolution Neural Networks

CNN fundamentals

The power of Convolution Neural Networks

Understanding Convolution Neural Networks

Convolution operation

Standard convolution operation

Convolving with stride

Convolving with padding

Transposed convolution

Pooling operation

Max pooling

Max pooling with stride

Average pooling

Fully connected layers

Putting everything together

Exercise - image classification on MNIST with CNN

About the data

Implementing the CNN

Analyzing the predictions produced with a CNN

Using CNNs for sentence classification

CNN structure

Data transformation

The convolution operation

Pooling over time

Implementation - sentence classification with CNNs

Chapter 6: Recurrent Neural Networks

Understanding Recurrent Neural Networks

The problem with feed-forward neural networks

Modeling with Recurrent Neural Networks

Technical description of a Recurrent Neural Network

Backpropagation Through Time

How backpropagation works

Why we cannot use BP directly for RNNs

Backpropagation Through Time - training RNNs

Truncated BPTT - training RNNs efficiently

Limitations of BPTT - vanishing and exploding gradients

Applications of RNNs

One-to-one RNNs

One-to-many RNNs

Many-to-one RNNs

Many-to-many RNNs

Generating text with RNNs

Defining hyperparameters

Unrolling the inputs over time for Truncated BPTT

Defining the validation dataset

Defining weights and biases

Defining state persisting variables

Calculating the hidden states and outputs with unrolled inputs

Calculating the loss.

Resetting state at the beginning of a new segment of text

Calculating validation output

Calculating gradients and optimizing

Outputting a freshly generated chunk of text

Evaluating text results output from the RNN

Perplexity - measuring the quality of the text result

Recurrent Neural Networks with Context Features - RNNs with longer memory

Technical description of the RNN-CF

Implementing the RNN-CF

Defining the RNN-CF hyperparameters

Defining input and output placeholders

Defining weights of the RNN-CF

Variables and operations for maintaining hidden and context states

Calculating output

Calculating the loss

Computing test output

Computing the gradients and optimizing

Text generated with the RNN-CF

Chapter 7: Long Short-Term Memory Networks

Understanding Long Short-Term Memory Networks

What is an LSTM?

LSTMs in more detail

How LSTMs differ from standard RNNs

How LSTMs solve the vanishing gradient problem

Improving LSTMs

Greedy sampling

Beam search

Using word vectors

Bidirectional LSTMs (BiLSTM)

Other variants of LSTMs

Peephole connections

Gated Recurrent Units

Chapter 8: Applications of LSTM - Generating Text

Our data

About the dataset

Preprocessing data

Implementing an LSTM

Defining parameters

Defining an LSTM cell and its operations

Defining inputs and labels

Defining sequential calculations required to process sequential data

Defining the optimizer

Decaying learning rate over time

Making predictions

Calculating perplexity (loss)

Resetting states

Greedy sampling to break unimodality

Generating new text

Example generated text

Comparing LSTMs to LSTMs with peephole connections and GRUs

Standard LSTM

Review.

Example generated text

Gated Recurrent Units (GRUs)

Review

The code

LSTMs with peepholes

Training and validation perplexities over time

Improving LSTMs - beam search

Implementing beam search

Examples generated with beam search

Improving LSTMs - generating text with words instead of n-grams

The curse of dimensionality

Word2vec to the rescue

Generating text with Word2vec

Examples generated with LSTM-Word2vec and beam search

Perplexity over time

Using the TensorFlow RNN API

Chapter 9: Applications of LSTM - Image Caption Generation

Getting to know the data

ILSVRC ImageNet dataset

The MS-COCO dataset

The machine learning pipeline for image caption generation

Extracting image features with CNNs

Implementation - loading weights and inferencing with VGG-16

Building and updating variables

Preprocessing inputs

Inferring VGG-16

Extracting vectorized representations of images

Predicting class probabilities with VGG-16

Learning word embeddings

Preparing captions for feeding into LSTMs

Generating data for LSTMs

Defining the LSTM

Evaluating the results quantitatively

BLEU

ROUGE

METEOR

CIDEr

BLEU-4 over time for our model

Captions generated for test images

Using TensorFlow RNN API with pretrained GloVe word vectors

Loading GloVe word vectors

Cleaning data

Using pretrained embeddings with TensorFlow RNN API

Defining the pretrained embedding layer and the adaptation layer

Defining the LSTM cell and softmax layer

Defining inputs and outputs

Processing images and text differently

Defining the LSTM output calculation

Defining the logits and predictions

Defining the sequence loss

Summary.

Chapter 10: Sequence-to-Sequence Learning - Neural Machine Translation.

Notes:

Description based on print version record.

ISBN:

9781788477758

1788477758

OCLC:

1039700926