Mastering Transformers : The Journey from BERT to Large Language Models and Stable Diffusion / Savas Yildirim and Meysam Asgari-Chenaghlu.

Format:

Book

Author/Creator:

Yıldırım, Savaş, author.

Asgari-Chenaghlu, Meysam, author.

Language:

English

Subjects (All):

Natural language processing (Computer science).

Physical Description:

1 online resource (462 pages)

Edition:

Second edition.

Place of Publication:

Birmingham, England : Packt Publishing, [2024]

Biography/History:

Yldrm Savas: Savas Yldrm graduated from the Istanbul Technical University Department of Computer Engineering and holds a Ph. D. degree in Natural Language Processing (NLP). Currently, he is an associate professor at the Istanbul Bilgi University, Turkey, and is a visiting researcher at the Ryerson University, Canada. He is a proactive lecturer and researcher with more than 20 years of experience teaching courses on machine learning, deep learning, and NLP. He has significantly contributed to the Turkish NLP community by developing a lot of open source software and resources. He also provides comprehensive consultancy to AI companies on their R&D projects. In his spare time, he writes and directs short films, and enjoys practicing yoga. Chenaghlu Meysam Asgari-: Meysam Asgari-Chenaghlu is an AI manager at Carbon Consulting and is also a Ph. D. candidate at the University of Tabriz. He has been a consultant for Turkey's leading telecommunication and banking companies. He has also worked on various projects, including natural language understanding and semantic search.

Summary:

Explore transformer-based language models from BERT to GPT, delving into NLP and computer vision tasks, while tackling challenges effectively Key Features Understand the complexity of deep learning architecture and transformers architecture Create solutions to industrial natural language processing (NLP) and computer vision (CV) problems Explore challenges in the preparation process, such as problem and language-specific dataset transformation Purchase of the print or Kindle book includes a free PDF eBook Book Description Transformer-based language models such as BERT, T5, GPT, DALL-E, and ChatGPT have dominated NLP studies and become a new paradigm. Thanks to their accurate and fast fine-tuning capabilities, transformer-based language models have been able to outperform traditional machine learning-based approaches for many challenging natural language understanding (NLU) problems. Aside from NLP, a fast-growing area in multimodal learning and generative AI has recently been established, showing promising results. Mastering Transformers will help you understand and implement multimodal solutions, including text-to-image. Computer vision solutions that are based on transformers are also explained in the book. You'll get started by understanding various transformer models before learning how to train different autoregressive language models such as GPT and XLNet. The book will also get you up to speed with boosting model performance, as well as tracking model training using the TensorBoard toolkit. In the later chapters, you'll focus on using vision transformers to solve computer vision problems. Finally, you'll discover how to harness the power of transformers to model time series data and for predicting. By the end of this transformers book, you'll have an understanding of transformer models and how to use them to solve challenges in NLP and CV. What you will learn Focus on solving simple-to-complex NLP problems with Python Discover how to solve classification/regression problems with traditional NLP approaches Train a language model and explore how to fine-tune models to the downstream tasks Understand how to use transformers for generative AI and computer vision tasks Build transformer-based NLP apps with the Python transformers library Focus on language generation such as machine translation and conversational AI in any language Speed up transformer model inference to reduce latency Who this book is for This book is for deep learning researchers, hands-on practitioners, and ML/NLP researchers. Educators, as well as students who have a good command of programming subjects, knowledge in the field of machine learning and artificial intelligence, and who want to develop apps in the field of NLP as well as multimodal tasks will also benefit from this book's hands-on approach. Knowledge of Python (or any programming language) and machine learning literature, as well as a basic understanding of computer science, are required.

Contents:

Cover

Title page

Copyright and Credits

Contributors

Table of Contents

Preface

Part 1: Recent Developments in the Field, Installations, and Hello World Applications

Chapter 1: From Bag-of-Words to the Transformers

Evolution of NLP approaches

Recalling traditional NLP approaches

Language modeling and generation

Leveraging DL

Considering the word order with RNN models

LSTMs and gated recurrent units

Contextual word embeddings and TL

Overview of the Transformer architecture

Attention mechanism

Multi-head attention mechanisms

Using TL with Transformers

Multimodal learning

Summary

References

Chapter 2: A Hands-On Introduction to the Subject

Technical requirements

Installing transformer with Anaconda

Installation on Linux

Installation on Windows

Installation on macOS

Installing TensorFlow, PyTorch, and Transformer

Installing and using Google Colab

Working with language models and tokenizers

Working with community-provided models

Working with multimodal transformers

Working with benchmarks and datasets

Important benchmarks

GLUE benchmark

SuperGLUE benchmark

XTREME benchmark

XGLUE benchmark

SQuAD benchmark

Accessing the datasets with an application programming interface

Data manipulation using the datasets library

Sorting, indexing, and shuffling

Caching and reusability

Dataset filter and map function

Processing data with the map function

Working with local files

Preparing a dataset for model training

Benchmarking for speed and memory

Part 2: Transformer Models: From Autoencoders to Autoregressive Models

Chapter 3: Autoencoding Language Models

BERT - one of the autoencoding language models

BERT language model pretraining tasks.

A deeper look into the BERT language model

Autoencoding language model training for any language

Sharing models with the community

Other autoencoding models

Introducing ALBERT

RoBERTa

ELECTRA

DeBERTa

Working with tokenization algorithms

BPE

WordPiece tokenization

Sentence piece tokenization

The tokenizers library

Chapter 4: From Generative Models to Large Language Models

An introduction to GLMs

Working with GLMs

GPT model family

Transformer-XL

XLNet

Working with text-to-text models

Multi-task learning with T5

Zero-Shot Text Generalization with T0

Another Denoising-Based Seq2Seq Model - BART

GLM training

NLG using AR models

Chapter 5: Fine-Tuning Language Models for Text Classification

Introduction to text classification

Fine-tuning a BERT model for single-sentence binary classification

Training a classification model with native PyTorch

Fine-tuning BERT for multi-class classification with custom datasets

Fine-tuning the BERT model for sentence-pair regression

Multilabel text classification

Utilizing run_glue.py to fine-tune the models

Chapter 6: Fine-Tuning Language Models for Token Classification

Introduction to token classification

Understanding NER

Understanding POS tagging

Understanding QA

Fine-tuning language models for NER

Question answering using token classification

Question answering for many tasks

Chapter 7: Text Representation

Introduction to sentence embeddings

Cross-encoder versus bi-encoder

Benchmarking sentence similarity models

Using BART for zero-shot learning

Semantic similarity experiment with FLAIR.

Average word embeddings

RNN-based document embeddings

Transformer-based BERT embeddings

SBERT embeddings

Text clustering with Sentence-BERT

Topic modeling with BERTopic

Semantic search with SBERT

Instruction fine-tuned embedding models

Further reading

Chapter 8: Boosting Model Performance

Improving performance with data augmentation

Character-level augmentation

Word-level augmentation

Sentence-level augmentation

Boosting IMDB text classification with augmentation

Adapting the model to the domain

Optimizing the parameters with HPO

Chapter 9: Parameter Efficient Fine-Tuning

Introduction to PEFT

Understanding Types of PEFT

Additive methods

Selective methods

Low-rank fine-tuning

Hands-on PEFT experiments

Fine-tuning a BERT checkpoint with adapter tuning

Efficiently fine-tune FLAN-T5 for an NLI task with Lora

Tuning with QLoRA

Part 3: Advanced Topics

Chapter 10: Large Language Models

Why large language models?

Importance of reward function

The instruction-following ability of LLMs

Fine-tuning large language models

Chapter 11: Explainable AI (XAI) in NLP

Interpreting attention heads

Visualizing attention heads with exBERT

Multiscale visualization of attention heads with BertViz

Understanding the inner parts of BERT with probing classifiers

Explain the model decision

Interpret Transformers' decision with LIME

Interpret Transformers' decision with SHAP

Chapter 12: Working with Efficient Transformers

Introduction to efficient, light, and fast transformers

Implementation for model size reduction.

Working with DistilBERT for knowledge distillation

Pruning transformers

Quantization

Working with efficient self-attention

Sparse attention with fixed patterns

Learnable patterns

Low-rank factorization, kernel methods, and other approaches

Easier quantization using bitsandbytes

Chapter 13: Cross-Lingual and Multilingual Language Modeling

Translation language modeling and cross-lingual knowledge sharing

XLM and mBERT

mBERT

XLM

Cross-lingual similarity tasks

Cross-lingual text similarity

Visualizing cross-lingual textual similarity

Cross-lingual classification

Cross-lingual zero-shot learning

Massive multilingual translation

Fine-tuning the performance of multilingual models

Chapter 14: Serving Transformer Models

FastAPI Transformer model serving

Dockerizing APIs

Faster Transformer model serving using TFX

Load testing using Locust

Faster inference using ONNX

SageMaker inference

Chapter 15: Model Tracking and Monitoring

Tracking model metrics

Tracking model training with TensorBoard

Tracking model training live with W&amp

B

Part 4: Transformers beyond NLP

Chapter 16: Vision Transformers

Vision transformers

Image classification using transformers

Semantic segmentation and object detection using transformers

Visual prompt models

Chapter 17: Multimodal Generative Transformers

Generative multimodal AI

Stable Diffusion for text-to-image generation

Stable Diffusion in action

Music generation using MusicGen

Text-to-speech generation using transformers

Summary.

Chapter 18: Revisiting Transformers Architecture for Time Series

Understanding time series concepts

Transformers and time series modeling

Index

Other Books You May Enjoy.

Notes:

Includes bibliographical references and index.

Description based on publisher supplied metadata and other sources.

Description based on print version record.

ISBN:

9781837631506

1837631506

OCLC:

1438954876