Python Data Cleaning Cookbook : Prepare Your Data for Analysis with Pandas, NumPy, Matplotlib, Scikit-Learn, and OpenAI / Michael Walker.

Format:

Book

Author/Creator:

Walker, Michael, 1701-1766, author.

Language:

English

Subjects (All):

Analysis of variance.

Data mining.

Database management.

Python (Computer program language).

Physical Description:

1 online resource (487 pages)

Edition:

Second edition.

Place of Publication:

Birmingham, England : Packt Publishing, [2024]

Summary:

Learn the intricacies of data description, issue identification, and practical problem-solving, armed with essential techniques and expert tips. Key Features Get to grips with new techniques for data preprocessing and cleaning for machine learning and NLP models Use new and updated AI tools and techniques for data cleaning tasks Clean, monitor, and validate large data volumes to diagnose problems using cutting-edge methodologies including Machine learning and AI Book Description Jumping into data analysis without proper data cleaning will certainly lead to incorrect results. The Python Data Cleaning Cookbook - Second Edition will show you tools and techniques for cleaning and handling data with Python for better outcomes.Fully updated to the latest version of Python and all relevant tools, this book will teach you how to manipulate and clean data to get it into a useful form. he current edition focuses on advanced techniques like machine learning and AI-specific approaches and tools for data cleaning along with the conventional ones. The book also delves into tips and techniques to process and clean data for ML, AI, and NLP models. You will learn how to filter and summarize data to gain insights and better understand what makes sense and what does not, along with discovering how to operate on data to address the issues you've identified. Next, you'll cover recipes for using supervised learning and Naive Bayes analysis to identify unexpected values and classification errors and generate visualizations for exploratory data analysis (EDA) to identify unexpected values. Finally, you'll build functions and classes that you can reuse without modification when you have new data.By the end of this Data Cleaning book, you'll know how to clean data and diagnose problems within it. What you will learn Using OpenAI tools for various data cleaning tasks Producing summaries of the attributes of datasets, columns, and rows Anticipating data-cleaning issues when importing tabular data into pandas Applying validation techniques for imported tabular data Improving your productivity in pandas by using method chaining Recognizing and resolving common issues like dates and IDs Setting up indexes to streamline data issue identification Using data cleaning to prepare your data for ML and AI models Who this book is for This book is for anyone looking for ways to handle messy, duplicate, and poor data using different Python tools and techniques. The book takes a recipe-based approach to help you to learn how to clean and manage data with practical examples.Working knowledge of Python programming is all you need to get the most out of the book. ]]>

Contents:

Cover

Copyright

Contributors

Table of Contents

Preface

Chapter 1: Anticipating Data Cleaning Issues When Importing Tabular Data with pandas

Technical requirements

Importing CSV files

Importing Excel files

Importing data from SQL databases

Importing SPSS, Stata, and SAS data

Importing R data

Persisting tabular data

Summary

Chapter 2: Anticipating Data Cleaning Issues When Working with HTML, JSON, and Spark Data

Importing simple JSON data

Importing more complicated JSON data from an API

Importing data from web pages

Working with Spark data

Persisting JSON data

Versioning data

Chapter 3: Taking the Measure of Your Data

Getting a first look at your data

Selecting and organizing columns

Selecting rows

Generating frequencies for categorical variables

Generating summary statistics for continuous variables

Using generative AI to display descriptive statistics

Chapter 4: Identifying Outliers in Subsets of Data

Identifying outliers with one variable

Identifying outliers and unexpected values in bivariate relationships

Using subsetting to examine logical inconsistencies in variable relationships

Using linear regression to identify data points with significant influence

Using k-nearest neighbors to find outliers

Using Isolation Forest to find anomalies

Using PandasAI to identify outliers

Chapter 5: Using Visualizations for the Identification of Unexpected Values

Using histograms to examine the distribution of continuous variables

Using boxplots to identify outliers for continuous variables

Using grouped boxplots to uncover unexpected values in a particular group.

Examining both distribution shape and outliers with violin plots

Using scatter plots to view bivariate relationships

Using line plots to examine trends in continuous variables

Generating a heat map based on a correlation matrix

Chapter 6: Cleaning and Exploring Data with Series Operations

Getting values from a pandas Series

Showing summary statistics for a pandas Series

Changing Series values

Changing Series values conditionally

Evaluating and cleaning string Series data

Working with dates

Using OpenAI for Series operations

Chapter 7: Identifying and Fixing Missing Values

Identifying missing values

Cleaning missing values

Imputing values with regression

Using k-nearest neighbors for imputation

Using random forest for imputation

Using PandasAI for imputation

Chapter 8: Encoding, Transforming, and Scaling Features

Creating training datasets and avoiding data leakage

Removing redundant or unhelpful features

Encoding categorical features: one-hot encoding

Encoding categorical features: ordinal encoding

Encoding categorical features with medium or high cardinality

Using mathematical transformations

Feature binning: equal width and equal frequency

k-means binning

Feature scaling

Chapter 9: Fixing Messy Data When Aggregating

Looping through data with itertuples (an anti-pattern)

Calculating summaries by group with NumPy arrays

Using groupby to organize data by groups

Using more complicated aggregation functions with groupby

Using user-defined functions and apply with groupby

Using groupby to change the unit of analysis of a DataFrame

Using pivot_table to change the unit of analysis of a DataFrame.

Summary

Chapter 10: Addressing Data Issues When Combining DataFrames

Combining DataFrames vertically

Doing one-to-one merges

Doing one-to-one merges by multiple columns

Doing one-to-many merges

Doing many-to-many merges

Developing a merge routine

Chapter 11: Tidying and Reshaping Data

Removing duplicated rows

Fixing many-to-many relationships

Using stack and melt to reshape data from wide to long format

Melting multiple groups of columns

Using unstack and pivot to reshape data from long to wide format

Chapter 12: Automate Data Cleaning with User-Defined Functions, Classes, and Pipelines

Functions for getting a first look at our data

Functions for displaying summary statistics and frequencies

Functions for identifying outliers and unexpected values

Functions for aggregating or combining data

Classes that contain the logic for updating Series values

Classes that handle non-tabular data structures

Functions for checking overall data quality

Pre-processing data with pipelines: a simple example

Pre-processing data with pipelines: a more complicated example

Packt Page

Other Books You May Enjoy

Index.

Notes:

Includes index.

Description based on publisher supplied metadata and other sources.

Description based on print version record.

ISBN:

1-80324-629-4