Data Engineering Best Practices : Architect Robust and Cost-Effective Data Solutions in the Cloud Era / Richard J. Schiller and David Larochelle.

Format:

Book

Author/Creator:

Schiller, Richard J., author.

LaRochelle, David, author.

Language:

English

Subjects (All):

Database management.

Big data--Processing.

Big data.

Cloud computing.

Agile software development.

Physical Description:

1 online resource (550 pages)

Edition:

First edition.

Place of Publication:

Birmingham, England : Packt Publishing, [2024]

Biography/History:

Schiller Richard J. : Richard J. Schiller is a chief architect, distinguished engineer, and startup entrepreneur with 40 years of experience delivering real-time large-scale data processing systems. He holds an MS in computer engineering from Columbia University's School of Engineering and Applied Science and a BA in computer science and applied mathematics. He has been involved with two prior successful startups and has co authored three patents. He is a hands-on systems developer and innovator. Larochelle David: David Larochelle has been involved in data engineering for startups, Fortune 500 companies, and research institutes. He holds a BS in computer science from the College of William & Mary, a Masters in computer science from the University of Virginia, and a Master's in communication from the University of Pennsylvania. David's career spans over 20 years, and his strong background has enabled him to work in a wide range of organizations, including startups, established companies, and research labs.

Summary:

Explore modern data engineering techniques and best practices to build scalable, efficient, and future-proof data processing systems across cloud platforms Key Features Architect and engineer optimized data solutions in the cloud with best practices for performance and cost-effectiveness Explore design patterns and use cases to balance roles, technology choices, and processes for a future-proof design Learn from experts to avoid common pitfalls in data engineering projects Purchase of the print or Kindle book includes a free PDF eBook Book Description Revolutionize your approach to data processing in the fast-paced business landscape with this essential guide to data engineering. Discover the power of scalable, efficient, and secure data solutions through expert guidance on data engineering principles and techniques. Written by two industry experts with over 60 years of combined experience, it offers deep insights into best practices, architecture, agile processes, and cloud-based pipelines. You'll start by defining the challenges data engineers face and understand how this agile and future-proof comprehensive data solution architecture addresses them. As you explore the extensive toolkit, mastering the capabilities of various instruments, you'll gain the knowledge needed for independent research. Covering everything you need, right from data engineering fundamentals, the guide uses real-world examples to illustrate potential solutions. It elevates your skills to architect scalable data systems, implement agile development processes, and design cloud-based data pipelines. The book further equips you with the knowledge to harness serverless computing and microservices to build resilient data applications. By the end, you'll be armed with the expertise to design and deliver high-performance data engineering solutions that are not only robust, efficient, and secure but also future-ready. What you will learn Architect scalable data solutions within a well-architected framework Implement agile software development processes tailored to your organization's needs Design cloud-based data pipelines for analytics, machine learning, and AI-ready data products Optimize data engineering capabilities to ensure performance and long-term business value Apply best practices for data security, privacy, and compliance Harness serverless computing and microservices to build resilient, scalable, and trustworthy data pipelines Who this book is for If you are a data engineer, ETL developer, or big data engineer who wants to master the principles and techniques of data engineering, this book is for you. A basic understanding of data engineering concepts, ETL processes, and big data technologies is expected. This book is also for professionals who want to explore advanced data engineering practices, including scalable data solutions, agile software development, and cloud-based data processing pipelines.

Contents:

Cover

Title Page

Copyright and Credits

Contributors

Table of Contents

Preface

Chapter 1: Overview of the Business Problem Statement

What is the business problem statement?

Anti-patterns to avoid

Patterns in the future-proof architecture

Future-proofing is …

Organization into zone considerations

Cloud limitations

The Intelligence Age

Use case definitions

The mission, the vision, and the strategy

Principles and the development life cycle

The architecture definition, best practices, and key considerations

The DataOps convergence

Summary

Chapter 2: A Data Engineer's Journey - Background Challenges

Challenge #1 - platform architectures change rapidly

Platform architectures in the 21st century

Impacts on business strategy

A flexible software development life cycle to manage platform risk

Challenge #2 - Total cost of ownership (TCO) is high

ETL architecture costs are high!

Buy versus build choices impact a solution's longevity

Challenge #3 - Evolving data repository patterns - identifying big rocks for data engineers

Intake, integration, and storage challenges in data engineering

Identifying the big rocks to be placed first into your design

Being able to handle technology hype

Chapter 3: A Data Engineer's Journey - IT's Vision and Mission

The vision

Develop the IT engineering vision

Vision summary

The mission and the IT strategy

IT's vision

IT's mission

IT mission summary

Principles, frameworks, and best practices

The architecture reflects the vision

Principles summary

Data engineering patterns for IT operability

What patterns are required and how are they specified?

Pattern summary

Chapter 4: Architecture Principles

Architecture principles overview

Architecture foundation.

Data lake, mesh, and fabric

Data immutability

Third party tool, cloud platform-as-a-service (PaaS), and framework integrations

Data mesh principles

Data mesh metadata

Data semantics in the data mesh

Data mesh, security, and tech stack considerations

What are the key foundational takeaways?

Architecture principles in depth

Principle #1 - Data lake as a centerpiece? No, implement the data journey!

Principle #2 - A data lake's immutable data is to remain explorable

Principle #3 - A data lake's immutable data remains available for analytics

Principle #4 - A data lake's sources are discoverable

Principle #5 - A data lake's tooling should be consistent with the architecture

Principle #6 - A data mesh defines data to be governed by domain-driven ownership

Principle #7 - A data mesh defines the data and derives insights as a product

Principle #8 - A data mesh defines data, information, and insights to be self-service

Principle #9 - A data mesh implements a federated governance processing system

Principle #10 - Metadata is associated with datasets and is relevant to the business

Principle #11 - Dataset lineage and at-rest metadata is subject to life cycle governance

Principle #12 - Datasets and metadata require cataloging and discovery services

Principle #13 - Semantic metadata guarantees correct business understanding at all stages in the data journey

Principle #14 - Data big rock architecture choices (time series, correction processing, security, privacy, and so on) are to be handled in the design early

Principle #15 - Implement foundational capabilities in the architecture framework first

Chapter 5: Architecture Framework - Conceptual Architecture Best Practices

Conceptual architecture overview

Best practice organization.

How does the conceptual architecture align with the logical architecture and physical architecture?

Conceptual architecture best practices

Conceptual architecture description

Conceptual architecture glossary

What are the data architecture's key issues identified in the conceptual architecture?

Best practice composition of the conceptual architecture

Conceptual to logical architecture mapping

Chapter 6: Architecture Framework - Logical Architecture Best Practices

Logical architecture overview

Organizing best practices

How does the logical architecture align with the conceptual and physical architecture?

Detailed capabilities of the ingestion zones

ETL data pipelines

Bronze standard datasets

Detailed capabilities of the transformation zones

Data quality features

Data lake house and warehouse

Gold and silver standard datasets

Detailed capabilities of the consumption zones

Data analytics

Accessing silver standard datasets from the consumption zone

Trade-offs between public cloud, on-premises, and multi-cloud

Cost of ingest or egress for cloud data

Cost of a dedicated network line to the point of service

Cost of provisioning

Cost of monitoring and observability

Hybrid or multi-cloud choices!

The benefits of a multi-cloud strategy

Chapter 7: Architecture Framework - Physical Architecture Best Practices

Physical architecture overview

Best practice organization

How does the physical architecture align with the logical and conceptual architecture?

How should the physical architecture align with the operational processes/capabilities of the solution?

Examples of physical reference architectures

Chapter 8: Software Engineering Best Practice Considerations

SBP 1 - follow the architecture!.

The core value of architectural integrity

The downstream impact of deviating

Ensuring adherence in your data engineering team

Continuous evolution and architecture

Conclusion

SBP 2 - implement Agile methodology for your organization!

Introduction to Agile methodology

Agile principles and their significance in data engineering

Benefits of implementing Agile in data engineering

Challenges and considerations in Agile data engineering

Steps to implement Agile in data engineering

Tools and Agile practices tailored for data engineering

SBP 3 - generate objectives and key results (OKRs)!

Introduction and deep dive into OKRs

Crafting data-centric OKRs

Potential challenges with OKRs in data engineering

Reviewing and iterating on OKRs in a data context

SBP 4 - implement data as a product!

SBP 5 - implement shift left testing (SLT) processes!

Understanding SLT

Benefits of SLT in data engineering

Implementing shift left testing

Specific shift left testing strategies for data engineering

Challenges in shift left testing for data engineering

Tools and technologies to facilitate shift left in data engineering

Synergy with other data best practices

SBP 6 - implement the difficult first!

The philosophy of tackling the hard tasks first

How data engineers can prioritize difficult tasks

Implementing difficult data tasks

SBP 7 - avoid premature optimization

The true cost of premature optimization

Recognizing and avoiding the trap in data engineering

Balancing performance needs and over-optimization in data engineering

SBP 8 - automate cloud code snippet deployments with standard deployment scripted wrappers

The importance of deployment automation.

The deployment model choices

Benefits of using scripted deployment wrappers

Version control - ensuring consistency and traceability

Relevance to data engineering in cloud environments

Practical implementation steps

Challenges and precautions

Synergy with other software and data best practices

SBP 9 - define and implement NFRs first

Distinguishing functional (FRs) from non-functional requirements (NFRs)

Relevance to data engineering

Key NFRs in cloud data engineering

Defining and implementing NFRs

Risks of neglecting early implementation of NFRs

SBP 10 - implement data journey journaling to facilitate future problem resolution

Challenges and considerations

SBP 11 - implement data journey pipelines that are experimental first!

Enabling data pipeline experimentation as datasets are readied

Releasing data like code

SBP 12 - choose languages with solid reasoning

Key languages in data engineering and their roles

The pressures and limitations imposed by PaaS offerings

Pitfalls to avoid

SBP 13 - drive scripting and PaaS code with parameterization using a secure configuration management repository tool

The power of parameterization and configuration management

The growth of configuration complexity

Why parameterize?

Configuration management repositories and configuration management databases (CMDBs)

Best practices for secure configuration management

SBP 14 - be prepared to prune dead code over time

The accumulation of dead code in software and PaaS systems

The unique challenge of PaaS service configurations

Pruning dead code

SBP 15 - if it doesn't fit, don't force it

use a microservice

PaaS and its boundaries

Microservices as a contingency strategy.

Challenges and considerations of this dual approach.

Notes:

Description based upon print version of record.

Includes bibliographical references and index.

Description based on publisher supplied metadata and other sources.

Description based on print version record.

ISBN:

9781803247366

1803247363

OCLC:

1455754493