Practical LLM Evaluation for Production Systems : Measure, monitor, and improve AI system reliability across training and inference / Ammar Mohanna, Indrajit Kar, Zonunfeli Ralte.

Format:

Book

Author/Creator:

Mohanna, Ammar.

Language:

English

Subjects (All):

Natural language generation (Computer science).

Artificial intelligence.

Physical Description:

1 online resource (488 pages)

Edition:

1st ed.

Place of Publication:

Birmingham : Packt Publishing, Limited, 2026.

Summary:

Build reliable Build reliable AI evaluation frameworks that measure quality, safety, grounding, and production readiness across modern LLM and SLM applicationsFree with your book: DRM-free PDF version + access to Packt's next-gen Reader* Key Features Design evaluation frameworks for LLMs, SLMs, multimodal, reasoning, and agentic AI systems.

Contents:

Intro

Practical LLM Evaluation for Production Systems

Measure, monitor, and improve AI system reliability across training and inference

Contributors

About the authors

About the reviewers

Table of Contents

Preface

Who this book is for

What this book covers

To get the most out of this book

Download the example code files

Download the color images

Conventions used

Get in touch

Share your thoughts

Free benefits with your book

How to unlock

Join our Discord and Reddit Space

1

Foundations of LLM Evaluation: Core Concepts and Primitives

What evaluation means for LLM products

Evaluation is a decision system, not a score

Quality is only one axis

Metrics are not decisions

What evaluation is not

Running example setup: The production assistant

Why traditional ML evaluation breaks for generative and agentic systems

Open-ended outputs break "single ground truth

Stochasticity breaks "one run = one answer

Prompt and context coupling break the model-centric worldview

Agents break static input-output assumptions

Defining "working" in production: quality, safety, cost, reliability

The four-axis definition of working (Q/S/C/R)

Turning product goals into measurable signals

Risk tiers and severity mapping

Definition of "done" for the running assistant, plus rollback criteria

Evaluation primitives: the shared vocabulary

Evaluation constructs: from primitives to decisions

Evaluators, metrics, actions, and thresholds

Metadata and versioning

Primitives and what to log

Building test suites and datasets

Slice-based sets: Coverage that reflects reality

Gold sets: What "gold" means for generative tasks

Adversarial sets: Intentional stress tests

Synthetic sets: Scale with caution.

Evaluator types

Deterministic validators

Reference-based metrics

LLM-as-judge

Human evaluation operations

Repeatability and statistical sanity under nondeterminism

What repeatability means for stochastic systems

Practical variance controls

Statistical hygiene for product teams

Evaluator drift

Regression detection under noise

From primitives to pipelines: Offline evaluation, online evaluation, regression gates

Offline evaluation pipeline

Online evaluation pipeline

Regression gates: turning metrics into deployment decisions

Linking offline and online: closed-loop improvement

Benchmarks vs. production evaluation: What transfers, what misleads, and why

Evaluation granularity

Tooling patterns: Harness design, versioning, dashboards, trace storage

Harness design principles

Versioning as an evaluation requirement

Dashboards for decisions, not decoration

Trace storage for debugging and evaluation

Tooling as the bridge from concepts to operations

Running example: An end-to-end evaluation blueprint for a production assistant

Step 1: Define the system tuple

Step 2: Define representative cases

Step 3: Organize cases into suites and slices

Step 4: Choose evaluators by failure mode

Step 5: Run the offline evaluation pipeline

Step 6: Apply gates and connect offline to online

Step 7: Keep the blueprint lightweight at first

Common failure modes in real evaluation programs

Chapter checklist: Minimum Viable Evaluation System

Skills recap aligned to the book's promises

Get this book's PDF version and more

2

Building Reliable Text-Only LLMs Through Training-Time Evaluation

Text-only system boundary, training stages, and training-time evaluation in practice

Defining the text-only system boundary

Training stages require different evaluation lenses.

The training-time evaluation loop

Data quality gates: Provenance, deduplication, toxicity, format consistency, instruction hygiene

Provenance and deduplication

Toxicity and safety content screening

Format consistency and parser stability

Instruction hygiene

Coverage and representativeness: Domain, language, intents, and long-tail slices

Coverage is slice completeness, not more tokens

Long-tail behavior and edge-case budgets

Leakage and contamination: Overlap, benchmark contamination, and memorization risk

What counts as leakage in practice

Benchmark contamination is leakage with higher stakes

Detecting overlap at scale

Pretraining objective validation: What loss can and cannot tell you

Instruction tuning evaluation: Instruction-following, constraint adherence, and structured outputs

Treat instruction following as a set of behaviors

Constraint adherence: evaluate rules as explicit checks where possible

Structured outputs as a first-class axis

Preference tuning evaluation: Trade-offs, over-refusal, and mode collapse

Make trade-offs explicit with a small drift scorecard

Over-refusal is a slice problem, not a global problem

Mode collapse: "safe but unhelpful" replies

Verbosity drift and cost drift

Gating preference tuning outcomes

Safety alignment during training: Harmful completion rate and jailbreak robustness

Task-specific fine-tuning: Accuracy, error taxonomies, and edge-case budgets

Generalization and robustness: Out-of-distribution (OOD) sets, paraphrase stability, and adversarial suites

Efficiency signals that matter later: Tokens, lengths, and compute implications

Release readiness: Thresholds, model cards, dataset cards, and changelogs

Case study: Selecting a stronger base model versus fine-tuning

Summary

3.

Controlling Text-Only LLM Behavior at Inference Time

The inference pipeline and why it needs controls

Input validation and intake policies

Prompt governance as versioned product code

Output contracts and the bounded reject-repair fallback loop

Decoding policy and stability under nondeterminism

Runtime evaluation with calibrated judges and human sampling

Tool calling correctness and evidence-bound claims

Retrieval-augmented generation controls and groundedness

Safety controls as slice-based gates

Performance and cost as first-class budgets

Monitoring, drift, and incident response

Rollout patterns: Shadow, canary, and rollback

End-to-end inference control blueprint for the running assistant

4

Grounding and Reliability in Vision Language Models During Training

What makes a vision language model a different evaluation problem?

Anatomy of a vision-language model as an evaluation target

Defining grounding: From intuition to measurable property

What transfers from text-only evaluation and what breaks

Production tasks, user journeys, and the failure modes they expose

Mapping VLM production tasks to evaluation requirements

A production-oriented failure taxonomy for VLMs

Building evaluation data for visual grounding

Gold set design for VLMs: Beyond image-caption pairs

Synthetic and programmatically generated evaluation data

Adversarial and stress-test probes for grounding

Signals and metrics for training-time VLM evaluation

Grounding fidelity metrics

Hallucination detection and quantification

Robustness and consistency signals

Cross-modal alignment metrics

Pretraining objective validation: Contrastive signal health and in-training retrieval monitoring

Composing a training-time evaluation dashboard.

Evaluator selection and calibration

Human evaluation for visual grounding: Design and pitfalls

Deterministic and programmatic evaluators

Model-based graders for VLM evaluation

From signals to training decisions: Thresholds, gates, and readiness

Setting thresholds for grounding and hallucination metrics

Multimodal safety evaluation as a hard training gate

Checkpoint selection and training stage gates

Efficiency signals as a parallel deployment gate

Diagnosing and responding to evaluation failures

Tooling patterns for VLM training-time evaluation

Evaluation harness architecture for VLMs

Versioning, traceability, and reproducibility

Dashboards, regression reports, and communication artifacts

Case study: Training-Time evaluation for a production visual document understanding VLM

System definition and task requirements

Failure taxonomy applied to document understanding

Evaluation data, metrics, and evaluator choices in practice

Thresholds, gate decisions, and a training retrospective

5

Evaluating Visual Grounding and Reliability at Inference Time

The VLM inference pipeline

Image intake and input validation

Visual context handling: What the model actually sees

Prompt governance for multimodal inputs

Output contracts for grounded structured outputs

Decoding policy and stability under visual nondeterminism

Runtime grounding evaluation with calibrated graders and human sampling

Evidence-bound visual claims

Visual grounding controls and groundedness at runtime

Grounding when retrieval is added to the VLM pipeline

Multimodal safety controls as runtime slice-based gates

Performance and cost as first-class budgets at production resolutions

Monitoring, visual drift, and incident response

Rollout patterns for VLM systems: shadow, canary, rollback.

End-to-end inference control blueprint for document-understanding VLM.

Notes:

Description based upon print version of record.

Description based on publisher supplied metadata and other sources.

ISBN:

9781807423889

OCLC:

1601985310