Dynamic programming : finite states / Thomas Sargent, New York University, John Stachurski, Australian National University.

Format:

Book

Author/Creator:

Sargent, Thomas J., author.

Stachurski, John, 1969- author.

Language:

English

Subjects (All):

Economics--Mathematical models.

Economics.

Dynamic programming.

Physical Description:

1 online resource (xi, 368 pages) : digital, PDF file(s).

Edition:

1st ed.

Place of Publication:

Cambridge : Cambridge University Press, 2025.

Summary:

Dynamic programming (DP) is a sub-field of optimization concerned with sequential decision making over time. The essential ideas of DP have been adopted in many applications, from robotics and AI to the sequencing of DNA. It is used around the world to control aircraft, route shipping, test products, recommend information on media platforms and solve major research problems. Dynamic Programming: Finite States treats the theory of dynamic programming and its applications in economics, finance, and operations research. It contains classical results on dynamic programming as well as extensions created by researchers and practitioners as they wrestle with formulating and solving dynamic models that can explain patterns observed in data. Adopting an abstract framework that provides great generality, this book facilitates rapid progress to the research frontier by combining rigorous theory with numerous applications, many solved exercises, and detailed open-source computer code.

Contents:

Cover

Half-title page

Title page

Copyright page

Dedication

Contents

Preface

Common Symbols

Common Abbreviations

1 Introduction

1.1 Bellman Equations

1.1.1 Finite-Horizon Job Search

1.1.1.1 A Two-Period Problem

1.1.1.2 Comments on Information

1.1.1.3 Value Functions

1.1.1.4 Three Periods

1.1.2 Infinite Horizon

1.2 Stability and Contractions

1.2.1 Vector Space

1.2.1.1 Real and Complex Vectors

1.2.1.2 Norms

1.2.1.3 Equivalence of Vector Norms

1.2.1.4 Matrices and Neumann Series

1.2.2 Nonlinear Systems

1.2.2.1 Fixed Points

1.2.2.2 Global Stability

1.2.2.3 Banach's Fixed-Point Theorem

1.2.3 Successive Approximation

1.2.3.1 Iteration

1.2.3.2 A One-Dimensional Example

1.2.4 Finite-Dimensional Function Space

1.2.4.1 Pointwise Operations on Functions

1.2.4.2 Functions versus Vectors

1.2.4.3 Distributions

1.3 Infinite-Horizon Job Search

1.3.1 Values and Policies

1.3.1.1 Optimal Choices

1.3.1.2 The Bellman Operator

1.3.1.3 Optimal Policies

1.3.2 Computation

1.3.2.1 Value Function Iteration

1.3.2.2 Computing the Continuation Value Directly

1.4 Chapter Notes

2 Operators and Fixed Points

2.1 Stability

2.1.1 Conjugate Maps

2.1.1.1 Conjugacy

2.1.1.2 Topological Conjugacy

2.1.2 Local Stability

2.1.3 Convergence Rates

2.1.4 Gradient-Based Methods

2.1.4.1 Newton Fixed-Point Iteration

2.1.4.2 Rates of Convergence

2.1.4.3 Speed versus Robustness

2.1.4.4 Parallelization

2.2 Order

2.2.1 Partial Orders

2.2.1.1 Partially Ordered Sets

2.2.1.2 Least and Greatest Elements

2.2.1.3 Sup and Inf

2.2.2 The Case of Pointwise Order

2.2.2.1 Suprema and Infima under a Pointwise Order

2.2.2.2 Inequalities and Identities

2.2.3 Order-Preserving Maps

2.2.3.1 Definition.

2.2.3.2 Increasing and Decreasing Functions

2.2.3.3 Blackwell's Condition

2.2.4 Stochastic Dominance

2.2.5 Parametric Monotonicity

2.3 Matrices and Operators

2.3.1 Nonnegative Matrices

2.3.1.1 Nonnegative Matrices and Their Powers

2.3.1.2 A Local Spectral Radius Result

2.3.1.3 Markov Matrices

2.3.2 A Lake Model

2.3.3 Linear Operators

2.3.3.1 Matrices versus Linear Operators

2.3.3.2 Linear Operators on Function Space

2.3.3.3 Computational Issues

2.3.3.4 Positive Operators and Markov Operators

2.4 Chapter Notes

3 Markov Dynamics

3.1 Foundations

3.1.1 Markov Chains

3.1.1.1 Defining Markov Chains

3.1.1.2 Application: S-s Dynamics

3.1.1.3 Higher Order Transition Matrices

3.1.2 Stationarity and Ergodicity

3.1.2.1 Application: Day Laborer

3.1.3 Approximation

3.2 Conditional Expectations

3.2.1 Mathematical Expectations

3.2.1.1 Conditional Expectations

3.2.1.2 The Law of Iterated Expectations

3.2.1.3 Monotone Markov Chains

3.2.2 Geometric Sums

3.2.2.1 Theory

3.2.2.2 Application: Valuation of Firms

3.2.2.3 Application: Valuing Consumption Streams

3.3 Job Search Revisited

3.3.1 Job Search with Markov State

3.3.1.1 Value Function Iteration

3.3.1.2 Continuation Values

3.3.2 Job Search with Separation

3.4 Chapter Notes

4 Optimal Stopping

4.1 Introduction to Optimal Stopping

4.1.1 Theory

4.1.1.1 The Stopping Problem

4.1.1.2 Lifetime Values

4.1.1.3 Policy Operators

4.1.1.4 The Value Function

4.1.1.5 The Bellman Operator

4.1.1.6 Optimal Policies

4.1.1.7 Value Function Iteration

4.1.2 Firm Valuation with Exit

4.1.2.1 Optional Exit

4.1.2.2 Exit versus No-Exit

4.1.3 Monotonicity

4.1.3.1 Monotone Values

4.1.3.2 Monotone Actions

4.1.4 Continuation Values

4.1.4.1 The Continuation Value Operator.

4.1.4.2 Dimensionality Reduction

4.1.4.3 Application to Firm Value

4.2 Further Applications

4.2.1 American Options

4.2.2 Research and Development

4.2.2.1 Constant R&amp

D Costs

4.2.2.2 IID R&amp

4.3 Chapter Notes

5 Markov Decision Processes

5.1 Definition and Properties

5.1.1 The MDP Model

5.1.2 Examples

5.1.2.1 A Renewal Problem

5.1.2.2 Optimal Inventory Management

5.1.2.3 Example: Cake Eating

5.1.2.4 Example: Optimal Stopping

5.1.3 Optimality

5.1.3.1 Policies and Lifetime Values

5.1.3.2 Defining Optimality

5.1.3.3 Optimality Theory

5.1.4 Algorithms

5.1.4.1 Value Function Iteration

5.1.4.2 Howard Policy Iteration

5.1.4.3 HPI as Newton Iteration

5.1.4.4 Optimistic Policy Iteration

5.2 Applications

5.2.1 Optimal Inventories

5.2.2 Optimal Savings with Labor Income

5.2.2.1 MDP Representation

5.2.2.2 Implementation

5.2.2.3 Timing

5.2.2.4 Outputs

5.2.3 Optimal Investment

5.2.3.1 Problem Description

5.2.3.2 MDP Representation

5.2.3.3 Implementation

5.3 Modified Bellman Equations

5.3.1 Structural Estimation

5.3.1.1 What Is Structural Estimation?

5.3.1.2 An Illustration

5.3.1.3 Expected Value Functions

5.3.1.4 Optimality via EV Methods

5.3.2 The Gumbel Max Trick

5.3.3 Optimal Savings with Stochastic Returns on Wealth

5.3.4 Q-Factors

5.3.5 Operator Factorizations

5.3.5.1 Refactoring the Bellman Operator

5.3.5.2 Refactorizations and Optimality

5.3.5.3 Refactored OPI

5.4 Chapter Notes

6 Stochastic Discounting

6.1 Time-Varying Discount Factors

6.1.1 Valuation

6.1.1.1 Motivation

6.1.1.2 Theory

6.1.2 Testing the Spectral Radius Condition

6.1.2.1 Spectral Radii via Expectations

6.1.2.2 Necessary Conditions

6.1.3 Fixed-Point Results

6.1.3.1 Long-Run Contractions.

6.1.3.2 A Spectral Radius Condition

6.1.3.3 A Generalized Blackwell Condition

6.2 Optimality with State-Dependent Discounting

6.2.1 MDPs with State-Dependent Discounting

6.2.1.1 Setup

6.2.1.2 Finite Lifetime Values

6.2.1.3 Optimality

6.2.1.4 Algorithms

6.2.1.5 Exogenous Discounting

6.2.1.6 Comments on the Spectral Radius Condition

6.2.2 Inventory Management Revisited

6.3 Asset Pricing

6.3.1 Introduction to Asset Pricing

6.3.1.1 Risk-Neutral Pricing?

6.3.1.2 A Stochastic Discount Factor

6.3.1.3 A General Specification

6.3.1.4 Markov Pricing

6.3.1.5 Pricing a Stationary Dividend Stream

6.3.1.6 Forward Sum Representation

6.3.2 Nonstationary Dividends

6.3.2.1 Price-Dividend Ratios

6.3.2.2 Application: Markov Growth with a Lucas SDF

6.3.3 Incomplete Markets

6.4 Chapter Notes

7 Nonlinear Valuation

7.1 Beyond Contraction Maps

7.1.1 Knaster-Tarski for Function Space

7.1.2 Concavity, Convexity, and Stability

7.1.2.1 The One-Dimensional Case

7.1.2.2 The Multidimensional Case

7.1.3 A Power-Transformed Affine Equation

7.2 Recursive Preferences

7.2.1 Motivation: Optimal Savings

7.2.1.1 A Recursive View of a Standard Model

7.2.1.2 Limitations of Time Additive Preferences

7.2.2 Risk-Sensitive Preferences

7.2.2.1 Lifetime Utility

7.2.2.2 Risk-Adjusted Expectation

7.2.2.3 Existence and Uniqueness

7.2.2.4 The Gaussian Case

7.2.3 Epstein-Zin Preferences

7.2.3.1 Specification

7.2.3.2 Properties of the Koopmans Operator

7.2.3.3 Proof of the Stability Result

7.2.3.4 Why Not Use Contractivity?

7.3 General Representations

7.3.1 Koopmans Operators

7.3.1.1 Certainty Equivalents

7.3.1.2 Properties

7.3.1.3 Monotonicity

7.3.1.4 Aggregation

7.3.1.5 Building Koopmans Operators

7.3.1.6 Comments on CES Aggregation.

7.3.1.7 Lifetime Value

7.3.1.8 Monotone Lifetime Values

7.3.2 A Blackwell-Type Condition

7.3.2.1 Blackwell Aggregators

7.3.2.2 The Risk-Sensitive Case

7.3.2.3 Quantile Preferences

7.3.3 Uzawa Aggregation

7.3.3.1 The Case of Conditional Expectation

7.3.3.2 Stability via Concavity

7.3.3.3 Epstein-Zin Preferences with State-Dependent Discounting

7.4 Chapter Notes

8 Recursive Decision Processes

8.1 Definition and Properties

8.1.1 Defining RDPs

8.1.2 Lifetime Value

8.1.2.1 Policies and Value

8.1.2.2 Uniqueness and Stability

8.1.2.3 Continuity

8.1.3 Optimality

8.1.3.1 Greedy Policies

8.1.3.2 Algorithms

8.1.3.3 Optimality

8.1.3.4 Comments on the Optimality Theorem

8.1.3.5 Nonstationary Policies

8.1.3.6 Bounded RDPs

8.1.4 Topologically Conjugate RDPs

8.1.4.1 Application: Epstein-Zin RDPs

8.2 Types of RDPs

8.2.1 Contracting RDPs

8.2.1.1 Definition and Examples

8.2.1.2 Error Bounds

8.2.1.3 A Blackwell-Type Condition

8.2.1.4 Application: Job Search with Quantile Preferences

8.2.1.5 Application: Optimal Default

8.2.2 Eventually Contracting RDPs

8.2.2.1 Definition and Properties

8.2.2.2 Optimality for MDPs with State-Dependent Discounting

8.2.3 Convex and Concave RDPs

8.2.3.1 Definitions and Optimality

8.2.3.2 Application to MDPs

8.3 Further Applications

8.3.1 Risk-Sensitive RDPs

8.3.1.1 Optimality Results

8.3.1.2 Risk-Sensitive Job Search

8.3.2 Adversarial Agents

8.3.2.1 Optimality

8.3.2.2 A Perturbed MDP Problem

8.3.3 Ambiguity and Robustness

8.3.3.1 Robust Control

8.3.3.2 Robustness and Adversarial Agents

8.3.3.3 Connection to Risk-Sensitive Preferences

8.3.4 Smooth Ambiguity

8.3.5 Minimization

8.3.5.1 Application: Shortest Paths

8.3.5.2 Application: Negative Discount Rate Optimality.

8.4 Chapter Notes.

Notes:

Title from publisher's bibliographic system (viewed on 18 Jul 2025).

Description based on publisher supplied metadata and other sources.

ISBN:

1-009-54077-7

1-009-54078-5

OCLC:

1528031102