| |
| |
| Preface | |
| |
| |
| |
| Mathematical Review | |
| |
| |
| |
| Methods of Proof and Some Notation | |
| |
| |
| |
| Methods of Proof | |
| |
| |
| |
| Notation | |
| |
| |
| Exercises | |
| |
| |
| |
| Vector Spaces and Matrices | |
| |
| |
| |
| Vector and Matrix | |
| |
| |
| |
| Rank of a Matrix | |
| |
| |
| |
| Linear Equations | |
| |
| |
| |
| Inner Products and Norms | |
| |
| |
| Exercises | |
| |
| |
| |
| Transformations | |
| |
| |
| |
| Linear Transformations | |
| |
| |
| |
| Eigenvalues and Eigenvectors | |
| |
| |
| |
| Orthogonal Projections | |
| |
| |
| |
| Quadratic Forms | |
| |
| |
| |
| Matrix Norms | |
| |
| |
| Exercises | |
| |
| |
| |
| Concepts from Geometry | |
| |
| |
| |
| Line Segments | |
| |
| |
| |
| Hyperplanes and Linear Varieties | |
| |
| |
| |
| Convex Sets | |
| |
| |
| |
| Neighborhoods | |
| |
| |
| |
| Polytopes and Polyhedra | |
| |
| |
| Exercises | |
| |
| |
| |
| Elements of Calculus | |
| |
| |
| |
| Sequences and Limits | |
| |
| |
| |
| Differentiability | |
| |
| |
| |
| The Derivative Matrix | |
| |
| |
| |
| Differentiation Rules | |
| |
| |
| |
| Level Sets and Gradients | |
| |
| |
| |
| Taylor Series | |
| |
| |
| Exercises | |
| |
| |
| |
| Unconstrained Optimization | |
| |
| |
| |
| Basics of Set-Constrained and Unconstrained Optimization | |
| |
| |
| |
| Introduction | |
| |
| |
| |
| Conditions for Local Minimizers | |
| |
| |
| Exercises | |
| |
| |
| |
| One-Dimensional Search Methods | |
| |
| |
| |
| Golden Section Search | |
| |
| |
| |
| Fibonacci Search | |
| |
| |
| |
| Newton's Method | |
| |
| |
| |
| Secant Method | |
| |
| |
| |
| Remarks on Line Search Methods | |
| |
| |
| Exercises | |
| |
| |
| |
| Gradient Methods | |
| |
| |
| |
| Introduction | |
| |
| |
| |
| The Method of Steepest Descent | |
| |
| |
| |
| Analysis of Gradient Methods | |
| |
| |
| Exercises | |
| |
| |
| |
| Newton's Method | |
| |
| |
| |
| Introduction | |
| |
| |
| |
| Analysis of Newton's Method | |
| |
| |
| |
| Levenberg-Marquardt Modification | |
| |
| |
| |
| Newton's Method for Nonlinear Least Squares | |
| |
| |
| Exercises | |
| |
| |
| |
| Conjugate Direction Methods | |
| |
| |
| |
| Introduction | |
| |
| |
| |
| The Conjugate Direction Algorithm | |
| |
| |
| |
| The Conjugate Gradient Algorithm | |
| |
| |
| |
| The Conjugate Gradient Algorithm for Nonquadratic | |
| |
| |
| Problems | |
| |
| |
| Exercises | |
| |
| |
| |
| Quasi-Newton Methods | |
| |
| |
| |
| Introduction | |
| |
| |
| |
| Approximating the Inverse Hessian | |
| |
| |
| |
| The Rank One Correction Formula | |
| |
| |
| |
| The DFP Algorithm | |
| |
| |
| |
| The BFGS Algorithm | |
| |
| |
| Exercises | |
| |
| |
| |
| Solving Linear Equations | |
| |
| |
| |
| Least-Squares Analysis | |
| |
| |
| |
| The Recursive Least-Squares Algorithm | |
| |
| |
| |
| Solution to a Linear Equation with Minimum Norm | |
| |
| |
| |
| Kaczmarz's Algorithm | |
| |
| |
| |
| Solving Linear Equations in General | |
| |
| |
| Exercises | |
| |
| |
| |
| Unconstrained Optimization and Neural Networks | |
| |
| |
| |
| Introduction | |
| |
| |
| |
| Single-Neuron Training | |
| |
| |
| |
| The Backpropagation Algorithm | |
| |
| |
| Exercises | |
| |
| |
| |
| Global Search Algorithms | |
| |
| |
| |
| Introduction | |
| |
| |
| |
| The Nelder-Mead Simplex Algorithm | |
| |
| |
| |
| Simulated Annealing | |
| |
| |
| |
| Particle Swarm Optimization | |
| |
| |
| |
| Genetic Algorithms | |
| |
| |
| Exercises | |
| |
| |
| |
| Linear Programming | |
| |
| |
| |
| Introduction to Linear Programming | |
| |
| |
| |
| Brief History of Linear Programming | |
| |
| |
| |
| Simple Examples of Linear Programs | |
| |
| |
| |
| Two-Dimensional Linear Programs | |
| |
| |
| |
| Convex Polyhedra and Linear Programming | |
| |
| |
| |
| Standard Form Linear Programs | |
| |
| |
| |
| Basic Solutions | |
| |
| |
| |
| Properties of Basic Solutions | |
| |
| |
| |
| Geometric View of Linear Programs | |
| |
| |
| Exercises | |
| |
| |
| |
| Simplex Method | |
| |
| |
| |
| Solving Linear Equations Using Row Operations | |
| |
| |
| |
| The Canonical Augmented Matrix | |
| |
| |
| |
| Updating the Augmented Matrix | |
| |
| |
| |
| The Simplex Algorithm | |
| |
| |
| |
| Matrix Form of the Simplex Method | |
| |
| |
| |
| Two-Phase Simplex Method | |
| |
| |
| |
| Revised Simplex Method | |
| |
| |
| Exercises | |
| |
| |
| |
| Duality | |
| |
| |
| |
| Dual Linear Programs | |
| |
| |
| |
| Properties of Dual Problems | |
| |
| |
| Exercises | |
| |
| |
| |
| Nonsimplex Methods | |
| |
| |
| |
| Introduction | |
| |
| |
| |
| Khachiyan's Method | |
| |
| |
| |
| Affine Scaling Method | |
| |
| |
| |
| Karmarkar's Method | |
| |
| |
| Exercises | |
| |
| |
| |
| Nonlinear Constrained Optimization | |
| |
| |
| |
| Problems with Equality Constraints | |
| |
| |
| |
| Introduction | |
| |
| |
| |
| Problem Formulation | |
| |
| |
| |
| Tangent and Normal Spaces | |
| |
| |
| |
| Lagrange Condition | |
| |
| |
| |
| Second-Order Conditions | |
| |
| |
| |
| Minimizing Quadratics Subject to Linear Constraints | |
| |
| |
| Exercises | |
| |
| |
| |
| Problems with Inequality Constraints | |
| |
| |
| |
| Karush-Kuhn-Tucker Condition | |
| |
| |
| |
| Second-Order Conditions | |
| |
| |
| Exercises | |
| |
| |
| |
| Convex Optimization Problems | |
| |
| |
| |
| Introduction | |
| |
| |
| |
| Convex Functions | |
| |
| |
| |
| Convex Optimization Problems | |
| |
| |
| |
| Semidefinite Programming | |
| |
| |
| Exercises | |
| |
| |
| |
| Algorithms for Constrained Optimization | |
| |
| |
| |
| Introduction | |
| |
| |
| |
| Projections | |
| |
| |
| |
| Projected Gradient Methods with Linear Constraints | |
| |
| |
| |
| Lagrangian Algorithms | |
| |
| |
| |
| Penalty Methods | |
| |
| |
| Exercises | |
| |
| |
| |
| Multiobjective Optimization | |
| |
| |
| |
| Introduction | |
| |
| |
| |
| Pareto Solutions | |
| |
| |
| |
| Computing the Pareto Front | |
| |
| |
| |
| From Multiobjective to Single-Objective Optimization | |
| |
| |
| |
| Uncertain Linear Programming Problems | |
| |
| |
| Exercises | |
| |
| |
| References | |
| |
| |
| Index | |