FANDOM


Optimization

This page contains resources about Mathematical Optimization and Operations Research.

More specific information is included in each subfield.

Subfields and ConceptsEdit

See Category:Optimization for some of  its subfields.

  • Cost Function / Loss Function / Objective Function
  • Convex Optimization
    • Linear Programming
  • Quadratic Programming
  • Nonlinear Programming
    • Karush–Kuhn–Tucker (KKT) conditions
  • Combinatorial Optimization
    • Integer Programming
    • Dynamic Programming / Optimal Control
      • Deterministic Optimal Control
      • Stochastic Optimal Control
      • Lyapunov Optimization
    • Greedy Algorithm
    • Travelling Salesman Problem (TSP)
  • Online Convex Optimization
    • Mini-Batch Learning
  • Variational Analysis‎
    • Calculus of variations
  • Robust Optimization
  • Lagrange Multipliers
  • Online Optimization
  • Iterative Methods
    • Expectation-Maximization (EM) Algorithm
    • Levenberg–Marquardt Algorithm
    • Iteratively Reweighted Least Squares
    • Nonlinear Least Squares
    • Subgradient Methods (used for Convex Minimization)
    • Gradient Descent / Steepest Descent
    • Krylov Subspace Methods
      • Conjugate Gradient Method
      • Biconjugate Gradient stabilized (BiCGSTAB) Method
      • Arnoldi Method
      • Lanczos Method
      • Generalized Minimal Residual (GMRES) Method
    • Broyden–Fletcher–Goldfarb–Shanno (BFGS) Algorithm
    • Resilient Backpropagation (Rprop)
  • Bayesian Optimization
  • Stochastic Optimization
    • Gradient Descent Methods (either full-batch or mini-batch or both)
      • Stochastic Gradient Descent (SGD)
      • SGD with Momentum
      • Averaged SGD
      • AdaDelta
      • AdaGrad
      • Adam
      • RMSprop
      • Nesterov’s Accelerated Gradient Descent
      • Projected Gradient Descent
      • Particle Mirror Descent (PMD)
      • Regularized Dual Averaging (RDA)
      • Follow the regularised leader (FTRL)
      • Online Gradient Descent
      • Adaptive Online Gradient Descent
      • Natural Gradient Descent
    • Stochastic Gradient Fisher Scoring
    • Stochastic Gradient Langevin Dynamics (SGLD)
    • Stochastic Gradient Hamiltonian Monte Carlo (SGHMC)
    • Stochastic Gradient Riemann Hamiltonian Monte Carlo (SGRHMC)
    • Stochastic Gradient Markov Chain Monte Carlo (SGMCMC)
    • Stochastic Gradient Nose-Hoover Thermostat (SGNHT)
    • Relativistic Stochastic Gradient Descent / Relativistic Monte Carlo
  • Stochastic Approximation
    • Robbins-Monro Algorithm (using noisy estimates of the gradient)
  • Metaheuristics
  • Inverse Problems

Online CoursesEdit

Video LecturesEdit


Lecture NotesEdit

IntroductoryEdit

SpecializedEdit

BooksEdit

IntroductoryEdit

  • Chong, E. K., & Zak, S. H. (2013). An Introduction to Optimization. John Wiley & Sons.
  • Luenberger, D. G., & Ye, Y. (2008). Linear and Nonlinear Programming. Springer.

SpecializedEdit

  • Bertsekas, D. P. (2017). Dynamic Programming and Optimal Control (Vol. 1). 4th Ed. Athena Scientific.
  • Hazan, E. (2015). Introduction to Online Convex Optimization. Foundations and Trends® in Optimization2(3-4), 157-325.
  • Bertsekas, D. P. (2015). Convex Optimization Algorithms. Athena Scientific.
  • Puterman, M. L. (2014). Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons.
  • Fletcher, R. (2013). Practical Methods of Optimization. John Wiley & Sons.
  • Shalev-Shwartz, S. (2011). Online Learning and Online Convex Optimization.Foundations and Trends® in Machine Learning4(2), 107-194.
  • Sra, S., Nowozin, S., & Wright, S. J. (2012). Optimization for machine learning. MIT Press.
  • Luke, S. (2009). Essentials of Metaheuristics. Raleigh: Lulu.
  • Press, W. H. (2007). Numerical Recipes 3rd edition: The Art of Scientific Computing. Cambridge University Press.
  • Nocedal, J., & Wright, S. J. (2006). Numerical Optimization. Springer.
  • Boyd, S. P., & Vandenberghe, L. (2004). Convex Optimization. Cambridge university Press.
  • Nesterov, Y., & Nesterov, I. E. (2004). Introductory Lectures on Convex Optimization: A Basic Course (Vol. 87). Springer.
  • Saad, Y. (2003). Iterative Methods for Sparse Linear Systems. Siam.
  • Vogel, C. R. (2002). Computational Methods for Inverse Problems. Siam.
  • Kelley, C. T. (1999). Iterative Methods for Optimization. Siam.
  • Dennis Jr, J. E., & Schnabel, R. B. (1996). Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Siam.
  • Rustagi, J. S. (1994). Optimization techniques in statistics. Elsevier.

SoftwareEdit

See List of Optimization Software  for the complete list.

See alsoEdit

Other ResourcesEdit