Numerical Methods and Optimization
Keywords:
Numerical methods, optimization, nonlinear optimization, numerical analysis, gradient descent, Newton method, quasi-Newton method, conjugate gradient, trust-region methods, constrained optimization, computational mathematics, convergence analysisAbstract
Numerical methods and optimization can be recognized as two very closely related disciplines within the field of
applied mathematics, computational science, engineering, economics, operations research, and decision science.
Numerical methods provide systematic procedures for obtaining approximate solutions to mathematical problems
when analytical solutions are not available or difficult to compute. However, optimization provides a framework for
finding the best possible solution based on the predefined criteria. The present-day complexity of scientific and
industrial problems has only increased the importance of using good numerical optimization methods that can deal
with nonlinearities, restrictions, high dimensionality of decision making spaces, uncertainties, and computational
limitations. The current paper studies the theory, operation, and practical applications of the most common numerical
methods of optimization with more focus on gradient-based techniques, Newton and quasi-Newton methods, conjugate
gradient methods, trust regions methods, sequential quadratic programming methods, and methods of optimization
without any use of derivatives. This research observed a computational method that utilized mathematical formulation,
algorithm selection, numerical implementation, convergence analysis, comparative testing, and evaluation of
computational results. The math model developed serves as an example of nonlinear constrained optimization that
will be helpful in evaluating the methods applied in the study in terms of optimization performance, including objective
function reduction, convergence, effort required in calculations, feasibility, and stability. The results showed that
successful numerical optimization depends on both the theoretical basis used in developing the algorithm and other
aspects such as initialization, scaling, stopping criteria, conditioning, constraint handling, and precision of
computations. The conclusion of the study is that it is possible to come up with an effective method for solving complex
real-life optimization problems by using integrated numerical optimization including the formulation of mathematical
models, pre-processing, choice of the algorithm, adaptive parameter control, validation, etc.
