AI News, Control: The "Uncle Fester" of the Data Science Family (part 2--Optimization)

Control: The "Uncle Fester" of the Data Science Family (part 2--Optimization)

By some miracle of Nature, or cosmic mathematical coincidence, there are three extremely important problems that end up being identical:  

Many businesses have ridden bad forecasts into bankruptcy not even realizing that they were depending on a linear model to describe a nonlinear environment.

Usually getting the exact answer to the wrong problem started out being a way to get an approximate answer to the right problem, but somewhere along the line we lost our way and forgot which problem we were actually trying to solve.

Since regression, root finding, and optimization are the same, the optimization problem is a good one to use to describe the twists in the road that affects all of these problems.

By the way, this is our old friend the root finding problem, which probably isn't a surprise since we said that problem was somehow equivalent to optimization.

Setting  f'(x)=0  is a good start, and it goes a long way, but there are lots of special cases and in practice all of the interesting problems involve some and usually several of these conditions.

ii) Asymptotic convergence to a minima: What if the value of f(x) continues to diminish for all time, like with the line, except in some limiting “asymptotic” way, like where f(x)=e-x.

Quartics are one of the simplest cases of multiple extrema (or at least of multiple minima), and it is true that quartics can be solved analytically, like the quadratic formula except with lots more terms.

In practice, we use higher-order polynomials to approximate more complex functions and particularly to approximate data series acquired through observations.

Say, for example, we are packing for a trip and we want to know the best number of socks and shoes and shirts to take with us and still fit everything in one bag.

Say we are selling “Grade A” almonds and we want make the mix at minimum cost, but the USDA says that you can only sell almonds as “Grade A” if there are fewer than 4% splits and less than 0.5% rot, and that 80% of them need to be larger than 20mm.

The constraints, taken together, create a “feasible region” where all constraints are satisfied, like in the figure below where all of the constraints are linear.

Like if you want your stock portfolio to give a certain return with minimum overall risk (image from the open source PyFolio project).

The current state-of-the-art typically applies heuristics--rules of thumb--that, in the right hands, can be used to get very good approximations to optimal solutions.

This is a problem where dropping the marble in the tube would actually work, so by some useful measure, this is an easy problem: the derivatives are all defined and there are no local minima.

But in line search, you would find the bottom of the tube (not the coil), and nudge your way along the tube finding the bottom of the tube again and again, a little further ahead perhaps millions of times before you got to the bottom of the coil.

Some variations of line search are better than others at finding their way down the coil, but the issue is that the line search doesn’t know it’s a coil.

Most combinatorial optimization uses a variation on the trust region method where we define a family of possible solutions that are in some sense “near” our initial guess, and we see if any of these are better than the guess.

We maintain a population of “fit” individuals or chromosomes (meaning that they give low values for the objective function) and then we select pairs of fit chromosomes to “recombine” in a process of Darwinian evolution, giving the most fit offspring a higher likelihood of producing their own offspring.

The trust region is the line between to the two 'parents' in chromosome space, and contains the set of chromosomes that could be created by combining the two parents.

These “genetic algorithms” sound hopelessly slow, but while they are tedious they do work for some problems, so long as the evolutionary process is designed and implemented by skilled practitioners who understand the nuances of the specific problem.

A similar heuristic (simulated annealing) mimics the energy minimization process that occurs when a glass or metal is cooled very slowly.

The optimization examples we’ve given are “one shot” problems: pack a bag, mix a batch of nuts, make a skateboard ramp, design a portfolio, etc.

Optimization Calculus - Fence Problems, Cylinder, Volume of Box, Minimum Distance & Norman Window

This calculus video tutorial explains how to solve optimization problems such as the fence problem along the river, fence problem with cost, cylinder problem, ...

❖ Optimization Problem #1 ❖

Thanks to all of you who support me on Patreon. You da real mvps! $1 per month helps!! :) !! Optimization Problem #2 ..

Optimization Problem #4 - Max Area Enclosed by Rectangular Fence

Thanks to all of you who support me on Patreon. You da real mvps! $1 per month helps!! :) !! Optimization Problem #4 ..

Optimization: profit | Applications of derivatives | AP Calculus AB | Khan Academy

Who knows, you may end up running a shoe factory one day. So it might not be a bad idea to know how to maximize profits. Created by Sal Khan. Watch the next ...

Optimization: cost of materials | Applications of derivatives | AP Calculus AB | Khan Academy

With all the storage you might have to handle for your shoe factory, I bet you'd also like to be able to minize the cost of storage. Created by Sal Khan. Watch the ...

Solving Optimization Problems using Derivatives

This tutorial demonstrates the solutions to 5 typical optimization problems using the first derivative to identify relative max or min values for a problem.

Position, Velocity, Acceleration using Derivatives

Thanks to all of you who support me on Patreon. You da real mvps! $1 per month helps!! :) !! In this video, I discuss the ..

Revenue Maximizing #1 - Optimization Word Problem (Calculus) - Quick Explanation!

Get more practice + worked examples at:

❖ Finding Intervals of Increase/Decrease Local Max/Mins ❖

Thanks to all of you who support me on Patreon. You da real mvps! $1 per month helps!! :) !! For more FREE math videos, ..

Increasing/Decreasing + Local Max and Mins using First Derivative Test

Thanks to all of you who support me on Patreon. You da real mvps! $1 per month helps!! :) !! Buy My Book: ' 1001 Calculus ..