14.7 - Multivariable Optimizations

Remember

Recall from Calculus 1 that if $f(x)$ has a local maximum/minimum at a point $x=c$, and if $f^{\prime }(c)$ exists, then $f^{\prime }(c)=0$.

MatPlotLib Graph

Python

import matplotlib.pyplot as plt import numpy as np # Define the function f(x) and its derivative f'(x) def f(x): return np.sin(x) # Example of a function with a local max and min def df(x): return np.cos(x) # Derivative of sin(x) # Create x values x_vals = np.linspace(0, 2 * np.pi, 500) # Points of local max and min (pi/2 and 3pi/2 for sin(x)) x_max = np.pi / 2 x_min = 3 * np.pi / 2 y_max = f(x_max) y_min = f(x_min) # Tangent lines at local max and min tangent_max_x = np.linspace(x_max - 1, x_max + 1, 100) tangent_max_y = df(x_max) * (tangent_max_x - x_max) + y_max tangent_min_x = np.linspace(x_min - 1, x_min + 1, 100) tangent_min_y = df(x_min) * (tangent_min_x - x_min) + y_min # Create plot plt.figure(figsize=(6, 6)) plt.plot(x_vals, f(x_vals), label=r'$f(x) = \sin(x)$', color='black') # Plot points of local max and min plt.scatter([x_max, x_min], [y_max, y_min], color='black', zorder=5) # Plot tangent lines plt.plot(tangent_max_x, tangent_max_y, color='red', linestyle='-', linewidth=2) plt.plot(tangent_min_x, tangent_min_y, color='red', linestyle='-', linewidth=2) # Add labels and title plt.text(x_max + 0.1, y_max + 0.01, 'Local Max', fontsize=12) plt.text(x_min + 0.1, y_min - 0.3, 'Local Min', fontsize=12) plt.title(r'Demonstration of Local Max and Min of $f(x) = \sin(x)$') # Add axes plt.axhline(0, color='black',linewidth=0.5) plt.axvline(0, color='black',linewidth=0.5) # Show plot plt.show()

Output

Calculus 1: local extrema correspond to horizontal tangent lines

But what about multivariable functions in the 3D plane?

Desmos

---

Extrema in 3D

Suppose a local extremum occurs at point P on the surface $z=f(x,y)$

This means that P must simultaneously be an extremum on every trace curve passing through P

MatPlotLib Graph

Python

import matplotlib.pyplot as plt import numpy as np from mpl_toolkits.mplot3d import Axes3D from mpl_toolkits.mplot3d import Axes3D # Define the 2-variable function f(x, y) def f_xy(x, y): return -(x**2 + y**2) + 4 # Example of a function with a local maximum at (0, 0) # Create x and y values x_vals = np.linspace(-2, 2, 400) y_vals = np.linspace(-2, 2, 400) x_vals, y_vals = np.meshgrid(x_vals, y_vals) # Calculate z values for the surface z_vals = f_xy(x_vals, y_vals) # Coordinates for local extremum P P_x = 0 P_y = 0 P_z = f_xy(P_x, P_y) # plot the surface fig = plt.figure(figsize=(8, 8)) ax = fig.add_subplot(111, projection='3d') # Plot surface ax.plot_surface(x_vals, y_vals, z_vals, cmap='Blues', alpha=0.6, edgecolor='none') # Plot point P ax.scatter(P_x, P_y, P_z, color='purple', s=100, zorder=5) # Highlight the axes and projections (a, b, and P) ax.plot([P_x, P_x], [P_y, P_y], [0, P_z], color='black', linestyle='--') # Vertical line to the surface ax.plot([P_x, P_x], [P_y, 0], [0, 0], color='black', linestyle='--') # Projection on x-z plane ax.plot([P_x, 0], [P_y, P_y], [0, 0], color='black', linestyle='--') # Projection on y-z plane # Cross-sectional lines in red on x and y planes ax.plot(x_vals[0], [P_y] * len(x_vals[0]), f_xy(x_vals[0], P_y), color='red', linestyle='-', linewidth=2) # Along y=0 ax.plot([P_x] * len(y_vals[:, 0]), y_vals[:, 0], f_xy(P_x, y_vals[:, 0]), color='red', linestyle='-', linewidth=2) # Along x=0 # Add labels ax.text(0.1, 0, P_z, 'P', color='purple', fontsize=12) ax.text(2, 0, 0, 'a', fontsize=12) ax.text(0, 2, 0, 'b', fontsize=12) ax.text(0, 0, 4.2, r'$z=f(x, y)$', fontsize=14) # Set labels ax.set_xlabel('X') ax.set_ylabel('Y') ax.set_zlabel('Z') # Set view angle for better perspective ax.view_init(45, 235) # Show plot plt.show()

Output

In Other Words:

The directional derivative at P must equal zero in every possible direction, as the tangent plane must be horizontal!

$D\vec{u}f(a,b)=0\to \nabla f(a,b)\cdot \vec{u}=0$

For all $\vec{u}$, resulting in:

$\nabla f(a,b)=0$

What this means:

This says that if $f(a,b)$ is a local maximum or minimum, and if the first order partial derivative exists, then the gradient vector at $(a,b)$ must equal zero.

• $f(a,b)$ is a local maximum
  • this means that $f(a,b)\ge f(x,y)$ for all $(x,y)$ in some neighborhood of $(a,b)$

At $(a,b)$, $D\vec{u}f(a,b)=0$ for all $\vec{u}$ in some neighborhood of $(a,b)$

---

Remarks

1. This does not say if $\nabla f(a,b)=0$, then $(a,b)$ corresponds to a local extremum. However, it does suggest that we can find the local extrema by looking for where $\nabla f=0$
2. More specifically, "$\nabla f=0$" means $⟨f_x,f_y⟩=⟨0,0⟩$, ie. $f_x=0$ AND $f_y=0$.
3. The points where $\nabla f=0$ are called Critical Points. So, just like in Calculus 1, the local extrema “candidates” occur at Critical Points

---

Problems

Example

Find the local extrema of $f(x,y)=x^2+y^2-2x-6y+14$

Solution

To approach this problem, we need to start by finding the Critical Points:

$\nabla f=0\to ⟨2x-2,2y-6⟩=⟨0,0⟩$

$\to 2x-2=0and2y-6=0$

$\to x=1,y=3$

Observe that $f(1,3)=1+9-2-18+14=4$, and by completing the square, we get that $f(x,y)=[(x-1{)}^2-1]+[(y-3{)}^2-9]+14$, resulting in:

$=(x-1{)}^2+(y-3{)}^2+4$

Because $(x-1{)}^2$ and $(y-3{)}^2$ are always non-negative, $f(x,y)\ge 4$ for all $(x,y)$. Therefore, $f(1,3)=4$ is a local minimum

---

Saddle Points

Saddle points are points where the gradient is zero, but the point is neither a maximum nor a minimum.

Python

import matplotlib.pyplot as plt import numpy as np from mpl_toolkits.mplot3d import Axes3D # Create a grid of points x = np.linspace(-5, 5, 100) y = np.linspace(-5, 5, 100) X, Y = np.meshgrid(x, y) # Define the saddle function Z = X**2 - Y**2 # Classic saddle point equation # Create a 3D plot fig = plt.figure() ax = fig.add_subplot(111, projection='3d') # Plot the surface ax.plot_surface(X, Y, Z, cmap='coolwarm') # Mark the saddle point at (0, 0) saddle_x, saddle_y = 0, 0 ax.scatter(saddle_x, saddle_y, saddle_x**2 - saddle_y**2, color='r', s=100, label='Saddle Point') # Set labels ax.set_xlabel('X axis') ax.set_ylabel('Y axis') ax.set_zlabel('Z axis') ax.set_title('Saddle Point: $z = x^2 - y^2$') # Show plot plt.show()

Output

Saddle Points present a unique challenge in optimization, as they are neither maxima nor minima. They are points where the gradient is zero, but the point is neither a maximum nor a minimum.

---

Second Derivative Test

Hessian Matrix

The Hessian Matrix is a square matrix of second-order partial derivatives of a scalar-valued function.

$H=\left[\begin{array}{cc}f\ast xx& f\ast xy\\ f\ast yx& f\ast yy\end{array}\right]$

We need to know the eigenvalues of the Hessian Matrix to determine the nature of the critical point.

$det(H-\lambda I)=0$

Now that we know this, we can move on tho the Second Partial Derivative Test.

Second Partial Derivative Test

Suppose the second order partial derivative $f(x,y)$ are continuous, and that $(a,b)$ is a Critical Point.

1. If $det(H)>0$ and $f_{xx}(a,b)>0$, then $f(a,b)$ is a local minimum.

2. If the Hessain Matrix is positive definite, then the point is a local minimum.

3. If the Hessina Matrix is negative definite, then the point is a local maximum.

   • This becomes a saddle point if the determinant is negative.

4. If the determinant is zero, the test is inconclusive.