2D problem example ¶
Test case example ¶
In order to demonstrate how surrogate modeling can outperform random sampling, we set up an easy problem consisting of an inverse 2-dimensional Gaussian profile. This tutorial provides nice and easy instructions on how to generate and plot 3D Gaussian distributions. In this example we use Numpy to generate the distribution and matplotlib to plot it.
[1]:
import numpy
def bivariate_normal_pdf():
X = numpy.arange(-4,4.1,0.5)
Y = numpy.arange(-4,4.1,0.5)
X, Y = numpy.meshgrid(X, Y)
R = numpy.sqrt(X**2 + Y**2)
Z = 0.4 - (1. / numpy.sqrt(2 * numpy.pi)) * numpy.exp(-.5*R**2)
return X, Y, Z
Let’s consider the
x
values to lie between 0 and 1, similar to what the learning rate in machine learning, and the
y
values to lie between 1 and 100, which could be the values that take the number of nodes in a neural network layer.
[2]:
import plotly
import plotly.graph_objects as go
x,y,z = bivariate_normal_pdf()
fig = go.Figure(data=[go.Surface(x=x, y=y, z=z)])
fig.update_traces(contours_z=dict(show=True, usecolormap=True, highlightcolor="limegreen", project_z=True))
fig.update_layout(title="3D bi-variate Gaussian distribution", autosize=True, scene_camera_eye=dict(x=1.5, y=-1.5, z=1.5), margin=dict(l=0, r=0, b=0, t=0))
fig.write_html("../_static/plotly/distribution.html")
Objective function ¶
We can now define an objective function that will be used as the input problem where variables x and y will be fed into and z will be returned. Given random values of
x
and
y
, the obbective function below will rescale those values and output the value
z
(one could think of the loss from training a neural network) within the relevant parameter space the distribution was drawn from.
[1]:
import numpy
def problem(X,Y,**kwargs):
R = numpy.sqrt(X**2 + Y**2)
Z = 0.4 - (1. / numpy.sqrt(2 * numpy.pi)) * numpy.exp(-.5*R**2)
return Z
Surrogate modeling ¶
Gaussian Process ¶
The code below will execute a random sampling of 3 different parameter sets and then does a surrogate modeling over 17 iterations.
[ ]:
import hyppo
config = {
'trainer':problem,
'prms':{
'nevals' : 3,
'names' : [ 'X', 'Y'],
'mult' : [0.01,0.01],
'xlow' : [-400,-400],
'xup' : [ 400, 400],
},
'hpo':{
'loops':27,
'surrogate':'gp',
}
}
hyppo.inline_job(config,run_mode=2,loops=20,store_path='2d_problem/gp')
[1]:
import hyppo
import pandas
hpo_path = 'logs/2d_problem/gp/002/logs/'
evaluation = hyppo.extract('%s/evaluation*_01.log' % hpo_path).sort_values(by=['nevals'])
surrogate = hyppo.extract('%s/surrogate*_01.log' % hpo_path).sort_values(by=['nevals'])
[ ]:
import os
import numpy
X = numpy.arange(-4,4.1,0.01)
Y = numpy.arange(-4,4.1,0.01)
hyppo.model_visualization_2d(X,Y,hpo_path,evaluation,surrogate,imax=15)
os.system('convert -delay 50 -loop 0 '+hpo_path+'/plots/iter_*.png ../_static/animation/gp2d.gif')
Radial Basis Function ¶
[ ]:
import hyppo
config = {
'trainer':problem,
'prms':{
'nevals' : 3,
'names' : [ 'X', 'Y'],
'mult' : [0.01,0.01],
'xlow' : [-400,-400],
'xup' : [ 400, 400],
},
'hpo':{
'loops':27,
'surrogate':'rbf',
'polynomial':'quadratic',
}
}
hyppo.inline_job(config,run_mode=2,loops=20,store_path='2d_problem/rbf')
[4]:
import hyppo
import pandas
hpo_path = 'logs/2d_problem/rbf/002/logs/'
evaluation = hyppo.extract('%s/evaluation*_01.log' % hpo_path).sort_values(by=['nevals'])
surrogate = hyppo.extract('%s/surrogate*_01.log' % hpo_path).sort_values(by=['nevals'])
[ ]:
import os
import numpy
X = numpy.arange(-4,4.1,0.01)
Y = numpy.arange(-4,4.1,0.01)
hyppo.model_visualization_2d(X,Y,hpo_path,evaluation,surrogate,imax=15)
os.system('convert -delay 50 -loop 0 '+hpo_path+'/plots/iter_*.png ../_static/animation/rbf2d.gif')
Random sampling ¶
[ ]:
import hyppo
config = {
'trainer':problem,
'prms':{
'nevals' : 30,
'names' : [ 'X', 'Y'],
'mult' : [0.01,0.01],
'xlow' : [-400,-400],
'xup' : [ 400, 400],
},
}
hyppo.inline_job(config,run_mode=0,loops=20,store_path='2d_problem/random')
Convergence plots ¶
In this page, we compare the performance of doing surrogate modeling with random sampling. The comparison can be done by plotting what we call a “convergence plot” where we plot the best value of the loss found so far after each evaluated set of parameters.
[6]:
import hyppo
import matplotlib.pyplot as plt
from matplotlib.ticker import FormatStrFormatter
plt.style.use('seaborn')
plt.figure(figsize=(6,4),dpi=200)
for i,(path,color) in enumerate([['logs/2d_problem/gp','blue'],['logs/2d_problem/rbf','green'],['logs/2d_problem/random','red']]):
mean_loss, mean_sdev = hyppo.get_convergence(path,with_surrogate=(i<2),sigma=True)
plt.plot(mean_loss,color=color,lw=1,zorder=1,label=path,drawstyle='steps-post')
plt.fill_between(range(len(mean_loss)),mean_loss-mean_sdev/2,mean_loss+mean_sdev/2,
alpha=0.1,lw=0,color=color,step='post')
plt.xlim(xmin=0)
plt.ylim(ymin=0)
plt.ylabel('Loss')
plt.xlabel('Index of function evaluations')
plt.legend(loc='upper right')
plt.axvline(3,color='k',lw=1,ls='dashed')
plt.tight_layout(h_pad=0.1)
plt.show()
