Hyperparameter Set

Available Hyperparameters

Parameter

Type

Description

Default

Models / Applications

activation

str

Activation function after each hidden layer

relu

CNN / LSTM / MLP / RNN

batch

int

Number of sample in single batch

8

CNN / LSTM / MLP / RNN

dropout

float

Dropout rate (between 0 and 1)

0

CNN / LSTM / MLP / RNN

epochs

int

Number of training epochs

10

CNN / LSTM / MLP / RNN

factor

int

Factor by which the number of channels is multiplied for each channel.

2

CNN

fc_activation

str

Activation function for post-convolution fully-connect layer

relu

CNN

fc_dropout

float

Dropout rate (between 0 and 1) for post-convolution fully-connected layer

0

CNN

fc_nodes

int

Number of flattened nodes in post-convolution fully-connected layer

10

CNN

filter

int

Number of filters per hidden layer

2

CNN

kernel

int

Kernel size for each hidden layer

2

CNN

lag

int

Time lag

0

Time-series forecasting

layers

int

Number of hidden layers

1

CNN / LSTM / MLP / RNN

loss

str

Loss function

mean_squared_error

CNN / LSTM / MLP / RNN

maxpool

int

2D max pooling (applied only if it is not 0)

0

CNN

nodes

int

Number of nodes per hidden layer

10

LSTM / MLP / RNN

optimizer

str

Optimizer function

Adam

CNN / LSTM / MLP / RNN

padding

str

Padding

valid

CNN

recurrent_activation

str

Inner recurrent activation which actualizes inner memory cell

relu

LSTM / RNN

recurrent_dropout

float

Inner recurrent dropout rate (between 0 and 1)

0

LSTM / RNN

stride

int

Strides of the convolution along the height and width.

1

CNN

Note

Some hyperparameters such as activation , dropout , nodes , filter , kernel , maxpool , padding and recurrent_activation are currently layer-independent, meaning that the variable will be applied to each layer. In order to ease the creation of the neural network, these hyperparameters will be converted into a list replicating the value by the number of hidden layers.

Categorical hyperparameters

The activation , fc_activation , loss , optimizer , padding and recurrent_activation hyperparameters are currently fixed parameter. The user can specify a different value from the configuration file. In the section below, we list the available options for both activation and loss functions as well as the optimizer.

Categorical Hyperparameters

Activation functions

Algorithm

Parameter

PyTorch function

Tensorflow function

Exponential Linear Unit

elu

torch.nn.ELU

tf.keras.activations.elu

Exponential

exponential

n/a

tf.keras.activations.exponential

Hard sigmoid

hard_sigmoid

torch.nn.Hardsigmoid

tf.keras.activations.hard_sigmoid

Linear (pass-through)

linear

torch.nn.Linear

tf.keras.activations.linear

Rectified Linear Unit

relu

torch.nn.ReLU

tf.keras.activations.relu

Scaled Exponential Linear Unit

selu

torch.nn.SELU

tf.keras.activations.selu

Sigmoid

sigmoid

torch.nn.Sigmoid

tf.keras.activations.sigmoid

Softmax

softmax

torch.nn.Softmax

tf.keras.activations.softmax

Softplus

softplus

torch.nn.Softplus

tf.keras.activations.softplus

Softsign

softsign

torch.nn.Softsign

tf.keras.activations.softsign

Tanh

tanh

torch.nn.Tanh

tf.keras.activations.tanh

Loss functions

Algorithm

Parameter

PyTorch function

Tensorflow function

Binary Cross Entropy*

binary_crossentropy

torch.nn.BCELoss

tf.keras.losses.BinaryCrossentropy

Categorical Cross Entropy

categorical_crossentropy

torch.nn.CrossEntropyLoss

tf.keras.losses.CategoricalCrossentropy
tf.keras.losses.SparseCategoricalCrossentropy

Categorical Hinge

categorical_hinge

n/a

tf.keras.losses.CategoricalHinge

Cosine Similarity

cosine_proximity

n/a

tf.keras.losses.cosine_similarity

Hinge,

hinge

n/a

tf.keras.losses.Hinge

Kullback-Leibler divergence

kullback_leibler_divergence

torch.nn.KLDivLoss

tf.keras.losses.KLDivergence

Logarithm of the hyperbolic cosine of the prediction error

logcosh

n/a/

tf.keras.losses.log_cosh

Mean Absolute Error

mean_absolute_error

torch.nn.L1Loss

tf.keras.losses.MeanAbsoluteError

Mean Absolute Percentage Error

mean_absolute_percentage_error

n/a/

tf.keras.losses.MeanAbsolutePercentageError

Mean Squared Error

mean_squared_error

torch.nn.MSELoss

tf.keras.losses.MeanSquaredError

Mean Squared Logarithmic Error

mean_squared_logarithmic_error

n/a

tf.keras.losses.MeanSquaredLogarithmicError

Poisson Loss

poisson

n/a

tf.keras.losses.Poisson

Squared hinge

squared_hinge

n/a

tf.keras.losses.SquaredHinge

One-hot label representation

When doing image classification with Tensorflow and when the output labels from the dataset are integers (like it is the case with the CIFAR10 dataset), the Sparse Categorical Crossentropy loss function should be used instead of the standard Categorical Cross Entropy (see here for more information). To enforce such loss function to be used, the loss parameter within the configuration file can be set to sparse_categorical_crossentropy (see this section on how to do it).

Optimizers

Algorithm

Parameter

PyTorch function

Tensorflow function

Adadelta

Adadelta

torch.optim.Adadelta

tf.keras.optimizers.Adadelta

Adagrad

Adagrad

torch.optim.Adagrad

tf.keras.optimizers.Adagrad

Adaptive Movement Estimation

Adam

torch.optim.Adam

tf.keras.optimizers.Adam

Adamax

Adamax

torch.optim.Adamax

tf.keras.optimizers.Adamax

Nesterov-accelerated Adaptive Moment Estimation

Nadam

n/a

tf.keras.optimizers.Nadam

Root Mean Square Propagation

RMSprop

torch.optim.RMSprop

tf.keras.optimizers.RMSprop

Stochastic Gradient Descent

sgd

torch.optim.SGD

tf.keras.optimizers.SGD

Software Modules

Save/Load into picke file

hyppo.hyperparams. get_hyperprms ( trainer , x_sc = None , record = None , library = None , ** kwargs ) [source]

This function handles the recording and/or loading of the compelte hyperparameter set ready-to-use for training.

Parameters :
x_sc dict

Dictionary of random hyperparameter values.

record bool

Path to pickle file to save/load the hyperparameter set.

Returns :
hyperprms dict

Complete hyperparameter set containing both the random values for the parameter to evaluate and the rest of the fixed parameters.

Complete hyperparameter set

hyppo.hyperparams. set_hyperparams ( random_set , library , ** kwargs ) [source]

This function merges into a single dictionary the hyperparameters that will be evaluated with the one that will be fixed. Pre-defined default values will be used for the fixed hyperparameters. The hardcoded default values can be changed by the user through the configuration file using the default option.

Parameters :
random_set dict

Random values for each hyperparameter to be evaluated

Returns :
hpo_set dict

Complete hyperparameter set

Examples

>>> from hyppo.hyperparams import set_hyperparams
>>> set_hyperparams({'layers':3,'nodes':20},library='tf')
{'activation': ['relu', 'relu', 'relu'],
 'batch': 8,
 'dropout': [0, 0, 0],
 'epochs': 10,
 'fc_activation': 'relu',
 'fc_dropout': 0,
 'fc_nodes': 10,
 'filter': [2, 2, 2],
 'kernel': [2, 2, 2],
 'lag': 0,
 'layers': 3,
 'loss': 'mean_squared_error',
 'loss_args': {},
 'maxpool': [0, 0, 0],
 'nodes': [20, 20, 20],
 'opt_args': {},
 'optimizer': 'Adam',
 'padding': ['valid', 'valid', 'valid'],
 'recurrent_activation': ['relu', 'relu', 'relu'],
 'recurrent_dropout': 0,
 'stride': 1}