lumin.nn.callbacks package¶
Submodules¶
lumin.nn.callbacks.callback module¶

class
lumin.nn.callbacks.callback.
Callback
(model=None, plot_settings=<lumin.plotting.plot_settings.PlotSettings object>)[source]¶ Bases:
lumin.nn.callbacks.abs_callback.AbsCallback
Base callback class from which other callbacks should inherit.
 Parameters
model (
Optional
[AbsModel
]) – model to refer to during trainingplot_settings (
PlotSettings
) – PlotSettings class

set_model
(model)[source]¶ Sets the callback’s model in order to allow the callback to access and adjust model parameters
 Parameters
model (
AbsModel
) – model to refer to during training Return type
None

set_plot_settings
(plot_settings)[source]¶ Sets the plot settings for any plots produced by the callback
 Parameters
plot_settings (
PlotSettings
) – PlotSettings class Return type
None
lumin.nn.callbacks.cyclic_callbacks module¶

class
lumin.nn.callbacks.cyclic_callbacks.
AbsCyclicCallback
(interp, param_range, cycle_mult=1, decrease_param=False, scale=1, model=None, nb=None, plot_settings=<lumin.plotting.plot_settings.PlotSettings object>)[source]¶ Bases:
lumin.nn.callbacks.callback.Callback
Abstract class for callbacks affecting lr or mom
 Parameters
interp (
str
) – string representation of interpolation function. Either ‘linear’ or ‘cosine’.param_range (
Tuple
[float
,float
]) – minimum and maximum values for parametercycle_mult (
int
) – multiplicative factor for adjusting the cycle length after each cycle. E.g cycle_mult=1 keeps the same cycle length, cycle_mult=2 doubles the cycle length after each cycle.decrease_param (
bool
) – whether to begin by decreasing the parameter, otherwise begin by increasing itscale (
int
) – multiplicative factor for setting the initial number of epochs per cycle. E.g scale=1 means 1 epoch per cycle, scale=5 means 5 epochs per cycle.model (
Optional
[AbsModel
]) – model to refer to during trainingnb (
Optional
[int
]) – number of minibatches (iterations) to expect per epochplot_settings (
PlotSettings
) – PlotSettings class

on_batch_begin
(**kargs)[source]¶ Computes the new value for the optimiser parameter and returns it
 Return type
float
 Returns
new value for optimiser parameter

on_batch_end
(**kargs)[source]¶ Increments the callback’s progress through the cycle
 Return type
None

on_epoch_begin
(**kargs)[source]¶ Ensures the cycle_end flag is false when the epoch starts
 Return type
None

class
lumin.nn.callbacks.cyclic_callbacks.
CycleLR
(lr_range, interp='cosine', cycle_mult=1, decrease_param='auto', scale=1, model=None, nb=None, plot_settings=<lumin.plotting.plot_settings.PlotSettings object>)[source]¶ Bases:
lumin.nn.callbacks.cyclic_callbacks.AbsCyclicCallback
Callback to cycle learning rate during training according to either: cosine interpolation for SGDR https://arxiv.org/abs/1608.03983 or linear interpolation for Smith cycling https://arxiv.org/abs/1506.01186
 Parameters
lr_range (
Tuple
[float
,float
]) – tuple of initial and final LRsinterp (
str
) – ‘cosine’ or ‘linear’ interpolationcycle_mult (
int
) – Multiplicative constant for altering the cycle length after each complete cycledecrease_param (
Union
[str
,bool
]) – whether to increase or decrease the LR (effectively reverses lr_range order), ‘auto’ selects according to interpscale (
int
) – Multiplicative constant for altering the length of a cycle. 1 corresponds to one cycle = one (sub)epochmodel (
Optional
[AbsModel
]) –Model
to alter, alternatively callset_model()
.nb (
Optional
[int
]) – Number of batches in a (sub)epochplot_settings (
PlotSettings
) –PlotSettings
class to control figure appearance
 Examples::
>>> cosine_lr = CycleLR(lr_range=(0, 2e3), cycle_mult=2, scale=1, ... interp='cosine', nb=100) >>> >>> cyclical_lr = CycleLR(lr_range=(2e4, 2e3), cycle_mult=1, scale=5, interp='linear', nb=100)

class
lumin.nn.callbacks.cyclic_callbacks.
CycleMom
(mom_range, interp='cosine', cycle_mult=1, decrease_param='auto', scale=1, model=None, nb=None, plot_settings=<lumin.plotting.plot_settings.PlotSettings object>)[source]¶ Bases:
lumin.nn.callbacks.cyclic_callbacks.AbsCyclicCallback
Callback to cycle momentum (beta 1) during training according to either: cosine interpolation for SGDR https://arxiv.org/abs/1608.03983 or linear interpolation for Smith cycling https://arxiv.org/abs/1506.01186 By default is set to evolve in opposite direction to learning rate, a la https://arxiv.org/abs/1803.09820
 Parameters
mom_range (
Tuple
[float
,float
]) – tuple of initial and final momentainterp (
str
) – ‘cosine’ or ‘linear’ interpolationcycle_mult (
int
) – Multiplicative constant for altering the cycle length after each complete cycledecrease_param (
Union
[str
,bool
]) – whether to increase or decrease the momentum (effectively reverses mom_range order), ‘auto’ selects according to interpscale (
int
) – Multiplicative constant for altering the length of a cycle. 1 corresponds to one cycle = one (sub)epochmodel (
Optional
[AbsModel
]) –Model
to alter, alternatively callset_model()
nb (
Optional
[int
]) – Number of batches in a (sub)epochplot_settings (
PlotSettings
) –PlotSettings
class to control figure appearance
 Examples::
>>> cyclical_mom = CycleMom(mom_range=(0.85 0.95), cycle_mult=1, ... scale=5, interp='linear', nb=100)

class
lumin.nn.callbacks.cyclic_callbacks.
OneCycle
(lengths, lr_range, mom_range=(0.85, 0.95), interp='cosine', model=None, nb=None, plot_settings=<lumin.plotting.plot_settings.PlotSettings object>)[source]¶ Bases:
lumin.nn.callbacks.cyclic_callbacks.AbsCyclicCallback
Callback implementing Smith 1cycle evolution for lr and momentum (beta_1) https://arxiv.org/abs/1803.09820 Default interpolation uses fastaistyle cosine function. Automatically triggers early stopping on cycle completion.
 Parameters
lengths (
Tuple
[int
,int
]) – tuple of number of (sub)epochs in first and second stages of cyclelr_range (
List
[float
]) – tuple of initial and final LRsmom_range (
Tuple
[float
,float
]) – tuple of initial and final momentainterp (
str
) – ‘cosine’ or ‘linear’ interpolationmodel (
Optional
[AbsModel
]) –Model
to alter, alternatively callset_model()
nb (
Optional
[int
]) – Number of batches in a (sub)epochplot_settings (
PlotSettings
) –PlotSettings
class to control figure appearance
 Examples::
>>> onecycle = OneCycle(lengths=(15, 30), lr_range=[1e4, 1e2], ... mom_range=(0.85, 0.95), interp='cosine', nb=100)
lumin.nn.callbacks.data_callbacks module¶

class
lumin.nn.callbacks.data_callbacks.
BinaryLabelSmooth
(coefs=0, model=None)[source]¶ Bases:
lumin.nn.callbacks.callback.Callback
Callback for applying label smoothing to binary classes, based on https://arxiv.org/abs/1512.00567 Applies smoothing during both training and inference.
 Parameters
coefs (
Union
[float
,Tuple
[float
,float
]]) – Smoothing coefficients: 0>coef[0] 1>1coef[1]. if passed float, coef[0]=coef[1]model (
Optional
[AbsModel
]) – not used, only for compatability
 Examples::
>>> lbl_smooth = BinaryLabelSmooth(0.1) >>> >>> lbl_smooth = BinaryLabelSmooth((0.1, 0.02))

class
lumin.nn.callbacks.data_callbacks.
SequentialReweight
(reweight_func, scale=0.1, model=None)[source]¶ Bases:
lumin.nn.callbacks.callback.Callback
Caution
Experiemntal proceedure
During ensemble training, sequentially reweight training data in last validation fold based on prediction performance of last trained model. Reweighting highlights data which are easier or more difficult to predict to the next model being trained.
 Parameters
reweight_func (
Callable
[[Tensor
,Tensor
],Tensor
]) – callable function returning a tensor of same shape as targets, ideally quantifying modelprediction performancescale (
float
) – multiplicative factor for rescaling returned tensor of reweight_funcmodel (
Optional
[AbsModel
]) –Model
to provide predictions, alternatively callset_model()
 Examples::
>>> seq_reweight = SequentialReweight( ... reweight_func=nn.BCELoss(reduction='none'), scale=0.1)

on_train_end
(fy, val_id, bs=None, **kargs)[source]¶ Reweighs the validation fold once training is finished
 Parameters
fy (
FoldYielder
) – FoldYielder providing the training and validation datafold_id – Fold index which was used for validation
 Return type
None

class
lumin.nn.callbacks.data_callbacks.
SequentialReweightClasses
(reweight_func, scale=0.1, model=None)[source]¶ Bases:
lumin.nn.callbacks.data_callbacks.SequentialReweight
Caution
Experiemntal proceedure
Version of
SequentialReweight
designed for classification, which renormalises class weights to original weightsum after reweighting During ensemble training, sequentially reweight training data in last validation fold based on prediction performance of last trained model. Reweighting highlights data which are easier or more difficult to predict to the next model being trained. Parameters
reweight_func (
Callable
[[Tensor
,Tensor
],Tensor
]) – callable function returning a tensor of same shape as targets, ideally quantifying modelprediction performancescale (
float
) – multiplicative factor for rescaling returned tensor of reweight_funcmodel (
Optional
[AbsModel
]) –Model
to provide predictions, alternatively callset_model()
 Examples::
>>> seq_reweight = SequentialReweight( ... reweight_func=nn.BCELoss(reduction='none'), scale=0.1)

class
lumin.nn.callbacks.data_callbacks.
BootstrapResample
(n_folds, bag_each_time=False, reweight=True, model=None)[source]¶ Bases:
lumin.nn.callbacks.callback.Callback
Callback for bootstrap sampling new training datasets from original training data during (ensemble) training.
 Parameters
n_folds (
int
) – the number of folds present in trainingFoldYielder
bag_each_time (
bool
) – whether to sample a new set for each subepoch or to use the same sample each timereweight (
bool
) – whether to reweight the sampleed data to mathch the weight sum (per class) of the original datamodel (
Optional
[AbsModel
]) – not used, only for compatability
 Examples::
>>> bs_resample BootstrapResample(n_folds=len(train_fy))

on_epoch_begin
(by, **kargs)[source]¶ Resamples training data for new epoch
 Parameters
by (
BatchYielder
) – BatchYielder providing data for the upcoming epoch Return type
None

class
lumin.nn.callbacks.data_callbacks.
FeatureSubsample
(cont_feats, model=None)[source]¶ Bases:
lumin.nn.callbacks.callback.Callback
Callback for training a model on a random subsample of the range of possible input features. Only subsamples continuous features. Number of continuous inputs infered from model. Associated
Model
will automatically mask its inputs during inference; simply provide inputs with the same number of columns as trainig data.Attention
This callback is now depreciated in favour of passing cont_subsample_rate and guaranteed_feats to
ModelBuilder
as these offer greater functionality and are compatable with using aMultiBlock
body. Will be removed in V0.5.Caution
This callback is incompatable with using a
MultiBlock
body Parameters
cont_feats (
List
[str
]) – list of all continuous features in input data. Order must match.model (
Optional
[AbsModel
]) –Model
being trained, alternatively callset_model()
 Examples::
>>> feat_subsample = FeatureSubsample(cont_feats=['pT', 'eta', 'phi'])

class
lumin.nn.callbacks.data_callbacks.
ParametrisedPrediction
(feats, param_feat, param_val, model=None)[source]¶ Bases:
lumin.nn.callbacks.callback.Callback
Callback for running predictions for a parametersied network (https://arxiv.org/abs/1601.07913); one which has been trained using one of more inputs which represent e.g. different hypotheses for the classes such as an unknown mass of some new particle. In such a scenario, multiple signal datasets could be used for training, with background receiving a random mass. During prediction one then needs to set these parametrisation features all to the same values to evaluat the model’s response for that hypothesis. This callback can be passed to the predict method of the model/ensemble to adjust the parametrisation features to the desired values.
 Parameters
feats (
List
[str
]) – list of feature names used during training (in the same order)param_feat (
str
) – the feature name which is to be adjustedparam_val (
float
) – the value to which to set the paramertisation featuremodel (
Optional
[AbsModel
]) – unused, purely for compatability, just leave it as None
 Examples::
>>> mass_param = ParametrisedPrediction(train_feats, 'res_mass', 300) >>> model.predict(fold_yeilder, pred_name=f'pred_mass_300', callbacks=[mass_param]) >>> >>> mass_param = ParametrisedPrediction(train_feats, 'res_mass', 300) >>> spin_param = ParametrisedPrediction(train_feats, 'spin', 1) >>> model.predict(fold_yeilder, pred_name=f'pred_mass_300', callbacks=[mass_param, spin_param])
lumin.nn.callbacks.loss_callbacks module¶

class
lumin.nn.callbacks.loss_callbacks.
GradClip
(clip, clip_norm=True, model=None)[source]¶ Bases:
lumin.nn.callbacks.callback.Callback
Callback for clipping gradients by norm or value.
 Parameters
clip (
float
) – value to clip atclip_norm (
bool
) – whether to clip according to norm (torch.nn.utils.clip_grad_norm_) or value (torch.nn.utils.clip_grad_value_)model (
Optional
[AbsModel
]) –Model
with parameters to clip gradients, alternatively callset_model()
 Examples::
>>> grad_clip = GradClip(1e5)
lumin.nn.callbacks.lsuv_init module¶
This file contains code modfied from https://github.com/duchaaiki/LSUVpytorch which is made available under the following BSD 2Clause “Simplified” Licence: Copyright (C) 2017, Dmytro Mishkin All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The Apache Licence 2.0 underwhich the majority of the rest of LUMIN is distributed does not apply to the code within this file.

class
lumin.nn.callbacks.lsuv_init.
LsuvInit
(needed_std=1.0, std_tol=0.1, max_attempts=10, do_orthonorm=True, verbose=False, model=None)[source]¶ Bases:
lumin.nn.callbacks.callback.Callback
Applies LayerSequential UnitVariance (LSUV) initialisation to model, as per Mishkin & Matas 2016 https://arxiv.org/abs/1511.06422. When training begins for the first time, Conv1D, Conv2D, and Linear modules in the model will be LSUV initialised using the BatchYielder inputs. This involves initialising the weights with orthonormal matirces and then iteratively scaling them such that the stadndar deviation of the layer outputs is equal to a desired value, within some tolerance.
 Parameters
needed_std (
float
) – desired standard deviation of layer outputsstd_tol (
float
) – tolerance for matching standard deviation with targetmax_attempts (
int
) – number of times to attempt weight scaling per layerdo_orthonorm (
bool
) – whether to apply orthonormal initialisation first, or rescale the exisiting valuesverbose (
bool
) – whether to print out details of the rescalingmodel (
Optional
[AbsModel
]) –Model
to provide parameters, alternatively callset_model()
 Example::
>>> lsuv = LsuvInit() >>> >>> lsuv = LsuvInit(verbose=True) >>> >>> lsuv = LsuvInit(needed_std=0.5, std_tol=0.01, max_attempts=100, do_orthonorm=True)

on_epoch_begin
(by, **kargs)[source]¶ If the LSUV process has yet to run, then it will run using all of the input data provided by the BatchYielder
 Parameters
by (
BatchYielder
) – BatchYielder providing data for the upcoming epoch Return type
None
lumin.nn.callbacks.model_callbacks module¶

class
lumin.nn.callbacks.model_callbacks.
SWA
(start_epoch, renewal_period=1, model=None, val_fold=None, cyclic_callback=None, update_on_cycle_end=None, verbose=False, plot_settings=<lumin.plotting.plot_settings.PlotSettings object>)[source]¶ Bases:
lumin.nn.callbacks.model_callbacks.AbsModelCallback
Callback providing Stochastic Weight Averaging based on (https://arxiv.org/abs/1803.05407) This adapted version allows the tracking of a pair of average models in order to avoid having to hardcode a specific start point for averaging:
Model average x0 will begin to be tracked start_epoch (sub)epochs/cycles after training begins.
cycle_since_replacement is set to 1
Renewal_period (sub)epochs/cycles later, a second average x1 will be tracked.
At the next renewal period, the performance of x0 and x1 will be compared on data contained in val_fold.
 If x0 is better than x1:
x1 is replaced by a copy of the current model
cycle_since_replacement is increased by 1
renewal_period is multiplied by cycle_since_replacement
 Else:
x0 is replaced by x1
x1 is replaced by a copy of the current model
cycle_since_replacement is set to 1
renewal_period is set back to its original value
Additonally, will optionally (default True) lockin to any cyclical callbacks to only update at the end of a cycle.
 Parameters
start_epoch (
int
) – (sub)epoch/cycle to begin averagingrenewal_period (
int
) – How often to check performance of averages, and renew tracking of least performantmodel (
Optional
[AbsModel
]) –Model
to provide parameters, alternatively callset_model()
val_fold (
Optional
[Dict
[str
,ndarray
]]) – Dictionary containing inputs, targets, and weights (or None) as Numpy arrayscyclic_callback (
Optional
[AbsCyclicCallback
]) – Optional for any cyclical callback which is runningupdate_on_cycle_end (
Optional
[bool
]) – Whether to lock in to the cyclic callback and only update at the end of a cycle. Default yes, if cyclic callback present.verbose (
bool
) – Whether to print out update information for testing and operation confirmationplot_settings (
PlotSettings
) –PlotSettings
class to control figure appearance
 Examples::
>>> swa = SWA(start_epoch=5, renewal_period=5)

get_loss
()[source]¶ Evaluates SWA model and returns loss
 Return type
float
 Returns
Loss on validation fold for oldest SWA average

class
lumin.nn.callbacks.model_callbacks.
AbsModelCallback
(model=None, val_fold=None, cyclic_callback=None, update_on_cycle_end=None, plot_settings=<lumin.plotting.plot_settings.PlotSettings object>)[source]¶ Bases:
lumin.nn.callbacks.callback.Callback
Abstract class for callbacks which provide alternative models during training
 Parameters
model (
Optional
[AbsModel
]) –Model
to provide parameters, alternatively callset_model()
val_fold (
Optional
[Dict
[str
,ndarray
]]) – Dictionary containing inputs, targets, and weights (or None) as Numpy arrayscyclic_callback (
Optional
[AbsCyclicCallback
]) – Optional for any cyclical callback which is runningupdate_on_cycle_end (
Optional
[bool
]) – Whether to lock in to the cyclic callback and only update at the end of a cycle. Default yes, if cyclic callback present.plot_settings (
PlotSettings
) –PlotSettings
class to control figure appearance
lumin.nn.callbacks.opt_callbacks module¶

class
lumin.nn.callbacks.opt_callbacks.
LRFinder
(nb, lr_bounds=[1e07, 10], model=None, plot_settings=<lumin.plotting.plot_settings.PlotSettings object>)[source]¶ Bases:
lumin.nn.callbacks.callback.Callback
Callback class for Smith learningrate range test (https://arxiv.org/abs/1803.09820)
 Parameters
nb (
int
) – number of batches in a (sub)epochlr_bounds (
Tuple
[float
,float
]) – tuple of initial and final LRmodel (
Optional
[AbsModel
]) –Model
to alter, alternatively callset_model()
plot_settings (
PlotSettings
) –PlotSettings
class to control figure appearance

on_batch_end
(loss, **kargs)[source]¶ Records loss and increments LR
 Parameters
loss (
float
) – training loss for most recent batch Return type
None