lumin.nn.models package

Subpackages

• lumin.nn.models.blocks package
• lumin.nn.models.layers package

Submodules

lumin.nn.models.helpers module

class lumin.nn.models.helpers.CatEmbedder(cat_names, cat_szs, emb_szs=None, max_emb_sz=50, emb_load_path=None)

Bases: object

Helper class for embedding categorical features. Designed to be passed to ModelBuilder. Note that the classmethod from_fy() may be used to instantiate an CatEmbedder from a FoldYielder.

Parameters:

• cat_names (List[str]) – list of names of catgorical features in order in which they will be passed as inputs columns

• cat_szs (List[int]) – list of cardinalities (number of unique elements) for each feature

• emb_szs (Optional[List[int]]) – Optional list of embedding sizes for each feature. If None, will use min(max_emb_sz, (1+sz)//2)

• max_emb_sz (int) – Maximum size of embedding if emb_szs is None

• emb_load_path (Union[Path, str, None]) – if not None, will cause ModelBuilder to attempt to load pretrained embeddings from path

Examples::

>>> cat_embedder = CatEmbedder(cat_names=['n_jets', 'channel'],
                               cat_szs=[5, 3])
>>>
>>> cat_embedder = CatEmbedder(cat_names=['n_jets', 'channel'],
                               cat_szs=[5, 3], emb_szs=[2, 2])
>>>
>>> cat_embedder = CatEmbedder(cat_names=['n_jets', 'channel'],
                               cat_szs=[5, 3], emb_szs=[2, 2],
                               emb_load_path=Path('weights'))

calc_emb_szs()

Method used to set sizes of embeddings for each categorical feature when no embedding sizes are explicitly passed Uses rule of thumb of min(50, (1+cardinality)/2)

Return type:

None

classmethod from_fy(fy, emb_szs=None, max_emb_sz=50, emb_load_path=None)

Instantiate an CatEmbedder from a FoldYielder, i.e. avoid having to pass cat_names and cat_szs.

Parameters:

• fy (FoldYielder) – FoldYielder with training data

• emb_szs (Optional[List[int]]) – Optional list of embedding sizes for each feature. If None, will use min(max_emb_sz, (1+sz)//2)

• max_emb_sz (int) – Maximum size of embedding if emb_szs is None

• emb_load_path (Union[Path, str, None]) – if not None, will cause ModelBuilder to attempt to load pretrained embeddings from path

Returns:

CatEmbedder

Examples::

>>> cat_embedder = CatEmbedder.from_fy(train_fy)
>>>
>>> cat_embedder = CatEmbedder.from_fy(train_fy, emb_szs=[2, 2])
>>>
>>> cat_embedder = CatEmbedder.from_fy(
        train_fy, emb_szs=[2, 2],
        emb_load_path=Path('weights'))

lumin.nn.models.initialisations module

lumin.nn.models.initialisations.lookup_normal_init(act, fan_in=None, fan_out=None)

Lookup for weight initialisation using Normal distributions

Parameters:

• act (str) – string representation of activation function

• fan_in (Optional[int]) – number of inputs to neuron

• fan_out (Optional[int]) – number of outputs from neuron

Return type:

Callable[[Tensor], None]

Returns:

Callable to initialise weight tensor

lumin.nn.models.initialisations.lookup_uniform_init(act, fan_in=None, fan_out=None)

Lookup weight initialisation using Uniform distributions

Parameters:

• act (str) – string representation of activation function

• fan_in (Optional[int]) – number of inputs to neuron

• fan_out (Optional[int]) – number of outputs from neuron

Return type:

Callable[[Tensor], None]

Returns:

Callable to initialise weight tensor

lumin.nn.models.model module

class lumin.nn.models.model.Model(model_builder=None)

Bases: AbsModel

Wrapper class to handle training and inference of NNs created via a ModelBuilder. Note that saved models can be instantiated direcly via from_save() classmethod.

# TODO: Improve mask description & user-friendlyness, change to indicate that ‘masked’ inputs are actually the ones which are used

Parameters:

model_builder (Optional[ModelBuilder]) – ModelBuilder which will construct the network, loss, optimiser, and input mask

Examples::

>>> model = Model(model_builder)

evaluate(inputs, targets=None, weights=None, bs=None, batch_yielder_type=<class 'lumin.nn.data.batch_yielder.BatchYielder'>)

Compute loss on provided data.

Parameters:

• inputs (Union[ndarray, Tensor, Tuple, BatchYielder]) – input data, or BatchYielder with input, target, and weight data

• targets (Union[ndarray, Tensor, None]) – targets, not required if BatchYielder is passed to inputs

• weights (Union[ndarray, Tensor, None]) – Optional weights, not required if BatchYielder, or no weights should be considered

• bs (Optional[int]) – batch size to use. If None, will evaluate all data at once

• batch_yielder_type (Type[BatchYielder]) – Class of BatchYielder to instantiate to yield inputs

Return type:

float

Returns:

(weighted) loss of model predictions on provided data

export2onnx(name, bs=1)

Export network to ONNX format. Note that ONNX expects a fixed batch size (bs) which is the number of datapoints your wish to pass through the model concurrently.

Parameters:

• name (str) – filename for exported file

• bs (int) – batch size for exported models

Return type:

None

export2tfpb(name, bs=1)

Export network to Tensorflow ProtocolBuffer format, via ONNX. Note that ONNX expects a fixed batch size (bs) which is the number of datapoints your wish to pass through the model concurrently.

Parameters:

• name (str) – filename for exported file

• bs (int) – batch size for exported models

Return type:

None

fit(n_epochs, fy, bs, bulk_move=True, train_on_weights=True, trn_idxs=None, val_idx=None, cbs=None, cb_savepath=Path('train_weights'), model_bar=None, visible_bar=True)

Fit network to training data according to the model’s loss and optimiser.

Training continues until: - All of the training folds are used n_epoch number of times; - Or a callback triggers training to stop, e.g. OneCycle,

or EarlyStopping

Parameters:

• n_epochs (int) – number of epochs for which to train

• fy (FoldYielder) – FoldYielder containing training and validation data

• bs (int) – Batch size

• bulk_move (bool) – if true, will optimise for speed by using more RAM and VRAM

• train_on_weights (bool) – whether to actually use data weights, if present

• trn_idxs (Optional[List[int]]) – Fold indexes in fy to use for training. If not set, will use all folds except val_idx

• val_idx (Optional[int]) – Fold index in fy to use for validation. If not set, will not compute validation losses

• cbs (Union[AbsCallback, List[AbsCallback], None]) – list of instantiated callbacks to adjust training. Will be called in order listed.

• cb_savepath (Path) – General save directory for any callbacks which require saving models and other information (accessible from fit_params),

• model_bar (Optional[ConsoleMasterBar]) – Optional master_bar for aligning progress bars, i.e. if training multiple models

Return type:

List[AbsCallback]

Returns:

List of all callbacks used during training

freeze_layers()

Make parameters untrainable

Return type:

None

classmethod from_save(name, model_builder)

Instantiated a Model and load saved state from file.

Parameters:

• name (str) – name of file containing saved state

• model_builder (ModelBuilder) – ModelBuilder which was used to construct the network

Return type:

AbsModel

Returns:

Instantiated Model with network weights, optimiser state, and input mask loaded from saved state

Examples::

>>> model = Model.from_save('weights/model.h5', model_builder)

get_feat_importance(fy, bs=None, eval_metric=None, savename=None, settings=<lumin.plotting.plot_settings.PlotSettings object>)

Call get_nn_feat_importance() passing this Model and provided arguments

Parameters:

• fy (FoldYielder) – FoldYielder interfacing to data used to train model

• bs (Optional[int]) – If set, will evaluate model in batches of data, rather than all at once

• eval_metric (Optional[EvalMetric]) – Optional EvalMetric to use to quantify performance in place of loss

• savename (Optional[str]) – Optional name of file to which to save the plot of feature importances

• settings (PlotSettings) – PlotSettings class to control figure appearance

Return type:

DataFrame

get_lr()

Get learning rate of optimiser

Return type:

float

Returns:

learning rate of optimiser

get_mom()

Get momentum/beta_1 of optimiser

Return type:

float

Returns:

momentum/beta_1 of optimiser

get_out_size()

Get number of outputs of model

Return type:

int

Returns:

Number of outputs of model

get_param_count(trainable=True)

Return number of parameters in model.

Parameters:

trainable (bool) – if true (default) only count trainable parameters

Return type:

int

Returns:

NUmber of (trainable) parameters in model

get_weights()

Get state_dict of weights for network

Return type:

OrderedDict

Returns:

state_dict of weights for network

load(name, model_builder=None)

Load model, optimiser, and input mask states from file

Parameters:

• name (str) – name of save file

• model_builder (Optional[ModelBuilder]) – if Model was not initialised with a ModelBuilder, you will need to pass one here

Return type:

None

predict(inputs, as_np=True, pred_name='pred', pred_cb=<lumin.nn.callbacks.pred_handlers.PredHandler object>, cbs=None, bs=None, batch_yielder_type=<class 'lumin.nn.data.batch_yielder.BatchYielder'>)

Apply model to inputed data and compute predictions.

Parameters:

• inputs (Union[ndarray, DataFrame, Tensor, FoldYielder]) – input data as Numpy array, Pandas DataFrame, or tensor on device, or FoldYielder interfacing to data

• as_np (bool) – whether to return predictions as Numpy array (otherwise tensor) if inputs are a Numpy array, Pandas DataFrame, or tensor

• pred_name (str) – name of group to which to save predictions if inputs are a FoldYielder

• pred_cb (PredHandler) – PredHandler callback to determin how predictions are computed. Default simply returns the model predictions. Other uses could be e.g. running argmax on a multiclass classifier

• cbs (Optional[List[AbsCallback]]) – list of any instantiated callbacks to use during prediction

• bs (Optional[int]) – if not None, will run prediction in batches of specified size to save of memory

• batch_yielder_type (Type[BatchYielder]) – Class of BatchYielder to instantiate to yield inputs

Return type:

Union[ndarray, Tensor, None]

Returns:

if inputs are a Numpy array, Pandas DataFrame, or tensor, will return predicitions as either array or tensor

save(name)

Save model, optimiser, and input mask states to file

Parameters:

name (str) – name of save file

Return type:

None

set_input_mask(mask)

Mask input columns by only using input columns whose indices are listed in mask

Parameters:

mask (ndarray) – array of column indices to use from all input columns

Return type:

None

set_lr(lr)

set learning rate of optimiser

Parameters:

lr (float) – learning rate of optimiser

Return type:

None

set_mom(mom)

Set momentum/beta_1 of optimiser

Parameters:

mom (float) – momentum/beta_1 of optimiser

Return type:

None

set_weights(weights)

Set state_dict of weights for network

Parameters:

weights (OrderedDict) – state_dict of weights for network

Return type:

None

unfreeze_layers()

Make parameters trainable

Return type:

None

lumin.nn.models.model_builder module

class lumin.nn.models.model_builder.ModelBuilder(objective, n_out, cont_feats=None, model_args=None, opt_args=None, cat_embedder=None, cont_subsample_rate=None, guaranteed_feats=None, loss='auto', head=<class 'lumin.nn.models.blocks.head.CatEmbHead'>, body=<class 'lumin.nn.models.blocks.body.FullyConnected'>, tail=<class 'lumin.nn.models.blocks.tail.ClassRegMulti'>, lookup_init=<function lookup_normal_init>, lookup_act=<function lookup_act>, pretrain_file=None, freeze_head=False, freeze_body=False, freeze_tail=False)

Bases: object

Class to build models to specified architecture on demand along with an optimiser.

Parameters:

• objective (str) – string representation of network objective, i.e. ‘classification’, ‘regression’, ‘multiclass’

• n_out (int) – number of outputs required

• cont_feats (Optional[List[str]]) – list of names of continuous input features

• model_args (Optional[Dict[str, Dict[str, Any]]]) – dictionary of dictionaries of keyword arguments to pass to head, body, and tail to control architrcture

• opt_args (Optional[Dict[str, Any]]) – dictionary of arguments to pass to optimiser. Missing kargs will be filled with default values. Currently, only ADAM (default), and SGD are available.

• cat_embedder (Optional[CatEmbedder]) – CatEmbedder for embedding categorical inputs

• cont_subsample_rate (Optional[float]) – if between in range (0, 1), will randomly select a fraction of continuous features (rounded upwards) to use as inputs

• guaranteed_feats (Optional[List[str]]) – if subsampling features, will always include the features listed here, which count towards the subsample fraction

• loss (Any) – either and uninstantiated loss class, or leave as ‘auto’ to select loss according to objective

• head (Callable[[Any], AbsHead]) – uninstantiated class which can receive input data and upscale it to model width

• body (Callable[[Any], AbsBody]) – uninstantiated class which implements the main bulk of the model’s hidden layers

• tail (Callable[[Any], AbsTail]) – uninstantiated class which scales the body to the required number of outputs and implements any final activation function and output scaling

• lookup_init (Callable[[str, Optional[int], Optional[int]], Callable[[Tensor], None]]) – function taking choice of activation function, number of inputs, and number of outputs an returning a function to initialise layer weights.

• lookup_act (Callable[[str], Module]) – function taking choice of activation function and returning an activation function layer

• pretrain_file (Optional[str]) – if set, will load saved parameters for entire network from saved model

• freeze_head (bool) – whether to start with the head parameters set to untrainable

• freeze_body (bool) – whether to start with the body parameters set to untrainable

Examples::

>>> model_builder = ModelBuilder(objective='classifier',
>>>                              cont_feats=cont_feats, n_out=1,
>>>                              model_args={'body':{'depth':4,
>>>                                                  'width':100}})
>>>
>>> min_targs = np.min(targets, axis=0).reshape(targets.shape[1],1)
>>> max_targs = np.max(targets, axis=0).reshape(targets.shape[1],1)
>>> min_targs[min_targs > 0] *=0.8
>>> min_targs[min_targs < 0] *=1.2
>>> max_targs[max_targs > 0] *=1.2
>>> max_targs[max_targs < 0] *=0.8
>>> y_range = np.hstack((min_targs, max_targs))
>>> model_builder = ModelBuilder(
>>>     objective='regression', cont_feats=cont_feats, n_out=6,
>>>     cat_embedder=CatEmbedder.from_fy(train_fy),
>>>     model_args={'body':{'depth':4, 'width':100},
>>>                 'tail':{y_range=y_range})
>>>
>>> model_builder = ModelBuilder(objective='multiclassifier',
>>>                              cont_feats=cont_feats, n_out=5,
>>>                              model_args={'body':{'width':100,
>>>                                                  'depth':6,
>>>                                                  'do':0.1,
>>>                                                  'res':True}})
>>>
>>> model_builder = ModelBuilder(objective='classifier',
>>>                              cont_feats=cont_feats, n_out=1,
>>>                              model_args={'body':{'depth':4,
>>>                                                  'width':100}},
>>>                              opt_args={'opt':'sgd',
>>>                                        'momentum':0.8,
>>>                                        'weight_decay':1e-5},
>>>                              loss=partial(SignificanceLoss,
>>>                                           sig_weight=sig_weight,
>>>                                           bkg_weight=bkg_weight,
>>>                                           func=calc_ams_torch))

build_model()

Construct entire network module

Return type:

Module

Returns:

Instantiated nn.Module

classmethod from_model_builder(model_builder, pretrain_file=None, freeze_head=False, freeze_body=False, freeze_tail=False, loss=None, opt_args=None)

Instantiate a ModelBuilder from an exisitng ModelBuilder, but with options to adjust loss, optimiser, pretraining, and module freezing

Parameters:

• model_builder – existing ModelBuilder or filename for a pickled ModelBuilder

• pretrain_file (Optional[str]) – if set, will load saved parameters for entire network from saved model

• freeze_head (bool) – whether to start with the head parameters set to untrainable

• freeze_body (bool) – whether to start with the body parameters set to untrainable

• freeze_tail (bool) – whether to start with the tail parameters set to untrainable

• loss (Optional[Any]) – either and uninstantiated loss class, or leave as ‘auto’ to select loss according to objective

• opt_args (Optional[Dict[str, Any]]) – dictionary of arguments to pass to optimiser. Missing kargs will be filled with default values. Choice of optimiser (‘opt’) keyword can either be set by passing the string name (e.g. ‘adam’ ), but only ADAM and SGD are available this way, or by passing an uninstantiated optimiser (e.g. torch.optim.Adam). If no optimser is set, then it defaults to ADAM. Additional keyword arguments can be set, and these will be passed tot he optimiser during instantiation

Returns:

Instantiated ModelBuilder

Examples::

>>> new_model_builder = ModelBuilder.from_model_builder(
>>>     ModelBuidler)
>>>
>>> new_model_builder = ModelBuilder.from_model_builder(
>>>     ModelBuidler, loss=partial(
>>>         SignificanceLoss, sig_weight=sig_weight,
>>>         bkg_weight=bkg_weight, func=calc_ams_torch))
>>>
>>> new_model_builder = ModelBuilder.from_model_builder(
>>>     'weights/model_builder.pkl',
>>>     opt_args={'opt':'sgd', 'momentum':0.8, 'weight_decay':1e-5})
>>>
>>> new_model_builder = ModelBuilder.from_model_builder(
>>>     'weights/model_builder.pkl',
>>>     opt_args={'opt':torch.optim.Adam,
...               'momentum':0.8,
...               'weight_decay':1e-5})

get_body(n_in, feat_map)

Construct body module

Return type:

AbsBody

Returns:

Instantiated body nn.Module

get_head()

Construct head module

Return type:

AbsHead

Returns:

Instantiated head nn.Module

get_model()

Construct model, loss, and optimiser, optionally loading pretrained weights

Return type:

Tuple[Module, Optimizer, Callable[[], Module], Optional[ndarray]]

Returns:

Instantiated network, optimiser linked to model parameters, uninstantiated loss, and optional input mask

get_out_size()

Get number of outputs of model

Return type:

int

Returns:

number of outputs of network

get_tail(n_in)

Construct tail module

Return type:

Module

Returns:

Instantiated tail nn.Module

load_pretrained(model)

Load model weights from pretrained file

Parameters:

model (Module) – instantiated model, i.e. return of build_model()

Returns:

model with weights loaded

set_lr(lr)

Set learning rate for all model parameters

Parameters:

lr (float) – learning rate

Return type:

None

Module contents