lumin.nn.metrics package¶
Submodules¶
lumin.nn.metrics.class_eval module¶
-
class
lumin.nn.metrics.class_eval.
AMS
(n_total, wgt_name, targ_name='targets', br=0, syst_unc_b=0, use_quick_scan=True)[source]¶ Bases:
lumin.nn.metrics.eval_metric.EvalMetric
Class to compute maximum Approximate Median Significance (https://arxiv.org/abs/1007.1727) using classifier which directly predicts the class of data in a binary classifiaction problem. AMS is computed on a single fold of data provided by a
FoldYielder
and automatically reweights data by event multiplicity to account missing weights.- Parameters
n_total (
int
) – total number of events in entire data setwgt_name (
str
) – name of weight group in fold file to use. N.B. if you have reweighted to balance classes, be sure to use the un-reweighted weights.targ_name (
str
) – name of target group in fold filebr (
float
) – constant bias offset for background yieldsyst_unc_b (
float
) – fractional systematic uncertainty on background yielduse_quick_scan (
bool
) – whether to optimise AMS by theams_scan_quick()
method (fast but suffers floating point precision) if False useams_scan_slow()
(slower but more accurate)
- Examples::
>>> ams_metric = AMS(n_total=250000, br=10, wgt_name='gen_orig_weight') >>> >>> ams_metric = AMS(n_total=250000, syst_unc_b=0.1, ... wgt_name='gen_orig_weight', use_quick_scan=False)
-
evaluate
(fy, idx, y_pred)[source]¶ Compute maximum AMS on fold using provided predictions.
- Parameters
fy (
FoldYielder
) –FoldYielder
interfacing to dataidx (
int
) – fold index corresponding to fold for which y_pred was computedy_pred (
ndarray
) – predictions for fold
- Return type
float
- Returns
Maximum AMS computed on reweighted data from fold
- Examples::
>>> ams = ams_metric.evaluate(train_fy, val_id, val_preds)
-
class
lumin.nn.metrics.class_eval.
MultiAMS
(n_total, wgt_name, targ_name, zero_preds, one_preds, br=0, syst_unc_b=0, use_quick_scan=True)[source]¶ Bases:
lumin.nn.metrics.class_eval.AMS
Class to compute maximum Approximate Median Significance (https://arxiv.org/abs/1007.1727) using classifier which predicts the class of data in a multiclass classifiaction problem which can be reduced to a binary classification problem AMS is computed on a single fold of data provided by a
FoldYielder
and automatically reweights data by event multiplicity to account missing weights.- Parameters
n_total (
int
) – total number of events in entire data setwgt_name (
str
) – name of weight group in fold file to use. N.B. if you have reweighted to balance classes, be sure to use the un-reweighted weights.targ_name (
str
) – name of target group in fold file which indicates whether the event is signal or backgroundzero_preds (
List
[str
]) – list of predicted classes which correspond to class 0 in the form pred_[i], where i is a NN output indexone_preds (
List
[str
]) – list of predicted classes which correspond to class 1 in the form pred_[i], where i is a NN output indexbr (
float
) – constant bias offset for background yieldsyst_unc_b (
float
) – fractional systematic uncertainty on background yielduse_quick_scan (
bool
) – whether to optimise AMS by theams_scan_quick()
method (fast but suffers floating point precision) if False useams_scan_slow()
(slower but more accurate)
- Examples::
>>> ams_metric = MultiAMS(n_total=250000, br=10, targ_name='gen_target', ... wgt_name='gen_orig_weight', ... zero_preds=['pred_0', 'pred_1', 'pred_2'], ... one_preds=['pred_3']) >>> >>> ams_metric = MultiAMS(n_total=250000, syst_unc_b=0.1, ... targ_name='gen_target', ... wgt_name='gen_orig_weight', ... use_quick_scan=False, ... zero_preds=['pred_0', 'pred_1', 'pred_2'], ... one_preds=['pred_3'])
-
evaluate
(fy, idx, y_pred)[source]¶ Compute maximum AMS on fold using provided predictions.
- Parameters
fy (
FoldYielder
) –FoldYielder
interfacing to dataidx (
int
) – fold index corresponding to fold for which y_pred was computedy_pred (
ndarray
) – predictions for fold
- Return type
float
- Returns
Maximum AMS computed on reweighted data from fold
- Examples::
>>> ams = ams_metric.evaluate(train_fy, val_id, val_preds)
-
class
lumin.nn.metrics.class_eval.
BinaryAccuracy
(threshold=0.5, targ_name='targets', wgt_name=None)[source]¶ Bases:
lumin.nn.metrics.eval_metric.EvalMetric
Computes and returns the accuracy of a single-output model for binary classification tasks.
- Parameters
threshold (
float
) – minimum value of model prediction that will be considered a prediction of class 1. Values below this threshold will be considered predictions of class 0. Default = 0.5.wgt_name (
Optional
[str
]) – name of weight group in fold file to use.targ_name (
str
) – name of target group in fold file which indicates whether the event is class 0 or 1
- Examples::
>>> acc_metric = BinaryAccuracy() >>> >>> acc_metric = BinaryAccuracy(threshold=0.8, wgt_name='weights')
-
evaluate
(fy, idx, y_pred)[source]¶ Computes the (weighted) accuracy for a set of targets and predictions for a given threshold.
- Parameters
fy (
FoldYielder
) –FoldYielder
interfacing to dataidx (
int
) – fold index corresponding to fold for which y_pred was computedy_pred (
ndarray
) – predictions for fold
- Return type
float
- Returns
The (weighted) accuracy for the specified threshold
- Examples::
>>> acc = acc_metric.evaluate(train_fy, val_id, val_preds)
-
class
lumin.nn.metrics.class_eval.
RocAucScore
(average='macro', max_fpr=None, multi_class='raise', targ_name='targets', wgt_name=None)[source]¶ Bases:
lumin.nn.metrics.eval_metric.EvalMetric
Computes and returns the area under the Receiver Operator Characteristic curve (ROC AUC) of a classifier model.
- Parameters
average (
Optional
[str
]) –As per scikit-learn. {‘micro’, ‘macro’, ‘samples’, ‘weighted’} or None, default=’macro’ If
None
, the scores for each class are returned. Otherwise, this determines the type of averaging performed on the data: Note: multiclass ROC AUC currently only handles the ‘macro’ and ‘weighted’ averages.'micro'
:Calculate metrics globally by considering each element of the label indicator matrix as a label.
'macro'
:Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account.
'weighted'
:Calculate metrics for each label, and find their average, weighted by support (the number of true instances for each label).
'samples'
:Calculate metrics for each instance, and find their average.
Will be ignored when
y_true
is binary.max_fpr (
Optional
[float
]) – As per scikit-learn. float > 0 and <= 1, default=None If notNone
, the standardized partial AUC over the range [0, max_fpr] is returned. For the multiclass case,max_fpr
, should be either equal toNone
or1.0
as AUC ROC partial computation currently is not supported for multiclass.multi_class (
str
) –As per scikit-learn. {‘raise’, ‘ovr’, ‘ovo’}, default=’raise’ Multiclass only. Determines the type of configuration to use. The default value raises an error, so either
'ovr'
or'ovo'
must be passed explicitly.'ovr'
:Computes the AUC of each class against the rest. This treats the multiclass case in the same way as the multilabel case. Sensitive to class imbalance even when
average == 'macro'
, because class imbalance affects the composition of each of the ‘rest’ groupings.'ovo'
:Computes the average AUC of all possible pairwise combinations of classes. Insensitive to class imbalance when
average == 'macro'
.
wgt_name (
Optional
[str
]) – name of weight group in fold file to use.targ_name (
str
) – name of target group in fold file which indicates whether the event is class 0 or 1
- Examples::
>>> auc_metric = RocAucScore() >>> >>> auc_metric = RocAucScore(wgt_name='weights') >>> >>> auc_metric = RocAucScore(max_fpr=0.2, wgt_name='weights') >>> >>> auc_metric = RocAucScore(multi_class='ovo', wgt_name='weights')
-
evaluate
(fy, idx, y_pred)[source]¶ Computes the (weighted) (averaged) ROC AUC for a set of targets and predictions.
- Parameters
fy (
FoldYielder
) –FoldYielder
interfacing to dataidx (
int
) – fold index corresponding to fold for which y_pred was computedy_pred (
ndarray
) – predictions for fold
- Return type
float
- Returns
The (weighted) (averaged) ROC AUC for the specified threshold
- Examples::
>>> auc = auc_metric.evaluate(train_fy, val_id, val_preds)
lumin.nn.metrics.eval_metric module¶
-
class
lumin.nn.metrics.eval_metric.
EvalMetric
(targ_name='targets', wgt_name=None)[source]¶ Bases:
abc.ABC
Abstract class for evaluating performance of a model using some metric
- Parameters
targ_name (
str
) – name of group in fold file containing regression targetswgt_name (
Optional
[str
]) – name of group in fold file containing datapoint weights
-
abstract
evaluate
(fy, idx, y_pred)[source]¶ Evaluate the required metric for a given fold and set of predictions
- Parameters
fy (
FoldYielder
) –FoldYielder
interfacing to dataidx (
int
) – fold index corresponding to fold for which y_pred was computedy_pred (
ndarray
) – predictions for fold
- Return type
float
- Returns
metric value
-
get_df
(fy, idx, y_pred)[source]¶ Returns a DataFrame for the given fold containing targets, weights, and predictions
- Parameters
fy (
FoldYielder
) –FoldYielder
interfacing to dataidx (
int
) – fold index corresponding to fold for which y_pred was computedy_pred (
ndarray
) – predictions for fold
- Return type
DataFrame
- Returns
DataFrame for the given fold containing targets, weights, and predictions
lumin.nn.metrics.reg_eval module¶
-
class
lumin.nn.metrics.reg_eval.
RegPull
(return_mean, use_bootstrap=False, use_weights=True, use_pull=True, targ_name='targets', wgt_name=None)[source]¶ Bases:
lumin.nn.metrics.eval_metric.EvalMetric
Compute mean or standard deviation of delta or pull of some feature which is being directly regressed to. Optionally, use bootstrap resampling on validation data.
- Parameters
return_mean (
bool
) – whether to return the mean or the standard deviationuse_bootstrap (
bool
) – whether to bootstrap resamples validation fold when computing statisiticuse_weights (
bool
) – whether to actually use weights if wgt_name is setuse_pull (
bool
) – whether to return the pull (differences / targets) or delta (differences)targ_name (
str
) – name of group in fold file containing regression targetswgt_name (
Optional
[str
]) – name of group in fold file containing datapoint weights
- Examples::
>>> mean_pull = RegPull(return_mean=True, use_bootstrap=True, ... use_pull=True) >>> >>> std_delta = RegPull(return_mean=False, use_bootstrap=True, ... use_pull=False) >>> >>> mean_pull = RegPull(return_mean=True, use_bootstrap=False, ... use_pull=True, wgt_name='weights')
-
evaluate
(fy, idx, y_pred)[source]¶ Compute statisitic on fold using provided predictions.
- Parameters
fy (
FoldYielder
) –FoldYielder
interfacing to dataidx (
int
) – fold index corresponding to fold for which y_pred was computedy_pred (
ndarray
) – predictions for fold
- Return type
float
- Returns
Statistic set in initialisation computed on the chsoen fold
- Examples::
>>> mean = mean_pull.evaluate(train_fy, val_id, val_preds)
-
class
lumin.nn.metrics.reg_eval.
RegAsProxyPull
(proxy_func, return_mean, use_bootstrap=False, use_weights=True, use_pull=True, targ_name='targets', wgt_name=None)[source]¶ Bases:
lumin.nn.metrics.reg_eval.RegPull
Compute mean or standard deviation of delta or pull of some feature which is being indirectly regressed to via a proxy function. Optionally, use bootstrap resampling on validation data.
- Parameters
proxy_func (
Callable
[[DataFrame
],None
]) – function which acts on regression predictions and adds pred and gen_target columns to the Pandas DataFrame it is passed which contains prediction columns pred_{i}return_mean (
bool
) – whether to return the mean or the standard deviationuse_bootstrap (
bool
) – whether to bootstrap resamples validation fold when computing statisiticuse_weights (
bool
) – whether to actually use weights if wgt_name is setuse_pull (
bool
) – whether to return the pull (differences / targets) or delta (differences)targ_name (
str
) – name of group in fold file containing regression targetswgt_name (
Optional
[str
]) – name of group in fold file containing datapoint weights
- Examples::
>>> def reg_proxy_func(df): >>> df['pred'] = calc_pair_mass(df, (1.77682, 1.77682), ... {targ[targ.find('_t')+3:]: ... f'pred_{i}' for i, targ ... in enumerate(targ_feats)}) >>> df['gen_target'] = 125 >>> >>> std_delta = RegAsProxyPull(proxy_func=reg_proxy_func, ... return_mean=False, use_pull=False)
-
evaluate
(fy, idx, y_pred)[source]¶ Compute statisitic on fold using provided predictions.
- Parameters
fy (
FoldYielder
) –FoldYielder
interfacing to dataidx (
int
) – fold index corresponding to fold for which y_pred was computedy_pred (
ndarray
) – predictions for fold
- Return type
float
- Returns
Statistic set in initialisation computed on the chsoen fold
- Examples::
>>> mean = mean_pull.evaluate(train_fy, val_id, val_preds)