Group Scores

The GroupScores class is an extension of the Scores class to also keep track of group membership.

• We only support one group. To perform analysis with respect to multiple groups simultaneously, one can create a new group variable that combines all original groups.

• All per-group metric functions will return arrays with \(G\) as the first dimension, where \(G\) is the number of different groups. We can use GroupScores.groups to correlate the array indices with group names. Arrays will play nicely with the existing vectorisation features in the library.

class GroupScores(pos, neg, *, pos_groups, neg_groups, score_class='pos', equal_class='pos', group_names=None, is_sorted=False)

Parameters:

• pos – Scores for positive samples.

• neg – Scores for negative samples.

• pos_groups – Group labels for positive samples.

• neg_groups – Group labels for negative samples.

• score_class (Union[BinaryLabel, str]) – Do scores indicate membership of the positive or the negative class?

• equal_class (Union[BinaryLabel, str]) – Do samples with score equal to the threshold get assigned to the positive or negative class?

• group_names (Optional[Any]) – Explicitly provided group names. If provided, this list will be used as is and not sorted.

• is_sorted (bool) – If True, we assume the scores are already sorted. Can be used to speed up initialisation.

bootstrap_sample(config=BootstrapConfig(nb_samples=1000, bootstrap_method='bca', sampling_method='dynamic', stratified_sampling=None, smoothing=False, ratio=None))

Creates one bootstrap sample by sampling with the specified configuration.

Parameters:

config (BootstrapConfig) – Bootstrap configuration.

Returns:

GroupScores object with the sampled scores.

Return type:

GroupScores

static from_labels(labels, scores, groups, *, pos_label=1, score_class='pos', equal_class='pos', is_sorted=False)

Parameters:

• labels – Array with sample labels.

• scores – Array with sample scores.

• groups – Array with group labels.

• pos_label (Any) – The label of the positive class. All other labels are treated as negative labels.

• score_class (Union[BinaryLabel, str]) – Do scores indicate membership of the positive or the negative class?

• equal_class (Union[BinaryLabel, str]) – Do samples with score equal to the threshold get assigned to the positive or negative class?

• is_sorted (bool) – If True, we assume the scores are already sorted. Can be used to speed up initialisation.

Returns:

A GroupScores instance.

Return type:

GroupScores

group_acceptance_rate(threshold)

Per-group Acceptance Rate at threshold(s).

Alias for group_topr().

group_cm(threshold)

Computes per-group confusion matrices at the given threshold(s).

Parameters:

threshold – Can be a scalar or array-like.

Returns:

A binary ConfusionMatrix of shape (G, T, 2, 2), where G is number of groups and T is shape of threshold array.

Return type:

ConfusionMatrix

group_far(threshold)

Per-group False Acceptance Rate at threshold(s).

Alias for group_fpr().

group_fnr(threshold)

Per-group False Negative Rate at threshold(s).

group_fpr(threshold)

Per-group False Positive Rate at threshold(s).

group_frr(threshold)

Per-group False Rejection Rate at threshold(s).

Alias for group_fnr().

group_rejection_rate(threshold)

Per-group Rejection Rate at threshold(s).

Alias for group_tonr().

group_tar(threshold)

Per-group True Acceptance Rate at threshold(s).

Alias for group_tpr().

group_tnr(threshold)

Per-group True Negative Rate at threshold(s).

group_tonr(threshold)

Per-group Test Outcome Negative Rate at threshold(s).

group_topr(threshold)

Per-group Test Outcome Positive Rate at threshold(s).

group_tpr(threshold)

Per-group True Positive Rate at threshold(s).

Return type:

ndarray

group_trr(threshold)

Per-group True Rejection Rate at threshold(s).

Alias for group_tnr().

swap()

Swaps positive and negative scores. Also reverses the decision logic, so that fpr of original scores equals fnr of reversed scores.

Returns:

GroupScores object with positive and negative scores reversed.

Return type:

GroupScores

groupwise(metric)

Converts a metric to a group-wise metric. A metric is can be a string indicating a member function of the Scores class or a callable with signature:

metric(sample: Scores, **kwargs) -> np.ndarray

The group-wise metric is a function:

groupwise_metric(sample: GroupScores, **kwargs) -> np.ndarray

returning an array of shape \((G, Y)\), where \(G\) is the number of groups and metric returns an array of shape \(Y\).

Return type:

Callable[[…], ndarray]