Confusion Matrix

class ConfusionMatrix(labels=None, predictions=None, *, weights=None, matrix=None, classes=None, binary=False)

Confusion matrix class with support for vectorized confusion matrices.

>>> labels      = [2, 0, 2, 2, 0, 1, 1, 2, 2, 0, 1, 2]
>>> predictions = [0, 0, 2, 1, 0, 2, 1, 0, 2, 0, 2, 2]
>>> cm = ConfusionMatrix(labels=labels, predictions=predictions)
>>> cm.classes
[0, 1, 2]
>>> cm2 = ConfusionMatrix(matrix={"a": {"a": 1, "b":2}, "b": {"a": 0, "b": 5}})
>>> cm2.classes
["a", "b"]

A confusion matrix can be created in two ways:

• From labels, predictions and (optionally) weights

• From a matrix already containing necessary information

Creating a confusion matrix from labels and predictions

If classes are not provided, we use all values that appear as either labels or predictions, in sorted order, as classes.

The dtype of the confusion matrix is int if no weights vector is provided and the same as the dtype of the weights vector otherwise. The default weight is 1 for all samples.

Creating a confusion matrix from a matrix

If matrix is a dict of dicts, we expect all entries to have the same set of keys, which will be used as classes. In this case the classes parameter can only be used to reorder the classes.

If matrix is a pandas DataFrame, the index and column names are used as classes. We expect them to be the same (up to reordering). The classes parameter can only be used to reorder the classes.

All other input types we attempt to convert to numpy arrays via np.asarray. The class names are either taken from the provided parameter or set to 0, …, N.

Vectorized confusion matrices

When creating the confusion matrix from a matrix, we can use a matrix of shape (…, N, N) to represent a vectorized confusion matrix.

Binary confusion matrices

Binary confusion matrices are two-class confusion matrices, with the classes in the order [positive, negative]. The default labels are [1, 0]. Note that this is different from the default labels for a two-class non-binary confusion matrix, for which the default labels would be [0, 1].

The main difference is that for a binary confusion matrix metrics, such as TPR, etc. return scalar values, while for general confusion matrices they return one value per class.

Parameters:

binary (bool) –

acceptance_rate(as_dict=False)

Acceptance Rate. Alias for topr().

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

accuracy()

Accuracy

Return type:

float

class_accuracy(as_dict=False)

Class Accuracy

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

class_error_rate(as_dict=False)

Class Error Rate

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

error_rate()

Error Rate

Return type:

float

far(as_dict=False)

False Acceptance Rate. Alias for fpr().

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

far_ci(alpha=0.05, *, as_dict=False)

False Acceptance Rate confidence interval. Alias for fpr_ci().

Parameters:

• alpha (float) –

• as_dict (bool) –

Return type:

Union[dict, float, ndarray]

fdr(as_dict=False)

False Discovery Rate

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

fn(as_dict=False)

False Negatives

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

fnr(as_dict=False)

False Negative Rate

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

fnr_ci(alpha=0.05, *, as_dict=False)

False Negative Rate confidence interval

Parameters:

• alpha (float) –

• as_dict (bool) –

Return type:

Union[dict, float, ndarray]

for_(as_dict=False)

False Omission Rate

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

fp(as_dict=False)

False Positives

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

fpr(as_dict=False)

False Positive Rate

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

fpr_ci(alpha=0.05, *, as_dict=False)

False Positive Rate confidence interval

Parameters:

• alpha (float) –

• as_dict (bool) –

Return type:

Union[dict, float, ndarray]

frr(as_dict=False)

False Rejection Rate. Alias for fnr().

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

frr_ci(alpha=0.05, *, as_dict=False)

False Rejection Rate confidence interval. Alias for fnr_ci().

Parameters:

• alpha (float) –

• as_dict (bool) –

Return type:

Union[dict, float, ndarray]

n(as_dict=False)

Condition Negative

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

property nb_classes: int

Number of classes of confusion matrix

npv(as_dict=False)

Negative Predictive Value

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

one_vs_all()

Binarizes the confusion matrix using one-vs-all strategy.

For an input confusion matrix of shape (…, N, N) with N classes, the output is a vectorized binary confusion matrix of shape (…, N, 2, 2), where cm[…, j] is the confusion matrix of class j (pos) against all other classes (neg).

The one-vs-all operation is not idempotent. If we start with a binary confusion matrix, we will generate a vector of two matrices, so each class gets to play the role of the positive class.

Returns:

Vectorized binary confusion matrix of shape (…, N, 2, 2).

Return type:

ConfusionMatrix

p(as_dict=False)

Condition Positve

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

pop()

Population

Return type:

float

ppv(as_dict=False)

Positive Predictive Value

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

rejection_rate(as_dict=False)

Rejection Rate. Alias for tonr().

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

tar(as_dict=False)

True Acceptance Rate. Alias for tpr().

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

tar_ci(alpha=0.05, *, as_dict=False)

True Acceptance Rate confidence interval. Alias for tpr_ci().

Parameters:

• alpha (float) –

• as_dict (bool) –

Return type:

Union[dict, float, ndarray]

tn(as_dict=False)

True Negatives

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

tnr(as_dict=False)

True Negative Rate

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

tnr_ci(alpha=0.05, *, as_dict=False)

True Negative Rate confidence interval

Parameters:

• alpha (float) –

• as_dict (bool) –

Return type:

Union[dict, float, ndarray]

ton(as_dict=False)

Test Outcome Negative

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

tonr(as_dict=False)

Test Outcome Negative Rate.

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

top(as_dict=False)

Test Outcome Positive

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

topr(as_dict=False)

Test Outcome Positive Rate.

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

tp(as_dict=False)

True Positves

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

tpr(as_dict=False)

True Positve Rate

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

tpr_ci(alpha=0.05, *, as_dict=False)

True Positive Rate confidence interval

Parameters:

• alpha (float) –

• as_dict (bool) –

Return type:

Union[dict, float, ndarray]

trr(as_dict=False)

True Rejection Rate. Alias for tnr().

Parameters:

as_dict (bool) –

Return type:

Union[dict, float, ndarray]

trr_ci(alpha=0.05, *, as_dict=False)

True Rejection Rate confidence interval. Alias for tnr_ci().

Parameters:

• alpha (float) –

• as_dict (bool) –

Return type:

Union[dict, float, ndarray]