How to plot confusion matrix for prefetched dataset in Tensorflow

In Tensorflow, a common task is to plot a confusion matrix for a prefetched dataset. This is a good way to visualize the model's performance and identify any potential problems. In this article, we'll look at the basics of how to plot a confusion matrix for a tupled dataset. It's important to remember that this matrix is only a rough representation of the data; it does not represent actual data.

How to plot confusion matrix for prefetched dataset in Tensorflow

Solution 1:

Disclaimer: this won't work for shuffled datasets. I will update this answer as soon as I can.

You can use tf.stack to concatenate all the dataset values. Like so:

true_categories = tf.concat([y for x, y in test_dataset], axis=0)

For reproducibility, let's say you have a dataset, a neural network, and a training loop:

import tensorflow_datasets as tfds
import tensorflow as tf
from sklearn.metrics import confusion_matrix

data, info = tfds.load('iris', split='train',
                       as_supervised=True,
                       shuffle_files=True,
                       with_info=True)

AUTOTUNE = tf.data.experimental.AUTOTUNE

train_dataset = data.take(120).batch(4).prefetch(buffer_size=AUTOTUNE)
test_dataset = data.skip(120).take(30).batch(4).prefetch(buffer_size=AUTOTUNE)

model = tf.keras.Sequential([
    tf.keras.layers.Dense(8, activation='relu'),
    tf.keras.layers.Dense(16, activation='relu'),
    tf.keras.layers.Dense(info.features['label'].num_classes, activation='softmax')
    ])

model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', 
              metrics='accuracy')

history = model.fit(train_dataset, validation_data=test_dataset, epochs=50, verbose=0)

Now that your model has been fitted, you can predict the test set:

y_pred = model.predict(test_dataset)

array([[2.2177568e-05, 3.0841196e-01, 6.9156587e-01],
       [4.3539176e-06, 1.2779665e-01, 8.7219906e-01],
       [1.0816366e-03, 9.2667454e-01, 7.2243840e-02],
       [9.9921310e-01, 7.8686583e-04, 9.8775059e-09]], dtype=float32)

This is going to be a (n_samples, 3) array because we're working with three categories. We want a (n_samples, 1) array for sklearn.metrics.confusion_matrix, so take the argmax:

predicted_categories = tf.argmax(y_pred, axis=1)

<tf.Tensor: shape=(30,), dtype=int64, numpy=
array([2, 2, 2, 0, 2, 2, 2, 2, 1, 1, 2, 0, 0, 2, 1, 1, 1, 2, 0, 2, 1, 2,
       1, 0, 2, 0, 1, 2, 1, 0], dtype=int64)>

Then, we can take all the y values from the prefetch dataset:

true_categories = tf.concat([y for x, y in test_dataset], axis=0)

[<tf.Tensor: shape=(4,), dtype=int64, numpy=array([1, 1, 1, 0], dtype=int64)>,
 <tf.Tensor: shape=(4,), dtype=int64, numpy=array([2, 2, 2, 2], dtype=int64)>,
 <tf.Tensor: shape=(4,), dtype=int64, numpy=array([1, 1, 1, 0], dtype=int64)>,
 <tf.Tensor: shape=(4,), dtype=int64, numpy=array([0, 2, 1, 1], dtype=int64)>,
 <tf.Tensor: shape=(4,), dtype=int64, numpy=array([1, 2, 0, 2], dtype=int64)>,
 <tf.Tensor: shape=(4,), dtype=int64, numpy=array([1, 2, 1, 0], dtype=int64)>,
 <tf.Tensor: shape=(4,), dtype=int64, numpy=array([2, 0, 1, 2], dtype=int64)>,
 <tf.Tensor: shape=(2,), dtype=int64, numpy=array([1, 0], dtype=int64)>]

Then, you are ready to get the confusion matrix:

confusion_matrix(predicted_categories, true_categories)

array([[ 9,  0,  0],
       [ 0,  9,  0],
       [ 0,  2, 10]], dtype=int64)

(9 + 9 + 10) / 30 = 0.933 is the accuracy score. It corresponds to model.evaluate(test_dataset):

8/8 [==============================] - 0s 785us/step - loss: 0.1907 - accuracy: 0.9333

Also the results are consistent with sklearn.metrics.classification_report:

              precision    recall  f1-score   support
           0       1.00      1.00      1.00         8
           1       0.82      1.00      0.90         9
           2       1.00      0.85      0.92        13
    accuracy                           0.93        30
   macro avg       0.94      0.95      0.94        30
weighted avg       0.95      0.93      0.93        30

Here's the entire code:

import tensorflow_datasets as tfds
import tensorflow as tf
from sklearn.metrics import confusion_matrix

data, info = tfds.load('iris', split='train',
                       as_supervised=True,
                       shuffle_files=True,
                       with_info=True)

AUTOTUNE = tf.data.experimental.AUTOTUNE

train_dataset = data.take(120).batch(4).prefetch(buffer_size=AUTOTUNE)
test_dataset = data.skip(120).take(30).batch(4).prefetch(buffer_size=AUTOTUNE)

model = tf.keras.Sequential([
    tf.keras.layers.Dense(8, activation='relu'),
    tf.keras.layers.Dense(16, activation='relu'),
    tf.keras.layers.Dense(info.features['label'].num_classes, activation='softmax')
    ])

model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', 
              metrics='accuracy')

history = model.fit(train_dataset, validation_data=test_dataset, epochs=50, verbose=0)

y_pred = model.predict(test_dataset)

predicted_categories = tf.argmax(y_pred, axis=1)

true_categories = tf.concat([y for x, y in test_dataset], axis=0)

confusion_matrix(predicted_categories, true_categories)

Solution 2:

This code will work with shuffled tf.data.Dataset

y_pred = []  # store predicted labels
y_true = []  # store true labels

# iterate over the dataset
for image_batch, label_batch in dataset:   # use dataset.unbatch() with repeat
   # append true labels
   y_true.append(label_batch)
   # compute predictions
   preds = model.predict(image_batch)
   # append predicted labels
   y_pred.append(np.argmax(preds, axis = - 1))

# convert the true and predicted labels into tensors
correct_labels = tf.concat([item for item in y_true], axis = 0)
predicted_labels = tf.concat([item for item in y_pred], axis = 0)

Thank you for reading the article.