A definition

Metrics are a human-understandable measures of model quality whereas the loss is the machine's. They are based on your validation set and are what you really care about, whereas the loss is "a measure of performance" that the training system can use to update weights automatically.

A good choice for loss is a function "that is easy for stochastic gradient descent (SGD) to use, whereas a good choies for your metrics are functions that your business users will care about. Seldom are they the same because most metrics don't provide smooth gradients that SGD can use to update your model's weights.

Note: Again, they are based on your validation/test sets (not your training set). Ultimately, we want to have a model that generalizes well to inputs it was not trained on, and this is what our validation/test sets represent. This is how we relay our model quality.

Examples

There are a whole list of metrics built into the fastai library, see here. Below I begin a listing of the most common ones as they come up in the fastbook (and from personal experience).

error rate = "the proportion of images that were incorrectly identified." 1

accuracy = the proportation of images that were correctly identified (1 - error rate)


Metrics to use based on task

Metric Multiclass classification Multilabel classification Regression
error rate Yes Yes* No
accuracy Yes Yes* No

* indicates that other metrics may be better for the given task.


1. "Chaper 1: Your Deep Learning Journey". In The Fastbook p.19