What does Transfer Learning mean?

Transfer learning means reusing a model that was already trained on a large dataset for a new task. The lower layers are kept as they are, because they recognize general features such as edges, shapes, or word meanings that every application needs. Only the upper layers, tailored to the specific task, get retrained.

The starting point is a pre-trained model — an image model trained on ImageNet, say, or a language model trained on a large body of text. The lower layers are frozen, meaning their weights are excluded from further adjustment. The upper part is replaced with new layers and trained on the organization’s own dataset, often with a few hundred to a few thousand examples. With fine-tuning, the frozen layers are then adjusted too, using a small learning rate.

Transfer learning pays off where an organization’s own dataset is too small for training from scratch. Examples include recognizing proprietary components in images, classifying domain-specific text, and analyzing medical scans. If the new task differs strongly from the original training material, the benefit drops noticeably.

The advantage over training from scratch lies in the demand for data and compute time. Instead of millions of examples, a few hundred suffice, and a training run takes hours instead of weeks. That puts a custom model within reach even for organizations without large datasets or their own compute clusters.

An adopted model brings along the properties of its original training, including any existing biases and gaps. The origin, license, and training basis of the source model therefore belong in the documentation of the resulting system. For models from open collections, both are not always stated.