Creations tend to inherit its creators' traits.

Continual learning

Continual learning is a machine learning approach where a model learns from a stream of data with changing distributions, aiming to transfer knowledge between different tasks. It is designed to enable models to adapt to new information without forgetting previously learned knowledge, mimicking how humans learn and adapt.

Think of traditional AI learning like a chef who memorizes and perfects their set menus, then then sticking strictly to what they learned, never adding new dishes. By contrast, continual learning is like a street-food chef who keeps experimenting every day, constantly tweaking recipes, and retiring stale ones on the fly to suit evolving tastes.

Catastrophic forgetting

Self-supervised learning

Supervised learning (SL) can become very costly in time and money because it relies on large volumes of high-quality, human-labeled data. Comparing the hypothetical prices between labelling 1 million and 1 billion images:

• 1M: 1,000,000 images * 10 seconds per image * 3600 seconds * $15 per hour = $41,667

• 1B: 1,000,000,000 images * 10 seconds per image * 3600 seconds * $15 per hour = $41,666,667

Self-supervised learning (self-SL)

Self-supervised learning: pretext tasks

Self-SL pretext tasks:

• Generative: Predict part of the input signal.

• Discriminative: Predict something about the input signal.

• Multimodal: Use some additional signal in addition to RGB images.

Generative modelling aims to learn how the data is produced to can create brand new samples, whereas SSL want to learn good features by making up a task from the data itself, so that it can then do whatever it 'likes' with those features.

To evaluate an Self-SL model, check whether it can:

1. Learn good feature extractors from self-supervised pretext tasks, e.g., predicting image rotations.

2. Attach a shallow network on the feature extractor, then train it on target task with small amount of labeled data.