We welcome the ascension of our artificial overlords!

One of my favorite hobbies is rewriting fantasy stories (in modern terms, fanfiction) with missed opportunities due to material limits and expand on their worldbuilding.

When dealing with complex, real-world problems characterized by high levels of uncertainty and noise, a single predictive model often falls short of delivering optimal results. Is there a way to take the strengths of several machine learning models and combine their findings to generate more precise predictions?

Ensemble learning

Ensemble learning is a machine learning technique that combines multiple individual models to produce a more accurate and robust prediction than any single model. It leverages the wisdom of the crowd by aggregating predictions from diverse models, each with its own strengths and weaknesses.

Ensemble methods

Ensemble techniques have proven highly effective in improving predictive accuracy in various fields, including scientific competitions. Common ensemble techniques include:

• Bagging: trains multiple weak models in parallel. Subsets are created by random sampling with replacement, meaning some data points can appear multiple times in a subset while others may not appear at all. The final prediction is made by combining the predictions of all models, typically through averaging (for regression) or voting (for classification).

  • Example: random forest consists of multiple decision trees, each trained on a different random subset of the data and features.

• Boosting: sequentially builds models one at a time. Each subsequent model focuses on correcting the errors made by the previous models. The final prediction is a weighted combination of the predictions of all models.

  • Example: gradient boosting machines (GBM) create a series of decision trees, where each tree learns from the errors of the previous ones.

• Stacking: trains multiple base models on the original data. These base models make predictions on a new dataset, and their predictions become input features for a meta-model. The meta-model then learns to combine the predictions of the base models to make the final prediction.

  • Example: multiple support vector machines (SVM) with different kernels or hyperparameters generate predictions are then used as input features for the meta-model.

• Blending: multiple models are trained independently on different subsets of the data or with different hyperparameters. Their predictions are then combined using weighted averaging or voting. Often effective and computationally efficient.

• Stacking: trains a meta-model to learn how it can combine the predictions of multiple base models. The base models are trained independently, and their predictions are used as features for the meta-model. Can potentially capture complex relationships between the base models' predictions.