We welcome the ascension of our artificial overlords!

"When you have a chance to make a choice, make one that you know you won't regret." - Trailblazer from Honkai: Star Rail.

Certain problem domains demand a model that can not only accurately predict outcomes but also provide insights into the decision-making process. Traditional statistical methods often fall short in offering transparent and interpretable results. This is where a specific type of model excels, capable of mimicking human decision-making by creating a series of sequential rules.

Decision tree

Decision trees offer a human-interpretable model for classification and regression tasks. They construct a tree-like model where each internal node represents a test on an attribute, each branch represents the outcome of the test, and each leaf node represents a class label or a value. The components of a decision tree include:

• Root Node: the tree starts with a root node containing the entire dataset.

• Feature Selection: the algorithm selects the best attribute to split the data based on an impurity measure (e.g., information gain, Gini index).

• Node Splitting: the dataset is divided into subsets based on the chosen attribute's values.

• Recursive Process: the process is repeated for each subset until a stopping criterion is met (e.g., maximum depth, minimum number of samples per leaf).

Evaluation metrics for decision trees

To determine the best feature for splitting the data, decision tree algorithms employ impurity measures. In decision trees, impurity quantifies the disorder or randomness of the data points in a specific node. Common impurity measures include:

• Information Gain: measures the reduction in entropy – the uncertainty about class distribution within a dataset – after a dataset is split on an attribute. It quantifies how much information we gain about the target variable by splitting the data.

• Gini Index: measures the probability of incorrectly classifying a randomly chosen element in the dataset if it were randomly labeled according to the distribution of labels in the subset.

By iteratively selecting the best feature to split the data at each node and applying impurity measures, decision trees can effectively learn complex patterns from the training data.

Regression tree

Regression trees are a variant of decision trees specifically designed for predicting continuous numerical values rather than categorical labels. The core concept remains the same: creating a tree-like structure based on recursive partitioning of the data. They both have a root node, internal nodes, branches, and leaf nodes.

The key differences between the decision tree and regression tree are:

• Target Variable: regression trees predict a continuous value, while classification trees predict a categorical label.

• Evaluation Metric: instead of impurity measures like information gain or Gini index, regression trees often use metrics like mean squared error (MSE) or mean absolute error (MAE) to evaluate the quality of splits.

• Leaf Node Prediction: the prediction at each leaf node is the average of the target variable values for the training samples that fall into that region.

The image below depicts a regression tree model. It visualizes the decision-making process for predicting a continuous numerical value based on a single feature (x). For instance, if the input value of x is less than or equal to 0.496, the model follows the left branch, and further decisions are made based on subsequent nodes.

The mean squared error (MSE) values at each node indicate the prediction accuracy within that region. A lower MSE values imply a better fit.

Decision tree depth and complexity

As the decision tree depth grows, the model becomes more complex, better capturing the underlying linear relationship and reducing the impact of noise. This is evident in the closer alignment of predicted values to the true values as the depth increases.

While deeper trees can improve fit, they also increase the risk of overfitting. An overly complex tree might capture noise rather than the underlying signal, leading to poor generalization performance on new data. A shallow tree might underfit the data, exhibiting high bias and low variance. Conversely, a deep tree might overfit, leading to low bias but high variance. The optimal tree depth aims to balance these two factors.

Cart decision tree

CART (Classification and Regression Trees) is a versatile algorithm capable of handling both classification and regression problems. It constructs a binary tree, meaning each node has exactly two child nodes. Its key characteristics are:

• Binary Splits: at each internal node, the dataset is divided into two subsets based on a single feature and a threshold value.

• Impurity Measure: to determine the best split, CART employs an impurity measure. For classification, Gini impurity or information gain is commonly used. For regression, mean squared error (MSE) is preferred.

• Recursive Partitioning: the process of splitting the data into subsets continues until a stopping criterion is met (e.g., maximum depth, minimum number of samples in a leaf).

• Pruning: to prevent overfitting, CART often includes a pruning step to remove unnecessary branches from the tree.

CART builds a binary decision tree by iteratively splitting the data based on the feature that maximizes information gain or minimizes impurity. The resulting tree can be used for both classification and regression tasks, making it a flexible and widely used algorithm.

Considerations for using decision tree models

Decision trees are a popular machine learning algorithm due to their interpretability and ability to handle both categorical and numerical data, but they are still a model built for a niche function and has shortcomings when compared to several alternatives for machine learning.

• Advantages

  • Simplicity and Interpretability: their human-readable structure makes them easily understandable, facilitating model explainability.

  • Efficiency: decision-making at each node is straightforward, leading to relatively low computational costs.

  • Versatility: capable of handling both categorical and numerical data.

• Disadvantages

  • Prone to Overfitting: without proper regularization, decision trees can become overly complex, capturing noise in the data rather than underlying patterns.

  • Instability: small changes in the training data can lead to significantly different tree structures.

  • Limited Ability to Capture Complex Relationships: decision trees might struggle to capture complex patterns or interactions among features.