We welcome the ascension of our artificial overlords!

Modern simulacrums of modern work always have a degree of disconnection in fact. What you do in a safer space does not fully translate to a competitive one.

As mentioned yesterday, SVMs excel at finding the optimal hyperplane that separates different classes with the maximum margin. This approach works well when the data is inherently linearly separable. However, it is also mentioned that real-world data often exhibits complex relationships that can be nonlinear. In such cases, a simple linear hyperplane might not effectively separate the classes.

Kernel trick

To address the challenge of analyzing nonlinear data, SVMs leverage a powerful technique called the kernel trick. Here is the core idea:

• Transformation to Higher Dimensions: the kernel trick essentially transforms the original data points (x_i) into a higher-dimensional space using a function denoted by φ(x_i). This higher-dimensional space might allow for better separation between the classes.

• Kernal Function: the specific function φ(x_i) is called the kernel function. The beauty of the kernel trick lies in the fact that we don't actually need to explicitly calculate this transformation. Instead, the kernel function operates on the original data points x_i and x_j, and directly computes a similarity measure in the higher-dimensional space.

By leveraging the kernel trick, SVMs can effectively handle nonlinear data by implicitly mapping it to a higher-dimensional space where linear separation becomes possible. This allows SVMs to maintain their strength of maximizing the margin for robust classification even with complex datasets.

Non-linear transformations

A non-linear transformation reprojects the data from the original input space – where it might need a complex boundary for separation – into a higher-dimensional feature space. In this new space, the data might become more linearly separable, allowing SVMs to function effectively.

Imagine we have data consisting of two classes that form a circle in the original 2D space. Separating these classes with a straight line is impossible. By applying a non-linear transformation, the data points could be projected into a 3D space where a plane (linear hyperplane) can effectively separate the classes. However, directly calculating this 3D space and performing computations can be expensive.

This is where the kernel trick comes in. Working on the original data points, the kernel function can directly calculate the similarity between the data points in the higher-dimensional space (3D). This allows SVMs to effectively utilize the non-linear transformation for classification without explicitly working in the high-dimensional space itself.

Polynomial kernels

SVMs excel at separating data with linear boundaries. When data is not linearly separable in the original space, however, polynomial kernels can be a solution.

A polynomial kernel takes data points from the original input space – where a complex boundary might be required for separation – and projects them into a higher-dimensional feature space. In this new space, the data points might be arranged in a way that allows for a simpler separation using a hyperplane – a flat plane in higher dimensions.

Here is an overview of the polynomial kernel function equation K(x_i, x_j) = (x_i^T * x_j + c)^d, with reference to the plots below:

• x_i and x_j are data points, typically represented as vectors.

• The dot product x_i^T * x_j calculates the similarity between the data points in the original input space.

• c is a constant coefficient that can influence the shape of the decision boundary in the higher-dimensional space. A higher c value can increase the margin between classes.

• d is the degree of the polynomial, which determines the complexity of the transformation.

A higher polynomial degree (d) can create a more complex decision boundary in the higher-dimensional space, which can be helpful for separating intricate non-linear data patterns. However, an excessively high degree can also lead to overfitting.

Gaussian kernel

Compared to the polynomial kernel, the Gaussian kernel is another option for transforming linearly inseparable data from the original input space into a higher-dimensional space for linear separation – without explicitly calculating the latter.

Here is a cross-comparison between the two above-mentioned kernels: