Common Supervised Machine Learning Algorithms

Two Types of Prediction Problems

Supervised machine learning algorithms take input data and map it to output data in order to provide predictions or probability distributions. These algorithms attempt to solve two main types of prediction problems: regression and classification.

Regression Problems

The output of a regression algorithm is a continuous real value that shows a predicted distribution over time. This type of algorithm uses input data to create a model and outputs a numerical value. Examples include:

Classification Problems

Classification problems take input data and categorize that data using discrete labels. They take input data and predict the most probable category. Examples include:

Common Algorithms

The following algorithms are some of the most commonly used in the field. They each attempt to solve one or both of the prediction problems discussed above.

Linear Regression

Linear regression algorithms are used for regression problems. Based on features, this algorithm predicts continuous values and creates a linear representation of data. In a data plot with an x- and y-axis, the regression line minimizes the difference from line to points.

Logistic Regression

Logistic regression is used for classification problems and predicts a binary classification by creating an s curve. While this classification creates a curve, it actually outputs a discrete yes or no result.

Decision Tree

Decision trees are used for classification problems. In a training stage, data is split into homogenous groups and subgroups. Groups are based on common features that share a higher predictive power. As an example, Groups A, B, and C could have subgroups A-1 and A-2; B-1 and B-2; and C-1 and C-2.

Example of a decision tree

In this example, a decision tree is used to predict if it will rain tomorrow, using values of humidity, cloud ceiling, and temperature.

An algorithm in the training stage takes weather data over a period of time and identifies common factors related to the probability of rain. Input weather data is then correlated to whether it rained on the following day. In the example below, a decision tree shows the number of times it rained (yes) or didn't (no) over the course of 50 days.

These 50 days of data are first divided into a yes/no binary: Rain tomorrow? In this dataset, it rained 30 days and did not rain on 20.

The first branch in the tree is related to cloud ceiling values of low, none, and high. Here is the breakdown of days it rained or did not rain, connected to each of the cloud ceiling values:

In a decision tree, group data by feature and relation to outcome. Divide the groupings of your data until you are as close as possible to a pure subgroup where 100% of the records have the same outcome. For example, in this dataset there was only one time that there was no cloud ceiling and it went on to rain the next day. Based on this data, the algorithm will predict no rain for the next day if it gets data with no cloud ceiling.

Another branch of values divides the days with low and high cloud ceiling according to humidity levels. The results are as follows:

As you can see, if cloud ceiling is low and humidity is high, rain is predicted. If cloud ceiling is high and humidity low, rain is also predicted.

One further branch of values divides the days with low and high humidity according to temperature. The results are as follows:

If cloud ceiling, humidity, and temperature are high, showing a 90% chance of rain, the algorithm will output a prediction of rain for the following day.

Strengths and weaknesses of decision trees

Decision trees are good at finding complex relationships between pieces of data. In this example, humidity affects rain unless cloud ceiling is not present.

Weaknesses are that decision trees are prone to overfitting a specific set of data, thereby not being applicable to data in a broader context.

Random Forest

A random forest is used for classification and regression problems.

A random forest is an ensemble of decision trees that each "vote" for one class. Each tree receives a bias that makes its vote count for more or less. The class with the most votes is selected as the output of the model.

For example, with the question Will it rain tomorrow? each tree only sees a random subset of the features and votes yes or no. In this example, three trees vote yes and one votes no.

Strengths and weaknesses of random forests

Strengths are that similar to a decision tree, random forests find complex relationships among pieces of data. Weaknesses are that it is computationally expensive when there are a lot of columns.

Neural Network

Neural networks are for classification and regression problems. They are commonly used to classify data by feeding it through a set of interconnected "neurons."

In this example, data is coming in from a table containing measurements of different flowers. The neural network sees three columns: stem length, petal width, and petal length. It then predicts the species of the flower.

Three layers are part of this neural network:

Each layer has what are referred to as neurons. Each neuron in the input layer typically represents one feature and corresponds to a class. In this example, data about a yellow flower is fed into the network and outputs a value of .08 for Rose, .26 for Lily, and .97 for Daffodil.

About the hidden layer

Backpropagation

In the training stage, neural networks compare the result of their predictions with the actual result and gradually update the weights and biases of each synapse until they reach their optimal accuracy.

Strengths and weaknesses of neural networks

Neural networks are an extremely powerful method for finding almost any kind of pattern in data. A weakness is that it may focus on the wrong pattern in the data, and it is hard to interpret and debug.

Close this window to return to the main page.