Regression

Standard regression problems

In a regression problem, we would typically have some input vectors x and some desired output values y. Note that, differently from classification problems, here the output values y are not restricted to be class labels, but can rather be continuous variables or vectors.

Models

Linear Regression

Let's say we have some univariate, continuous sets of input data, and a corresponding univariate, continuous set of output data, such as a set of points in R². A simple linear regression is able to fit a line relating the input variables to the output variables in which the minimum-squared-error of the line and the actual output points is minimum.

// Declare some sample test data.
double[] inputs = { 80, 60, 10, 20, 30 };
double[] outputs = { 20, 40, 30, 50, 60 };

// Use Ordinary Least Squares to learn the regression
OrdinaryLeastSquares ols = new OrdinaryLeastSquares();

// Use OLS to learn the simple linear regression
SimpleLinearRegression regression = ols.Learn(inputs, outputs);

// Compute the output for a given input:
double y = regression.Transform(85); // The answer will be 28.088

// We can also extract the slope and the intercept term
// for the line. Those will be -0.26 and 50.5, respectively.
double s = regression.Slope;     // -0.264706
double c = regression.Intercept; // 50.588235

See Simple Linear Regression

Multivariate Linear Regression

The multivariate linear regression is a generalization of the multiple linear regression. In the multivariate linear regression, not only the input variables are multivariate, but also are the output dependent variables.

In the following example, we will perform a regression of a 2-dimensional output variable over a 3-dimensional input variable.

double[][] inputs = 
{
    // variables:  x1  x2  x3
    new double[] {  1,  1,  1 }, // input sample 1
    new double[] {  2,  1,  1 }, // input sample 2
    new double[] {  3,  1,  1 }, // input sample 3
};

double[][] outputs = 
{
    // variables:  y1  y2
    new double[] {  2,  3 }, // corresponding output to sample 1
    new double[] {  4,  6 }, // corresponding output to sample 2
    new double[] {  6,  9 }, // corresponding output to sample 3
};

With a quick eye inspection, it is possible to see that the first output variable y1 is always the double of the first input variable. The second output variable y2 is always the triple of the first input variable. The other input variables are unused. Nevertheless, we will fit a multivariate regression model and confirm the validity of our impressions:

// Use Ordinary Least Squares to create the regression
OrdinaryLeastSquares ols = new OrdinaryLeastSquares();

// Now, compute the multivariate linear regression:
MultivariateLinearRegression regression = ols.Learn(inputs, outputs);

// We can obtain predictions using
double[][] predictions = regression.Transform(inputs);

// The prediction error is
double error = new SquareLoss(outputs).Loss(predictions); // 0

See Multivariate Linear Regression

Multiple Linear Regression

We will try to model a plane as an equation in the form "ax + by + c = z". We have two input variables (x and y) and we will be trying to find two parameters a and b and an intercept term c.

// We will use Ordinary Least Squares to create a
// linear regression model with an intercept term
var ols = new OrdinaryLeastSquares()
{
    UseIntercept = true
};

// Now suppose you have some points
double[][] inputs = 
{
    new double[] { 1, 1 },
    new double[] { 0, 1 },
    new double[] { 1, 0 },
    new double[] { 0, 0 },
};

// located in the same Z (z = 1)
double[] outputs = { 1, 1, 1, 1 };

// Use Ordinary Least Squares to estimate a regression model
MultipleLinearRegression regression = ols.Learn(inputs, outputs);

// As result, we will be given the following:
double a = regression.Coefficients[0]; // a = 0
double b = regression.Coefficients[1]; // b = 0
double c = regression.Intercept; // c = 1

// This is the plane described by the equation
// ax + by + c = z => 0x + 0y + 1 = z => 1 = z.

// We can compute the predicted points using
double[] predicted = regression.Transform(inputs);

// And the squared error loss using 
double error = new SquareLoss(outputs).Loss(predicted);

See Multiple Linear Regression and Partial Least Squares

Logistic Regression

See Logistic regression, Logistic Regression Analysis and Generalized Linear Models.

Multinomial Logistic Regression (Softmax)

See Multinomial Logistic Regression.

Support Vector Machines

See Sequential Minimal Optimization for Regression, L1-regularized logistic regression, L2-regularized logistic regression in the dual and L2-regularized L2-loss logistic regression.

Neural Networks

See Levenberg-Marquardt with Bayesian Regularization and Resilient Backpropagation.

Variations

Regression models censored in time

See Cox's Proportional Hazards Model