Gaussian 2009 Tutorial How To Implement Gaussian

We will build up deeper understanding on how to implement Gaussian process regression from scratch on a toy example.

This might not mean much at this moment so lets dig a bit deeper in its meaning.

Even if the starting point is known, there are several directions in which the processes can evolve.

We can simulate this process over time t in 1 dimension d by starting out at position 0 and move the particle over a certain amount of time Delta t with a random distance Delta d from the previous position.The random distance is sampled from a.

Sampling Delta d from this normal distribution is noted as Delta d sim mathcalN(0, Delta t).

The position d(t) at time t evolves as d(t Delta t) d(t) Delta d.

We can sample a realization of a function from a stochastic process.

However each realized function can be different due to the randomness of the stochastic process.

In fact, the Brownian motion process can be reformulated as a Gaussian process.

Since functions can have an infinite input domain, the Gaussian process can be interpreted as an infinite dimensional Gaussian random variable.

By choosing a specific kernel function k it is possible to set.

Note that the exponentiated quadratic covariance decreases exponentially the further away the function values x are from eachother.

We do this by drawing correlated samples from a 41-dimensional Gaussian mathcalN(0, k(X, X)) with X X1, ldots, X41.

This is because these marginals come from a Gaussian process with as prior the exponentiated quadratic covariance which adds prior information that points close to eachother in the input space X must be close to eachother in the output space y.

This can be done with the help of the posterior distribution p(mathbfy2 mid mathbfy1,X1,X2).

Since they are jointly Gaussian and we have a finite number of samples we can write.

Notice that the mean of the posterior predictions mu21 of a Gaussian process are weighted averages of the observed variables mathbfy1, where the weighting is based on the coveriance function k.

The variance sigma22 of these predictions is then the diagonal of the covariance matrix Sigma21.

The top figure shows the distribution where the red line is the posterior mean, the grey area is the 95 prediction interval, the black dots are the observations (X1,mathbfy1).

The prediction interval is computed from the standard deviation sigma21, which is the square root of the diagonal of the covariance matrix.

The bottom figure shows 5 realizations (sampled functions) from this distribution.

We can make predictions from noisy observations f(X1) mathbfy1 epsilon, by modelling the noise epsilon as Gaussian noise with variance sigmaepsilon2.

Note that the noise only changes kernel values on the diagonal (white noise is independently distributed).