Regularizer

Regularization prevent overfitting and introduce additional information (prior knowledge) to solve an ill-posed problem.

Regularizers implement the following main methods:

value(r::Regularizer)

Compute the value of the regularizer.

gradient(r::Regularizer)

Compute the gradient of the regularizer.

The following regularizers are implemented:

L2reg(w::Vector, λ::Float64)

Implements an \(L^2\)-norm regularization of the weight vector w of the decision function:

\[\begin{split}\Omega({\bf w})&=\frac{1}{2\lambda}\|{\bf w}\|^2,\end{split}\]

where the regularization parameter \(\lambda\) controls the influence of the regularizer.

Note

The \(L^2\)-norm regularization corresponds to Gaussian prior assumption of \({\bf w}\sim\mathcal{N}({\bf 0},\lambda{\bf I})\).