How can I control classifier parameter optimization?
PRTools has some built-in possibilities to optimize parameters used for defining classifiers. For instance, the quadratic classifier based on the assumption of normal distributions, qdc, has some parameters to control the regularization. They adapt the estimated covariance matrix into the direction of a diagonal matrix based on the feature variances, or the average feature variance. A call like
A = qdc(A,alf)
regularizes the covariance matrix by a weighted average with a diagonal matrix. The weights are 1-alf and alf. Users may try a few values like 1e-2 and 1e-3, or, instead, set alf = NaN. In this case PRTools optimizes alf by at most 20 calls to the Matlab optimizer fminbnd, on an interval set inside qdc. The criterion is a 5-fold cross-validation error, see below for details. A soft error criterion is used while a crisp error causes problems as minor changes in alf may not cause a difference in the error.This procedure may slow down training by a factor of 100 (5 x 20).
It is also possible to optimize more than one parameter simultaneously. In this case not a full grid search is performed, but the parameters are optimized one after the other in a on order specified by the user. Defaults are used for parameters not yet optimized.
This procedure is controlled by the PRTools routine regoptc, called when one of the parameters is NaN. This might not be programmed in all functions. Moreover, users may want to control some settings. This can be solved by calling regoptc externally. Here is an example based on the PRTools support vector classifier svc. We will simultaneously optimize the degree of the polynomial kernel and the trade-off parameter C.
A = gendatb; % dataset
pars = {'p',NaN,NaN}; % initial parameters. NaNs will be optimized
defs = {'p',1,1}; % set defaults for all classifier parameters
parmin_max = [0, 0; 1,10; 1e-2, 1e2]; % set interval for every parameter
par_order = [2 3 1]; % optimization order of parameters
realint = [1 0 1]; % 2nd par is integer, others real
testfunc =testc([],'soft'); % test function
W = regoptc(a,'svc',pars,defs,par_order,parmin_max,testfunc,realint);
delfigs; scatterd(A); plotc(W)
getopt_pars % show final parameter values
This example usually finds a 3rd order polynomial kernel and a trade-off parameter C that is somewhat larger than 1. The test function should be a mapping that can be called like E = A*W*testfunc for a computation of the criterion to be minimized.
The call to regoptc is somewhat inconsistent. It requires that the classifier function is given by a string and the test function by a mapping. This is needed as the regoptc needs to know what parameters have to be optimized. Parameter values to be tested are determined by fminbnd (or nfminbnd for integer parameters). The function to be minimized is an error estimated by cross validation using the PRTools routine prcrossval. The user specified mapping testfunc (see the above example) is used inside this routine. The number of folds and the number of repetitions are determined by the globals REGOPT_NFOLDS and REGOPT_REPS, preset by 5 and 1. The maximum number of iterations for fminbnd and nfminbnd is set by REGOPT_ITERMAX, preset by 20. Users may change these values by prglobal.
