### How can I control classifier parameter optimization?

PRTools has some built-in possibilities to optimize parameters used for defining classifiers. For instance, the quadratic classifier based on the assumption of normal distributions, `qdc`

, has some parameters to control the regularization. They adapt the estimated covariance matrix into the direction of a diagonal matrix based on the feature variances, or the average feature variance. A call like

`A = qdc(A,alf)`

regularizes the covariance matrix by a weighted average with a diagonal matrix. The weights are `1-alf`

and `alf`

. Users may try a few values like 1e-2 and 1e-3, or, instead, set `alf`

= `NaN`

. In this case PRTools optimizes `alf`

by at most 20 calls to the Matlab optimizer `fminbnd`

, on an interval set inside `qdc`

. The criterion is a 5-fold cross-validation error, see below for details. A soft error criterion is used while a crisp error causes problems as minor changes in `alf`

may not cause a difference in the error.This procedure may slow down training by a factor of 100 (5 x 20).

It is also possible to optimize more than one parameter simultaneously. In this case not a full grid search is performed, but the parameters are optimized one after the other in a on order specified by the user. Defaults are used for parameters not yet optimized.

This procedure is controlled by the PRTools routine `regoptc`

, called when one of the parameters is `NaN`

. This might not be programmed in all functions. Moreover, users may want to control some settings. This can be solved by calling `regoptc`

externally. Here is an example based on the PRTools support vector classifier `svc`

. We will simultaneously optimize the degree of the polynomial kernel and the trade-off parameter C.

`A = gendatb; % dataset`

pars = {'p',NaN,NaN}; % initial parameters. NaNs will be optimized

defs = {'p',1,1}; % set defaults for all classifier parameters

parmin_max = [0, 0; 1,10; 1e-2, 1e2]; % set interval for every parameter

par_order = [2 3 1]; % optimization order of parameters

realint = [1 0 1]; % 2nd par is integer, others real

testfunc =`testc([],'soft')`

; % test function

W = regoptc(a,'svc',pars,defs,par_order,parmin_max,testfunc,realint);

delfigs; scatterd(A); plotc(W)

getopt_pars % show final parameter values

This example usually finds a 3rd order polynomial kernel and a trade-off parameter C that is somewhat larger than 1. The test function should be a mapping that can be called like `E = A*W*testfunc`

for a computation of the criterion to be minimized.

The call to regoptc is somewhat inconsistent. It requires that the classifier function is given by a string and the test function by a mapping. This is needed as the regoptc needs to know what parameters have to be optimized. Parameter values to be tested are determined by `fminbnd`

(or `nfminbnd`

for integer parameters). The function to be minimized is an error estimated by cross validation using the PRTools routine `prcrossval`

. The user specified mapping `testfunc`

(see the above example) is used inside this routine. The number of folds and the number of repetitions are determined by the globals `REGOPT_NFOLDS`

and `REGOPT_REPS`

, preset by 5 and 1. The maximum number of iterations for `fminbnd`

and `nfminbnd`

is set `by REGOPT_ITERMAX`

, preset by 20. Users may change these values by `prglobal`

.