PRTools examples: Combining Classifiers

The construction of new, dedicated classifiers by combining traditional ones available in a toolbox is an importent technique for almost every advanced application. It is assumed that the reader is familiar with the introductory sections of the user guide:

• The PRTools elements
• The basics of the classifiers
• Operations on classifiers

Combining classifiers is an extensive area. PRTools offers a large set of combining rules and constructs. Here just a few examples are presented.

• Stacked combining
• Parallel combining
• Sequential combining
• Fixed combiners
• Trained combiners

Stacked combining

By stacked combining a set of classifiers in the same space are combined. They may be trained by the same or by different training sets. A test object to be classified is applied to all classifiers and results are combined. Preferably we combine posteriors or confidences. So classc should be appliedStacked combining is defined by horizontal concatenation of classifiers.

T = gendatb;              % train set
S = gendatb([1000 1000]); % test set
W1 = T*parzenc*classc;    % train first classifier
W2 = T*fisherc*classc;    % train second classifier
W3 = T*qdc*classc;        % train third classifier
W = [W1 W2 W3];           % stacked combining
Cmax = W*maxc;            % max combiner
Cmin = W*minc;            % min combiner
Cmean = W*meanc;          % mean combiner
Cprod = W*prodc;          % product combiner
testc(S,{Cmac,Cmin,Cmean,Cproc}) % test

The max combiner and the min combiner yield the same result for two-class problems. Several shortcuts are possible, e.g. for the last 5 lines:

testc(S,W*{maxc,minc,meanc,prodc})

or for the entire sequence:

testc(S,T*[parzenc,fisherc,qdc]*classc*{maxc,minc,meanc,prodc})

Note that in this last construct the untrained classifiers are combined.

Parallel combining

Parallel combining combines classifiers in different feature spaces. They may be trained by different datasets as well as by the same objects represented in all spaces. Test objects, however, should be represented in all spaces, otherwise they cannot be applied to the classifiers to be combined. In the example we use three of the mfeat datasets. They refer to the same objects. In the generation of train and test sets the alignment of objects should be maintained. This needs some care. It is done here by resetting the random generator.

A1 = mfeat_kar;
A2 = mfeat_mor;
A3 = mfeat_fac;
randreset; [T1,S1] = gendat(A1,0.5);
randreset; [T2,S2] = gendat(A2,0.5);
randreset; [T3,S3] = gendat(A3,0.5);
W1 = T1*ldc*classc; % train first classifier
W2 = T2*dtc*classc; % train second classifier
W3 = T3*(pcam([],10)*qdc*classc); % third classifier
W = [W1; W2; W3];    % parallel combining by vertical concatenation
testc([S1 S2 S3],{W*maxc,W*minc,W*meanc,W*prodc});

Note that in the last statement the test objects in the three space have to be concatenated horizontally. Effectively this implies that the spaces are concatenated. The three classifiers, however, live in separate subspaces of this extended space.

Sequential combining

Classifiers can be sequentially combined with various types of mappings, mainly used for obtaining better feature spaces. Surprisingly it is also possible to combine classifiers sequentially. In this way a second classifier operates in the output space, e.g. the class confidences, of the first.

A = gendatm
w1 = fisherc;
w2 = qdc;
w3 = qdc*classc*fisherc;
w4 = fisherc*classc*qdc;
prcrossval(A,{w1,w2,w3,w4},2,5)

Occasionally sequentially combined classifiers like w3 and w4 perform better, especially in multi-class problems. This has not been researched yet.

Fixed combiners

Fixed combiners operate in a data independent way on the classification matrix of the combined classifier. This is always a horizontally concatenated combination of the outputs of the $$n$$ base classifiers. Several columns point to the same class, one for each base classifier. The fixed combining rule reduced the set of $$n c$$ columns ($$c$$ is the number of classes) to just $$c$$ columns.

Above some examples of fixed combining rules are applied. The total set of PRTools combiners is available here.

Trained combiners

The combining rule applied to the classification matrix resulting from the combined classifier may be adapted to a training set. This can be the same dataset as used for training the base classifiers. It may also be a fresh dataset, if available, to avoid bias. Training by the same dataset is straightforward and can be done in the same lines as by other sequential sets of mappings, e.g.

A = gendath([200 200]);
W = [fisherc,nmc,svc]*fisherc;
W = setname(W,'TrainedCC');
prcrossval(A,{fisherc,nmc,svc,W},5,2)

Here we use a standard classifier (fisherc) for combining the base classifiers. Some of the special combining rules are trainable, e.g. dcsc, modselc, naivebcc and mlrc.

elements: datasets datafiles cells and doubles mappings classifiers mapping types.
operations: datasets datafiles cells and doubles mappings classifiers stacked parallel sequential dyadic.
user commands: datasets representation classifiers evaluation clustering examples support routines.
introductory examples: Introduction Scatterplots Datasets Datafiles Mappings Classifiers Evaluation Learning curves Feature curves Dimension reduction Combining classifiers Dissimilarities.
advanced examples.