This application is the national phase under 35 U.S.C. § 371 of PCT International Application No. PCT/DK99/00340 which has an International filing date of Jun. 21, 1999, which designated the United States of America.
1. Field of the Invention
The present invention relates generally to n-tuple or RAM based neural network classification systems and, more particularly, to n-tuple or RAM based classification systems where the decision criteria applied to obtain the output scores and compare these output scores to obtain a classification are determined during a training process.
2. Description of the Prior Art
A known way of classifying objects or patterns represented by electric signals or binary codes and, more precisely, by vectors of signals applied to the inputs of neural network classification systems lies in the implementation of a so-called learning or training phase. This phase generally consists of the configuration of a classification network that fulfils a function of performing the envisaged classification as efficiently as possible by using one or more sets of signals, called learning or training sets, where the membership of each of these signals in one of the classes in which it is desired to classify them is known. This method is known as supervised learning or learning with a teacher.
A subclass of classification networks using supervised learning are networks using memory-based learning. Here, one of the oldest memory-based networks is the “n-tuple network” proposed by Bledsoe and Browning (Bledsoe, W. W. and Browning, 1, 1959, “Pattern recognition and reading by machine”, Proceedings of the Eastern Joint Computer Conference, pp. 225–232) and more recently described by Morciniec and Rohwer (Morciniec, M. and Rohwer, R., 1996, “A theoretical and experimental account of n-tuple classifier performance”, Neural Comp., pp. 629–642).
One of the benefits of such a memory-based system is a very fast computation time, both during the learning phase and during classification. For the known types of n-tuple networks, which is also known as “RAM networks” or “weightless neural networks”, learning may be accomplished by recording features of patterns in a random-access memory (RAM), which requires just one presentation of the training set(s) to the system.
The training procedure for a conventional RAM based neural network is described by Jørgensen (co-inventor of this invention) et al. in a contribution to a recent book on RAM based neural networks (T. M. Jørgensen, S. S. Christensen, and C. Liisberg, “Cross-validation and information measures for RAM based neural networks,” RAM-based neural networks, J. Austin, ed., World Scientific, London, pp. 78–88, 1998). The contribution describes how the RAM based neural network may be considered as comprising a number of Look Up Tables (LUTs). Each LUT may probe a subset of a binary input data vector. In the conventional scheme the bits to be used are selected at random. The sampled bit sequence is used to construct an address. This address corresponds to a specific entry (column) in the LUT. The number of rows in the LUT corresponds to the number of possible classes. For each class the output can take on the values 0 or 1. A value of 1 corresponds to a vote on that specific class. When performing a classification, an input vector is sampled, the output vectors from all LUTs are added, and subsequently a winner takes all decision is made to classify the input vector. In order to perform a simple training of the network, the output values may initially be set to 0. For each example in the training set, the following steps should then be carried out:
Present the input vector and the target class to the network, for all LUTs calculate their corresponding column entries, and set the output value of the target class to 1 in all the “active” columns.
By use of such a training strategy it may be guaranteed that each training pattern always obtains the maximum number of votes on the true class. As a result such a network makes no misclassification on the training set, but ambiguous decisions may occur. Here, the generalisation capability of the network is directly related to the number of input bits for each LUT. If a LUT samples all input bits then it will act as a pure memory device and no generalisation will be provided. As the number of input bits is reduced the generalisation is increased at an expense of an increasing number of ambiguous decisions. Furthermore, the classification and generalisation performances of a LUT are highly dependent on the actual subset of input bits probed. The purpose of an “intelligent” training procedure is thus to select the most appropriate subsets of input data.
Jørgensen et al. further describes what is named a “leave-one-out cross-validation test” which suggests a method for selecting an optimal number of input connections to use per LUT in order to obtain a low classification error rate with a short overall computation time. In order to perform such a cross-validation test it is necessary to obtain a knowledge of the actual number of training examples that have visited or addressed the cell or element corresponding to the addressed column and class. It is therefore suggested that these numbers are stored in the LUTs. It is also suggested by Jørgensen et al. how the LUTs in the network can be selected in a more optimum way by successively training new sets of LUTs and performing cross validation test on each LUT. Thus, it is known to have a RAM network in which the LUTs are selected by presenting the training set to the system several times.
The output vector from the RAM network contains a number of output scores, one for each possible class. As mentioned above a decision is normally made by classifying an example in to the class having the largest output score. This simple winner-takes-all (WTA) scheme assures that the true class of a training examples cannot lose to one of the other classes. One problem with the RAM net classification scheme is that it often behaves poorly when trained on a training set where the distribution of examples between the training classes are highly skewed. Accordingly there is a need for understanding the influence of the composition of the training material on the behaviour of the RAM classification system as well as a general understanding of the influence of specific parameters of the architecture on the performance. From such an understanding it could be possible to modify the classification scheme to improve its performance and competitiveness with other schemes. Such improvements of the RAM based classification systems is provided according to the present invention.
Recently Thomas Martini Jørgensen and Christian Linneberg (inventors of this invention) have provided a statistical framework that have made it possible to make a theoretical analysis that relates the expected output scores of the n-tuple net to the stochastic parameters of the example distributions, the number of available training examples, and the number of address lines n used for each LUT or n-tuple. From the obtained expressions, they have been able to study the behaviour of the architecture in different scenarios. Furthermore, they have based on the theoretical results come up with proposals for modifying the n-tuple classification scheme in order to make it operate as a close approximation to the maximum a posteriori or maximum likelihood estimator. The resulting modified decision criteria can for example deal with the so-called skewed class prior problem causing the n-tuple net to often behave poorly when trained on a training set where the distribution of examples between the training classes are highly skewed. Accordingly the proposed changes of the classification scheme provides an essential improvement of the architecture. The suggested changes in decision criteria are not only applicable to the original n-tuple architecture based on random memorisation. It also applies to extended n-tuple schemes, some of which use a more optimal selection of the address lines and some of which apply an extended weight scheme.
According to a first aspect of the present invention there is provided a method for training a computer classification system which can be defined by a network comprising a number of n-tuples or Look Up Tables (LUTs), with each n-tuple or LUT comprising a number of rows corresponding to at least a subset of possible classes and further comprising a number of columns being addressed by signals or elements of sampled training input data examples, each column being defined by a vector having cells with values, said method comprising determining the column vector cell values based on one or more training sets of input data examples for different classes so that at least part of the cells comprise or point to information based on the number of times the corresponding cell address is sampled from one or more sets of training input examples. The method further comprises determining one or more output score functions for evaluation of at least one output score value per class, and/or determining one or more decision rules to be used in combination with at least part of the obtained output score values to determine a winning class.
It is preferred that the output score values are evaluated or determined based on the information of at least part of the determined column vector cell values.
According to the present invention it is preferred that the output score functions and/or the decision rules are determined based on the information of at least part of the determined column vector cell values.
It is also preferred to determine the output score functions from a family of output score functions determined by a set of parameter values. Thus, the output score functions may be determined either from the set of parameter values, from the information of at least part of the determined column vector cell values or from both the set of parameter values and the information of at least part of the determined column vector cell values.
It should be understood that the training procedure of the present invention may be considered a two step training procedure. The first step may comprise determining the column vector cell values, while the second step may comprise determining the output score functions and/or the decision rules.
As already mentioned, the column vector cells are determined based on one or more training sets of input data examples of known classes, but the output score functions and/or the decision rules may be determined based on a validation set of input data examples of known classes. Here the validation set may be equal to or part of the training set(s), but the validation set may also be a set of examples not included in the training set(s).
According to the present invention the training and/or validation input data examples may preferably be presented to the network as input signal vectors.
It is preferred that determination of the output score functions is performed so as to allow different ways of using the contents of the column vector cells in calculating the output scores used to find the winning class amongst two or more classes. The way the contents of the column vector cells are used to obtain the score of one class might depend on which class(es) it is compared with.
It is also preferred that the decision rules used when comparing two or more classes in the output space are allowed to deviate from the decision rules corresponding to a WTA decision. Changing the decision rules for choosing two or more classes is equivalent to allowing individual transformation of the class output scores and keeping a WTA comparison. These corresponding transformations might depend on which class(es) a given class is compared with.
The determination of how the output score functions may be calculated from the column vector cell values, as well as the determination of how many output score functions to use and/or the determination of the decision rules to be applied on the output score values may comprise the initialisation of one or more sets of output score functions and/or decision rules.
Furthermore it is preferred to adjust at least part of the output score functions and/or the decision rules based on an information measure evaluating the performance on the validation example set. If the validation set equals the training set or part of the training set it is preferred to use a leave-one-out cross-validation evaluation or extensions of this concept.
In order to determine or adjust the output score functions and the decision rules according to the present invention, the column cell values should be determined. Here, it is preferred that at least part of the column cell values are determined as a function of the number of times the corresponding cell address is sampled from the set(s) of training input examples. Alternatively, the information of the column cells may be determined so that the maximum column cell value is 1, but at least part of the cells have an associated value being a function of the number of times the corresponding cell address is sampled from the training set(s) of input examples. Preferably, the column vector cell values are determined and stored in storing means before the determination or adjustment of the output score functions and/or the decision rules.
According to the present invention, a preferred way of determining the column vector cell values may comprise the training steps of
However, it should be understood that the present invention also covers embodiments where the information of the column cells is determined by alternative functions of the number of times the cell has been addressed by the input training set(s). Thus, the cell information does not need to comprise a count of all the times the cell has been addressed, but may for example comprise an indication of when the cell has been visited zero times, once, more than once, and/or twice and more than twice and so on.
In order to determine the output score functions and/or the decision rules, it is preferred to adjust these output score functions and/or decision rules, which adjustment process may comprise one or more iteration steps. The adjustment of the output score functions and/or the decision rules may comprise the steps of determining a global quality value based on at least part of the column vector cell values, determining if the global quality value fulfils a required quality criterion, and adjusting at least part of output score functions and/or part of the decision rules until the global quality criterion is fulfilled.
The adjustment process may also include determination of a local quality value for each sampled validation input example, with one or more adjustments being performed if the local quality value does not fulfil a specified or required local quality criterion for the selected input example. As an example the adjustment of the output score functions and/or the decision rules may comprise the steps of
Preferably, steps (b)–(d) of the above mentioned adjustment process may be carried out for all examples of the validation set(s).
The local and/or global quality value may be defined as functions of at least part of the column cells.
It should be understood that when adjusting the output score functions and/or decision rules by use of one or more quality values each with a corresponding quality criterion, it may be preferred to stop the adjustment iteration process if a quality criterion is not fulfilled after a given number of iterations.
It should also be understood that during the adjustment process the adjusted output score functions and/or decision rules are preferably stored after each adjustment, and when the adjustment process includes the determination of a global quality value, the step of determination of the global quality value may further be followed by separately storing the hereby obtained output score functions and/or decision rules or classification system configuration values if the determined global quality value is closer to fulfil the global quality criterion than the global quality value corresponding to previously separately stored output score functions and/or decision rules or configuration values.
A main reason for training a classification system according to an embodiment of the present invention is to obtain a high confidence in a subsequent classification process of an input example of an unknown class.
Thus, according to a further aspect of the present invention, there is also provided a method of classifying input data examples into at least one of a plurality of classes using a computer classification system configured according to any of the above described methods of the present invention, whereby column cell values for each n-tuple or LUT and output score functions and/or decision rules are determined using on one or more training or validation sets of input data examples, said method comprising
The present invention also provides training and classification systems according to the above described methods of training and classification.
Thus, according to the present invention there is provided a system for training a computer classification system which can be defined by a network comprising a stored number of n-tuples or Look Up Tables (LUTs), with each n-tuple or LUT comprising a number of rows corresponding to at least a subset of possible classes and further comprising a number of columns being addressed by signals or elements of sampled training input data examples, each column being defined by a vector having cells with values, said system comprising
Here, it is preferred that the means for determining the output score functions and/or decision rules is adapted to determine these functions and/or rules based on the information of at least part of the determined column vector cell values.
The means for determining the output score functions may be adapted to determine such functions from a family of output score functions determined by a set of parameter values. Thus, the means for determining the output score functions may be adapted to determine such functions either from the set of parameter values, from the information of at least part of the determined column vector cell values or from both the set of parameter values and the information of at least part of the determined column vector cell values.
According to the present invention the means for determining the output score functions and/or the decision rules may be adapted to determine such functions and/or rules based on a validation set of input data examples of known classes. Here the validation set may be equal to or part of the training set(s) used for determining the column cell values, but the validation set may also be a set of examples not included in the training set(s).
In order to determine the output score functions and decision rules according to a preferred embodiment of the present invention, the means for determining the output score functions and decision rules may comprise
As already discussed above the column cell values should be determined in order to determine the output score functions and decision rules. Here, it is preferred that the means for determining the column vector cell values is adapted to determine these values as a function of the number of times the corresponding cell address is sampled from the set(s) of training input examples. Alternatively, the means for determining the column vector cell values may be adapted to determine these cell values so that the maximum value is 1, but at least part of the cells have an associated value being a function of the number of times the corresponding cell address is sampled from the training set(s) of input examples.
According to an embodiment of the present invention it is preferred that when a training input data example belonging to a known class is applied to the classification network thereby addressing one or more column vectors, the means for determining the column vector cell values is adapted to increment the value or vote of the cells of the addressed column vector(s) corresponding to the row(s) of the known class, said value preferably being incremented by one.
For the adjustment process of the output score functions and decision rules it is preferred that the means for adjusting output score functions and/or decision rules is adapted to
As an example of a preferred embodiment according to the present invention, the means for adjusting output score functions and decision rules may be adapted to
The means for adjusting the output score functions and decision rules may further be adapted to stop the iteration process if the global quality criterion is not fulfilled after a given number of iterations. In a preferred embodiment, the means for storing n-tuples or LUTs comprises means for storing adjusted output score functions and decision rules and separate means for storing best so far output score functions and decision rules or best so far classification system configuration values. Here, the means for adjusting the output score functions and decision rules may further be adapted to replace previously separately stored best so far output score functions and decision rules with obtained adjusted output score functions and decision rules if the determined global quality value is closer to fulfil the global quality criterion than the global quality value corresponding to previously separately stored best so far output score functions and decision rules. Thus, even if the system should not be able to fulfil the global quality criterion within a given number of iterations, the system may always comprise the “best so far” system configuration.
According to a further aspect of the present invention there is also provided a system for classifying input data examples of unknown classes into at least one of a plurality of classes, said system comprising:
It should be understood that it is preferred that the cell values of the column vectors and the output score functions and/or decision rules of the classification system according to the present invention are determined by use of a training system according to any of the above described systems. Accordingly, the column vector cell values and the output score functions and/or decision rules may be determined during a training process according to any of the above described methods.
For a better understanding of the present invention and in order to show how the same may be carried into effect, reference will now be made by way of example to the accompanying drawings in which:
In the following a more detailed description of the architecture and concept of a classification system according to the present invention will be given including an example of a training process of the column cells of the architecture and an example of a classification process. Furthermore, different examples of learning processes for the output score functions and the decision rules according to embodiments of the present invention are described.
Notation
The notation used in the following description and examples is as follows:
Description of Architecture and Concept
In the following references are made to
A RAM-net or LUT-net consists of a number of Look Up Tables (LUTs) (1.3). Let the number of LUTs be denoted NLUT. An example of an input data vector {overscore (y)} to be classified may be presented to an input module (1.1) of the LUT network. Each LUT may sample a part of the input data, where different numbers of input signals may be sampled for different LUTs (1.2) (in principle it is also possible to have one LUT sampling the whole input space). The outputs of the LUTs may be fed (1.4) to an output module (1.5) of the RAM classification network.
In
The viC-values of the activated LUT columns (2.6) may be fed (1.4) to the output module (1.5), where one or more output scores may be calculated for each class and where these output scores in combinations with a number of decision rules determine the winning class.
Let {overscore (x)}∈X denote an input data example used for training and let {overscore (y)} denote an input data example not belonging to the training set. Let C({overscore (x)}) denote the class to which {overscore (x)} belongs. The class assignment given to the example {overscore (y)} is then obtained by calculating one or more output scores for each class. The output scores obtained for class C is calculated as functions of the viC numbers addressed by the example {overscore (y)} but will in general also depend on a number of parameters {overscore (β)}. Let the mth output score of class C be denoted SC,m(viC,{overscore (β)}) A classification is obtained by combining the obtained output scores from all classes with a number of decision rules. The effect of the decision rules is to define regions in the output score space that must be addressed by the output score values to obtain a given winner class. The set of decision rules is denoted Ξ and corresponds to a set of decision borders.
Initial Training of the Architecture
The flow chart of
Before the trained network can be used for classification the output score functions and the decision rules must be initialised.
Classification of an Unknown Input Example
When the RAM network of the present invention has been trained to thereby determine values for the column cells whereby the LUTs may be defined, the network may be used for classifying an unknown input data example.
In a preferred example according to the present invention, the classification is performed by using the decision rules Ξ and the output scores obtained from the output score functions. Let the decision function invoking Ξ and the output scores be denoted D(•). The winning class can then be written as:
Winner Class=D(Ξ, S1,1, S1,2, . . . S1,j, . . . S2,1, . . . S2,k, . . . S1,m)
Adjustment of Output Score Function Parameter {overscore (β)} and Adjustment of Decision Rules Ξ
Usually the initially determined values of {overscore (β)} and the initial set of rules Ξ will not present the optimal choices. Thus, according to a preferred embodiment of the present invention, an optimisation or adjustment of the {overscore (β)} values and the Ξ rules should be performed.
In order to select or adjust the parameters {overscore (β)} and the rules Ξ to improve the performance of the classification system, it is suggested according to an embodiment of the invention to define proper quality functions for measuring the performance of the {overscore (β)}-values and the Ξ-rules. Thus, a local quality function QL({overscore (v)},{overscore (x)}, X,{overscore (β)},Ξ) may be defined, where {overscore (v)} denotes a vector containing all viC elements of the LUT network. The local quality function may give a confidence measure of the output classification of a specific example {overscore (x)}. If the quality value does not satisfy a given criterion the {overscore (β)} values and the Ξ rules are adjusted to make the quality value satisfy or closer to satisfying the criterion (if possible).
Furthermore a global quality function: QG({overscore (v)}, X, {overscore (β)}, Ξ) may be defined. The global quality function may measure the performance of the input training set as a whole.
20 This example illustrates an optimisation procedure for adjusting the decision rules Ξ.
We consider Nc training classes. The class label c is an integer running from 1 to Nc.
For each class c we define a single output score function:
where δi,j is Kroneckers delta (δi,j=1 if i=j and 0 otherwise), and
The expression for the output score function illustrates a possible family of functions determined by a parameter vector {overscore (β)}. This example, however, will only illustrate a procedure for adjusting the decision rules Ξ, and not {overscore (β)}. For simplicity of notation we therefore initialise all values in {overscore (β)} to one. We then have:
With this choice of {overscore (β)} the possible output values for Sc are the integers from 0 to NLUT (both inclusive).
The leave-one-out cross-validation score or vote-count on a given class c is:
where CT({overscore (x)}) denotes the true class of example {overscore (x)}.
For all possible inter-class combinations (c1,c2), (c1∈{1,2 . . . Nc},c2∈{1,2, . . . Nc})^(c1≠c2) we wish to determine a suitable decision border in the score space spanned by the two classes. The matrix Bc
Each matrix element contains one of the following three values: c1,c2 and kAMB, where kAMB is a constant different from c1 and c2. Here we use kAMB=0. The two output score values S1 and S2 obtained for class c1 and class c2, respectively, are used to address the element bS
The decision rules are initialised to correspond to a WTA decision. This corresponds to having a decision border along the diagonal in the matrix Bc
A strategy for adjusting the initialised decision border according to an information measure that uses the va
Create the cost matrix Mc
αc
A minimal-cost path from m0,0 to mN
// Loop through all entries in the cost matrix in reverse order:
The dynamic programming approach can be extended with regularisation terms, which constraint the shape of the border.
An alternative method for determining the decision border could be to fit a B-spline with two control points in such a way that the associated cost is minimised.
Using the decision borders determined from the strategy outlined above an example can now be classified in the following manner:
A leave-one-out cross-validation test using the decision borders determined from the strategy outlined above is obtained in the following manner:
With reference to
In the above case one would as alternative stop criterion use a criterion that only allows two loops through the adjustment scheme.
This example illustrates an optimisation procedure for adjusting {overscore (β)}.
For each class we again define a single output score
With these score values the example is now classified as belonging to the class with the label found from
In this example we use {overscore (β)}=(k1,k2, . . . ,kN
For practical use, the kMAX-value will depend upon the skewness of the class priors and the number of address-lines used in the RAM net system.
This example also illustrates an optimisation procedure for adjusting {overscore (β)} but with the use of a local quality function QL.
For each class we now define as many output scores as there are competing classes, i.e. Nc−1 output scores:
With these score values a decision is made in the following manner
In this example we use
We also initialise the Ξ rules to describe a WTA decision when comparing the output scores from the different classes.
The foregoing description of preferred exemplary embodiments of the invention has been presented for the purpose of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and obviously many modifications and variations are possible in light of the present invention to those skilled in the art. All such modifications which retain the basic underlying principles disclosed and claimed herein are within the scope of this invention.
Number | Date | Country | Kind |
---|---|---|---|
1998 00883 | Jun 1998 | DK | national |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/DK99/00340 | 6/21/1999 | WO | 00 | 3/15/2001 |
Publishing Document | Publishing Date | Country | Kind |
---|---|---|---|
WO99/67694 | 12/29/1999 | WO | A |
Number | Name | Date | Kind |
---|---|---|---|
4975975 | Filipski | Dec 1990 | A |
5359699 | Tong et al. | Oct 1994 | A |
5555344 | Zūnkler | Sep 1996 | A |
5790754 | Mozer et al. | Aug 1998 | A |
5893058 | Kosaka | Apr 1999 | A |
6021387 | Mozer et al. | Feb 2000 | A |
6169981 | Werbos | Jan 2001 | B1 |
6238342 | Feleppa et al. | May 2001 | B1 |
6571227 | Agrafiotis et al. | May 2003 | B1 |
Number | Date | Country |
---|---|---|
0935212 | Aug 1999 | EP |
9200572 | Jan 1992 | WO |