Patent Grant
7827179
The present invention relates to a data clustering system, a data clustering method, and a data clustering program.
Hereinafter, in a method and apparatus that classify a plurality of data according to some criterion, a process of grouping substantially the same data is referred to as “duplicate record verification”. The “substantially the same” indicates, e.g., a relationship between “NEC” and “enu-i-shi” or “1-chome 2-banchi” and “1-2”, that is, a relationship between two data which do not completely coincide with each other in terms of digital data due to Japanese orthographic variation but can be determined to be the same by a human. In the case where duplication of records in a single database or between a plurality of databases is checked, if all combinations are checked, the number of combinations is explosively increased as the number of records is increased, with the result that enormous processing time is required in the duplication check processing.
In a conventional approach, the duplicate record verification is performed by combining a method (rough narrowing-down) with a lower accuracy but with a lighter processing load and a method (detailed narrowing-down) with a heavy processing load but with a high accuracy.
For example, in the case where duplication check is carried out for 100,000 records, the rough narrowing-down method is used first to combine the 100,000 records into a large number of blocks each including several thousand to several ten thousand records and then the detailed narrowing-down method is applied to each block.
As a technique for the rough narrowing-down method, there is known a Sorted Neighborhood Method (disclosed in Non-Patent Document 1) and the like.
As a technique for the detailed narrowing-down method, there is known a method using an edit distance (disclosed in Non-Patent Document 2) and the like. The edit distance is used for measuring similarity between character strings. As a similar technique, there are known a phonetic distance, input (typewriter) distance, and the like.
In the detailed narrowing-down method, similarity or distance between records is calculated followed by setting of a supplied threshold value, whereby the records are combined into groups. The grouped records may be determined as duplication record candidates by a human, or may directly be determined to be duplicated records. Thus, by combining the rough narrowing-down method and detailed narrowing-down method, efficiency of duplicate record verification processing has been improved.
Non-Patent Document 1: M. A. Hernandez and other one, Real-world Data is Dirty: Data Cleansing and the Merge/Purge Problem, Journal of Data Mining and Knowledge Discovery, Vol. 1, 1998
Non-Patent Document 2: M. Hernandez and other one, The Merge/Purge Problem for Large Databases, Proceedings of the 1995 ACM SIGMOD International Conference of DATA (SIGMOD 1995), 1995, pages 127 to 138
The rough narrowing-down method tends to be superior in processing speed but inferior in accuracy, while the detailed narrowing-down method tends to be superior in accuracy but inferior in processing speed. Therefore, when the number of records to be verified is increased, it is necessary to narrow down more records in the rough narrowing-down method in order to order to prevent the processing time from increasing, with the result that accuracy is degraded. Thus, a large number of records cannot be narrowed down in the rough narrowing-down method in order to maintain accuracy, thus incurring an increase in the processing time.
That is, a first problem is that as the number of records to be verified is increased, it becomes difficult to prevent the processing speed from decreasing while maintaining accuracy.
The second problem is that it is difficult to adequately set a threshold value for narrowing down the records in the rough narrowing-down method, which involves trial and error process. That is, if the threshold value for the narrowing-down in the rough narrowing-down method is too small, it is necessary to check a large number of combinations in the subsequent detailed narrowing-down method, which reduces the effect of the rough narrowing-down method. On the other hand, if the threshold value for the narrowing-down is too large, records that are supposed to be determined as the same group by tighter checking are eliminated from duplication record candidates in the stage of the rough narrowing-down method.
The present invention is made in view of the above problems, an object thereof is to prevent the processing speed from decreasing while maintaining accuracy even when the number of records to be verified is increased.
Another object of the present invention is to eliminate the need for the trial and error approach to the determination of a threshold value in the rough narrowing-down method.
A first invention to solve the above problems is a data clustering system characterized by comprising: a simple similarity calculation section that receives a block containing at least one data and a block generation condition and divides the block based on the block generation condition; a block division section that repeatedly performs block division processing that allows the simple similarity calculation section to divide the block by supplying the block and block generation condition thereto to create a hierarchical relationship between the blocks obtained by the block division processing; a block storage section that stores blocks and the hierarchical relationship between the blocks; a similarity calculation section that calculates the similarity between combinations of data contained in a block; and a block integration section that allows the similarity calculation section to calculate the similarity between combination of data contained in a block and executes processing for combining groups each of which contains at least one data based on the calculation result of the similarity calculation section along the hierarchical relationship between blocks stored in the block storage section.
A second invention to solve the above problems is the data clustering system according to the first invention, characterized in that when performing additional block division processing, the block division section uses, as a new block generation condition, a block generation condition tighter than the original block generation condition.
A third invention to solve the above problems is the data clustering system according to the first or second invention, characterized in that the simple similarity calculation section uses a method with a lower accuracy but with a lower calculation load to calculate the simple similarity between data, and the similarity calculation section uses a method with a higher calculation load but with a higher accuracy to calculate the similarity between data.
A fourth invention to solve the above problems is the data clustering system according to the first, second, or third invention, characterized in that the block integration section uses representative data of groups to allow the similarity to be calculated and determines whether the groups can be combined or not.
A fifth invention to solve the above problems is the data clustering system according to the fourth invention, characterized in that the similarity calculation section determines “integration possible” “integration impossible” or “continuous check” as the result of the similarity calculation based on the similarity between the data combination and, when the result is “continuous check” and the data that have been subjected to the similarity calculation are represented data, calculates the similarity between combination of different data contained in the groups to which the representative data belong and determines “integration possible” or “integration impossible”.
A sixth invention to solve the above problems is the data clustering system according to the first, second, third, fourth, or fifth invention, characterized in that when moving to a new block along the hierarchical relationship between blocks, the block integration section adds a check mark to a group containing only one data to exclude representative data of the group added with the check mark from the subsequent similarity calculation between the combination of data performed by the similarity calculation section.
A seventh invention to solve the above problems is the data clustering system according to the first, second, third, fourth, fifth, or sixth invention, characterized in that the data is a record on a database system.
An eighth invention to solve the above problems is a data clustering method, comprising the steps of: repeatedly performing block division processing that divides a block containing at least one data based on a block generation condition to create a hierarchical relationship between the blocks obtained by the block division processing; storing blocks and the hierarchical relationship between the blocks; and calculating the similarity between combination of data contained in a block and executing processing for combining groups each of which contains at least one data based on the calculation result of the similarity calculation along the hierarchical relationship between the stored blocks.
A ninth invention to solve the above problems is the data clustering method according to the eighth invention, characterized in that when performing additional block division processing, the step of creating the hierarchical relationship between blocks uses, as a new block generation condition, a block generation condition tighter than the original block generation condition.
A tenth invention to solve the above problems is the data clustering method according to the eighth or ninth invention, characterized in that the step of creating the hierarchical relationship between blocks divides the block based on simple similarity obtained using a method with a lower accuracy but with a lower calculation load and block generation condition, and the step of executing processing for combining groups uses a method with a higher calculation load but with a higher accuracy to calculate the similarity.
An eleventh invention to solve the above problems is the data clustering method according to the eighth, ninth, or tenth invention, characterized in that the step of executing processing for combining groups uses representative data of groups to calculate the similarity and determines whether the groups can be combined or not.
A twelfth invention to solve the above problems is the data clustering method according to the eleventh invention, characterized in that the step of executing processing for combining groups determines “integration possible” “integration impossible” or “continuous check” as the result of the similarity calculation based on the similarity between the data combination and, when the result is “continuous check” and the data that have been subjected to the similarity calculation are represented data, calculates the similarity between combination of different data contained in the groups to which the representative data belong to and determines “integration possible” or “integration impossible”.
A thirteenth invention to solve the above problems is the data clustering method according to the eighth, ninth, tenth, eleventh, or twelfth invention, characterized in that when moving to a new block along the hierarchical relationship between blocks, the step of executing processing for combining groups adds a check mark to a group containing only one data to exclude representative data of the group added with the check mark from the subsequent similarity calculation between the combination of data performed by the similarity calculation section.
A fourteenth invention to solve the above problems is the data clustering method according to the eighth, ninth, tenth, eleventh, twelfth, or thirteenth invention, characterized in that the data is a record on a database system.
A fifteenth invention to solve the above problems is a data clustering program characterized by allowing a computer to execute: simple similarity calculation processing that divides a block containing at least one data based on a block generation condition; block division processing that repeatedly performs block division processing that allows the simple similarity calculation processing to divide the block by supplying the block and block generation condition thereto to create a hierarchical relationship between the blocks obtained by the block division processing; block storage processing that stores blocks and the hierarchical relationship between the blocks; similarity calculation processing that calculates the similarity between combinations of data contained in a block; and block integration processing that allows the similarity calculation section to calculate the similarity between combination of data contained in a block and executes processing for combining groups each of which contains at least one data based on the calculation result of the similarity calculation section along the hierarchical relationship between blocks stored in the block storage section.
A sixteenth invention to solve the above problems is the data clustering program according to the fifteenth invention, characterized in that when performing additional block division processing, the block division processing uses, as a new block generation condition, a block generation condition tighter than the original block generation condition.
A seventeenth invention to solve the above problems is the data clustering program according to the fifteenth or sixteenth invention, characterized in that the simple similarity calculation processing uses a method with a lower accuracy but with a lower calculation load to calculate the simple similarity between data, and the similarity calculation processing uses a method with a higher calculation load but with a higher accuracy to calculate the similarity between data.
An eighteenth invention to solve the above problems is the data clustering program according to the fifteenth, sixteenth, or seventeenth invention, characterized in that the block integration processing uses representative data of groups to allow the similarity to be calculated and determines whether the groups can be combined or not.
A nineteenth invention to solve the above problems is the data clustering program according to the seventeenth invention, characterized in that the similarity calculation processing determines “integration possible” “integration impossible” or “continuous check” as the result of the similarity calculation based on the similarity between the data combination and, when the result is “continuous check” and the data that have been subjected to the similarity calculation are represented data, calculates the similarity between combination of different data contained in the groups to which the representative data belong and determines “integration possible” or “integration impossible”.
A twentieth invention to solve the above problems is the data clustering program according to the fifteenth, sixteenth, seventeenth, eighteenth, or nineteenth invention, characterized in that when moving to a new block along the hierarchical relationship between blocks, the block integration processing adds a check mark to a group containing only one data to exclude representative data of the group added with the check mark from the subsequent similarity calculation between the combination of data performed in the similarity calculation processing.
A twenty-first invention to solve the above problems is the data clustering program according to the fifteenth, sixteenth, seventeenth, eighteenth, nineteenth, or twentieth invention, characterized in that the data is a record on a database system.
By adopting the above configuration, data is divided into small-sized blocks while tightening a division condition in a stepwise manner and, afterward, the blocks are integrated while determining whether or not to allow the integration. Thus, it is possible to check starting from data combinations that are likely to be integrated, thereby reducing the number of combinations while maintaining accuracy. As a result, the first and second objects of the present invention can be achieved.
A first advantage is that it is possible to prevent the processing speed from decreasing while maintaining accuracy. Data is divided into small-sized blocks while tightening a division condition in a stepwise manner and, the blocks are integrated in the reverse order in which the data is divided into the blocks while determining whether or not to allow the integration. Thus, it is possible to check starting from data combinations that are likely to be integrated, thereby reducing the number of combinations while maintaining accuracy.
A second advantage is that it is possible to eliminate the need for the trial and error approach to the determination of parameters for the clustering. This is, data is divided into small-sized blocks while tightening a division condition in a stepwise manner, and the blocks are integrated in the reverse order in which the data is divided into the blocks while determining whether or not to allow the integration, which eliminates the need of previous determination of the level of a block generation condition in the rough narrowing-down method, which has been required in a conventional approach.
A first embodiment of the present invention will be described in detail by referring to the accompanying drawings.
Hereinafter, “database” is sometimes referred to as “DB”.
A target database 1 is so-called a database and stores a plurality of data. The database 1 may be a single database or a database obtained by virtually merging a plurality of databases.
The DB access section 2 performs data input/output from and to the target database 1. When reading data from the target database 1, the DB access section 2 combines the read data into one block supplies it to the block division section 3. The one block contains one or more data. When receiving information from the block integration section 5 to be described later, the DB access section 2 writes the received information in the target database 1.
Here, a description will be given of details of the block. In the present invention, terms “block” and “group” are used to represent one containing one or more data. The group is created through processing performed by the block integration section 5 and similarity calculation section 6 and contains one or more data. One of the data contained in the group is set as representative data. The block is constituted by one or more data, one or more groups, or combination thereof.
The block division section 3 divides a block received from the DB access section 2 into sufficiently small blocks. The block division processing is made by supplying the block and block generation condition to the simple similarity calculation section 4. Upon receiving the block and block generation condition, the simple similarity calculation section 4 generates blocks and returns them to the block division section 3 (details will be described later). The block division section 3 checks the returned blocks. If a supplied block is not sufficiently small, the block division section 3 supplies this block and a new block generation condition which is one step tighter than the original one to the simple similarity calculation section 4 for another block division processing.
The number of data belonging to one block is used as a criterion for determining whether the block is sufficiently small or not. For example, the block division section 3 does not determine so if the number of data belonging to a block is 20 or more. This determination may be made based on the ratio of the number of data belonging to a block relative to the number of all data. For example, if the number of data belonging to a block is 1% or more, the block is further divided. The determination is not limited to the above. The determination criterion may be input by a user, may be read from a setting file, or may previously be determined.
The block generation condition is specified, on a case-by-case basis, according to a method for calculating simple similarity. For example, a method for calculating the simple similarity as a numerical value may be used. In this case, a certain threshold value is set as the block generation condition, and a data combination having a similarity exceeding the threshold value is combined into the same block. In order to make the block generation condition one step tighter, a one step larger threshold value is used.
The simple similarity calculation section 4 receives the block and block generation condition from the block division section 3, calculates the simple similarity between data combinations contained in the block, generates blocks based on the calculation result and block generation condition, and returns the generated blocks to the block division section 3. At this time, the simple similarity calculation section 4 uses the rough narrowing-down method, i.e., a method with a higher processing speed but with a lower accuracy to calculate the simple similarity. A known method can be used for the calculation of the simple similarity.
The block division section 3 stores a hierarchical relationship and blocks generated in the course of the division processing in the block storage section 8.
Before explaining the details of the block integration section 5 and similarity calculation section 6, the “group” used in processing performed by the block integration section 5 and similarity calculation section 6 will here be described. As described above, a group contains one or more data, and one of the data contained in the group is set as representative data.
Various methods can be used to select the representative data. For example, data contained in a group are sorted by some sort of key, and data positioned at, e.g., the center or head is selected as the representative data. In the case where only one data is contained in a group, the one data is selected as the representative data. The selection of the representative data need not be made at the group formation time, but may be made at the time when the representative data is required for calculation.
The block integration section 5 integrates blocks and groups in the direction from a hierarchically deeper position to shallower position based on the hierarchical relationship between the blocks stored in the block storage section 8.
The block integration section 5 checks whether groups contained in a block can be integrated or not. To this end, the block integration section 5 supplies combinations of representative data belonging to the groups to the similarity calculation section 6. The similarity calculation section 6 calculates the similarity between the representative data. When determining that target groups can be integrated with each other, the similarity calculation section 6 returns “integration possible” to the block integration section 5; while when determining that target groups cannot be integrated, the similarity calculation section 6 returns “integration impossible”. Upon receiving a result from the similarity calculation section 6, the block integration section 5 integrates the groups corresponding to the combination of representative data that has been determined to be “integration possible”.
An example of the block integration processing performed by the block integration section 5 will be described with reference to
The similarity calculation section 6 receives the data combinations from the block integration section 5, calculates the similarity between them, and determines whether integration of each data combination can be made or not. As described above, when determining that target groups can be integrated, the similarity calculation section 6 returns “integration possible” to the block integration section 5; while when determining that target groups cannot be integrated, the similarity calculation section 6 returns “integration impossible”. At this time, the similarity calculation section 6 uses the detailed narrowing-down method, i.e., a method with a higher accuracy but with a lower processing speed to calculate the similarity. A known method, e.g., one using the edit distance, can be used for the calculation of the similarity. Operation of the similarity calculation section 6 will be described later.
At the time point when there exists no block to be integrated, the block integration section 5 ends the processing. The similar data integrated into a group at this time is regarded as a cluster.
The block storage section 8 stores the blocks supplied from the block division section 3 and hierarchical relationship between them. The block integration section 5 refers to the stored hierarchical relationship to perform the block integration processing and corrects the hierarchical relationship, on a case-by-case basis, at the time of the block integration.
Operation of the first embodiment will next be described.
Next, with reference to
The block division section 3 checks the blocks returned from the simple similarity calculation section 4 and confirms whether the original block has been divided (step K3). When the block has been divided, the block division section confirms the size of each block (step K4). In the case where the size of one returned block is too large, the block division section 3 makes the block generation condition one step tighter (step K5) and supplies once again the one block and a new block generation condition to the simple similarity calculation section 4 (step K1). The block division section 3 repeats the above operation until the size of the block returned from the simple similarity calculation section 4 becomes sufficiently small. In the case where the size of one block is sufficiently small, the block division section 3 stores, as the hierarchical relationship between blocks, information indicating how the original block has been divided (step K7) and checks the size of another block (step K9). When there exists no block to be divided, the block division section 3 ends this processing (step K10).
Operation of the simple similarity calculation section 4 will next be described. The simple similarity calculation section 4 generates blocks from data belonging to the supplied block according to the block generation condition. For example, the simple similarity calculation section 4 combines a data group having a simple similarity with the data specified in the block generation condition, which exceeds a threshold value into one block. Alternatively, a data group that satisfies the block generation condition may be combined into one block. Further alternatively, data satisfying the block generation condition may be searched for from data belonging to the supplied block. If a new block needs to be generated at this time, the simple similarity calculation section 4 generates new one. Note that one data may belong to a plurality of blocks.
As described above, through the processing performed by the block division section 3 and simple similarity calculation section 4, a hierarchical relationship between blocks as shown in
Upon receiving a set of the data combinations, the similarity calculation section 6 calculates the similarity between each data combination and returns “integration possible” or “integration impossible” to the block integration section 5.
The block integration section 5 checks the result returned from the similarity calculation section 6 (step T2) and performs group formation processing based on the result. That is, when the result is “integration possible”, the block integration section 5 integrates the relevant groups into one group (step T3).
After completion of the group formation processing for the target block, the block integration section 5 integrates blocks (step T4), and shifts to the processing for the next block (step T7). The block integration section 5 integrates blocks and groups in the direction from a hierarchically deeper position to shallower position based on the hierarchical relationship between blocks (as shown in
After the completion of the above processing, the block integration section 5 supplies the group formation result to the DB access section 2. The DB access section 2 registers the received result in the database 1.
In the case where there is any data combination determined to be “continuous check”, the similarity calculation section 6 checks whether the elements of the data combination are representative data or not (step J3). When determining that the elements of the data combination are representative data, the similarity calculation section 6 carries out a detailed check of whether groups containing the representative data can be integrated or not (step J4). That is, the similarity calculation section 6 calculates the similarity between all combinations of data contained in the groups. Then, the similarity calculation section 6 classifies the calculation result once again (step J7). In this manner, the similarity calculation section 6 calculates a results indicative of whether target groups including the representative data are “integration possible” or “integration impossible” and returns the result to the block integration section 5 (steps J5 and J6).
The block storage section 8 in the first embodiment is realized by, e.g., a semiconductor memory. The DB access section 2, block division section 3, simple similarity calculation section 4, block integration section 5, and similarity calculation section 6 may be realized by a hardware having the respective functions, or by a computer operating according to, e.g., a data clustering program. In this case, the data clustering program is provided by means of a computer readable medium such as a magnetic disk or semiconductor memory and read by the computer at its start-up time, allowing the computer to operate according to the data clustering program to thereby function as the DB access section 2, block division section 3, simple similarity calculation section 4, block integration section 5, and similarity calculation section 6.
As described above, according to the first embodiment, the block division section 3 hierarchically divides each block while tightening the block generation condition in a stepwise manner, and the block integration section 5 integrates blocks and groups in the direction from a hierarchically deeper position to shallower position while determining whether integration of each group set can be made or not. Only when a detailed check is required, the similarity between combinations of data contained in target groups is calculated. Thus, it is possible to perform the check starting from data combinations that are likely to be integrated and to omit the calculation of combinations for which a detailed check is not required. Therefore, it is possible to reduce the number of combinations while maintaining clustering accuracy.
When the rough narrowing-down method and detailed narrowing-down method are used in combination in a conventional approach, it is difficult to determine a threshold value for narrowing down the records, which involves trial and error process. On the other hand, in the present invention, each block is divided while tightening the block generation condition in a stepwise manner, and, based on the obtained hierarchical relationship between blocks, blocks and groups are integrated while determining whether integration of each group set can be made or not. This eliminates the need for the trial and error approach to the determination of a threshold value for narrowing down the records in the rough narrowing-down method.
Further, when performing detailed check for combinations determined to be “continuous check”, the block integration section 5 need not check combinations of all data belonging to two target groups but may skip further check if there is any combination having a similarity exceeding a set similarity threshold value. Further, since a high similarity value can be expected within one group, a method that mutually checks combinations of the representative data in one group and all data contained in other group can be adopted. This further reduces the number of combinations whose similarity is to be calculated and prevents the processing speed from decreasing.
In the second embodiment, “check mark” can be added to a group. The “check mark” is not added when a new group is generated but added in the course of the processing performed by the block integration section 500.
Operation of the block integration section 500 will be described with reference to
According to the second embodiment of the present invention, the block integration section 500 adds “check mark” to the group containing only one data in the course of the block integration processing and skips the similarity calculation between combinations of the data in the group added with “check mark” and representative data of another group. As the hierarchical position becomes shallower, the block division condition set in the block division section 3 becomes looser, so that the similarity between data to be integrated next and data contained in the block obtained after one block integration processing is low. Therefore, even if the similarity calculation with the data to be integrated next is omitted in the course of block integration processing, clustering accuracy is not degraded, but, conversely, it is possible to significantly reduce the number of combinations whose similarity is to be checked and thereby to reduce the processing time.
A group to which the “check mark” is to be added need not be a group which has not been integrated by the integration processing performed across one or more hierarchies, but may be a group which has not been integrated by the integration processing performed across two or more, or, three or more hierarchies. Such a condition need not previously be determined but may be input by a user or may be read from a setting file.
A first example of the present invention will be described below with reference to the accompanying drawings. The following example corresponds to the first embodiment of the present invention.
In the present example, data is stored in a database shown in
The simple similarity calculation section 4 in the present example uses NAME field and ADDRESS field to calculate the simple similarity. However, the database and field to be used for the simple similarity calculation may be determined and input by a user, or may be read from a setting file. Concretely, all fields may be used for the calculation or only predetermined fields such as NAME, ADDRESS, and PHONE NUMBER may be used.
Further, as described in the first embodiment, a known method may be used for the simple similarity calculation. In the present example, a method that combines records whose first some characters coincide with one another into a block. With this method, there generated a block containing records whose first one character of the NAME field coincide with one another, a block containing records whose first one character of the ADDRESS field coincide with one another, and the like. Note that one record may be contained in a plurality of different blocks.
The DB access section 2 reads a database as shown in
The block generation condition in the present example is, e.g., “coincidence of first one character of NAME field” or “coincidence of first one character of ADDRESS field”. The block generation condition may differ depending on the simple similarity calculation method.
As shown in
The block division section 3 checks whether the size of each of the blocks returned from the simple similarity calculation section 4 is sufficiently small. It is assumed, in the present example, that a block containing 100 or more records should be further divided. That is, in the present example, the block division section 3 checks the number of records contained in each block returned from the simple similarity calculation section 4 and supplies once again a block containing 100 or more records, if exists, to the simple similarity calculation section 4 together with a block generation condition that has been tightened by one step. The simple similarity calculation section 4 returns blocks obtained by dividing the supplied block to the block division section 3. The block generation condition that has been tightened by one step is, e.g., “coincidence of first two characters of the current filed” or “coincidence of first one character of another field”, and the like. How the condition is made tighter may be input by a user, may be read from a setting file, or may previously be determined.
In the case where the block cannot be divided any more even if the block division condition is made one step tighter, the condition is further tightened. In the case where the block cannot still be divided, the simple similarity calculation section 4 notifies the block division section 3 of information indicating that the supplied block cannot be divided no more to prevent the block division section 3 from trying to divide the current block. A case where the block cannot be divided no more even if the block division condition is made tighter is, e.g., “complete coincidence of values of the current field”, “complete coincidence of values of all fields”, and the like.
It is assumed, for example, that a block “one character of ADDRESS field is [O] or [Dai]” has 111 records. The block division section 3 supplies block generation conditions “block containing records whose first one character of ADDRESS field is [O] or [Dai]” and “coincidence of first two characters of ADDRESS field” to the simple similarity calculation section 4, and the simple similarity calculation section 4 returns blocks obtained by dividing the supplied block to the block division section 3. If the block cannot be divided, (that is, all the records in the block belongs to the same block) under the block generation condition “coincidence of first two characters of ADDRESS field”, the block generation condition is made tighter in a stepwise manner (“coincidence of first three characters of ADDRESS field”, “coincidence of first four characters of ADDRESS field”, . . . ). It is assumed, for example, that in the case where the block is divided under a block generation condition “coincidence of first two characters of ADDRESS field”, a block containing 110 records whose first two characters of ADDRESS field is [O-saka] (the number of phrases separated by hyphens correspond to the number of characters) and block containing 1 record whose first two characters of ADDRESS field is [Dai-tou] are obtained. Since the block containing 110 records whose first two characters of ADDRESS field is [O-saka] satisfies the block generation condition specifying that a block containing 100 or more records should be further divided, it is further divided. It is assumed, for example, that the block is divided under the condition “coincidence of first three characters of ADDRESS field”, a block containing 101 records whose first three characters of ADDRESS field is [O-saka-hu] and block containing 9 records whose first three characters of ADDRESS field is [O-saka-shi] are obtained. Since the block containing 101 records whose first three characters of ADDRESS field is [O-saka-hu] satisfies the block generation condition specifying that a block containing 100 or more records should be further divided, it is further divided. For example, in the case where the block cannot be divided under a block generation condition “coincidence of first four characters of ADDRESS field”, the block is then divided under a block generation condition “coincidence of first five characters of ADDRESS field”. It is assumed that the block containing 101 records whose first three characters of ADDRESS field is [O-saka-hu] is divided into a block containing 90 records whose first four characters of ADDRESS field is [O-saka-hu-o] (e.g., data representing “O-saka-hu-o-saka-shi”) and a block containing 11 records whose first characters of ADDRESS field is [O-saka-hu-mori-guchi-shi] (e.g., data representing “O-saka-hu-mori-guchi-shi”). In this manner, the block containing 111 records whose first one character of ADDRESS field is [O] or [Dai] is divided into the block containing 90 records whose first four characters of ADDRESS field is [O-saka-hu-o], block containing 11 records whose first characters of ADDRESS field is [O-saka-hu-mori], block containing 9 records whose first three characters of ADDRESS field is [O-saka-shi], and block containing 1 record whose first two characters of ADDRESS field is [Dai-tou], as shown in
The block division section 3 stores, as the hierarchical relationship between blocks, information indicating how the original block has been divided in the block storage section 8. For example, the information has a tree structure as shown in
After completion of the block division processing performed by the block division section 3, the block integration section starts its processing. The block integration section 5 supplies combinations of records belonging to each of the blocks stored in the block storage section 8 to the similarity calculation section 6. The similarity calculation section 6 calculates the similarity between the received records. If the calculated similarity exceeds a threshold value, the similarity calculation section 6 notifies the block integration section 5 that the received records belong to the same group.
Various methods including one using the edit distance are known as a calculation method of the similarity. In this example, any of the methods is used to calculate the similarity according to the procedure as described above. The threshold value may be input by a user, may be read from a setting file, or may previously be determined. Every time the similarity of one combination has been calculated, the similarity value or information indicating that the similarity thereof does not exceed the threshold value is recorded so as not to perform the same calculation afterward.
It is assumed that if the similarity between a record whose ID is 001 and record whose ID is 002 exceeds the threshold value and similarity between the record whose ID is 002 and record whose ID is 003 exceeds the threshold value, the records 001, 002, and 003 are determined to belong to the same group even if the similarity between the record 001 and record 003 does not exceed the threshold value.
For simplification, the following description will focus on a group contained in the block “first four characters of ADDRESS field is [O-saka-hu-o]” in the tree structure shown in
Here, a description is given of the processing in the block “first four characters of ADDRESS field is [O-saka-hu-o]”. It is assumed that records 1 to 70 can be combined into one group. Similarly, in the block “first four characters of ADDRESS field is [O-saka-hu-mori]”, records 91 to 95 can be combined into one group. Each of other records is assumed to constitute one group.
The block integration section 5 supplies combinations of records 1 to 90 (combination of records 1 and 2, combination of records 2 and 3, . . . ) in the block “first four characters of ADDRESS field is [O-saka-hu-o]” to the similarity calculation section 6 for the similarity calculation. The similarity calculation section 6 returns information indicating whether the received records belong to the same group to the block integration section 5. In the case of the block “first four characters of ADDRESS field is [O-saka-hu-o]”, the similarity calculation section 6 returns information indicating that the records 1 to 70 can be combined into one group by respectively determining that records 1 and 2 can be combined into the same group, records 2 and 3 can be combined into the same group, . . . . Similarly, in the case of the block “first four characters of ADDRESS field is [O-saka-hu-mori]”, the similarity calculation section 6 returns information indicating that the records 91 to 95 can be combined into one group by respectively determining that records 91 and 92 can be combined into the same group, records 92 and 93 can be combined into the same group, . . . . The block integration section 5 acquires such information to combine the records belonging to the block “first four characters of ADDRESS field is [O-saka-hu-o]” into 21 groups as shown in
After the group formation, the block integration section 5 selects a representative record for each group. Any method may be used for selecting the representative record. For example, records contained in a duplication candidate group are sorted by some sort of key, and a record positioned at, e.g., the center or head is selected as the representative record. In the case where only one record is contained in a group, the one record is selected as the representative record. It is assumed here that a record 1 is selected as the representative record from the group containing records 1 to 70, and record 91 is selected as the representative record from the group containing records 91 to 95. In other groups, each of which contains only one record, the one record itself serves as the representative record.
The block integration section 5 supplies a combination of the representative records to the similarity calculation section 6 so as to check whether target groups can be integrated or not. The similarity calculation section 6 calculates the similarity between the representative records to determine whether the calculated similarity exceeds the threshold value and returns the determination result to the block integration section 5.
After completing the abovementioned processing for the block “first four characters of ADDRESS field is [O-saka-hu-o]” and block “first four characters of ADDRESS field is [O-saka-hu-mori]”, the block integration section 5 integrates the above blocks and shifts to the integration processing for their parent block in the tree structure, i.e., a block “first three characters of ADDRESS field is [O-saka-hu]”. Assuming that the block “first four characters of ADDRESS field is [O-saka-hu-o]” is as shown in
As described above, the block integration section 5 integrates blocks in the order from a deeper position in the tree structure. As the hierarchical position becomes shallower, the block division condition set in the block division section 3 becomes looser. Thus, as for a group that has not been integrated across, e.g., two or more hierarchies and has still only one record in the course of the block integration processing, it is possible to skip similarity calculation with the representative record of another group.
The above condition is not limited to two or more hierarchies, but may be one or more hierarchies or three or more hierarchies, and need not previously be determined but may be input by a user and may be read from a setting file. By skipping the similarity calculation for the group containing only one record, it is possible to significantly reduce the number of combinations to be checked, thereby reducing the processing time. In fact, there is a fear that a combination supposed to be combined into the same group may be ignored to reduce clustering accuracy. However, the similarity calculation is carried out in the order from a hierarchically deeper position, i.e., in the order from combinations under the tighter block division condition, so that accuracy equivalent or superior to the conventional method in which the rough narrowing-down method and detailed narrowing-down method are used in combination can be expected.
Here, assume a case where a given group has become a group containing only one record in the course of the integration processing performed on the ADDRESS field and similarity check with the representative record of another group is skipped. In this case, if the record is supposed to be contained in a duplicate candidate group with another record, it is likely that two records are combined into the same group in the course of the integration processing performed on, e.g., the NAME field. Therefore, it is believed that accuracy is not degraded when the similarity calculation for the group containing only one record is skipped.
The block integration section 5 combines groups into one when the similarity between the representative records thereof exceeds the threshold value. However, it is likely that even if the similarity between one combination of represented records does not exceed the threshold value, the similarity between another combination of records in the same group may exceed the threshold value. Any combination of records whose similarity exceeds the threshold value is supposed to exist within a group. In most cases, a value of the similarity is believed to be comparatively large, if not exceed, in every combination of the records within a group. Therefore, in the case where the similarity between the representative records does not exceed the threshold value, a detailed check is carried out than only when the similarity value thereof is comparatively large. This comparatively large value serving as a lower similarity threshold value may be input by a user, may be read from a setting file, or may previously be determined. For example, the lower similarity threshold value may be determined by doubling the margin of the similarity threshold value. More specifically, assuming that the similarity value ranges from 0 to 100 and that the similarity threshold value is 90, the margin of the threshold value is 10. Then, the margin of the threshold value is increased to 20, whereby the lower similarity threshold value is set to 80. The level of the margin may be input by a user, may be read from a setting file, or may previously be determined.
In order to check whether groups can be integrated or not, the block integration section 5 supplies combinations of representative records belonging to the groups to the similarity calculation section 6. The similarity calculation section 6 calculates the similarity between the representative records so as to determine whether the similarity between them exceeds the threshold value. When the similarity does not exceed the threshold value but has a comparatively large value, the similarity calculation section 6 determines that it is likely that the representative records may be combined into the same group and calculates the similarity between combinations of all records contained in the two groups to which the representative records belong.
At this time, the similarity calculation section 6 need not check combinations of all records in the two groups but may end the processing if there is any combination whose similarity exceeds the threshold value. Further, since a high similarity value can be expected within one group, a method that mutually checks combinations of the representative record in one group and all records contained in other group can be adopted. Further, the following configuration may be adopted, wherein: when the similarity does not exceed the threshold value but has a comparatively large value, the similarity calculation section 6 returns information indicating that it is likely that representative records may be combined into the same group; and upon receiving the information, the block integration section 5 supplies combinations of all records in two groups to which the representative records belong to the similarity calculation section 6.
After completion of the block integration processing in the manner as described above, the block integration section 5 registers an obtained result in the database 1 through the DB access section 2. The result may be interpreted as duplicate records and registered in the database 1 by overwriting the original database, or may be registered as a new database.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2005-254821 | Sep 2005 | JP | national |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/JP2006/317319 | 9/1/2006 | WO | 00 | 2/29/2008 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2007/026870 | 3/8/2007 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 6424746 | Nishida | Jul 2002 | B1 |
| 20070156401 | Nagano et al. | Jul 2007 | A1 |
| Number | Date | Country |
|---|---|---|
| 07-085070 | Mar 1995 | JP |
| 07-141390 | Jun 1995 | JP |
| 08-147320 | Jun 1996 | JP |
| 11-203415 | Jul 1999 | JP |
| 2002-183171 | Jun 2002 | JP |
| 2003-316796 | Nov 2003 | JP |
| 2004-029969 | Jan 2004 | JP |
| 2005-128961 | May 2005 | JP |
| Number | Date | Country | |
|---|---|---|---|
| 20090271420 A1 | Oct 2009 | US |
