Patent Application
20050267915
The present invention relates to a method and apparatus for recognizing specific type of information files.
The information is usually stored and archived in the form of files. Similarly, the information broadly spreading on Internet is also distributed and transmitted in the form of Web files. With the fast development of the Internet, the amount of Web file information is increasingly growing up and accounts for a substantial proportion, thus making more significant the importance of the information processing techniques on Internet such as classification and retrieval of Web files. Also with the fast development of networks, the subscribers' demands for online information are getting diverse. Generally, the searching method based on string matching could well satisfy the subscribers' requirements for searching refined information. However, classification or recognition of some file groups characterized by information types is not so satisfying.
Today, with the high speed development of networks, information carried by Web pages is getting highly integrated and the content thereof is getting more and more complicated and diverse. Many information contents such as hyper link and hyper media information have become indispensable parts of the Web pages. It increased the amount of transmittable information and improved the user interfaces to a certain extent, on the other hand, it renders the structures of Web pages complicated, brings about various topics in the Web information and adds noise to the main information contents. Heretofore, many researchers engaging in Web information processing proposed various Web information-blocking method in an attempt to accurately understand and extract main information, such as:
Ziv Bar-Yossef and Sridhar Rajagopalan 2002. Template Detection via Data Mining and its Applications. In Proceedings of the WWW2002, May 7-11, 2002, Honolulu, Hi., USA.
Shian-Hua Lin, Jan-Ming Ho 2002. Discovering Informative Content Blocks from Web Documents. SIGKDD '02, Jul. 23-26, 2002, Edmonton, Alberta, Canada.
As is well known, in the Web information, the information carried on Web is organized and expressed by HTML description language, and the Web information is interpreted and displayed to the end users with Web browsers. Seemingly, this kind of information flow is a linear text information flow, but actually, the Web information flow has certain organization structures. The composition structure analysis of Web file, which is also a key technique of Web page information processing, shall be conducted prior to processing of Web information. In the Web pages, the page contents are organized with HTML description language, and the information structure thereof can be mapped to a DOM (Document Object Model) tree with HTML Tag and Web text information as its nodes. The existing browsers display Web pages by parsing DOM tree structure of Web pages. Text information in Web pages is organized with information to be conveyed with Tags defined in HTML. Structure trees of Web information can be processed by parsing the functional attributes of the tags. (Ziv Bar-Yossef 2002) proposed a relatively simple heuristic page blocking method that partitions Web pages based on semantic consistency of information by using DOM tree and different attributes of HTML Tags, so as to separate different information topics. (Shian-Hua Lin 2002) proposed a method for detecting and partitioning information blocks of Web pages by utilizing HTML Tags such as <Table>. It can be seen that both methods partition Web pages by using different attributes of HTML Tags in order to extract desired information contents of the users.
In order to address the above-mentioned problem in classifying and recognizing file group characterized by information type, the present invention provides a method and apparatus for recognizing specific type of information files, which can conduct a file type-based recognition on Web pages collected from Internet or file groups stored in related storage apparatus. Based on the fact that files of the same type have attributes specific thereto that can be effectively utilized in file type recognition, the invention groups the input files, which achieves an effect of pre-classification of file samples, and contributes to the improvement of recognition precision. In an aspect of the invention, there is provided a file recognition apparatus, which comprises: a file grouping section for classifying the files to be recognized by types in the viewpoints such as URL and author names, and grouping the files based on their attributes, so that the subsequent recognition modules can conduct recognition based on the file attributes of each groups, the file grouping section also serves an effect of pre-classification of the samples, and improves the ultimate recognition precision of the system; a file type recognition section for extracting main information blocks of a file based on inherent DOM structure of the Web page and attributes of HTML Tags, and determining the information type, such as lyric, log and BBS, of the file, the file type recognition section recognizes file types based on characteristics specific to the above-mentioned specific information, such as key words, punctuation marks, document structure and repetition of contents; and a file-type-recognition correction section for correcting, in consideration of recognition precision of whole files in conjunction with recognition results of each individual files, all file recognition results of the group, with special attention paid to the overall recognition accuracy of all files in the group, so as to improve the overall recognition precision of all files.
Preferably, in the file recognition apparatus of the invention, the file type recognition section comprises a main-information-block extraction unit for extracting main information block from files and removing noise components that have no significance to the file.
Preferably, in the file recognition apparatus of the invention, the file-type-recognition correction section summarizes the recognition result of each file in current file subgroup, calculates a ratio of number of files recognized as positive example to the number of files in current subgroup by taking the current file subgroup as an unit, and determining the current file subgroup by comparing the ratio to a predetermined threshold value.
In another aspect of the invention, there is provided a file recognition method for recognizing a specific information type with respect to a file group collected from the Internet or stored in other storage apparatus, the method comprising steps of: classifying the files to be recognized by file types from a predetermined viewpoint; recognizing the types of the files based on characteristics specific to the specific information type; and correcting the recognition result of each file in consideration of the recognition precision of all files in the group.
Preferably, in the file recognition method of the invention, the step of recognizing further comprises a step of removing noise components that have no significance to the file, and extracting only the main part.
Preferably, in the file recognition method of the invention, the step of correcting summarizes the recognition result of each file in current file subgroup, calculates a ratio of number of files recognized as positive example to the number of files in current subgroup by taking the current file subgroup as an unit, and determine the current file subgroup by comparing the ratio to a predetermined threshold value.
An embodiment of the apparatus for recognizing specific type of information files of the invention and the reorganization method used therein will be described with reference to the drawings, with the reorganization of lyric pages as an example.
(1) File Grouping Section
First of all, this file grouping section conducts a file type classification on the input file groups, which are Web pages collected from the Internet or file groups stored in other storage apparatus, based on various viewpoints such as URL and author names.
In most of the conventional systems, all files to be recognized are equal to the recognition system, and the system recognizes and determines each individual file with the same method and resources. This is basically reasonable in the viewpoint of system modeling and is fair to each files to be recognized. However, there are certain associations among files in practical applications, and such associations exhibit in form of specific file attributes, while the conventional systems failed to make use of this characteristic. The file grouping section of this invention is just based on this consideration, and classifies files in different viewpoints such as URLs and author names and takes respective classes as input of the system. Thus the individual files can be associated and the system can conduct recognition based on common attributes of each group.
From the viewpoint of the system overall recognition function, the file grouping section bring to an effect of pre-classification of the input samples, which contributes to the improvement of the ultimate overall recognition precision of the system.
(2) File Type Recognition Section
In the file type recognition section, the structure information of the DOM trees and the attributes of HTML Tags are fully exploited to extract main information blocks from complicated Web pages. In this case, the invention adopts a method for extracting main information block from Web page based on web page template information, in order to remove the interference of noise components to reorganization of the web main information and therefore to improve the reorganization precision of the system.
The file type recognition section extracts main information block of the file based on inherent DOM structure of the Web pages and attributes of HTML Tags, and determines the specific information type (lyric information) of the file based on the main information contents. Then it uses characteristics specific to lyric information which is a type of specific type information, such as key words, punctuation marks, document structure and repetition of contents, to recognizing file type.
1. As a key technology in Web page information processing, the file-DOM-tree representation unit realizes the mapping of linear flow of a Web page source code to DOM tree structure of the Web file, and underlies the subsequent file structure analysis. As is known, Web pages, in which the information contents to be conveyed are formatted with HTML description language, consists of HTML Tag information, notes information and main information to be conveyed. The notes information is of no help to the structure analysis, while the Tag information contains abundant structure information. In the DOM tree, information to be conveyed by Web pages usually exists in the form of leaves with the node attribute thereof being text attribute.
2. The information-blocks-of-leaf-node-in-DOM-tree merging unit realizes delimitation and positioning between different information blocks in a Web page. The HTML source files of Web page files are displayed to users after being interpreted by a browser. From the viewpoint of display effect, the organization of information has certain structure and different text information aggregate to a certain extent in different locations in the Web page, i.e., exist in form of information blocks. There are also certain associations among corresponding nodes on DOM tree of the Web page. This merging unit realizes the merging of information blocks as follows.
In order to find out relationship between information blocks with HTML DOM tree, the DOM tree need to be processed first to eliminate irrelevant information nodes such as script nodes, and to mark out significant nodes. The following is the merging method for information blocks:
(a) Defining Relevant Symbols Used in the Algorithm
(b) Traversing the Entire DOM Tree for the Web Page with a Depth-First Postorder and checking each node in the following way:
Step 1:
All the DN nodes will be canceled up to now.
Step 2:
A relatively compact Web Page DOM tree can be obtained after canceling unreasonable nodes in the tree. Now, if we cascade contents of all leaf nodes of different child tree, we can find that each string stands for an information string, i.e., the Web Page information block.
3. The data-structure-of-information-block-of-DOM-tree representation unit converts the Web page information as node-merged into a data structure of web page information blocks. After being processed by the information-blocks-of-leaf-node-in-DOM-tree merging unit, the Web page information is divided into different information blocks. For the purpose of the subsequent extraction of template information block, the processed DOM tree information contents are copied to the data structure of the DOM tree information blocks. This data structure is a chain table structure in which each node stores one information block content of the Web page. The data-structure-of-information-block-of-DOM-tree representation unit copies all leaf nodes of corresponding information block child tree in the processed DOM tree sequentially to the nodes of chain table, in an order of from left to right.
4. The similarity-of-string-in-information-block calculation unit calculates the similarity between two strings. The similarity between strings is defined as the similarity degree of the two strings as calculated. A double type variable lying within the range of [0,1] is used to denote the similarity, 0 for no similarity and 1 for identical strings. In this calculation unit, similarity calculation is accomplished by calculating edit-distance of two strings. Three edit operations for characters: insertion, canceling and swapping, are defined, and operation function costs of these three operations are set to 1. Then dynamic programming method will be applied to calculate their similarity.
5. The template-information-block extraction unit extracts template information for Web page training set (two representative Web pages). After processing of the above-mentioned units, data structure of DOM tree information block corresponding to the training set Web pages (such as the two input chain tables Table_1 and Table_2 shown in
1. The specific algorithm for the current-Web-page-file-DOM-tree representation unit is the same as that for the file-DOM-tree representation unit of the template-information-for-file-subgroup extraction unit.
2. The specific algorithm for the leaf-nodes-in-DOM-tree-for-current-Web-page merging unit is the same as that for the Information-blocks-of-leaf-node-in-DOM-tree merging unit of the template-information-for-file-subgroup extraction unit.
3. The specific algorithm for the information-block-in-current-Web-page-file representation unit is the same as that for the Data-structure-of-information-block-of-DOM-tree representation unit of the template-information-for-file-subgroup extraction unit.
4. The specific algorithm for the similarity-of-string-in-information-block calculation unit is the same as that for the information block strings similarity calculation unit of the template-information-for-file-subgroup extraction unit.
5. The main-information-block-of-Web-page extraction unit extracts the main information block from the Web page information.
After processing of the above-mentioned units, data structure of information block of DOM tree corresponding to the current Web page (such as the input chain table Web_Table shown in
The characteristic-information recognition unit employing key word/counter key word screen matching searches and matches the main information block with key word characteristics and calculates the key work score of this Web page and saves it in the characteristic information variables. Three vectors, Tc, Tf and Tw are defined, where Tc is key word vector, Tf is appearance frequency vector of the key word in the current main information block and Tw is weight vector of the key word. After searching and matching each main information block, the current value of Tf can be obtained and the inner product Tc·Tf·Tw, i.e., the characteristic word score of the current Web page main information block, can be computed. The score is stored in the characteristic information variables for further determination.
The above key word searching and matching process uses the complete matching technology of string and therefore tends to ignore the error accumulation when the matched information isn't the “string sub-set” of non-key word information and the non-key word information expresses another semanteme. The “counter key word screen algorithm” is proposed to address this problem, i.e., matching with “key word matching algorithm” after pre-matching possible key word information of this kind.
Linking-characteristics-of-information-block extraction unit implements the summarizing analysis for chain table of main information block. In the linking-characteristics-of-information-block extraction unit, the length of the link text and the text length of current main information block are counted and the ratio of these two lengths is calculated. The result is saved in the characteristic variables for further determination.
The sectioning-characteristic-information-of-information-block extraction unit implements summarization of line segmentation information of the main information block. The number of sub-segment in each line is counted, the average number of line segment in the current main information block is obtained and saved in the characteristic variables for further determination. In this case, the line sub-segment is defined as the character segment in text information separated by one or more spaces.
The text-repetition-characteristic-information-of-information-block extraction unit implements the summarizing analysis of text repetition of the main information block. Firstly, it orders all lines in current main information block in unit of line according to text contents. Secondly, from the first line, it calculates similarity of each neighboring lines' text contents in turn and saves the calculation results in corresponding temporary variables. Finally, it counts the number of line information similarity that are bigger than a threshold and saves the information in characteristic variables for further determination.
The text-punctuation-mark-characteristic-information-of-information-block extraction unit implements the summarizing analysis of the punctuation mark characteristic information of main information block. It counts predetermined punctuation marks in the current main information block contents and saves the information in characteristic information variables for further determination.
The text-length-characteristic-information-of-information-block extraction unit implements the summarizing analysis of text length of main information block and saves the characteristic information in the characteristic information variables for further determination.
The comprehensive determining unit implements comprehensive determination of parameter values saved in characteristic information variables. This unit defines three parameters representing three performance levels for each characteristic information including key word, information block association, line segmentation of information block, text repetition of information block, text punctuation mark of information block and text length of information block, respectively, as shown in the following table:
The values can be selected based on predetermined threshold values, and the type of main information blocks can be determined with a heuristic rule. In this embodiment, the following heuristic rule are adopted:
All files with the characteristic information variable determined based on the current information block matching the above-mentioned rules are determined as positive example recognition results, otherwise as negative example recognition results.
(3) File-Type-Recognition Correction Unit
The file-type-recognition correction section corrects all reorganization results in the current group in consideration of the overall recognition results of files in the same group and in conjunction with recognition results of each individual files, with special attention paid to the overall recognition accuracy of all files in the group. Specifically, the file-type-recognition correction section summarizes recognition results for each file in current file subgroup, takes the current file subgroup as an unit and calculates the “correct recognition rate” of this subgroup, i.e., the ratio of number of files recognized as positive example to the number of files in current subgroup, and makes a determination with respect to the current file subgroup based on a predetermined threshold value.
An embodiment of the reorganization apparatus and method according to the invention has been described by taking the reorganization of lyric web pages as an example. However, the invention is not limited to the reorganization of lyric web pages, and instead can be applied to all kind of information files. In addition, details as described above are merely illustrative and for providing a better understanding of the invention. Various modifications and variations can be made to the reorganization apparatus and method according to the invention within the scope as defined in the claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2004-100383575 | May 2004 | CN | national |
