Patent Grant
7739279
1. Field of Invention
This invention relates to information retrieval
2. Description of Related Art
References, reports, technical papers and other documents are increasingly presented in digital formats. Authors of new works frequently make links or connections between portions of the new work and other documents. However, most conventional document authoring systems require that these links be created manually. Some conventional recommender systems, such as Google's Find Similar automatically select documents similar to the first document in a list of documents. These conventional recommender systems select related documents based on the documents statistical similarity to the selected first document. Thus, oft-mentioned but less relevant terms in the first document can distort the selection of additional relevant documents.
Thus, systems and methods that determine relevant information based on the document structure would be useful. Systems and methods are provided for determining relevant information from a document based on the structure of the document. A document is selected and structural elements within the document having a dominance relationship are determined. A first location within the document is selected. The structural element surrounding the first location is determined and the surrounding and non-surrounding structural elements are characterized. Additional documents are associated with the first location in the surrounding structural element based on the characterizations of the surrounding structural element and the non-surrounding structural elements. Systems and methods of dynamically determining annotations for images based on document structure are also provided. Aspect of the present invention also provide a computer readable storage medium comprising computer readable program code embodied on the computer readable storage medium, the computer readable program code useable to program a computer for determining relevant information comprising the steps of: determining a document, identifying at least two structural elements in the document having a dominance relationship, selecting a first location in the document, determining the structural element surrounding the first location, characterizing the surrounding structural element, characterizing the non-surrounding structural elements, and associating at least one additional document with the surrounding structural element based on the characterization of the surrounding and non-surrounding structural elements.
The user of the communications enabled personal computer 300 selects a document 1000 from the information repository 200. The selected document 1000 is forwarded via communications links 99 to the system for determining relevant information 100. At least two structural elements are determined for the selected document 1000. A structural element is a structurally related group of units where the units are atomic objects within a theory of discourse analysis. Thus, in one exemplary embodiment, atomic units are discourse constituents defined by the Unified Linguistic Discourse Model (ULDM) theory of discourse analysis. In various other exemplary embodiments, sentences, paragraphs or other units specified by a theory of discourse analysis may also be used without departing from the scope of this invention.
The user of the communications-enabled personal computer 300 selects a first location within the selected document 1000. The structural elements surrounding the selected first location are determined based on the theory of discourse analysis. In various exemplary embodiments, a location within the selected document 1000 is selected by highlighting a word or position, using a mouse gesture and/or using various other known or later developed input or selection methods. The structural element surrounding the selected word or position is then selected. In an exemplary ULDM based embodiment, if the first location is located within a sub-ordination, co-ordination or n-ary discourse unit, the subordination, co-ordination or n-ary discourse unit surrounding the first location is selected as the surrounding structural element.
A characterization of the surrounding structural element is determined. The characterization of the surrounding structural elements include, but is not limited to, creating a vector based on the frequency of terms in the structural element, performing latent semantic analysis (LSA) on the structural element and/or any known or later developed metric that uses explicit or implicit information about the document structure. Thus, in one exemplary embodiment, the explicit use of titles and/or a particular presentation style or sequence of the document author is an explicit indication of document structure. Systems and methods for determining document structure are further described in Polanyi et al., Systems and Methods for Hybrid Text Summarization, U.S. patent application Ser. No. 10/684,508, filed Oct. 15, 2003, herein incorporated by reference in its entirety. Implicit document structure is structure derived from linguistic or visual information presented in the document, analysis of document content or the like.
A characterization of each of the other or non-surrounding structural elements in the document is determined. As discussed above, the characterization of non-surrounding structural elements is based on term frequency occurrence, latent semantic analysis and/or various known or later developed methods of determining a metric.
Additional documents are determined based on the characterization of the surrounding and non-surrounding structural elements. In one exemplary embodiment according to this invention, a first set of documents is selected based on the similarity of the documents to the surrounding structural element. The documents in the first set of documents that are also similar to the non-surrounding structural elements are removed from the list of additional documents, downgraded or the like. The reduced list of additional documents reflects documents that are similar to the user selected location in the first document but which are dissimilar to other portions of the first document. In various exemplary embodiments according to this invention, the first location in the document is associated with text or image based information. However, it will be apparent that in still other embodiments, the first location is associated with audio, video and/or any other type of information for which a similarity measure can be computed.
The reduced list of additional documents similar to the surrounding structural element is then forwarded over the communications links 99 to the user of communication-enabled personal computer 300. In still other exemplary embodiments, the additional document most similar to the surrounding structural element is automatically associated with the selected first location. Thus, in one embodiment, images representative of the selected first location are identified.
In step S300, one or more structural elements in the document are determined. The structural elements are determined based on the Unified Linguistic Discourse Model (ULDM) of Polanyi et al, Rhetorical Structures Theory (RST), Discourse Structures Theory (DST), Structure Discourse Representation Theory (SDRT) and or various other known or later developed theories that identify a dominance relationship between structural elements. After the structural elements have been determined, control continues to step S400 where a first location in the document is determined.
The first location is determined manually or automatically under program control. Explicit user action such as a mouse or pen gesture, keyboard input, highlighting and the like can be used to manually select the first location. However, implicit user input can also be used to select a first location based on eye or head tracking and/or other indications of interest. The first location may point to an image, a portion of the text, a video file, an audio file and/or any other type of information for which a similarity measure can be computed. After the first location has been determined, control continues to step S500.
In step S500, the structural element surrounding the first location is selected. For example, in an exemplary ULDM based embodiment, the structural element is one of the discourse constituents identified by the ULDM. However, sentences, paragraphs and/or any known or later identified structural element having a dominance relationship between elements may also be used in the practice of this invention.
In other exemplary embodiments, the structural elements are determined based on Rhetorical Structures Theory, Discourse Structures Theory and or various other theories of analysis capable of identifying dominance relationships between structural elements. Thus, in one exemplary embodiment, the surrounding and non-surrounding structural elements associated with the first location are based on the nucleus and satellite elements identified by Rhetorical Structures Theory. Once the surrounding structural element is determined, control continues to step S600.
A characterization of the surrounding structural element is determined in step S600. In various embodiments, the surrounding structural element characterization is determined by analyzing the structural element with latent semantic analysis. Latent semantic analysis produces words representative of the surrounding structural element. A vector representation of the structural element is then determined. After the surrounding structural element has been characterized, control continues to step S700.
In step S700, characterizations of the non-surrounding structural elements are determined. The characterizations are based on latent semantic analysis or any other method of determining a representation of the non-surrounding structural elements. After the characterization of the non-surrounding structural elements has been determined, control continues to step S800 where at least one additional document is determined based on the characterization of the surrounding and non-surrounding structural elements.
In one embodiment, at least one additional document is determined by selecting documents most similar to the surrounding structural element and most dis-similar from the non-surrounding structural elements. Documents more similar to the surrounding structural element and less similar to the non-surrounding structures are selected since these documents are most likely to be useful in signaling the difference in meaning between the information content of the surrounding structural element and the information content of the non-surrounding structural element. After the at least one additional document has been selected, control continues to step S900.
In step S900, a determination is made as to whether the session has ended. If it is determined that the session has not ended, control jumps immediately to step S400. The steps S400-S900 are repeated until it is determined that the user has ended the session. Control then continues to step S1000 and the process ends.
A user of the communications-enabled personal computer 300 selects a document 1000 from the information repository 200. The selected document 1000 is forwarded over communications links 99 and displayed on the communications-enabled personal computer 300. The user of the communications-enabled personal computer 300 selects a first location within the selected document 1000 by highlighting, using a mouse click, voice selection and/or various known and/or later developed input or selection methods. The first location may be associated with image, text, audio, video or any other known or later developed type of information.
The first location information and the selected document 1000 are forwarded over the communications links 99 to the input/output circuit 10 of the system for determining relevant information 100. The selected document 1000 and first location information are stored in memory 20. The processor then activates the document structure manager 40 to determine the structure of the stored document. In one embodiment subordination, n-ary and co-ordination discourse constituents of the document are identified according to the ULDM theory of discourse analysis. However, it will be apparent that Rhetorical Structures Theory (RST), Discourse Structures Theory (DST), Structure Discourse Representation Theory (SDRT) and/or any known or later developed method of determining dominant and subordinate structural elements in the selected document 1000 may also be used in the practice of this invention.
The processor 30 retrieves the stored location information from memory 20 and activates the structural element manager 50 to determine the structural element surrounding the location in the selected document 1000. The processor 30 activates the characterization manager 60 to determine characterizations of the surrounding and non-surrounding structural elements. In various embodiments, the characterizations are based on latent semantic analysis, representative vectors and the like.
The processor 30 then activates the search manager 70 to determine additional documents based on the characterization of the surrounding and non-surrounding structural elements. For example, in one embodiment, the search manager 70 identifies additional documents based on similarity to the characterization of the surrounding structural element and dissimilarity to the characterization of the non-surrounding structural elements. In various other embodiments, the N-best additional documents are associated with the first location. N can be any number greater than 0. For example, with N=3, three additional documents are presented as candidate documents to be associated with the first location.
In one exemplary embodiment according to this invention, the user indicates a first location 440 by pointing to, highlighting or otherwise selecting the “Jane's” portion within the second structural element 420. The second structural element surrounding the selected word “Jane's” is identified. A vector or concept representation is associated with the second or surrounding structural element 420. The vector or concept representation captures the meaning associated with the terms “Ted and Jane's marriage”.
Concept vectors or meaning representations are also associated with the non-surrounding first and third structural elements 410, 430. Concept vectors or meaning representations capture the meaning of the phrase “Jane's first trip to Mexico” from the first phrase and “Jane's last trip to Europe” from the last phrase. The vectors or concept representations are used to select additional documents for association with the first location that are similar in meaning to the surrounding structural element but which are dissimilar to the non-surrounding structural elements. Thus, additional documents would be conceptually similar to Ted and Jane's marriage but would differ from documents describing Jane's trip to Mexico and Jane's last trip to Europe. This facilitates the selection of additional documents that fit within the conceptual development of the document. For example, in various embodiments, illustrative pictures for a selected concept are proposed using the systems and methods according to this invention. Similarly, in another embodiment, illustrative text captions are proposed for a given image.
The second row of the data structure for storing concept vectors 500 contains the value “INFO.ELEMENT[2]”, and “Ted/Jane/marriage”. This value reflects a meaning representation created from the salient concepts in the second structural element identified as “TED AND JANE”S MARRIAGE”.
The third row of the data structure for storing concept vectors 500 contains the value “INFO.ELEMENT[3]”, and “Jane/trip/Europe” indicating the salient elements of the third structural element “JANE”S LAST TRIP TO EUROPE”. The third concept vector or meaning representation is similar to the first concept vector since the first and third structural elements both relate to trips by “Jane”. All three structural elements relate to “Jane” but the second structural element relates to the additional concepts of “marriage” and “Ted”. The third structural element relates to the additional concepts of “trip” and “Europe”.
Thus, additional documents relating to the concepts of “Ted”, “Jane” and “marriage” and which are most dissimilar to the concepts, “trip” “Mexico” and “Europe” are selected from the information repository. For example, the likelihood of presenting pictures associated with concept vectors from Jane's trips to Mexico and Europe are decreased while the likelihood of selecting pictures annotated with concept vectors describing Jane's marriage to Ted, is increased. In other exemplary embodiments, the most likely document is dynamically associated with the surrounding structural element.
The second row contains a second structural element 731, “They were married in New York City in 1984”. The second structural element 731 is associated with the concept vector “Jane/Ted/marriage”. In various exemplary embodiments, the concept vector substitutes “Jane/Ted” for “they” in the structural element. The third row contains the third structural element 732 “Ted and Jane now live in Maine with their two children, Jessica and Timothy”. The third structural element 732 is associated with the concept vector “Ted/Jane/live/Maine”. A first location 740 within the document is determined by highlighting the portion “married”. The first location is surrounded by the second structural element 731.
In one exemplary embodiment according to this invention, vector or meaning representations characterizing the second structural element 731 and vector or meaning representations characterizing the non-surrounding first and third structural elements 730, 732 are used to identify additional documents. That is, additional documents 720 and/or portions 721, 722, 723 of additional documents are associated with vector or meaning representations. Additional documents that are similar to the vector or meaning representation of the second structural element 731 surrounding the first location 740 and dissimilar to the concept vector or meaning representation of the first and third non-surrounding structural elements 730, 732, are selected. The N-best additional documents or document portions, for example document portion 721, are displayed and/or automatically associated with the first location 740.
The user indicates a first location 840 within the document 810 using a mouse or pen gesture, an eye movement, voice recognition and/or any other input or selection method. The structural element 831 surrounding the indicated and/or selected first location 840 is determined. Characterizations of the surrounding and non-surrounding structural elements are also determined. A search is performed for additional image documents 821-823 which are characterized as similar to the surrounding structural element 830 and which are dissimilar to the non-surrounding structural elements. The image documents 821, 822, 823 may be separate image documents or portions of a document 820. The image documents 821-823 may be annotated manually and/or automatically. The resultant additional image documents are representative of the meaning of the surrounding structural element 830.
The second image2920 embedded within the document is selected. The structural element surrounding the second image 2920 is determined. In this case, the surrounding structural element is the two phrase unit “The roadway of the bridge is 246 feet above sea level while the towers rise to 746 feet. The roadway is suspended from the towers via suspension cables.” An exemplary concept vector for second image2920 includes the terms “Golden Gate Bridge” 911, “towers” 921, “rise” 922 and “746 feet” 923. The concept vector describes the meaning or concepts of the images. This allows the images to be more easily re-used while authoring or viewing other documents.
The first row of the exemplary data structure for storing image concept vectors 1000 contains the value “IMAGE1” in the image identifier portion 1010. This value uniquely identifies the image to the system for determining relevant content.
The information source portion 1020 of the exemplary data structure for storing image concept vectors 1000 contains the value “INFO.ELEMENT[1]=“The Golden Gate Bridge links the city and county of San Francisco with Mann County.” This reflects the structural element within which “IMAGE1” is embedded.
The concept vector portion 1030 contains the value “GOLDEN GATE BRIDGE, SAN FRANCISCO, MARIN COUNTY”. This indicates the concept vector that is used to represent the image in other associations. The concept vector is based on or derived from the structural representation surrounding “IMAGE1”.
The last row of the exemplary data structure for storing image concept vectors 1000 contains the value “IMAGE2” in the image identifier portion 1010 uniquely identifying the image to the system.
The information source portion 1020 of the exemplary data structure for storing image concept vectors 1000 contains the value “INFO.ELEMENT[1]=“The roadway of the bridge is 246 feet above sea level while the towers rise to 746 feet. The roadway is suspended from the towers via suspension cables.” This reflects the two-phrase structural element within which “IMAGE2” is embedded.
The concept vector portion 1030 contains the value “GOLDEN GATE BRIDGE, TOWER, RISE, 746 FEET”. This indicates that a concept vector will be used to represent “IMAGE2” that indicates the image reflects information about how far the Golden Gate Bridge towers rises.
Each of the circuits 10-70 of the system for determining relevant information 100 can be implemented as portions of a suitably programmed general-purpose computer. Alternatively, circuits 10-70 of the system for determining relevant information 100 outlined above can be implemented as physically distinct hardware circuits within an ASIC, or using a FPGA, a PDL, a PLA or a PAL, or using discrete logic elements or discrete circuit elements. The particular form each of the circuits 10-70 of the determining relevant information 100 outlined above will take is a design choice and will be obvious and predicable to those skilled in the art.
Moreover, the determining relevant information 100 and/or each of the various circuits discussed above can each be implemented as software routines, managers or objects executing on a programmed general purpose computer, a special purpose computer, a microprocessor or the like. In this case, the determining relevant information 100 and/or each of the various circuits discussed above can each be implemented as one or more routines embedded in the communications network, as a resource residing on a server, or the like. The determining relevant information 100 and the various circuits discussed above can also be implemented by physically incorporating the determining relevant information 100 into software and/or a hardware system, such as the hardware and software systems of a web server or a client device.
As shown in
The communication links 99 shown in
Further, it should be appreciated that the communication links 99 can be wired or wireless links to a network. The network can be a local area network, a wide area network, an intranet, the Internet, or any other distributed processing and storage network.
While this invention has been described in conjunction with the exemplary embodiments outlined above, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, the exemplary embodiments of the invention, as set forth above, are intended to be illustrative, not limiting. Various changes may be made without departing from the spirit and scope of the invention.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6766316 | Caudill et al. | Jul 2004 | B2 |
| 7113943 | Bradford et al. | Sep 2006 | B2 |
| 20030233224 | Marchisio et al. | Dec 2003 | A1 |
| 20040095376 | Graham et al. | May 2004 | A1 |
| 20050086592 | Polanyi et al. | Apr 2005 | A1 |
| 20070022072 | Kao et al. | Jan 2007 | A1 |
| Number | Date | Country | |
|---|---|---|---|
| 20070143098 A1 | Jun 2007 | US |
