Patent Application
20080059488
a illustrates an example of the document referenced in
b illustrates an exemplary embodiment of a positional representation generated from the document of
c illustrates an exemplary embodiment of a positional representation generated from the document of
a and
In general, exemplary embodiments of the invention as described in further detail hereafter include systems and methods for providing an efficient technique of organizing documents prior to indexing by a search engine. This facilitates seeking the exact location of a search term once a relevant document has been located by making use of a positional representation of a document. The positional representation also facilitates efficient inverted indexing of documents by search engines. In addition, the positional representation greatly simplifies the computation of the inverted index, making it compatible with the native indexing structures used by state-of-the-art search engines. Positional representations of documents are equivalent lossless representations of those documents. The positional representations are essentially compressed versions of the original documents and typically occupy less memory than the original documents, resulting in reduced storage requirements.
Exemplary systems and methods for organizing documents prior to indexing by a search engine will now be discussed in further detail with reference to illustrative embodiments of
A document is input (110) into the system (100) and is passed to the term identifying and position determining module (120) which identifies all of the unique terms in the document and their respective positions. The unique terms may be one or more words or an annotation. This information is passed to the record generation module (130) which builds a record for each of the unique terms having positional information derived from the respective positions. The record generation module (130) combines the records into a positional representation data structure and stores the data structure into the positional representation database (140). Additional documents are input (110) as necessary, each creating a new entry in the positional representation database (140). The index generation module (150) processes the positional representations stored in the positional representation database (140) to generate an inverted index.
In a typical inverted index, for each term, a list of documents in which that term appears is stored. The inverted index is often generated by traversing directly through each of the documents. However, generating the inverted from positional representations of those documents is less time consuming because each positional representation is smaller and better organized than the original document.
The inverted index is stored in the inverted index database (160). When a user enters a search query (170), the query processing module receives the query, and retrieves the inverted index from the inverted index database (160). The query processing module (160) traverses the inverted index until it determines a document is most relevant to the entered query, retrieves the positional representation that corresponds to the document from the positional representation database (140) and returns the relevant document, advanced to a relevant position (190) based on the retrieved positional representation.
a illustrates an example of the document referenced in
b illustrates an exemplary embodiment of a positional representation generated from the document of
Referring to
c illustrates an exemplary embodiment of a positional representation generated from the document of
Referring to
The positional representations embodied in
The positional representations embodied in
The positional representations may be stored in a database, main memory, cache, hard disk, etc. When a positional representation is stored as a file, the filename may correspond to the document the positional representation was converted from. As an example, the document having filename ‘text.pdf’ may be converted into a positional representation having filename ‘text.pdf.pr’. Since the original file name can be discerned from filename of the positional representation, the positional representation need not contain a document identifier.
A positional representation may also include annotations from the document. An annotation is extra information associated with a particular point in a document or a particular section, sentence, term, image, audio clip, video clip, etc., and is typically not visible to a user unless specifically requested. As an example, the term ‘Sally’ in
Multimedia documents may contain a mixture of text along with embedded images, audio clips, video, etc. For these documents, an XML descriptor or similar format descriptor in addition to the source document is typically created. The descriptor is the one that is typically indexed by the search engines rather than the source itself. For a multimedia document, the positional representation is generated from the XML descriptor instead of the document.
Annotations become especially important in multimedia documents since although much of the document may be in binary, it may be interspersed with annotations that identify frames or scenes. A conventional search for a particular actor might return a relevant multimedia document, but not where in the document the actor appears. The times that the actor appears in the movie can be derived from the positions of the annotations.
a and
a, illustrates how the inverted index is generated using a flowchart. In a first step, a collection of documents are input 310. In a second step, each of the documents in the collection are converted to a positional representation 320. In a final step the resulting positional representations are used to generate an inverted index 330.
The inverted index 370 can then be used with any search application such as a search engine or search middleware to retrieve documents relevant to an entered query. Once a user enters a search term, a search engine can traverse through the entries in the inverted index. Upon finding a matching entry in the inverted index for the search term, documents relevant to the search term can be identified and potentially viewed by the user. When one of the relevant documents is not readily available in its original form, it can be re-created by translating its positional representation. Since all of the unique terms and their positions are known, it is a trivial matter for an application do the translation.
When a relevant document is displayed, the presentation of the document can be automatically advanced to any of the occurrences of the search time by using the positional information stored in the positional representation of the relevant document. As an example, a lengthy document having several occurrences of the search term ‘spinal meningitis’ could be automatically scrolled to the first or subsequent occurrence of the term. When the relevant document is a multimedia document, the presentation of the document in time can be advanced to the location of the search term. As an example, an mpeg movie document having an occurrence of an annotation of ‘finale’ could automatically be advanced to the finale in the movie.
According to an exemplary embodiment of the present invention an apparatus is provided that includes a processor for converting a document to a positional representation. The processor extracts each of the unique terms from the document and their respective occurrence positions in the document. The processor next generates entries for each of the unique terms. Each of the entries includes a unique term and positional information which can be used to derive the positions where that unique term appears in the document. The positional information may be the positions of the unique terms, offsets of occurrences of the unique terms or some combination thereof. Once the processor has completed generation of the entries, it combines the entries into a data structure known as a positional representation. The positional representation may also include a document identifier to identify the document the positional representation was derived from.
In the interest of clarity, not all features of an actual implementation are described in this specification. It will be appreciated that in the development of any such actual embodiment, numerous implementation-specific decisions must be made to achieve the developers' specific goals, such as compliance with system-related and business-related constraints, which will vary from one implementation to another. Moreover, it will be appreciated that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking for those of ordinary skill in the art having the benefit of this disclosure.
While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawings and are herein described in detail. It should be understood, however, that the description herein of specific embodiments is not intended to limit the invention to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the invention as defined by the appended claims. It should be understood that the systems and methods described herein may be implemented in various forms of hardware, software, firmware, or a combination thereof.
The particular embodiments disclosed above are illustrative only, as the invention may be modified and practiced in different but equivalent manners apparent to those skilled in the art having the benefit of the teachings herein. Furthermore, no limitations are intended to the details of design herein shown, other than as described in the claims below. It is therefore evident that the particular embodiments disclosed above may be altered or modified and all such variations are considered within the scope and spirit of the invention. Accordingly, the protection sought herein is as set forth in the claims below.
