The present invention relates to the processing of electronic text generally.
Since its introduction to the public in the late 20th century, email has become a popular and widely used form of communication both at home and in the workplace. In addition to the advantages email introduced to the realm of interpersonal communications, by making the delivery of written messages quicker and more convenient, email further introduced completely new benefits to the exchange of written messages.
For example, the “Reply” and “Forward” functions available to email users introduced the “discussion thread”. Reference is now made to
Person Y then replies, as indicated by arrow 15, to Person X, by using a conventional email “Reply” function. Use of the “Reply” function generates email 11, which contains root message text 20, and to which Person Y adds his reply text 21. This first reply is defined as Round 2 of email exchange 19.
Person X then replies to Person Y using the “Reply” function, as indicated by arrow 17, in which case his reply email 12 contains root message text 20, first reply text 21 and new reply text 22. This second reply is defined as Round 3 of email exchange 19.
One benefit of the email discussion thread is that it provides running documentation of a discussion occurring between two or more people. At any time it is possible to read the entire discussion thread beginning from the root message, and thus obtain a full picture of what was discussed, and which contributions to the discussion were made by whom, without a laborious search for documents.
Unfortunately, discussion threads significantly increase the volume of messages to be stored and processed by an email administration system. For example, the processes of indexing and searching message volumes become increasingly cumbersome with increasing message volume size.
The subject matter regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of the specification. The invention, however, both as to organization and method of operation, together with objects, features, and advantages thereof, may best be understood by reference to the following detailed description when read with the accompanying drawings in which:
a and 11b are graphical illustrations of the root lookup and last offspring data for the exemplary compact email threads of
It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements.
The present invention provides a novel method of indexing and searching large volumes of electronic message transmissions (EMTs).
There is provided according to embodiments of the invention a method, which is carried out by storing information describing configurations of discussion threads formed of respective series of EMTs that are exchanged among at least two individuals, the discussion threads having a root EMT, zero or more reply EMTs, and a last offspring EMT. The method is further carried out by compacting the EMT discussion threads, and indexing the compacted EMT discussion threads.
According to an aspect of the method storing information includes enumerating the EMTs by assigning numerical identifiers to the EMTs according to creation times thereof, and determining respective thread types of the EMT discussion. The thread type is selected from a linear thread type, wherein each EMT contains a full text of each preceding EMT of its respective series of EMTs, and a conjoined thread type that includes a set of EMT discussion threads that share a common root EMT. The method is further carried out by identifying the root EMT of EMT discussion threads, respectively, and identifying the last offspring EMT in the EMT discussion threads, respectively.
According to a further aspect of the method, enumerating includes assigning consecutive numerical identifiers to successive EMTs within the EMT discussion threads, wherein for respective EMT discussion threads the root EMT thereof has a lowest numerical identifier, and the last offspring EMT thereof has a highest numerical identifier.
According to still another aspect of the method, each EMT includes meta-data and new content, which the EMT adds to its discussion thread and wherein compacting includes identifying meta-data and new content of each EMT, and wherein indexing includes indexing the meta-data and the new content for the EMTs.
According to an additional aspect of the method, indexing includes distinguishing between occurrences of a term in the meta-data and in content of each EMT.
In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the present invention.
Applicants have realized that a significant portion of the volume of email messages in an email management system may be largely due to the repetition of messages in threaded discussions. Applicants have realized that similar threaded discussions are also common in newsgroups. The present invention may be operable for all systems which have threaded discussions.
Applicants have further realized that for well-ordered threads, the pattern in which messages are repeated is typical, as shown in
Reference is now made to
Electronic messages in general, of which email is one type, and of which newsgroup postings are another type, may have meta-data associated with each message transmission, as recorded by the server sending and receiving the messages.
Returning now to
As may be seen, in an email volume containing emails 10, 11 and 12, text 20 occurs three times, text 21 occurs two times, and text 22 occurs once. The present invention may be a search engine which may generally conserve email administration system resources by exploiting the structure of email threads to index each of text sections 20, 21 and 22 one time only. The search engine may also search the index and rank search results according to retrieval policies based on the singularities of message thread structures, improving the effectiveness of the search and the quality of the results.
The search engine disclosed in the present invention may also be operable for all other types of electronic message transmission (EMT) volumes which are comprised of discussion threads, such as newsgroup postings.
A preferred embodiment of the present invention may be as shown in
Returning now to
Thread processor 42 may analyze threads edt1 . . . edtn to ascertain their structures, assign identification numbers to each EMT, and compile thread structure reference tables, described in more detail hereinbelow, defining the structures of the EMT threads. Thread structure information processed by thread processor 42 may be stored in thread management database 43. The output of thread processor 42 may be a compact EMT volume 56, which may be smaller in size than original volume 50. The reduced EMTs which form compact EMT volume 56 may consist of the new text contribution of each EMT in a thread and its meta-data, and may not include repetitions of text from preceding EMTs in the thread. Indexer 44 may create a traditional inverted index 58 for compact EMT volume 56. Query manager 46 may process queries 52 input into search engine 40 by accessing thread management database 43 and index 58. Query manager 46 may return search results 54 in response to query 52.
EMT threads edt1 . . . edtn may have characteristic structural configurations, e.g. linear or conjoined. The exemplary email thread shown in
As shown in
In a conjoined thread set configuration, such as that depicted in
Reference is now made to
For linear EMT threads lt1 . . . ltn, docID assigner 60 may assign consecutive numerical IDs (docIDs) to sequential EMTs on the basis of document creation time stored in the EMT meta-data. For conjoined thread sets ct1 . . . ctn, DocID assigner 60 may use Depth First Search (DFS) numbering, as is well known in the art, on the directed tree implied by the thread-set, starting from the root document.
Root lookup table compiler 62 may analyze threads edt1 . . . edtn and may compile a root lookup table 67 listing the root EMT docID for every assigned docID. Last offspring lookup table compiler 64 may analyze threads edt1 . . . edtn and may compile a last offspring lookup table 68 listing the last offspring EMT docID for every assigned docID. The last offspring EMT of EMT X is defined as the EMT having the highest docID of all EMTs of which X is an ancestor. Taken together, root lookup table 67 and last offspring table 68 may effectively provide complete structural descriptions for all email threads edt1 . . . edtn.
Thread type determiner 65 may then analyze the data in root lookup table 67 and last offspring lookup table 68 to determine whether a group of EMTs belonging to a thread edti form a linear thread lti or a conjoined thread set cti. Thread type determiner 65 may compile this information in thread type lookup table 69 in which the value “L” (linear) or “J” (conjoined) may be assigned to each Root docID in root lookup table 67. Thread type determiner 65 may identify that a root docID is the root of a linear thread if the root docID is common to a group of consecutive docIDs which also share a common Last Offspring docID. Thread type determiner 65 may identify that a root docID is the root of a conjoined thread set if the root docID is common to a group of consecutive docIDs which have differing Last Offspring docIDs.
It is noted that the thread type data may be joined, as shown by arrow 61, to root lookup table 67 so that each docID in the system may be identifiable as belonging to a linear thread or to a conjoined thread set.
Compact EMT compiler 66 may compile compact EMT volume 56 by associating, with each assigned docID, the meta-data and the new text content associated with that docID.
Exemplary compact email volumes, 70 and 72, for exemplary linear and conjoined email threads lt1 and ct1 respectively, are shown in
The reduction in volume realized by compact EMT compiler 66 may be seen by comparing the original volumes of email threads lt1 and ct1 and compact volumes 70 and 72, respectively, represented graphically in
Detailed representations of compact email volumes 70 and 72 are shown in
Similarly, compact email volume 72 for conjoined thread ct1, as shown graphically in
Reference is now made to
A cursory review of these four tables may indicate how the structures of both linear and conjoined threads, and the distinction between them, may be completely described by the combination of root lookup table 67 and last offspring lookup table 68. Out of a volume of random emails 50, a thread may be distinguished as a group of emails having the same root email, such as emails 10, 11 and 12 all sharing root email 10 as shown in table 74 of
Last offspring lookup table 68 may provide the information necessary to distinguish linear threads from conjoined thread sets. A group of consecutively numbered emails which share their root email and their last offspring email is defined as a linear thread. As shown in
Reference is now briefly made to
As explained previously in the discussion of
Reference is now made to
The value recorded in the second field indicating the location of the token in the docID may refer to a word count position in the meta-data of the EMT if the value in the third field is an “M”, or to a word count position in the content of the EMT if the value in the third field is a “C”.
Reference is now made to
In
Reference is now made to
Query manager 46 may accomplish the task of guided systematic and selective searching of EMT volume 50, by beginning at a certain start point, i.e., by selecting one message candidate, and then, in an iterative process, using the rules and conditions prescribed in the algorithms disclosed hereinbelow to validate or disqualify that candidate, and to determine the next candidate to be checked. Query manager 46 may conserve system resources by skipping around message volume 50 to select candidates as dictated by the thread-savvy algorithms.
In accordance with a preferred embodiment of the present invention, query manager 46 may conduct a selective, time and resource efficient search of a message volume 50 as described above without any loss of recall, that is, without failing to retrieve qualifying results to a query due to the shortcuts taken in the indexing and search processes.
Returning now to
As shown in
Candidate enumerator 122 may employ postings iteration manager 120 in an iterative process in which, at any given time, one docID, the CandidateMessage, may be under consideration for qualifying as a search result for query 52. Postings iteration manager 120 may traverse posting lists PL1 . . . PLn of required terms R1 . . . Rn, optional terms O1 . . . On and forbidden terms F1 . . . Fn of query 52. Candidate enumerator 122 may maintain three message pointers—CandidateMessage, CandidateRoot, and LastOffspring. The pointer CandidateMessage may point to the document ID docIDi under consideration for candidacy as a response to query 52 at any given time. The pointer CandidateRoot may point to the root docID of docIDi, and the pointer Last Offspring may point to the last offspring docID of docIDi. The iterative process of candidate enumeration and examination for qualification performed by CE 122 and PIM 120 is discussed in greater detail with respect to
A message meeting the search criteria, i.e. a qualifying candidate QCi, may proceed to candidate scorer 124 for scoring. Qualifying candidates QC1 . . . QCn and their associated scores, ScQC1 . . . ScQCn may proceed to candidate ranker 126 for ranking.
Candidate scorer 124 may assign scores to qualifying candidates QC1 . . . QCn, with ‘better’ candidates receiving higher scores. For example, a candidate email containing a certain number of optional terms O1 . . . On may be assigned a higher score than a candidate email containing a lesser number of optional terms O1 . . . On. Candidate ranker 126 may assess the candidate scores on the basis of the retrieval policies of search engine 40, and may assess which qualifying candidates may be returned by the search engine as results to query 52. The retrieval policies of search engine 40 may be system or user defined.
Reference is now made to
In the example shown in
Candidate enumerator 122 may then access root lookup table 67 to determine the root of the candidate message (step G3 in
In a preferred embodiment of the present invention, candidate enumerator 122 may consult root lookup table 67 to ascertain whether the candidate message is part of a linear or conjoined thread (step G4 in
As shown in
The two conditions may be:
These conditions imply that when the postings iterator of a posting list of a required term PIRi is on a message with docID X, all other posting lists may be safely advanced to a position at or beyond Root [X] without any loss of recall.
Returning to the example shown in
Candidate enumerator 122 may then consider occurrence 156, of required term R3 in docID 105, as located by postings iterator PIR3. However, since occurrence 156 is a meta-occurrence, it must fulfill condition La, and since the candidate message under consideration is docID 115 and not docID 105, condition La is not satisfied. Condition Lb is also not satisfied by occurrence 156 because condition Lb requires a content occurrence and not a meta occurrence. Postings iterator PIR3 may then advance to occurrence 158 of term R3 on PLR3. This occurrence is found to satisfy condition Lb because it is a content occurrence at docID 105 falling in the required range of 100 to 115.
Returning now to
During step L3, CE 122 may determine if the candidate message under consideration is limited to CandidateMessage, or if all docIDs in the range {CandidateMessage . . . LastOffspring[CMsg]} may also be candidates. The last offspring of the candidate message, LastOffspring[CMsg] may be determined by CE 122 either by accessing last offspring lookup table 68, or by finding the highest docID in root lookup table 67 which has the same root as CandidateMessage. Referring briefly to
Returning now to
CE 122 may then proceed to verify (step L5) the absence of forbidden terms in CandidateMessage or in the range {CandidateMessage . . . LastOffspring[CMsg]}. CE 122 may examine the occurrences of all forbidden terms F1 . . . Fn which fall within the range [CandidateRoot, . . . CandidateMessage] to determine if CandidateMessage or ranges of messages in the thread to which CandidateMessage belongs may be disqualified for containing forbidden terms. Candidate enumerator 122 may disqualify candidate messages according to the following rules:
Candidate enumerator 122 may proceed to step L4 and choose a new candidate message if all candidate messages are disqualified in step L5 for containing forbidden terms. Qualifying candidate messages not disqualified for containing forbidden terms may proceed to candidate scorer 124 (step G5).
Candidate scorer 124 may assign scores to qualifying candidates on the basis of all the occurrences of query terms R1 . . . Rn and O1 . . . On in the message by iterating through query term occurrences in the range [CandidateRoot, . . . CandidateMessage] of all term posting lists. All content occurrences in the range may contribute to the score of a qualifying candidate QC, but meta occurrences may only contribute to the score of a qualifying candidate if they occur in the qualifying candidate itself.
Once candidate enumerator 122 verifies a qualifying candidate or candidates, CE 122 may proceed to step L6. In step L6, candidate enumerator 122 may choose the next candidate message in consideration of the retrieval policy of search engine 40. For example, the retrieval policy of search engine 40 may dictate that it is sufficient to return as search results, only the first message in each thread which satisfies query 52. In this case, after scoring one qualifying candidate, candidate enumerator may start searching for the next candidate beginning from the next thread, i.e., candidate enumerator 122 may skip all messages in the thread of the current qualifying candidate.
Alternatively, if the retrieval policy of search engine 40 dictates that all relevant messages in the thread should be returned as search results, the search may be continued from CandidateMessage+1. This method may allow search engine 40 to easily identify the highest scoring message of the thread—all thread candidates may be enumerated sequentially, since they may be indexed with consecutive docIDs. Furthermore, scores for successive docIDs in a thread may be computed simply by candidate scorer 124 on the basis of the scores of preceding docIDs, that is, according to a methodology based on the patterns of the email thread, in the following way: It is assumed that the score S(k) of message k was just computed by candidate scorer 124, and that the next CandidateMessage may be k+j. Since the content of each message may be fully contained in the text of the following messages, the score S(k+j) of message (k+j) may equal to:
S(k+j)=S(k)+CS(k+1, . . . , k+j)−MS(k)+MS(k+j)
where CS indicates the score contributed by content occurrences and MS indicates the score contributed by meta occurrences.
Returning now to step G4 in
The two conditions may be:
These conditions imply that when the postings iterator PIRi of a posting list PLRi of a required term Ri is on a message with docID k, all other posting lists may be safely advanced to a position satisfying the following two conditions without any loss of recall:
CE 122 may then proceed to step J4 if it determines that every required term of query 52 meets one of conditions Ja or Jb. CE 122 may proceed to step J5 and choose a new candidate message if at least one required term of query 52 does not meet condition Ja or Jb.
During step J4, CE 122 may determine if the candidate message under consideration is limited to CandidateMessage, or if all docIDs in the range {CandidateMessage . . . LastOffspring [CMsg]} may also be candidates. CE 122 may consider all docIDs in this range if condition Jb in step J4 applies to all posting lists (i.e., a match with no meta-occurrences has been identified). In this case it may be guaranteed that all messages in the thread whose docID is no larger than LastOffspring [CandidateMessage] may also be candidates.
CE 122 may also use the following rule to improve searching efficiency: If an occurrence of a query term in docID X satisfies CandidateRoot<X<CandidateMessage but LastOffspring[X]<CandidateMessage, the posting list of the term may be advanced to the docID numbered LastOffspring[X]+1.
Candidate enumerator 122 may then proceed (step J6) to verify the absence of forbidden terms in CandidateMessage or in the range {CandidateMessage . . . LastOffspring[CMsg]}. CE 122 may examine the occurrences of all forbidden terms F1 . . . Fn that fall within the range [CandidateRoot, . . . CandidateMessage] to determine if CandidateMessage or ranges of messages in the thread to which CandidateMessage belongs may be disqualified for containing forbidden terms. Candidate enumerator 122 may disqualify messages according to the following rules:
CE enumerator 122 may proceed to step J5 and choose a new candidate message if all candidate messages are disqualified in step J6 for containing forbidden terms. Qualifying candidate messages not disqualified for containing forbidden terms may proceed to candidate scorer 124 (step G5).
Candidate score assessor 124 may assign scores to qualifying candidates on the basis of all the occurrences of query terms R1 . . . Rn and O1 . . . On the message by iterating through query term occurrences in the range [CandidateRoot, . . . CandidateMessage], of all term posting lists. All content occurrences in docIDs whose LastOffspring≧Candidate Message may contribute to the score of the qualifying candidate, but meta occurrences may only contribute to the score of the qualifying candidate if they occur in the qualifying candidate itself.
Once candidate enumerator 122 verifies a qualifying candidate or candidates, CE 122 may proceed to step J7. In step J7, candidate enumerator 122 may choose the next candidate message in consideration of the retrieval policy of search engine 40. For example, the retrieval policy of search engine 40 may dictate that it is sufficient to return as search results, only the first message in each thread which satisfies query 52. In this case, after scoring one qualifying candidate, candidate enumerator may start searching for the next candidate beginning from the next thread, i.e., candidate enumerator 122 may skip all messages in the thread of the current qualifying candidate, and begin searching at the docID numbered LastOffspring[CandidateRoot]+1.
Alternatively, if the retrieval policy of search engine 40 dictates that all relevant messages in the thread should be returned as search results, the search may be continued from CandidateMessage+1. This method may allow search engine 40 to easily identify the highest scoring message of the thread set—all thread candidates may be enumerated sequentially, since they may be indexed with consecutive docIDs.
For scoring, operations may depend on whether CandidateMessage+1 refers to CandidateMessage, (i.e., CandidateMessage is an ancestor of CandidateMessage+1). If CandidateMessage+1 refers to CandidateMessage, scores for successive docIDs may be computed simply by candidate scorer 124 as described previously for linear threads.
However, if the last scored CandidateMessage has no offspring, the CandidateMessage counter may be advanced by one, and all posting lists may be set to the root of the new CandidateMessage, in which case some posting lists may be rewound.
Alternatively, search engine 40 may follow a hybrid approach and may return one message per qualifying thread in the thread set. This may be achieved by advancing the next candidate to LastOffspring[CandidateMessage]+1.
In an additional preferred embodiment of the present invention, illustrated in
Objects and processes illustrated in
Virtual cursor layer 130 may operate as if it were superimposed over the posting iterators within postings iteration manager 120. Virtual cursor layer 130 may thus provide knowledge of the thread structure of the EMT volume being queried, as described by the data stored in thread management database 43, which may include root lookup table 67 and last offspring table 68 (
The employment of virtual cursor layer 130 may isolate the query process from the EMT volume thread structure model, and may thus allow the EMT volume thread structure model to be used in a query process while the query process may remain oblivious to the details of the thread structure and its implementation in candidate enumeration. This embodiment of the present invention may thereby allow thread-aware searches to be conducted within query processes which have not been specially adapted to this particular type of search method.
Virtual cursor layer 130 may be provided by creating a “positive” virtual cursor for each required term, and a “negative” virtual cursor for each forbidden term in query 52. Algorithms may be provided for the positive and negative versions of two basic cursor methods next ( ) and fwdBeyond ( ) as well as for the method fwdShare( ). These algorithms, PositiveVirtual::next ( ), PositiveVirtual::fwdBeyond ( ), NegativeVirtual::next ( ) NegativeVirtual::fwdBeyond ( ) and Physical::fwdShare ( ) may dictate the movements of the virtual cursors and the underlying posting iterators, thereby enumerating candidate EMTs. Pseudocode for the algorithms is shown in
Algorithms 140, 142, 144, 146 and 148 shown in
In the algorithms shown in
Algorithm 142, for positive fwdBeyond(d), as shown in
Algorithm 144, for negative next ( ), as shown in
Algorithm 146, for negative fwdBeyond(d) forwards the virtual cursor to the next docID at or beyond docID D that does not contain the term Ti. As shown in line 6 of algorithm 146 fwdShare(d) is called to position CP on the next docID which shares term Ti with docID D. Then, as shown in line 14, next( ) is called to position the virtual cursor on the next document that does not contain term Ti.
Algorithm 148, for fwdShare(d) strives to forward the physical cursor so that it shares term Ti with docID D. If there is no such document, it returns with the cursor positioned on the first docID beyond D. This is accomplished, as shown in line 1 of algorithm 148, by looping until the physical cursor moves beyond D or to a posting that shares term Ti with docID D. The movement of the physical cursor depends on where the cursor lies. As shown in
In the additional preferred embodiment of the present invention illustrated in
While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes, and equivalents will now occur to those of ordinary skill in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.
This application is a divisional of U.S. application Ser. No. 12/484,213, filed 14 Jun. 2009, which is a continuation of application Ser. No. 11/200,969, filed Aug. 10, 2005 (now U.S. Pat. No. 7,565,347).
Number | Name | Date | Kind |
---|---|---|---|
4496173 | Roche et al. | Jan 1985 | A |
6167402 | Yeager | Dec 2000 | A |
6789083 | Thelen | Sep 2004 | B2 |
7062772 | Underseth et al. | Jun 2006 | B2 |
7383274 | Pearce et al. | Jun 2008 | B2 |
8131726 | Broder et al. | Mar 2012 | B2 |
20020107877 | Whiting et al. | Aug 2002 | A1 |
20020199096 | Wenocur et al. | Dec 2002 | A1 |
20030135495 | Vagnozzi | Jul 2003 | A1 |
20040044735 | Hoblit | Mar 2004 | A1 |
20050240558 | Gil et al. | Oct 2005 | A1 |
20060212457 | Pearce et al. | Sep 2006 | A1 |
Number | Date | Country | |
---|---|---|---|
20120259834 A1 | Oct 2012 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 12484213 | Jun 2009 | US |
Child | 13494044 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 11200969 | Aug 2005 | US |
Child | 12484213 | US |