Patent Application
20050055340
The increasing need for finding relevant data over the Internet has produced a number of categories of data searching techniques and technology over wide area networks and in particular the Internet. Many of these techniques are included in patents and publications provided by well known industry leaders in the Internet searching business including Google™, Northern Light®, and Inktomi® (used by Yahoo!®). Various aspects of these techniques will be discussed below.
With more than 4 billion Internet sites in existence, the problem of developing an effective search engine is paramount. Even though some searching techniques may provide for effective cursory searching based on input terms, the information returned to the user may still be inadequate for guidance, because of the layers of information under an entrance page. For example, a large institution such as a government, corporation, or non profit organization may easily have more than 100,000 pages or documents on one single top-level domain uniform resource locator (URL) and at least a few thousand under a single sublevel.
As can be appreciated by those skilled in the art, the instructions for searching for specific information over a large network with a limited data set, such as on a single institutional site may have different structural and architectural characteristics than instructions for searching over a nearly indefinite number of Internet pages. Attempts to organize this information may be the product of many interdisciplinary technologies ranging from library science to electrical engineering to archival taxonomy.
One very popular method for data mining, is the “scoring” method. Google, Inc. of Mountain View, Calif. has several published U.S. Patent Applications including 2001/0123988 entitled “Methods and Apparatus for Employing Usage Statistics in Document Retrieval” by Dean et al. and 2001/0133481 entitled “Methods and Apparatus for Providing Search Results in Response to an Ambiguous Search Query.” Both of these patent applications are hereby incorporated by reference in order to illustrate the background to the present invention.
As can be appreciated, one of the drawbacks of the “scoring” method is that like any statistical method, it can be artificially “skewed” by either a disproportionate group of users or other manipulable technique. Mechanisms can be put into place to account for these factors, the technological advances and otherwise “skewable” techniques. For example, U.S. Pat. No. 6,269,361 issued to Davis, et al. and assigned to GoTo.com of Pasadena, Calif. describes such a technique for influencing a place in the list of a search engine. As needed to detail the problem of influencing search results, this document is hereby incorporated by reference.
Google® owns other technology related to data searching techniques. For example, a recently issued U.S. Pat. No. 6,526,440 entitled “Ranking Search Results by Reranking the Results Based on Local Interconectivity by Krishna Bharat teaches the use of connectivity to determine “relevance.” However, these results are subject to “statistical” problems, although it may require an immense “effort” on the part of any single unsavory entity to intentionally skew such data in its favor. For example, a single URL, used by an entity and of particular usefulness (i.e, relevance) to the majority of people may be overtaken by an entity's URL that uses many different URLs to connect to that link, allowing manipulation by entities who may benefit from the use of “click-throughs,” mainly the sale of advertising space or pop-up screens. FIGS. 1A-C illustrate some of the various searching techniques used by this entity.
As such, scoring techniques for finding relevant documents can learn only by statistical inferences and connectivity and require a manual detection of manipulations or irregularities. For example, many URLs can point to a single site or page, which can skew the “popular” use of the statistic. Furthermore, it is assumed that “relevance” for looking for a document begs the question as to “whom is it relevant to?” The above-described methods may be useful for persons looking for the result “relevant” to a majority of people or even a well defined subset of persons. However, users with unusual profiles or searching techniques may be excluded from effectively using these methods in looking for relevant documents over the Internet. The importance of relative criteria in searching the Internet for relevant information is not just a philosophical question, but lends itself to very practical concerns about the heuristics of the search.
There are other types of intelligent searching techniques that attempt use principles of artificial intelligence as they apply to natural language processing. U.S. Pat. No. 6,430,551 by Thelen et al. and assigned to Phillips Electronics of the Netherlands, uses pattern recognition techniques, as such
Neural networks are both a conceptual framework and a practical computing application developed in the attempt to teach computers how to model brain functioning (or other biological models) in the areas of pattern recognition of speech and vision processing. The concept of neural network computing originally applied to pattern recognition studies. The concept of neural computing requires that “rules” generated by a high level structure (such as a brain) are implemented at the “nerve” level (or the data input) to process the incoming data properly. Training mechanisms for the use of neural networks over the Internet for use in analyzing financial market data include U.S. Pat. No. 6,247,001 entitled “Method of Training a Neural Network” by Tresp et al. currently assigned to Siemens of Munich Germany, and hereby incorporated by reference.
Another adaptive intelligence mechanism applied to complex computing problems is the genetic algorithm. Genetic algorithms are components of larger computing solutions (i.e. a larger algorithm) that are usually able to adapt and combine in other algorithms. Genetic algorithms are known to those skilled in the art for various purposes, and their description may be referenced by any number of textbooks on the subject, including Introduction to Genetic Algorithms, by Melanie Mitchell (MIT Press 1996), which is hereby incorporated by reference for purposes of teaching the implementation of genetic algorithms or components. Such algorithms are also taught in U.S. Pat. No. 6,182,057, which is hereby incorporated by reference.
Bayesian logic is also referred to as fuzzy logic, which has been the focus of many types of intelligence-based computing for a couple of decades. In its most simplified form fuzzy logic is a technique for defining members of sets based on contingent and relative variables. Fuzzy logic therefore plays crucial roles in machine learning techniques where adaptation is required. The use of multiple intelligence computing techniques simultaneously has been discussed in the recent literature. The concept of the neuro-fuzzy and/or fuzzy-neuro systems is discussed at length in Fuzzy Engineering Expert Systems with Neural Network Applications by A. B. Badirui and J. Y. Cheung (John Wiley & Sons, 2002) and Soft Computing: Integrating Evolutionary, Neural and Fuzzy Systems, by A. Tettamanzi and M. Tomassini (Springer 2001). These two references are incorporated by reference in order to teach the various techniques of developing and configuring neural networks, fuzzy logic, genetic algorithms and expert systems in general. Some textual references have noted that neural networks may not be good for searching algorithm applications mainly because neural network rules are implemented a low levels, which may be impractical with data input as complex as natural language expressions, which are typically used in an internet search. Such a concept is discussed in Evolutionary Algorithms for Data Mining, by Alex Freitas, Springer, 1998, p. 4, which is hereby incorporated by reference.
An example of multiple use of artificial intelligence techniques over networks is described in U.S. Pat. No. 6,327,550 entitled “Method and Apparatus for System State Monitoring Using Pattern Recognition and Neural Networks” by Vinberg et al. and currently assigned to Computer Associates Think, Inc., of Islandia, N.Y. The Vinberg reference teaches the use of state vectors as they would be applied to networks. Other interactive multiple intelligence mechanisms are described in U.S. Pat. No. 5,249,259 (“Genetic Algorithms for Designing Neural Networks”) and U.S. Pat. No. 5,727,130 (“Genetic Algorithm for Constructing and Tuning Fuzzy Logic System”) neither of which teachers multiple interactive for data mining over networks per se. Both of these documents are incorporated by reference. However, none of these multiple intelligence node systems is particularly well suited for use in a search processing system over the Internet to find relevant documents or pages.
The present invention provides solutions to the above-listed shortcomings by providing adaptive structures, such as fuzzy logic and genetic algorithms or modules to a neural network architecture in order to improve the capacity and trainability of the neural network for computing a relevant search result based on a large set of search criteria. By allowing the search criteria to be processed in a neural network, the system of the present invention can process information that would normally be too computationally complex to resolve.
The present invention is particularly effective at minimizing the organization and processing of massive amounts of data in order to find appropriate resources (i.e. documents or pages) in reponse for a search inquiry. One of the advantages of the present invention is that particular rules and application may be applied at several different levels to reduce the search and computing time. For example, the fuzzy neurode implements two complementary technologies at the lowest level and may prevent the processing of massive amounts of irrelevant information at the computational level. The adaptive genetic components may detect particular successful or unsuccessful searching configurations of the neural network and combine with other searching configurations where similar patterns have been detected. Finally, fuzzy logic and computation rules based on prior search results, user and situational data and manual or automated feedback mechanisms serve to teach the intelligence components of the present invention more efficient and accurate searching mechanisms.
The present invention can be better understood by the following diagrams and illustrations. However, as can be appreciated by those skilled in the art, the components of the present invention may be implemented in a variety of forms including virtual and physical as well as implementing what appear in the drawings as single units on multiple computing devices. Thus, the drawings are not meant to be limiting, but are provided for better understanding of the components and the interactions between the components.
FIGS. 1A-C represent prior art examples of search engine techniques on documentscoring systems, document accesses or links.
The present invention takes advantage of a virtual or actual neural network data searching system combined with the additional artificial techniques of using expert rules and fuzzy logic in search operations conducted over a large body of data collected from the Internet or other WAN. The present invention takes advantage of the power of the neural network in order to process higher level searching constructs instead of simple inputs. However, by processing data through a neural network on complex searching constructs, the system can provide many advantages in providing accuracy and customization.
The present invention must be able to access a large pool of data collected from the Internet. Because these large pools of data are commercially available, it is expected that in a preferred embodiment of the invention that this data is purchased from a third party. Referring now to
Referring now to
The data resource module(s) 50 are accessed by a search processing system 100 through a series of actual and virtual connections 55 which may be through any number of communications links such as T1, Ethernet, DSL, etc. However, in alternate embodiments of the invention, this access may be virtual where the data is simply duplicated in a more accessible location, such as where the search processing system 100 is located. The virtual duplication 50′ of the data resource module 50 in another location is shown in
Referring now to
Referring now to
The above structures are described as virtual structures even though they may be physically embodied in a specific device or in separate computer readable mediums. As can be appreciated by those skilled in the art, the modular descriptions of the various structures or components allow for an understanding of the computational architecture of one of the embodiments of the invention. Furthermore, there is no requirement that any one module be executed by a single computer or that all the modules be on the same computer. Neural network processing often benefits from parallel processing, which can include parallel processing on one device or multiple devices. In fact, throughout the specification the structures may be implemented in a virtual fashion. Those skilled in the art will readily recognize that there will be advantages to various implementations of the present invention. For the sake of simplicity, in a first embodiment and the examples illustrated all the modules will be located and executed on a single computational device.
Although it will not be discussed further, the components of the search processing system are stored and implemented on at least one computation device 102, which will most likely have storage or access to storage of a variety of different types. The details of the one or more computation device 102 on which the search processing system 100 is implemented are not particularly important to the present invention unless there are details which would affect the performance of many of the inventive steps and structure which are described below. It can be assumed that all the components of the search processing system 100 are executable on the one or more computational devices 102 and that data and instructions between components and modeules of the system 100 are shared through communication mechanisms included in the computational devices 102. These can be internal busses, external communication structures such as T1, Ethernet, wireless LAN, virtual data sharing, internal or external parameter passing in programming languages, access to a common internal or external databases among other communication and/or data sharing mechanisms.
Referring now to
As can be appreciated not all levels are necessary for the operation of the present invention. Although the collection of a large array of data allows the neural network to function optimally over the course of a large number of searches, in addition to developing learning rules which may apply at both low and high levels.
Similarly the neural network nodes 110(l) . . . 110(n) receive the appropriate information from the weighted neurode or set of neurodes via an “axon” even though the nodes 110(n) may be part of the same executable instructions as the neurodes 101(n) which gather the data. The discrete nature of these structures is useful in implementing the multiple AI processes involved in the present invention as may be appreciated by those skilled in the art.
Referring now to
The relevant preliminary search result is then generated in step 1175 from the neural network 120, which receives data from (or has already “learned” from) the low-level neural input in step 1150. The preliminary search result is subject to a high-level modification from the expert rule 140 and fuzzy logic 160 modules in step 1178. In step 1180 the search results are delivered to the user through the interface 180. Simultaneously, any data for learning instructions is generated in step 1190. Learning by the search processor 100 is described in detail below. It is anticipated that step 1178 will become decreasingly necessary for each time the learning instructions are generated in step 1190. The application of expert rules and fuzzy logic at the low level in step 1150 saves considerable computational resources over applying them at higher levels. The process of generating a search result is described below, but as can be appreciated by those skilled in the art, may be executed in many different ways without departing from the spirit and scope of invention.
Parametric control (including user) data 510(i) . . . 510(n) will generally be macro level data that defines the behavior of the entire search engine. The parametric data may be based on an individual user's preferences or conditions that may be easily determined by the interface 180. This data may include items entered by the user such as financial situation, content preferences, geographic location, etc. Automatic parametric data may include weather, stock market results, the particular user of the interface, detected inquiries to the user's credit card and any number of variables which may influence the manner in which the search may be conducted. The table below helps define one aspect of the present invention.
The “spider review” shown above is then an effective way to describe a summary for samples of the “neural input” for the present invention. However, the list in the above table is by no means exhaustive, but meant to be illustrative only. As can be appreciated by those skilled in the art, the advantage of using a neural network to get data at such a low level is in representing fairly complicated search constructs in a large number of standardized, normalized or standardizable data inputs for processing. In the table above there at least 53 spider nerve inputs and 12 human review inputs.
Human review input, such as design quality and content issues that may be computationally difficult to calculate from the neural network may be stored as data in each of the modules. As more data is generated by the search processing system 100, the computer will be able to apply the human rules to its own learning generated from the data and will also learn other rules on its own. For example, a pattern recognition algorithm may apply to a URL with a large amount of pop-up advertising although undetectable by the search system 100. The common characteristics or “neural patterns” from the spider review will alert the system that such patterns correspond to the same one as the human-reviewed URL with a large amount of advertising.
FIGS. 11A-D show the functions of the neurodes at the data input level or neurode level 105.
The general process of providing feedback through the expert rules module 140 or the fuzzy logic module 160 is shown in
In
As can be appreciated by those skilled in the art,
However as can be appreciated by those skilled in the art, parameters may or may not need to be relevant in all cases, the network accommodates this and determines predictability in a massively parallel distribution of search knowledge. In such a case where a parametric or user data is not initially deemed to be relevant the search processing ignores one or more pieces of parametric data 510(n) based on search criteria and applied rules in the expert rule module 140. This acts as a pre-search fuzzy set, i.e. the set of parameters used in the search is limited by the “category” of the search.
However, as can be appreciated by those skilled in the art, even 20 or 30 neural input parameters with 3 states each may quickly become unmanageable computationally complex and inaccessible. The advantage of the fuzzy logic being located at the neural input 101(n) or 110(n) is processed is that control over the computational aspects of potentially massive amounts of input data. Search inputs 300 which is particularly sensitive to certain parameters 510(n) can be adapted to become a fuzzy neurode, instead of the neuro-fuzzy processors. The process of applying a set of expert rules in the expert rule module 140 is shown by
Referring now to
The neural network processor 120 allows for the inclusion of personal data in the decision process like previous consumer behavior to add predictive ability to what a relevant search return would be. The personalization of the personal data for determining relevancy is a key enhancement compared to current art, which for the most part return globally relevant returns and acts to enhance the machine intelligence self-training. For example, a person in China searches for Soy Sauce and a Restaurateur in Manhattan searches for Soy Sauce. The returns will be quite different because the search processing system 100 via the neural network 120 recognizes important determinants of relevancy for each individual searcher from the application of the expert rules based on these parameters. As stated above, the effect of thee personal and parametric data 510(n) may be processed at multiple levels either directly the neural network “neurodes” (low-level), the function gates (mid-level), or post neural processing (high-level). Thus, in the above example, items in the neural network which respond to the “geographic” neurode, which would be “reweighted” based on personal geography, for a low-level implementation.
The search processing system 100 of the e present invention is melded with human review, spiders, genetic (sub)algorithms, fuzzy inference engines and expert systems which are comprised of sets of adaptable expert rules. The sets of rules applied by the expert rule module 140 may be preliminary global rules which are rules that are still being adapted. There may also be global rules which are rules which are the product of many adaptations and have been tested. Expert rules or subsystems may be implemented at multiple levels. An example of this is where an expert (sub)system determines the presence of documents which result in spam. Anti-spam parameter 510(n) will resulting the expert system being loaded into the fuzzy logic module and applied at a low-level input so that data on documents which results in spam is not processed by the search engine at the neural network level.
In part 2, the fuzzy logic module sends weighing instructions to neurode inputs N(1)—N(4) either based on instructions from the search engine or the relevant parameters 510(1) and 510(2). Of course, the search system 100 could associate the two parameters 510(1), 510(2) (market conditions and income) with search inquiry terms “risk-free” and “bond” or “funds.” As can be appreciated by those skilled in the art, association of information may lead to reduced processing time by eliminating nodal information that may not be particularly relevant. For example, in the automated spider review in Table 1.1, holiday input may not be particularly weighted with importance while searching for financial services (on the other hand, the expression Easter may be related to tax season).
In part 3 documents are compared to an itemized truth table with scoring 299 from a previous search which is further put together with relevant human input data for category H(4), that is a “reputable or authoritative site” (see Table 1.1 below) in this example. Thus, the summation or weighted scoring may be one mechanism to determine appropriate search results, but the H(4) criteria in this case overrides the scoring and will not allow for high scoring matches which are not authoritative. Thus, a “target score” match 399 is made by the machine learning mechanism and noted for future use of “225” or “175” but not the “200” score. Furthermore, from part 3, the machine learns that H(4) is generally present when N(3) and N(4) signals are present, but N(1) appears to be less relevant and no positive N(2) data was returned which met the threshold condition. The search results may be presented to the user by high score and reputability in Part 4. However, in part 4, the search processing system “learns” that neurode input N(3) and N(4) are likely indicators of this human input attribute H(4) and adapts such learning for the next appropriate search task and a preliminary global rule may be put into the expert rule module 140. In step 5, for the next search of the type the presence of N(3) and N(4) will be give larger weights or N(1) may also be reduced in weight. Thus repeated learning of this type will nearly eliminate N(1) as relevant. However, the system eliminates N(2) from the neural connection for the next search of this type.
The above table is illustrative of the human input and expert rules as implemented in the present invention in a particular embodiment. These are only illustrative to the example. The expert rules are nonflexible during an intrasearch criteria applicable to the computation of the neural input although these expert rules may be detected at the input node 105 or function gate level 115 as well. However, as shown above, the expert rules are clearly adaptable in the machine learning system of the invention.
FIGS. 24A-D show sample feedback mechanisms for learning.
Similarly in
FIGS. 25A-C are illustrative of a simple examples genetic algorithms as they may be adapted or combined as the result of machine learning. These algorithms may be global to the whole search system 100 or used only by one component They are virtually stored in virtual storage 198 on one or more computing machines 102, such that they may be accessed by any component of the search system 100.
However many health related sites will tag certain pages as adult content in order to allow for sensitivity to the marketplace. Thus, the adult content blocker is able to learn manually from human input or from machine learning that the adult content neurode is not accurate for this user's purposes. The genetic algorithm determines that a health related neural input or a human input of reputable site will negate the effects of the adult content blocker for the search term “breast.” However, the algorithm may apply to other search terms that will draw both adult and non adult content. Thus, the genetic algorithm will adapt the neural input in addition to combining with other search algorithms which may apply to a common category of expression for anatomical parts or the genetic component may adapt through a neural pattern recognition.
The machine cannot innately understand that “string” “strang” and “strung” are all related terms. Thus it has reduced the importance of the 4th position (the vowel) more than half after the third search. However, the search term “strong” is not related and returns only 25% overlapping search results R4. Thus, the machine learns that the 4th position neurode is now more relevant and adjusts the weight by a factor of 1.3. The search “streng” returns no results, boosting the weight by 1.6 and “stryng” returns only R6 which allows for a weight factor of 1.1. So at the end of the six searches, the 4th position neurode is only a bit less than 1 (0.96), which indicates that the search “STR” X “NG” will result in the 4th position being slightly less of a search input factor. The learning mechanism may have enough data on this search type that it makes the preliminary global rule of the weights, a global rule.
Table 1.1 shows a series of example “spider level” search which may be implemented in as neural input from the data collection system 50. As can be appreciated, the existence of data collection services which may be purchased for use with the present invention or generated by the categorized different than the examples in table 1.2 provide. However, the 50 or so criteria described may provide for an example of types of search criteria would be processed as neural input adapted by learning mechanisms.
The actual calculation of the search results would be highly dependent on the particular implementation of the invention. Certainly a “score” based on weights of the neural inputs would be one embodiment of the invention. Of course, the advantage of the present invention over the prior art is the fuzzy logic or expert rules that may be implemented at different levels. Thus, a weighted score based on the neural input may adapt by a number of fuzzy mechanisms at a number of computational points.
A “score” from the generation of a result from a neural network is not an accurate description, however, pattern recognition (discussed below) which is the predominant computational solution in many neural networks may not be appropriate.
There is redundancy in the predictive models provided in the present invention. This means that the search processor tolerates absent and poor data very well compared with the prior art. As can be appreciated the minimally tolerated input for returning an accurate search answer will be reduced by the machine learning over time.
The above examples and embodiments are meant to be illustrative only and are not exhaustive. As can be appreciated by those skilled in the art, many of the structures described can be virtual or physical and combined in one machine or several. Furthermore, the modularity of any given component must be appreciated. For example, three (or any number of) neurodes may be combined into a single one if the AI search processor determines that it is appropriate under the circumstances. Thus, the scope of the invention should not be limited to the example provided above, but rather to the spirit of the invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2,395,905 | Jul 2002 | CA | national |
This application claims priority under 35 U.S.C. 119(e) to U.S. Provisional Patent Application ______ filed Mar. 3, 2003, entitled NEURALLY-PROCESSED SEARCH ENGINE WITH FUZZY AND LEARNING PROCESSES IMPLEMENTED AT MULTIPLE LEVELS by Scott Dresden, which is hereby incorporated by reference in its entirety for all purposes.
| Number | Date | Country | |
|---|---|---|---|
| 60451237 | Mar 2003 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 10390590 | Mar 2003 | US |
| Child | 10664787 | Sep 2003 | US |
