Patent Application
20240119095
A computer-implemented search engine conventionally receives a query from a client computing device operated by user, searches a computer-readable index of items (webpages, images, videos, etc.) based upon the query, identifies items from the index that are relevant to the query, ranks the items to create a ranked list, and returns a search engine results page (SERP) to the client computing device, where the SERP includes the ranked list of items. While search engine technology is continuously improving with respect to inferring intent of users based upon queries issued by the users, a conventional search engine is limited to identifying items that have been indexed and/or stored by the search engine based upon a user query. When the search engine receives a query set forth by a user that accurately reflects an information retrieval intent of the user, but the search engine has not index and/or stored items that are relevant to that intent, the search engine either returns nothing to the user or returns results that are irrelevant.
The following is a brief summary of subject matter that is described in greater detail herein. This summary is not intended to be limiting as to the scope of the claims.
Described herein are various technologies pertaining returning search results to a user in response to receipt of a query from the user, where the search results can include both: 1) items that have been previously indexed and/or stored; and 2) content that has been generated by a computer-implemented model based upon the query. With more particularity, a computing system receives a query from an application executing on a client computing device that is in network communication with the computing system. The application can be a web browser, a mobile search engine application, a productivity application (such as a word processing application, an email application, a spreadsheet application, a slide presentation application, etc.).
Based upon the query, the computing system searches a computer-readable index of items to identify items that are related to the query. The items can be webpages, images, videos, etc. The computing system can identify an item indexed in the computer-readable index based upon the query. In addition, the computing system can perform operations related to generating content based upon the query (where, in the context of content generation, the query may be referred to as a prompt). In an example, a search engine results page (SERP) includes the item identified by the computing system as being relevant to the query and a selectable button that indicates that content can be generated based upon the query. The computing system transmits the SERP to the client computing device, and upon receiving an indication that a user has selected the selectable button in the SERP, the computing system provides the query as input to a computer-implemented model that generates content based upon textual input. For instance, the computer-implemented model is a diffuser model that generates images, video, text, etc. based upon textual input (and optionally other input, such as an image, video, etc., such that the input is multi-modal). The computer-implemented model generates content based upon the query, and the computing system transmits the content to the client computing device for presentment to the user by way of the application.
In another example, upon receipt of the query, the computing system searches the computer-readable index for items that are relevant to the query and the computing system provides the query as input to the computer-implemented model. In such an example, the SERP includes both: 1) an item identified by the computing system as being relevant to the query; and 2) content generated by the computer-implemented model.
Further, the computing system can search the computer-readable index based upon the content generated by the computer-implemented model. More particularly, the computing system can transmit the content generated by the computer-implemented model to the client computing device for presentment to the user by way of the application. The client computing device receives an indication from the user that a search is to be performed based upon the content. More specifically, the content can have feature values, and the computing system can search the computer-readable index based upon the feature values. The computing system identifies items that are indexed in the computer-readable index based upon the feature values of the content generated by the computer-implemented model. Hence, the computing system can identify pre-existing items that are similar to the content that was generated (on-demand) based upon the query. The computing system transmits the identified items to the client computing device for presentment to the user by way of the application executing on the client computing device.
While the examples set forth above have referenced a SERP, it is emphasized that the technologies described herein are not limited to inclusion in search engines. Rather, the technologies describe herein can be used in connection with productivity applications (e.g., when a user is searching for content to include in a word processing document, a slideshow presentation, etc.), and the claims are not to be limited to search engines unless expressly recited in such claims.
The above summary presents a simplified summary in order to provide a basic understanding of some aspects of the systems and/or methods discussed herein. This summary is not an extensive overview of the systems and/or methods discussed herein. It is not intended to identify key/critical elements or to delineate the scope of such systems and/or methods. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.
Various technologies pertaining to searching for content and generating content based upon a query are now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. It may be evident, however, that such aspect(s) may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more aspects. Further, it is to be understood that functionality that is described as being carried out by certain system components may be performed by multiple components. Similarly, for instance, a component may be configured to perform functionality that is described as being carried out by multiple components.
Moreover, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or clear from the context, the phrase “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, the phrase “X employs A or B” is satisfied by any of the following instances: X employs A; X employs B; or X employs both A and B. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or clear from the context to be directed to a singular form.
Further, as used herein, the terms “component” and “system” are intended to encompass computer-readable data storage that is configured with computer-executable instructions that cause certain functionality to be performed when executed by a processor. The computer-executable instructions may include a routine, a function, or the like. It is also to be understood that a component or system may be localized on a single device or distributed across several devices. Further, as used herein, the term “exemplary” is intended to mean serving as an illustration or example of something and is not intended to indicate a preference.
Described herein are various technologies pertaining to searching for content and generating content based upon a query. With more particularity, a computing system receives a query from an application executing on a client computing device, where the query is set forth to the client computing device by a user thereof. In an example, the computing system executes a general-purpose search engine, and the user is providing the query to the search engine. Based upon the query, the computing system searches a computer-readable index of items, where the items may be webpages, images, videos, etc. Additionally, the computing system provides the query to a computer-implemented model that generates contents based upon textual input. In an example, the computer-implemented model is a diffuser model that generates images based upon input, where the input can be textual or multi-modal. The computer-implemented model generates content based upon the query (e.g., a prompt), and the computing system returns search results to the client computing device. In an example, the search results include an item identified in the search of the computer-readable index as well as the content generated by the computer-implemented model.
The query may be a relatively complex query that expresses an information retrieval intent that can be identified by the computing system; however, there may be few (or no) items indexed by the computing system that satisfy the information retrieval intent. An example query is “picture of a baseball player swinging a baseball bat at a soccer ball”. In this example, the query is well-formed (the information retrieval intent of user can be readily ascertained). An image that precisely matches the information retrieval intent of the user, however, may not be indexed by the computing system (or may not be readily located in an index of images based upon the textual query). Utilizing the technologies described herein, the information retrieval intent of the user who issued the query can be satisfied by providing the query to the computer-implemented model (as a prompt), which can generate the requested image based upon the query. Further, the computing system can receive an indication that the generated content is to be employed as a query (in a visual search). Accordingly, values for features of the generated content can be computed, where such values can be employed to search a computer readable index of items (images, videos, etc.). The computing system can search the computer readable index based upon the values for the features of the generated content and can identify and item in the computer readable index based upon such values. Put differently, the computing system can identify items indexed in the computer readable index based upon the content that was generated by the computer implemented model, where the identified items are similar to the generated content. Continuing with the example above, the computer implemented model can generate an image of a baseball player swinging a baseball bat at a soccer ball, and values for features of such generated image can be employed by the computing system to identify previously indexed images that are similar to such image. Thus, if the user were searching for a pre-existing image of a baseball player swinging a bat at a soccer ball, the computing system can identify such image based upon the image generated by the computer implemented model.
Referring now to
The computing system 102 includes a processor 110 and memory 112, where the memory 112 has instructions loaded therein that are executed by the processor 110. The computing system 102 also includes a data store 114, where the data store 114 comprises a document index 116, an image index 118, and search history data 120. While not shown, the data store 114 may also include a video index or an index of another type of item. The document index 116 is a computer-readable index that indexes documents by terms that pertain to such documents. For instance, the documents are webpages that are indexed by terms included in the webpages, indexed by terms in metadata assigned to the webpages, and so forth. The image index 118 is a computer-readable index that indexes images by terms and/or feature values that pertain to the images. The search history data 120 can identify numbers of searches performed by users of the computing system 102, numbers of days that users performed searches through use of the computing system 102, etc.
The memory 112 of the computing system 102 includes a search system 122 that searches the document index 116, the image index 118, and/or other suitable computer-readable indices (such as a video index) based upon queries received by the computing system 102. The search system 122 additionally generates contents based upon queries received by the computing system 102, as will be described in greater detail below. The search system 122 returns search results to client computing devices that transmit queries to the computing system 102, where the search results include items identified based upon searches of indices in the data store 114 and/or content generated based upon the queries.
The search system 122 optionally includes an authorizer module 124 that ascertains whether content can be generated on behalf of the user. Generating content based upon textual or multi-modal input is computationally expensive, and the authorizer module 124 can restrict content generation to users who are relatively frequent users of the search system 122. Accordingly, prior to the search system 122 generating content based upon a query received from a client computing device operated by the user, the authorizer module 124 can access the search history data 120 and grant or deny a request to generate content based upon information in the search history data 120 assigned to the user. When the authorizer module 124 determines that the user has set forth a number of queries to the search system 122 over a window of time that is greater than a threshold, the authorizer module 124 can authorize generation of content based upon a query received from a client computing device operated by the user. In another example, when the authorizer module 124 determines that the user has set forth queries to the search system 122 during a number of days that is above a predefined threshold, the authorizer module 124 can authorize generation of content based upon a query received from the client computing device operated by the user.
The computing system 102 further optionally includes a query classifier 126 that assigns classification labels to queries received by the computing system 102, where a classification label assigned to a query by the query classifier 126 indicates whether content is to be (dynamically) generated based upon the query. The query classifier 126 can be a binary classifier, where a first label that is assignable to a query by the query classifier 126 indicates that content is not to be generated based upon the query, while a second label that is assignable to the query by the query classifier 126 indicates that content is to be generated based upon the query. The query classifier 126 can be trained based upon queries having labels assigned thereto, where the labels indicate whether or not content is to be generated based upon the queries. Some queries are well-suited for generating content, such as relatively long and detailed queries like “video of a fish jumping out of the water while holding a banjo”. Other queries are not well-suited for generating content, such as a query requesting a fact or a query pertaining to a particular webpage, such as “how tall was George Washington” or “US Congress webpage”. The query classifier 126 is trained to distinguish between queries that are well-suited for generating content and queries that are not well-suited for generating content.
The search system 122 comprises an index search module 128 that searches the document index 116 and/or the image index 118 (or other suitable computer-readable index) based upon queries received from client computing devices. The index search module 128 can identify items from the indices 116-118 based upon the queries and can further rank identified items to form ranked lists of items.
The search system 122 further comprises a content generator module 130 that generates contents based upon queries received by the search system 122. With more particularity, the content generator module 130 includes a computer-implemented model that generates contents based upon textual input or multi-modal input. In an example, the computer-implemented model is a diffuser model. The computer-implemented model, in one example, generates images based upon prompts. The computer-implemented model, in another example, generates videos based upon prompts. In still yet another example, the computer-implemented model generates text and/or audio based upon prompts.
The search system 122 further optionally includes a feature identifier module 132 that computes values for features of contents generated by the content generator module 130. For example, images in the image index 118 can be indexed by values of features of such images. When the content generator module 130 generates an image based upon textual input, the feature identifier module 132 can compute values for the features of such image. Values of features of images can be indicative of colors existent in the images, location of certain features in the images, and so forth.
The search system 122 also includes a suggestion module 134 that generates query suggestions based upon one or more of: 1) contents generated by the content generator module 130; 2) search results identified by the index search module 128; 3) previous queries issued to the search system 122, and so forth. In an example, queries suggested by the suggestion module 134 can be queries that are well-suited for use by the content generator module 130 to generate content.
The search system 122 also includes a visual search module 136 that can perform a visual search, where the visual search module 136 uses an image generated by the content generator module 130 as a basis for searching the image index 118. As noted above, the feature identifier module 132 can compute values for features of an image generated by the content generator module 130. The visual search module 136 can search the image index 118 based upon the values of such features and can identify images that are similar to the image generated by the content generator module 130.
The search system 122 also includes a supplemental content module 138 that can identify supplemental content based upon at least one of a query received by the search system 122, a search result identified by the index search module 128, or a value of a feature of content generated by the content generator module 130. In an example, the supplemental content is an electronic advertisement that is presented together with search results.
An example of operation of the search system 122 is now set forth. The search system 122 receives a query set forth by the user 108 of the client computing device 104, where the query is received by way of the application executing on the client computing device 104. Optionally, the authorizer module 124 (prior to or in response to receipt of the query) searches the search history data 120 based upon an identifier of the user 108 to ascertain whether the user is authorized to have content generated on their behalf. When the authorizer module 124 determines that content is not to be created on behalf of the user 108, the search system 122 does not provide the query to the content generator module 130.
When the authorizer module 124 determines that content can be created for the user 108, the search system 122 optionally provides the query to the query classifier 126. The query classifier 126 assigns a classification label to the query, where the classification label indicates whether or not the query is to be provided to the content generator module 130. When the classification label indicates that the query is not to be provided to the content generator module 130, the search system 122 does not provide the query to the content generator module 130.
Regardless of the output of the authorizer module 124 and the output of the query classifier module 126, the index search module 128 searches a computer-readable index (the document index 116, the image index 118, or other suitable index in the data store 114) based upon the query. The index search module 128 can identify an item indexed in the computer-readable index based upon the query (e.g., the index search module 128 identifies the item as being relevant to the query).
When the authorizer module 124 determines that content can be created for the user 108, and further when the query classifier 126 assigns a classification label to the query that indicates that the query is to be provided to the content generator module 130, the search system 122 provides the query to the content generator module 130. The content generator module 130 generates content based upon the query. In an example, when the query is “baseball player hitting a soccer ball with a baseball bat”, the content generator module 130 generates an image that includes a baseball player hitting a soccer ball with a baseball bat.
The search system 122 can return search results to the client computing device 104 for presentment by way of a graphical user interface of the application executing on the client computing device 104, where the search results include at least one of the item identified as being relevant to the query by the index search module 128 or the content generated by the content generator module 130 based upon the query. For example, the search system 122 may be included in a general-purpose search engine, and the search results can be included in a search engine results page (SERP) that is provided to the client computing device 104 for presentment to the user 108 by way of the application executing on the client computing device 104.
As mentioned previously, the content generator module 130 may utilize a significant amount of computing resources when generating the content based upon the query. Accordingly, it may take some amount of time for the content generator module 130 to generate the content (e.g., 1-5 seconds). The search system 122 can generate animations for presenting the content generated by the content generator module 130 together with the item identified by the index search module 128 based upon the query. For example, the search system 122 constructs a SERP to include a region for displaying content generated by the content generator module 130, where generated content “fades in” over time (such that the user 108 does not see a blank portion of a SERP until generation of the content is completed). In another example, the suggestion module 134 generates query suggestions based upon the query received from the client computing device 104, the item identified by the index search module 128 as being relevant to the query, previous query submitted by the user 108 or other users of these search system 122, etc. Query suggestions generated by the suggestion module 134 can be presented in the GUI of the application executing on the client computing device 104 until the content generator module 130 completes generation of the content. Thereafter, the search system 122 can update the information presented to the user 108 to include the content generated by the content generator module 130.
The feature identifier module 132 computes values for features of the content generated by the content generator module 130, where the values for the features can reflect items depicted in the content, named entities in the content, locations of items in the content, gradients in the content, and so forth. When the content is an image, the feature identifier module 132 can update the image index 118 to index the image generated by the content generator module 130 by the values of the features computed by the feature identifier module 132. Similar operations can be performed when the content is a video, text, multi-modal, etc.
In some cases, the user 108 may wish to employ content generated by the content generator module 130 to locate an item included in one of the indices 116-118 stored in the data store 114. Thus, the search system 122 can receive a message from the client computing device 104 that indicates that the search system 122 is to search one or more of the indices 116-118 based upon the content generated by the content generator module 130. This may occur when the user 108 desires to find pre-existing content that is similar to the content generated by the content generator module 130. Upon receipt of such message, the visual search module 136, for example, can search the image index 118 based upon the values of the features of the content generated by the content generator module 130. The visual search module 136 can identify an image indexed in the image index 118 based upon the search of such index 118, and the search system 122 can return the identified image to the client computing device 104 as an updated search result. Hence, again, the search system 122 can employ the content generated by the content generator module 130 in connection with searching for content that is similar to the content generated by the content generator module 130.
The supplemental content module 138 can identify supplemental content, such as an electronic advertisement, based upon the query received by the search system 122, the item identified as being relevant to the query by the index search module 128, a value of a feature of the content generated by the content generator module 130, and so forth.
In summary, then, the search system 122 can search for existing (indexed) items based upon the query received from the client computing device 104 and can additionally (dynamically) generate content based upon the query received from the client computing device 104. This may be particularly advantageous in scenarios where the information retrieval intent is clearly expressed in the query but an item that closely aligns with such intent is not indexed in a computer readable index that is accessible to the search system 122. The technologies described herein pertain to improvements over conventional search systems, which are limited to searching for pre-existing (indexed) items.
Now referring to
The modules 124-138 operate as described above with respect to
The content generator module 130 generates the content based upon the query and transmits the content to one of the computing system 102 or the client computing device 104. When the second computing system 202 transmits the content to the computing system 102, the search system 122 includes the generated content in search results and transmits the search results to the client computing device 104. When the second computing system 202 transmits the generated content to the client computing device 104, the application executing on the client computing device 104 can incorporate the content into a GUI of the application.
Now referring solely to
In the example illustrated in
Referring now to
Moreover, the acts described herein may be computer-executable instructions that can be implemented by one or more processors and/or stored on a computer-readable medium or media. The computer-executable instructions can include a routine, a sub-routine, programs, a thread of execution, and/or the like. Still further, results of acts of the methodologies can be stored in a computer-readable medium, displayed on a display device, and/or the like.
Referring now solely to
AT 708, a search result is identified from the computer-readable index based upon the search. At 710, content is generated (dynamically) based upon the query, where the content is generated by a computer-implemented model. For instance, the computer-implemented model is a diffuser model. At 712, the search result identified at 708 and the content generated at 710 are returned to the client computing device. The methodology 700 completes at 714.
Now referring to
At 810, a determination is made as to whether the query is well-suited for provision to a computer-implemented model that generates content based upon textual input. When it is determined at 810 that the query is not well-suited for provision to the computer-implemented model, the methodology 800 proceeds to 812 and the search result is returned to the client computing device.
When it is determined at 810 that the query is well-suited for provision to the computer-implemented model, the methodology 800 proceeds to 814, where the search result and a selectable button are returned to the client computing device. Put differently, the search result is returned to the client computing device and a selectable button is caused to be presented on a display of the client computing device. The selectable button notifies a user who issued the query that selection of such button will result in content being generated based upon the query.
At 816, a determination is made as to whether the selectable button has been selected by the user. When it is determined at 816 that the selectable button has been selected, the methodology 800 proceeds to 818, where the query is provided to the computer-implemented model that generates content based upon textual input. At 820, generated content is obtained from the computer-implemented model, and at 822 the generated content is returned to the client computing device.
The methodology 800 completes at 824 upon the search result being returned to the client computing device at 812, upon a determination being made at 816 that the selectable button has not been selected, or upon return of the generated content to the client computing device at 822.
At 910, a determination is made as to whether the query is well-suited for provision to a computer-implemented model that dynamically generates content based upon textual input. When it is determined at 910 that the query is not well-suited for provision to the computer-implemented model, the methodology 900 proceeds to 912, where the search result identified at 908 is returned to the client computing device. When it is determined at 910 that the query is well-suited for provision to the computer-implemented model, search history for the issuer of the query is accessed at 914. At 916, a determination is made as to whether content is to be generated based upon the search history for the issuer of the query. When it is determined that content is to be generated, the methodology 900 proceeds to 918, where the query is provided to a computer-implemented model that generates content based upon textual input. At 920, the generated content is obtained from the computer-implemented model, and at 922 the generated content and the search result identified at 908 are returned to the client computing device. The methodology 900 completes at 924 upon a determination at 910 that the query is not well-suited for provision to the computer-implemented model, upon a determination at 916 that content is not to be generated based upon the search history for the issuer of the query, or a upon the generated a content and search result being returned to the client computing device at 922.
While the methodology 900 illustrates the determination 910 occurring prior to the determination 916, it is to be understood that determination 916 may occur prior to the determination 910.
With reference now to
At 1010, values for features of the content generated by the computer implemented model are computed. At 1012, a request to use the content as a query is received from the client computing device. At 1014, a computer-readable index of items is searched based upon the values of the features of the content computed at 1010. At 1016, an item is identified based upon the search, and at 1018 the item is returned to the client computing device. The methodology 1000 completes at 1020.
Referring now to
The computing device 1100 additionally includes a data store 1108 that is accessible by the processor 1102 by way of the system bus 1106. The data store 1108 may include executable instructions, computer-readable indices, etc. The computing device 1100 also includes an input interface 1110 that allows external devices to communicate with the computing device 1100. For instance, the input interface 1110 may be used to receive instructions from an external computer device, from a user, etc. The computing device 1100 also includes an output interface 1112 that interfaces the computing device 1100 with one or more external devices. For example, the computing device 1100 may display text, images, etc. by way of the output interface 1112.
It is contemplated that the external devices that communicate with the computing device 1100 via the input interface 1110 and the output interface 1112 can be included in an environment that provides substantially any type of user interface with which a user can interact. Examples of user interface types include graphical user interfaces, natural user interfaces, and so forth. For instance, a graphical user interface may accept input from a user employing input device(s) such as a keyboard, mouse, remote control, or the like and provide output on an output device such as a display. Further, a natural user interface may enable a user to interact with the computing device 1100 in a manner free from constraints imposed by input device such as keyboards, mice, remote controls, and the like. Rather, a natural user interface can rely on speech recognition, touch and stylus recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, voice and speech, vision, touch, gestures, machine intelligence, and so forth.
Additionally, while illustrated as a single system, it is to be understood that the computing device 1100 may be a distributed system. Thus, for instance, several devices may be in communication by way of a network connection and may collectively perform tasks described as being performed by the computing device 1100.
Various functions described herein can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer-readable storage media. A computer-readable storage media can be any available storage media that can be accessed by a computer. By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc (BD), where disks usually reproduce data magnetically and discs usually reproduce data optically with lasers. Further, a propagated signal is not included within the scope of computer-readable storage media. Computer-readable media also includes communication media including any medium that facilitates transfer of a computer program from one place to another. A connection, for instance, can be a communication medium. For example, if the software is transmitted from a web site, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave are included in the definition of communication medium. Combinations of the above should also be included within the scope of computer-readable media.
Alternatively, or in addition, the functionally described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.
Technologies described herein relate to at least the following examples.
What has been described above includes examples of one or more embodiments. It is, of course, not possible to describe every conceivable modification and alteration of the above devices or methodologies for purposes of describing the aforementioned aspects, but one of ordinary skill in the art can recognize that many further modifications and permutations of various aspects are possible. Accordingly, the described aspects are intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims. Furthermore, to the extent that the term “includes” is used in either the detailed description or the claims, such term is intended to be inclusive in a manner similar to the term “comprising” as “comprising” is interpreted when employed as a transitional word in a claim.
