Patent Application
20240028661
Browsers are applications that allow users to navigate to different websites and to display content, or web pages, generated by the website. Such content can include web page elements (e.g., mark-up language elements, document object model (DOM) objects) that are rendered by the browser to generate a presentation of the web page for display.
Implementations described herein are related to enabling a user to modify web page elements displayed by the web browser in a way not anticipated by the creators of the web page elements and without needing to understand the internal structure of the web page. Specially, disclosed implementations take as input a natural language request to modify a presentation of the current webpage as displayed by the browser, identifies data elements within the webpage (mark-up language elements, DOM elements, etc.) and modifies the appearance of the responsive data elements in accordance with the request. In some implementations, the proposed system may access data in a datastore to determine which elements to modify.
In one general aspect, a computer-based method can include receiving, from a user of a web browser, query data representing a request to modify a presentation of a web page as displayed by the web browser, the web page including structured data elements and the request identifying a criterion. The computer-based method can also include analyzing the web page to identify a structured data element of the structured data elements that satisfies the criterion. The computer-based method can further include applying a modification to the structured data element satisfying the criterion to produce a modified presentation of the web page as displayed by the web browser.
In another general aspect, a computer program product comprises a non-transitory storage medium, the computer program product including code that, when executed by processing circuitry of a computing device, causes the processing circuitry to perform a method. The method can include receiving, from a user of a web browser, query data representing a request to modify a presentation of a web page as displayed by the web browser, the web page including structured data elements, and the request identifying a criterion. The method can also include analyzing the web page to identify a structured data element of the structured data elements that satisfies the criterion. The method can further include applying a modification to the structured data element satisfying the criterion to produce a modified presentation of the web page as displayed by the web browser.
In another general aspect, an electronic apparatus comprises memory and controlling circuitry coupled to the memory. The controlling circuitry can be configured to receive, from a user of a web browser, query data representing a request to modify a presentation of a web page as displayed by the web browser, the web page including structured data elements and the request identifying a criterion. The controlling circuitry can also be configured to analyze the web page to identify a structured data element of the structured data elements that satisfies the criterion. The controlling circuitry can further be configured to modify the structured data elements satisfying the criterion to produce a modified presentation of the web page as displayed by the web browser.
The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.
Users of web pages frequently wish to modify those web pages to extract further information than what is being presented on the web pages. For example, consider a web meeting application that arranges identifiers (e.g., names, avatars) of meeting participants in a column aside a window showing a live stream of the meeting. In the conventional approaches to modifying a web page, the user would only be able to make changes to the interface implemented by the application developer. If the user wished to emphasize (e.g., highlight, bold, blink, etc.) the identifiers of those participants that report to the same manager as the user, the user would not be able to because the web developer did not allow for the user to do so. Disclosed implementations can implement such a request by enabling the user to modify a web page even when the web developer did not design the web page with such modifications built in. The enablement may take the form of a natural language query that may be interpreted by a natural language processor in conjunction with semantic information in the web page.
The query represents a request to modify web page elements according to a criterion. The criterion may identify a target of an emphasis operation (e.g., highlighting) based on a structured data element (e.g., semantic markup element, table element, list, etc.) representing the target; the web browser determines which structured data elements may be relevant to the target of the emphasis operation. The natural language processor is then configured to determine the criterion based on the query and semantic information. For example, for the query, “highlight meeting participants who have the same manager as me,” the criterion is “same manager as me.”
To identify the target of the emphasis operation, however, the disclosed implementations may seek to access an external dataset. Along these lines, there may be associations between the structured data elements and external datasets that make it possible to determine the association via a natural language query that includes the criterion. For example, in the above query, the disclosed implementations may involve identifying and searching/sorting through a human resources database to locate the managers of the participants and determine those participants represented by structural data elements satisfying the criterion. This provides a user with the ability to modify any web page with semantic information through which the web browser may interpret the data in the web page because the data is in a machine-readable format through semantic metadata.
Some approaches to modifying browser application user interfaces include browser extensions that enable a user to manipulate web elements of a user interface. These approaches, however, are brittle to changes of the interface by creators of respective browser applications and require some level of programming experience (understanding detailed information about how the web application is structured and understanding browser script, e.g., JavaScript®. Accordingly, most users cannot feasibly modify browser application views, and for those that can, the modification can break, e.g. when the creator or provider of the webpage changes or redesigns the structure and arrangement of elements include in the web page, without notice, requiring the modification to be rebuilt.
A technical problem with the above-described conventional approaches is that while web page creators may provide some options to the user to customize the web application user interface, those customizations are limited and static. Conventional third-party approaches to modifying web application user interfaces have controls that enable a user to manipulate elements of a user interface. The conventional approaches to modifying web application user interfaces (e.g., with extensions) are brittle to changes of the interface by creators of respective browser applications. Therefore, it is not feasible for a user of a conventional browser application to augment web application views without knowing detailed information about how the web application is structured, and manually creating brittle custom browser extensions (in a browser script, e.g., JavaScript®) to modify the web page elements.
In accordance with the implementations described herein, a technical solution to the above-described technical problem includes enabling a user to modify a web page as displayed by the web browser in a way not anticipated, e.g., beyond any options provided to the user, by the creators of the web application and without needing to understand the internal structure of the web page. For example, improved techniques implemented as part of the technical solution would allow a user to emphasize the names of participants having the same manager without knowing anything about the architecture or programming code associated with the web meeting application, as shown in
In the improved techniques, a browser enables the user to modify (e.g., augment) a presentation of the web page as displayed by the browser by parsing a natural language query, identifying a criterion in the query, analyzing structured data elements on the web page, and potentially one or more known external datasets, to evaluate the criterion and identify structured data elements to modify. The browser accomplishes this by receiving a natural language query provided by the user that includes the criterion.
A natural language query is processed using a natural language processor. Natural language processing is concerned with interactions between computers and human language, in particular how to program computers to process and analyze large amounts of natural language data. The goal is a computer capable of “understanding” the contents of documents, including the contextual nuances of the language within them. A natural language processor can accurately extract information and insights contained in the documents as well as categorize and organize the documents themselves. Processing a query using a natural language processor, when the query is written in natural language, is advantageous because the processor, in conjunction with markup language information in the web page, can interpret a user's intent far more accurately than a conventional query parser.
The query represents a request to modify, e.g., structured data elements, that satisfy the criterion. The criterion may identify an emphasis operation and a target of the emphasis operation. The browser analyzes the web page elements to identify elements, including data structured elements (such as semantic markup elements, table elements, lists, etc.). The browser determines which structured data elements may be relevant to the target of the emphasis operation. For example, some web page elements may be used to identify other web page elements to which the emphasis operation is applied. These elements (to which the target operation is applied) may be referred to as web elements having structured data elements that satisfy the criterion. The browser may also, based on the query criterion, identify and retrieve the one or more known external datasets, which may be stored locally (including an open browser tab and/or a dataset associated with the user, such as a shopping list or list of interests) or on a server other than that hosting the web page. The browser uses information from the one or more external datasets, along with the semantic information, to modify a presentation of the web page as intended by the user.
A technical advantage of disclosed implementations is that the improved techniques enable the user to modify a presentation of the web page without requiring the user to be aware of the internal structure and elements of the web page and/or requiring that the user creating a specialized browser extension on her own, e.g., through the use of natural language commands. Specifically, implementations can identify structured data elements in the web page identified by the semantic information and can cross-reference the structured data elements with the external datastore to identify structured data elements responsive to the request. Implementations can modify these responsive data elements in the current web page to change the appearance of those elements. Moreover, to ensure privacy, data from the external datastore remains local to the computer running the browser and is not sent to a server associated with the web application. Thus, the modification of the presentation of the web page can be accomplished without sending any data to a server hosting the web page.
In some implementations, the query is input by the user via a keyboard input into a text box input or the like. In some implementations, the query is input by the user via speech, i.e., spoken words from the user input into an application configured to capture and parse the speech into query components (e.g., criterion, targets, modification). In some implementations, the speech is displayed to the user via a speech-to-text application.
In some implementations, the processing circuitry 120 processes the query using a natural language processor. The browser application that receives a natural language query from the participant may use existing natural language processing capabilities (part-of-speech tagging, entity identification, etc.) related to semantic representations, language understanding and question-answering for a search application. The improved techniques leverage such capabilities to understand the intent of natural language requests, in order to translate a query to instructions for fetching data or augmenting the web application interface.
A web browser is application software running on processing circuitry (e.g., processing circuitry 120) for accessing web pages. When a user requests a web page from a website, the web browser retrieves content from a web server renders and displays the content in a browser window on the user's device. A web application is identified by a uniform resource locator (URL) input by the user. A web browser may display multiple web pages within a single user interface; each such web page is activated using a tab, e.g., the top of the user interface. A user of the web browser may activate a different web page by clicking on its tab on the user interface.
A web application is an application developed and stored on a web server by application creators and run within a web browser. The web application defines the web elements rendered by the browser to generate the user interface. Upon receiving a request to access the browser application from a user device (i.e., via a URL), the web server transmits code interpreted (rendered) and run within a web browser on the user device.
A web page element is an object defined by the web application. These include visible elements (e.g., participant lists and participant streaming images in a web meeting application) and non-visible elements (e.g., JavaScript). For example, in the above-discussed web meeting application, the web page elements 150 and 160 contain visual information about the meeting participants and participant names. In a browser application, some web page elements include application data (e.g., meeting participant identifiers, avatars, and photos) and other forms of data including semantic information (e.g., person, movie, director.).
A structured data element is a data object stored in a data structure that provides a hierarchal description of information in a web element and is organized according to a data model. In some implementations, structured data elements conform to a tabular format with relationship between different rows and columns. In this way, different rows and columns may be sorted or searched according to a query. Examples of structured data elements in the context of modifying web page elements are discussed with regard to
Semantic markup information is markup that reinforces the semantics, or meaning, of the information in web pages and browser applications rather than merely to define its presentation or look. Semantic markup information is processed by traditional web browsers as well as by many other user agents. In addition, other, more detailed semantic markup information is provided in the Semantic Web which includes Resource Description Framework (RDF), Web Ontology Language (OWL), and Extensible Markup Language (XML). HTML describes documents and the links between them. RDF, OWL, and XML, by contrast, can describe arbitrary things such as people, meetings, or cooking ingredients.
In some implementations, the semantic information in the web page is expressed according to the W3C Consortium standard for Semantic Web. Semantic Web allows web pages to encode web page elements with semantics (i.e., meaning). For example,
A modification of a presentation of a web page as displayed by a web browser in this context is a change to one or more web elements at the browser that change the appearance of content as displayed within the web browser. In some implementations, the modification is an augmentation to the structured data element in the web browser, e.g., an annotation such as an underline, an asterisk, or the like. In some implementations, the modification is an emphasis such as text italics, change in text color, change in text font, change in text font size, change in background color, bolded text, underlined text, a highlight, or the like. The modification can be referred to as a visual pointer because the modification visually distinguishes the modified element. Thus, a modification can be added web page elements at the browser that, when rendered by the browser, effect a change in appearance of a web page element. In some implementations, the modification is a removal or deletion of displayed data in the user interface or web element. In some implementations, when the structured data element satisfying the criterion includes a table, the modification represents a sort order operation in which, upon application of this modification, table rows are rearranged in the table to produce a modified sort order. In some implementations, when the structured data element satisfying a criterion from a subsequent query includes a table row of list element, the modification includes a resorting (reordering) of the table rows.
In some implementations, a modification may be selected from a library of prespecified modifications, i.e., predefined emphasis operations. Such a library may include modification identifiers and, for each modification identifier, instructions for effecting the modification. For example, a modification identifier for bold can be associated with instructions for applying a markup element (e.g., <b>) to a web page element that satisfies the criterion, where a modification identifier for highlight can be associated with instructions for applying a markup element to change a background color of the web page element that satisfies the criterion. Modification identifiers may be text (e.g., “bold”) or may be any string, number, or combination of these that corresponds to text that may appear in the query. The library can also be used when the query is unclear about the modification. For example, if a query states “show me the participants having the same manager as me,” then the processing circuitry may select a modification at random from the library and implement its instructions. In some implementations, the improved techniques include a default modification in case the query renders the modification as indeterminate. Example instructions include adding markup tags for implementing the prespecified modification. In some implementations, the instructions can be customized by the user (e.g., enabling the user to select the highlight color, the default modification, etc.).
In some implementations, the processing circuitry may be configured to provide an interface that enables the user to select a modification from two or more modifications (e.g., selectable modifications). For example, a query that includes highlight may correspond to two or more modifications in the library, e.g., highlight in blue or highlight in yellow. In such an implementation, the user is enabled to select a modification from the two or more modifications (also referred to as selectable modifications or candidate modifications). For example, the candidate modifications may be displayed to the user in a scrollable list, and the user may click on/select the desired modification.
In some implementations, a user may be enabled to choose a default modification that is made when the user does not specify a modification. Such a default modification, in some implementations, is saved to a profile associated with the user and/or with the device so that the default modification is enabled for the user when using any web application in the web browser. Moreover, the user may save their most recent modification to a buffer (e.g., cache memory) or other storage for application to a subsequent web page.
In some implementations, a modification may take the form of a deemphasis of web page elements that do not satisfy the criterion. For example, a deemphasis can take the form of a lightening or fading of text color, a reversion from bold text to normal text, an undoing of an emphasis, a decrease in the size of font text, and the like.
Query data represents a query, or a request to modify a user interface or web element. The query data in this context includes an identification of data (e.g., web page elements) in a web page to be modified, a criterion for identifying those structured data elements to be modified, and a modification to be performed on the presentation of the web page. In some implementations, the query data is transmitted to a natural language processor for identification of the data, criterion, and modification. For example, a natural language processor may identify the modification in the query data. The natural language processor may then transmit an identifier representing the modification to a processor on which the web browser runs.
A criterion is a condition satisfied by the data to be modified, and not satisfied by the data that is not to be modified. In some implementations, the reverse occurs. In the above-discussed web meeting application example, the criterion is “same manager as me.” This implies that the processing circuitry will identify an external dataset that includes information about the participants' managers and compare each participant's manager with the participant making the query. For example, the processing circuitry may identify, in the external dataset, all of the participants listed in the database having the same manager as the participant. In another example, the processing circuitry may identify all of the participants listed in the database not having the same manager as the participant.
As shown in
In some examples, a web page of a browser application is provided as web page server 105 associated with a domain name. An example web site is a collection of web pages formatted in an appropriate machine-readable language, e.g., a markup language such as hypertext markup language (HTML), that can contain text, images, multimedia content, and programming elements, e.g., scripts. The text, images, multimedia content, and programming elements are examples of web page elements. Each web site is maintained by a publisher, e.g., an entity that manages and/or owns the web site. Web page server 105 can be static or dynamic.
In some examples, web page server 105 provides data over the network 102 and is associated with a resource address, e.g., a uniform resource locator (URL). In some examples, web page server 105 can provide a web site, which includes digital content, such as web pages, word processing documents, and portable document format (PDF) documents, images, video, and feed sources, among other appropriate digital content. The web page server 105 can store content, e.g., words, phrases, images and sounds and may include embedded information, e.g., meta information and hyperlinks, and/or embedded instructions, e.g., scripts.
In some examples, a user device 106 is an electronic device that is under control of a user and is capable of requesting and receiving resources from the web page server 105 over the network 102. Example user devices 106 include personal computers, mobile computing devices, e.g., smartphones, wearable devices, smart televisions, and/or tablet computing devices that can send and receive data over the network 102. As used throughout this document, the term mobile computing device (“mobile device”) refers to a user device that is configured to communicate over a mobile communications network. A smartphone, e.g., a phone that is enabled to communicate over the Internet, is an example of a mobile device, as are wearables and other smart devices such as smart speakers. A user device 106 typically includes a user application, e.g., a web browser, to facilitate the sending and receiving of data over the network 102.
In some implementations in which there are multiple user devices in the network 102, each of the multiple user devices is associated with a respective user. In such a case, a modification to a web page initiated by a first user device associated with a first user may be shared with a second user device associated with a second user. For example, at least some or all of the participants of the web meeting shown in
In some implementations in which there are multiple user devices in the network 102, each of the multiple devices may make modifications to the web page along the lines disclosed herein. As disclosed above, each of the multiple users on the respective user devices may select other users with whom to share the modifications to the web page. The modification data generated by the processor 120 may be transmitted to other user devices in the network 102. In this case, when the modification to the web page is effected within a browser extension, that browser extension running on browsers of the other user devices receives the modification data and modifies the web page according to the received modification data.
As shown in
As shown in
The web page elements 112, 113, 114 are objects displayed by the web browser in the presentation of the web page 111. The web page elements 112, 113, 114 each include data displayed to the user as discussed with respect to
As shown in
In some implementations, the web page elements 112, 113, 114 are associated with a document model (e.g., document object model) that includes a set of nodes representing the structured data elements 115, 116, 117. In the example above, the items may be list elements, e.g., identified by the <li> HTML tag. In the example above, the items may be in a separate table row, e.g., with each item being in a column labeled “ingredient” and appearing with other columns, such as measurement type, measurement amount, and/or calories. In the example above, the items may have semantic markup, e.g., a metadata tag indicating each item is an ingredient in a recipe.
In an example, a user may submit a query 108, “mark ingredients in my shopping list that are on sale at a grocery store close to me.” The processing circuitry 120 performs a natural language processing operation on the query and, in conjunction with semantic data on the web page, determines that the intent of the user is to access grocery information for the user, e.g., associate with a user profile or an account of the user. It is noted that some shopping list data as described here may be private and is not to be shared with the public. Accordingly, the processing circuitry 120 should be able to modify the user interface without sending private data, or any other data, to the web page server 105, for example.
Moreover, the criterion the user has specified in the query 108 is that an ingredient identifier listed in the web page element 112 matches a food identifier in a grocery store database identifying that food as being on sale. Accordingly, the processing circuitry accesses the structured data elements 115 to obtain the ingredient identifiers of the structured data elements 115 and compares these food identifiers with the food identifiers in the grocery database. The processing circuitry identifies the matches between the food identifiers and the ingredient identifiers, as shown in the example of
The query data also specifies a particular modification. The modification may be specific, such as “highlight” or “turn blue”. The modification may be general, i.e., “mark” in the example query 108 or “show me”. The system may include a plurality of predefined modification identifiers, which correspond to the text string in the query data. (e.g., “highlight,” “italicize,” “bold,” “underline,” etc.). These predefined modifications may include a template with a fixed portion and a variable portion, e.g., “turn *” with the variable portion matching (corresponding to) what follows the fixed portion. The variable portion may represent a set of predefined values, such as colors. Thus, “turn red” or “make blue” may be modification identifiers that correspond with a predefined modification. Some predefined terms in the query may be translated to a modification identifier. For example, the processing circuitry 120 may recognize “mark” in a query as a non-specific modification request, and translate “mark” (the non-specific modification request) to “highlight” (a modification identifier). This translation may be determined by randomly selecting the modification identifier from among the predefined modification identifiers. This translation may be determined by selecting a default modification identifier. This translation may be determined by rules or by a machine-learned classifier that performs a machine-learning-based classification operation to identify “mark” as being most likely to refer to highlighting. The modification identifier of highlighting may be associated with instructions for applying a background color mark-up tag to the web page element(s) that satisfy the criterion. Accordingly, the processing circuitry 120 transmits instructions in the form of modification data 109 to highlight the structured data elements 115 for which the ingredient identifier matches a food identifier in the grocery database. Now the user interface 111 shows the user which ingredients are on sale at a local grocery store.
In some implementations, the processing circuitry 120 obtains the external dataset from an external dataset available to the user device 106 and not on a public network. For example, the external dataset 122 may be local storage on the user device 106. Alternatively, the external dataset 122 may be an external disk drive directly connected to the user device 106 or over a wireless LAN.
In some implementations, the processing circuitry 120 obtains the external dataset from an external server 121 over a network 103. In some implementations, the network 103 is not distinct from the network 102. In some implementations, the network 103 is separate from the network 102. In some implementations, the network 103 includes a virtual private network (VPN) for accessing private data.
The processing circuitry 120 includes a network interface 222, one or more processing units 224, memory 226, and a display interface 228. The network interface 222 includes, for example, Ethernet adaptors, Token Ring adaptors, and the like, for converting electronic and/or optical signals received from the network to electronic form for use by the processing circuitry 120. The set of processing units 224 include one or more processing chips and/or assemblies. The memory 226 includes both volatile memory (e.g., RAM) and non-volatile (non-transitory) memory, such as one or more ROMs, disk drives, solid state drives, and the like. The set of processing units 224 and the memory 226 together form controlling circuitry, which is configured and arranged to carry out various methods and functions as described herein. The display interface 228 is configured to provide data to a display device for rendering and display to a user.
In some implementations, one or more of the components of the processing circuitry 120 can be, or can include processors (e.g., processing units 224) configured to process instructions stored in the memory 226. Examples of such instructions as depicted in
The query manager 230 is configured to receive query data 231. In some implementations, the query manager 230 receives the query data 231 over the network interface 222, i.e., over a network. In some implementations, the query manager 230 receives the query data 231 directly as user input, e.g., from user device 106. As shown in
The query data 231 represents a query (i.e., a request) to modify the user interface web page elements according to a criterion. In some implementations, the query data 231 takes the form of a string of characters. In some implementations, the query data 231 takes an alternative form, e.g., a graphic, an image, an audio clip, a video, etc. In some implementations, the query data 231 is written in a form appropriate for natural language processing.
The NLP manager 232 is configured to process the query data 231 using a natural language processing algorithm to produce NLP data 233. The NLP algorithm works primarily with syntax (i.e., structure) of the query represented by the query data 231 and semantics (i.e., meaning) as represented by semantic data, e.g., semantic data 244. On the syntax side, the NLP algorithm may be configured to lemmatize words and relate them to their basic forms, e.g., part-of-speech analysis. In some implementations, the NLP algorithm includes breaking apart sentences, inducing grammatical structures, and segmenting words. On the semantics side, the NLP algorithm may perform entity and entity attribute analysis and recognition (including entity names and synonyms).
The NLP data 233 represents output of running the NLP algorithm of the NLP manager 232. The NLP data 233 is in a form that is interpretable by the processing circuitry 120 for identifying structured data elements from which the user interface 111 (
The criterion data 234 represents a criterion by which the structured data elements from which the user interface 111 is to be modified are identified. For example, for the request “show me the participants having the same manager as me,” the NLP manager 232 may deduce that the criterion represented by the criterion data 234 identifies participant names in an external dataset relevant to the request as having the same manager as the user. In another example, for the request “mark ingredients that are on sale at a nearby grocery store,” the NLP manager 232 may deduce that the criterion represented by the criterion data 234 identifies food identifiers in a grocery database which are identified as being on sale. Generally, the criterion data 234 indicates an aspect of the external dataset that is to be compared to a field in the structured data elements.
The structured data element 235 represents references to structured data elements (e.g., structured data elements 115 of
In some implementations, the query manager 230 identifies structured data elements by identifying table elements, list elements, or semantic markup elements. For example, HTML table elements that may be identified include tags such as <tr> (define a row of a table), <th> (define a header), and <td> (define a cell in the table). HTML list elements that may be identified include tags such as <ul> for an unordered (bulleted) list, <ol> for an ordered list, and <dl> (definition list). HTML semantic markup items defining semantic markup information include descriptors of what each element represents. For example, an HTML tag <div itemtype=“http://schema.org/Restaurant”><spanproperty=“rating”>4</span><span property=“telephone”>867-5309</span></div> informs a natural language processor that a value of a first field in a structured data element represents a restaurant as defined in the schema.org hierarchy. The restaurant has a rating value of 4 and a telephone number of 867-5309. Accordingly, the semantic markup information places the rating and phone number in the context of a restaurant. Once the structured data elements are identified via these table elements, list elements, or semantic markup elements, the query manager 230 compares the structured data elements to the criterion as described with regard to
The web page 300 has a top-level page 302 that represents the web application as a whole and is displayed in a user interface (i.e., user interface 111). In the case shown in
The chat section 304 is configured to enable the participants to send private or public messages to other participants during a web meeting. As shown in
The message list 308 includes structured data elements 310(1) and 310(2), each representing data (e.g., metadata) describing each message. As shown in
The participant list 306 is configured to include all participants in the web meeting. In some implementations, the participant list 306 may change in real time according to which participants enter or leave the web meeting while the web meeting is being conducted. In some implementations, the participant list 306 is static. The participant list 306 includes structured data elements 312(1), 312(2), 312(3), and 312(4), each representing data (e.g., metadata) describing each participant. As shown in
It is noted that the structured data elements of a web page element, when a document model with which the web page elements are associated is a tree structure, are represented by a set of nodes of the tree structure. That is, the document model associated with a web page element describes a relationship of the structured data elements with each other.
Returning to
The external dataset identifier data 237 represents an external dataset (e.g., external dataset 122 of
In some implementations, for the above example, the human resources database is stored locally and the identifier provides a location of the human resources database in storage. In some implementations, the human resources database is accessed over a network (e.g., network 103) at an external server (e.g., external server 121). In some implementations, the human resources database is private and may only be accessed from the external server using various data protection schemes (e.g., passwords, two-way authentication, knowledge-based authentication, or the like). Further detail regarding an external dataset is discussed with respect to
In some implementations, the external dataset is associated with a profile of the user. In some implementations, the profile is associated with a user name/password. In some implementations, the profile of the user is stored with web browser information and may be accessed through web browser preferences. For example, a profile of the user stored may include a browsing history, a set of bookmarks, browser themes, avatars, security options, and the like. Based on such a profile, the external dataset manager 250 may select a particular external dataset. For example, if a user has a bookmark pointing to a grocery store web site, the external dataset manager 250 may choose to extract data from that website.
In some implementations, the external dataset is a search result returned form a search engine index, which can search a document repository and/or knowledge graph (entity/fact repository). In some implementations, the processing circuitry may submit a query (e.g., second query or search engine query) to the search engine that includes terms and/or phrases from the request and/or from the structured data elements. For example, a user looking at a web page with a recipe may provide a request to “highlight ingredients with gluten”. After identifying structured elements for ingredients in the web page elements, the system can submit a query to the search engine for each element, e.g., a query of ‘milk has gluten’, ‘cornmeal has gluten’, etc. The system may receive the search results and parse the result to determine whether that ingredient should be modified (highlighted). Some search results include a onebox or short answer result to such factual queries. Accordingly, an external dataset may be selected based on its presence in a search engine index.
In some implementations, the external dataset includes data originating from another open browser tab. Such data is technically external data, as it is external to the browser tab of interest.
The example external dataset 400 is represented in
Consider the case in which the member represented by database element 432 is the user forming the above-discussed query or request, “show the participants having the same manager as me.” The query manager 230 has determined via the NLP algorithm that the query intent indicated the human resource database as the external dataset. The query manager 230 determines the participants in the participant list reporting to the same participant as the member represented by database element 432. As shown in
As shown in
As shown in
It is noted that the external datasets 400 and 500 discussed with regard to
In some implementations, the modification of a web page, web page elements, or structured data elements are not applied based on an identification of an external dataset such as external datasets 400 and 500. In such an implementation, the modification is based on data local to the user device 106 (
Returning to
The web browser data 241 represents the data used and/or displayed by the web browser that may be accessed for the purpose of modifying the user interface of the web browser. As shown in
Each of the web page elements 242(1 . . . M), e.g., 242(1), is an object displayed by a web browser in a user interface in a browser window, based on data received from a web page server (e.g., web page server 105). For example, in the above-discussed web meeting application, the web page elements 150 and 160 contain visual information about the meeting participants and participant names. In a browser application, some web page elements include application data (e.g., meeting participant identifiers, avatars, and photos) and other forms of data including semantic information (e.g., first name, last name, email address, etc.). In some implementations, the web page elements 150 and 160 are received from the web page server in the form of data and instructions for displaying.
As show in
The semantic data 244 represents semantic information embedded in the web page including semantic metadata. To enable the encoding of semantic information with the data, technologies such as Resource Description Framework (RDF) and Web Ontology Language (OWL) are used. These technologies are used to formally represent metadata. For example, ontology can describe concepts, relationships between entities, and categories of things. These embedded semantics offer significant advantages such as reasoning over data and operating with heterogeneous data sources.
For example, in the web meeting application example above, the data representing the participant list (e.g., Participant List 306 of
The external dataset manager 250 is configured to identify and obtain external dataset data 251 from a variety of sources. In some implementations, the external dataset manager 250, before any user query is received by the query manager 230, is configured to access sources of information from local sources, i.e., attached or internal hard drive, or remote sources via the network interface 222. In some implementations, the external dataset manager 250 assigns unique identifiers to each external dataset so accessed and obtained.
In some implementations, the external dataset manager 250 is configured to access external dataset data 251 in response to the query manager 230 receiving query data, e.g., query 108 of
The external dataset data 251 includes information relevant to user queries, for example the organizational chart in
The web page element modification manager 260 is configured to perform the modifications to the web page elements of a web page according to web page element modification data 271, as determined from query data received by the query manager 230. In some implementations, the web page element modification manager 260 is implemented as part of the browser or as a browser extension capable to modify the context of webpages as they are displayed. In some implementations, the web page element modification manager 260 may alter the appearance of pixels displaying the web page elements to be modified. In some implementations, the web page element modification manager 260 may generate other effects for the web page elements, e.g., increasing brightness of pixels in a region to be modified or generating animation effects such as blinking or simulating waves over the region. In some implementations, the web page element modification manager 260 modify the web page such that web page elements are re-sorted, e.g., reordering or rearranging table rows or list elements. In some implementations, the web page element modification manager 260 removes objects from the display as a modification.
In some implementations, the page element modification manager 260 selects a particular modification based on the query from a library or repository of modifications, as represented by library data 272 in
The components (e.g., modules, processing units 224) of the user device 106 can be configured to operate based on one or more platforms (e.g., one or more similar or different platforms) that can include one or more types of hardware, software, firmware, operating systems, runtime libraries, and/or so forth. In some implementations, the components of the processing circuitry 120 can be configured to operate within a cluster of devices (e.g., a server farm). In such an implementation, the functionality and processing of the components of the processing circuitry 120 can be distributed to several devices of the cluster of devices.
The components of the processing circuitry 120 can be, or can include, any type of hardware and/or software configured to process attributes. In some implementations, one or more portions of the components shown in the components of the processing circuitry 120 in
Although not shown, in some implementations, the components of the processing circuitry 120 (or portions thereof) can be configured to operate within, for example, a data center (e.g., a cloud computing environment), a computer system, one or more server/host devices, and/or so forth. In some implementations, the components of the processing circuitry 120 (or portions thereof) can be configured to operate within a network. Thus, the components of the processing circuitry 120 (or portions thereof) can be configured to function within various types of network environments that can include one or more devices and/or one or more server devices. For example, the network can be, or can include, a local area network (LAN), a wide area network (WAN), and/or so forth. The network can be, or can include, a wireless network and/or wireless network implemented using, for example, gateway devices, bridges, switches, and/or so forth. The network can include one or more segments and/or can have portions based on various protocols such as Internet Protocol (IP) and/or a proprietary protocol. The network can include at least a portion of the Internet.
In some implementations, one or more of the components of the processing circuitry 120 can be, or can include, processors configured to process instructions stored in a memory. For example, a query manager 230 (and/or a portion thereof), a web browser manager 240 (and/or a portion thereof), an external dataset manager 250 (and/or a portion thereof), and a web page element manager 260 (and/or a portion thereof).
In some implementations, the memory 226 can be any type of memory such as a random-access memory, a disk drive memory, flash memory, and/or so forth. In some implementations, the memory 226 can be implemented as more than one memory component (e.g., more than one RAM component or disk drive memory) associated with the components of the processing circuitry 120. In some implementations, the memory 226 can be a database memory. In some implementations, the memory 226 can be, or can include, a non-local memory. For example, the memory 226 can be, or can include, a memory shared by multiple devices (not shown). In some implementations, the memory 226 can be associated with a server device (not shown) within a network and configured to serve the components of the processing circuitry 120. As illustrated in
At 602, the query manager 230 receives, from a user of a web browser, query data (e.g., query 108) representing a request to modify a presentation of a web page (e.g., web page elements 112) as displayed by the web browser in a user interface (e.g., user interface 111), the web page elements including structured data elements (e.g., structured data elements 115), the request identifying a criterion (e.g., criterion data 234). For example, the query manager 230 may receive a user query, “highlight the ingredients that are on sale at a grocery store close to me.” Using a NLP algorithm, the query manager 230 determines that the user intends to look at grocery price data and identify items in the ingredient list that are on sale.
At 604, the query manager 230 analyzes the web page to identify a structured data element (e.g., structured data element 118) of the structured data elements that satisfies the criterion. For example, the criterion determined from the query in the example above is “ingredient identifiers corresponding to (matching) food identifiers in a grocery list of food items that are on sale.” The query manager 230 then identifies and accesses an external dataset managed by the external dataset manager 250 and performs the comparison. When there is an ingredient identifier matching a food identifier of the grocery list, the query manager 230 identifies the structured data element having that ingredient identifier as a field value. The query manager 230, after applying the NLP and semantic markup info to the query, identifies the criterion as “Product identifiers==foodIdentifiers AND foodIdentifiers.isOnSale=TRUE.” The NLP identifies this criterion as such using semantic markup information such as <div itemscope itemtype=“https://schema.org/SomeProducts”itemid=“#product”>.”
At 606, the web page modification manager 260 applies a modification (e.g., modification data 236) to the structured data element satisfying the criterion to produce a modified presentation of the web page (e.g., modified web page element 160) as displayed by the web browser. For example, once the structured data element satisfying the criterion and the modification have been identified, the web page element modification manager 260 performs the prescribed modification (e.g., emphasis operation) on a displayed representation of the structured data element, e.g., highlighting, checking, bolding, italicizing, blinking, reordering, and the like.
In some implementations, the page modification manager 260 applies modifications selected from a library of modifications. For example, such a library may have available for selection modifications such as those listed above: highlighting, checking, bolding, italicizing, blinking, and the like.
As shown in
Computing device 700 includes a processor 702, memory 704, a storage device 706, a high-speed interface 708 connecting to memory 704 and high-speed expansion ports 710, and a low speed interface 712 connecting to low speed bus 714 and storage device 706. Each of the components 702, 704, 706, 708, 710, and 712, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 702 can process instructions for execution within the computing device 700, including instructions stored in the memory 504 or on the storage device 706 to display graphical information for a GUI on an external input/output device, such as display 716 coupled to high speed interface 708. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices 700 may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).
The memory 704 stores information within the computing device 700. In one implementation, the memory 704 is a volatile memory unit or units. In another implementation, the memory 704 is a non-volatile memory unit or units. The memory 704 may also be another form of computer-readable medium, such as a magnetic or optical disk.
The storage device 706 is capable of providing mass storage for the computing device 700. In one implementation, the storage device 706 may be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. A computer program product can be tangibly embodied in an information carrier. The computer program product may also contain instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 704, the storage device 506, or memory on processor 702.
The high speed controller 708 manages bandwidth-intensive operations for the computing device 700, while the low speed controller 712 manages lower bandwidth-intensive operations. Such allocation of functions are examples only. In one implementation, the high-speed controller 708 is coupled to memory 704, display 716 (e.g., through a graphics processor or accelerator), and to high-speed expansion ports 710, which may accept various expansion cards (not shown). In the implementation, low-speed controller 712 is coupled to storage device 706 and low-speed expansion port 714. The low-speed expansion port, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.
The computing device 700 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 720, or multiple times in a group of such servers. It may also be implemented as part of a rack server system 724. In addition, it may be implemented in a personal computer such as a laptop computer 722. Alternatively, components from computing device 700 may be combined with other components in a mobile device (not shown), such as device 750. Each of such devices may contain one or more of computing device 700, 750, and an entire system may be made up of multiple computing devices 700, 750 communicating with each other.
Computing device 750 includes a processor 752, memory 764, an input/output device such as a display 754, a communication interface 766, and a transceiver 768, among other components. The device 750 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 750, 752, 764, 754, 766, and 768, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.
The processor 752 can execute instructions within the computing device 750, including instructions stored in the memory 764. The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor may provide, for example, for coordination of the other components of the device 750, such as control of user interfaces, applications run by device 750, and wireless communication by device 750.
Processor 752 may communicate with a user through control interface 758 and display interface 756 coupled to a display 754. The display 754 may be, for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display) or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface 756 may comprise appropriate circuitry for driving the display 754 to present graphical and other information to a user. The control interface 758 may receive commands from a user and convert them for submission to the processor 752. In addition, an external interface 760 may be provided in communication with processor 752, so as to enable near area communication of device 750 with other devices. External interface 760 may provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.
The memory 764 stores information within the computing device 750. The memory 764 can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Expansion memory 774 may also be provided and connected to device 750 through expansion interface 772, which may include, for example, a SIMM (Single In Line Memory Module) card interface. Such expansion memory 774 may provide extra storage space for device 750 or may also store applications or other information for device 750. Specifically, expansion memory 774 may include instructions to carry out or supplement the processes described above and may include secure information also. Thus, for example, expansion memory 774 may be provided as a security module for device 750 and may be programmed with instructions that permit secure use of device 750. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.
The memory may include, for example, flash memory and/or NVRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 764, expansion memory 774, or memory on processor 752, that may be received, for example, over transceiver 768 or external interface 762.
Device 750 may communicate wirelessly through communication interface 766, which may include digital signal processing circuitry where necessary. Communication interface 766 may provide for communications under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others. Such communication may occur, for example, through radio-frequency transceiver 768. In addition, short-range communication may occur, such as using a Bluetooth, WiFi, or other such transceiver (not shown). In addition, GPS (Global Positioning System) receiver module 770 may provide additional navigation- and location-related wireless data to device 750, which may be used as appropriate by applications running on device 750.
Device 750 may also communicate audibly using audio codec 760, which may receive spoken information from a user and convert it to usable digital information. Audio codec 760 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of device 750. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may also include sound generated by applications operating on device 750.
The computing device 750 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone 780. It may also be implemented as part of a smart phone 782, personal digital assistant, or other similar mobile device.
Distributed computing system 800 may include any number of computing devices 880. Computing devices 880 may include a server or rack servers, mainframes, etc. communicating over a local or wide-area network, dedicated optical links, modems, bridges, routers, switches, wired or wireless networks, etc.
In some implementations, each computing device may include multiple racks. For example, computing device 880a includes multiple racks 858a-858n. Each rack may include one or more processors, such as processors 852a-852n and 862a-862n. The processors may include data processors, network attached storage devices, and other computer controlled devices. In some implementations, one processor may operate as a master processor and control the scheduling and data distribution tasks. Processors may be interconnected through one or more rack switches 858, and one or more racks may be connected through switch 878. Switch 878 may handle communications between multiple connected computing devices 800.
Each rack may include memory, such as memory 854 and memory 864, and storage, such as 856 and 866. Storage 856 and 866 may provide mass storage and may include volatile or non-volatile storage, such as network-attached disks, floppy disks, hard disks, optical disks, tapes, flash memory or other similar solid state memory devices, or an array of devices, including devices in a storage area network or other configurations. Storage 856 or 866 may be shared between multiple processors, multiple racks, or multiple computing devices and may include a computer-readable medium storing instructions executable by one or more of the processors. Memory 854 and 864 may include, e.g., volatile memory unit or units, a non-volatile memory unit or units, and/or other forms of computer-readable media, such as a magnetic or optical disks, flash memory, cache, Random Access Memory (RAM), Read Only Memory (ROM), and combinations thereof. Memory, such as memory 854 may also be shared between processors 852a-852n. Data structures, such as an index, may be stored, for example, across storage 856 and memory 854. Computing device 800 may include other components not shown, such as controllers, buses, input/output devices, communications modules, etc.
An entire system, such as electronic environment 100, may be made up of multiple computing devices 800 communicating with each other. For example, device 880a may communicate with devices 880b, 880c, and 880d, and these may collectively be known as electronic environment 100. As another example, processing circuitry 120 of
In some implementations, the request includes a natural language query and the structured data elements of the web page elements include semantic markup information; in such implementations, analyzing the web page elements includes determining a term in the natural language query based on the semantic markup information.
In some implementations, the modification is obtained from a library of modifications available to the user. In some implementations, the library of modifications includes a highlighting modification.
In some implementations, the improved techniques include determining that the criterion includes an element corresponding to information in an external dataset; obtaining data from the external dataset that corresponds to the element; and using the data from the external dataset to identify the structured data element that satisfies the criterion. In some implementations, the external dataset is obtained from a server different from a server transmitting the web page elements to the web browser. In some implementations, the external dataset is associated with a profile of the user. In some implementations, the external dataset is associated with a search engine index. In some implementations, the external dataset includes an open browser tab in the web browser.
In some implementations, the external dataset includes private data originating from a private data resource. In some implementations, the web page elements are received from a web page server, and the improved techniques include preventing the private data from being transmitted to the web page server.
In some implementations, analyzing the web page elements includes identifying the structured data elements from the web page elements by identifying table elements, list elements, semantic markup elements, or accessibility data; and comparing the structured data elements to the criterion. In some implementations, the request further identifies an emphasis operation, the modified user interface applying the emphasis operation to the structured data element satisfying the criterion. In some implementations, the emphasis operation maps to one operation of a plurality of predefined emphasis operations. In some implementations, the web page elements include a table with table elements and the structured data element satisfying the criterion is in a region of the table. In such implementations, producing the modified user interface further includes changing an appearance of the region. In some implementations, the web page elements are associated with a document model that includes a set of nodes representing the structured data elements of the web page elements. In such implementations, modifying the structured data element includes editing a node of the document model represented by the structured data element that satisfies the criterion to change an appearance of the structured data element.
In some implementations, obtaining data from the external dataset that corresponds to the element includes identifying a uniform resource locator (URL) associated with the element; and accessing the data at a web resource determined from the URL.
In some implementations, the user belongs to an organization; and obtaining data from the external dataset that corresponds to the element includes receiving the data from a resource associated with the organization at a point in time prior to receiving the query data
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.
These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” “computer-readable medium” refers to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), and the Internet.
The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the specification.
It will also be understood that when an element is referred to as being on, connected to, electrically connected to, coupled to, or electrically coupled to another element, it may be directly on, connected or coupled to the other element, or one or more intervening elements may be present. In contrast, when an element is referred to as being directly on, directly connected to or directly coupled to another element, there are no intervening elements present. Although the terms directly on, directly connected to, or directly coupled to may not be used throughout the detailed description, elements that are shown as being directly on, directly connected or directly coupled can be referred to as such. The claims of the application may be amended to recite example relationships described in the specification or shown in the figures.
While certain features of the described implementations have been illustrated as described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the scope of the implementations. It should be understood that they have been presented by way of example only, not limitation, and various changes in form and details may be made. Any portion of the apparatus and/or methods described herein may be combined in any combination, except mutually exclusive combinations. The implementations described herein can include various combinations and/or sub-combinations of the functions, components and/or features of the different implementations described.
In addition, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. In addition, other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Accordingly, other implementations are within the scope of the following claims.
