Patent Application
20180081973
The present application relates generally to computers and computer applications, and more particularly to automatic triggering of search queries.
Plethora of electronic information currently exist, which users may consume, for example, by going online, navigating the World Wide Web or the Internet. For instance, information may be available in the form of electronic books, web site, or even a text generated from a system transcribing speech in a meeting. The vast amount of available information also comes with information that may not be understandable by users. For example, there may be text such as words, phrases, expressions that is not known to a viewing individual, and make it difficult for the individual to fully comprehend the content. Searching for dictionary may be time consuming and may not be consistent with a particular user's knowledge.
A method and system of automatically triggering a search query may be provided. The method, in one aspect, may monitor a user interaction with a computer device, the user interaction comprising at least user consuming material via the computer device. The method may also include identifying content of the material the user is consuming. The method may further include capturing via at least one sensor device, a physiological pattern generated by the user while consuming the content. The method may also include, responsive to detecting that the physiological pattern represents an outlier, determining a term in the content for providing additional information, the term determined from the content based on user knowledge, frequency of the term appearing in previous content consumed by the user and popularity of the term. The method may also include executing a search query with the determined term. The method may further include filtering a result of the search query based on the user's knowledge. The method may also include presenting via the computer device the filtered result and at least one of the previous content containing the term the user has previously consumed.
A system of automatically triggering a search query, in one aspect, may include at least one hardware processor. A consumed content database stores summaries of consumed content and may be coupled to the hardware processor. The hardware processor may monitor a user interaction with a display device, the user interaction comprising at least user consuming material displayed on the display device. The hardware processor may identify content of the material the user is consuming. The hardware processor may capture via at least one sensor device, a physiological pattern generated by the user while consuming the content. The hardware processor, responsive to detecting that the physiological pattern represents an outlier, may determine a term in the content for providing additional information, the term determined from the content based on user knowledge, frequency of the term appearing in previous content consumed by the user and popularity of the term. The hardware processor may execute a search query with the determined term in the consumed content database and an internet. The hardware processor may filter a result of the search query based on the user's knowledge. The hardware processor may present the filtered result and at least one of the previous content containing the term the user has previously consumed via the display device.
A computer readable storage medium storing a program of instructions executable by a machine to perform one or more methods described herein also may be provided.
Further features as well as the structure and operation of various embodiments are described in detail below with reference to the accompanying drawings. In the drawings, like reference numbers indicate identical or functionally similar elements.
In one aspect, a system and method are disclosed that detect behavioral and physiological patterns of a user while the user is consuming content. Behavioral and physiological patterns of a user may be detected that indicate that the user does not know a term or requires additional information in order to fully understand the content being consumed. Examples of behavioral and physiological patterns may include but are not limited to pupil dilation, heart beat frequency, skin conductance, for example, detected via a sensor device. Examples of the content the user consumes may include but are not limited to text, image, audio, and video. The system and method in one embodiment may catalog the behavioral and physiological patterns and trigger search queries to provide unknown information automatically. The queries may be customized based on the user's prior knowledge. The system and method in one embodiment generates a personalized search query based on physiological and/or behavioral patterns. The personalized search query may be constructed also based on the user's knowledge.
The system and method may include cataloging physiological and/or behavioral patterns while users are consuming content and collecting terms and/or subjects that the user has familiarity with. Responsive to detecting a physiological or behavioral pattern related to information finding actions, the system and method of the present disclosure in one embodiment triggers search queries considering definitions for the identified unknown terms and personalized examples of the unknown term combined with subjects that the user is familiar with.
The system and method of the present disclosure in one embodiment may implement a machine learning technique to detect a physiological pattern. The system, for example, in one embodiment may retrieve content from sources which are consumed by the user. The system may verify the frequency with which each word is visualized by the user. For example, if a user enjoys cooking, the user may visit several sites with recipes. In these recipes, common ingredients may be there such as “salt”, “pepper”, and others. The system can determine the frequency of these words or terms occurring in the website and create a database with words or terms, and the frequency with which they appear in the documents consumed by the user. For instance, words or terms occurring for a threshold number of times may be selected. These databases are used for machine learning techniques as they require data to increase their accuracy.
At 104, content of the material the user is consuming is identified. The content may be identified, for example, based on the data that is displayed on a screen or display device the user is viewing. The content may be also identified by following or detecting user's eye sight or field of view, which may be sensed via a sensor device. A system of the present disclosure, for example, may capture the content being consumed by the user, for example, part of document appearing on the screen, for example, by utilizing eye sight detection sensor to detect the area of the document on the screen the user is focusing on. As an example, cameras from mobile devices such as smartphones, tablets, laptops can be used for eye tracking.
At 106, physiological pattern generated by the user while consuming the content is captured via at least one sensor device. For example, a system of the present disclosure in one embodiment may capture behavioral and/or physiological pattern(s) generated by the user using a sensor for measuring the behavioral and/or physiological pattern. Such sensor or sensors may be deployed on stand-alone devices, for example, eye tracker sensor, microphone, or on the device being used by the user to consume content, for example, sensors of smartphone.
The system in one embodiment may identify via one or more sensors abnormal behavioral and/or physiological pattern or patterns, for example, mouse device movement, pupil dilation, heart beat frequency, skin conductance, generated by the user while the user consumes the content. In another aspect, the system is trained, for example, via classification algorithm or another machine learning algorithm to identify configurations of physiological pattern or patterns that indicate whether that particular the user does not know a term or requires additional information in order to fully understand the content being consumed. For instance, while a user is reading a document, a machine learning algorithm can detect whether the user is having difficulty understanding terms the user is not familiar with. As an example, a machine learning technique can use the following features: speed to consume a sentence (based on eye tracking) and complexity of the sentence (number of terms user is not familiar with based on previous consumed material). While the user consumes the material, the system in one embodiment verifies the speed the user is reading sentences. If for a given sentence the speed is slower than usual for that user and the complexity of the terms in the sentence (for that user) is higher than what the user knows, a query to look up the meaning of the term (the user has little or no familiarity) is triggered.
At 108, responsive to detecting that the physiological pattern represents an outlier, a term in the content is determined for providing additional information. The term is determined based on user's knowledge, word frequency, and popularity of the term. For instance, the system in one embodiment may capture the content being presented to the user and identify the pieces which are more likely to require clarifications. I one embodiment, one or more terms may be selected from the content based on user knowledge, frequency of the term or terms appearing in previous content consumed by the user and popularity of the term or terms. As another example, the system may identify one or more words in the region being visualized by the user with smallest visualization frequency, for example, as determined in a training phase. For example, the words in the region which the user has not previously seen or infrequently seen (as determined based on a threshold frequency value) may be input into a search operation or search engine.
At 110, a search query is executed with the determined term. For example, based on identified content that may need more clarification, the system in one embodiment may trigger a search operation using those elements (e.g., term or terms) as input data. In one embodiment, executing a search query may include performing a search for the terms in sources such as content that has been already consumed and saved previously, performing an Internet search, for example, in encyclopedia web sites, social networks, web sites having books, web sites having publications, other Internet sites, and other sources.
At 112, the result returned from the search query may be filtered based on user's knowledge and/or background, for example, so as to provide a focused result that helps the user's understanding.
At 114, the filtered result may be presented. Previous content the user has previously consumed and that contains the term may be presented via the computer device also. The system in one embodiment may present clarification messages to the user, which for example, include a result or results from search queries. For example, the system in one embodiment may return the meaning or synonym(s) of the term or word together with one or more examples containing the word that have been visualized by the user before, for example, to help the user to recall previous understanding of the word. The meaning of the term and previous content that includes the term (also collectively referred to as additional information) may be presented, for example, by presenting an interactive element on the computer device, or on the user interface of the computer device, for example, a display on which the user is consuming the content. Examples of the interactive element, for example, a user interface element, may include a slide bar, tooltips icon, or another user interface element embedded or incorporated into the user interface display. In one embodiment, responsive to detecting that the interactive element is selected, the additional information may be presented or displayed on the user interface display or display device.
In one embodiment, the result of the query may be filtered or sorted according to the user's knowledge of the term or subject. For instance, if a query is triggered for a person with information technology (IT) background, the system in one embodiment may identify an explanation and/or example where the query result appears in this context, for example, providing results of already consumed content, which can be referenced for easier understanding. For example, queries involving the word “Cloud” will have as results phrases about Cloud Computing and virtualization, for example, with links to sites that provide the information such as thesaurus and encyclopedia sites, with previously read IT text. The result will not have phrases such as “a visible collection of particles of water”. In one embodiment, the system may adjust the depth of the content presented to the user based on the behavioral and/or physiological signals. For instance, if the number of user's blinks per time period exceeds a threshold value, for example, with an accelerated heart beat exceeding a threshold value, more comprehensive explanation may be presented to the user than would have been presented to a user who for example, blinked less the a threshold number of times when viewing a given unknown term or a region where the given unknown term is shown on a display.
In one embodiment, the system may trigger a query proactively, considering physiological feedback, users' knowledge and popularity of the terms. The system in one embodiment may identify automatically (or anticipate automatically) search requests the user may conduct based on sensed data. The level of user's own knowledge may be identified and used in determining the way the query results are presented, for example, a term is explained using an example that easier for the user to understand. Thus, in one embodiment, the system may explore user's knowledge for selecting information and presenting the information to the user.
At 116, it is determined whether the user accessed the additional information, for example, the filtered result with the previous content that may provide clarifications for the user's understanding of the content being consumed. For instance, it is detected whether the interactive element such as the slide bar, tooltip or another user interface element has been selected or pressed, for example, by user using an input device, responsive to which selection the additional information is displayed or presented for user access.
At 118, responsive to detecting that the user accessed the additional information, consumption of the additional information by the user is registered. For example, the information that the user consumed the additional information may be stored in a database. The database may also store the additional information.
At 206, the generated content summary is stored in a database in a storage device.
At 208, interaction and physiological traces while consuming content are detected, for example, via one or more sensors, and stored in an interaction and physiological traces database in a storage device. For example, mouse movements while the user is consuming (e.g., viewing) content may be detected and stored. For instance, the amount of movement of the mouse, the speed of movement, the mouse movement or displacement pattern with timing, may be detected and stored. Heart beat of the user, eye movement, pupil dilation, and/or skin conductance while consuming content, may be also detected and stored. For example, an operating system of the computer device with which the user is interacting, may constantly or continuously capture information associated with the user's interaction, for example, with touch screens, and mouse movement on a display device. In one embodiment, each event is captured together with its type description and timestamp. Other information may also be captured. As another example, biometric sensors may also collect periodically information about key performance indicators (KPIs) being measured. For instance, an eye movement tracker identifies the position in the screen being gazed by the user as well as the moment (or time-stamp) in which this event is taking place; heart beat sensors register periodically (e.g., every second) the current frequency. With time stamps, the system one can cross-relate the collected information in order to infer the biological reaction while consuming a specific piece of content.
At 210, patterns involving content, interaction, and physiological data are identified and stored. For instance, a user reading a text having terms the user is familiar with may perform this task while blinking 20 times per minute and with 90 heart beats per minute. In one embodiment, average values for metrics captured by other sensors can be similarly collected and inferred. The identified patterns are used as a basis for triggering search queries. For instance, based on the detected physiological patterns, an average physiological pattern may be established for the user. Any future physiological pattern that deviates from the average physiological pattern may be detected as an outlier. For instance, if significant deviations from the average values associated with the user are identified, an outlier is detected. Whether a deviation is significant may be determined based on comparing the deviation with a threshold value. For example, if the deviation exceeds the threshold value, it is determined that the deviating pattern is an outlier. Such threshold value may be configurable.
In one embodiment, the hardware processor 302 presents an interactive user interface element 314 on the display device 306, and responsive to detecting a selection of the interactive user interface element, the hardware processor 302 presents the filtered result and at least one of the previous content containing the term the user has previously consumed.
The additional information database 312 stores registration of consumption of additional information and coupled to the hardware processor. Responsive to detecting a selection of the interactive user interface element, the hardware processor 302 registers and stores user consumption of the filtered result and at least one of the previous content containing the term the user has previously consumed in the additional information database 312.
Once the terms are filtered and ranked (e.g., shown in
As another example, consider the following education scenario. The system in one embodiment may monitor user interaction with a device, for example, a user using an augmented reality device (e.g., a glass) for reading a book or a user interacting with multiple materials (e.g., book, blackboard) during a class. The system may detect behavioral or physiological pattern from the user in encountering a region in the book, detect unknown terms in the region and trigger search for explanations and/or definitions of the unknown terms. For example, the system may detect outliers in the consumption of content indicating probable unknown terms. The system may trigger a search for explanations and/or definitions and place the found information adjacent to the unknown terms, supporting the user in the task of understanding.
Yet another example may be in a meeting scenario. The system in one embodiment may capture audio that generates the transcription of the meeting in real-time. Based on the knowledge of the user, the system may identify terms and/or expressions the user is likely to need additional information. For example, the knowledge of the user is inferred based on the corpus the user consumes; more specifically, for each word, the number of times a user saw that content is used as KPI to indicate familiarity.
The system may trigger a search considering the identified term or terms and presents additional information to the user, for example, as a notification in the user's smartphone.
Still another use scenario may include using text and/or video during a travel. Consider that a user is using an augmented reality device (e.g., a glass). During the travel the user interacts with multiple materials (e.g., signs, texts). The system, for example, coupled to the augmented reality device, in one embodiment may detect outliers in the consumption of content indicating unknown terms or probable unknown terms. The system may trigger a search for explanation and/or definition of the unknown terms, placing the additional information next to unknown terms.
The computer system may be described in the general context of computer system executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. The computer system may be practiced in distributed cloud computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.
The components of computer system may include, but are not limited to, one or more processors or processing units 12, a system memory 16, and a bus 14 that couples various system components including system memory 16 to processor 12. The processor 12 may include a module 30 that performs the methods described herein. The module 30 may be programmed into the integrated circuits of the processor 12, or loaded from memory 16, storage device 18, or network 24 or combinations thereof.
Bus 14 may represent one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnects (PCI) bus.
Computer system may include a variety of computer system readable media. Such media may be any available media that is accessible by computer system, and it may include both volatile and non-volatile media, removable and non-removable media.
System memory 16 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) and/or cache memory or others. Computer system may further include other removable/non-removable, volatile/non-volatile computer system storage media. By way of example only, storage system 18 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (e.g., a “hard drive”). Although not shown, a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a “floppy disk”), and an optical disk drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media can be provided. In such instances, each can be connected to bus 14 by one or more data media interfaces.
Computer system may also communicate with one or more external devices 26 such as a keyboard, a pointing device, a display 28, etc.; one or more devices that enable a user to interact with computer system; and/or any devices (e.g., network card, modem, etc.) that enable computer system to communicate with one or more other computing devices. Such communication can occur via Input/Output (I/O) interfaces 20.
Still yet, computer system can communicate with one or more networks 24 such as a local area network (LAN), a general wide area network (WAN), and/or a public network (e.g., the Internet) via network adapter 22. As depicted, network adapter 22 communicates with the other components of computer system via bus 14. It should be understood that although not shown, other hardware and/or software components could be used in conjunction with computer system. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.
The present invention may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.
The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.
Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.
Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.
Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.
These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
The corresponding structures, materials, acts, and equivalents of all means or step plus function elements, if any, in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.
