Patent Grant
11720324
This application is a 371 of International Application No. PCT/KR2019/000039 filed on Jan. 2, 2019, which claims priority to Korean Patent Application No. 10-2018-0001862 filed on Jan. 5, 2018, the disclosures of which are herein incorporated by reference in their entirety.
Various embodiments of the present invention relate to an apparatus and a method for displaying an electronic document for processing a user's voice command in an electronic device.
With development of information communication technology and semiconductor technology, an electronic device may provide various functions such as a broadcast function, a wireless Internet function, a camera function, a user authentication function, or an electronic payment function. The electronic device may receive a user's voice command through an input device (e.g., a microphone) and provide a function corresponding to the user's voice command so that the user may use various functions more easily.
An electronic device may display an electronic document (e.g., a web page) including a select field (e.g., a login select field) for executing a function connected to an input field (e.g., an ID, a password input field) where a user may input data. In addition, the electronic device may analyze a use's voice command and thus display data in the input field of the electronic document or execute the function corresponding to the select field. However, since the electronic document displays a plurality of contents together besides information (e.g., a text, an image) related to the input field or the select field, the user may have difficulty in accurately identifying the voice command indicating the input field or the select field. In addition, as the user does not accurately obtain the voice command corresponding to the function provided by the electronic document, the electronic device may receive the user's voice command which is input ambiguously, and may not accurately obtain a user's intent which matches a voice command received using information displayed on the electronic document.
Various embodiments of the present invention may provide an apparatus and a method for displaying an electronic document including guide information for processing a user's voice command in an electronic device.
According to various embodiments of the invention, an electronic device includes an input device; a display; and a processor, wherein the processor may be configured to detect a voice command of a user using the input device, if outputting an electronic document corresponding to the voice command, identify at least one input field in the electronic document, determine guide information based on information of the at least one input field, and display the electronic document including the guide information using the display.
According to various embodiments of the invention, an operating method of an electronic device may include detecting a voice command of a use, if outputting an electronic document corresponding to the voice command, identifying at least one input field in the electronic document, determining guide information based on information of the at least one input field, and displaying the electronic document including the guide information.
An electronic device according to various embodiments may enable a user to easily input a voice command by displaying an electronic document including guide information, and accurately obtain a user's intent corresponding to the voice command.
Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings. In describing the embodiment of the present invention, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. Terms to be described are terms defined in consideration of the functions of the present invention, which may vary according to a user's or operator's intent or practice. Hence, their definition should be made based on contents throughout this specification.
The processor 120 may execute, for example, software (e.g., a program 140) to control at least one other component (e.g., a hardware or software component) of the electronic device 101 coupled with the processor 120, and may perform various data processing or computation. According to one embodiment, as at least part of the data processing or computation, the processor 120 may load a command or data received from another component (e.g., the sensor module 176 or the communication module 190) in volatile memory 132, process the command or the data stored in the volatile memory 132, and store resulting data in non-volatile memory 134. According to an embodiment, the processor 120 may include a main processor 121 (e.g., a central processing unit (CPU) or an application processor (AP)), and an auxiliary processor 123 (e.g., a graphics processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operable independently from, or in conjunction with, the main processor 121. Additionally or alternatively, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or to be specific to a specified function. The auxiliary processor 123 may be implemented as separate from, or as part of the main processor 121.
The auxiliary processor 123 may control at least some of functions or states related to at least one component (e.g., the display device 160, the sensor module 176, or the communication module 190) among the components of the electronic device 101, instead of the main processor 121 while the main processor 121 is in an inactive (e.g., sleep) state, or together with the main processor 121 while the main processor 121 is in an active state (e.g., executing an application). According to an embodiment, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., the camera module 180 or the communication module 190) functionally related to the auxiliary processor 123. The memory 130 may store various data used by at least one component (e.g., the processor 120 or the sensor module 176) of the electronic device 101. The various data may include, for example, software (e.g., the program 140) and input data or output data for a command related thereto. The memory 130 may include the volatile memory 132 or the non-volatile memory 134.
The program 140 may be stored in the memory 130 as software, and may include, for example, an operating system (OS) 142, middleware 144, or an application 146.
The input device 150 may receive a command or data to be used by other component (e.g., the processor 120) of the electronic device 101, from the outside (e.g., a user) of the electronic device 101. The input device 150 may include, for example, a microphone, a mouse, or a keyboard.
The sound output device 155 may output sound signals to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker may be used for general purposes, such as playing multimedia or playing record, and the receiver may be used for an incoming calls. According to an embodiment, the receiver may be implemented as separate from, or as part of the speaker.
The display device 160 may visually provide information to the outside (e.g., a user) of the electronic device 101. The display device 160 may include, for example, a display, a hologram device, or a projector and control circuitry to control a corresponding one of the display, hologram device, and projector. According to an embodiment, the display device 160 may include touch circuitry adapted to detect a touch, or sensor circuitry (e.g., a pressure sensor) adapted to measure the intensity of force incurred by the touch.
The audio module 170 may convert a sound into an electrical signal and vice versa. According to an embodiment, the audio module 170 may obtain the sound via the input device 150, or output the sound via the sound output device 155 or a headphone of an external electronic device (e.g., an electronic device 102) directly (e.g., wired) or wirelessly coupled with the electronic device 101.
The sensor module 176 may detect an operational state (e.g., power or temperature) of the electronic device 101 or an environmental state (e.g., a state of a user) external to the electronic device 101, and then generate an electrical signal or data value corresponding to the detected state. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.
The interface 177 may support one or more specified protocols to be used for the electronic device 101 to be coupled with the external electronic device (e.g., the electronic device 102) directly (e.g., wired) or wirelessly. According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.
A connecting terminal 178 may include a connector via which the electronic device 101 may be physically connected with the external electronic device (e.g., the electronic device 102). According to an embodiment, the connecting terminal 178 may include, for example, a HDMI connector, a USB connector, a SD card connector, or an audio connector (e.g., a headphone connector),
The haptic module 179 may convert an electrical signal into a mechanical stimulus (e.g., a vibration or a movement) or electrical stimulus which may be recognized by a user via his tactile sensation or kinesthetic sensation. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electric stimulator.
The camera module 180 may capture a still image or moving images. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.
The power management module 188 may manage power supplied to the electronic device 101. According to one embodiment, the power management module 188 may be implemented as at least part of, for example, a power management integrated circuit (PMIC).
The battery 189 may supply power to at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a primary cell which is not rechargeable, a secondary cell which is rechargeable, or a fuel cell.
The communication module 190 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 101 and the external electronic device (e.g., the electronic device 102, the electronic device 104, or the server 108) and performing communication via the established communication channel. The communication module 190 may include one or more communication processors that are operable independently from the processor 120 (e.g., the application processor (AP)) and supports a direct (e.g., wired) communication or a wireless communication. According to an embodiment, the communication module 190 may include a wireless communication module 192 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). A corresponding one of these communication modules may communicate with the external electronic device via the first network 198 (e.g., a short-range communication network, such as BLUETOOTH, wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA)) or the second network 199 (e.g., a long-range communication network, such as a cellular network, the Internet, or a computer network (e.g., LAN or wide area network (WAN)). These various types of communication modules may be implemented as a single component (e.g., a single chip), or may be implemented as multi components (e.g., multi chips) separate from each other.
The wireless communication module 192 may identify and authenticate the electronic device 101 in a communication network, such as the first network 198 or the second network 199, using subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the subscriber identification module 196.
The antenna module 197 may transmit or receive a signal or power to or from the outside (e.g., the external electronic device) of the electronic device 101. According to an embodiment, the antenna module 197 may include one or more antennas, and, therefrom, at least one antenna appropriate for a communication scheme used in the communication network, such as the first network 198 or the second network 199, may be selected, for example, by the communication module 190 (e.g., the wireless communication module 192). The signal or the power may then be transmitted or received between the communication module 190 and the external electronic device via the selected at least one antenna.
At least some of the above-described components may be coupled mutually and communicate signals (e.g., commands or data) therebetween via an inter-peripheral communication scheme (e.g., a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)).
According to an embodiment, commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 via the server 108 coupled with the second network 199. Each of the electronic devices 102 and 104 may be a device of a same type as, or a different type, from the electronic device 101. According to an embodiment, all or some of operations to be executed at the electronic device 101 may be executed at one or more of the external electronic devices 102, 104, or 108. For example, if the electronic device 101 should perform a function or a service automatically, or in response to a request from a user or another device, the electronic device 101, instead of, or in addition to, executing the function or the service, may request the one or more external electronic devices to perform at least part of the function or the service. The one or more external electronic devices receiving the request may perform the at least part of the function or the service requested, or an additional function or an additional service related to the request, and transfer an outcome of the performing to the electronic device 101. The electronic device 101 may provide the outcome, with or without further processing of the outcome, as at least part of a reply to the request. To that end, a cloud computing, distributed computing, or client-server computing technology may be used, for example.
Before describing an embodiment of the present invention, an integrated intelligence system to which an embodiment of the present invention may be applied will be described.
Referring to
The user terminal 100 may provide a user with a required service through an app (or an application program) (e.g., an alarm app, a message app, or a picture (gallery) app) stored in the user terminal 100. For example, the user terminal 100 may execute and operate a different app through an intelligence app (or, a voice recognition app) stored in the user terminal 100. The user terminal 100 may receive a user input for executing and operating the different app through the intelligence app of the user terminal 100. The user input may be received through, for example, a physical button, a touch pad, a voice input, or a remote input. According to an embodiment, the user terminal 100 may a device of various types. The electronic device may include, for example, at least one of a portable communication device (e.g., a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. The user terminal 100 according to the embodiment of the present disclosure is not limited to the aforementioned devices.
According to an embodiment, the user terminal 100 may receive a user's speech as a user input. The user terminal 100 may receive the user's speech and generate a command to operate the app based on the user's speech. Hence, the user terminal 100 may operate the app using the command.
According to an embodiment, the intelligence server 200 may receive a user voice input from the user terminal 100 through a communication network (e.g., the network 199 of
The term “path rule” of the present disclosure may generally indicate, but is not limited to, a sequence of states for an electronic device to perform a task requested by the user. For example, the path rule may include information relating to the sequence of the states. The task may be, for example, any action which the intelligence app may provide. The task may include creating a schedule, sending a picture to a desired counterpart, or providing weather information. The user terminal 100 may perform the task, by sequentially having at least one or more states (e.g., action states of the user terminal 100).
According to an embodiment, the path rule may be provided or generated by an artificial intelligent (AI) system. The AI system may be a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN)), or a recurrent neural networks (RNN). As another example, the AI system may be a combination thereof or other AI system. According to an embodiment, the path rule may be selected from a set of defined path rules, or may be generated in real time in response to a user request. For example, the AI system may select at least one path rule among a plurality of defined path rules, or generate a path rule dynamically (or in real time). As another example, the user terminal 100 may use a hybrid system to provide the path rule.
According to an embodiment, the user terminal 100 may execute the action, and display a screen corresponding to the state of the user terminal 100 which executes the action on a display. For another example, the user terminal 100 may execute the action, and may not display a result of performing the action on the display. The user terminal 100 may, for example, execute a plurality of actions, and display only some results of the plurality of the actions on the display. The user terminal 100 may, for example, display only a result of executing the last action on the display. As another example, the user terminal 100 may receive a user's input and display a result of executing the action on the display.
According to an embodiment, the personal information server 300 may include a database which stores user information. For example, the personal information server 300 may receive user information (e.g., context information, or application execution) from the user terminal 100 and store it in the database. The intelligence server 200 may receive the user information from the personal information server 300 through a communication network and use it to generate a path rule for the user input. According to an embodiment, the user terminal 100 may receive user information from the personal information server 300 over the communication network and use it as information for managing the database.
According to an embodiment, the suggestion server 400 may include a database which stores information relating to a function in a terminal or introduction of an application or a function to be provided. For example, the suggestion server 400 may receive user information of the user terminal 100 from the personal information server 300 and include a database of functions which the user may use. The user terminal 100 may receive information of the function to be provided from the suggestion server 400 over the communication network and provide the information to the user.
Referring to
According to an embodiment, the input module 110 may receive a user input from the user. For example, the input module 110 may receive the user input from a connected external device (e.g., a keyboard or a headset). As another example, the input module 110 may include a touch screen (e.g., a touch screen display) coupled with the display 161. As another example, the input module 110 may include a hardware key (or, a physical key) positioned in the user terminal 100 (or, the housing of the user terminal 100).
According to an embodiment, the input module 110 may include a microphone for receiving a user's speech as a voice signal. For example, the input module 110 includes a speech input system, and may receive a user's speech as a voice signal through the speech input system. The microphone may be exposed through, for example, a part (e.g., a first part) of the housing.
According to an embodiment, the display 161 may display an execution screen of an image or a video, and/or an application. For example, the display 161 may display a graphic user interface (GUI) of an app. According to an embodiment, the display 161 may be exposed through a part (e.g., a second part) of the housing.
According to an embodiment, the speaker 156 may output a voice signal. For example, the speaker 156 may output the voice signal generated in the user terminal 100 to outside. According to an embodiment, the speaker 156 may be exposed through a part (e.g., a third part) of the housing.
According to an embodiment, the memory 130 may store a plurality of apps (or, application programs) 131 and 133. The plurality of the apps 131 and 133 may be, for example, programs for performing a function corresponding to a user's input. According to an embodiment, the memory 130 may store an intelligence agent 135, an execution manager module 137 or an intelligence service module 139. The intelligence agent 135, the execution manager module 137, or the intelligence service module 139 may be, for example, a framework (or an application framework) for processing the received user input (e.g., the user speech).
According to an embodiment, the memory 130 may include a database for storing information required to recognize a user input. For example, the memory 130 may include a log database capable for storing log information. As another example, the memory 130 may include a persona database for storing user information.
According to an embodiment, the memory 130 may store the plurality of the apps 131 and 133, and the plurality of the apps 131 and 133 may be loaded and executed in the memory 130. For example, the plurality of the apps 131 and 133 stored in the memory 130 may be loaded and driven by the execution manager module 137. The plurality of the apps 131 and 133 may include execution service modules 131a and 133a for performing functions. In an embodiment, the plurality of the apps 131 and 133 may execute a plurality of actions (e.g., a sequence of states) 131b and 133b through the execution service modules 131a and 133a to perform functions. In other words, the execution service modules 131a and 133a may be enabled by the execution manager module 137, and may execute the plurality of the actions 131b and 133b.
According to an embodiment, if the actions 131b and 133b of the plurality of the apps 131 and 133 are executed, an execution state screen according to the execution of the actions 131b and 133b may be displayed on the display 161. The execution state screen may be, for example, a screen in which the actions 131b and 133b are completed. The execution state screen may be, as another example, a screen in which the execution of the actions 131b and 133b is partial landing (e.g., if parameters necessary for the actions 131b and 133b are not input).
According to an embodiment, the execution service modules 131a and 133a may execute the actions 131b and 133b according to a path rule. For example, the execution service modules 131a and 133a may be enabled by the execution manager module 137, receive an execution request according to the path rule from the execution manager module 137, and execute functions of the plurality of the apps 131 and 133, by conducting the actions 131b and 133b in response to the execution requests. If the executions of the actions 131b and 133b are completed, the execution service modules 131a and 133a may transfer completion information to the execution manager module 137.
According to an embodiment, if the plurality of the actions 131b and 133b is executed in the plurality of the apps 131 and 133, the plurality of the actions 131b and 133b may be executed in sequence. If the execution of one action (e.g., action 1 of the first app 131, or action 1 of the second app 133) is completed, the execution service modules 131a and 133a may open a next action (e.g., action 2 of the first app 131, or action 2 of the second app 133), and transmit completion information to the execution manager module 137. Herein, opening an arbitrary action may be understood as switching the arbitrary action to an executable state, or preparing the execution of the arbitrary operation. In other words, if the arbitrary action is not opened, the corresponding action may be not executed. If the completion information is received, the execution manager module 137 may transmit an execution request for a next action (e.g., action 2 of the first app 131, action 2 of the second app 133) to the execution service module. According to an embodiment, if the plurality of the apps 131 and 133 is executed, the plurality of the apps 131 and 133 may be sequentially executed. For example, if the execution of the last action (e.g., action 3 of the first app 131) of the first app 131 is completed and the completion information is received, the execution manager module 153 may transmit the execution request of the first action (e.g., action 1 of the second app 133) of the second app 133 to the execution service 133a.
According to an embodiment, if the plurality of the actions 131b and 133b is executed in the plurality of the apps 131 and 133, result screens according to the execution of the plurality of the executed actions 131b and 133b respectively may be displayed on the display 161. According to an embodiment, only some of the plurality of the result screens according to the execution of the plurality of the executed actions 131b and 133b may be displayed on the display 161.
According to an embodiment, the memory 130 may store an intelligence app (e.g., a voice recognition app) interworking with the intelligence agent 135. The app interworking with the intelligence agent 135 may receive and process a user's speech as a voice signal. According to an embodiment, the app interworking with the intelligence agent 135 may be operated by a particular input (e.g., an input through a hardware key, an input through a touch screen, a particular voice input) inputted through the input module 110.
According to an embodiment, the intelligence agent 135, the execution manager module 137, or the intelligence service module 139 stored in the memory 130 may be executed by the processor 120. The function of the intelligence agent 135, the execution manager module 137, or the intelligence service module 139 may be implemented by the processor 120. According to an embodiment, the function of the intelligence agent 135, the execution manager module 137, or the intelligence service module 139 may be carried out by the processor 120. According to an embodiment, the intelligence agent 135, the execution manager module 137, or the intelligence service module 139 stored in the memory 130 may be implemented by hardware as well as software.
According to an embodiment, the processor 120 may control an overall operation of the user terminal 100. For example, the processor 120 may receive a user input by controlling the input module 110. The processor 120 may display an image by controlling the display 161. The processor 120 may output a voice signal by controlling the speaker 156. The processor 120 may execute a program by controlling the memory 130, or fetch or store necessary information.
According to an embodiment, the processor 120 may execute the intelligence agent 135, the execution manager module 137, or the intelligence service module 139 stored in the memory 130. Accordingly, the processor 120 may implement the function of the intelligence agent 135, the execution manager module 137, or the intelligence service module 139.
According to an embodiment, the processor 120 may generate a command for operating an app based on a voice signal received as a user input by executing the intelligence agent 135. According to an embodiment, the processor 120 may execute the plurality of the apps 131 and 133 stored in the memory 130 according to the generated command by executing the execution manager module 137. According to an embodiment, the processor 120 may manage user's information by executing the intelligence service module 139, and process a user input using the user's information.
The processor 120 may transmit the user input received through the input module 110 to the intelligence server 200 by executing the intelligence agent 135, and process the user input through the intelligence server 200.
According to an embodiment, the processor 120 may preprocess the user input before transmitting the user input to the intelligence server 200 by executing the intelligence agent 135. According to an embodiment, to preprocess the user input, the intelligence agent 135 may include an adaptive echo canceller (AEC) module, a noise suppression (NS) module, an end-point detection (EPD) module, or an automatic gain control (AGC) module. The AEC module may cancel echo included in the user input. The NS module may suppress background noise included in the user input. The EPD module may detect an end-point of a user voice included in the user input, and find a portion where the user's voice is present using the detected end-point. The AGC module may recognize the user input, and adjust a volume of the user input to be suitable to process the recognized user input. According to an embodiment, the processor 120 may execute all the preprocessing configuration for performance, but the processor 120 in another embodiment may execute part of the preprocessing configuration to operate with low power.
According to an embodiment, the intelligence agent 135 may execute a wake up recognition module stored in the memory 130 to recognize a user's call. Accordingly, the processor 120 may recognize a wake up command of the user through the wakeup recognition module, and execute the intelligence agent 135 for receiving a user input if receiving the wake up command. The wake up recognition module may be implemented with a low-power processor (e.g., a processor included in an audio codec). According to an embodiment, if receiving a user input through a hardware key, the processor 120 may execute the intelligence agent 135. If the intelligence agent 135 is executed, an intelligence app (e.g., a voice recognition app) interworking with the intelligence agent 135 may be executed.
According to an embodiment, the intelligence agent 135 may include a voice recognition module for executing a user input. The processor 120 may recognize a user input for execute an action in the app through the voice recognition module. For example, the processor 120 may recognize a limited user (voice) input (e.g., a speech such as “click” for executing a photographing action if a camera app is running) for executing an action such as the wake up command in the plurality of the apps 131 and 133 through the voice recognition module. The processor 120 may assist the intelligence server 200 and thus recognize and rapidly process a user command processable in the user terminal 100 through the voice recognition module. According to an embodiment, the voice recognition module of the intelligence agent 135 for executing the user input may be implemented at an app processor.
According to an embodiment, the voice recognition module (including the voice recognition module of the wake up module) of the intelligence agent 135 may recognize a user input using an algorithm for recognizing a voice. The algorithm used to recognize the voice may be at least one of, for example, a hidden markov model (HMM) algorithm, an artificial neural network (ANN) algorithm, or a dynamic time warping (DTW) algorithm.
According to an embodiment, the processor 120 may convert the user's voice input to text data by executing the intelligence agent 135. For example, the processor 120 may transmit a user's voice to the intelligence server 200 through the intelligence agent 135, and receive text data corresponding to the user's voice from the intelligence server 200. Thus, the processor 120 may display the converted text data on the display 161.
According to an embodiment, the processor 120 may receive a path rule from the intelligence server 200 by executing the intelligence agent 135. According to an embodiment, the processor 120 may transfer the path rule to the execution manager module 137 through the intelligence agent 135.
According to an embodiment, the processor 120 may transfer to the intelligence service module 139 an execution result log according to the path rule received from the intelligence server 200 by executing the intelligence agent 135, and the transferred execution result log may be accumulated and managed in user's preference information of a persona module 139b.
According to an embodiment, the processor 120 may execute the plurality of the apps 131 and 133 by receiving a path rule from the intelligence agent 135 by executing the execution manager module 137, and enable the plurality of the apps 131 and 133 to execute the actions 131b and 133b included in the path rule. For example, the processor 120 may transmit command information (e.g., path rule information) for executing the actions 131b and 133b to the plurality of the apps 131 and 133 through the execution manager module 137, and receive completion information of the actions 131b and 133b from the plurality of the apps 131 and 133.
According to an embodiment, the processor 120 may transfer command information (e.g., path rule information) for executing the actions 131b and 133b of the plurality of the apps 131 and 133 between the intelligence agent 135 and the plurality of the apps 131 and 133 by executing the execution manager module 137. The processor 120 may bind the plurality of the apps 131 and 133 to execute according to the path rule through the execution manager module 137, and transfer command information (e.g., path rule information) of the actions 131b and 133b included in the path rule to the plurality of the apps 131 and 133. For example, the processor 120 may sequentially execute the actions 131b and 133b of the plurality of the apps 131 and 133 according to the path rule, by sequentially transmitting the actions 131b and 133b included in the path rule to the plurality of the apps 131 and 133 through the execution manager module 137.
According to an embodiment, the processor 120 may manage an execution state of the actions 131b and 133b of the plurality of the apps 131 and 133 by executing the execution manager module 137. For example, the processor 120 may receive information of the execution state of the actions 131b and 133b from the plurality of the apps 131 and 133 through the execution manager module 137. If the execution state of the actions 131b and 133b is, for example, partial landing (e.g., if a parameter necessary for the actions 131b and 133b is not inputted), the processor 120 may transfer information of the partial landing to the intelligence agent 135 through the execution manager module 137. The processor 120 may request the user to input necessary information (e.g., parameter information), using the information received through the intelligence agent 135. If the execution state of the actions 131b and 133b is, for example, an operating state, the processor 120 may receive speech from the user through the intelligence agent 135. The processor 120 may transfer the plurality of the apps 131 and 133 being executed through the execution manager module 137 and information of the execution state of the plurality of the apps 131 and 133 to the intelligence agent 135. The processor 120 may transmit the user speech to the intelligence server 200 through the intelligence agent 135. The processor 120 may receive parameter information of the user's speech from the intelligence server 200 through the intelligence agent 135. The processor 120 may transfer the parameter information received through the intelligence agent 135 to the execution manager module 137. The execution manager module 137 may change the parameter of the actions 131b and 133b with a new parameter using the received parameter information.
According to an embodiment, the processor 120 may transfer parameter information included in the path rule to the plurality of the apps 131 and 133 by executing the execution manager module 137. If the plurality of the apps 131 and 133 is sequentially executed according to the path rule, the execution manager module 137 may transfer the parameter information included in the path rule from one app to another app.
According to an embodiment, the processor 120 may receive a plurality of path rules by executing the execution manager module 137. The processor 120 may select the plurality of the path rules based on user's speech through the execution manager module 137. For example, if the user's speech has specifies some app 131 for executing some action 131a through the execution manager module 137, but does not specify another app 133 for executing the remaining action 133b, the processor 120 may receive a plurality of different path rules by which the same app 131 (e.g., a gallery app) for executing some action 131a is executed and a different app 133 (e.g., a message app, a telegram app) for executing the remaining action 133b is executed. The processor 120 may execute, for example, the same actions 131b and 133b (e.g., consecutive identical actions 131b and 133b) of the plurality of the path rules through the execution manager module 137. If executing up to the same action, the processor 120 may display a state screen for selecting the different apps 131 and 133 included in the plurality of the path rules respectively through the execution manager module 137 on the display 161.
According to an embodiment, the intelligence service module 139 may include a context module 139a, a persona module 139b, or a suggestion module 139c.
The processor 120 may collect current states of the plurality of the apps 131 and 133 from the plurality of the apps 131 and 133 by executing the context module 139a. For example, the processor 120 may receive context information indicating the current states of the plurality of the apps 131 and 133 by executing the context module 139a, and collect the current states of the plurality of the apps 131 and 133 through the received context information.
The processor 120 may manage personal information of the user who uses the user terminal 100 by executing the persona module 139b. For example, the processor 120 may collect use information and execution result of the user terminal 100 by executing the persona module 139b, and manage the user's personal information using the collected use information and execution result of the user terminal 100.
The processor 120 may predict a user's intent by executing the suggestion module 139c, and recommend a command to the user based on the user's intent. For example, the processor 120 may recommend a command to the user according to a user's current state (e.g., a time, a place, a situation and an app) by executing the suggestion module 139c.
Referring to
According to an embodiment, the user terminal 100 may execute the intelligence app for recognizing a voice through a hardware key 112. For example, if receiving the user input through the hardware key 112, the user terminal 100 may display a user interface (UI) 162 of the intelligence app on the display 161. The user may, for example, touch a voice recognition button 162a in the UI 162 of the intelligence app to input 161b a voice while the UI 162 of the intelligence app is displayed on the display 161. The user may, as another example, input 161b a voice by continuously pressing the hardware key 112 to input 161b the voice.
According to an embodiment, the user terminal 100 may execute an intelligence app for recognizing a voice through a microphone 111. For example, if a designated voice 161a (e.g., wake up!) is inputted through the microphone 111, the user terminal 100 may display the UI 162 of the intelligence app on the display 161.
Referring to
According to an embodiment, the processor 120 may receive a plurality of context information from the plurality of the apps 131 and 133 through the context module 139a. The context information may be, for example, information relating to the plurality of the apps 131 and 133 executed most recently. The context information may be, as another example, information relating to the current state within the plurality of the apps 131 and 133 (e.g., information of a corresponding picture if the user views a picture in a gallery).
According to an embodiment, the processor 120 may receive context information indicating the current state of the user terminal 100 from a device platform. For example, the context information may include general context information, user context information, or device context information.
According to an embodiment, the general context information may include general information of the user terminal 100. The general context information may be identified through an internal algorithm by receiving data through a sensor hub of the device platform. For example, the general context information may include information of a current space and time. The information of the current space and time may include, for example, current time or current location information of the user terminal 100. The current time may be identified through the time on the user terminal 100, and the current location information may be identified through a global positioning system (GPS). As another example, the general context information may include physical motion information. The physical motion information may include information relating to, for example, walking, running, or driving. The physical motion information may be identified through a motion sensor. The driving information may identify driving through the motion sensor, or identify boarding and parking by sensing Bluetooth connection of a vehicle. As another example, the general context information may include user activity information. The user activity information may include information relating to, for example, commuting, shopping, or traveling. The user activity information may be identified using information relating to a place which the user or the app registers in a database.
According to an embodiment, the user context information may include information of the user. For example, the user context information may include information of a user's emotional state. The emotional state information may include information relating to, for example, happiness, sadness, or anger of the user. As another example, the user context information may include information of the user's current state. The current state information may include information relating to, for example, an interest or intent (e.g., shopping).
According to an embodiment, the device context information may include state information of the user terminal 100. For example, the device context information may include information of the path rule executed by the execution manager module 137. As another example, the device information may include battery information. The battery information may be identified through, for example, charging and discharging states of the battery. As another example, the device information may include information of a connected device or network. The connected device information may be identified through, for example, a communication interface to which the device is connected.
Referring to
According to an embodiment, the processor 120 may provide a hint to the user by executing the hint provision module 139c-1. For example, the processor 120 may receive a hint generated from the context hint generation module 139c-2, the reuse hint generation module 139c-5, or the introduction hint generation module 139c-6 through the hint provision module 139c-1 and provide the hint to a user.
According to an embodiment, the processor 120 may generate a hint to be recommended according to the current state by executing the condition checking module 139c-3 or the condition model module 139c-4. The processor 120 may receive information corresponding to the current state by executing the condition checking module 139c-3, and set a condition model using the received information by executing the condition model module 139c-4. For example, the processor 120 may obtain a time, a location, a situation and an app being used if the hint is provided to the user by executing the condition model module 139c-4, and provide the user with hint which is likely to be used under a corresponding condition in descending order of priority.
According to an embodiment, the processor 120 may generate a hint to be recommended based on a frequency of use by executing the reuse hint generation module 139c-5. For example, the processor 120 may generate a hint based on a user's use pattern by executing the reuse hint generation module 139c-5.
According to an embodiment, the introduction hint generation module 139c-6 may generate a hint which introduces the user to a new function or a function used frequently by other user. For example, the hint for introducing the new function may include introduction (e.g., an operating method) of the intelligence agent 135.
According to another embodiment, the context hint generation module 139c-2, the condition checking module 139c-3, the condition model module 139c-4, the reuse hint generation module 139c-5, or the introduction hint generation module 139c-6 of the suggestion module 139c may be included in the personal information server 300. For example, the processor 120 may receive a hint from the context hint generation module 139c-2, the reuse hint generation module 139c-5, or the introduction hint generation module 139c-6 of the user personal information server 300 through the hint provision module 139c-1 of the suggestion module 139c and provide the received hint to the user.
According to an embodiment, the user terminal 100 may provide a hint according to a series of the following processes. For example, if receiving a hint provision request from the intelligence agent 135, the processor 120 may transmit a hint generation request to the context hint generation module 139c-2 through the hint provision module 139c-1. If receiving the hint generation request, the processor 120 may receive information corresponding to a current state from the context module 139a and the persona module 139b through the condition checking module 139c-3. The processor 120 may transfer the information received through the condition checking module 139c-3 to the condition model module 139c-4, and assign a priority to hints among hints provided to the user in descending order of usability using the information through the condition model module 139c-4. The processor 120 may identify the condition through the context hint generation module 139c-2 ({circle around (6)}), and generate a hint corresponding to the current state. The processor 120 may transfer the generated hint to the hint provision module 139c-1 through the context hint generation module 139c-2. The processor 120 may arrange the hint according to a designated rule through the hint provision module 139c-1, and transfer the hint to the intelligence agent 135.
According to an embodiment, the processor 120 may generate a plurality of context hints through the hint provision module 139c-1, and designate the priority to the plurality of the context hints according to a designated rule. According to an embodiment, the processor 120 may provide the user first with a context hint of the high priority among the plurality of the context hints through the hint provision module 139c-1.
According to an embodiment, the user terminal 100 may suggest a hint according to the frequency of the use. For example, if receiving the hint provision request from the intelligence agent 135, the processor 120 may transfer a hint generation request to the reuse hint generation module 139c-5 through the hint provision module 139c-1. If receiving the hint generation request, the processor 120 may receive user information from the persona module 139b through the reuse hint generation module 139c-5. For example, the processor 120 may receive a path rule included in user's preference information of the persona module 139b, a parameter included in the path rule, an execution frequency of an app, or time and space information where the app is used through the reuse hint generation module 139c-5. The processor 120 may generate a hint corresponding to the received user information through the reuse hint generation module 139c-5. The processor 120 may transmit the generated hint to the hint provision module 139c-1 through the reuse hint generation module 139c-5. The processor 120 may arrange the hint through the hint provision module 139c-1, and transfer the hint to the intelligence agent 135.
According to an embodiment, the user terminal 100 may suggest a hint for a new function. For example, if receiving a hint provision request from the intelligence agent 135, the processor 120 may transfer a hint generation request to the introduction hint generation module 139c-6 through the hint provision module 139c-1. The processor 120 may receive information of a function to be introduced from the suggestion server 400 by transferring the introduction hint provision request from the suggestion server 400 through the introduction hint generation module 139c-6. The suggestion server 400 may, for example, store information of the function to be introduced, and a hint list for the function to be introduced may be updated by a service operator. The processor 120 may transfer the generated hint to the hint provision module 139c-1 through the introduction hint generation module 139c-6. The processor 120 may arrange the hint through the hint provision module 139c-1, and transmit the hint to the intelligence agent 135 ({circle around (6)}).
Hence, the processor 120 may provide the user with the hint generated in the context hint generation module 139c-2, the reuse hint generation module 139c-5, or the introduction hint generation module 139c-6 through the suggestion module 139c. For example, the processor 120 may display the generated hint in an app which operates the intelligence agent 135 through the suggestion module 139c, and receive an input for selecting the hint from the user through the app.
Referring to
The NLU module 220 or the path planner module 230 of the intelligence server 200 may generate a path rule.
According to an embodiment, the ASR module 210 may convert a user input received from the user terminal 100 to text data.
According to an embodiment, the ASR module 210 may convert a user input received from the user terminal 100 to text data. For example, the ASR module 210 may include a speech recognition module. The speech recognition module may include an acoustic model or a language model. For example, the acoustic model may include information related to utterance, and the language model may include unit phoneme information or information relating to a combination of the unit phoneme information. The speech recognition module may convert the user speech to the text data by using the utterance information or the combination information of the unit phoneme information. The information relating to the acoustic model or the language model may be stored in an automatic speech recognition database (ASR DB) 211.
According to an embodiment, the NLU module 220 may obtain user's intent by conducting syntactic analysis or semantic analysis. The syntactic analysis may divide the user input into syntactic units (e.g., words, phrases, morphemes, etc.), and obtain which syntactic elements the divided units are. The semantic analysis may be performed by using semantic matching, rule matching, or formula matching. Thus, the NLU module 220 may obtain a domain, an intent, or a parameter (or a slot) required to the user input to express the intent.
According to an embodiment, the NLU module 220 may determine the user's intent and the parameter by using the matching rule which is divided into the domain, the intend, and the parameter (or a slot) required to obtain the intent. For example, the one domain (e.g., an alarm) may include a plurality of intents (e.g., alarm setting or alarm release, etc.), and one intent may include a plurality of parameters (e.g., a time, the number of iterations, an alarm sound, etc.). For example, a plurality of rules may include, for example, one or more necessary parameters. The matching rule may be stored in a natural language understanding database (NLU DB) 221.
According to an embodiment, the NLU module 220 may obtain meaning of words extracted from the user input using linguistic features (e.g., grammatical elements) such as morphemes or phrases, and determine the user's intent by matching the obtained meaning of the words to the domain and the intent. For example, the NLU module 220 may determine the user intent by calculating how many words extracted from the user input are included in each of the domain and the intent. According to an embodiment, the NLU module 220 may determine a parameter of the user input by using the word used to obtain the intent. According to an embodiment, the NLU module 220 may determine the user intent by using the NLU DB 221 storing the linguistic features to obtain the intent of the user input. According to another embodiment, the NLU module 220 may determine the user intent by using a personal language model (PLM). For example, the NLU module 220 may determine the user intent by using personalized information (e.g., a contact list or a music list). For example, the PLM may be stored in, for example, the NLU DB 221. According to an embodiment, not only the NLU module 220 but also the ASR module 210 may recognize a user's voice by referring to the PLM stored in the NLU DB 221.
According to an embodiment, the NLU module 220 may generate a path rule based on the intent of the user input and the parameter. For example, the NLU module 220 may select an app to be executed based on the intent of the user input, and determine an action to be executed in the selected app. The NLU module 220 may generate the path rule by determining the parameter corresponding to the determined action. According to an embodiment, the path rule generated by the NLU module 220 may include information relating to the app to be executed, the action (e.g., at least one or more states) to be executed in the app, and a parameter required to execute the action.
According to an embodiment, the NLU module 220 may generate one path rule, or a plurality of path rules based on the intent of the user input and the parameter. For example, the NLU module 220 may receive a path rule set corresponding to the user terminal 100 from the path planner module 230, and determine the path rule by mapping the intent of the user input and the parameter to the received path rule set.
According to another embodiment, the NLU module 220 may generate one path rule or a plurality of path rules by determining the app to be executed, the action to be executed in the app, and the parameter required to execute the action based on the intent of the user input and the parameter. For example, the NLU module 220 may generate the path rule by arranging the app to be executed and the action to be executed in the app in the form of ontology or a graph model according to the intent of the user input using information of the user terminal 100. For example, the generated path rule may be stored in, for example, a path rule database (PR DB) 231 through the path planner module 230. The generated path rule may be added to the path rule set of the DB 231.
According to an embodiment, the NLU module 220 may select at least one path rule of the plurality of the generated path rules. For example, the NLU module 220 may select an optimal path rule of the plurality of the path rules. As another example, if only some action is specified based on the user speech, the NLU module 220 may select a plurality of path rules. The NLU module 220 may determine one path rule of the plurality of the path rules based on a user's additional input.
According to an embodiment, the NLU module 220 may transmit the path rule to the user terminal 100 in response to a request for the user input. For example, the NLU module 220 may transmit one path rule corresponding to the user input to the user terminal 100. As another example, the NLU module 220 may transmit the plurality of path rules corresponding to the user input to the user terminal 100. For example, if only some action is specified based on the user speech, the plurality of path rules may be generated by the NLU module 220.
According to an embodiment, the path planner module 230 may select at least one path rule of the plurality of the path rules.
According to an embodiment, the path planner module 230 may transfer the path rule set including the plurality of the path rules to the NLU module 220. The plurality of the path rules of the path rule set may be stored in the PR DB 231 connected to the path planner module 230 in the form of a table. For example, the path planner module 230 may transfer the path rule set corresponding to the information (e.g., OS information or app information) of the user terminal 100 received from the intelligence agent 151 to the NLU module 220. For example, a table stored in the PR DB 231 may be stored for, for example, each domain or each version of the domain.
According to an embodiment, the path planner module 230 may select one path rule or the plurality of path rules from the path rule set and transfer to the NLU module 220. For example, the path planner module 230 may select one path rule or a plurality of path rules by matching the user's intent and the parameter to the path rule set corresponding to the user terminal 100 and transfer to the NLU module 220.
According to an embodiment, the path planner module 230 may generate one path rule or a plurality of path rules by using the user intent and the parameter. For example, the path planner module 230 may generate one path rule or a plurality of path rules by determining the app to be executed and the action to be executed in the app based on the user intent and the parameter. According to an embodiment, the path planner module 230 may store the generated path rule in the PR DB 231.
According to an embodiment, the path planner module 230 may store the path rule generated by the NLU module 220 in the PR DB 231. The generated path rule may be added to the path rule set stored in the PR DB 231.
According to an embodiment, the table stored in the PR DB 231 may include a plurality of path rules or a plurality of path rule sets. The plurality of the path rules or the plurality of the path rule sets may reflect a kind, a version, a type, or characteristics of a device which performs each path rule.
According to an embodiment, the DM module 240 may determine whether the user intent obtained by the NLU module 220 is clear. For example, the DM module 240 may determine whether the user intent is clear based on whether the information of the parameter is sufficient. The DM module 240 may determine whether the parameter obtained by the NLU module 220 is sufficient to perform a task. According to an embodiment, if the user intent is not clear, the DM module 240 may perform a feedback for requesting necessary information from the user. For example, the DM module 240 may perform a feedback for requesting parameter information for obtaining the user intent.
According to an embodiment, the DM module 240 may include a content provider module. If executing an action based on the intent obtained by the NLU module 220 and the parameter, the content provider module may generate a result of performing a task corresponding to the user input. According to an embodiment, the DM module 240 may transmit the result generated at the content provider module as a response to the user input to the user terminal 100.
According to an embodiment, the NLG 250 may change designated information into a text form. The information changed into the text form may be in the form of a natural language speech. For example, the designated information may be information of an additional input, information for guiding completion of the action corresponding to the user input, or information for guiding the additional input of the user (e.g., feedback information relating to the user input). The information changed into the text form may be transmitted to the user terminal 100 and displayed on the display 120, or may be transmitted to the TTS module 260 and changed into a voice form.
According to an embodiment, the TTS module 260 may change information of the text form to information of the voice form. The TTS module 260 may receive the information of the text form from the NLG module 250, change the information of the text form to the information of the voice form, and transmit to the user terminal 100. The user terminal 100 may output the information of the voice form to the speaker 156.
According to an embodiment, the NLU module 220, the path planner module 230, and the DM module 240 may be implemented as one module. For example, the NLU module 220, the path planner module 230 and the DM module 240 may be implemented as one module to determine the user intent and the parameter, and to generate a response (e.g., a path rule) corresponding to the determined user intent and the parameter. Hence, the generated response may be transmitted to the user terminal 100.
Referring to
According to an embodiment, the PR DB 231 of the path planner module 230 may store the path rule set for performing the function of the app. The path rule set may include a plurality of path rules including a plurality of actions (e.g., a sequence of states). The plurality of the path rules may sequentially arrange actions executed according to a parameter inputted to the plurality of actions respectively. According to an embodiment, the plurality of the path rules may be configured in the form of ontology or a graph model and stored in the PR DB 231.
According to an embodiment, the NLU module 220 may select an optimal path rule A-B1-C3-D-F from the plurality of the path rules A-B1-C1, A-B1-C2, A-B1-C3-D-F, and A-B1-C3-D-E-F corresponding to the intent of the user input and the parameter.
According to an embodiment, if there is no path rule perfectly matched to the user input, the NLU module 220 may transfer a plurality of rules to the user terminal 100. For example, the NLU module 220 may select a path rule (e.g., A-B1) partially corresponding to the user input. The NLU module 220 may select one or more path rules (e.g., A-B1-C1, A-B1-C2, A-B1-C3-D-F, and A-B1-C3-D-E-F) including the path rule (e.g., A-B1) partially corresponding to the user input and transfer to the user terminal 100.
According to an embodiment, the NLU module 220 may select one of the plurality of the path rules based on an additional input of the user terminal 100, and transfer the selected one path rule to the user terminal 100. For example, the NLU module 220 may select one path rule (e.g., A-B1-C3-D-F) from the plurality of the path rules (e.g., A-B1-C1, A-B1-C2, A-B1-C3-D-F, and A-B1-C3-D-E-F) according to the user input (e.g., an input for selecting C3) additionally inputted at the user terminal 100 and transmit to the user terminal 100.
According to yet another embodiment, the NLU module 220 may determine a user's intent and a parameter corresponding to the user input (e.g., an input for selecting C3) additionally inputted to the user terminal 100 through the NLU module 220, and transmit the determined user intent or parameter to the user terminal 100. The user terminal 100 may select one path rule (e.g., A-B1-C3-D-F) from the plurality of the path rules (e.g., A-B1-C1, A-B1-C2, A-B1-C3-D-F, and A-B1-C3-D-E-F), based on the transmitted intent or the parameter.
Thus, the user terminal 100 may finish the actions of the plurality of the apps 131 and 133 according to the selected one path rule.
According to an embodiment, if a user input in which information is insufficient is received at the intelligence server 200, the NLU module 220 may generate a path rule partially corresponding to the received user input. For example, the NLU module 220 may transmit the partially corresponding path rule to the intelligence agent 131. The processor 120 may receive the path rule by executing the intelligence agent 135, and transfer the partially corresponding path rule to the execution manager module 137. The processor 120 may execute the first app 131 according to the path rule through the execution manager module 137. The processor 120 may execute the first app 131 through the execution manager module 137 and transmit information of the insufficient parameter to the intelligence agent 135. The processor 120 may request an additional input from the user using the information of the insufficient parameter through the intelligence agent 135. If receiving the additional input by the user through the intelligence agent 135, the processor 120 may transmit and process the additional input to the intelligence server 200. The NLU module 220 may generate a path rule added based on the intent of the user input additionally inputted and parameter information and transmit to the intelligence agent 135. The processor 120 may execute the second app 133 by transmitting the path rule to the execution manager module 137 through the intelligence agent 135.
According to an embodiment, if a user input with some information omitted is received at the intelligence server 200, the NLU module 220 may transmit a user information request to the personal information server 300. The personal information server 300 may transmit information of the user inputting the user input stored in the persona database to the NLU module 220. The NLU module 220 may select a path rule corresponding to the user input with some action omitted using the user information. Thus, even if the user input with some information omitted is received at the intelligence server 200, the NLU module 220 may determine a path rule corresponding to the user input by receiving an additional input by requesting the omitted information or by using the user information.
Table 1 attached below may show an exemplary form of a path rule related to a task requested by the user according to an embodiment.
Referring to Table 1, a path rule generated or selected at an intelligence server (e.g., the intelligence server 200) according to a user speech (e.g., “Share the photo”) may include at least one state 25, 26, 27, 28, 29 or 30. For example, the at least one state (e.g., any one action state of the terminal) may correspond to at least one of picture app execution (pictureView) 25, picture search function execution (searchView) 26, search result display screen output (searchViewResult) 27, search result display screen output with photo non selected (SearchEmptySelectedView) 28, search result display screen output with at least one photo selected (SearchSelectedView) 29, or shared application selection screen output (CrossShare) 30.
According to an embodiment, the parameter information of the path rule may correspond to at least one state. For example, the at least one photo may be included in the selected search result display screen output 29 state.
The task (e.g., “Share the photo!”) requested by the user may be performed according to the execution result of the path rule including a sequence of states 25, 26, 27, 28, and 29.
Referring to
According to an embodiment, the processor 120 may transmit the information received from the plurality of the apps 131 and 133, the execution manager module 137, or the context module 139a to the suggestion module 139c through the persona module 139b. For example, the processor 120 may transfer the data stored in the operation log database or context database to the suggestion module 139c through the persona module 139b.
According to an embodiment, the processor 120 may transmit the information received from the plurality of the apps 131 and 133, the execution manager module 137, or the context module 139a to the personal information server 300 through the persona module 139b. For example, the processor 120 may periodically transmit the data accumulated and stored in the operation log database or context database to the personal information server 300 through the persona module 139b.
According to an embodiment, the processor 120 may transfer the data stored in the operation log database or the context database to the suggestion module 139c through the persona module 139b. User information generated through the persona module 139b may be stored in a persona database. The persona module 139b may periodically transmit the user information stored in the persona database to the personal information server 300. According to an embodiment, the information transmitted to the personal information server 300 through the persona module 139b may be stored in the persona database. The personal information server 300 may infer user information required to generate a path rule of the intelligence server 200 using the information stored in the persona database.
According to an embodiment, the user information inferred using the information transmitted through the persona module 139b may include profile information or preference information. The profile information or the preference information may be inferred through a user's account and the accumulated information.
The profile information may include user's personal information. For example, the profile information may include user's population statistics information. The population statistics information may include, for example, a use's gender or age. As another example, the profile information may include life event information. The life event information may be inferred by, for example, comparing log information with a life event model, and reinforced by analyzing a behavior pattern. As another example, the profile information may include interest information. The interest information may include, for example, shopping goods of interest or a field of interest (e.g., sports, politics, etc.). As another example, the profile information may include activity region information. The activity region information may include, for example, information relating to a home or a workplace. The activity region information may include not only location information of the place but also information relating to regions recorded with priorities based on an accumulated time of stay and a visit count. As another example, the profile information may include activity time information. The activity time information may include, for example, information relating to a wakeup time, a commute time, or a sleep time. The commute time information may be inferred using the activity region information (e.g., home or workplace information). The sleep time information may be inferred through an unused time of the user terminal 100.
The preference information may include user's preference information. For example, the preference information may include app preference information. The app preference may be inferred through, for example, a use record (e.g., a use record per time or place) of an app. The app preference may be used to determine an app to be executed according to a user's current state (e.g., a time or pa lace). For example, the preference information may include contacts preference information. The contacts preference may be inferred by, for example, analyzing contact frequency (e.g., contact frequency per time or per place) information of contacts. The contacts preference may be used to determine contacts to contact according to the user's current state (e.g., contact to an overlapping name). As another example, the preference information may include setting information. The setting information may be inferred by, for example, analyzing setting frequency (e.g., frequency set as a setting value by time or place) information of a particular setting value. The setting information may be used to set the particular setting value according to the user's current state (e.g., a time, a place, or a context). As another example, the preference information may include place preference. The place preference may be inferred through, for example, a record of visits to a particular place (e.g., a record of visits per time). The place preference may be used to determine a place which the user is visiting according to the user's current state (e.g., time). For example, the preference information may include command preference. The command preference may be inferred through, for example, a command use frequency (e.g., a use frequency per time or place). The command preference may be used to determine a command pattern to be used according to the user's current state (e.g., time or place). In particular, the command preference may include information of a menu selected most by the user in the current state of the app executed by analyzing log information.
The electronic device according to various embodiments disclosed in the present document may be a device of various types. The electronic device may include at least one of, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. The electronic device according to an embodiment of the disclosure is not limited to those described above.
It should be appreciated that various embodiments of the disclosure and the terms used therein are not intended to limit the techniques set forth in the disclosure to particular embodiments, but include various changes, equivalents, and/or alternatives of corresponding embodiments. With regard to descriptions of drawings, like reference numerals may be used for similar components. The singular expression may include a plural expression, unless the context clearly dictates otherwise. In the present disclosure, an expression such as “A or B”, “at least one of A and/or B”, “A, B, or C”, or “at least one of A, B, and/or C” may include all possible combinations of items listed. Expressions such as “first”, “second”, “primarily” or “secondary” may represent corresponding elements regardless of order or importance, are merely used to distinguish one element from other element, and do not limit corresponding elements. If it is described that an element (e.g., a first element) is “(operatively or communicatively) coupled” to or “connected” to another element (e.g., a second element), the element may be directly connected to the other element or may be connected through another element (e.g., a third element).
The term “module” used in the present disclosure includes a unit configured in hardware, software, or firmware, and may interchangeably be used with a term, for example, logic, logic block, part, or circuit. A module may be a single integral part or a minimum unit or part for performing one or more functions. For example, the module may be configured with an application-specific integrated circuit (ASIC).
Various embodiments of the present disclosure may be implemented as software (e.g., the program 140) including commands stored in machine (e.g., computer)-readable storage media (e.g., an internal memory 136 or an external memory 138). The machine is a device for invoking a command stored in the storage medium and operating according to the invoked command, and may include an electronic device (e.g., the electronic device 101) according to embodiments disclosed. If the command is executed by a processor (e.g., the processor 120), the processor may perform a function corresponding to the command on its own or using other components under control of the processor. The command may include a code generated or executed by a compiler or an interpreter. The machine-readable storage media may be provided in the form of a non-transitory storage medium. Herein, the term ‘non-transitory’ simply means that the storage medium does not include a signal and is tangible, but does not differentiate semi-permanently or temporarily storing data in the storage medium.
According to an embodiment, a method according to various embodiments of the present disclosure may be included and provided in a computer program product. The computer program product may be traded as commodities between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or online through an application store (e.g., Playstore™). If distributed online, at least part of the computer program product may be at least temporarily stored in a storage medium such as a server of a manufacturer, a server of the application store, or a memory of a relay server, or may be generated temporarily.
Each component (e.g., a module or program) according to various embodiments may be configured a single or multiple entities, some sub component of the above-mentioned corresponding sub components may be omitted, or other sub component may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, to perform functions performed by corresponding components respectively before being integrated in the same or similar manner. According to various embodiments, operations performed by modules, programs, or other components may be carried out sequentially, in parallel, repeatedly, heuristically, or at least some operations may be executed in a different order, or omitted, or other operation may be added.
According to various embodiments of the present invention, an electronic device (e.g., the electronic device 101 of
According to various embodiments, the processor may be configured to display data in the at least one input field based on at least part of the voice command using the display, and if there is a blank input field (e.g., a password input field 1309 of
According to various embodiments, the information of the input field may include texts or identification information related to the at least one input field.
According to various embodiments, the processor may be configured to extract part of the texts related to the at least one input field based on a priority, and determine the extract part as the guide information.
According to various embodiments, the processor may be configured to extract guide information corresponding to any one input field of the at least one input field based on a priority, and if the detected guide information overlaps with guide information corresponding to other input field of the at least one input field, detect other guide information corresponding to the any one input field based on a next priority.
According to various embodiments, the processor may configured to extract an additional voice command of the user using the input device, if part of the additional voice command matches the guide information, display data in an input field corresponding to the matched guide information using the display, and if the part of the additional voice command does not match the guide information, display a guide message using the display.
According to various embodiments, the processor may be configured to apply a graphic effect to the guide information such that the guide information is highlighted relative to at least one object in the electronic document.
According to various embodiments, the processor may be configured to detect an additional voice command of the user using the input device, and if the additional voice command contains a continue command, sequentially display data contained the additional voice command in the at least one input field based on the continue command using the display.
According to various embodiments, the processor may be configured to detect an additional voice command of the user using the input device, and if detecting a modify command in the additional voice command, change data displayed in the at least one input field based on the modify command.
According to various embodiments, the processor may be configured to detect an additional voice command of the user using the input device, display data contained the additional voice command in the at least one input field corresponding to the additional voice command using the display, if there is no blank input field in the at least one input field, output additional guide information, detect a voice command corresponding to the additional guide information, and execute a function corresponding to the voice command corresponding to the additional guide information.
Referring to
The electronic device 101 (e.g., the processor 120 of
The electronic device 101 (e.g., the processor 120 of
According to an embodiment, if there is no need to output the electronic document corresponding to the voice command 1103, the electronic device 101 may terminate the corresponding procedure or execute a function corresponding to the voice command. For example, if a voice command (e.g., Tell me the meaning of evaluation) of the user 1101 is a function (e.g., outputting English word meaning) not related to the electronic document unlike the drawing, the electronic device 101 may immediately execute the function corresponding to the voice command without outputting the electronic document.
According to an embodiment, if outputting the electronic document corresponding to the voice command 1103, the electronic device 101 (e.g., the processor 120 of
The electronic device 101 (e.g., the processor 120 of
The electronic device 101 (e.g., the processor 120 of
According to an embodiment, the electronic device 101 may add a graphic effect so that the guide information may be more easily recognized by the user. For example, the electronic device 101 may additionally display a rectangular figure surrounding the guide information and a shading effect filling the figure on the display 161 as shown in
According to an embodiment, the electronic device 101 may apply a graphic effect to the guide information so that the user may easily obtain an operation (e.g., changing data of a reservation related input field) which may be currently performed by the user.
According to an embodiment, the electronic device 101 may determine the name of the input field as the guide information. For example, as shown in
According to an embodiment, the electronic device 101 may determine a text not related to the input field as the guide information. For example, as shown in
The electronic device 101 may display the electronic document including the guide information and then output a message 1104 (e.g., Confirm the reservation) requesting the user 1101 to identify reservation information. As another example, the electronic device 101 may display a message requesting to identify the reservation information on the display 161. Next, the electronic device 101 may display a next electronic document (e.g., a payment web page) for the reservation if receiving an input for requesting a ticket reservation from the user.
The following description explains an operation of displaying data in the input field in operation 1011 of
Referring to
The electronic device 101 (e.g., the processor 120 of
According to an embodiment, if determining that there is no blank input field, the electronic device 101 (e.g., the processor 120 of
According to an embodiment, if determining that the blank input field exists, the electronic device 101 (e.g., the processor 120 of
According to an embodiment, if there is no use history corresponding to the blank input field, the electronic device 101 (e.g., the processor 120 of
According to an embodiment, if there is the use history corresponding to the blank input field, the electronic device 101 (e.g., the processor 120 of
The electronic device 101 (e.g., the processor 120 of
The following description explains an operation of determining guide information in operation 1009 of
Referring to
The electronic device 101 (e.g., the processor 120 of
In operation 1405, the electronic device 101 (e.g., the processor 120 of
According to an embodiment, if determining that the detected guide information does not overlap with the guide information corresponding to other input field, the electronic device 101 (e.g., the processor 120 of
According to an embodiment, if determining that the detected guide information overlaps with the guide information corresponding to other input field, the electronic device 101 (e.g., the processor 120 of
In operation 1409, the electronic device 101 (e.g., the processor 120 of
The electronic device 101 (e.g., the processor 120 of
The following description explains an operation of displaying data in an input field based on an additional voice command in operation 1011 of
Referring to
In operation 1503, the electronic device 101 (e.g., the processor 120 of
According to an embodiment, if part of the additional voice command matches the guide information, the electronic device 101 (e.g., the processor 120 of
According to an embodiment, if part of the additional voice command does not match the guide information, the electronic device 101 (e.g., the processor 120 of
The electronic device 101 (e.g., the processor 120 of
The electronic device 101 (e.g., the processor 120 of
The electronic device 101 (e.g., the processor 120 of
The following description explains an operation of displaying an electronic document based on an additional voice command in operation 1011 of
Referring to
The electronic device 101 (e.g., the processor 120 of
According to an embodiment, the electronic device 101 may display data in a next input field on the display 161 if detecting a silence section exceeding a designated time in the user's voice command. For example, if the user inputs a voice command (e.g., “Suwon silence 2017.11.11”), the electronic device 101 detecting the silence section may skip the arrival station input field 1909 and and display data in the departure date input field 1911 on the display 161.
According to an embodiment, the continue command may be activated if a plurality of input fields allowing data input is sequentially displayed. For example, if a plurality of input fields 1907, 1909, 1911, 1913, and 1915 is sequentially displayed from top to bottom of a display area of the display 161 as shown in
According to an embodiment, if the continue command is not included in the additional voice command, the electronic device 101 (e.g., the processor 120 of
According to an embodiment, if the continue command is included in the additional voice command, the electronic device 101 (e.g., the processor 120 of
The electronic device 101 (e.g., the processor 120 of
According to an embodiment, if not detecting the modify command, the electronic device 101 (e.g., the processor 120 of
According to an embodiment, if detecting the modify command, the electronic device 101 (e.g., the processor 120 of
The following description explains an operation of outputting additional guide information in operation 1011 of
Referring to
The electronic device 101 (e.g., the processor 120 of
According to an embodiment, the electronic device 101 may output additional guide information in operation 2007 even if there is the blank input field. For example, if the blank input field is not an essential element in executing a function (e.g., subscription) corresponding to a voice command (e.g., if an input field for inputting whether the user is married is blank in an electronic document for subscription), the electronic device 101 may output additional guide information.
According to an embodiment, if there is no blank input field, the electronic device 101 (e.g., the processor 120 of
The electronic device 101 (e.g., the processor 120 of
According to an embodiment, if not detecting the voice command corresponding to the additional guide information 2115, the electronic device 101 (e.g., the processor 120 of
According to an embodiment, if detecting the voice command corresponding to the additional guide information 2115, the electronic device 101 (e.g., the processor 120 of
Referring to
According to various embodiment of the invention, an operating method of an electronic device (e.g., the electronic device 101 of
According to various embodiments, displaying the electronic document may include displaying data in the at least one input field based on at least part of the voice command, and if there is a blank input field in the at least one input field (e.g., the password input field 1309 of
According to various embodiments, the information of the input field may include texts or identification information related to the at least one input field.
According to various embodiments, determining guide information may include extracting part of the texts related to the at least one input field based on a priority, and determining the extract part as the guide information.
According to various embodiments, determining guide information may include extracting guide information corresponding to any one input field of the at least one input field based on a priority, and if the detected guide information overlaps with guide information corresponding to other input field of the at least one input field, extracting other guide information corresponding to the any one input field based on a next priority.
According to various embodiments, the method may further include extracting an additional voice command of the user using the input device, if part of the additional voice command matches the guide information, displaying data in an input field corresponding to the matched guide information, and if the part of the additional voice command does not match the guide information, displaying a guide message.
According to various embodiments, the method may include applying a graphic effect to the guide information such that the guide information is highlighted relative to at least one object in the electronic document.
According to various embodiments, the method may further include detecting an additional voice command of the user, and if the additional voice command contains a continue command, sequentially displaying data contained the additional voice command in the at least one input field based on the continue command.
According to various embodiments, the method may further include detecting an additional voice command of the user, and if detecting a modify command in the additional voice command, changing data displayed in the at least one input field based on the modify command.
According to various embodiments, the method may further include detecting an additional voice command of the user, displaying data contained the additional voice command in the at least one input field corresponding to the additional voice command, if there is no blank input field in the at least one input field, outputting additional guide information, detecting a voice command corresponding to the additional guide information, and executing a function corresponding to the voice command corresponding to the additional guide information.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2018-0001862 | Jan 2018 | KR | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/KR2019/000039 | 1/2/2019 | WO |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2019/135599 | 7/11/2019 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 8718242 | Bangalore et al. | May 2014 | B2 |
| 9836192 | Katsuranis | Dec 2017 | B2 |
| 20040044523 | Sakai et al. | Mar 2004 | A1 |
| 20050071172 | James | Mar 2005 | A1 |
| 20070213984 | Ativanichayaphong et al. | Sep 2007 | A1 |
| 20090171669 | Engelsma | Jul 2009 | A1 |
| 20090306980 | Shin | Dec 2009 | A1 |
| 20110085191 | Takato | Apr 2011 | A1 |
| 20120313849 | Bak | Dec 2012 | A1 |
| 20130033643 | Kim | Feb 2013 | A1 |
| 20140191949 | Park et al. | Jul 2014 | A1 |
| 20140195243 | Cha | Jul 2014 | A1 |
| 20140196087 | Park et al. | Jul 2014 | A1 |
| 20140343950 | Simpson | Nov 2014 | A1 |
| 20170351486 | Won | Dec 2017 | A1 |
| 20180121881 | Kumar | May 2018 | A1 |
| 20180181545 | Qu | Jun 2018 | A1 |
| 20180329677 | Gruber | Nov 2018 | A1 |
| 20180336010 | Mukherjee | Nov 2018 | A1 |
| Number | Date | Country |
|---|---|---|
| 4006338 | Nov 2007 | JP |
| 2010217628 | Sep 2010 | JP |
| 10-2006-0020101 | Mar 2006 | KR |
| 10-2009-0115599 | Nov 2009 | KR |
| 10-2010-0011786 | Feb 2010 | KR |
| 10-0988397 | Oct 2010 | KR |
| 10-1066732 | Sep 2011 | KR |
| 10-2012-0040936 | Apr 2012 | KR |
| 10-1207435 | Dec 2012 | KR |
| 10-2013-0016024 | Feb 2013 | KR |
| 10-2014-0089847 | Jul 2014 | KR |
| 10-2015-0072625 | Jun 2015 | KR |
| 10-2016-0039830 | Apr 2016 | KR |
| Entry |
|---|
| FreedomSpeech (Youtube: https://www.youtube.com/watch?v=DEISEronSIQ, 58 seconds duration, pp. 1-3, published 2014). (Year: 2014). |
| International Search Report dated Apr. 8, 2019 in connection with International Patent Application No. PCT/KR2019/000039, 2 pages. |
| Written Opinion of the International Searching Authority dated Apr. 8, 2019 in connection with International Patent Application No. PCT/KR2019/000039, 5 pages. |
| Korean Intellectual Property Office, “Office Action,” dated Jul. 11, 2022, in connection with Korean Patent Application No. 10-2018-0001862, 10 pages. |
| Korean Intellectual Property Office, “Notice of Patent Grant,” dated Jan. 5, 2023, in connection with Korean Patent Application No. 10-2018-0001862, 5 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20210089269 A1 | Mar 2021 | US |
