Patent Application
20200050345
The present technology relates to an information processing apparatus, a method, and a program.
Patent Literature 1 discloses a projector including a camera. For example, Patent Literature 1, a paragraph of 0096, describes that a user performs a hand gesture on a table on which an image is projected and an input is performed by recognizing the gesture. On the other hand, Patent Literature 2, paragraphs of 0048 to 0064, discloses a technology about virtual sound image localization.
Patent Literature 1: Japanese Patent Application Laid-open No. 2013-257686
Patent Literature 2: 2011-188287
Information equipment such as a personal computer (hereinafter may be represented by PC) and a smartphone typically includes a display device optimized for displaying information (for example, liquid crystal screen, etc.). However, in recent years, it is contemplated that a flat surface or a space such as a table (see Patent Literature 1) and a wall, for example, to which displaying information is not intended, is used as a display surface of information.
In this way, if a physical interface of visually transmitting information to the user is not optimized, there is a problem that it is difficult to efficiently take out information presented in a certain past point by returning to the point presented.
The present technology is made in view of the above-mentioned circumstances, and it is an object of the present technology to provide a user interface capable of efficiently accessing to information presented in the past.
An aspect of the present technology to achieve the object is an information processing apparatus includes a detection unit, a control unit, and an output unit.
The detection unit detects behavior of a user.
The control unit determines a current interest object of the user depending on the detected behavior and sequentially generates display information including content corresponding to the determined interest object.
The output unit outputs the generated display information.
The control unit optimizes a display mode of the display information depending on a change in the determined interest object on a basis of content of an interest object in the past of the user outputted from the output unit and content of the current interest object.
According to the information processing apparatus, only when a user takes any action (behavior including talk, finger tap, and the like), a keyword corresponding thereto is selected and a display screen optimized is generated by selecting one of pieces of information presented in the past. Thus, there is provided a user interface capable of efficiently accessing to information presented in the past.
In a case where content outputted in the past from the output unit is determined as the current interest object on a basis of the behavior of the user, the control unit may include the content outputted in the past determined as the current interest object in the display information and optimize the display mode of the display information.
In this case, if the behavior of the user points the content outputted in the past (for example, talk such as “a little while ago”), the content based on the behavior of the user detected can be extracted as information about a center of a topic by the user.
The output unit may generate the display information such that each display element corresponding to the content outputted in the past from the output unit is outputted in an output order by linking with lines.
By linking a before and later relationship among a plurality of content items outputted in the past with lines, it is possible to indicate in a visible form.
The control unit may optimize the display mode of the display information by narrowing the content outputted in the past determined as the current interest object.
By narrowing the current interest objects of the user, further optimized presentation becomes possible.
The control unit may generate the display information such that the display element corresponding to the content outputted in the past determined as the current interest object is arranged at a center of a line of sight of the user.
In this case, a layout of the display element in the GUI can be sequentially optimized.
The control unit may include content belonging to a layer common to the content outputted in the past determined as the current interest object together with the content outputted in the past determined as the current interest object in the display information.
The layout of the display element in the GUI can be further optimized.
The control unit may generate the display information showing that a plurality of content items belongs to a common layer by a visual effect and/or a common frame.
The layout of the display element in the GUI can be further optimized.
The detection unit may input a voice of the user. In addition, in this case, the control unit may determine the current interest object by analyzing the voice of the user inputted from the detection unit.
By providing a natural input method such as a user talk (or a talk between users), it is possible to provide an effective user interface without placing a burden on the user.
Other aspect of the present technology to achieve the object is an information processing method including:
a step of detecting behavior of a user;
a step of determining a current interest object of the user depending on the detected behavior;
a step of sequentially generating display information including content corresponding to the determined interest object;
a step of outputting the generated display information; and
a step of optimizing a display mode of the display information depending on a change in the determined interest object on a basis of content of an interest object in the past of the user outputted from the output unit and content of the current interest object.
Also, by the information processing method, there are provided the effects similar to the information processing apparatus.
Other aspect of the present technology to achieve the object is a program causing an information processing apparatus to execute
a step of detecting behavior of a user;
a step of determining a current interest object of the user depending on the detected behavior;
a step of sequentially generating display information including content corresponding to the determined interest object;
a step of outputting the generated display information; and
a step of optimizing a display mode of the display information depending on a change in the determined interest object on a basis of content of an interest object in the past of the user outputted from the output unit and content of the current interest object.
Also, by the information processing method, there are provided the effects similar to the information processing apparatus.
As described above, according to the present technology, a user interface capable of efficiently accessing to information presented in the past can be provided.
It should be noted that the effects described here are not necessarily limitative and may be any of effects described in the present disclosure.
Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.
The embodiments of the present technology will be described in the following order.
The hardware of the information processing apparatus 1 shown may configure a processing control unit to which a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, and a RAM (Random Access Memory) 13 are connected via a bus 14 similar to a multi-purpose computer. The CPU 11 also has a function to sequentially generate display information (for example, whole projection screen).
The processing control unit configured of a variety of hardware connected to the bus 14 is further connected to an image sensor 15, a microphone 16, a projector 17, a speaker 18, and a communication unit 19 via the bus 14 capable of performing information communication each other.
The image sensor 15 has an imaging function and the microphone 16 has a voice input function. The image sensor 15 and the microphone 16 configures a detection unit 20. The image sensor 15 senses an environment (obstacle, color on table) and the like in addition to behavior of a user. The detection unit 20 detects the behavior of the user.
The projector 17 has a function to project a display image and the speaker 18 has a voice output function. The projector 17 and the speaker 18 configures an output unit 21.
The communication unit 19 is an input-output interface communicating with external devices. It may be considered that the communication unit 19 belongs to both of the detection unit 20 and the output unit 21.
In
In
Responses to the behavior of the user inputted from the detection unit 20 are performed by the GUI displayed by the projector 17 and the voice outputted from the speaker 18.
A display output and a voice output may be partly performed by an external apparatus.
As shown in
In a case where the hardware configuration in this embodiment is an integrated neck strap type as shown in
Note that as shown in
Next, a software configuration of the information processing apparatus 1 will be described.
The ROM 12 stores an image processing SW11, a voice recognition SW12, a natural language analysis SW13, an interactive UI core SW14, a graphical user interface SW15, and a sound image localization and voice output SW16, as shown in
The image processing SW11 provides functions of figuring out a center of a line of sight of the user, figuring out an area including no obstacle on the display surface 201, or figuring out a position to which the user points on the basis of a captured image of the image sensor 15.
The voice recognition SW12 is software of recognizing a user's voice collected by the microphone 16. The natural language analysis SW13 is software of analyzing the recognized voice as a natural language and converting it into text data, string data, or the like that can be handled by the information processing apparatus 1.
The interactive UI core 14 provides a function to make up the agent that integrates a user's talk and other peripheral information and presents information to the user for a user interaction. Hereinafter, such an agent is referred to as “user agent”.
The graphical user interface SW15 provides a function of presenting effectively, user-friendly, and visually the information from the user agent. The graphical user interface SW15 performs output by a process that the display screen is generated by the CPU 11 and the display screen is projected by the projector 17. Note that a graphical user interface provided by the graphical user interface SW15 is hereinafter described simply as “GUI”.
The sound image localization and voice output SW16 provides a function of outputting the output of the user agent from the speaker 18 as a sound. The sound image localization and voice output SW16 performs sound image localization so that the voice is outputted from a display element expressing the user agent in the GUI and the user can sense it. Specifically, the technology described in Patent Literature 2, for example.
A software group including the interactive UI core SW14 as a central core shown in
Note that the above-described software configuration is only one example, and the interactive UI core SW 14 and the graphical user interface SW 15 may be configured as integrated software. Also, part of software may be provided from an external cloud as SaaS.
(1-3. Description about GUI Components)
Next, with respect to
Refer to
Refer to
Refer to
Refer to
As described above, the GUI in this embodiment has a variety types of information presented to the user and presents by different GUI components depending on the types. The user is made aware of a difference in types.
(1-4. Description about GUI and Basic Processing)
In the above-described GUI components, the respective nodes are connected by paths as shown in
The CPU 11 may further adjust a color of the GUI representing the history of the user operation. For example, a display color is adjusted depending on a color of the display surface 201 imaged by the image sensor 15. In this case, if a black table is a background and a background color is black, displaying is performed in a white frame and while characters. Other than that, a display color is adjusted on the basis of the real objects imaged by the image sensor 15, for example. In this case, if red apple is recognized, it avoids to display the GUI with a color near red and green.
In a case where the image sensor 15 captures that the real object is moved while the GUI is displayed, the CPU 11 adjusts again an arrangement of each GUI component depending on the moved real objects.
The CPU 11 recognizes or estimates the position of user's eyes on the basis of the captured image of the image sensor 15. For example, if only a hand is imaged, a position of a face is estimated from a position of the hand. Then, the CPU 11 adjusts the arrangement of the GUI component so as not to arrange the real object between the recognized or estimated position of the user's eyes and the GUI to be displayed. For example, in a case where a tall object such as a vase, the GUI is displayed before the vase.
The basic processing of generating the GUI shown in
First, the CPU 11 displays the agent icon (ST11). The agent icon (
Next, the behavior of the user with respect to the CPU 11 is detected by the detection unit 20 (ST12). The behavior of the user herein referred includes the user's talk and action. There may be a plurality of the users.
Next, the CPU 11 selects the keyword depending on the behavior of the user (ST13). The keyword is determined on the basis of a word included in the talk, for example.
Next, the CPU 11 searches new information or extracts past information on the basis of the selected keyword (ST14). The CPU 11 decides which to select searching the new information or extracting the past information on the bases of the keyword.
Details about the ST13 and ST14 will be described below. Here, searching the new information is selected. In this case, it transits from
When the screen like
ST12 to ST16 are flows that a user interface returns a response with respect to a user input. If the user further talks (for example, Search ####), ST12 to ST16 are repeated again and a display screen like
With reference to
As shown in
By a start command from the user, the agent icon corresponding to the icon of the user agent is displayed and continues to stay on a common display surface during a series of interaction time (
In the stat that the agent icon is displayed, when the user issues a specific request, the path is extended from the center of the agent icon and main answer information is displayed from the node positioned at an end as a starting point (
If derived items are present from an item of the main answer information, other path or paths extend from the same node as the main answer information, and the derived items are displayed from the node at the end as starting points. The paths are referred to as branch lines or related lines. The branch lines or the related lines are shown in
(1-5. Description about Content of Current Interest Object)
As shown in
The response information directly returned from the user interface (main answer information) is referred to as “content of current interest object” (see
Next, in ST14, optimization processing of the display screen executed by the CPU 11 will be described with reference to
The optimization processing of the display screen shown in
The CPU 11 determines whether or not a focus is moved (ST151). If Yes, the CPU 11 centers the “content of current interest object” on the center of the line of sight of the user (ST152). Here, the center of the line of sight of the user is a point determined as the center of the line of sight of the user on the basis of the information inputted by the image sensor 15.
Note that the GUI components corresponding to information groups that are not focused keep linking with paths and are gradually moved and animated.
Next, the CPU 11 makes the information far from the “content of current interest object” inconspicuous (ST154). The far information is defined as information linked via two or more nodes from the “content of current interest object”, for example.
With reference to
In this case, so as not to difficult to see an indicator by the user's finger, the “content of current interest object” moves the place (
Next, with reference to
In such a case, if the CPU 11 analyses the captured image inputted from the image sensor 15 and detects that the user's hand covers a part of the GUI component, a part or all of the GUI component is deformed.
Next, a method of controlling information presented by the CPU 11 in the past (shown by GUI component in
Each piece information I (x) shown in
The information I (x) holds the node ID and a node ID of connected other node. In addition, the information I (x) may hold search results, in a case where the external search engines, the photo databases, the storage of the information processing apparatus 1 are searched in the past. There may be a part of the search results.
The information I (x) also holds the information of the corresponding GUI component. For example, in a case where the information I (x) is a pointer for the search results of the external search engines on the Internet, the information corresponds to the “Document” and holds the information that the GUI component of the “data (Document)” shown in
The information I (x) holds the icon as necessary. For example, in a case where the information I (x) is a pointer for the search results of photo storage, the information I (x) holds those picked up from some of the search results and iconized at a low resolution by the CPU 11.
Since the CPU 11 manages the information I in the data structure shown in
The information I (x) holds the keyword to tag the information. There may be a plurality of keywords. An example of the keyword is a search keyword used when the search result is presented. A specific place and a specific thing (for example, “food”, etc.) are set as the keywords automatically set by the CPU 11. For example, in ST13 of
Accordingly, the CPU 11 can invoke the information I (x) by using the keyword to tag the information I (x). In fact, processing of extracting the information about a center of a topic by the user, i.e., the focused information, by using the keyword will be hereinafter described with reference to
(1-8. Processing Executed and GUI Movement when Information Presented in the Past is Accessed)
As shown in
In a case where it is determined that the current interest object is not outputted in the past in ST141 (ST142, No), the CPU 11 interprets that the behavior of the user designates a new information search and the processing of the new information search is executed (ST147).
As the processing of the new information search, a website or other data base (including photo storage and life logs on which a user's health condition is recorded) is searched on the basis of the keyword extracted in ST13.
In a case where it is determined that the content of current interest object is outputted in the past ST141 (ST142, Yes), the CPU 11 searches the content corresponding to the content of current interest object from a plurality of pieces of information I (x) presented in the past on the basis of the keyword and the behavior of the user extracted in ST13 (ST143). In this embodiment, an example of the “pieces of the information I (x) presented in the past” is the whole information I, i.e., all information already presented is to be searched in ST143.
Then, the CPU 11 extracts the information I (x) searched in ST143 (ST144). The CPU 11 moves a focus on the extracted information I (x) (ST146). However, in this embodiment, processing of narrowing an object to be searched pointed by the information I (x) is performed beforehand (ST145).
As described with reference to
Note that in a case where the information I (x) is the search of the web site, the tagging keyword can be a keyword inputted to a search query to a search engine.
As a specific mode of narrowing in ST145, in this embodiment, in a case where it returns to other presented information B from some information A, it uses a method of narrowing by using a keyword to tag information through to the information A and the information B.
As described above, by performing the processing along
As shown in
Furthermore, the CPU 11 may use the search keyword generalized and abstracted as the tagging keyword. For example, in context of consulting with the user agent about a lunch in a trip, when there are talks such as “Soba” and “Udon”, a word “food” by generalizing Soba is set to the keyword to tag the information I (Soba) as well as “Soba”.
Similarly, “Udon” and “food” are set to the keyword to tag the information I (Udon). Thus, the CPU 11 becomes capable of extracting a plurality of pieces of information I (x) by the keyword “food”. A cluster of the plurality of pieces of information extracted by the same keyword is referred to as a “layer” in this embodiment. The information processing apparatus 1 according to this embodiment uses the layer of the information extracted by, in particular, the generalized keyword.
It will be described with reference to conceptual diagrams of the layers in
In this embodiment, when the CPU 11 optimizes and displays the information I (1), the CPU 11 also searches in background and holds information relating to the information I (1) such as information I (12) and information I (13). The conceptual diagram of
When the CPU 11 searches the new information in ST147, the CPU 11 stores the information I (x) to the RAM 13 or the like as the pointer of the searched information for management. In this management, a tag is used. The CPU 11 manages that the pieces of information I having a common tagging keyword are capable of arranging in the same layer.
In
In this embodiment, the pieces of the information connected by the paths from the user agent icon UAI to the focused information are managed to belong to the “main layer”, but it is not limited thereto. For example, when the information I (11) is focused in
On the other hand, the information other than those is managed to belong a sublayer. In addition, the focus is moved, information groups belonging to the main layer are replaced.
Furthermore, in this embodiment, optimized display screens are generated by presenting the pieces of information belonging to the same layer with common visual effects or presenting them by surrounding with common frames. The visual effects include, for example, a color, transmittance, a bold, and the like. Examples of the common frames include those shown in
With the above-described configurations, the user becomes easily aware of a focus movement across the layers and becomes easily perceive the information when it returns to the past information or the like. In other words, ease of use is improved.
In addition, the information presented in the layers L2 and L3 is referred to as the derived items in
Thus, a display intensity difference among the respective layers allows thinking of the user to be clear. Note that by intentionally displaying the third layer and the second layer in an illegible status, only the presence may be allowed to be recognized by the user.
Furthermore, in this embodiment, relating to information acquired from outside, etc. and presented or display elements (GUI elements) corresponding thereto, a display position and a layout are dynamically and sequentially changed so as not to constantly block the view taking the relationship between the user's lines of sight and the finger position. The optimization of the display screen is sequentially performed.
In this example, the information processing apparatus 1 selects nouns appeared in the user's talk for the tagging keyword. In addition, the information processing apparatus 1 interprets a voice tone or content of the talk as a command to the user agent. Furthermore, the information processing apparatus 1 holds words including positive terms such as “want to see”, “want to go”, and “curious about” as user's thinking information on the agent.
An example situation is as follows: two people of a user A and a user B chat over a table and the agent is activated, too. Under the situation, the two users are talked including the agent provided by the information processing apparatus 1 as follows: (
Here, words such as the “mythological island of demons” and the “where” are extracted as the keywords.
The CPU 11 recognizes the talk of the user B here as the command to the user agent on the basis of the voice tone and the talk content.
Here, the information processing apparatus 1 recognizes that a time condition of “a little while ago” is present in the user's talk and extracts the related photo group shown in
Various modifications and alterations of the above-described embodiments may be available. For example, a part of an output destination of the output unit 21 of the information processing apparatus 1 can be changed. For example, as shown in
In other words, in a highly confidential operation step such as the money settlement processing and a personal information input, an operation screen is temporarily sent to a registered personal terminal screen to encourage the operation, its history is displayed on the original common display surface after the operation is completed.
With this consiguration, personal information can be protected even in an operation in a public environment. At the time of operation of the money settlement processing, the personal information input, and the like, it temporarily moves to the personal terminal screen to cause a relay operation. Thus, the common display surface and the private display surface are used in combination to thereby ensuring confidentiality.
In the above-described embodiments, the display surface 201 is set to the table or the wall. However, in order to illustrate that the spirit of the present disclosure is not limited thereto, hereinafter an embodiment to which the spirit of the present disclosure is applied is shown by an inside of an automobile.
In
An example of a specific issue in the use scene includes that visibility is poor even if the GUI is projected in a case where sunshine is strong or in a case where forward scenery through the windshield is too bright such as while snow scenery.
Then, the CPU 11 divides insides of the displayable area into an “appropriate display area” and an “inappropriate display area” on the basis of the image captured by the image sensor 15. And, the GUI is displayed only on the “appropriate display area” of these.
The CPU 11 determines an area with a low contrast because of sunshine glare, too bright forward scenery, etc. is determined as the “inappropriate display area”. On the other hand, an area with a high contrast because of sunlight diminished by shading with an exterior building, sufficiently low brightness of the forward scenery through the windshield, or the like is determined as the “appropriate display area”.
Note that the “displayable area” may not only be the windshield but also be the dashboard in front of the assistant driver's seat. The CPU 11 determines the dashboard in front of the assistant driver's seat as the “appropriate display area” as long as there are no hindrance circumstances such as the sunshine glare.
Sensing of brightness and coloring of the display surface 201 is performed by the image sensor 15, which is not limited thereto. A dedicated sensor may be arranged inside or outside a vehicle. As compared with the configuration that the image sensor 15 that is an internal camera of the vehicle also functions as a sunshine sensor, a configuration that the dedicated sunshine sensor is added can further improve accuracy of determining the brightness and the coloring.
In this modification, arrangement of the GUI may be controlled as follows: The image sensor 15 senses a position of a human face in the assistant driver's seat. The CPU 11 adjusts the display position (mainly, height) of the GUI to a height of eyes of the human in the assistant driver's seat on the basis of the position of the human face in the assistant driver's seat.
In addition, an area having a background relatively unchanged is sensed by using the internal camera or an external camera of the vehicle such as the image sensor 15 and the sunshine sensor. Inside the area, an area in the “displayable area” specified on the basis of visibility from the position of the line of sight of the user seated on the assistant driver's seat may be determined as the “appropriate display area” by the CPU 11.
In the above-described embodiments and examples, the search results are not sequentially held in time-series, but histories having layer structures are held by linking the respective search result data items with nodes and paths and further classifying by search context (search branch), information theme, or the like. Only the layer of the theme to be focused in that time is shown to the user. The user is visually attracted on the point to be currently focused by moving the node and drawing animation of the path.
According to the embodiments and examples, it becomes easy to recognize a current interest search path in a space not optimized for the information search. Furthermore, returning to a branch point in the past in the middle of the search, searching again from the branch point in the past by changing a consideration viewpoint, or the like becomes possible through natural conversation with the agent (without using complex commands and GUIs).
In addition, according to the embodiments, since operation procedures specific to a variety of services are performed instead thereof and the user can concentrate on expression of a request and confirmation and utilization of a result, a plurality of services can be used by a single procedure.
In addition, according to the embodiments, since it becomes possible to always make focus only on the most important information at the time from a vast amount of related information, a thinking load of the user can be minimized.
In addition, according to the embodiments, by a 3D layered screening method, a consideration path of the user is held. There is provided an interface capable of always returning to an optimal branched point as necessary. It is possible to effectively assist reconsideration, comparison, and confirmation by the user.
In addition, according to the embodiments, by holding and tracing procedure lines, the most important item is always displayed in front of the user's eyes in accordance with a use place, and the layout is dynamically adjusted so as not to interfere a view including the GUI and a finger at the time of inputting with a finger. Accordingly, visibility is optimized in accordance with a location and a status.
Note that the present technology may also have the following structures.
(1)
An information processing apparatus, including:
a detection unit detecting behavior of a user;
a control unit determining a current interest object of the user depending on the detected behavior and sequentially generating display information including content corresponding to the determined interest object; and
an output unit outputting the generated display information,
the control unit optimizing a display mode of the display information depending on a change in the determined interest object on a basis of content of an interest object in the past of the user outputted from the output unit and content of the current interest object.
(2)
The information processing apparatus according to (1), in which
in a case where content outputted in the past from the output unit is determined as the current interest object on a basis of the behavior of the user, the control unit includes the content outputted in the past determined as the current interest object in the display information and optimizes the display mode of the display information.
(3)
The information processing apparatus according to (2), in which
the output unit generates the display information such that each display element corresponding to the content outputted in the past from the output unit is outputted in an output order by linking with lines.
(4)
The information processing apparatus according to (2) or (3), in which
the control unit optimizes the display mode of the display information by narrowing the content outputted in the past determined as the current interest object.
(5)
The information processing apparatus according to any of (2) to (4), in which
the control unit generates the display information such that the display element corresponding to the content outputted in the past determined as the current interest object is arranged at a center of a line of sight of the user.
(6)
The information processing apparatus according to any of (2) to (5), in which
the control unit includes content belonging to a layer common to the content outputted in the past determined as the current interest object together with the content outputted in the past determined as the current interest object in the display information.
(7)
The information processing apparatus according to (6), in which
the control unit generates the display information showing that a plurality of content items belongs to a common layer by a visual effect and/or a common frame.
(8)
The information processing apparatus according to any of (1) to (7), in which
the detection unit inputs a voice of the user, and the control unit determines the current interest object by analyzing the voice of the user inputted from the detection unit.
(9)
An information processing method, including:
a step of detecting behavior of a user;
a step of determining a current interest object of the user depending on the detected behavior;
a step of sequentially generating display information including content corresponding to the determined interest object;
a step of outputting the generated display information; and
a step of optimizing a display mode of the display information depending on a change in the determined interest object on a basis of content of an interest object in the past of the user outputted from the output unit and content of the current interest object.
(10)
A program causing an information processing apparatus to execute
a step of detecting behavior of a user;
a step of determining a current interest object of the user depending on the detected behavior;
a step of sequentially generating display information including content corresponding to the determined interest object;
a step of outputting the generated display information; and
a step of optimizing a display mode of the display information depending on a change in the determined interest object on a basis of content of an interest object in the past of the user outputted from the output unit and content of the current interest object.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2017-034175 | Feb 2017 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2017/045630 | 12/20/2017 | WO | 00 |
