Patent Grant
10627912
This application claims the benefit of Japanese Priority Patent Application JP 2016-099393 filed May 18, 2016, the entire contents of which are incorporated herein by reference.
The present disclosure relates to an information processing apparatus, an information processing system, and an information processing method.
In recent years, users can operate an information processing apparatus by various operation methods. For example, an information processing apparatus has been developed which detects a user's gesture to allow a user's operation.
JP 2010-191826A discloses an information processing apparatus which allows a user's operation on the basis of the user's gesture as described above. The information processing apparatus disclosed in JP 2010-191826A includes an imaging device, and the imaging device detects the user's gesture.
There may be a case where the user performs an erroneous operation when performing an operation on the information processing apparatus. The same goes for a case where the user performs a gesture operation. It is therefore desirable to provide a mechanism for allowing the user to cancel, when the user has performed a gesture operation, the operation based on the gesture.
According to an embodiment of the present disclosure, there is provided an information processing apparatus including: a process execution unit configured to execute a process relating to a user's gesture recognized on a basis of information from a sensor. The process execution unit determines, during a period after the gesture has been recognized, whether or not there is an input of operation information based on a user's operation and the process execution unit refrains from executing the process relating to the recognized gesture when there is an input of the operation information during the period.
According to an embodiment of the present disclosure, there is provided an information processing system including: a process execution unit configured to execute a process relating to a user's gesture recognized on a basis of information from a sensor. The process execution unit determines, during a period after the gesture has been recognized, whether or not there is an input of operation information based on a user's operation, and the process execution unit refrains from executing the process relating to the recognized gesture when there is an input of the operation information during the period.
According to an embodiment of the present disclosure, there is provided an information processing method including: executing a process relating to a user's gesture recognized on a basis of information from a sensor; determining, during a period after the gesture has been recognized, whether or not there is an input of operation information based on a user's operation; and refraining from executing the process relating to the recognized gesture when there is an input of the operation information during the period.
According to the present disclosure described above, when the user has performed an erroneous operation on the basis of a gesture, the uses can cancel the erroneous operation.
Note that the effects described above are not necessarily limitative. With or in the place of the above effects, there may be achieved any one of the effects described in this specification or other effects that may be grasped from this specification.
Hereinafter, (a) preferred embodiment(s) of the present disclosure will be described in detail with reference to the appended drawings. In this specification and the appended drawings, structural elements that have substantially the same function and structure are denoted with the same reference numerals, and repeated explanation of these structural elements is omitted.
Note that description will be provided in the following order.
The cell-phone 200, including a loud speaker and a microphone, can communicate with other terminals by communicating with a base station included in a mobile communication network. In addition, the cell-phone 200 includes a display unit for displaying information, as well as a touch panel formed on the display unit. A user performs various operations relating to functions or applications of the cell-phone 200 by performing operations using the touch panel. Note that the component used for operating the cell-phone 200, which is not limited to a touch panel may be a component, such as a direction key, a hard button, or a jog dial.
In addition, the cell-phone 200 has various functions and applications besides the call function, some of which may be realized by communicating with the server 400 via the communication network 300. For example, email, short message, social networking service, photography, music reproduction, browsing function, map display, and the like, may be included as types of functions and applications of the cell-phone 200.
The server 400 is connected to the cell-phone 200 via the communication network 300. The server 400 executes processes in accordance with applications or functions of the cell-phone 200. For sample, the server 400 processes voice data transmitted from the cell-phone 200.
In the aforementioned system configuration, a user can operate functions of the cell-phone 200 or applications being executed in the cell-phone 200 by performing various operations on the small-size terminal 100, with the small-size terminal 100 being attached to the user's ear.
<1-2 Configuration of Small-Size Terminal>
The communication system of the embodiment of the present disclosure has thus been outlined above. In the following, there will be described a configuration of the small-size terminal 100 of the embodiment of the present disclosure.
The appearance of the small-size terminal 100 has been described above. In the following, there will be described a configuration of the information processing system of the present embodiment.
The user can perform a plurality of different operations on the button 102. For example, the user may press the button 102 once briefly, or twice briefly, as different operation methods performed by the user on the button 102. Alternatively, the user may press the button 102 for a predetermined time period, as a different operation method performed by user on the button 102. Note that the button 102 may be an electrostatic or a pressure-sensitive touch panel, or may be a physical key. Alternatively, there may be one or more buttons 102, the number of which is not limited. However, a smaller number of the buttons 102 are preferred due to the limited space for arranging the buttons 102 in a small-size terminal, such as the small-size terminal 100 of the present embodiment.
The processing unit 104, which is connected to each part of the small-size terminal 100, processes information from each part. For example, the processing unit 104 detects a signal resulting from the user's operation to the button 102. In addition the processing unit 104 processes voice data received from the cell-phone 200 via the short-distance communication unit 114, and outputs the processed voice data to the loud speaker 108. In addition, the processing unit 104 processes the voice received by the microphone 110 to generate voice data, and transmits the voice data to the cell-phone 200 via the short-distance communication unit 114.
In addition, the processing unit 104 processes a signal from the sensor 112. The sensor 112 includes a proximity sensor 116, an acceleration sensor 118, and a gyro sensor 120. The processing unit 104 detects that the small-size terminal 100 is attached to the user's ear, on the basis of a detection signal generated by the proximity sensor. In addition, the processing unit 104 processes signals output from the acceleration sensor 118 and the gyro sensor 120, and transmits the processed signal to the cell-phone 200. The battery 106 supplies electric power stored therein to each part of the small-size terminal 100, and each part operates by the supplied power.
The loud speaker 108 outputs voice on the basis of the voice data processed by the processing unit 104. When, for example, the call function is being used in the cell-phone 200, the loud speaker 108 outputs the voice of the call on the basis of voice data transmitted from the cell-phone 200. In addition, the microphone 110 receives voice and sends an audio signal to the processing unit 104. For example, the microphone 110 receives voice uttered by the user and sends an audio signal to the processing unit 104.
Note that, the proximity sensor 116 may be, for example, a proximity sensor capable of detecting a target object without touching the object. The proximity sensor may detect a target object using infrared light, supersonic wave, or electromagnetic wave, for example. In yet another example, the proximity sensor may detect a target object using variation of eddy current due to electromagnetic induction or variation of electrostatic capacitance caused by approaching of the target object.
In addition, the acceleration sensor 118 detects acceleration applied to the small-size terminal 100. When, for example, the small-size terminal 100 is attached to the user, the acceleration sensor 118 detects the acceleration when the user shakes his/her neck. There are various types of the acceleration sensor 118, such as optical or semiconductor, and the small-size terminal 100 of the present embodiment may include any type of acceleration sensor.
In addition, the gyro sensor 120 detects an angular velocity and an angular acceleration of the small-size terminal 100. When, for example, the small-size terminal 100 is attached to the user, the gyro sensor 120 detects the angular velocity and the angular acceleration when the user shakes his/her neck. There are also various types of the gyro sensor 120, such as hydraulic or optical, as with the acceleration sensor 118, and the small-size terminal 100 of the present embodiment may include any type of gyro sensor.
In addition, the short-distance communication unit 114 uses a short-distance wireless communication interface, such as Bluetooth (registered trademark), to communicate with other devices, such as the cell-phone 200. Since the small-size terminal 100 uses the low-capacity battery 106, it is preferred to use a power-saving wireless communication interface, such as Bluetooth Low Energy. Note that, the short-distance communication unit 114 is not limited to the aforementioned interface and may be a short-distance wireless communication interface such as ZigBee (registered trademark).
Note that the small-size terminal 100 and the cell-phone 200 are connected to each other by pairing. Pairing is performed by searching for devices located within a reachable range of radio wave and selecting a device that the user wants to connect from the detected devices. As for pairing, the small-size terminal 100 and the cell-phone 200 may be configured to authenticate each other using a password. The cell-phone 200 and the small-size terminal 100 are allowed to communicate with each other by performing pairing in the aforementioned manner.
<1-3. Configuration of Cell-Phone and Server>
The configuration of the small-size terminal 100 has been described above. In the following, there will be described a configuration of the cell-phone 200 of the present embodiment. The cell-phone 200 includes a processing unit 202, a short-distance communication unit 204, and a cellular communication unit 206, as illustrated in
The short-distance communication unit 204 uses a short-distance wireless communication interface to communicate, with other devices, such as the small-size terminal 100. The short-distance communication unit 204 may be a short-distance wireless communication interface, such as Bluetooth or ZigBee as described above. The cellular communication unit 200 is used for communicating with devices located remotely compared with the short-distance communication unit 204. For example, the cellular communication unit 200 may be an interface based on a communication standard specified by 3GPP, such as Long Term Evolution.
The voice generation unit 308 generates various voice data. When, for example, a call function is being executed in the cell-phone 200, the voice generation unit 208 generates voice data of the communication party. In addition, the voice generation unit 208 generates voice data based on an application being executed in the cell-phone 200. Note that, the generated voice data may be sent to the small-size terminal 100 via the short-distance communication unit 204.
The gesture recognition unit 210 recognizes the user's gesture on the basis of the signal output by the acceleration sensor 118 and/or the gyro sensor 120. A gesture may be recognized by, for example, comparing a pattern of a signal of the acceleration sensor 118 and/or the gyro sensor 120 associated with a preliminarily stored predetermined gesture with a signal output by the acceleration sensor 118 and/or the gyro sensor 120 and received by the gesture recognition unit 210. The gesture recognized by the gesture recognition unit 210 may be, for example, the user's shaking of his/her head sideways or lengthways with the small-size terminal 100 being attached to the user's ear. The gesture recognition unit 210 may recognize that the user has performed an affirmative operation by shaking his/her head lengthways. Alternatively, the gesture recognition unit 210 may recognize that the user has performed a negative operation by shaking his/her head sideways.
The button operation recognition unit 212 recognizes that the user has operated the button 102 of the small-size terminal 100. For example, the user's operation of the button 102 may be recognized by sending of a button operation signal from the small-size terminal 100 to the cell-phone 200 when the user operates the button 102.
The voice operation recognition unit 214 recognizes a voice operation performed by the user. For example, the microphone 110 of the small-size terminal 100 receives the user's voice, and the small-size terminal 100 transmits the user's voice data to the cell-phone 200. The cell-phone 200 which has received the voice data transmits the voice data to the server 400. A voice analysis unit 404 of the server 400 analyzes the received voice data and sends an analysis result to the cell-phone 200. The voice operation recognition unit 214 recognizes the voice operation performed by the user on the basis of the received analysts result.
The process execution unit 216 executes various processes. For example, the process execution unit 216 executes an application installed in the cell-phone 200. In addition, the process execution unit 216 receives operation information from the gesture recognition unit 210, the button operation recognition unit 212, and the voice operation recognition unit 214, and performs a process based on the operation information, on the application being executed.
Note that, the analysis of voice data in the aforementioned example is performed in the voice analysis unit 404 of the server 400. However the analysis of voice data performed in the voice analysis unit 404 of the server 400 may be performed in the voice operation recognition unit 214 of the cell-phone 200.
Note that, the gesture recognized by the gesture recognition unit 210 may be user's tilting of his/her head to the left or right. When, for example, the gesture recognition unit 210 recognizes that the user has tilted his/her head to the left or right, the process execution unit 216 may fast forward or rewind a music to be reproduced in a music reproduction application. Alternatively, the gesture recognized by the gesture recognition unit 210 may be the user's pushing of his/her face forward. When, for example, the gesture recognition unit 210 recognizes that the user has pushed his/her face forward, the process execution unit 216 may magnify a photograph in a photograph display application. Alternatively, the gesture recognized by the gesture recognition unit 202 may the user's pulling of his/her face backward. When, for example, the gesture recognition mat 210 recognizes that the user has pulled his/her face backward, the process execution unit 216 may shrink the photograph in the photograph display application.
<<2. Exemplary Process Relating to an Activation Timing of a Sensor for Recognizing a Gesture Operation>>
A system configuration of the present embodiment has been described above. In the following, there will be described an exemplary process in an information processing system of the present embodiment.
First, at S102, the cell-phone 200 receives a message. On this occasion, the application being executed by the process execution unit 216 sends a notification indicating message reception to the voice generation unit 208, the gesture recognition unit 210, and the voice operation recognition unit 214. Next, at S104, the voice generation unit 208 which has received the message from the application generates voice data to order to audibly notify the user of the reception of the message at S102. The cell-phone 200 then transmits the generated voice data to the small-size terminal 100, and the processing unit 104 of the small-size terminal 100 reproduces the received voice data. Here, the content of the voice data to be reproduced may be, for example, “A message has been received. Will it be read?”.
At S106, the voice operation recognition unit 214 which has received the notification from the application transitions to a voice operation waiting state to wait for a voice operation from the user. In parallel with the processing at S106, the sensor of the small-size terminal 100 is activated at S108. This may be realized by generation of an activation signal by the gesture recognition unit 210 which has received the notification from the application, and transmission of the activation signal to the small/size terminal 100.
At S108, the acceleration sensor 118 and/or the gyro sensor 120 to be used for gesture recognition are activated. At S110, the gesture recognition unit 210 transitions to a gesture waiting state to wait for a gesture operation from the user, in synchronization with the activation of the acceleration sensor 118 and/or the gyro sensor 120. Note that the activation of the acceleration sensor 118 and/or the gyro sensor 120 may be realized by sending, to the gesture recognition unit 210, information indicating that the voice operation recognition unit 214 has transitioned to the voice operation waiting state. On this occasion, the activation of the acceleration sensor 118 and/or the gyro sensor 120 synchronizes with the transition of the voice operation recognition unit 214 to the voice operation waiting state.
Next, at S112, a sound effect is reproduced indicating that the gesture recognition unit 210 is in the gesture waiting state. The sound effect allows the user to recognize when the gesture recognition unit 210 has transitioned to the gesture waiting state. Next, at S114, the gesture recognition unit 210 determines whether or not the user has performed a gesture. When the gesture recognition unit 210 recognizes a gesture at S114, the process flow proceeds to S116.
At S116, operation information relating to the gesture recognized at S114 is temporarily stored in a storage unit which is not illustrated. Next, at S118, the sound effect indicating that the gesture recognition unit 210 is in the gesture waiting state is stopped. Then, at S120, the gesture recognition unit 210 releases the gesture waiting state and terminates gesture recognition. Accordingly, the acceleration sensor 118 and/or the gyro sensor 120 transition to a dormant state at S122.
Next, at S124, the process execution unit 216 executes, on the application, a process relating to the gesture temporarily stored at S116. Specifically, when the content of the voice data reproduced at S104 is “A message has been received. Will it be read?” and the user has performed a gesture of shaking his/her head lengthways at S114, the process execution unit 210 executes a process of reading the received message at S124.
When the gesture recognition unit 210 does not recognize a gesture at S114, the process flow proceeds to S126. At S126, the gesture recognition unit 210 determines whether or not a predetermined time period has elapsed. When the gesture recognition unit 210 that a predetermined time period has elapsed at S126, the process flow proceeds to S128.
At S128, the sound effect indicating that the gesture recognition unit 210 is in the gesture waiting state is stopped. Then, at S130, the gesture recognition unit 210 releases the gesture waiting state and terminates gesture recognition. Accordingly, the acceleration sensor 118 and/or the gyro sensor 120 transition to a dormant state at S132.
As has been described above, the acceleration sensor 118 and/or the gyro sensor 120 are activated on the basis of reception of a notification indicating message reception from the application by the gesture recognition unit 210. Actuating the acceleration sensor 118 and/or the gyro sensor 120 on the basis of the notification from the application as thus described causes the sensor 112 to be activated at an appropriate timing with regard to the operation of the application, whereby it is possible to reduce power consumption of the small-size terminal 100. This is very advantageous for a terminal having the low-capacity battery 106, such as the small-size terminal 100.
In addition to the aforementioned processing, the sensor 112 to be activated may be selected in accordance with the notification from application or the type of application. When, for example, the gesture to be recognized on the basis of the notification from application or the type of application is a simple gesture, either the acceleration sensor 118 or the gyro sensor 120 may be activated. Alternatively, when a complicated gesture is the gesture to be recognized, both the acceleration sensor 118 and the gyro sensor 120 may be activated. On this occasion, the notification from application or the type thereof may be stored in association with the gesture to be recognized in accordance with the notification or the type.
Note that, the notification from application in the aforementioned example is a notification indicating that a message has been received. In addition, the gesture to be recognized by the notification from the application is a user's gesture of shaking his/her head lengthways (affirmative response) or a user's gesture of shaking his/her head sideways (negative response).
Alternatively, a mode of the sensor 112 to be activated may be selected on the basis of the notification from application or the type of application. The mode of the sensor 112 to be selected may be, for example, a mode having a different sampling frequency. A mode with a high sampling frequency is selected when a gesture with a fast motion is recognized, whereas a mode with a low sampling frequency is selected when a gesture with a slow motion is recognized. As thus described, selection of the sensor 112 or selection of the mode of the sensor 112 to be activated in accordance with the gesture to be recognized allows further reduction in power consumption of the small-size terminal 100.
Note that, in the aforementioned embodiment, a sound effect is reproduced at S112, which indicates that the gesture recognition unit 210 is in a gesture waiting state. However, the sound effect may be reproduced while the user is performing a gesture, i.e., while the acceleration sensor 118 and/or the gyro sensor 120 are detecting acceleration, angular velocity, or angular acceleration. Alternatively, vibration may be used instead of a sound effect to indicate that the gesture recognition unit 210 is in a gesture waiting state. Alternatively, a message may be reproduced instead of a sound effect. For example, the message may be something like “Gesture recognition will be started”. Alternatively, a message may likewise be reproduced when gesture recognition is terminated. On this occasion, a message, such as “gesture recognition will be terminated”, for example, may be reproduced.
Alternatively, when the user performs a voice operation instead of a gesture operation in the aforementioned exemplary process, the acceleration sensor 118 and/or the gyro sensor 120 may transition to a dormant state in synchronization with the recognition of the voice operation.
<<3. An Exemplary Process in which a Gesture Operation is Recognized in Parallel with Another Operation Method>>
An exemplary process relating to an activation timing of the sensor 112 used for gesture operation in the present embodiment has been described above. In the following, there will be described an exemplary process of recognizing a gesture operation and another operation method. In the information processing system of the present embodiment, recognition of a voice operation and a button operation is performed in parallel with recognition of a gesture operation. Accordingly, the user can perform various operations by combining a plurality of operation methods.
The cell-phone 200 receives a message at S202 in a similar manner to the exemplary process in
In parallel with the processing at S204, the acceleration sensor 118 and/or the gyro sensor 120 of the small-size terminal 100 are activated at S206, and the gesture recognition unit 210 transitions to a gesture waiting state to wait for a gesture operation from the user. Additionally, in parallel with the processing at S204 and S206, the button operation recognition unit 212 transitions to a button operation waiting state to wait for a button operation from the user (S208).
Then, at S210, the gesture recognition unit 210 determines whether or not the user has performed a gesture. When the gesture recognition unit 210 recognizes a gesture at S210, the process flow proceeds to S212. At S212, operation information relating to the gesture recognized at S210 is temporarily stored. Note that, when the gesture recognition unit 210 does not recognize a gesture at S210, the process flow proceeds to S214. At S214, the gesture recognition unit 210 determines whether or not a predetermined time period has elapsed. When the gesture recognition unit 210 determines at S214 that a predetermined time period has elapsed, the process is terminated.
Next, at S216, the button operation recognition unit 212 determines whether or not the user has operated the button 102. When the button operation recognition unit 212 recognizes at S216 that the user has operated the button 102, the button operation recognition unit 212 sends operation information relating to a button operation to the process execution unit 216, and the process execution unit 216 executes a process relating to the button operation (S218).
When the button operation recognition unit 212 does not recognize at S216 that the user has operated the button 102, the process flow proceeds to S220. At S220, the voice operation recognition unit 214 determines whether or not the user has performed a voice operation. When the voice operation recognition unit 214 recognizes at S220 that the user has performed a voice operation, the voice operation recognition unit 214 sends operation information relating to the voice operation to the process execution unit 216, and the process execution unit 216 executes a process relating to the voice operation (S222).
When the voice operation recognition unit 214 does not recognize at S220 that the user has performed a voice operation, the process flow proceeds to S224. At S224, the gesture recognition unit 210 sends operation information relating to the gesture temporarily stored at S212 to the process execution unit 216, and the process execution unit 216 executes a process relating to the gesture operation.
An exemplary process when gesture operation is performed first has been described above. However, the order of performing operations is not limited to the example described in
First, there will be described a relation with another operation method in a case where a button operation is performed earlier. When a button operation is performed earlier and subsequently a gesture operation is performed, a process relating to the button operation is executed in preference to the gesture operation. On this occasion, a sound effect indicating that the gesture recognition unit 210 is in a gesture waiting state may be stopped at the time point when button operation is performed. Similarly, when a button operation is performed earlier and subsequently a voice operation is performed, a process relating to the button operation is executed in preference to the voice operation.
Next, there will be described a relation with another operation method in a case where a gesture operation is performed earlier. When a gesture operation is performed earlier and subsequently a button operation is performed, a process relating to the button operation is executed in preference to the gesture operation. Similarly, when a gesture operation is performed earlier and subsequently a voice operation is performed, a process relating to the voice operation is executed in preference to the gesture operation.
Next, there will be described a relation with another operation method in a case where a voice operation is performed earlier. When a voice operation is performed earlier and subsequently a button operation is performed, a process relating to the button operation is executed in preference to the voice operation. Similarly, when a voice operation is performed earlier and subsequently a gesture operation is performed, a process relating to the voice operation is executed in preference to the gesture operation.
The priority of each operation method is not limited to the aforementioned example. In addition, the priority of each operation method may be changed. For example, weighting is performed on each operation method, and the process execution unit 216 may execute a process relating to each operation method in consideration of the weighting. The weighting of each operation method may be set by the user. Alternatively, weighting of each operation method may be set in accordance with the recognition precision of each operation method.
Note that, respective operation methods described above may be used in combination. For example, a button operation may be combined with a gesture operation. When a music reproduction application is being executed by the process execution unit 216, the user's tilting of his/her head to the right while touching the button 102 may allow the process execution unit 216 to execute a process of reproducing the next music.
Alternatively, a voice operation may be combined with a gesture operation. When a message application is being executed by the process execution unit 216, the user's shaking of his/her head lengthways while uttering “OK” may allow the process execution unit 216 to input the letters “OK” in an emphasized manner. In addition, the process execution unit 216 may input an image (stamp) corresponding to the letters “OK” instead of the process of emphasizing the letters.
Alternatively, a voice operation may be combined with a button operation. When a message application is being executed by the process execution unit 216, the user's uttering “OK” white touching the button 102 may allow the process execution unit 216 to input the letters “OK” in an emphasized manner. Alternatively, the process execution unit 216 may input an image (stamp) corresponding to the letters “OK” instead of the process of emphasizing the letters.
In the information processing system of the present embodiment described above, a gesture operation, a voice operation, and a button operation are performed in parallel. Accordingly, the user can perform various operations using a plurality of operation methods.
<<4. An Exemplary Process in which a Recognized Gesture Operation is Canceled>>
An exemplary process in which a gesture operation, a voice operation, and a button operation are performed in parallel has been described above. In the following, there will be described an exemplary process of canceling a gesture operation. There is a possibility of misrecognizing a gesture operation and therefore it is preferred to prepare a process of canceling the recognized gesture operation.
At S302, the small-size terminal 100 reproduces a voice “Reply with OK?”, for example, in order to cause the user to determine whether or not to reply to a message. At S306, the gesture recognition unit 210 transitions to a gesture waiting state to wait for a gesture operation from the user.
Then, at S306, the gesture recognition unit 210 determines whether or not the user has performed a gesture. When the gesture recognition unit 210 recognizes a gesture at S306, the process flow proceeds to S308. Note that, in the following, there will be described an exemplary process in a case where the gesture recognized at S306 is the user's gesture of shaking his/her head lengthways (nod).
At S308, the gesture recognition unit 210 sends operation information relating to the gesture recognized at S306 to the process execution unit 216, and the process execution unit 216 stores the operation information relating to the recognized gesture. Note that, when the gesture recognition unit 210 does not recognize a gesture at S306, the process flow proceeds to S310. At S310, the gesture recognition unit 210 determines whether or not a predetermined time period has elapsed. When the gesture recognition unit 210 determines at S310 that a predetermined time period has elapsed, the process is terminated.
Next, at S312, a voice notification is performed on the basis of the gesture recognized at S306. When the gesture recognized at S306 is a gesture relating to a “nod” indicating an affirmative response, a voice notification such as “Reply in three seconds”, for example, is performed at S312.
Next, at S314 the gesture recognition unit 210 transitions to a cancellation gesture waiting state to recognize a cancellation gesture for canceling the operation relating to the gesture recognized at S306. For example, the cancellation gesture may be the user's gesture of shaking his/her head sideways. Here the cancellation gesture is a predetermined specific gesture.
Next at S316, the gesture recognition unit 210 determines whether or not the user has performed a cancellation gesture. When the gesture recognition unit 210 recognizes a cancellation gesture at S316, the gesture recognition unit 210 terminates gesture recognition (S318). Then, at S320, the gesture recognition unit 210 sends operation information relating to the cancellation gesture recognized at S316 to the process execution unit 216, and the process execution unit 216 executes a cancellation process. Specifically, the process execution unit 216 does not execute the process relating to the gesture stored at S308. In other words, the operation relating to the gesture recognized at S306 is canceled. Then, at S322, the small-size terminal 100 performs voice notification in order to notify the user that the operation recognized at S306 has been canceled. The content of the voice reproduced at S322 may be for example, “Reply canceled”.
When the gesture recognition unit 210 does not recognize a cancellation gesture at S316, the process flow proceeds to S324. At S324, the gesture recognition unit 210 determines whether or not a predetermined time period has elapsed. The time period at S324 is a cancellation period set for the user to cancel the operation relating to the gesture recognized at S306. Further, during the cancellation period, the process execution unit 216 waits for an input of operation information relating to the cancellation gesture from the gesture recognition unit 210.
When the gesture recognition unit 210 determines at S324 that a predetermined time period has elapsed, the process flow proceeds to S326. At S326, the gesture recognition unit 210 terminates gesture recognition and notifies the process execution 216 that the cancellation period has elapsed. Then, at S328, the process execution unit 216 executes a process relating to the gesture stored at S308. At S330, a voice notification such as “replied”, for example, is performed in order to indicate that a process relating to the gesture operation performed by the user at S306 has been performed.
Note that, the process execution unit 216 may dynamically set the length of the cancellation period for waiting for an input of operation information relating to the cancellation gesture from the gesture recognition unit 210. For example, the process, execution unit 216 may set the length of the cancellation period on the basis of information acquired from an application being executed.
On this occasion, the process execution unit 216 may set the length of the cancellation period on the basis of the type of the application being executed. For example, the process execution unit 216 may set the length of the cancellation period to be long for a message transmission and reception application, and set the length of the cancellation period to be short for a music reproduction application. This is because, in transmission of a message, the user may transmit important information and therefore the user determines transmission of a message more carefully. On the other hand, this is because the user suffers no significant disadvantage even if the user performs an erroneous operation in music reproduction. Accordingly, there is set a cancellation period of an appropriate length in accordance with the type of the application being executed.
Alternatively, the process execution unit 216 may set the length of the cancellation period on the basis of the state of the application being executed. For example, the process execution unit 216 may set the length of the cancellation period to be long for an operation that terminates moving image photographing in a moving image photographing application, and set the length of the cancellation period to be short for an operation that starts moving image photographing. In other words, the process execution unit 216 sets different lengths for the cancellation periods for an operation in a state before starting moving image photographing and an operation in a state during moving image photographing. This is because, it is generally expected to photograph the motion of the subject quickly when starting moving image photographing whereas, when terminating moving image photographing, it suffices to later delete a moving image photographed for an unnecessarily long time. Accordingly, there is set a cancellation period of an appropriate length in accordance with the state of the application being executed.
Alternatively, the process execution unit 216 may set the length of the cancellation period on the basis of the type of the notification of the application being executed. For example, in a message transmission and reception application, the process execution unit 216 may set the length of the cancellation period to be long for an operation to a notification asking the user whether or not to transmit a message, and set the length of the cancellation period to be short for an operation to a notification asking the user whether or not to read a message. This is because the user determines transmission of a message more carefully, whereas the user suffers no significant disadvantage even if the user performs an erroneous operation when reading a message. Accordingly, there is a set a cancellation period of an appropriate length in accordance with the type of the notification of the application being executed.
Alternatively, the process execution unit 216 may set the length of the cancellation period on the basis of the magnitude of acceleration, angular velocity, or angular acceleration acquired from the acceleration sensor 118 and the gyro sensor 120. For example, when the acceleration, angular velocity, or angular acceleration acquired from the acceleration sensor 118 and the gyro sensor 120 is large, the process execution unit 216 determines that user is moving and sets the length of the cancellation period to be long. This is because, the user performs many erroneous operations when moving. Accordingly, there is set a cancellation period of an appropriate length in accordance with the user's moving state.
Alternatively, the user may set the cancellation period by operating the operation unit of the cell-phone 200. Alternatively, the cancellation period may not be provided depending on type or state of application, or notification from application. This is because it is preferred that no cancellation period is provided for an operation desired to be performed quickly. For example, it is desirable that no cancellation period is provided for a notification “music to be reproduced?” from the application so that a process relating to the operation is immediately performed.
Further, in
Alternatively, the operation information relating to the cancellation gesture at S316 may be generated when the gesture recognition unit 210 detects acceleration, angular velocity, or angular acceleration equal to or higher than a predetermined value. For example, a cancellation gesture may be performed by the user's fierce shaking of his/her head sideways. Accordingly, erroneous cancellation operations may be reduced.
An exemplary process of canceling a recognized gesture operation has been described above. Accordingly, in a case where a gesture operation has been misrecognized or the user has performed an erroneous operation, the user can cancel the once-recognized gesture operation.
Alternatively, the process to be canceled in the aforementioned example may be other than a process relating to a gesture operation. For example, a process relating to a recognized voice operation may be canceled by recognition of a cancellation gesture after the voice operation has been recognized. Alternatively, a process relating to a recognized button operation may be canceled by recognition of a cancellation gesture after the button operation has been recognized.
<<5. An Exemplary Process in which a Gesture Operation is Recognized with Machine Learning>>
An exemplary process to cancel a recognized gesture operation has been described above. In the following, there will be described an exemplary process of recognizing a gesture operation using a result of learning by machine learning. In recognition of a gesture operation, the precision of recognition may be raised by use of a result of learning by machine learning. However, the recognition precision of a gesture operation with machine learning may be insufficient. Therefore, the information processing system of the present embodiment performs recognition of a gesture operation on the basis of a particular rule, in addition to recognition of a gesture operation by machine learning.
The rule determination unit 220 preliminarily stores the acceleration pattern illustrated in
The aforementioned rule will be described more specifically in the following. For example, acceleration before starting a gesture is set as (x0, y0, z0), and acceleration when terminating the gesture is set as (x1, y1, z1). The rule determination unit 220 may use as a rule that the value of difference |(x1−x0), (y1−y0), (z1−z0)| between magnitudes of the two accelerations is equal to or smaller than a predetermined threshold value.
When the rule determination unit 220 recognizes a gesture at S404, the process flow proceeds to S406. At S406, the gesture recognition unit 210 sends operation information relating to the recognized gesture to the process execution unit 216, and the process execution unit 216 executes a process relating to the recognized gesture. When the rule determination unit 220 does not recognize a gesture at S404, the process is terminated. On this occasion, the process execution unit 216 does not execute a process relating to the gesture recognized by the machine learning determination unit 218 at S402.
As has been described above, a gesture is recognized by the machine learning determination unit 218 and the rule determination unit 220 in the gesture recognition unit 210 of the present embodiment. Accordingly, misrecognition of gestures by machine learning is reduced.
<<6. An Exemplary Process in which a Different Operation is Recognized in Accordance with the Strength of a Gesture to be Recognized>>
An exemplary process in which a gesture is recognized on the basis of a result of learning with machine learning and a predetermined rule has been described above. In the following, there will be described an exemplary process in which the operation to the recognized gesture varies in accordance with the strength of the gesture to be recognized. For example, the user may fiercely shake his/her head sideways when expressing strong negation, or may slowly shake his/her head sideways when expressing weak negation. In the information processing system of the present embodiment, the user can perform different operations in accordance with the strength of such gestures.
When the user slowly shakes his/her head sideways, the acceleration sensor 118 detects acceleration of a magnitude that does not exceed the threshold value T, as illustrated in the range A of
The process execution unit 210 which has received the operation information relating to the first gesture and the operation information relating to the second gesture executes a different process for each piece of the operation information. For example, the process relating to the first operation information may be a process of inputting “NO” with a usual size and color in the message input operation. On the other hand, the process relating to the second operation information may be a process of inputting “NO” with a size larger than usual and red color in the message input operation. As thus described in the aforementioned example, the user can input a strong negative message by fiercely shaking his/her head.
As has been described above, the user can perform different operations by performing gestures with different strengths. Note that, different operations are not limited to the aforementioned example. In the call application, for example, the process execution unit 216 may execute a “hold-on” operation when the user has performed the first gesture described above. Alternatively, the process execution unit 210 may execute a “hang up” operation when the user has performed the second gesture.
Further, an example has been described above in which the user can perform the first gesture and the second gesture by performing gestures with different strengths. However, there may be set a plurality of the aforementioned threshold values T, for example, different threshold values T1, T2 and T3 in ascending order, so as to allow other different gestures to be recognized. For example, a first gesture may be recognized when acceleration, angular velocity, or angular acceleration which is smaller than the threshold value T1 is detected. Alternatively, a second gesture may be recognized when acceleration angular velocity, or angular acceleration which is larger than the threshold value T1 and smaller than the threshold value T2 is detected. Alternatively, a third gesture may be recognized when acceleration, angular velocity, or angular acceleration which is larger than the threshold value T2 and smaller than the threshold value T3 is detected. Alternatively a fourth gesture may be recognized when acceleration, angular velocity, or angular acceleration which is larger than the threshold value T3 is detected. As thus described, there may be a set a plurality of threshold values for acceleration, angular velocity, or angular acceleration to be detected, allowing a plurality of gesture operations to be recognized in accordance with the set threshold value.
Further, different gesture operations are recognized in accordance with the strength of gestures in the aforementioned example. However, different gesture operations may also be recognized in accordance with the number of times a gesture is recognized. For example, it is conceivable that the user shakes his/her head many times when expressing strong negation. Therefore, the process execution unit 216 may execute the process relating to the second gesture when a same gesture has been recognized not less than a predetermined number of times. Alternatively, the process execution unit 216 may execute the process relating to the first gesture when a same gesture has been recognized less than a predetermined number of times. Alternatively, there may be set a plurality of threshold values with regard to the number of times a gesture is detected, and a plurality of gesture operations may be recognized in accordance with the number of times of set threshold value.
<<7. An Exemplary Process in which a Timeout Period is Set for Recognizing a Gesture>>
An exemplary process in which an operation of a gesture recognized in accordance with the strength of the recognized gesture varies has been described above. Note that, in the following, there will be described an exemplary process in which a timeout period for recognizing a gesture is set. In gesture recognition, there is a case where the user shakes his/her head sideways for a plurality of times, for example. On this occasion, there is a possibility that, although the user wants to perform an operation once, the operation may be performed for a plurality of times. In the present embodiment, a timeout period relating to a gesture recognition is set in order to prevent an operation from being performed against the user's intention.
A first-time gesture is then recognized at a time point denoted by P1. Subsequently, a change of acceleration caused by a plurality of times of similar gestures perforated by the user is detected. On this occasion, the gesture recognition unit 210 determines whether or not the timeout period has elapsed. When the gesture recognition unit 210 determines that the timeout period has not elapsed, the gesture recognition unit 210 does not recognize gestures performed during the period. Then, after the timeout period has elapsed, the gesture recognition unit 210 starts recognizing a gesture. Accordingly, the gesture recognition unit 210 recognizes a second-time gesture at a time point P2 after the timeout period has elapsed.
As has been described above, setting a timeout period prevents a process from being performed for a number of times unwanted by the user.
<<8. Supplement>>
It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.
In the aforementioned embodiment, for example, recognition of gestures, recognition of button operations, and recognition of voice operations are performed by the cell-phone 200. However, recognition of gestures, recognition of button operations, and recognition of voice operations may be performed by the server 400. Specifically, the small-size terminal 100 transmits, to the server 400 via a network, signals from the sensor 112, audio signals from the microphone 110, and operation signals of the button 102. The server 400 which has been received respective signals processes the received signals to perform recognition of gestures, recognition of button operations, and recognition of voice operations. The server 400 then transmits the result of recognition to the small-size terminal 100. The small-size terminal 100 then executes a process to an application in accordance with the received recognition result.
Alternatively, recognition of gestures, recognition of button operations, and recognition of voice operations may be performed by the small-size terminal 100. On this occasion, the processing unit 104 of the small-size terminal 100 processes the signals from respective parts to perform recognition of gestures, recognition of button operations, and recognition of voice operations. The small-size terminal 100 then executes a process to an application in accordance with the received recognition result. The small-size terminal 100 may be configured to execute all the processes as described above. Note that, in the aforementioned example, the small-size terminal 100 may be directly connected to the server 400 via the communication network 300 as illustrated in
Alternatively, the small-size terminal 100 may calculate a likelihood of gesture recognition and transmit the likelihood to the cell-phone 200, so that the cell-phone 200 may determine the gesture recognition using the likelihood of the gesture recognition calculated by the small-size terminal 100 and a predetermined threshold value. In this example, the small-size terminal 100 and the cell-phone 200 may be connected through a communication path such as Bluetooth, Wi-Fi, or a wired network.
The small-size terminal 100 executes a process relating to gesture recognition on the basis of information from the sensor 112 and generates an intermediate result of gesture recognition. When N types of gesture recognition are performed, an exemplary intermediate result may be a data set having listed therein a compatible value (threshold value) for each gesture for example, when two types of gestures, namely “nod” and “sideways shake” are recognized, the intermediate result may be expressed as “nod:0.8”, and “sideways shake:0.2”, for example, the intermediate result, which is a pair of a gesture and score generated in the small-size terminal 100, is transmitted to the cell-phone 200. The cell-phone 200 may determine the threshold value in accordance with the setting value, personalized policy for each user, or the situation of the user, the small-size terminal 100, or the cell-phone 200, and determine whether or not to process an operation relating to gesture recognition.
Further, in the aforementioned embodiment, the small-size terminal 100 is an information processing apparatus attached to the user's ear. However, the small-size terminal 100 is not limited thereto. For example, the small-size terminal 100 may be a wristband-type wearable terminal.
In addition, there may be provided a computer program for operating the processing units 104 and 202 in the manner described above. Alternatively, there may be provided a storage medium having such a program stored therein.
<<9. Conclusion>>
As has been discussed above, the sensor 112 used for gesture recognition in the information processing system according to the embodiment of the present disclosure is activated on the basis of a notification from an application. In addition, the sensor 112 used for gesture recognition is activated in synchronization with a voice operation. As a result, it is possible to reduce power consumption in the small-size terminal 100.
Additionally, the information processing system according to the embodiment of the present disclosure can wait for operations relating to gesture recognition, voice operations, and button operations in parallel. In addition, the user can perform operations combining operations relating to gesture recognition, voice operations, and button operations. Therefore, the user can perform various operations.
In addition, the information processing system according to the embodiment of the present disclosure executes a process of cancelling an operation relating to a once-recognized gesture. Accordingly, the user, even when having performed an erroneous operation, can cancel the operation. In addition, there is dynamically set a cancellation period for accepting a process of canceling an operation relating to a recognized gesture in the information processing system according to the embodiment of the present disclosure. Accordingly, there is set an appropriate cancellation period in accordance with the state or type of the application being executed.
Further, the effects described in this specification are merely illustrative or exemplified effects, and are not limitative. That is, with or in the place of the above effects, the technology according to the present disclosure may achieve other effects that are clear to those skilled in the art from the description of this specification.
Additionally, the present technology may also be configured as below.
(1) An information processing apparatus including:
a process execution unit configured to execute a process relating to a user's gesture recognized on a basis of information from a sensor, wherein
the process execution unit determines, during a period after the gesture has been recognized, whether or not there is an input of particular operation information based on a user's operation, and
the process execution unit retrains from executing the process relating to the recognized gesture when there is an input of the operation information during the period.
(2) The information processing apparatus according to (1), wherein
the process execution unit dynamically sets a length of the period.
(3) The information processing apparatus according to (2), wherein
the process execution unit executes an application,
the process execution unit executes a process relating to the gesture as an operation to the application, and
the process execution unit sets the period on a basis of information relating to the application acquired from the application.
(4) The information processing apparatus according to (3), wherein
the process execution unit sets the period in accordance with a type of the application.
(5) The information processing apparatus according to (3), wherein
the process execution unit sets the period in accordance with a state of the application.
(6) The information processing apparatus according to (3), wherein
the process execution unit sets the period in accordance with a type of notification of the application.
(7) The information processing apparatus according to (2), wherein
the sensor is an acceleration sensor or a gyro sensor, and
the process execution unit sets the period in accordance with a magnitude of acceleration or angular velocity acquired from the sensor.
(8) The information processing apparatus according to any one of (1) to (7), including:
a gesture recognition unit configured to recognize the gesture, wherein
the operation information is generated on a basis of a gesture recognized by the gesture recognition unit.
(9) The information processing apparatus according to any one of (1) to (8), including:
a gesture recognition unit configured to recognize the gesture, wherein
the sensor is an acceleration sensor or a gyro sensor, and
the operation information during the period is generated when the gesture recognition unit detects acceleration or angular velocity which is equal to or higher than a predetermined value.
(10) The information processing apparatus according to any one of (1) to (9), wherein
the process execution unit executes an application, and
the sensor is activated on a basis of a notification from the application being executed by the process execution unit.
(11) The information processing apparatus according to any one of (1) to (10), wherein
a mode of a sensor to be activated is selected in accordance with a notification from the application or a type of the application.
(12) The information processing apparatus according to any one of (1) to (10), wherein
the sensor includes a plurality of sensors, and
a sensor to be activated is selected in accordance with a notification from the application or a type of the application.
(13) The information processing apparatus according to any one of (1) to (12), including:
a gesture recognition unit configured to recognize the gesture, wherein
the sensor is an acceleration sensor or a gyro sensor, and
a gesture recognition unit recognizes a first gesture when acceleration or angular velocity which is smaller than a predetermined value is detected, and recognizes a second gesture when acceleration or angular velocity which is larger than the predetermined value is detected.
(14) The information processing apparatus according to any one of (1) to (13), including:
a gesture recognition unit configured to recognize the gesture, wherein
the gesture recognition unit recognizes a first gesture when a same gesture is recognized a smaller number of times than a predetermined number of times, and recognizes a second gesture when a same gesture is recognized a larger number of times than the predetermined number of times.
(15) The information processing apparatus according to any one of (1) to (14), including:
a gesture recognition unit configured to recognize the gesture, wherein
the gesture recognition unit includes
the gesture recognition unit recognizes the gesture when both the machine learning determination unit and the rule determination unit recognize the gesture.
(16) An information processing system including:
a process execution unit configured to execute a process relating to a user's gesture recognized on a basis of information from a sensor, wherein
the process execution unit determines, during a period after the gesture has been recognized, whether or not there is an input of operation information based on a user's operation, and
the process execution unit refrains from executing the process relating to the recognized gesture when there is an input of the operation information during the period.
(17) An information processing method including:
executing a process relating to a user's gesture recognized on a basis of information from a sensor;
determining, during a period after the gesture has been recognized, whether or not there is an input of operation information based on a user's operation; and
refraining from executing the process relating to the recognized gesture when there is an input of the operation information during the period.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2016-099393 | May 2016 | JP | national |
| Number | Name | Date | Kind |
|---|---|---|---|
| 20100125816 | Bezos | May 2010 | A1 |
| 20110317874 | Ikenoue | Dec 2011 | A1 |
| 20120218190 | Pechanec | Aug 2012 | A1 |
| 20120242604 | Kato | Sep 2012 | A1 |
| 20140059428 | Jeong | Feb 2014 | A1 |
| 20140267004 | Brickner | Sep 2014 | A1 |
| 20140267022 | Kim | Sep 2014 | A1 |
| 20150035773 | Fujii | Feb 2015 | A1 |
| 20150234572 | Arita | Aug 2015 | A1 |
| 20150268789 | Liao | Sep 2015 | A1 |
| 20160154624 | Son | Jun 2016 | A1 |
| Number | Date | Country |
|---|---|---|
| 2010-191826 | Sep 2010 | JP |
| Number | Date | Country | |
|---|---|---|---|
| 20170336873 A1 | Nov 2017 | US |
