This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2013-055019, filed on Mar. 18, 2013, the entire contents of which are incorporated herein by reference.
The embodiments discussed herein are related to an information processing device.
A technology has been proposed in which actions such as gestures, voice messages and the like performed by a user are detected at an information processing device, such as a personal computer (PC), a tablet terminal or the like, as operational instructions for the information processing device. Another technology has been proposed in which, for example, a gesture that a user performs while looking at a camera is detected as being intended as a gesture by the user, while a gesture that is performed by the user while facing in a direction other than at the camera is determined not to be intended as a gesture by the user.
According to an aspect of the embodiments, an information processing device includes: a processor; and a memory storing instructions, which when executed by the processor perform a procedure, the procedure including: (a) acquiring concentration level information relating to a degree of concentration of a user on the device; (b) acquiring action information that directly or indirectly represents an action of the user; (c) setting a first threshold that is for determining, in accordance with the concentration level information acquired in (a), whether or not the action information acquired in (b) is an operational instruction; and (d) determining, using the first threshold set in (c), whether or not the action of the user represented by the action information acquired in (b) is intended as an operational instruction by the user.
The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention.
The inventors have discovered that the sizes of actions performed as operational instructions (for example, the size of a gesture, the volume of a voice giving a voice message or the like) tend to be smaller when a user is concentrating on the operation than when the user is not concentrating on the operation. Errors in determinations of whether actions are operational instructions or not may be reduced by, in accordance with a level of concentration by a person, altering a threshold for determining whether the size of an action indicates an operational instruction. Herebelow, examples of embodiments of the disclosed technology are described in detail, referring to the attached drawings.
First Exemplary Embodiment
The concentration level information acquisition section 12 acquires concentration level information representing a physical quantity that is related to whether or not a user is concentrating. A position of a line of sight of the user may be mentioned as an example of a physical quantity related to whether the user is concentrating. In this case, the concentration level information acquisition section 12 acquires sightline position information representing a sightline position of the user to serve as the concentration level information. Body movements of the user (for example, movements of the head of the user) may be mentioned as another example of the physical quantity related to whether the user is concentrating. In this case, the concentration level information acquisition section 12 acquires body movement information representing body movements of the user to serve as the concentration level information. The sightline position information, body movement information or the like may be obtained by acquiring a video image, which is obtained by imaging the user with an imaging unit equipped with a function for capturing video images, and analyzing the acquired video image. Hereinafter, a mode in which sightline position information is acquired to serve as the concentration level information is principally described.
The action information acquisition section 14 acquires action information that represents, directly or indirectly, actions by a user that are performed to give operational instructions. For example, when an operational instruction is performed by the user moving their hand(s) and performing a predetermined gesture, the action information acquisition section 14 acquires the video image obtained by imaging of the user by the imaging section to serve as action information that directly represents the action by the user. The action information acquisition section 14 may acquire, instead of the video image, results extracted by image recognition of the hand of the user in the video image. As a further example, when an operational instruction is performed by a user performing a voice action of uttering a predetermined voice message, the action information acquisition section 14 acquires voice information, which is obtained by the voice message being detected by a voice detection unit, to serve as action information that directly represents the action by the user. The action information acquisition section 14 may acquire, instead of the voice information, results of voice recognition of the voice information. Hereinafter, a mode in which an operational instruction is performed by a user moving their hand to perform a predetermined gesture is principally described.
On the basis of the concentration level information acquired by the concentration level information acquisition section 12, the concentration level determination section 16 determines whether or not the user is concentrating.
As described above, when a person is concentrating on giving an operational instruction to an information processing device, the size of the action performed as the operational instruction (for example, the scale of a gesture or the volume of the voice giving a voice message) tends to be smaller. Accordingly, the threshold determination command section 18 acquires a determination result from the concentration level determination section 16 and, depending on whether it is determined by the concentration level determination section 16 that the user is in a concentrating state or not, the threshold determination command section 18 determines a first threshold that is for determining whether actions performed by the user are intended as operational instructions or not. The operational instruction determination section 20 uses the first threshold determined by the threshold determination command section 18 to determine whether or not the action by the user represented by the action information acquired by the action information acquisition section 14 is intended as an operational instruction by the user.
The information processing device 10 may be realized by, for example, a computer 30 illustrated in
The memory section 36 may be realized by a hard disk drive (HDD), flash memory or the like. An instruction recognition program 50 for causing the computer 30 to function as the information processing device 10 is memorized in the memory section 36. The CPU 32 reads the instruction recognition program 50 from the memory section 36 and loads the instruction recognition program 50 into the memory 34, and processes included in the instruction recognition program 50 are executed in sequence.
The instruction recognition program 50 includes a concentration level information acquisition process 52, a concentration level determination process 54, an action information acquisition process 56, a threshold determination command process 58 and an operational instruction determination process 60. By executing the concentration level information acquisition process 52, the CPU 32 operates as the concentration level information acquisition section 12 illustrated in
Thus, the computer 30 executing the instruction recognition program 50 functions as the information processing device 10. The instruction recognition program 50 is an example of an information processing program of the disclosed technology. The computer 30 may be a personal computer (PC) or a smart terminal that is a portable information processing device incorporating the functions of a portable information terminal (a personal digital assistant (PDA)), or the like.
The information processing device 10 may be realized by, for example, a semiconductor integrated circuit, and more specifically by an application-specific integrated circuit (ASIC) or the like.
When a user is giving operational instructions by performing predetermined gestures with their hands or uttering predetermined voice messages, there may be hand movements or voice utterances that are not intended to be operational instructions or of which the user is unaware. Therefore, in order to prevent mistaken recognition of unintentional hand movements or voice utterances as operational instructions, it is common for large movements and slight movements outside a pre-specified range, or voice utterances with volumes below a predetermined value, to not be recognized as operational instructions.
However, as mentioned above, it is observed that the size of gestures performed as operational instructions or the volume of voice messages uttered as operational instructions tends to be smaller when a person is concentrating on giving operational instructions to an information processing device. As a result, when the user is concentrating on the input of operational instructions to the information processing device, determinations may be made that gestures performed by the user are not intended as gestures by the user.
In consideration of the above, the information processing device 10 according to the first exemplary embodiment carries out the instruction recognition processing illustrated in
Then, in step 102, on the basis of the concentration level information acquired by the concentration level information acquisition section 12, the concentration level determination section 16 determines a concentration level F, which represents a degree to which the user is concentrating. If the sightline position is shifting as illustrated in
F=f(A−B) (1)
Here, function f may be a function that outputs the reciprocal of the input value as illustrated in the following expression (2), and may be an exponential function as illustrated in the following expression (3). Function f may be a function whose output value decreases as the input value increases, and may be a function that, using a table, alters the output value so as to decrease in discrete amounts in response to increases in the input value (A−B).
F=f(A−B)=1/(A−B) (2)
F=f(A−B)=a·exp(−(A−B)2) (3)
In step 104, in accordance with the concentration level F determined by the concentration level determination section 16, the threshold determination command section 18 specifies an determination in a threshold value TH for the action information that directly or indirectly represents actions by the user. The threshold value TH is an example of the first threshold of the disclosed technology. For example, as illustrated in the following expression (4), the threshold value TH may be the reciprocal of the concentration level F.
TH=1/F (4)
Thus, when the sightline position of the user shifts, for example, as illustrated in
In step 106, the action information acquisition section 14 acquires the video image obtained by imaging of the user by the imaging unit 46, to serve as action information directly representing an action by the user. By analyzing the acquired video image, the action information acquisition section 14 recognizes a gesture performed by the user moving their hand. For this gesture recognition, for example, the technology recited in JP-A No. 2011-76255 may be employed.
In step 108, the operational instruction determination section 20 compares a size of the action (gesture) by the user represented by the action information acquired by the action information acquisition section 14 with the threshold value TH for which determinations have been specified by the threshold determination command section 18. Then, in step 110, the operational instruction determination section 20 makes a determination as to whether an operational instruction has been performed by the user on the basis of the result of the comparison between the size of the action (gesture) by the user and the threshold value TH.
For example, in a period in which variations in the sightline position of the user are relatively large, the threshold value TH has a relatively large value. Therefore, as indicated by reference numeral 82 in
If the result of the determination in step 110 is negative, the instruction recognition processing ends. On the other hand, if the result of the determination in step 110 is affirmative, the processing advances to step 112. In step 112, the operational instruction determination section 20 executes processing in accordance with the operational instruction from the user. This processing in accordance with the operational instruction by the user may be, for example, processing to open an arbitrary file, link or the like, processing to close an arbitrary file, link or the like that has been opened, processing to move the position of a cursor, and so forth.
Thus, a user may give operational instructions to the information processing device 10 by moving their hand(s) and performing gestures. Moreover, even when the sizes of actions (gestures) by the user are small because the user is concentrating thereon, actions (gestures) that are intended as operational instructions by the user may be recognized as operational instructions.
Next, a second exemplary embodiment of the disclosed technology is described. Structures of the second exemplary embodiment are the same as in the first exemplary embodiment, so the same reference numerals are assigned to the respective sections and descriptions of the structures are not given. Herebelow, operation of the second exemplary embodiment is described with reference to
In the instruction recognition processing according to the second exemplary embodiment, after the concentration level determination section 16 determines the concentration level F in step 102, the processing advances to step 114. In step 114, the concentration level determination section 16 compares the concentration level F with a threshold value THF relating to a pre-specified concentration level, and makes a determination as to whether the concentration level F is at least the threshold value THF. When the concentration level F is equal to or greater than the threshold value THF, the result of the determination in step 114 is affirmative, and the processing from step 104 is carried out (specifying a determination in the threshold value TH with the threshold determination command section 18, acquiring action information with the action information acquisition section 14, and performing an operational instruction determination with the operational instruction determination section 20).
On the other hand, when the concentration level F is less than the threshold value THF, the result of the determination in step 114 is negative, and the instruction recognition processing ends. In this case, the processing from step 104 onward (specifying a determination in the threshold value TH with the threshold determination command section 18, acquiring action information with the action information acquisition section 14, and performing an operational instruction determination with the operational instruction determination section 20) is omitted. Thus, in a period prior to an action detection interval depicted in
Thus, in this second exemplary embodiment, similarly to the first exemplary embodiment, when a user is concentrating and the size of actions (gestures) by the user is small, actions (gestures) that are intended as operational instructions by the user may be recognized as operational instructions. Moreover, in the second exemplary embodiment, action information is not acquired and operational instruction determinations are not performed in a period in which the concentration level F is less than the threshold value THF. Therefore, even if the user moves their hand greatly in a state in which the user is not concentrating, misrecognition of these actions as operational instructions may be avoided. Furthermore, because the acquisition of action information is suspended when the user is not concentrating and is not expected to be giving operational instructions, a processing load on the computer may be moderated.
Next, a third exemplary embodiment of the disclosed technology is described. Structures of the third exemplary embodiment are the same as in the first exemplary embodiment, so the same reference numerals are assigned to the respective sections and descriptions of the structures are not given. Herebelow, operation of the third exemplary embodiment is described with reference to
In the instruction recognition processing according to the third exemplary embodiment, the same as in the second exemplary embodiment, the concentration level F is determined by the concentration level determination section 16 in step 102, and then a determination is made by the concentration level determination section 16 in step 114 as to whether the concentration level F is at least the threshold value THF. When the concentration level F is equal to or greater than the threshold value THF, the result of the determination in step 114 is affirmative and the processing advances to step 116.
In step 116, the concentration level determination section 16 makes a determination as to whether a sightline position represented by the sightline position information acquired by the concentration level information acquisition section 12 is disposed inside an active window of windows displayed at the display unit 40. The determination in step 116 may be implemented by acquiring a display range of the active window displayed at the display unit 40 and determining whether or not the sightline position falls within the display range of the active window.
For example, as illustrated in
On the other hand, as illustrated by the example in
Thus, in this third exemplary embodiment, whether a user is concentrating may be determined more accurately, by determining whether or not the user is concentrating by making a determination as to whether a sightline position of the user is disposed within an active window. Furthermore, in the third exemplary embodiment too, actions (gestures) that are intended as operational instructions by the user when the user is concentrating may be recognized as operational instructions. Moreover, in the third exemplary embodiment, even if the user unintentionally moves their hand greatly in a state in which the user is not concentrating, misrecognition of these actions as operational instructions may be avoided.
Next, a fourth exemplary embodiment of the disclosed technology is described. Parts that are the same as in the first to third exemplary embodiments are assigned the same reference numerals and descriptions thereof are not given.
An information processing device 22 according to the fourth exemplary embodiment is illustrated in
The distance estimation section 24 estimates a distance of the user on the basis of, for example, the size of an image region that corresponds to the hand of the user in an image obtained by imaging of the user by the imaging unit 46. Instead of the size of an image region corresponding to the hand of the user, the distance of the user may be estimated on the basis of the size of an image region corresponding to the face of the user or the like. As a further example, when the imaging unit 46 has a structure that is provided with a plural number of imaging components, instead of estimating the distance of the user on the basis of the size of an image region, the distance of the user may be estimated on the basis of a difference in positions of an image region in the plural images that are respectively imaged by the plural imaging components. Further yet, instead of the distance of the user being estimated on the basis of an image, the distance of the user may be estimated using, for example, infrared beams or the like.
In accordance with the distance of the user estimated by the distance estimation section 24, the normalization section 26 normalizes the size of actions by the user represented by the action information acquired by the action information acquisition section 14, or normalizes the threshold value TH determined by the threshold determination command section 18. For example, when the user moves their hand and performs a predetermined gesture, the size of a change in position of an image region corresponding to the hand of the user in the video image is smaller when the distance of the user is larger. Therefore, when the normalization section 26 normalizes the size of actions by the user, the normalization section 26 normalizes (corrects) the size of actions by the user such that the size of actions by the user is larger in accordance with larger distances from the user. When the normalization section 26 normalizes the threshold value TH, the normalization section 26 normalizes (corrects) the threshold value TH such that the threshold value TH is larger in accordance with larger distances from the user.
The instruction recognition program 50 according to the fourth exemplary embodiment further includes a distance estimation process 62 and a normalization process 64, as illustrated by broken lines in
Now, as operations of the fourth exemplary embodiment, portions of the instruction recognition processing according to the fourth exemplary embodiment that differ from the instruction recognition processing described in the first exemplary embodiment (
Two methods may be considered for estimating the distance of a user on the basis of the size of an image region corresponding to the hand of the user. In a first method, a plural number of templates that are, for example, hand images with different sizes from one another are registered in advance in association with distances from users corresponding to the sizes of the hands in the individual templates. Hence, a template that is closest to the size of the image region corresponding to the hand of the user is selected from the plural templates by template matching. Then, the distance of a user that is associated with the selected template is extracted to serve as a result of estimation of the distance of the user. Thus, the distance of the user may be estimated from the size of the image region corresponding to the hand of the user.
In the second method, the distance of the user is estimated by evaluating the area of the image region corresponding to the hand of the user in an image obtained by imaging of the user by the imaging unit 46. More specifically, the area (number of pixels) of an image region corresponding to the hand of a user is counted for each of plural cases in which values of the distance of users are different from one another, and the areas (numbers of pixels) of the image regions counted for the respective cases are registered in advance in association with the distances from users. Hence, the image region corresponding to the hand of the user is extracted by extracting a region in an image that is the color of the skin of the user, and the number of pixels in the extracted image region is counted. Then, the area (number of pixels) closest to the counted number of pixels of the image region is selected from the plural registered areas (numbers of pixels), and the distance of a user that is associated with the selected area (number of pixels) is extracted to serve as a result of estimation of the distance of the user. Thus, the distance of the user may be estimated from the size of the image region corresponding to the hand of the user.
In the second method, if a focusing distance of the optical system of the imaging unit 46 is represented by f, an area is represented by x cm2, and the number of pixels in an image region when the hand of a user is imaged is represented by y, then the distance of the user z may be found by calculating the following expression (5).
z=f×√(x/y) (5)
In step 120, the normalization section 26 normalizes (corrects) the threshold value TH determined by the threshold determination command section 18 in accordance with the distance of the user estimated in step 118. That is, the normalization section 26 normalizes (corrects) the threshold value TH that has been determined by the threshold determination command section 18 such that the threshold value TH is smaller when the distance of the user is larger and the threshold value TH is larger when the distance of the user is smaller. This normalization (correction) may be implemented by, for example, multiplying the threshold value TH with a coefficient whose value decreases as the distance of the user increases. When the processing of step 120 has been performed, the processing advances to step 108 and the processing from step 108 onward is performed.
For example,
On the other hand,
Thus, in the fourth exemplary embodiment, the threshold value TH is normalized (corrected) such that the threshold value TH is smaller when the distance of the user is larger and the threshold value TH is larger when the distance of the user is smaller. Therefore, actions intended as operational instructions by the user may be accurately determined to be operational instructions without being affected by changes with the distance of the user in the size of actions by the user represented by the action information.
In the fourth exemplary embodiment, a mode is described in which the threshold value TH is normalized (corrected) in accordance with the distance of the user, but this is not restrictive of the disclosed technology. The normalization section 26 may normalize (correct) the size of the action by the user represented by the action information acquired by the action information acquisition section 14 in accordance with the distance of the user. A criterion for the size of the action by the user represented by the action information may be applied such that the size of the action by the user represented by the action information is larger when the distance of the user is larger and the size of the action by the user represented by the action information is smaller when the distance of the user is smaller. This may be implemented by, for example, multiplying the size of the action by the user represented by the action information with a coefficient whose value increases as the distance of the user increases. In this case too, actions intended as operational instructions by the user may be accurately determined to be operational instructions without being affected by changes with the distance of the user in the size of actions by the user represented by the action information.
Further, a mode has been described hereabove in which determinations in the threshold value TH are specified in accordance with the concentration level F of the user, but this is not restrictive of the disclosed technology. The threshold value TH may be determined in consideration of where the sightline position of the user is disposed. For example, in a state in which an icon is displayed at the display unit 40, when the sightline position of the user stays within the displayed icon, the threshold value TH described hereabove may be used without modification. However, when the sightline position of the user leaves the displayed icon, the concentration level F determined by the concentration level determination section 16 may be multiplied by a coefficient 1/x(x>1), and the threshold value TH may be found on the basis of the concentration level F multiplied by the coefficient 1/x. In this case, it is determined that the concentration level of the user is higher when the user is looking at the icon and that the concentration level of the user is lower when the user is not looking at the icon, and gestures performed as operational instructions by the user are determined accordingly. Therefore, operational instructions may be more accurately determined, taking better account of degrees of concentration by the user.
In the above mode in which it is determined whether the user is looking at an icon, when the sightline position of the user stays within the icon, the threshold value TH may also be set in consideration of the size of this icon. Specifically, if the size of the icon in which the sightline position of the user stays is represented by m, the concentration level F is found by the following expression (6).
F=f((A−B)/m) (6)
The threshold value TH is set on the basis of the concentration level F found from expression (6). In this mode, the value of the concentration level F is higher when the size of the icon at which the user is looking is smaller, and the value of the threshold value TH is smaller when the value of the concentration level F is higher, so actions by the user are more likely to be recognized as operational instructions. Thus, in this mode too, operational instructions may be more accurately determined, taking better account of degrees of concentration by the user.
Hereabove, as an example of the actions of a user, a case of the user performing gestures as operational instructions has been principally described, but this is not restrictive of the disclosed technology. The actions of the user may be, for example, voice actions of uttering predetermined voice messages. In this case, the action information acquisition section 14 acquires voice information representing a voice message given by the user to serve as action information that indirectly represents an action by the user. The operational instruction determination section 20 compares a volume of the voice message according to the voice action by the user, which is indirectly represented by the action information acquired by the action information acquisition section 14, with the threshold value TH, and thus makes a determination as to whether the voice action by the user is intended as an operational instruction by the user.
Hereabove, the sightline position of a user has been described as an example of the physical quantity that is related to whether or not the user is concentrating, but this is not restrictive of the disclosed technology. This physical quantity may be, for example, a body movement of the user. As an example of a body movement, movements of the head of the user may be employed. For example, the technology recited in P. Viola and M. Jones, “Robust real-time face detection”, Int. J. of Computer Vision. Vol. 57, no. 2, pp. 137-154, 2004 may be employed. In this case, as an example of the concentration level information, the concentration level information acquisition section 12 acquires body movement information representing movements of the head of the user. The concentration level determination section 16 determines that the user is concentrating when the body movements represented by the body movement information acquired by the concentration level information acquisition section 12 are smaller than a pre-specified threshold.
Hereabove, a mode is described in which the instruction recognition program 50, which is an example of an information processing program relating to the disclosed technology, is pre-memorized (installed) in the memory section 36, but this is not restrictive. The information processing program relating to the disclosed technology may be provided in a form that is recorded on a recording medium such as a CD-ROM, a DVD-ROM or the like.
All references, patent applications and technical specifications cited in the present specification are incorporated by reference into the present specification to the same extent as if the individual references, patent applications and technical specifications were specifically and individually recited as being incorporated by reference.
The disclosed technology has an effect in one regard in that an action intended as an operational instruction by a user may be determined to be an operational instruction even when the user is concentrating thereon.
All examples and conditional language provided herein are intended for the pedagogical purposes of aiding the reader in understanding the invention and the concepts contributed by the inventor to further the art, and are not to be construed as limitations to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although one or more embodiments of the present invention have been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.
Number | Date | Country | Kind |
---|---|---|---|
2013-055019 | Mar 2013 | JP | national |