This application is based upon and claims the benefit of priority from the prior Japanese Patent Application No. 2011-141029, filed on Jun. 24, 2011, the entire contents of which are incorporated herein by reference.
Embodiments described herein relate generally to an information processing apparatus, a computer implemented method for processing information and a medium storing a computer program for processing information.
Conventionally, there is known an information processing apparatus (for example, cellular phones) that includes an input device such as a touch panel or a keyboard as well as an acceleration sensor detecting static acceleration. Such information processing apparatus controls movement of a cursor or display of a 3D (three-dimensional) image in accordance with the output of the acceleration sensor.
Moreover, there is known a user interface apparatus that obtains the facial image of a user, estimates the direction of the user's face based on the obtained facial image, and executes a predetermined input process in accordance with the estimated direction of the face.
However, the direction of a human's face has high flexibility in a horizontal direction, but has low flexibility in a vertical direction. For example, when the direction of the face is largely turned to the upper in the horizontal direction, a screen is out of the user's sight. That is, the technology of operating the information processing apparatus in accordance with only the direction of the face is degraded in operability.
Embodiments will now be explained with reference to the accompanying drawings.
In general, according to one embodiment an information processing apparatus includes a camera, a first feature region detector, a second feature region detector, a plurality of operation command generators and a selector. The camera obtains an image of at least a part of a human body and generates image data of the obtained image. The first feature region detector detects a first feature region including a feature part of the human body from the image data and generates first feature region information defining the first feature region. The second feature region detector generates second feature region information defining a second feature region corresponding to the first feature region in a virtual space based on the first feature region information. The operation command generators generate operation commands corresponding to a plurality of partial spaces in the virtual space. The selector selects one of the operation command generators based on the second feature region information.
In a first embodiment, the basic functions of an information processing apparatus 10 will be described.
For example, the controller 12 is a computer processor. The controller 12 executes control programs stored in the memory 14 to realize the functions of a body position calculator 122, a UI (user interface) controller 124, and an application controller 126. Further, the controller 12 executes an application program stored in the memory 14 to realize the application 17.
The functions realized by the controller 12 of the first embodiment will be described.
The body position calculator 122 includes a first feature region detector 122A and a second feature region detector 122B. The first feature region detector 122A detects a first feature region including a body feature part from image data of the image obtained by the camera 18. The first feature region is a region which is defined in a first space of the image data. The second feature region detector 122B detects a second feature region corresponding to the first feature region. The second feature region is a region which is defined in a second space different from the first space.
The UI controller 124 includes a selector 124A and a plurality of operation command generators 124B. In
The application controller 126 controls the application 17 based on the operation commands outputted from the selected operation command generators 124B. As a result, the application 17 executes a process in response to the movement of the feature part of the user.
Information processing of the first embodiment will be described.
<S300> The first feature region detector 122A controls the camera 18 so that the camera 18 obtains the image of the user. The camera 18 obtains the image of the user and generates image data including the obtained image of the user.
<S302> The first feature region detector 122A detects a first feature region R1 including a feature part of the human body of the user U from the image data IMG obtained in step S300 and generates first feature region information. The first feature region information is information which defines the first feature region R1.
For example, the first feature region detector 122A detects a feature part (for example, a head part of the user U) in the image data IMG by the use of a predetermined feature detection algorithm and generates the horizontal size, the vertical size and the coordinates of the center of the feature part as the first feature region information by the use of a template matching technique. More specifically, the first feature region detector 122A calculates a correlation value between a predetermined template image and an image expressed by the image data IMG while moving the template image and generates the horizontal size, the vertical size and the coordinates of the center of a region corresponding to a region with the highest correlation value as the first feature region information. The feature detection algorithm is one of an eigen space method, a partial space method, or the combination of a Haar like feature amount and an Adaboost algorithm.
<S304> The second feature region detector 122B generates second feature region information based on the first feature region information generated in step S302. The second feature region information is information which defines a second feature region R2 corresponding to the first feature region R1 in a virtual space (second space) different from the image space AB.
For example, the second feature region detector 122B generates the second feature region information using Equation 1. In Equation 1, Or1a and Or1b are coordinates in the horizontal direction (the A direction in
<S306> The selector 124A selects one of the plurality of operation command generators 124B(1) to 124B(3) based on the second feature region information.
<S500> The application controller 126 assigns the plurality of partial spaces in the virtual space XYZ to the plurality of operation command generators 124B, respectively, and notifies the selector 124A of the assignment result (a relation between the plurality of partial spaces and the plurality of operation command generators 124B).
<S502> The selector 124A calculates a duration time T using the photographed times of the plurality of image data IMG. The duration time T is a time in which the center Or2 of the second feature region continues to be located in one identical space of the partial spaces PS1 to PS3.
On the other hand,
<S504> The selector 124A compares the duration time T with a predetermined first time threshold value Th1. The first time threshold value Th1 is, for example, 500 msec. When the duration time T is greater than the first time threshold value Th1 (Yes in step S504), step S506 is executed. On the other hand, when the duration time T is equal to or less than the first time threshold value Th1 (No in step S504), the process returns to step S502. A loop from step S502 to step S504 is repeated until the duration time T exceeds the first time threshold value Th1.
<S506> The selector 124A generates a selection table based on the assignment result notified in step S500, and selects the operation command generator 124B corresponding to the partial space PS in which the center Or2 of the second feature region is located by the use of the generated selection table.
<S308> The operation command generator 124B selected in step S506 generates an operation command based on the movement of the center Or2 of the second feature region. The operation command generator 124B generates a predetermined operation command in accordance with a difference between the duration time T of the center Or2 of the second feature region and a predetermined second time threshold value Th2. The second time threshold value Th2 may be identical with or may be different from the first time threshold value Th1.
When the operation command generator 124B(1) is selected in step S506 and the duration time T is greater than the second time threshold value Th2, the operation command generator 124B(1) generates an operation command of “YES”. On the other hand, when the operation command generator 124B(1) is selected in step S506 and the duration time T is equal to or less than the second time threshold value Th2, the operation command generator 124B(1) generates an operation command of “undetermined”. For example, when the user U moves his or her head part so that the center Or2 of the second feature region is located in the partial space PS1 for a time greater than the second time threshold value Th2, the operation command of “YES” is generated.
When the operation command generator 124B(2) is selected in step S506 and the duration time T is greater than the second time threshold value Th2, the operation command generator 124B(2) generates an operation command of “NO”. On the other hand, when the operation command generator 124B(2) is selected in step S506 and the duration time T is equal to or less than the second time threshold value Th2, the operation command generator 124B(2) generates the operation command of “undetermined”. For example, when the user U moves his or her head part so that the center Or2 of the second feature region is located in the partial space PS2 for a time greater than the first time threshold value Th1 and the second time threshold value Th2, the operation command of “NO” is generated. For example, when the user U nods forward, the operation command of “YES” is generated. When the user U shakes his or her head part right and left, the operation command of “NO” is generated. When the user U continues to move his or her head part, the operation command of “undetermined” is generated.
When the operation command generator 124B(3) is selected in step S506, the operation command generator 124B(3) generates the operation command of “undetermined” irrespective of the duration time T. For example, when the user U moves his or her head part so that the center Or2 of the second feature region is located in the partial space PS3, the operation command of “undetermined” is generated.
<S310> The application controller 126 outputs the operation command generated in step S308 to the application 17. Thus, the application 17 is controlled in accordance with the movement of the feature part of the user U.
According to the first embodiment, the center of the feature part of a human body is mapped to a virtual space and an operation command is generated based on the movement of the center and the duration time in which the center is located in one of the plurality of partial spaces of the virtual space. Therefore, the operability of the information processing apparatus 10 can be improved in response to a movement of the human body. Specifically, the user U can operate the application 17 without touching the operating module 16 just by moving a feature part photographed by the camera 18 in an actual space.
In the first embodiment, the coordinates of the center of the second feature region in the virtual space XYZ is calculated by using the statistical values Uw and Uh of the feature part of the user U. Therefore, the same calculation equation (Equation 1) can be applied to the plurality of users U.
In the first embodiment, the feature part is mapped from the image space AB to the virtual space XYZ. Therefore, the user U can enter continuous operation commands even when the user U stops.
In the first embodiment, the rule of generating the operation commands can be assigned to each partial space PS.
Incidentally, in the first embodiment, the example of the two operation command generators 124B(1) and 124B(2) has hitherto been described. However, the number of the operation command generators 124B is not limited to two. Further, the example of the virtual space XYZ including the plurality of planar (two-dimensional) partial spaces PS has hitherto been described to facilitate the description. However, the virtual space XYZ may include a plurality of stereoscopic (three-dimensional) partial spaces PS.
In the first embodiment, in step S304, the first feature region detector 122A may detect the first feature region with a shape which is different from the rectangular shape instead of detecting the first feature region with the rectangular shape. The first feature region detector 122A may detect the first feature region with, for example, an elliptical shape. In this case, the first feature region detector 122A generates the minor axis length, the major axis length, a rotation angle of the minor axis and the coordinates of the center of the elliptical shape circumscribed with a feature part of the user U as the first feature region information.
In the first embodiment, in step S500, the application controller 126 may use a virtual space XYZ with an arbitrary three-dimensional shape such as a spherical shape, a conical shape, or an elliptical shape instead of the triangular virtual space XYZ.
In a second embodiment, the generation of the operation command will be described when a three-dimensional computer graphic is drawn as an example of an application function of the information processing apparatus 10. Further, the same description as that of the first embodiment will not be repeated.
When the operation command generator 124B(1) is selected in step S506 and the duration time T is greater than the second time threshold value Th2, the operation command generator 124B(1) generates an operation command of “viewpoint change (Pv)” having a viewpoint parameter Pv. On the other hand, when the operation command generator 124B(1) is selected in step S506 and the duration time T is equal to or less than the second time threshold value Th2, the operation command generator 124B(1) generates an operation command of “undetermined”. For example, when the user U moves his or her head part so that the center Or2 of the second feature region is located in the partial space PS1 for a time greater than the second time threshold value Th2, a viewpoint of the three-dimensional computer graphic displayed on the display 19 is changed in accordance with the viewpoint parameter Pv.
The operation command generator 124B(1) generates the viewpoint parameter Pv based on the position of the center Or2 of the second feature region. The viewpoint parameter Pv includes three viewpoint angles θ, φ, and r defining the viewpoint of the three-dimensional computer graphic. For example, the operation command generator 124B(1) generates the viewpoint parameter Pv using Equation 2. In Equation 2, Or2x, Or2y, and Or2z are the X coordinate, the Y coordinate, and the Z coordinate of the center Or2 of the second feature region, respectively. Further, α, β, and γ are predetermined coefficients and δ is a predetermined integer. For example, when the head of the user U gets close to the camera 18, the three-dimensional computer graphic is expanded and displayed. When the head of the user U gets distant from the camera 18, the three-dimensional computer graphic is reduced in size and displayed.
On the other hand, when the operation command generator 124B(2) is selected in step S506 and the duration time T is greater than the second time threshold value Th2, the operation command generator 124B(2) generates an operation command of “movement (Pm)” having a movement parameter Pm. On the other hand, when the operation command generator 124B(2) is selected in step S506 and the duration time T is equal to or less than the second time threshold value Th2, the operation command generator 124B(2) generates an operation command of “undetermined”. For example, when the user U moves his or her head part so that the center Or2 of the second feature region is located in the partial space PS1 for a time greater than the first time threshold value Th1 and the second time threshold value Th2, at least a portion of the three-dimensional computer graphic displayed on the display 19 is changed in accordance with the movement parameter Pm.
The operation command generator 124B(1) generates the movement parameter Pm based on the position of the center Or2 of the second feature region. The movement parameter Pm includes a movement vector (Pmx, Pmy, Pmz) of at least a portion of the three-dimensional computer graphic. For example, the operation command generator 124B(1) generates the movement parameter Pm using Equation 3. In Equation 3, Or2x, Or2y, and Or2z are the X coordinate, the Y coordinate, and the Z coordinate of the center Or2 of the second feature region, respectively. Further, CGx, CGy, and CGz are the X coordinate, the Y coordinate, and the Z coordinate of at least the portion of the three-dimensional computer graphic. For example, when the user U moves his or her head part, the eyes of an avatar expressed by the three-dimensional computer graphic move to follow the head part of the user. In this way, it can be configured that the avatar of the three-dimensional computer graphic keeps looking at the user.
According to the second embodiment, the three viewpoint angles θ, φ, and r or the movement vector (Pmx, Pmy, Pmz) of at least the portion of the three-dimensional computer graphic are calculated by applying predetermined coefficients and integers to the coordinates of the center Or2 of the second feature region. Therefore, the user U can operate the three-dimensional computer graphic more smoothly by moving a feature part.
Further, in the second embodiment, the operation command generator 124B(2) may generate an operation command of “viewpoint fixation” irrespective of the duration time T, instead of generating the movement vector (Pmx, Pmy, Pmz) of at least the portion of the three-dimensional computer graphic. In this case, when the user U moves his or her head part so that the center Or2 of the second feature region is located in the partial space PS2 for a time greater than the first time threshold Th1, the viewpoint of the three-dimensional computer graphic displayed on the display 19 is fixed.
In a third embodiment, the generation of operation commands for browsing digital books will be described as an example of an application function of the information processing apparatus 10. The same descriptions as those of the first and second embodiments will not be repeated.
When the operation command generator 124B(1) is selected in step S506 and the duration time T is greater than the second time threshold value Th2, the operation command generator 124B(1) generates an operation command of “viewport movement (Pvp1)” having a first viewport parameter Pvp1. On the other hand, when the operation command generator 124B(1) is selected in step S506 and the duration time T is equal to or less than the second time threshold value Th2, the operation command generator 124B(1) generates an operation command of “undetermined”. For example, when the user U moves his or her head part so that the center Or2 of the second feature region is located in the partial space PS1 for a time greater than the first time threshold value Th1 and the second time threshold value Th2, a viewport of a digital book displayed on the display 19 is moved in accordance with the first viewport parameter Pvp1. The viewport is a point which defines a planar image to be displayed on a screen when the user virtually browses a part of a large planar image.
The operation command generator 124B(1) generates the first viewport parameter Pvp1 based on the position of the center Or2 of the second feature region. The first viewport parameter Pvp1 includes a change amount (ΔVx, ΔVy) of the coordinates of the center of the viewport and a change amount (ΔVw) of a range of the viewport. For example, the operation command generator 124B(1) generates the first viewport parameter Pvp1 using Equation 4. In Equation 4, Or2x, Or2y, and Or2z are the X coordinate, the Y coordinate, and the Z coordinate of the center Or2 of the second feature region, respectively. Further, α and β are predetermined coefficients and γ is a predetermined integer. In Equation 4, the change amount (ΔVx, ΔVy) of the coordinates of the center of the viewport is calculated by a square of the coordinates (Or2x, Or2y, Or2z) of the center Or2 of the second feature region. For example, when the user U moves his or her head part, the viewport is moved at high speed in a location distant from the center, whereas the viewport is moved at low speed in a location close to the center.
On the other hand, when the operation command generator 124B(2) is selected in step S506 and the duration time T is greater than the second time threshold value Th2, the operation command generator 124B(2) generates an operation command of “viewport movement (Pvp2)”. On the other hand, when the operation command generator 124B(2) is selected in step S506 and the duration time T is equal to or less than the second time threshold value Th2, the operation command generator 124B(2) generates an operation command of “undetermined”. For example, when the user U moves his or her head part so that the center Or2 of the second feature region is located in the partial space PS2 for a time greater than the first time threshold value Th1 and the second time threshold value Th2, the viewport of the digital book displayed on the display 19 moves in accordance with the second viewport parameter Pvp2.
The operation command generator 124B(2) generates the second viewport parameter Pvp2 based on the position of the center Or2 of the second feature region. The second viewport parameter Pvp2 includes a change amount (ΔVx, ΔVy) of the coordinates of the center of the viewport and a change amount (ΔVw) in a range of the viewport. For example, the operation command generator 124B(2) generates the second viewport parameter Pvp2 using Equation 5. In Equation 5, Or2x, Or2y, and Or2z are the X coordinate, the Y coordinate, and the Z coordinate of the center Or2 of the second feature region, respectively. Further, α and β are predetermined coefficients and γ is a predetermined integer. In Equation 5, the change amount (ΔVx, ΔVy) of the coordinates of the center of the viewport is calculated by the coordinates (Or2x, Or2y, Or2z) of the center Or2 of the second feature region. For example, when the user U moves his or her head part, the viewport moves at a constant speed in proportion to a distance from the center.
According to the third embodiment, the viewport of the digital book can be moved at a speed in accordance with the distance from the center or a constant speed irrespective of the distance from the center by applying the predetermined coefficients and integers to the coordinates and the range of the center Or2 of the second feature region. In particular, when the center Or2 of the second feature region is located in the partial region PS1, the viewport can be moved at high speed at a location distant from the center, whereas the viewport can be moved at low speed at a location close to the center. When the center Or2 of the second feature region is located in the partial region PS2, the viewport can be moved stably.
In a fourth embodiment, the generation of operation commands for moving a mouse pointer will be described as an example of an application function of the information processing apparatus 10. The same descriptions as those of the first to third embodiments will not be repeated.
When the operation command generator 124B(1) is selected in step S506 and the duration time T is greater than the second time threshold value Th2, the operation command generator 124B(1) generates an operation command of “pointer movement (Pp1)” having a pointer parameter Pp. On the other hand, when the operation command generator 124B(1) is selected in step S506 and the duration time T is equal to or less than the second time threshold value Th2, the operation command generator 124B(1) generates an operation command of “undetermined”. For example, when the user U moves his or her head part so that the center Or2 of the second feature region is located in the partial space PS1 for a time greater than the first time threshold value Th1 and the second time threshold value Th2, a mouse pointer displayed on the display 19 is moved in accordance with the pointer parameter Pp.
The operation command generator 124B(1) generates the pointer parameter Pp based on the position of the center Or2 of the second feature region. The pointer parameter Pp includes a change amount (Δx, Δy) of the X and Y directions. For example, the operation command generator 124B(1) generates the pointer parameter Pp using Equation 6. In Equation 6, Or2x, Or2y, and Or2z are the X coordinate, the Y coordinate, and the Z coordinate of the center Or2 of the second feature region, respectively. Further, α is a predetermined coefficient and βth is a predetermined threshold value. In Equation 6, the change amount (Δx, Δy) of the coordinates of the mouse pointer is determined by a difference between the coordinates (Or2x, Or2y) of the center Or2 of the second feature region and the threshold value βth. For example, when the user U moves his or her head part, the mouse pointer is accelerated as the mouse pointer becomes distant from the center, whereas the mouse pointer is decelerated as the mouse pointer becomes close to the center. When the distance from the center is within a predetermined range, the mouse pointer stops.
On the other hand, when the operation command generator 124B(2) is selected in step S506 and the duration time T is greater than the second time threshold value Th2, the operation command generator 124B(2) generates an operation command of “stop”. On the other hand, when the operation command generator 124B(2) is selected in step S506 and the duration time T is equal to or less than the second time threshold value Th2, the operation command generator 124B(2) generates an operation command of “undetermined”. For example, when the user U moves his or her head part so that the center Or2 of the second feature region is located in the partial space PS2 for a time greater than the first time threshold value Th1 and the second time threshold value Th2, the mouse pointer is stopped.
According to the fourth embodiment, the mouse pointer can be accelerated, decelerated, or stopped in accordance with the distance from the center by calculating the pointer parameter Pp based on the relation between the coordinates (Or2x, Or2y) of the center Or2 of the second feature region and the threshold value βth. In particular, the usability can be improved by the threshold value βth when the distance between the head part of the user U and the center is within a predetermined range.
In a fifth embodiment, the generation of operation commands for reproducing a moving image will be described as an example of an application function of the information processing apparatus 10. The same descriptions as those of the first to fourth embodiments will not be repeated.
When the operation command generator 124B(1) is selected in step S506 and the duration time T is greater than the second time threshold value Th2, the operation command generator 124B(1) generates an operation command of “one-time speed”. On the other hand, when the operation command generator 124B(1) is selected in step S506 and the duration time T is equal to or less than the second time threshold value Th2, the operation command generator 124B(1) generates an operation command of “undetermined”. For example, when the user U moves his or her head part so that the center Or2 of the second feature region is located in the partial space PS1 for a time greater than the first time threshold value Th1 and the second time threshold value Th2, the moving image is reproduced at one-time speed.
On the other hand, when the operation command generator 124B(2) is selected in step S506 and the duration time T is greater than the second time threshold value Th2, the operation command generator 124B(2) generates an operation command of “reproduction speed change (Ps)” having a speed parameter Ps. On the other hand, when the operation command generator 124B(2) is selected in step S506 and the duration time T is equal to or less than the second time threshold value Th2, the operation command generator 124B(2) generates an operation command of “undetermined”. For example, when the user U moves his or her head part so that the center Or2 of the second feature region is located in the partial space PS2 for a time greater than the first time threshold value Th1 and the second time threshold value Th2, the reproduction speed of the moving image is changed in accordance with the speed parameter Ps.
The operation command generator 124B(2) generates the speed parameter Ps based on the position of the center Or2 of the second feature region. For example, the operation command generator 124B(2) generates the speed parameter Ps using Equation 7. In Equation 7, Or2x and Or2z are the X coordinate and the Z coordinate of the center Or2 of the second feature region, respectively. Further, α, β, γ, and δ are predetermined integers. In Equation 7, the speed parameter Ps is determined in accordance with the coordinates (Or2x, Or2z) of the center Or2 of the second feature region. For example, when the user U moves his or her head part right and left at a position distant from the camera 18, the reproduction speed is changed in accordance with the position of the head part. When the user U moves his or her head part at a position close to the camera 18, the reproduction speed becomes one-time speed.
According to the fifth embodiment, the reproduction speed can be controlled in accordance with the distance from the center by calculating the speed parameter Ps based on the coordinates (Or2x, Or2z) of the center Or2 of the second feature region.
The fifth embodiment can be applied not only to the reproduction speed of a moving image but also to the control of the reproduction speed of music.
At least a portion of the information processing apparatus 10 according to the above-described embodiments may be composed of hardware or software. When at least a portion of the information processing apparatus 10 is composed of software, a program for executing at least some functions of the information processing apparatus 10 may be stored in a recording medium, such as a flexible disk or a CD-ROM, and a computer may read and execute the program. The recording medium is not limited to a removable recording medium, such as a magnetic disk or an optical disk, but it may be a fixed recording medium, such as a hard disk or a memory.
In addition, the program for executing at least some functions of the information processing apparatus 10 according to the above-described embodiment may be distributed through a communication line (which includes wireless communication) such as the Internet. In addition, the program may be encoded, modulated, or compressed and then distributed by wired communication or wireless communication such as the Internet. Alternatively, the program may be stored in a recording medium, and the recording medium having the program stored therein may be distributed.
While certain embodiments have been described, these embodiments have been presented by way of example only, and are not intended to limit the scope of the inventions. Indeed, the novel methods and systems described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions and changes in the form of the methods and systems described herein may be made without departing from the spirit of the inventions. The accompanying claims and their equivalents are intended to cover such forms or modifications as would fall within the scope and spirit of the inventions.
Number | Date | Country | Kind |
---|---|---|---|
2011-141029 | Jun 2011 | JP | national |