Patent Grant
10928896
The present disclosure relates to an information processing apparatus, an information processing method, and a program.
In recent years, user interfaces allowing a user to operate through the line of sight by using line-of-sight detection technology such as an eye tracking technology are emerging. For example, the technology described in PTL 1 below can be cited as a technology concerning the user interface allowing the user to operate through the line of sight.
Patent Literature 1: JP 2009-64395A
For example, when an operation is performed through a user's line of sight, an apparatus that conducts a process based on a user's line of sight needs to conduct the process after determining which user's line of sight to use as the basis for conducting the process, or in other words, determining the operator who performs an operation through line of sight (hereinafter simply called the “operator” in some cases). However, a method of determining the operator in an apparatus like the above has not been established.
The present disclosure proposes a new and improved information processing and information processing method capable of determining an operator who performs an operation through line of sight, and conducting a process based on the line of sight of the determined operator.
According to the present disclosure, there is provided an information processing apparatus including: an image acquisition unit configured to acquire a captured image of users; a determination unit configured to determine an operator from among the users included in the acquired captured image; and a processing unit configured to conduct a process based on information about user line of sight corresponding to the determined operator.
According to the present disclosure, there is provided an information processing method executed by an information processing apparatus, the information processing method including: a step of acquiring a captured image of users; a step of determining an operator from among the users included in the acquired captured image; and a step of conducting a process based on information about user line of sight corresponding to the determined operator.
According to the present disclosure, it is possible to determine an operator who performs an operation through line of sight, and conduct a process based on the line of sight of the determined operator.
The above effect is not necessarily restrictive and together with the above effect or instead of the above effect, one of the effects shown in this specification or another effect grasped from this specification may be achieved.
A preferred embodiment of the present disclosure will be described in detail below with reference to the appended drawings. Note that in this specification and the drawings, the same reference signs are attached to elements having substantially the same function and configuration, thereby omitting duplicate descriptions.
The description will be provided in the order shown below:
1. Information Processing Method According to the Present Embodiment
2. Information Processing Apparatus According to the Present Embodiment
3. Program According to the Present Embodiment
Before describing the configuration of an information processing apparatus according to the present embodiment, an information processing method according to the present embodiment will first be described. The information processing method according to the present embodiment will be described by taking a case in which processing according to the information processing method according to the present embodiment is performed by an information processing apparatus according to the present embodiment as an example.
As discussed above, a method of deciding an operator who performs an operation through line of sight in an apparatus that conducts a process based on a user's line of sight has not been established. Herein, an operator according to the present embodiment refers to the user who performs an action that affects the behavior of the apparatus (or an application being executed), for example. Note that, as discussed later, as an example of a process according to an information processing method according to the present embodiment, an information processing apparatus according to the present embodiment is also capable of conducting a process that does not affect the behavior of the apparatus (or an application being executed) on the basis of the line of sight of a user not determined to be the operator.
Accordingly, an information processing apparatus according to the present embodiment acquires a captured image of users (image acquisition process), and determines the operator from among the users included in the acquired captured image (determination process). Subsequently, an information processing apparatus according to the present embodiment conducts a process on the basis of information about the line of sight of the user corresponding to the determined operator (execution process).
Herein, a captured image of users according to the present embodiment refers to a captured image that may include users, for example. Herein, a captured image of users according to the present embodiment simply will be called the “captured image”. The captured image according to the present embodiment may be a user-captured image viewable on a display screen, for example. In addition, the captured image according to the present embodiment may be generated by image capture in an imaging unit (discussed later) provided in an information processing apparatus according to the present embodiment, or an external imaging device. The captured image according to the present embodiment may be a moving image or a still image, for example.
The display screen according to the present embodiment is, for example, a display screen on which various images are displayed and toward which the user directs the line of sight. As the display screen according to the present embodiment, for example, the display screen of a display unit (described later) included in the information processing apparatus according to the present embodiment and the display screen of an external display apparatus (or an external display device) connected to the information processing apparatus according to the present embodiment wirelessly or via a cable can be cited.
In addition, information about user line of sight according to the present embodiment refers to information (data) about a user's eyes, such as the position of a user's line of sight on the display screen and a user's eye movements, for example. The information about user line of sight according to the present embodiment may be information about the position of the line of sight of a user and information about eye movements of a user, for example.
Here, the information about the position of the line of sight of the user according to the present embodiment is, for example, data showing the position of the line of sight of the user or data that can be used to identify the position of the line of sight of the user (or data that can be used to estimate the position of the line of sight of the user. This also applies below).
As the data showing the position of the line of sight of the user according to the present embodiment, for example, coordinate data showing the position of the line of sight of the user on the display screen can be cited. The position of the line of sight of the user on the display screen is represented by, for example, coordinates in a coordinate system in which a reference position of the display screen is set as its origin. The data showing the position of the line of sight of the user according to the present embodiment may include data indicating the direction of the line of sight (for example, data showing the angle with the display screen).
When coordinate data indicating the position of the line of sight of the user on the display screen is used as information about the position of the line of sight of the user according to the present embodiment, the information processing apparatus according to the present embodiment identifies the position of the line of sight of the user on the display screen by using, for example, coordinate data acquired from an external apparatus having identified (estimated) the position of the line of sight of the user by using line-of-sight detection technology and indicating the position of the line of sight of the user on the display screen. When the data indicating the direction of the line of sight is used as information about the position of the line of sight of the user according to the present embodiment, the information processing apparatus according to the present embodiment identifies the direction of the line of sight by using, for example, data indicating the direction of the line of sight acquired from the external apparatus.
It is possible to identify the position of the line of sight of the user and the direction of the line of sight of the user on the display screen by using the line of sight detected by using line-of-sight detection technology and the position of the user and the orientation of face with respect to the display screen detected from a captured image in which the direction in which images are displayed on the display screen is captured. However, the method of identifying the position of the line of sight of the user and the direction of the line of sight of the user on the display screen according to the present embodiment is not limited to the above method. For example, the information processing apparatus according to the present embodiment and the external apparatus can use any technology capable of identifying the position of the line of sight of the user and the direction of the line of sight of the user on the display screen.
As the line-of-sight detection technology according to the present embodiment, for example, a method of detecting the line of sight based on the position of a moving point (for example, a point corresponding to a moving portion in an eye such as the iris and the pupil) of an eye with respect to a reference point (for example, a point corresponding to a portion that does not move in the eye such as an eye's inner corner or corneal reflex) of the eye can be cited. However, the line-of-sight detection technology according to the present embodiment is not limited to the above technology and may be, for example, any line-of-sight detection technology capable of detecting the line of sight.
As the data that can be used to identify the position of the line of sight of the user according to the present embodiment, for example, captured image data in which the direction in which images (moving images or still images) are displayed on the display screen is imaged can be cited. The data that can be used to identify the position of the line of sight of the user according to the present embodiment may further include detection data of any sensor obtaining detection values that can be used to improve estimation accuracy of the position of the line of sight of the user such as detection data of an infrared sensor that detects infrared radiation in the direction in which images are displayed on the display screen.
When data that can be used to identify the position of the line of sight of the user is used as information about the position of the line of sight of the user according to the present embodiment, the information processing apparatus according to the present embodiment uses, for example, captured image data acquired by an imaging unit (described later) included in the local apparatus (hereinafter, referred to as the information processing apparatus according to the present embodiment) or an external imaging device. In the above case, the information processing apparatus according to the present embodiment may use, for example, detection data (example of data that can be used to identify the position of the line of sight of the user) acquired from a sensor that can be used to improve estimation accuracy of the position of the line of sight of the user included in the local apparatus or an external sensor. The information processing apparatus according to the present embodiment performs processing according to an identification method of the position of the line of sight of the user and the direction of the line of sight of the user on the display screen according to the present embodiment using, for example, data that can be used to identify the position of the line of sight of the user acquired as described above to identify the position of the line of sight of the user and the direction of the line of sight of the user on the display screen.
The information related to a user's eye movements according to the present embodiment may be, for example, data indicating the user's eye movements, or data that may be used to specify the user's eye movements (or data that may be used to estimate the user's eye movements. This applies similarly hereinafter.)
The data indicating a user's eye movements according to the present embodiment may be, for example, data indicating a predetermined eye movement, such as a single blink movement, multiple consecutive blink movements, or a wink movement (for example, data indicating a number or the like corresponding to a predetermined movement). In addition, the data that may be used to specify a user's eye movements according to the present embodiment may be, for example, captured image data depicting the direction in which an image (moving image or still image) is displayed on the display screen.
When data indicating a user's eye movements is used as the information related to a user's eye movements according to the present embodiment, the information processing apparatus according to the present embodiment determines that a predetermined eye movement has been performed by using data indicating the user's eye movements acquired from an external apparatus that specifies (or estimates) the user's eye movements on the basis of a captured image, for example.
Herein, for example, when a change in eye shape detected from a moving image (or a plurality of still images) depicting the direction in which an image is displayed on the display screen qualifies as a change in eye shape corresponding to a predetermined eye movement, it is possible to determine that the predetermined eye movement was performed. Note that the method of determining a predetermined eye movement according to the present embodiment is not limited to the above. For example, the information processing apparatus according to the present embodiment or an external apparatus is capable of using arbitrary technology enabling a determination that a predetermined eye movement was performed.
When data that may be used to specify a user's eye movements is used as the information related to a user's eye movements according to the present embodiment, the information processing apparatus according to the present embodiment uses captured image data (an example of data that may be used to specify a user's eye movements) acquired from an imaging unit (discussed later) provided in the local apparatus or an external imaging device, for example. The information processing apparatus according to the present embodiment uses the data that may be used to specify a user's eye movements acquired as above to conduct a process related to a method of determining a predetermined eye movement according to the present embodiment, and determine that the predetermined eye movement was performed, for example.
Hereinafter, processes according to an information processing method according to the present embodiment will be described more specifically.
(1) Image Acquisition Process
The information processing apparatus according to the present embodiment acquires the captured image according to the present embodiment. The information processing apparatus according to the present embodiment acquires the captured image according to the present embodiment by controlling image capture in an imaging unit (discussed later) provided in the information processing apparatus according to the present embodiment or an external imaging device, for example. The information processing apparatus according to the present embodiment controls image capture in the imaging unit (discussed later) or the like by transmitting control commands related image capture to the imaging unit (discussed later), the external imaging device, or the like via a communication unit (discussed later) or a connected external communication device.
Note that the image acquisition process according to the present embodiment is not limited to the above. For example, the information processing apparatus according to the present embodiment may also passively acquire the captured image according to the present embodiment transmitted from the imaging unit (discussed later) or the external imaging device.
(2) Determination Process
The information processing apparatus according to the present embodiment determines the operator from among the users included in a captured image acquired by the process of (1) above (image acquisition process). The information processing apparatus according to the present embodiment determines a single user or multiple users from among the users included in the captured image as the operator(s).
(2-1) First Example of Determination Process
The information processing apparatus according to the present embodiment determines the operator on the basis of the size of a face region detected from the captured image, for example.
Herein, the face region according to the present embodiment refers to a region including the face portion of a user in the captured image. The information processing apparatus according to the present embodiment detects the face region by detecting features such as the user's eyes, nose, mouth, and bone structure from the captured image, or by detecting a region resembling a luminance distribution and structure pattern of a face from the captured image, for example. Note that the method of detecting the face region according to the present embodiment is not limited to the above, and the information processing apparatus according to the present embodiment may also use arbitrary technology enabling the detection of a face from the captured image.
(2-1-1) Process in the Case of Determining Single User from Among Users Included in Captured Image as the Operator
The information processing apparatus according to the present embodiment determines the operator to be a single user corresponding to the face region having the largest face region size from among face regions detected from the captured image, for example.
At this point, regions having the same (or approximately the same) face region size may be included among the face regions detected from the captured image.
Accordingly, when there exist multiple face regions having the largest face region size among the face regions detected from the captured image, the information processing apparatus according to the present embodiment determines the operator to be the user corresponding to the face region detected earlier, for example.
By determining as the operator the user corresponding to the face region detected earlier as above, for example, the information processing apparatus according to the present embodiment is able to determine a single user as the operator, even when face regions are the same (or approximately the same) size.
Note that the method of determining the operator when there exist multiple face regions having the largest face region size among the face regions detected from the captured image is not limited to the above.
For example, the information processing apparatus according to the present embodiment may also determine the operator to be the user corresponding to the face region detected later, or determine from among the face regions detected from the captured image a single user as the operator by following a configured rule (such as randomly, for example).
In addition, the information processing apparatus according to the present embodiment may also determine a single user as the operator by combining one or multiple processes from among the determination process according to the second example discussed later to the determination process according to the fifth example discussed later, for example. By determining the operator according to a process combining the determination process according to the first example with a determination process according to another example, it becomes possible to prevent the operator from changing frequently, for example.
The information processing apparatus according to the present embodiment determines a single user as the operator on the basis of the size of a face region detected from the captured image as above, for example.
Note that the process in the case of determining a single user as the operator in the determination process according to the first example according to the present embodiment is not limited to the above.
For example, the information processing apparatus according to the present embodiment may also determine that a user who had been determined to be the operator on the basis of the size of a face region detected from the captured image is not the operator.
For example, the information processing apparatus according to the present embodiment computes a first difference value indicating the difference in the size of a face region corresponding to the user determined to be the operator from the size of a face region corresponding to a user not determined to be the operator (hereinafter called an “other user”) from among the users included in the captured image. Subsequently, when the first difference value is equal to or greater than a configured first threshold value (or when the first difference value is greater than the first threshold value), the information processing apparatus according to the present embodiment determines that the user who had been determined to be the operator is not the operator.
The first threshold value may be a fixed value configured in advance, or a variable value that may be set appropriately by user operations or the like, for example. The degree to which the user determined to be the operator continues to be the operator changes according to the magnitude of the configured value of the first threshold value. Specifically, in the case in which the value of the first threshold value is 0 (zero), the user who had been determined to be the operator is determined not to be the operator when the size of the face region corresponding to the user determined to be the operator becomes smaller than the size of a face region corresponding to another user (or when the size of the face region corresponding to the user determined to be the operator becomes less than or equal to the size of a face region corresponding to another user). Also, as the value of the first threshold value becomes larger, the value of the first difference value needed for the user who had been determined to be the operator to be determined not to be the operator becomes larger, and thus the user who had been determined to be the operator is less likely to be determined not to be the operator.
(2-1-2) Process in the Case of Determining Multiple Users from Among Users Included in Captured Image as the Operator
The information processing apparatus according to the present embodiment determines the operator to be users up to a configured number of people in order of largest face region size detected from the captured image, for example. More specifically, the information processing apparatus according to the present embodiment determines, as the operator, users up to a configured number of people or a number of users less than the configured number of people, in order of largest face region size detected from the captured image, for example.
Herein, the configured number of people in the determination process according to the first example may be fixed, or varied by user operations or the like.
In addition, when the configured number of people is exceeded as a result of operator candidates being selected in order of largest face region size detected from the captured image due to face regions being the same size, the information processing apparatus according to the present embodiment does not determine the operator to be a user corresponding to a face region detected later from among the face regions having the same face region size, for example. Note that the method of determining the operator in the above case obviously is not limited to the above.
The information processing apparatus according to the present embodiment determines multiple users as the operator on the basis of the sizes of face regions detected from the captured image as above, for example.
(2-2) Second Example of Determination Process
The information processing apparatus according to the present embodiment determines the operator on the basis of a distance, from the display screen, of a user corresponding to a face region detected from the captured image (hereinafter called the “distance corresponding to a face region”), for example. The information processing apparatus according to the present embodiment determines a single user or multiple users from among the users included in the captured image as the operator(s).
Herein, the “distance, from the display screen, of a user corresponding to a face region included in the captured image” according to the present embodiment is specified (or estimated) on the basis of a depth map captured by a method such as time of flight (TOF), for example. In addition, the information processing apparatus according to the present embodiment is also capable of specifying (or estimating) the “distance, from the display screen, of a user corresponding to a face region included in the captured image” according to the present embodiment on the basis of a face region detected from the captured image and a detection value from a depth sensor using infrared or the like, for example. In addition, the information processing apparatus according to the present embodiment may also specify the “distance, from the display screen, of a user corresponding to a face region included in the captured image” by specifying (or estimating) the coordinates of a face region using arbitrary technology, and computing the distance to the coordinates of a reference position, for example. Herein, the coordinates of the face region and the coordinates of the reference position are expressed as coordinates in a three-dimensional coordinate system made up of two axes representing a plane corresponding to the display screen and one axis representing the vertical direction with respect to the display screen, for example. Note that the method of specifying (or the method of estimating) the “distance, from the display screen, of a user corresponding to a face region included in the captured image” according to the present embodiment obviously is not limited to the above.
(2-2-1) Process in the Case of Determining Single User from Among Users Included in Captured Image as the Operator
The information processing apparatus according to the present embodiment determines the operator to be a single user corresponding to the face region having the shortest distance corresponding to a face region, for example.
At this point, distances corresponding to a face region having the same (or approximately the same) distance may be included among the distances corresponding to a face region according to the present embodiment which correspond to face regions detected from the captured image.
Accordingly, when there exist multiple distances corresponding to a face region having the same (or approximately the same) distance among the distances corresponding to a face region according to the present embodiment, the information processing apparatus according to the present embodiment determines the operator to be the user corresponding to the face region detected earlier, for example.
By determining as the operator the user corresponding to the face region detected earlier as above, for example, the information processing apparatus according to the present embodiment is able to determine a single user as the operator, even when there exist multiple distances corresponding to a face region having the same (or approximately the same) distance.
Note that the method of determining the operator when there exist multiple distances corresponding to a face region having the same (or approximately the same) distance among the distances corresponding to a face region according to the present embodiment which correspond to face regions detected from the captured image is not limited to the above.
For example, the information processing apparatus according to the present embodiment may also determine the operator to be the user corresponding to the face region detected later, or determine from among the face regions detected from the captured image a single user as the operator by following a configured rule (such as randomly, for example).
In addition, the information processing apparatus according to the present embodiment may also determine a single user as the operator by combining one or multiple processes from among the determination process according to the first example above and the determination process according to the third example discussed later to the determination process according to the fifth example discussed later, for example. By determining the operator according to a process combining the determination process according to the third example with a determination process according to another example, it becomes possible to prevent the operator from changing frequently, for example.
The information processing apparatus according to the present embodiment determines a single user as the operator on the basis of the distance corresponding to a face region according to the present embodiment, which corresponds to a face region detected from the captured image as above, for example.
Note that the process in the case of determining a single user as the operator in the determination process according to the second example according to the present embodiment is not limited to the above. For example, the information processing apparatus according to the present embodiment may also determine that a user who had been determined to be the operator on the basis of the length of the distance corresponding to a face region according to the present embodiment which corresponds to a face region captured from the captured image is not the operator.
For example, the information processing apparatus according to the present embodiment computes a second difference value indicating the difference in the distance corresponding to a face region corresponding to another user from the distance corresponding to a face region corresponding to the user determined to be the operator. Subsequently, when the second difference value is equal to or greater than a configured second threshold value (or when the second difference value is greater than the second threshold value), the information processing apparatus according to the present embodiment determines that the user who had determined to be the operator is not the operator.
The second threshold value may be a static value configured in advance, or a variable value that may be set appropriately by user operations or the like, for example. The degree to which the user determined to be the operator continues to be the operator changes according to the magnitude of the configured value of the second threshold value. Specifically, in the case in which the value of the second threshold is 0 (zero), the user who had been determined to be the operator is determined not to be the operator when the distance corresponding to the face region corresponding to the user determined to be the operator becomes shorter than the distance corresponding to a face region corresponding to another user (or when the distance corresponding to the face region corresponding to the user determined to be the operator becomes less than or equal to the distance corresponding to a face region corresponding to another user). Also, as the value of the second threshold value becomes larger, the value of the second difference value needed for the user who had been determined to be the operator to be determined not to be the operator becomes larger, and thus the user who had been determined to be the operator is less likely to be determined not to be the operator.
(2-2-2) Process in the Case of Determining Multiple Users from Among Users Included in Captured Image as the Operator
The information processing apparatus according to the present embodiment determines the operator to be users up to a configured number of people in order of shortest distance corresponding to a face region, for example. More specifically, the information processing apparatus according to the present embodiment determines, as the operator, users up to a configured number of people or a number of users less than the configured number of people, in order of shortest distance corresponding to a face region, for example.
Herein, the configured number of people in the determination process according to the second example may be fixed, or varied by user operations or the like.
In addition, when the configured number of people is exceeded as a result of operator candidates being selected in order of shortest distance corresponding to a face region due to distances corresponding to a face region according to the present embodiment which correspond to face regions detected from the captured image being the same, the information processing apparatus according to the present embodiment does not determine the operator to be a user corresponding to a face region detected later from among the face regions having the same face distance corresponding to a face region, for example. Note that the method of determining the operator in the above case obviously is not limited to the above.
The information processing apparatus according to the present embodiment determines multiple users as the operator on the basis of the distance corresponding to a face region according to the present embodiment which corresponds to a face region detected from the captured image as above, for example.
(2-3) Third Example of Determination Process
The information processing apparatus according to the present embodiment determines the operator on the basis of a predetermined gesture detected from the captured image, for example.
Herein, the predetermined gesture according to the present embodiment may be various gestures, such as a gesture of raising a hand, or a gesture of waving a hand, for example.
For example, in the case of detecting a gesture of raising a hand, the information processing apparatus according to the present embodiment respectively detects the face region and the hand from the captured image. Subsequently, if the detected hand exists within a region corresponding to the face region (a region configured to determine that a hand was raised), the information processing apparatus according to the present embodiment detects the gesture of raising a hand by determining that the user corresponding to the relevant face region raised a hand.
As another example, in the case of detecting a gesture of waving a hand, the information processing apparatus according to the present embodiment respectively detects the face region and the hand from the captured image. Subsequently, if the detected hand is detected within a region corresponding to the face region (a region configured to determine that a hand was waved), and the frequency of luminance change in the captured image is equal to or greater than a configured predetermined frequency (or the frequency of the luminance change is greater than the predetermined frequency), the information processing apparatus according to the present embodiment detects the gesture of waving a hand by determining that the user corresponding to the relevant face region waved a hand.
Note that the predetermined gesture according to the present embodiment and the method of detecting the predetermined gesture according to the present embodiment are not limited to the above. The information processing apparatus according to the present embodiment may also detect an arbitrary gesture, such as a gesture of pointing a finger, by using an arbitrary method enabling detection from the captured image, for example.
(2-3-1) Process in the Case of Determining Single User from Among Users Included in Captured Image as the Operator
The information processing apparatus according to the present embodiment determines, as the operator, a user for which a predetermined gesture was detected earlier from the captured image, for example.
Note that in the determination process according to the third example, the process of determining a single user as the operator is not limited to the above.
For example, when there exist multiple users for which a predetermined gesture was detected from the captured image, the information processing apparatus according to the present embodiment may determine the operator to be a user for which a predetermined gesture was detected later, for example. Also, in the above case, the information processing apparatus according to the present embodiment may also determine from among the users for which a predetermined gesture was detected from the captured image a single user as the operator by following a configured rule (such as randomly, for example). Furthermore, in the above case, the information processing apparatus according to the present embodiment may also determine a single user as the operator by combining the determination process according to the first example above, the determination process according to the second example above, the determination process according to the fourth example discussed later, and the determination process according to the fifth example discussed later, for example.
(2-3-2) Process in the Case of Determining Multiple Users from Among Users Included in Captured Image as the Operator
The information processing apparatus according to the present embodiment determines the operator to be users up to a configured number of people in order of a predetermined gesture being detected from the captured image, for example. More specifically, the information processing apparatus according to the present embodiment determines the operator to be users up to a configured number of people or a number of users less than the configured number of people, in order of a predetermined gesture being detected from the captured image, for example.
Herein, the configured number of people in the determination process according to the third example may be fixed, or varied by user operations or the like.
Note that in the determination process according to the third example, the process of determining multiple users as the operator is not limited to the above.
For example, the information processing apparatus according to the present embodiment may also determine the operator to be users up to the configured number of people selected from among the users for which a predetermined gesture was detected from the captured image by following a configured rule (such as randomly, for example).
(2-4) Fourth Example of Determination Process
When the position of a user's line of sight on the display screen is included in a configured region on the display screen, the information processing apparatus according to the present embodiment determines the operator to be the user corresponding to the relevant line of sight, for example.
Herein, the information processing apparatus according to the present embodiment uses the position of a line of sight of a user on the display screen, which is indicated by the information about the line of sight of a user according to the present embodiment. Also, the configured region on the display screen according to the present embodiment may be a fixed region configured in advance on the display screen, a region in which a predetermined object, such as an icon or character image, is displayed on the display screen, or a region configured by a user operation or the like on the display screen, for example.
(2-4-1) Process in the Case of Determining Single User from Among Users Included in Captured Image as the Operator
The information processing apparatus according to the present embodiment determines the operator to be a user for which the position of the user's line of sight was detected earlier within a configured region on the display screen, for example.
Note that in the determination process according to the fourth example, the process of determining a single user as the operator is not limited to the above.
For example, when there exist multiple users for which a line of sight was detected within the configured region on the display screen, the information processing apparatus according to the present embodiment may determine the operator to be a user for which the position of the user's line of sight was detected later within the configured region on the display screen, for example. Also, in the above case, the information processing apparatus according to the present embodiment may also determine, from among the users for which the position of the user's line of sight is included within the configured region on the display screen, a single user as the operator by following a configured rule (such as randomly, for example). Furthermore, in the above case, the information processing apparatus according to the present embodiment may also determine a single user as the operator by combining the determination process according to the first example above, the determination process according to the second example above, the determination process according to the third example above, and the determination process according to the fifth example discussed later, for example.
(2-4-2) Process in the Case of Determining Multiple Users from Among Users Included in Captured Image as the Operator
The information processing apparatus according to the present embodiment determines the operator to be users up to a configured number of people in order of the position of the user's line of sight being detected within a configured region on the display screen, for example. More specifically, the information processing apparatus according to the present embodiment determines the operator to be users up to a configured number of people or a number of users less than the configured number of people, in order of the position of the user's line of sight being detected within the configured region on the display screen, for example.
Herein, the configured number of people in the determination process according to the fourth example may be fixed, or varied by user operations or the like.
Note that in the determination process according to the fourth example, the process of determining multiple users as the operator is not limited to the above.
For example, the information processing apparatus according to the present embodiment may also determine the operator to be users up to the configured number of people selected from among the users for which the position of the user's line of sight is included within the configured region on the display screen by following a configured rule (such as randomly, for example).
(2-5) Fifth Example of Determination Process
The information processing apparatus according to the present embodiment identifies a user included a captured image on the basis of the captured image, for example. Subsequently, the information processing apparatus according to the present embodiment determines the operator on the basis of a ranking associated with the identified user.
Herein, a ranking according to the present embodiment refers to a value indicating an index by which the information processing apparatus according to the present embodiment preferentially determines the operator, for example. In the ranking according to the present embodiment, the priority of ranking may be high to the extent that the value is small, or the priority of ranking may be high to the extent that the value is large, for example.
More specifically, the information processing apparatus according to the present embodiment detects a face region from the captured image, and conducts a face recognition process on the detected face region to extract face information (data) indicating features of a user's face, for example. Subsequently, the information processing apparatus according to the present embodiment uses a table (or database) associating a user ID uniquely indicating a user with face information and the extracted face information to identify the user by specifying the user ID corresponding to the face information, for example.
Herein, the user ID uniquely indicating a user according to the present embodiment additionally may be associated with the execution state of an application and/or information related to the calibration of a user interface (UI), for example. The information related to UI calibration may be data indicating positions where objects such as icons are arranged on the display screen, for example. By additionally associating the user ID uniquely indicating a user according to the present embodiment with the execution state of an application or the like, the information processing apparatus according to the present embodiment is able to manage identifiable users in greater detail, and also provide to a identifiable user various services corresponding to that user.
Note that the method of identifying a user based on the captured image according to the present embodiment is not limited to the above.
For example, the information processing apparatus according to the present embodiment may also use a table (or a database) associating a user ID indicating a user type, such as an ID or the like indicating whether the user is an adult or a child, with face information and extracted face information to specify a user ID corresponding to a face region, and thereby specify a user type.
After users are identified, the information processing apparatus according to the present embodiment uses a table (or database) associating a user ID with a ranking value and the specified user ID to specify the ranking corresponding to the identified users. Subsequently, the information processing apparatus according to the present embodiment determines the operator on the basis of the specified ranking.
By determining the operator on the basis of a ranking as above, for example, the information processing apparatus according to the present embodiment is able to realize the following. Obviously, however, an example of a determination process according to the fifth example is not limited to the examples given below.
The information processing apparatus according to the present embodiment determines the operator to be the user with the highest ranking associated with a user identified on the basis of the captured image, for example.
Note that in the determination process according to the fifth example, the process of determining a single user as the operator is not limited to the above.
For example, when there exist multiple users having the highest ranking, the information processing apparatus according to the present embodiment may determine the operator to be the user whose face region was detected earlier from the captured image or the user identified earlier on the basis of the captured image from among the users having the highest ranking, for example. Also, in the above case, the information processing apparatus according to the present embodiment may also determine from among the users having the highest ranking a single user as the operator by following a configured rule (such as randomly, for example).
(2-5-2) Process in the Case of Determining Multiple Users from Among Users Included in Captured Image as the Operator
The information processing apparatus according to the present embodiment determines the operator to be users up to a configured number of people in order of highest ranking, for example. More specifically, the information processing apparatus according to the present embodiment determines the operator to be users up to a configured number of people or a number of users less than the configured number of people, in order of highest ranking, for example.
Herein, the configured number of people in the determination process according to the fifth example may be fixed, or varied by user operations or the like.
In addition, when the configured number of people is exceeded as a result of operator candidates being selected in order of highest ranking due to rankings being the same, the information processing apparatus according to the present embodiment does not determine the operator to be a user whose face region was detected later from the captured image or a user identified later on the basis of the captured image from among the users having the same ranking, for example. Note that the method of determining the operator in the above case obviously is not limited to the above.
(2-6) Sixth Example of Determination Process
The information processing apparatus according to the present embodiment may also determine the operator when speech indicating configured predetermined spoken content is additionally detected for a user determined to be the operator in each of the determination process according to the first example indicated in (2-1) above to the determination process according to the fifth example indicated in (2-5) above.
The information processing apparatus according to the present embodiment detects the speech indicating predetermined spoken content by performing speech recognition using source separation or source localization, for example. Herein, source separation according to the present embodiment refers to technology that extracts only speech of interest from among various sounds. Also, source localization according to the present embodiment refers to technology that measures the position (angle) of a sound source.
As a determination process according to the present embodiment, the information processing apparatus according to the present embodiment determines the operator from among users included in the captured image on the basis of the captured image by conducting one of the processes from the determination process according to the first example indicated in (2-1) above to the determination process according to the sixth example indicated in (2-6) above, for example.
Note that the process of determining the operator from among users included in the captured image on the basis of the captured image in a determination process according to the present embodiment is not limited to being from the determination process according to the first example indicated in (2-1) above to the determination process according to the sixth example indicated in (2-6) above. For example, the information processing apparatus according to the present embodiment may also determine the operator according to a detection order of faces detected from the captured image. Examples of determining the operator according to a face detection order include taking the operator to be the user whose face was detected first, or taking the operator to be users equal to a configured number of people in order of face detection, for example.
Also, the determination process according to the present embodiment is not limited to being a process of determining the operator from among users included in the captured image. The information processing apparatus according to the present embodiment is also capable of conducting one or more of the processes from the determination process according to the seventh example indicated below to the determination process according to the tenth example indicated below as the determination process according to the present embodiment, for example.
(2-7) Seventh Example of Determination Process
For example, the information processing apparatus according to the present embodiment configures an operation level for a user determined to be the operator.
Herein, the operation level according to the present embodiment refers to a value indicating an index related to a range of operations that may be performed using line of sight by the determined operator, for example. The operation level according to the present embodiment is associated with a range of operations that may be performed using line of sight by a table (or a database) associating the operation level according to the present embodiment with information about operations that may be performed using line of sight, for example. The information about operations according to the present embodiment may be, for example, various data for realizing operations, such as data indicating operated-related commands or parameters, or data for executing an operation-related application (such as an address where an application is stored, and parameters, for example).
When an operation level is configured in the determination process according to the present embodiment, the information processing apparatus according to the present embodiment conducts a process corresponding to the operator's line of sight on the basis of the configured operation level in an execution process according to the present embodiment discussed later. In other words, when an operation level is configured in the determination process according to the present embodiment, the information processing apparatus according to the present embodiment is able to dynamically change the processes that the determined operator may perform on the basis of the configured operation level, for example.
The information processing apparatus according to the present embodiment configures the configured operation level for a user determined to be the operator, for example.
In addition, when the user determined to be the operator is identified on the basis of the captured image, the information processing apparatus according to the present embodiment may configure an operation level corresponding to the identified user, for example. The information processing apparatus according to the present embodiment configures an operation level corresponding to the identified user by using a table (or database) associating a user ID with an operation level and the user ID corresponding to the user identified on the basis of the captured image, for example.
(2-8) Eighth Example of Determination Process
As a determination process according to the present embodiment, the information processing apparatus according to the present embodiment may also determine that a user who had been determined to be the operator is not the operator.
The information processing apparatus according to the present embodiment determines that a user who had been determined to be the operator is not the operator when information about the line of sight of a user corresponding to the user determined to be the operator cannot be acquired from the captured image, for example. As above, in the case of determining that a user is not the operator on the basis of information about the line of sight of a user corresponding to the user determined to be the operator, the information processing apparatus according to the present embodiment determines that the user who had been determined to be the operator is not the operator when the user determined to be the operator stops directing his or her line of sight towards the display screen, for example.
Note that the process of determining that a user who had been determined to be the operator is not the operator according to the present embodiment is not limited to the above.
For example, the cause for being unable to acquire information about the line of sight of a user corresponding to the user determined to be the operator from the captured image may be, for example, that the user determined to be the operator is no longer included in the captured image, or that the user determined to be the operator is included in the captured image, but is not looking at the display screen. Additionally, when information about the line of sight of a user cannot be acquired from the captured image because the user determined to be the operator is included in the captured image, but is not looking at the display screen, and the information processing apparatus according to the present embodiment determines that the user who had been determined to be the operator is not the operator, there is a risk of loss of convenience for the user who had been determined to be the operator.
Accordingly, even when information about the line of sight of a user corresponding to the user determined to be the operator cannot be acquired from the captured image, if the head of the user determined to be the operator is detected, the information processing apparatus according to the present embodiment does not determine that the user who had been determined to be the operator is not the operator.
The information processing apparatus according to the present embodiment detects a user's head from the captured image by detecting a shape corresponding to a head (such as a circular shape or an elliptical shape, for example) from the captured image, or by detecting luminance changes or the like in the captured image. The information processing apparatus according to the present embodiment detects the head of the user determined to be the operator by conducting a process related to the detection of a user's head on a partial region of the captured image that includes a region in which a face region corresponding to the user determined to the operator was detected, for example. Note that the process related to the detection of a user's head and the method of detecting the head of the user determined to be the operator according to the present embodiment are not limited to the above. For example, the information processing apparatus according to the present embodiment may also detect the head of the user determined to be the operator by using an arbitrary method and process enabling the detection of the head of the user determined to be the operator, such as a method that uses detection results from various sensors such as an infrared sensor.
As above, when the head of the user determined to be the operator is detected, by not determining that the user who had been determined to be the operator is not the operator, even if the user who had been determined to be the operator hypothetically looks away from the display screen, that user still remains the operator. Thus, as above, when the head of the user determined to be the operator is detected, by not determining that the user who had been determined to be the operator is not the operator, a reduction in the convenience of the user who had been determined to be the operator may be prevented.
(2-9) Ninth Example of Determination Process
The determination process according to the eighth example above illustrates a process in which the information processing apparatus according to the present embodiment determines that the user who had been determined to be the operator is not the operator, on the basis of information about the line of sight of a user. When the determination process according to the eighth example above is used, it is possible to change the operator determined by the information processing apparatus according to the present embodiment, such as by having the user who is the operator hide his or her face so that information about the line of sight of a user is not acquired, or by having the user who is the operator move to a position where his or her head is not detected from the captured image, for example.
However, the method of changing the operator according to the present embodiment is not limited to a method using the determination process according to the eighth example above. For example, the information processing apparatus according to the present embodiment may also actively change the operator from a user who had been determined to be the operator to another user, on the basis of a predetermined combination of gestures by the user determined to be the operator and the other user.
More specifically, the information processing apparatus according to the present embodiment detects a predetermined combination of gestures by the user determined to be the operator and the other user, for example. The information processing apparatus according to the present embodiment detects the predetermined combination of gestures by using a method related to arbitrary gesture recognition technology, such as a method that uses image processing on the captured image, or a method utilizing detection values from an arbitrary sensor such as a depth sensor, for example.
Subsequently, when the predetermined combination of gestures by the user determined to be the operator and the other user is detected from the captured image, the information processing apparatus according to the present embodiment changes the operator from the user who had been determined to be the operator to the other user.
When the predetermined combination of gestures by the user determined to be the operator and the other user, such as the high-five gesture illustrated in
By having the information processing apparatus according to the present embodiment conduct the determination process according to the ninth example as above, the users are able to change the operator intentionally by performing gestures, even when the number of users determined to be the operator has reached a configured upper limit.
(2-10) Tenth Example of Determination Process
When the number of users determined to be the operator has not reached a configured upper limit (or alternatively, when an upper limit on the number of users determined to be the operator is not configured; this applies similarly hereinafter), the information processing apparatus according to the present embodiment determines the operator to be a user newly included in the captured image as a result of conducting the process in the case of determining the operator to be multiple users in a process from the determination process according to the first example indicated in (2-1) above to the determination process according to the sixth example indicated in (2-6) above, for example.
Note that the process related to the determination of the operator in the case in which the number of users determined to be the operator has not reached a configured upper limit is not limited to the above.
For example, depending on the application executed in the execution process according to the present embodiment discussed later, immediately determining the operator to be a user newly included in the captured image is not desirable in some cases. Accordingly, the information processing apparatus according to the present embodiment may also conduct a process as given below, for example, when a user is newly included in the captured image while the number of users determined to be the operator has not reached a configured upper limit, for example. In addition, the information processing apparatus according to the present embodiment may also conduct a process selected by a user operation or the like from among the above process of determining the operator to be a user newly included in the captured image or the processes given below, for example.
The information processing apparatus according to the present embodiment conducts a process on the basis of the information about the line of sight of the user corresponding to the operator determined in the process of (2) above (determination process).
Herein, the process based on the information about the line of sight of a user according to the present embodiment may be various processes using the information about the line of sight of a user according to the present embodiment, such as a process of selecting an object existing at the position of the line of sight indicated by the information about the position of the line of sight of a user (an example of information about the line of sight of a user), a process of moving an object depending on the position of the line of sight indicated by the information about the position of the line of sight of a user, a process associated with an eye movement indicated by information about the eye movements of a user (an example of information about the line of sight of a user), and a process of controlling, on the basis of an eye movement indicated by information about the eye movements of a user, the execution state of an application or the like corresponding to the position of the line of sight indicated by the information about the position of the line of sight of a user, for example. In addition, the above object according to the present embodiment may be various objects displayed on the display screen, such as an icon, a cursor, a message box, and a text string or image for notifying the user, for example.
In addition, the information processing apparatus according to the present embodiment may also conduct, as the process based on the information about the line of sight of a user according to the present embodiment, a control process that causes an external apparatus communicable via a communication unit (discussed later) or a connected external communication device to conduct a process based on the information about the line of sight of the user corresponding to the operator, for example. In the case of conducting the above control process, the information processing apparatus according to the present embodiment causes the communication unit (discussed later) or the like to transmit to the external apparatus data including the information about the line of sight of the user corresponding to the operator determined in the process of (2) above (determination process) and a command causing a process to be conducted, for example.
In the case of conducting the above control process as the execution process according to the present embodiment, the information processing apparatus according to the present embodiment is able to utilize data which is acquired from the external apparatus and which indicates a result of the process based on the information about the line of sight of the user corresponding to the operator conducted in the external apparatus, for example.
Note that the execution process according to the present embodiment is not limited to the above. For example, the information processing apparatus according to the present embodiment may also conduct the execution process according to the first example indicated below or the execution process according to the second example indicated below as the execution process according to the present embodiment.
(3-1) First Example of Execution Process
When an operation level is configured for the user determined to be the operator in the process of (2) above (determination process), the information processing apparatus according to the present embodiment conducts a process based on information about the line of sight of a user within a range of processes associated with the relevant operation level, on the basis of the operation level configured for the user determined to be the operator, for example. In other words, in the case of conducting the execution process according to the first example, the information processing apparatus according to the present embodiment is able to dynamically change the available processes on the basis of the operation level configured for the user determined to be the operator, for example.
The information processing apparatus according to the present embodiment specifies information about an operation corresponding to a configured operation level in the process of (2) above (determination process) with a table (or a database) associating the operation level according to the present embodiment with information about operations that may be performed using line of sight, for example. By specifying information about operations corresponding to an operation level, it becomes possible to recognize a range of processes associated with an operation level. Subsequently, the information processing apparatus according to the present embodiment conducts a process on the basis of the information about the line of sight of a user, within the range of processes associated with the operation level. Obviously, however, an example of an execution process according to the first example is not limited to the example given above.
(3-2) Second Example of Execution Process
The information processing apparatus according to the present embodiment utilizes the line of sight of the other user not determined to be the operator in the process of (2) above (determination process), and conducts a process on the basis of information about the line of sight of the user corresponding to the other user, for example.
Herein, the process based on information about the line of sight of the user corresponding to the other user according to the present embodiment may be a process that excludes processing that would affect the behavior of the apparatus, for example. This is because the other user according to the present embodiment is not the user determined to be the operator.
To give a specific example, the process based on information about the line of sight of the user corresponding to the other user according to the present embodiment may be a process that causes an object indicating the position of the line of sight to be displayed at the position of the line of sight indicated by the information about the line of sight of the user corresponding to the other user, for example. As above, by causing an object indicating the position of the line of sight to be displayed at the position of the line of sight indicated by the information about the line of sight of the user corresponding to the other user, it becomes possible to visually express the endpoint of the other user's line of sight on the display screen.
Note that the execution process according to the second example is not limited to the above.
For example, when the process based on information about the line of sight of the user corresponding to the other user according to the present embodiment causes an object indicating the position of the line of sight to be displayed at the position of the line of sight indicated by the information about the line of sight of the user corresponding to the other user, if the endpoint of the other user's line of sight is always displayed on the display screen, there is a risk that the operator, the other user, or the like may feel irritated.
Accordingly, the information processing apparatus according to the present embodiment may also selectively conduct a process based on information about user line of sight corresponding to the other user. To give an example of the case in which the information processing apparatus according to the present embodiment causes an object indicating the position of the line of sight to be displayed at the position of the line of sight indicated by information about user line of sight corresponding to the other user, the information processing apparatus according to the present embodiment selectively causes an object indicating the position of the line of sight to be displayed at the position of the line of sight indicated by information about user line of sight corresponding to the other user, for example.
More specifically, the information processing apparatus according to the present embodiment conducts a process on the basis of the information about user line of sight when predetermined speech spoken by the other user is recognized as a result of conducting speech recognition on an acquired speech signal, for example. In addition, the information processing apparatus according to the present embodiment does not conduct a process based on the information about user line of sight when predetermined speech spoken by the other user is not recognized.
Herein, a speech signal according to the present embodiment is generated by a speech input device provided in the information processing apparatus according to the present embodiment or a speech input device external to the information processing apparatus according to the present embodiment. The information processing apparatus according to the present embodiment processes a speech signal acquired from the speech input device. Herein, the speech input device according to the present embodiment may be a speech input device capable of generating a speech signal corresponding to the speech of a specific user, such as a speech input device capable of conducting source separation or a speech input device capable of conducting source localization.
Additionally, the predetermined speech according to the present embodiment may be speech indicating a demonstrative pronoun such as “this” and “that” or the like. Obviously, however, the predetermined speech according to the present embodiment is not limited to being speech indicating a demonstrative pronoun.
As illustrated in
Additionally, as illustrated in
By having the endpoint of the line of sight of the user U2 who is the other user be displayed like the object O in
Note that, although not illustrated in
The information processing apparatus according to the present embodiment conducts the process of (1) above (image acquisition process), the process of (2) above (determination process), and the process of (3) above (execution process) as a process in accordance with an information processing method according to the present embodiment, for example. Herein, in the process of (2) above (determination process), the information processing apparatus according to the present embodiment determines the operator who performs operations by line of sight from among users included in the captured image acquired by the process of (1) above (image acquisition process). Subsequently, in the process of (3) above (execution process), the information processing apparatus according to the present embodiment conducts a process on the basis of the line of sight of the operator determined in the process of (2) above (determination process).
Consequently, the information processing apparatus according to the present embodiment conducts the process of (1) above (image acquisition process), the process of (2) above (determination process), and the process of (3) above (execution process) as a process in accordance with an information processing method according to the present embodiment, and thereby may determine the operator who performs operations by line of site, and conduct a process on the basis of the line of sight of the determined operator.
Next, an applied example of a process in accordance with the information processing method according to the present embodiment discussed above will be described.
In the case of determining a game participant, the information processing apparatus according to the present embodiment determines a participant (operator) by conducting one of the processes from the determination process according to the first example indicated in (2-1) above to the determination process according to the sixth example indicated in (2-6) above, for example.
In addition, when joining partway through or dropping out of the game, the information processing apparatus according to the present embodiment conducts a process as indicated in (a) and (b) below, for example.
(a) Case in which Information Processing Apparatus According to the Present Embodiment is Able to Conduct Face Recognition
(a-1) Case in which Information Processing Apparatus According to the Present Embodiment is Unable to Use Detection Result of User's Head
When there is a participating user whose face is no longer recognized partway through the game (such as when the relevant user is facing to the side or behind, for example), the information processing apparatus according to the present embodiment determines the participants in the game according to a mode configured from among the modes indicated below.
When there exists a participating user whose face is no longer detected partway through the game, and the head of the relevant user is detected (the case in which the relevant user is present within the angle of view of the imaging device or the like), the information processing apparatus according to the present embodiment maintains that user's status as a participant. In addition, when there exists a participating user whose face is no longer detected partway through the game, and the head of the relevant user is not detected, the information processing apparatus according to the present embodiment drops the relevant participating user from the game. For example in the example of
(b) Case in which Information Processing Apparatus According to the Present Embodiment is Unable to Conduct Face Recognition
(b-1) Case in which Information Processing Apparatus According to the Present Embodiment is Unable to Use Detection Result of User's Head
During the game, the information processing apparatus according to the present embodiment determines the participants to be n people (where n is a set integer equal to or greater than 1) in order of one's face being detected, for example.
When there is a participating user whose face is no longer detected (such as when the relevant user is facing to the side or behind, for example), the information processing apparatus according to the present embodiment determines that the next candidate user is a participant in the game, for example. Also, when the face is detected again, if the number of users whose face is currently detected (number of participants) does not satisfy n people, the information processing apparatus according to the present embodiment determines that the re-detected user is a participant. Also, when the face is detected again, if the number of users whose face is currently detected (number of participants) satisfies n people, the information processing apparatus according to the present embodiment does not determine that the re-detected user is a participant.
Note that information processing apparatus according to the present embodiment may also determine a participant (operator) by conducting the determination process according to the first example indicated in (2-1) above or the determination process according to the second example indicated in (2-2) above, for example.
(b-2) Case in which Information Processing Apparatus According to the Present Embodiment is Able to Use Detection Result of User's Head
The information processing apparatus according to the present embodiment conducts a process similar to (a-1) above, for example.
When a process in accordance with an information processing method according to the present embodiment is applied to the operation of a television set, the users U1 to U3 are able to use line of sight to perform operations as indicated below, for example.
An applied example of a process in accordance with an information processing method according to the present embodiment may be the first example indicated in (I) above to the third example indicated in (Ill) above, for example. Obviously, however, an applied example of a process in accordance with an information processing method according to the present embodiment is not limited to the first example indicated in (I) above to the third example indicated in (Ill) above.
Next, an example of the configuration of an information processing apparatus according to the present embodiment capable of performing the processing according to the information processing method according to the present embodiment described above will be described.
The information processing apparatus 100 may also include, for example, a ROM (Read Only Memory, not shown), a RAM (Random Access Memory, not shown), a storage unit (not shown), an operation unit (not shown) that can be operated by the user, and a display unit (not shown) that displays various screens on the display screen. The information processing apparatus 100 connects each of the above elements by, for example, a bus as a transmission path.
The ROM (not shown) stores programs used by the control unit 104 and control data such as operation parameters. The RAM (not shown) temporarily stores programs executed by the control unit 104 and the like.
The storage unit (not shown) is a storage means provided in the information processing apparatus 100, and stores various data, such as data in accordance with an information processing method according to the present embodiment such as a table (or a database) associating a user ID uniquely indicating a user with face information, and applications or the like. Herein, the storage unit (not shown) may be, for example, a magnetic recording medium such as a hard disk, or non-volatile memory such as flash memory. Additionally, the storage unit (not shown) may also be removable from the information processing apparatus 100.
As the operation unit (not shown), an operation input device described later can be cited. As the display unit (not shown), a display device described later can be cited.
(Hardware Configuration Example of the Information Processing Apparatus 100)
The MPU 150 is constituted of a processor configured by various operation circuits, such as a micro processing unit (MPU), or various processing circuits and functions as the control unit 104 that controls the whole information processing apparatus 100. The MPU 150 also plays the role of, for example, the image acquisition unit 110, the determination unit 112, and the processing unit 114 described later in the information processing apparatus 100.
The ROM 152 stores programs used by the MPU 150 and control data such as operation parameters. The RAM 154 temporarily stores programs executed by the MPU 150 and the like.
The recording medium 156 functions as a storage unit (not shown), and stores various data, such as data in accordance with an information processing method according to the present embodiment such as a table (or a database) associating a user ID uniquely indicating a user with face information, and applications or the like. As the recording medium 156, for example, a magnetic recording medium such as a hard disk and a nonvolatile memory such as a flash memory can be cited. The recording medium 156 may be removable from the information processing apparatus 100.
The input/output interface 158 connects, for example, the operation input device 160, the display device 162, and an imaging device 164. The operation input device 160 functions as an operation unit (not shown) and the display device 162 functions as a display unit (not shown). The imaging device 164 functions as a imaging unit (not shown). As the input/output interface 158, for example, a USB (Universal Serial Bus) terminal, a DVI (Digital Visual Interface) terminal, an HDMI (High-Definition Multimedia Interface) (registered trademark) terminal, and various processing circuits can be cited.
The operation input device 160 is, for example, included in the information processing apparatus 100 and connected to the input/output interface 158 inside the information processing apparatus 100. As the operation input device 160, for example, a button, a direction key, a rotary selector such as a jog dial, and a combination of these devices can be cited.
The display device 162 is, for example, included in the information processing apparatus 100 and connected to the input/output interface 158 inside the information processing apparatus 100. As the display device 162, for example, a liquid crystal display and an organic electro-luminescence display (also called an OLED display (Organic Light Emitting Diode Display)) can be cited.
The imaging device 164 is provided on the information processing apparatus 100 and is connected to the input/output interface 158 inside the information processing apparatus 100, for example. When the imaging device is provided, in the information processing apparatus 100, it becomes possible to process a captured image generated by imaging in the imaging device, determine the operator, obtain information about user line of sight on the basis of the captured image, and the like. Also, when the image device is provided, the information processing apparatus 100 is also capable of causing the display screen to display a captured image (or part of a captured image) generated by imaging in the imaging device, as indicated by U1 to U3 illustrated in
As the imaging device according to the present embodiment, for example, a lens/image sensor and a signal processing circuit can be cited. The lens/image sensor is constituted of, for example, an optical lens and an image sensor using a plurality of image sensors such as CMOS (Complementary Metal Oxide Semiconductor). The signal processing circuit includes, for example, an AGC (Automatic Gain Control) circuit or an ADC (Analog to Digital Converter) to convert an analog signal generated by the image sensor into a digital signal (image data). The signal processing circuit may also perform various kinds of signal processing, for example, the white balance correction processing, tone correction processing, gamma correction processing, YCbCr conversion processing, and edge enhancement processing.
It is needless to say that the input/output interface 158 can also be connected to an external device such as an operation input device (for example, a keyboard and a mouse), a display device, an imaging device as an external apparatus of the information processing apparatus 100. The display device 162 may be a device capable of both the display and user operations like, for example, a touch screen.
The communication interface 166 is a communication means included in the information processing apparatus 100 and functions as the communication unit 102 to communicate with an external device or an external apparatus such as an external imaging device, an external display device, and an external sensor via a network (or directly) wirelessly or through a wire. As the communication interface 164, for example, a communication antenna and RF (Radio Frequency) circuit (wireless communication), an IEEE802.15.1 port and transmitting/receiving circuit (wireless communication), an IEEE802.11 port and transmitting/receiving circuit (wireless communication), and a LAN (Local Area Network) terminal and transmitting/receiving circuit (wire communication) can be cited. As the network according to the present embodiment, for example, a wire network such as LAN and WAN (Wide Area Network), a wireless network such as wireless LAN (WLAN: Wireless Local Area Network) and wireless WAN (WWAN: Wireless Wide Area Network) via a base station, and the Internet using the communication protocol such as TCP/IP (Transmission Control Protocol/Internet Protocol) can be cited.
With the configuration shown in, for example,
For example, the information processing apparatus 100 additionally may be equipped with one or more sensors that fulfill the role of a detection unit (not illustrated) that obtains data which may be used in a process in accordance with an information processing method according to the present embodiment. The data which may be used in a process in accordance with an information processing method according to the present embodiment may be, for example, data which may be used to improve the estimation accuracy of the position of a user's line of sight, or data which may be used to specify (or estimate) the distance of a user from the display screen.
A sensor according to the present embodiment may be an arbitrary sensor capable of obtaining data which may be used in a process in accordance with an information processing method according to the present embodiment, such as an infrared sensor or a depth sensor, for example.
When configured to, for example, perform processing on a standalone basis, the information processing apparatus 100 may not include the communication interface 164. The information processing apparatus 100 may also be configured not to include the recording medium 156, the operation device 160, or the display device 162.
Referring to
As the communication unit 102, for example, a communication antenna and RF circuit and a LAN terminal and transmitting/receiving circuit can be cited, but the configuration of the communication unit 102 is not limited to the above example. For example, the communication unit 102 may adopt a configuration conforming to any standard capable of communication such as a USB terminal and transmitting/receiving circuit or any configuration capable of communicating with an external apparatus via a network.
The control unit 104 is made up of an MPU or the like, for example, and fulfills the role of controlling the information processing apparatus 100 overall. In addition, the control unit is equipped with the image acquisition unit 110, the determination unit 112, and the processing unit 114, and fulfills the leading role of conducting a process in accordance with an information processing method according to the present embodiment, for example.
The image acquisition unit 110 fulfills the leading role of conducting the process of (1) above (image acquisition process), and acquires a captured image according to the present embodiment.
The determination unit 112 fulfills the leading role of conducting the process of (2) above (determination process).
The determination unit 112 determines the operator from among users included in the captured image acquired by the image acquisition unit 110, for example. More specifically, the determination unit 112 determines the operator from among users included in the captured image on the basis of the captured image by conducting one of the processes from the determination process according to the first example indicated in (2-1) above to the determination process according to the sixth example indicated in (2-6) above, for example.
In addition, the determination unit 112 may also conduct one or more processes from among the determination process according to the seventh example indicated in (2-7) above to the determination process according to the tenth example indicated in (2-10) above.
The processing unit 114 fulfills the leading role of conducting the process of (3) above (execution process), and conducts a process on the basis of information about user line of sight corresponding to the operator determined in the determination unit 112.
In addition, the processing unit 114 may also conduct the execution process according to the first example indicated in (3-1) above, or the execution process according to the second example indicated in (3-2) above.
By being equipped with the image acquisition unit 110, the determination unit 112, and the processing unit 114, the control unit 104 leads the conducting of process in accordance with an information processing method according to the present embodiment, for example.
According to the configuration illustrated in
Consequently, according to the configuration illustrated in
Also with the configuration shown in, for example,
However, the configuration of the information processing apparatus according to the present embodiment is not limited to the configuration in
For example, the information processing apparatus according to the present embodiment can include one or two or more of the image acquisition unit 110, the determination unit 112, and the processing unit 114 shown in
The information processing apparatus according to the present embodiment may not include the communication unit 102 when communicating with an external device or an external apparatus via an external communication device having the function and configuration similar to those of the communication unit 102 or when configured to perform processing on a standalone basis.
Also, the information processing apparatus according to the present embodiment additionally may be equipped with an imaging unit (not shown) made up of an imaging device, for example. When the imaging unit (not shown) is provided, in the information processing apparatus according to the present embodiment, it becomes possible to process a captured image generated by imaging in the imaging unit (not shown), determine the operator, obtain information about user line of sight on the basis of the captured image, and the like. Also, when the image unit (not shown) is provided, the information processing apparatus according to the present embodiment is also capable of causing the display screen to display a captured image (or part of a captured image) generated in the imaging unit (not shown), as indicated by U1 to U3 illustrated in
Also, the information processing apparatus according to the present embodiment additionally may be equipped with a detection unit (not shown) made up of an arbitrary sensor capable of obtaining data which may be used in a process in accordance with an information processing method according to the present embodiment, for example. When the detection unit (not shown) is provided, the information processing apparatus according to the present embodiment uses detection values from the detection unit (not shown) in a process in accordance with an information processing method according to the present embodiment, for example.
In the foregoing, the information processing apparatus has been described as the present embodiment, but the present embodiment is not limited to such a form. The present embodiment can also be applied to various devices, for example, a TV set, a display apparatus, a tablet apparatus, a communication apparatus such as a mobile phone and smartphone, a video/music playback apparatus (or a video/music recording and playback apparatus), a game machine, and a computer such as a PC (Personal Computer). The present embodiment can also be applied to, for example, a processing IC (Integrated Circuit) that can be embedded in devices as described above.
The present embodiment may also be realized by a system including a plurality of apparatuses predicated on connection to a network (or communication between each apparatus) like, for example, cloud computing. That is, the above information processing apparatus according to the present embodiment can be realized as, for example, an information processing system including a plurality of apparatuses.
A program for causing a computer to function as an information processing apparatus according to the present embodiment (for example, a program enabling the execution of a process in accordance with an information processing method according to the present embodiment, such as the process of (1) above (image acquisition process), the process of (2) above (determination process), and the process of (3) above (execution process)) may be executed by a processor or the like in the computer, and thereby enable the computer to determine an operator who performs operations by line of sight, and conduct a process on the basis of the line of sight of the determined operator.
Also, effects achieved by the above processing according to the information processing method according to the present embodiment can be achieved by a program causing a computer to function as an information processing apparatus according to the present embodiment being performed by a processor or the like in the computer.
In the foregoing, the preferred embodiment of the present disclosure has been described in detail with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to the above examples. A person skilled in the art may find various alterations and modifications within the scope of the appended claims and it should be understood that they will naturally come under the technical scope of the present disclosure.
For example, the above shows that a program (computer program) causing a computer to function as an information processing apparatus according to the present embodiment is provided, but the present embodiment can further provide a recording medium caused to store the program.
The above configuration shows an example of the present embodiment and naturally comes under the technical scope of the present disclosure.
Effects described in this specification are only descriptive or illustrative and are not restrictive. That is, the technology according to the present disclosure can achieve other effects obvious to a person skilled in the art from the description of this specification, together with the above effects or instead of the above effects. Additionally, the present technology may also be configured as below.
(1)
An information processing apparatus including:
(2)
The information processing apparatus according to (1), wherein
(3)
The information processing apparatus according to (2), wherein
(4)
The information processing apparatus according to (3), wherein
(5)
The information processing apparatus according to (3) or (4), wherein
(6)
The information processing apparatus according to (2), wherein
(7)
The information processing apparatus according to (1), wherein
(8)
The information processing apparatus according to (7), wherein
(9)
The information processing apparatus according to (8), wherein
(10)
The information processing apparatus according to (8) or (9), wherein
(11)
The information processing apparatus according to (7), wherein
(12)
The information processing apparatus according to (1), wherein
(13)
The information processing apparatus according to (1), wherein
(14)
The information processing apparatus according to (1), wherein
(15)
The information processing apparatus according to any one of (1) to (14), wherein
(16)
The information processing apparatus according to any one of (1) to (15), wherein
(17)
The information processing apparatus according to any one of (1) to (16), wherein
(18)
The information processing apparatus according to (17), wherein
(19)
The information processing apparatus according to any one of (1) to (18), wherein
(20)
An information processing method executed by an information processing apparatus, the information processing method including:
| Number | Date | Country | Kind |
|---|---|---|---|
| 2013-190715 | Sep 2013 | JP | national |
The present application is a continuation application of U.S. patent application Ser. No. 14/917,244, filed Mar. 7, 2016, which is a National Stage of PCT/JP2014/067433, filed Jun. 30, 2014, and which claims the priority from Japanese Patent Application JP 2013-190715, filed Sep. 13, 2013, the entire content of which is hereby incorporated by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 6353764 | Imagawa et al. | Mar 2002 | B1 |
| 20060281969 | Wang et al. | Dec 2006 | A1 |
| 20080240563 | Takano | Oct 2008 | A1 |
| 20090074304 | Momosaki | Mar 2009 | A1 |
| 20090315869 | Sugihara et al. | Dec 2009 | A1 |
| 20100205667 | Anderson et al. | Aug 2010 | A1 |
| 20110032274 | Miyata | Feb 2011 | A1 |
| 20120106792 | Kang et al. | May 2012 | A1 |
| 20130243270 | Kamhi et al. | Sep 2013 | A1 |
| 20130307771 | Parker et al. | Nov 2013 | A1 |
| 20140111420 | Ann et al. | Apr 2014 | A1 |
| 20140157129 | Dinshaw et al. | Jun 2014 | A1 |
| 20150030214 | Fujiwara et al. | Jan 2015 | A1 |
| 20150116206 | Irie | Apr 2015 | A1 |
| 20160246366 | Huang et al. | Aug 2016 | A1 |
| Number | Date | Country |
|---|---|---|
| 1223391 | Jul 1999 | CN |
| 103826145 | May 2014 | CN |
| 0919906 | Jun 1999 | EP |
| 0919906 | Jun 1999 | EP |
| 2735938 | May 2014 | EP |
| 2850504 | Mar 2015 | EP |
| 11-327753 | Nov 1999 | JP |
| 2010-231355 | Oct 2010 | JP |
| 2010-238145 | Oct 2010 | JP |
| 2014-086085 | May 2014 | JP |
| 10-2014-0050484 | Apr 2014 | KR |
| 2013173148 | Nov 2013 | WO |
| 2014061916 | Apr 2014 | WO |
| Entry |
|---|
| Office Action for CN Patent Application No. 201480048803.2, dated Dec. 20, 2018, 07 pages of Office Action and 11 pages of English Translation. |
| Office Action for EP Patent Application No. 14843497.0, dated Aug. 6, 2018, 7 pages of Office Action. |
| Office Action for EP Patent Application No. 14843497.0, dated Aug. 6, 2018, 07 pages. |
| International Search Report and Written Opinion of PCT Application No. PCT/JP2014/067433, dated Aug. 5, 2014, 13 pages of English Translation and 11 pages of ISRWO. |
| International Preliminary Report on Patentability of PCT Application No. PCT/JP2014/067433, dated Mar. 24, 2016, 13 pages of English Translation and 08 pages of IPRP. |
| Notice of Allowance and Fees Due for U.S. Appl. No. 14/917,244, dated Sep. 27, 2018, 02 pages. |
| Notice of Allowance and Fees Due for U.S. Appl. No. 14/917,244, dated Jun. 18, 2018, 14 pages. |
| Advisory Action for U.S. Appl. No. 14/917,244, dated Feb. 6, 2018, 03 pages. |
| Final Rejection for U.S. Appl. No. 14/917,244, dated Nov. 30, 2017, 33 pages. |
| Non-Final Rejection for U.S. Appl. No. 14/917,244, dated Jun. 27, 2017, 32 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20190041985 A1 | Feb 2019 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 14917244 | US | |
| Child | 16154331 | US |
