Patent Application
20190155392
The present invention pertains to user interaction system.
The present invention further pertains to user interaction method.
The present invention further pertains to a portable pointing device.
The present invention still further relates to a storage medium.
WO2004047011 discloses a portable pointing device connected to a camera and sending pictures to a digital signal processor, capable of recognizing an object and a command given by the user by making a gesture with the portable pointing device, and controlling an electrical apparatus on the basis of this recognition. The cited document further suggests that motion can be determined by imaging successive pictures and applying a motion estimation algorithm.
For example a gesture in the form of an upward motion may be used to increase an audio volume of a controlled apparatus. A circular motion trajectory might be used to “rewind”, i.e. to cause a playback apparatus to return to an earlier point in time in the content that is reproduced.
It is an object to provide a user interaction system which is improved in that it enables a more efficient way of recognizing gestures made with the portable pointing device.
It is a further object to provide a user interaction method which is improved in that it enables a more efficient way of recognizing gestures made with the portable pointing device.
It is a still further object to provide a portable pointing device allowing for a more efficient way of recognizing gestures made therewith. It is a still further object to provide a storage medium having stored thereon a computer program enabling a more efficient way of recognizing gestures with a digital signal processor.
In accordance with the object first mentioned a user interaction system according to a first aspect of the invention is provided that comprises an electrical apparatus, a portable pointing device, and a digital signal processor. The portable pointing device is operable by a user for pointing to a region in space and includes a camera connected thereto for obtaining subsequent images from said space. The digital signal processor is capable of receiving and processing the subsequent images to determine a motion of said pointing device, and capable of transmitting user interface information to the electrical apparatus if it is determined that the motion as determined corresponds to a predetermined gesture. The user interface information represents control information for control of the electrical apparatus in accordance with the predetermined gesture. The electrical apparatus, e. g. a computer or a television, then performs an action corresponding to the user interaction command.
The user interaction system is characterized in that the digital signal processor comprises a pattern recognition module for recognizing a predetermined pattern in said subsequent images, a position estimation unit for estimating a position of said pattern in said subsequent images, and a gesture matching unit for matching a predetermined gesture on the basis of data indicative for differences of said position (position differences) in said subsequent images.
In the user interaction system according to the present invention the gesture matching unit compares the data indicative for differences of said position differences with respective position differences corresponding to respective predetermined gestures.
Typically the user indicates the beginning and end of a gesture made with the pointing device, for example by pressing and releasing an activation button.
It is however not necessary to postpone the comparison until a complete sequence of position differences is retrieved from the subsequent captured images. Instead immediately after capturing the second and each subsequent image, i.e. while the gesture is being performed, the gesture recognition process compares the determined position difference with the corresponding position differences known for the various predetermined gestures Therewith a fast response is possible. I.e. at the moment that the user indicates that s/he completed the gesture, the preceding position differences have already been processed to determine the most likely gesture that could have caused the determined position differences of the predetermined pattern between successive captured images, and a user interface information with the proper control command can be transmitted to the apparatus. It would even be possible to transmit a control command even before the user indication once the position differences received so far can only be attributed to a single gesture, i.e. the likelihood of the most likely gesture has reached a predetermined threshold. It has been found however that operation is more reliable if gesture recognition is continued until the user provides the indication that user that s/he completed the gesture or the end of the gesture is detected automatically.
Furthermore, the digital signal processor is capable of determining successive positions of a predetermined pattern in images obtained by the camera, and of estimating the position differences from these successive positions. This simplifies the estimation of the motions of the portable pointing device, in that it is no longer necessary to detect these motions by comparing the images in its entirety, but that it suffices to compare the positions predetermined pattern in the images taken by the camera. This is relatively easy due to the fact that a predetermined spot or shape or combinations thereof can be identified within the image. This is advantageous for a portable pointing device as less processing power is required and therewith longer battery lifetime is obtained.
In an embodiment of the user interaction system according to the first aspect the digital signal processor further includes a spatial transformation unit for applying a spatial transformation to the estimated position in order to obtain an estimated pointing position. This facilitates a more intuitive control by the user as the user can sometimes more easily imagine a gesture to be formed in terms of a trajectory followed by a pointing position than a trajectory followed by the position where the predetermined pattern will be detected in the camera image.
In order to further facilitate user control, a user interaction system is provided wherein the controlled electrical apparatus includes a display screen, and the controlled electrical apparatus is arranged to display the estimated pointing position on the display screen.
The predetermined pattern may result from any absolute reference in space that can be detected. For example if the apparatus to be controlled is a television or other device having a display screen for displaying active video content, the predetermined pattern to be detected is the portion in the camera image that comprises active video in contrast to a stationary environment
Alternatively, or in addition one or more beacons may be arranged in the space that radiate photon radiation. In that case the predetermined pattern to be recognized is a pattern resulting from the one or more beacons. This has the advantage that the operation of the system according of the first aspect becomes independent of the video content made available by the apparatus to be controlled.
Each of the one or more beacons may be associated with a respective controllable apparatus. In that case feedback may be provided to the user indicating which beacon is pointed at, as this implies which of the various apparatuses is currently under control. Such feedback may be provided for example visually by a lighting element arranged near the beacon pointed at, or by a display on the pointing device showing an overview of the various apparatuses and/or their beacons.
A beacon may be any photon-radiating means that provides for a detectable pattern (a dot, cluster or other arrangement of dots, a geometrical shape) in the camera image. Preferably the photon radiation emitted by the beacon is not visible to human beings, for example photon radiation having a wavelength in the infra red range.
In accordance with the object second mentioned above, a user interaction method is provided in a system comprising an electrical apparatus, a portable pointing device with a camera, and a digital signal processor. The method comprises
making a gesture with said portable pointing device,
capturing successive images with said camera comprised in the portable pointing device while making the gesture; and
receiving successive image data representing said successive images by said digital signal processor,
processing the successive image data by said digital processor,
transmitting user interface information to the electrical apparatus, which user interface information is constructed from said command identification data. The processing referred to above, includes
determining successive positions of a predetermined pattern in said successive images obtained by the camera,
identifying a gesture on the basis of difference data resulting from differences of said position in said successive images,
outputting command identification data representing a user interaction command corresponding with the identified gesture. The user interface information transmitted to the electrical apparatus is constructed from said command identification data.
In accordance with the object third mentioned above a portable pointing device is provided that is operable by a user for pointing to a region in space. The portable pointing device includes a camera connected thereto for obtaining images from said space and a digital signal processor. The digital signal processor is capable of receiving and processing the images, and capable of receiving and processing the subsequent images to identify a gesture made with said pointing device, and capable of transmitting user interface information to the electrical apparatus, said user interface information representing control information for said electrical apparatus corresponding to the identified gesture. The digital signal processor comprises a pattern recognition module for recognizing a predetermined pattern in said subsequent images and for estimating a position of said pattern in said subsequent images, and a gesture matching unit for identifying a gesture on the basis of difference data resulting from differences of said position in said subsequent images.
In an embodiment the digital signal processor further includes a spatial transformation unit for applying a spatial transformation to said estimated position in order to obtain an estimated position (pointing position) of a location pointed to by the user with the portable pointing device.
In an embodiment the electrical apparatus includes a display screen, and is arranged to display said estimated pointing position on said display screen.
In an embodiment the predetermined pattern to be recognized is a pattern resulting from photon radiation emitted by at least one beacon.
The camera of the portable pointing device is capable of detecting radiation from at least one beacon arranged in said space and the digital signal processor is capable of determining successive positions of a representation of the at least one beacon in images obtained by the camera, and of estimating the motion trajectory from said successive positions.
The detection of the at least one beacon may be facilitated by one or more of the measures in the embodiments presented below.
In an embodiment the at least one beacon radiates non-visible radiation and the camera is substantially insensitive to radiation other than the non-visible radiation radiated by the at least one beacon. This may be achieved in that the camera has sensor elements that are intrinsically insensitive to said radiation other than the non-visible radiation, or in other ways, for example by an optical filter arranged in front of the camera that selectively passes the non-visible radiation of the at least one beacon. For example the at least one beacon may be an IR beacon and the camera may be an IR camera.
If multiple beacons are used it may be desired to provide further features that facilitate identification of the contribution of the various beacons to predetermined patterns in the captured image. Regardless whether or not multiple beacons are used, this may also be relevant if photon radiation sources are present in the environment that could disturb a proper recognition of the predetermined patterns.
In an embodiment the at least one beacon comprises a driving unit to drive a photon radiation element of said beacon according to a time-modulated pattern and the digital signal processor includes a detector for detecting image data modulated according to that pattern. The intensity may for example be modulated according to a sine pattern having a modulation frequency, and the detector may be arranged to detect image data varying with this frequency and to ignore other image data. Any modulation pattern, for example a sine wave pattern may be used for modulation.
In an embodiment the at least one beacon comprises a photon radiation element capable of radiating photon radiation at mutually different wavelengths, and a driving unit to drive modulate wavelength of the photon radiation element according to a time-modulated pattern and the digital signal processor includes a detector for detecting image data modulated according to that pattern.
In an embodiment the at least one beacon is arranged to emit photon radiation according to a unique spatial pattern and/or a set of beacons is arranged to emit photon radiation according to a unique spatial pattern, and the digital signal processor is provided with a pattern recognition module that identifies the unique spatial pattern as the predetermined pattern within image data retrieved from the camera.
These and other aspects are described in more detail with reference to the drawings. Therein:
Like reference symbols in the various drawings indicate like elements unless otherwise indicated.
The portable pointing device 12, of which an embodiment is shown in more detail in
The digital signal processor 14 is capable of receiving and processing the images, and is capable of transmitting user interface information to the electrical apparatus, which is derived by processing the images. In the embodiments of
As shown in more detail in
In again another embodiment two or more of the portable pointing device 12, the digital signal processor 14 and the electrical apparatus 10 to be controlled may be coupled via a common wireless or wired network.
The digital signal processor 14 is capable of determining successive positions of a representation of the at least one beacon 16 in images obtained by the camera 122. The digital signal processor 14 can estimate the motion trajectory from these successive positions.
The digital signal processor or parts thereof may be implemented as an ASIC, which might be hardcoded. Alternatively the digital signal processor or parts thereof may be implemented as a generally programmable processor carrying out a program stored in storage medium. Also intermediate implementations are possible, in the form of processors having an instruction set for a restricted set of operations, reconfigurable processors and combinations thereof. In the embodiment shown in
therein Dt=Pi,t−Pi,t−1
In other embodiments the difference data Dt resulting from differences of said position Pi,t is obtained by applying a spatial transformation to said estimated position in order to obtain an estimated pointing position. Subsequently, the difference data Dt is determined from the difference in coordinates between subsequent pointing positions.
One relative position Dt or a sequence Dt, Dt+1, . . . ,Dt+n of these relative positions forms a motion characterizing signature MS.
One relative position indicative for an upward movement may for example represent a gesture to be used for turning on the electrical apparatus 10. Another relative position indicative for an opposite movement may for example represent a gesture to be used for turning off the electrical apparatus.
A motion characterizing signature MS composed of a sequence of relative positions may be used to extend the range of possible gestures. More complex gestures may be used for restricted control purposes. For example, a particular gesture only known by the user or by a restricted group of users may be used as a password to obtain exclusive access to the electrical apparatus 10.
A reliable relative position Dt may already be obtained with a single beacon 16, provided that the user takes care that s/he operates the portable pointing device 12 from a fixed position with respect to the single beacon 16 and holds the portable pointing device 12 in a fixed roll angle, as defined in
As indicated above, a single beacon suffices if the roll angle is fixed, or if effects of variations of the roll angle can be compensated using data indicative for an observed roll angle as detected by a roll angle detector. In the absence of a roll angle detector compensation for variations in the roll angle would still be possible if more than one beacon is used.
In an embodiment, the at least one beacon 16a, is part of a set of beacons 16a, 16b, etc. A pair of beacons 16a, 16b already is sufficient to determine a roll angle and to use the observed value of the roll angle for roll angle compensation. In this case each image IM,t results in a pair of positions Pi,t, one for each of the beacons. Nevertheless, more beacons may be used if desired. In this case each image IM,t results in three positions Pi,t, one for each of the beacons. As an alternative or additional measure, the beacons may have a characteristic ‘signature’, enabling the pattern recognition module 1421 to determine which image data results from each of the beacons. In that case it would already be sufficient to have two beacons, of which one is identifiable by its characteristic ‘signature’. The signature of the identifiable beacon may for example be provided in that a driving unit 162 drives a photon radiation element 161 of the beacon according to a time-modulated pattern.
For a proper functioning of the portable pointing device 12 it is sufficient that the digital signal processor 14 can adequately identify the at least one beacon 16 in images IM,t obtained by the camera 122.
In an embodiment the at least one beacon 16 radiates non-visible radiation and the camera 122 is substantially insensitive to radiation other than the non-visible radiation radiated by the at least one beacon 16. In this way no substantial image processing is necessary to identify the position of the beacon or beacons in the images IM,t. The image data IM,t may for example in a practical embodiment the beacon 16 or set of beacons 16a, 16b, (16c) radiate infra-red radiation.
[Delta] position determining module 1422 then determines the changes of those positions Pa, t; Pb, tin subsequent images. Alternatively, delta position determining module 1422 determines the change in position pointed to by the user, wherein the respective pointing positions are estimated from the positions Pa, t; Pb, t of the predetermined pattern detected in subsequent captured images. As long a the roll angle does not change substantially, the relative position determining module 1422 can easily determine which position identified in an image IMc,t corresponds to which position identified in a preceding image IMc, t−1. For example a leftmost point may always be identified as originating from a first beacon, a rightmost point may always be identified as originating from a second beacon, and an uppermost point may always be identified as originating from a third beacon. Being able to identify the beacons makes it possible to identify and correct for changes in distance and roll angle. Therewith sufficient information is available to determine the motion trajectory performed by the user in pointing position coordinates. If it is not possible to identify the beacons used in this way, because more substantial roll movements of the portable pointing device occur, the following alternative solutions are possible to identify the beacons.
According to a first solution an accelerometer is provided to determine the roll angle. Based on the therewith observed roll angle a compensation is applied to the data directly or indirectly obtained from the captured image data. For example a rotation operation compensating for the observed roll angle is applied to the captured image data, to obtain a roll angle compensated image which is further processed as if it were the captured image itself. Alternatively the position(s) found for the one or more beacons in the originally captured image may be corrected to compensate for the roll angle. Still alternatively a roll angle compensation may be applied in any further processing stage.
According to a second solution the images are sampled at a relatively high repetition rate, so that the points resulting from the same beacon in subsequent images IMc,t−1, IMc,t always are closer to each other than a point from that same beacon in image IMc,t to a point resulting from another beacon in the previous image IMc,t−1. In this way the path of points from a beacon can be tracked in the sequence of images IMc,t. For this purpose a tracking engine, for example including a Kalman filter or a particle filter may be applied. According to the second solution the beacons 16a, 16b, 16c are arranged at mutually different distances Dab, Dbc and Dac, as shown in
I.e. IMD(x,y,t)=ABS(IM(x,y,t)−IM(x,y,t-T/2)),
wherein x,y are the image coordinates.
The modulation method used for the beacon needs to have some different properties compared to possible interfering sources]. For example the modulation period is preferably selected relatively short in comparison to a period of intensity variations that may be caused by other photon radiation sources, e.g. monitors, TV-screens and light sources Alternatively more complex modulation patterns may be provided by driving unit 162 and detected by detector 143, which need not necessarily be at a high frequency.
In an embodiment as shown in
In a variation of the embodiment as partly shown in
In again another embodiment, as shown in
Alternatively a single beacon may be provided that emits photon radiation according to a unique spatial pattern to be detected. In this case the camera 122 may need a higher resolution than in case multiple beacons 16a, . . . , 16n are used that are distributed in space as in this case the dimensions of the spatial pattern that results in the captured image typically are smaller than the dimensions of a spatial pattern that result from a plurality of beacons distributed in space. Also one or more of the beacons as shown in
As a first step 51, a user makes a gesture with the portable pointing device 12. To that end the user may direct the portable pointing device 12 to an electrical apparatus to be controlled and move the point targeted at along an imaginary curve.
As a second step S2, while making the gesture, successive images are captured with the camera 122 comprised in the portable pointing device. In a third step S3, successive image data IM,t representing the successive images is received by a digital processor 14, which may be integrated in the pointing device 12 or may be provided elsewhere.
The successive image data IM,t is subsequently processed in a fourth step S4 by the digital processor. This processing may include the following processing steps. In a first processing step S41 successive positions Pa,t of a predetermined pattern are determined in the successive images. Typically the predetermined pattern results from a beacon or a plurality of beacons that are deliberately arranged in a space 20 where the portable pointing device 12 is used. As can be seen in
A second processing step S42 involves identifying a gesture on the basis of difference data resulting from differences of said position in said successive images. A particular implementation of this second processing step S42 is further shown in more detail in
The embodiment as illustrated in
As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, “or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).
Also, use of the “a” or “an” are employed to describe elements and components of the invention. This is done merely for convenience and to give a general sense of the invention. This description should be read to include one or at least one and the singular also includes the plural unless it is obvious that it is meant otherwise.
| Number | Date | Country | Kind |
|---|---|---|---|
| 14187126.9 | Sep 2014 | EP | regional |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2015/072530 | 9/30/2015 | WO | 00 |
