1. Field of the Invention
The present invention relates to a body gesture control system for operating electrical and electronic devices. More particularly, the present invention relates to the body gesture control system provided with a gesture change detection unit for operating electrical and electronic devices.
2. Description of the Related Art
In general, remote signaling devices with infrared or RF signals are widely used to control functions of various electrical and electronic devices, including electrical machines, VCRs, TVs and computer devices. However, there is a requirement for freehand control to interact with the electrical and electronic devices. Moreover, voice or hand gesture signals can also be used to control functions of electrical and electronic devices.
By way of example, U.S. Pat. No. 5,594,463, entitled “HAND GESTURE MACHINE CONTROL SYSTEM,” discloses a system for the control from a distance of machines having displays and including hand gesture detection in which the hand gesture causes movement of an on-screen hand icon over an on-screen machine control icon, with the hand icon moving the machine control icon in accordance with sensed hand movements to effectuate machine control. TV control led by hand signals includes detecting a single hand gesture and providing a hand icon on the screen along with the provision of icons representing TV controls such as volume, channel, color, density, etc. A television camera detects the hand in a noisy background through correlation techniques based on values of local image orientation. In order to trigger the system into operation, a trigger gesture such as the “how” sign is distinguished from the background through the utilization of orientation angle differences. However, the control system is designed for the control of on-screen hand icons by a hand gesture that fails to meet a requirement for reducing mis-recognition of gestures.
Furthermore, U.S. Pat. No. 6,002,808, entitled “HAND GESTURE CONTROL SYSTEM,” discloses a system is provided for rapidly recognizing hand gestures for the control of computer graphics. Particularly, image moment calculations are utilized to determine an overall equivalent rectangle corresponding to hand position, orientation and size, with size correlating to the width of the hand. A hole generated through the utilization of the touching of the forefinger with the thumb provides a special trigger gesture recognized through the corresponding hole in the binary representation of the hand. Image moments of images of other objects are detected for controlling or directing onscreen images. However, the control system is also designed for the control of onscreen images by a hand gesture that fails to meet a requirement for reducing mis-recognition of gestures.
As is described in greater detail below, the present invention provides a body gesture control system for operating electrical and electronic devices. A gesture change detection unit is provided to process body gesture images to precisely recognize a body gesture in such a way as to mitigate and overcome the above problem.
The primary objective of this invention is to provide a body gesture control system for operating electrical and electronic devices. An image processor device is used to process body gesture images with a gesture change detection unit to precisely recognize a body gesture. Accordingly, the body gesture control system is successful in reducing the possibility of mis-recognition among various types of the body gestures.
The hand gesture control system in accordance with an aspect of the present invention includes:
an image sensor device used to capture and generate at least two body gesture images of a user; and
an image processor device being in communication with the image sensor device for receiving image data, the image processor device being operated to process the captured body gesture images for recognizing the body gesture, the image processor device including:
an image calculation unit calculating gesture regions of the captured body gesture images;
a gesture change detection unit electrically connected with the image calculation unit, the gesture change detection unit being operated to detect changes of the captured body gesture images and to thereby determine a body gesture recognition signal; and
a signal transmission unit electrically connected with the gesture change detection unit, the signal transmission unit used to output the body gesture recognition signal received from the gesture change detection unit.
In a separate aspect of the present invention, the image sensor device is selected from a CCD or a CMOS.
In a further separate aspect of the present invention, the image calculation unit is selected from an image gradient calculation/binaryzation unit.
In yet a further separate aspect of the present invention, the image gradient calculation/binaryzation unit utilizes an edge detection method to detect an edge and to thereby calculate a gradient and a motion vector.
In yet a further separate aspect of the present invention, the edge detection method is implemented by a Sobel operator or a Laplacian operator.
In yet a further separate aspect of the present invention, the gesture change detection unit includes a trajectory calculation unit to calculate a gesture trajectory and a gesture direction, and further includes a gesture-judging unit to judge the gestures.
In yet a further separate aspect of the present invention, the gesture change detection unit further includes a line-of-sight angle calculation unit to calculate a user's sight angle.
In yet a further separate aspect of the present invention, the gesture change detection unit includes a skin-color filter array to detect a skin-color region.
In yet a further separate aspect of the present invention, the skin-color filter array is selected from a band-pass filter.
Further scope of the applicability of the present invention will become apparent from the detailed description given hereinafter. However, it should be understood that the detailed description and specific examples, while indicating preferred embodiments of the invention, are given by way of illustration only, since various modifications will become apparent to those skilled in the art from this detailed description.
The present invention will become more fully understood from the detailed description given hereinbelow and the accompanying drawings which are given by way of illustration only, and thus are not limitative of the present invention, and wherein:
It is noted that a body gesture control system for operating electrical and electronic devices in accordance with the preferred embodiments of the present invention is applicable for various devices, including electrical machines, VCRs, TVs and computer devices, which are not limitative of the present invention.
Throughout the specification, unless the context requires otherwise, the technical terms, as used herein, have the same meaning as one of ordinary skill commonly known in the art. The term body gesture means a gesture using body parts, including hands, fingers, head, lips, eyes, etc.
With continued reference to
Still referring to
Still referring to
Assume that the size of the body gesture image consists of M scanned lines and N scanned lines, namely a M×N determinant. The body gesture image is given by an IM×N matrix with gray level pixels I(x, y), where (x, y) is coordinates of pixels, 1≦x≦M, 1≦y≦N.
A Sobel operator is expressed as
To calculate gradients in the x direction and the y direction of the edges of a 3×3 grid, the gradient can be written in the form
With the notations Sobel_X(x, y) and Sobel_Y(x, y), the matrix of gradients of the body gesture image can be expressed as
In judging gradients in the x direction and the y direction of the edges of the 3×3 grid, two thresholds (two gradient values (i.e., focus value, FV)) are utilized to compare with gradients of Sobel_X(x, y) and Sobel_Y(x, y). In a FV binaryzation, the M×N matrix Sobel_show_data of the body gesture image is given by
Sobel_show_data=[Sobel_show_data(x,y)]M×N=[D(96<Sobel—X(x,y)<224)or D(96<Sobel—Y(x,y)<224)]M×N
where D is a logic judgment operator.
The M×N matrix Sobel_show_data of the body gesture image can be expressed as a binary matrix and written in the form
buf—t1=[buf—t1(x,y)]M×N=Sobel_show_data(t−1)
where t is currently capturing time and t−1 is previously capturing time.
With continued reference to
In calculating the gradient binary matrix, the sum of detection results in the x direction with 8 points (from x to x−7) can be expressed in the form
the sum in the y direction can then be expressed in the form
where eye_buf_sum(x, y) is total moving points.
By way of example, chroma components cb and cr received from the captured body gesture image of the image sensor device 1 to detect skin color utilized in the skin-color filter are in the form
then from which, when y is odd, the logically judging result buf_show(x, y) is in the form
buf_show(x,y)=D(20<eye_buf_sum(x,y))AND D(Cb_show—en(x,y)=1)AND D(Cr_show—en(x,y)=1)
when y is even, the logically judging result is in the form
buf_show_data=buf_show(x,y−1)
In trajectory calculation, by way of example, coordinates of the left hand are expressed in the form
The judgment operator D is used to determine whether an amount of skin-color pixels is greater than 2c. If the skin-color pixels are greater, 2c pixels are taken into account for average coordinate calculations. After modified, these coordinates become
xc(l)=FT_LEFT_show—X_temp=FT_LEFT_range—X_temp>>c
yc(l)=FT_LEFT_show—Y_temp=FT_LEFT_range—Y_temp>>c
where >>c means that c bits are rounded off.
In the center of trajectory point (xc(l), yc(l)), within the range of 120×120 pixels, and since these conditions take the form
xL(l)=xc(l)−60, xR(l)=xc(l)+60
yU(l)=yc(l)−60, yD(l)=yc(l)+60
the coordinates (x, y) become in the form
such that coordinates of the gesture trajectory are generated by the trajectory calculation unit 221. The judgment operator D is further used to determine whether an amount of skin-color pixels is greater than 2c. If the skin-color pixels are greater, 2c pixels are taken into account for average coordinate calculations.
With continued reference to
Turning now to
In a preferred embodiment, the gesture change detection unit 22 is used to calculate displacements, velocities, accelerations and forces of the body gestures with respect to an image sensor frame (i.e. body gesture image) captured in a different time by the image sensor device 1 so that accurate movements of the body gestures can be successfully detected, as is described in greater detail below.
To calculate movements from the first position T0 to the second position T1, the velocities of the finger touch region in the x direction are
vleft-edge(T0→T1)=(x2−x0)/(T1−T0)
vright-edge(T0→T1)=(x3−x1)/(T1−T0)
Similarly, the velocities of the finger touch region in the y direction are
vupper-edge(T0→T1)=(y2−y0)/(T1−T0)
vlower-edge(T0→T1)=(y3−y1)/(T1−T0)
In addition, widths of the finger touch regions in the first and second positions are (x1−x0) and (x3−x2) in the x direction. Accordingly, changes of the widths of the finger touch regions in the x direction are approximately expressed in the form
vwidth,x(T0→T1)≈[(x3−x2)−(x1−x0)]/(T1−T0)
Similarly, changes of the widths of the finger touch regions in the y direction are approximately expressed in the form
vwidth,y(T0→T1)≈[(y3−y2)−(y1−y0)]/(T1−T0)
Therefore, from the second position T1 to the third position T2, the velocities and the changes of widths of the finger touch region in the x and y directions can also be expressed by
vleft-edge(T1→T2)=(x4−x2)/(T2−T1)
vright-edge(T1→T2)=(x5−x3)/(T2−T1)
vupper-edge(T1→T2)=(y4−y2)/(T2−T1)
vlower-edge(T1→T2)=(y5−y3)/(T2−T1)
vwidth,x(T1→T2)≈[(x5−x4)−(x3−x2)]/(T2−T1)
vwidth,y(T1→T2)≈[(y5−y4)−(y3−y2)]/(T2−T1)
Then, the accelerations of the finger touch regions in the x and y directions are
aleft-edge(T0→T2)=[vleft-edge(T1→T2)−vleft-edge(T0→T1)]/(T2−T0)
aright-edge(T0→T2)=[vright-edge(T1→T2)−vright-edge(T0→T1)]/(T2−T0)
aupper-edge(T0→T2)=[vupper-edge(T1→T2)−vupper-edge(T0→T1)]/(T2−T0)
alower-edge(T0→T2)=[vlower-edge(T1→T2)−vlower-edge(T0→T1)]/(T2−T0)
It is the fact that a force of the finger gesture acting in the x and y directions is proportional to an acceleration value of the finger touch region represented if the mass of the finger is approximately constant.
Similarly, the acceleration values of the finger touch regions in the z direction representing a force of the finger gesture are given as
awidth,x(T0→T2)≈[vwidth,x(T1→T2)−vwidth,x(T0→T1)]/(T2−T0)
awidth,y(T0→T2)≈[vwidth,y(T1→T2)−vwidth,y(T0→T1)]/(T2−T0)
Accordingly, these above-mentioned equations are applied to calculate displacements, velocities, accelerations and forces of several body-part gestures, including finger-clicking gestures, clapping gestures, hand-waving gestures, nodding gestures and other body gestures. Preferably, the body gesture control system is used to simulate typewriting on a virtual keyboard or operating a virtual touch panel or mouse.
Although the invention has been described in detail with reference to its presently preferred embodiment(s), it will be understood by one of ordinary skill in the art that various modifications can be made without departing from the spirit and the scope of the invention, as set forth in the appended claims.
Number | Date | Country | Kind |
---|---|---|---|
99100855 A | Jan 2010 | TW | national |
Number | Name | Date | Kind |
---|---|---|---|
5594463 | Sakamoto | Jan 1997 | A |
6002808 | Freeman | Dec 1999 | A |
20090079813 | Hildreth | Mar 2009 | A1 |
20090217211 | Hildreth et al. | Aug 2009 | A1 |
Number | Date | Country | |
---|---|---|---|
20110170745 A1 | Jul 2011 | US |