GESTURE CONTROL DEVICE AND METHOD

Abstract

A device for recognizing control gestures and determining which one device out of a plurality is the target of control acquires images of a gesture from each electronic device. A three dimensional coordinate system for each image is established, and coordinate of a central point of each electronic device determined. Extent of gesture to the left and to the right at different depths is determined and a regression plane equation is calculated. A distance between the regression plane and center points of each electronic device is determined and the electronic device with the closest (the shortest distance) center point is determined as the target device of the control gesture. A gesture control method is also provided.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Taiwan Patent Application No. 105116709 filed on May 27, 2016.

FIELD

The subject matter herein generally relates to interface devices, and particularly to a gesture control device and method capable of determining an object to be controlled by gesture, among a plurality of electronic devices.

BACKGROUND

Electronic devices can be controlled by gestures. A gesture command usually controls one electronic device. However, a number of electronic devices may be close together, and it is difficult to determine which one of the electronic devices should be controlled by the gesture.

BRIEF DESCRIPTION OF THE DRAWINGS

Implementations of the present technology will now be described, by way of example only, with reference to the attached figures.

FIG. 1 is a block diagram illustrating an exemplary embodiment of an operating environment of a device with system for control by gesture.

FIG. 2 is a block diagram illustrating an exemplary embodiment of a gesture control system running in the device of FIG. 1.

FIG. 3 is a schematic diagram illustrating an exemplary embodiment of a working process of the device of FIG. 1.

FIG. 4 is a flowchart illustrating an exemplary embodiment of a gesture control method.

DETAILED DESCRIPTION

It will be appreciated that for simplicity and clarity of illustration, where appropriate, reference numerals have been repeated among the different figures to indicate corresponding or analogous elements. In addition, numerous specific details are set forth in order to provide a thorough understanding of the embodiments described herein. However, it will be understood by those of ordinary skill in the art that the embodiments described herein can be practiced without these specific details. In other instances, methods, procedures, and components have not been described in detail so as not to obscure the related relevant feature being described. The drawings are not necessarily to scale and the proportions of certain parts may be exaggerated to better illustrate details and features. The description is not to be considered as limiting the scope of the embodiments described herein.

The term “comprising” means “including, but not necessarily limited to”, it specifically indicates open-ended inclusion or membership in a so-described combination, group, series, and the like.

FIG. 1 illustrate an exemplary embodiment of an operating environment of a gesture control device 100. The gesture control device 100 can communicate with a number of electronic devices. The gesture control device 100 can determine which one of the electronic devices should be controlled by a gesture. In the exemplary embodiment, the electronic devices can be, but are not limited to, televisions, air conditioners, fridges, multimedia players, monitors, computers, and the like. The gesture control device 100 can communicate with the electronic devices wirelessly, for example by using WIFI, Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMAW-CDMA), CDMA2000, IMT Single Carrier, Enhanced Data Rates for GSM Evolution (EDGE), Long-Term Evolution, (LTE), Time-Division LTE, (TD-LTE), High Performance Radio Local Area Network, (HiperLAN), High Performance Radio Wide Area Network, (HiperWAN), Local Multipoint Distribution Service, (LMDS), Worldwide Interoperability for Microwave Access, (WiMAX), ZIGBEE, BLUETOOTH, Flash Orthogonal Frequency-Division Multiplexing, (Flash-OFDM), High Capacity Spatial Division Multiple Access, (HC-SDMA), Universal Mobile Telecommunications System, (UMTS), UMTS Time-Division Duplexing, (UMTS-TDD), Evolved High Speed Packet Access, (HSPA+), Time Division Synchronous Code Division Multiple Access, (TD-SCDMA), Evolution-Data Optimized, (EV-DO), Digital Enhanced Cordless Telecommunications, (DECT), or the like. The gesture control device 100 also can communicate with electronic devices by wires.

In the exemplary embodiment, a first electronic device 200 and a second electronic device 300 are taken as examples of communicating with the gesture control device 100. In the exemplary embodiment, the first electronic device 200 includes a first image capturing device 20, the second electronic device 300 includes a second image capturing device 30. The first image capturing device 20 is a depth camera configured to capture images of gestures in a first effective range R1, the second image capturing device 30 is a depth camera configured to capture gesture images in a second effective range R2. The first electronic device 200 and the second electronic device 300 can execute functions according to gestures captured by the image capturing devices 20 and/or 30.

In the exemplary embodiment, the gesture control device 100 can be, but is not limited to, a server, a communication device such as a Set Top Box, or an integrated chip or programming modules embedded in the first electronic device 200 or in the second electronic device 300.

Referring to FIG. 2, in the exemplary embodiment, the gesture control device 100 can include, but is not limited to, a storage device 11 and a processor 12. The storage device 11 can be, but is not limited to, an internal storage system, such as a flash memory, a random access memory (RAM) for temporary storage of information, and/or a read-only memory (ROM) for permanent storage of information. The storage device 11 can also be a storage system, such as a hard disk, a storage card, or a data storage medium. The processor 12 can be, but is not limited to, a central processing unit, a digital signal processor, or a single chip, for example.

In the exemplary embodiment, the storage device 11 can store a gesture control system 10. The gesture control system 10 can include a number of modules, which are collections of software instructions stored in the storage device 11 and executed by the processor 12. In the exemplary embodiment, the gesture control system 10 can include an acquiring module 101, an establishing module 102, a calculating module 103, and a determining module 104.

The acquiring module 101 acquires an image of a gesture from each of the first image capturing device 20 and/or the second image capturing device 30 and acquires an orientation of a gesture in the gesture image. In the exemplary embodiment, the gesture image can include a depth information as to objects in the images. In the exemplary embodiment, the gesture image can include a number of pictures, thus the orientation and motion in the gesture can be acquired according to a position of the gesture in different pictures. The orientation of the gesture can indicate that the gesture of a user has a directivity. The orientation is determined as ended if the gesture is stopped for a preset time interval. In the exemplary embodiment, the first image capturing device 20 and the second image capturing device 30 capture the gesture image when a gesture is detected in the effective area of the first image capturing device 20 and the second image capturing device 30.

The establishing module 102 establishes a three dimensional coordinate system for the gesture image, and determines a coordinate of a central point of each of the electronic devices (the first electronic device 200 and the second electronic device 300). In the exemplary embodiment, plane coordinates of the gesture image are determined as an X axis and a Y axis of the coordinate system, and the depth direction of the gesture image is determined as a Z axis of the coordinate system. In the exemplary embodiment, the coordinate of the center points of the electronic devices are predetermined according to a position of the image capturing device. For example, if the first electronic device 200 is a computer, and the image capturing device 20 is located in the middle of an upper edge of a screen of the first electronic device 200, the coordinate of the center point of the first electronic device 200 will be predetermined as the coordinate of the center point of the screen of the first electronic device 200.

The acquiring module 101 further determines a coordinate of a most extreme left-side horizontal position (left end) of the gesture in different depths and a coordinate of a most extreme right-side horizontal position (right end) of the gesture in different depths when the gesture is ended. For example, as shown in FIG. 3, a coordinate of a left end A₁ of the gesture in a depth Z₁ is (x′₁, y′₁, z₁), a coordinate of a right end B₁ of the gesture in the depth Z₁ is (x″₁, y″₁, z₁), a coordinate of the left end A₂ of the gesture in a depth Z₂ is (x′₂, y′₂, z₂), a coordinate of the right end B₂ of the gesture depth Z₂ is (x″₂, y″₂, z₂). Thus a coordinate of the left end A_n of the gesture in a depth Z_n is (x′_n, y′_n, z_n), and a coordinate of the right end B_n of the gesture depth Z_n is (x″_n, y″_n, z_n).

The calculating module 103 calculates a coordinate of a center point between the left end and the right end of the gesture in each different depth. For example, a coordinate of a center point C₁ in the depth Z₁ is

$\left(\frac{x_1^{\prime }+x_1^{″}}{2},\frac{y_1^{\prime }+y_1^{″}}{2},z_1\right),$

a coordinate of a center point C₂ in the depth Z₂ is

$\left(\frac{x_2^{\prime }+x_2^{″}}{2},\frac{y_2^{\prime }+y_2^{″}}{2},z_2\right),$

and a coordinate of a center point C_n in the depth Z_n is

$\left(\frac{x_n^{\prime }+x_n^{″}}{2},\frac{y_n^{\prime }+y_n^{″}}{2},z_n\right).$

The calculating module 103 calculates a regression plane equation according to the coordinate of the center points of the gesture image in different depths.

In the exemplary embodiment, the calculating module 103 calculates the regression plan equation by using a regression analysis method. By standardizing the coordinate of the center points, the calculating module 103 obtains the following formulas:

$x_k^{\prime }=\frac{x_k-\stackrel{\_}{x}}{S_x},y_k^{\prime }=\frac{y_k-\stackrel{\_}{y}}{S_y},z_k^{\prime }=\frac{z_k-\stackrel{\_}{z}}{S_z},\mathrm{wherein},\stackrel{\_}{x}=\frac{x_1+x_2+\dots +x_n}{n},S_x=\sqrt{\frac{\sum _{i=1}^n{\left(x_i-\stackrel{\_}{x}\right)}^2}{n-1}},\stackrel{\_}{y}=\frac{y_1+y_2+\dots +y_n}{n},S_y=\sqrt{\frac{\sum _{i=1}^n{\left(y_i-\stackrel{\_}{y}\right)}^2}{n-1}},\stackrel{\_}{z}=\frac{z_1+z_2+\dots +z_n}{n},S_z=\sqrt{\frac{\sum _{i=1}^n{\left(z_i-\stackrel{\_}{z}\right)}^2}{n-1}}.$

Setting a standardized plane equation as z′=ax′+by′+c, a residual error is e_i=z′_i−{circumflex over (z)}′_i, and {circumflex over (z)}′_i=ax′_i+by′_i+c is the result. The calculating module 103 further calculates a value of “a”, “b”, and “c” in above formula when a value of

$\sum _{i=1}^ne_i^2$

is minimal.

$\begin{array}{c}\sum _{i=1}^ne_i^2=\sum _{i=1}^n{\left({\mathrm{ax}}_i^{\prime }+{\mathrm{by}}_i^{\prime }+c-z_i^{\prime }\right)}^2=\sum _{i=1}^n{\left[\left({\mathrm{ax}}_i^{\prime }+{\mathrm{by}}_i^{\prime }-z_i^{\prime }\right)+c\right]}^2\\ =\sum _{i=1}^n{\left({\mathrm{ax}}_i^{\prime }+{\mathrm{by}}_i^{\prime }-z_i^{\prime }\right)}^2+2c\sum _{i=1}^n\left({\mathrm{ax}}_i^{\prime }+{\mathrm{by}}_i^{\prime }-z_i^{\prime }\right)+\sum _{i=1}^nc^2\\ =\sum _{i=1}^n{\left({\mathrm{ax}}_i^{\prime }+{\mathrm{by}}_i^{\prime }-z_i^{\prime }\right)}^2+2\mathrm{ac}\sum _{i=1}^nx_i^{\prime }+2\mathrm{bc}\sum _{i=1}^ny_i^{\prime }-2c\sum _{i=1}^nz_i^{\prime }+{\mathrm{nc}}^2\\ =\sum _{i=1}^n{\left({\mathrm{ax}}_i^{\prime }+{\mathrm{by}}_i^{\prime }-z_i^{\prime }\right)}^2+{\mathrm{nc}}^2,\end{array}$ $\mathrm{here},\stackrel{\_}{x^{\prime }}=0\iff \sum _{i=1}^nx_i^{\prime }=0,\stackrel{\_}{y^{\prime }}=0\iff \sum _{i=1}^ny_i^{\prime }=0,\stackrel{\_}{z^{\prime }}=0\iff \sum _{i=1}^nz_i^{\prime }=0.$

If C=0, above formula becomes:

$\sum _{i=1}^n{\left({\mathrm{ax}}_i^{\prime }+{\mathrm{by}}_i^{\prime }-z_i^{\prime }\right)}^2=\left(\sum _{i=1}^nx_i^{\mathrm{\prime 2}}\right)a^2+\left(2\sum _{i=1}^nx_i^{\prime }y_i^{\prime }\right)\mathrm{ab}+\left(\sum _{i=1}^ny_i^{\mathrm{\prime 2}}\right)b^2-2\left(\sum _{i=1}^nx_i^{\prime }z_i^{\prime }\right)a-2\left(\sum _{i=1}^ny_i^{\prime }z_i^{\prime }\right)b+\sum _{i=1}^nz_i^{\mathrm{\prime 2}}.$

By

$f\left(a,b\right)=\left(\sum _{i=1}^nx_i^{\mathrm{\prime 2}}\right)a^2+\left(2\sum _{i=1}^nx_i^{\prime }y_i^{\prime }\right)\mathrm{ab}+\left(\sum _{i=1}^ny_i^{\mathrm{\prime 2}}\right)b^2-2\left(\sum _{i=1}^nx_i^{\prime }z_i^{\prime }\right)a-2\left(\sum _{i=1}^ny_i^{\prime }z_i^{\prime }\right)b$

referring to a function

$+\sum _{i=1}^nz_i^{\mathrm{\prime 2}},$

that is: f(a, b)=Aa²+2Bab+Cb²+Da+Eb+F.

By using a Cauchy inequality algorithm:

$\left(\sum _{i=1}^nx_i^{\mathrm{\prime 2}}\right)\left(\sum _{i=1}^ny_i^{\mathrm{\prime 2}}\right)\ge {\left(\sum _{i=1}^nx_i^{\prime }y_i^{\prime }\right)}^2\Rightarrow B^2-\mathrm{AC}\le 0.$

Generally speaking, B²−AC ≠0.

According to a formula: if f(x, y)=ax²+2bxy+cy²+dx+ey+f, here a>0, and b²−ac≦0, f(x, y) is minimal when (x, y)=(h,k). In this condition,

$h=\frac{\begin{array}{cc}-d& 2b\\ -e& 2c\end{array}}{\begin{array}{cc}2a& 2b\\ 2b& 2c\end{array}}=\frac{-\mathrm{dc}+\mathrm{de}}{2\left(\mathrm{ac}-b^2\right)},k=\frac{\begin{array}{cc}2a& -d\\ 2b& -e\end{array}}{\begin{array}{cc}2a& 2b\\ 2b& 2c\end{array}}=\frac{-\mathrm{ae}+\mathrm{bd}}{2\left(\mathrm{ac}-b^2\right)}.$

Calculating the function f(a, b)=Aa²+2Bab+Cb²+Da+Eb+F ,

$a=\frac{\begin{array}{cc}-D& 2B\\ -E& 2C\end{array}}{\begin{array}{cc}2A& 2B\\ 2B& 2C\end{array}}=\frac{-\mathrm{DC}+\mathrm{DE}}{2\left(\mathrm{AC}-B^2\right)},b=\frac{\begin{array}{cc}2A& -D\\ 2B& -E\end{array}}{\begin{array}{cc}2A& 2B\\ 2B& 2C\end{array}}=\frac{-\mathrm{AE}+\mathrm{BD}}{2\left(\mathrm{AC}-B^2\right)}.\mathrm{Here},A=\sum _{i=1}^nx_i^{\mathrm{\prime 2}},B=\sum _{i=1}^nx_i^{\prime }y_i^{\prime },C=\sum _{i=1}^ny_i^{\mathrm{\prime 2}},D=\sum _{i=1}^nx_i^{\prime }z_i^{\prime },E=\sum _{i=1}^ny_i^{\prime }z_i^{\prime },F=\sum _{i=1}^nz_i^{\mathrm{\prime 2}}.$

The parameters “a” and “b” in above formula further can be converted to

$a=\frac{-\mathrm{DC}+\mathrm{DE}}{2\left(\mathrm{AC}-B^2\right)}=\frac{-\left(\sum _{i=1}^nx_i^{\prime }z_i^{\prime }\right)\left(\sum _{i=1}^ny_i^{\mathrm{\prime 2}}\right)+\left(\sum _{i=1}^nx_i^{\prime }z_i^{\prime }\right)\left(\sum _{i=1}^ny_i^{\prime }z_i^{\prime }\right)}{2\left[\left(\sum _{i=1}^nx_i^{\mathrm{\prime 2}}\right)\left(\sum _{i=1}^ny_i^{\mathrm{\prime 2}}\right)-{\left(\sum _{i=1}^nx_i^{\prime }y_i^{\prime }\right)}^2\right]}=\frac{-r_{\mathrm{xz}}r_{\mathrm{yy}}+r_{\mathrm{xz}}r_{\mathrm{yz}}}{2\left(r_{\mathrm{xx}}r_{\mathrm{yy}}-r_{\mathrm{xy}}^2\right)},b=\frac{-\mathrm{AE}+\mathrm{BD}}{2\left(\mathrm{AC}-B^2\right)}=\frac{-\left(\sum _{i=1}^nx_i^{\mathrm{\prime 2}}\right)\left(\sum _{i=1}^ny_i^{\prime }z_i^{\prime }\right)+\left(\sum _{i=1}^nx_i^{\prime }y_i^{\prime }\right)\left(\sum _{i=1}^nx_i^{\prime }z_i^{\prime }\right)}{2\left[\left(\sum _{i=1}^nx_i^{\mathrm{\prime 2}}\right)\left(\sum _{i=1}^ny_i^{\mathrm{\prime 2}}\right)-{\left(\sum _{i=1}^nx_i^{\prime }y_i^{\prime }\right)}^2\right]}=\frac{-r_{\mathrm{xx}}r_{\mathrm{yz}}+r_{\mathrm{xy}}r_{\mathrm{xz}}}{2\left(r_{\mathrm{xx}}r_{\mathrm{yy}}-r_{\mathrm{xy}}^2\right)}.$

The calculated regression plan equation is:

$\frac{z-\stackrel{\_}{z}}{S_z}=a\left(\frac{x-\stackrel{\_}{x}}{S_x}\right)+b\left(\frac{y-\stackrel{\_}{y}}{S_y}\right).$

The calculating module 103 further calculates a distance between the regression plan and the center points of each of the electronic devices (the first electronic device 200 and the second electronic device 300). For example, if the coordinate of the center point of the first electronic device 200 is (x₀, y₀, z₀), and if the regression plan equation is expanded to form px+qy+rz+s=0, the calculating module 103 calculates the distance between the center point of the first electronic device 100 and the regression plan according to the formula:

$\frac{{\mathrm{px}}_0+{\mathrm{qy}}_0+{\mathrm{rz}}_0+s}{\sqrt{p^2+q^2+r^2}}.$

The determining module 104 determines which one of the electronic devices (for example the first electronic device 200 or the second electronic device 300) is intended as the target of the gesture by determining which one of the distances between the regression plan and the center points of the electronic devices is less than a preset value. If the determining module 104 determines that a distance between the regression plan and the center point of one electronic device is less than the preset value, the determining module determines that such electronic device is the object to be controlled by the gesture. If the determining module 104 determines that a distance between the regression plan and the center point of the electronic device is equal to or greater than the preset value, the determining module determines that such electronic device is not the intended target object to be controlled.

A method for determining a target electronic device controlled by a gesture is illustrated in FIG. 4. The method is provided by way of example, as there are a variety of ways to carry out the method. Each block shown in FIG. 4 represent one or more processes, methods, or subroutines carried out in the example method. Additionally, the illustrated order of blocks is by example only and the order of the blocks can be changed. The example method can begin at block S40.

At block S40, an acquiring module of a gesture control device acquires an image of a gesture from an image capturing device of each electronic device communicated with the gesture control device, and acquires an orientation of a gesture in the image of the gesture. In the exemplary embodiment, the gesture image can include a depth information. The gesture image can include a number of pictures, the orientation and motion in the gesture is acquired according to a position of the gesture in different pictures. The orientation of the gesture can indicate that the gesture of a user has a directivity. The orientation is determined as ended if the gesture is stopped for a preset time interval.

At block S41, a establishing module establishes a three dimensional coordinate system for the gesture image, and determines a coordinate of a central point of each of the electronic devices communicated with the gesture control device. In the exemplary embodiment, the coordinate of the center point of the electronic devices are predetermined according to a position of the image capturing device.

At block S42, the acquiring module determines a coordinate of a most extreme left-side horizontal position (left end) of the gesture in different depths and a coordinate of a most extreme right-side horizontal position (right end) of the gesture in different depths when the gesture is ended. For example, as shown in FIG. 3, a coordinate of a left end A₁ of the gesture in a depth Z₁ is (x′₁, y′₁, z₁), a coordinate of a right end B₁ of the gesture in the depth Z₁ is (x″₁, y″₁, z₁), a coordinate of the left end A₂ of the gesture in a depth Z₂ is (x′₂, y′₂, z₂), a coordinate of the right end B₂ of the gesture depth Z₂ is (x″₂, y″₂, z₂). Thus, a coordinate of the left end A_n of the gesture in a depth Z_n is (x′_n, y′_n, z_n), and a coordinate of the right end B_n of the gesture depth Z_n is (x″_n, y″_n, z_n).

At block S43, the calculating module calculates a coordinate of a center point between the left end and the right end of the gesture in each different depth.

At block S44, the calculating module calculates a regression plane equation according to the coordinate of the center points of the gesture image in different depths. The method for calculating the regression plane equation is described previously.

At block S45, the calculating module calculates a distance between the regression plan and the center points of each of the electronic devices. The method for calculating the distance is described previously.

At block S46, the determining module determines whether the distance between the regression plan and the center point of one of the electronic device is less than a preset value. If the determining module determines that the distance between the regression plan and the center point of one of the electronic device is less than the preset value, the procedure goes to block S47. Otherwise, the procedure is ended.

At block S47, the determining module determines the electronic device is a target controlled by the gesture if the distance between the regression plan and the center point of this electronic device is less than the preset value.

It is believed that the present embodiments and their advantages will be understood from the foregoing description, and it will be apparent that various changes may be made thereto without departing from the spirit and scope of the disclosure or sacrificing all of its material advantages, the examples hereinbefore described merely being exemplary embodiments of the present disclosure.

Claims

1. A gesture control device to communicate with at least two electronic devices, each of the electronic devices comprising an image capturing device, the gesture control device comprising: at least one processor; andat least one storage device storing one or more programs, when executed by the at least one processor, the one or more programs cause the at least one processor to: acquire an image of a gesture from the image capturing device of each electronic device communicated with the gesture control device and acquire an orientation of a gesture in the image of the gesture;establish a three dimensional coordinate system for the gesture image, and determine a coordinate of a central point of each of the electronic devices communicated with the gesture control device;determine a coordinate of a left end of the gesture in different depths and a coordinate of a right end of the gesture in different depths when the gesture is ended;calculate a coordinate of a center point between the left end and the right end of the gesture in each different depths; calculate a regression plane equation according to the coordinate of the center points of the gesture image in different depths;calculate a distance between the regression plan and the center points of each of the electronic devices; anddetermine which one of the electronic devices is intended as a target controlled by the gesture by determining which one of the distances between the regression plan and the center point of the electronic device is less than a preset value.

2. The gesture control device of claim 1, wherein the gesture image comprises a plurality of pictures, the orientation of the gesture is acquired according to a position of the gesture in different pictures.

3. The gesture control device of claim 1, wherein the orientation is determined as ended if the gesture is stopped for a preset time interval.

4. The gesture control device of claim 1, wherein a formula for calculating the distance between the regression plan and the center point of the electronic device is:

5. A gesture control method applied in a gesture control device, the gesture control device configured to communicate with at least two electronic devices, each of the electronic devices comprising an image capturing device, the gesture control method comprising: acquiring a gesture image from the image capturing device of each electronic device communicated with the gesture control device and acquiring an orientation of a gesture in the gesture image;establishing a three dimensional coordinate system for the gesture image, and determining a coordinate of a central point of each of the electronic devices communicated with the gesture control device;determining a coordinate of a left end of the gesture in different depths and a coordinate of a right end of the gesture in different depths when the gesture is ended;calculating a regression plane equation according to the coordinate of the center points of the gesture image in different depths;calculating a distance between the regression plan and the center points of each of the electronic devices; anddetermining which one of the electronic devices is intended as a target controlled by the gesture by determining which one of the distances between the regression plan and the center point of the electronic device is less than a preset value.

6. The gesture control method of claim 5, wherein the gesture image comprises a plurality of pictures, the orientation of the gesture is acquired according to a position of the gesture in different pictures.

7. The gesture control method of claim 5, wherein the orientation is determined as ended if the gesture is stopped for a preset time interval.

8. The gesture control method of claim 5, wherein a formula for calculating the distance between the regression plan and the center point of the electronic device is: