Patent Application
20190163284
The present invention relates to an apparatus and method for a user to remotely control an electronic device, and in particular, to a camera-based remote control apparatus and method that may remotely control the operation of an electronic device using a virtual touch through a motion (gesture) of a user's fingertip with respect to the center of a camera.
Smartphone or other touch panel-equipped electronic devices are recently commonplace. Touch panel control eliminates the need of displaying a pointer on the screen as contrasted with existing mouse control for computers or others. To control a touch-controllable electronic device, the user may simply move and touch his finger on an icon on the screen without moving a cursor that is conventionally required for computers or other electronic devices. As such, touch panel technology enables quick control thanks to omission of cursor control.
Despite such advantageous, however, touch panel techniques require the user to directly touch the screen, rendering it unavailable for remote use. For remote control, touch-controllable electronic devices thus resort to other control devices, such as remote controllers.
Korean Patent Application Publication No. 2010-0129629 published on Dec. 9, 2010 (hereinafter, document 1) discloses a remote electronic device controller capable of generating a point at a precise position. According to the technique, two cameras are used to capture an image ahead of the display and generate a point at the spot where the extension line of the user's eye and finger meets the screen. However, such is very inconvenient in that a pointer should be inevitably generated by a pointer controller as a preliminary step for controlling the electronic device before the user's gesture is compared with a pre-stored pattern for specific operation.
Further, according to document 1, in order to detect the spot where the extension line of the user's eye and finger meets the screen, three-dimensional (3D) spatial coordinates are produced through 3D coordinates calculation. In this case, each of the user's eye, finger, and selection point for operation control should be detected through images captured by camera at different positions. Thus, it is difficult to precisely detect selection points, and detection of more precise selection points may increase the load of calculation.
Moreover, remote electronic device control by the reference technique is limited to electronic devices equipped with a large-screen display, e.g., large enough for control icons to be noticeable to the user positioned away therefrom. Thus, remote control is possible only on the front surface of the electronic device, and the user should be positioned in front of the front of the electronic device for control.
Further, operation control of an electronic device is performed by communication with a camera controller, and for remote control, the electronic device should be located at a fixed position. This limits the application to only stationary electronic devices.
Korean Patent Application Publication No. 10-2010-0129629, titled “Method for controlling operation of electronic device by motion detecting and electronic device adopting the same.”
Korean Patent No. 10-1235432, titled “Remote control apparatus and method using 3D modeled virtual touch of electronic device.”
The present invention has been conceived to address the above issues and aims to provide a camera-based, virtual touch-adopted remote control apparatus and method that may control the operation of an electronic device using a remote virtual touch through a motion (gesture) of a user's finger by performing a dominant eye setting using a camera whose center is in line with an extension line of the user's eye and finger.
Another object of the present invention is to provide a remote control apparatus and method capable of remotely controlling an electronic device located anywhere in a home using a virtual touch through a motion (or gesture) of a user's fingertip regardless of motion of the electronic device or position and direction between the user and electronic device.
Other objects of the present invention are not limited to the foregoing objects, and other objects will be apparent to one of ordinary skill in the art from the following detailed description.
To achieve the above objects, according to the present invention, a remote control device using a camera-based virtual touch comprises an image obtaining unit obtaining an image of a user's body through image capturing; a position calculator calculating coordinate data indicating positions of first coordinates (an eye) and second coordinates (a finger or pointer) of the user's body using the image received from the image obtaining unit—and a matching identifier identifying whether the positions of the first coordinates and the second coordinates of the user body identified based on the coordinate data produced by the position calculator with respect to a camera lens match an extension line, and if matching, determining that the user has a setting intention.
Preferably, the coordinate data produced by the position calculator is two-dimensional (20) or three-dimensional (3D) data and is coordinate data at a position viewed from the camera lens of the image obtaining unit.
Preferably, the remote control device further comprises a virtual touch setting unit determining that the user has the setting intention if the matching is identified by the matching identifier, determining whether there is a variation in the coordinate data of the first and second coordinates, and performing an operation set to correspond to varied contact coordinate data; and a driving controller, if a command code generated from the virtual touch setting unit is inputted, converting the command code into a control command signal for controlling driving of a corresponding electronic device, and sending the control command signal to the electronic device.
Preferably, the virtual touch setting unit includes a gesture detector, if the matching is identified by the matching identifier, detecting the variation in the coordinate data of the first and second coordinates obtained by the image obtaining unit and detecting the user's finger motion (gesture), an electronic device detector comparing electronic device information previously stored in an electronic device information database (DB) based on the gesture detected by the gesture detector to detect the corresponding electronic device, an electronic device setting unit setting a control signal according to control of driving of the electronic device detected based on the gesture detected by the gesture detector, and a virtual touch processor generating a command code for controlling driving of the electronic device based on the control signal set by the electronic device setting unit.
Preferably, the electronic device information DB stores product information including the manufacturer and model name of electronic devices provided in the home, protocol information applied to the control of operation of each device, and control method information per device.
Preferably, the control command signal sent from the electronic device driving controller is sent through a short-range wireless communication scheme including an infrared (IR) signal, Wi-Fi, or Bluetooth®.
Preferably, a remote control method using a camera-based virtual touch, comprising the steps of (A) calculating coordinate data using an image captured and received to obtain positions of first coordinates (an eye) and second coordinates (a finger or pointer) of a user's body.
(B) identifying whether the obtained first and second coordinates of the user's body match an extension line with respect to a camera lens,
(C) if matching, detecting a variation in the obtained coordinate data of the first and second coordinates to detect the users motion (gesture).
(D) if the matching is identified, immediately determining that the user has a setting intention to perform a dominant eye setting distinctively for the dominant eye setting.
Preferably, a remote control method using a camera-based virtual touch, comprising the steps of (a) calculating coordinate data using an image captured and received to obtain positions of first coordinates (an eye) and second coordinates (a finger or pointer) of a user's body,
(b) identifying whether the obtained first and second coordinates of the user's body match an extension line with respect to a camera lens,
(c) if matching, detecting a variation in the obtained coordinate data of the first and second coordinates to detect the user's motion (gesture).
(d) generating a control command signal for controlling driving of an electronic device based on the detected motion of a finger and sending the control command signal to the electronic device.
Preferably, the coordinate data is two-dimensional (2D) or three-dimensional (3D) data and is coordinate data at a position viewed from the camera lens.
Preferably, step (d) includes the steps of (d1) comparing electronic device information previously stored in an electronic device information database (DB) based on the detected gesture by performing the dominant eye setting and a consecutive operation to detect a corresponding electronic device and to set an electronic device to be controlled for operation through a virtual touch,
(d2) setting a control signal according to control of driving of the electronic device set through the user's motion (gesture) detected based on the 2D or 3D coordinate data,
(d3) converting into a control command signal for controlling of driving of the electronic device based on the set control signal and generating a control command code for a corresponding electronic device,
(d) converting the generated command code into a control command signal for controlling operation of the electronic device and sending the control command signal to the electronic device.
Preferably, step (d2) includes setting the second coordinates (finger or pointer) of the user moving in at least one direction of up, down, left, and right with respect to the extension line of the camera lens, the first coordinates (eye), and the second coordinates (finger or pointer) as a control signal indicating selection or control of a particular icon displayed on a display unit or a motion of a joystick, setting the second coordinates of the user moving along a straight line from left to right (or from right to left) or from down to up (or from up to down) with respect to the extension line of the camera lens, the first coordinates (eye), and the second coordinates (finger or pointer) as a control signal for controlling a left-to-right turn-on/off switch or up-to-down turn-on/off switch, and setting the second coordinates of the user moving along a curved line from down to up (or from up to down) with respect to the extension line of the camera lens, the first coordinates (eye), and the second coordinates (finger or pointer) as a control signal for controlling a degree of rotation for adjusting a magnitude through a rotating wheel.
Preferably, the control command signal sent in step (d4) is sent through a short-range wireless communication scheme including an infrared (IR) signal, Wi-Fi, Bluetooth, or ZigBee.
As described above, according to the present invention, the camera-based remote control apparatus and method using a virtual touch has the following effects.
First, a dominant eye setting is performed using a camera whose center is in line and meets an extension line of an eye and finger, allowing for easy and precise locating a touch point in line with the extension line of the eye and finger. Thus, a motion (gesture) of the users fingertip may be detected precisely and quickly.
Second, a particular electronic device may be rapidly controlled in such a manner as a virtual touch on the camera rather than virtually touching a control icon displayed on the display device using a pointer through the display device. Thus, the present invention may be applicable to all other electronic devices than those with no display device.
Third, research efforts for use of virtual touch so far have been limited to only applications using display devices. The present invention may provide a brand-new touch recognition scheme enabling precise control of electronic devices and use for virtual touch devices not equipped with a display.
Fourth, a remote virtual touch may be provided for devices requiring simple control through switching, such as curtains, lighting, or automatic door locks, as well as electronic devices to perform control corresponding to the virtual touch, allowing for various applications.
Other objectives, features, and advantages of the present invention will be apparent by a detailed description of embodiments taken in conjunction with the accompanying drawings.
A remote control apparatus and method using a camera-based virtual touch according to the present invention are described below in preferred embodiments thereof. However, the present disclosure is not limited to the embodiments disclosed herein, and various changes may be made thereto. The embodiments disclosed herein are provided only to inform one of ordinary skilled in the art of the category of the present disclosure. The present disclosure is defined only by the appended claims. The configurations disclosed in the specification and the drawings are mere examples and do not overall represent the technical spirit of the present invention. Therefore, various changes may be made thereto, and equivalents thereof also belong to the scope of the present invention.
As shown in
The image obtaining unit 110 may be a camera module, and the image obtaining unit 110 may include an image sensor, such as CCD or CMOS, which may detect an image and convert the detected image into an electrical image signal.
The position calculator 120 produces coordinate data indicating the position of a user's body, i.e., first coordinates (eye) and second coordinates (finger and pointer) using an image received from the image obtaining unit 110. The produced coordinate data is two dimensional (2D) or three-dimensional (3D) coordinate data and indicates coordinate data at a position viewed from a camera lens of the image obtaining unit 110.
Further, a finger among the user's body portions used to produce the coordinate data is a portion enabling accurate and sophisticate control. Accordingly, according to the present invention, it is very effective to use a tip of a thumb and/or index finger as first coordinate data. In the same point of view, the fingertip playing a role as the first coordinate data may be replaced with a pointer (e.g., a pen) with a sharp tip which is grasped by fingers. According to the present invention, only a center point of an eye of the user is used as second coordinate data. For example, if the user places and views his index finger before his eye, he will see the finger double. This happens because the user's eyes see different shapes of the index finger (due to an angular difference between both eyes of the user). However, the user will clearly see the index finger at only one eye. Further, if looking at the index finger intentionally with only one eye, the user may clearly see the index finger even not closing the other eye. This is why aiming is done with one eye closed in shooting, archery, or other sports requiring a high-level accuracy. The present invention adopts such principle that viewing a finger (first coordinates) or pointer tip (second coordinates) with only one eye allows its shape to be shown more clearly.
The matching identifier 130 identifies whether the position of the user's body portions, i.e., eye and finger, identified based on the coordinate data produced by the position calculator 120, matches an extension line with respect to the center of the camera lens. Such identification by the matching identifier 130 as to whether the position of the eye and finger image-captured by the camera matches the extension line may vest more clarity in analysis of captured images.
If such matching is identified by the matching identifier 130, the virtual touch setting unit 140 determines that the user has a setting intention, and the virtual touch setting unit 140 determines whether the coordinate data of the eye and finger is varied and performs a set operation to correspond to varied contact coordinate data.
To that end, the virtual touch setting unit 140 includes a gesture detector 142 that, when the matching is identified by the matching identifier 130, detects a variation in the coordinate data of the eye and finger obtained from the image obtaining unit 110 and detects a motion (gesture) of the user's finger, an electronic device detector 141 that performs comparison with electronic device information previously stored in an electronic device information database (DB) 145 based on the gesture detected by the gesture detector 142 to detect a corresponding electronic device, an electronic device setting unit 143 that sets a control signal according to control of an operation of the electronic device detected based on the gesture detected by the gesture detector 142, and a virtual touch processor 144 that generates a command code for controlling the operation of a corresponding electronic device 200 based on the control signal set by the electronic device setting unit 143.
The electronic device information DB 145 stores product information including the manufacturer and model name of electronic devices provided in the home, protocol information applied to the control of operation of each device, and control method information per device. The control method information refers to information, such as a motion of the user's finger and position of a touch, for controlling or adjusting the power of the electronic device, illuminance, sound volume, or temperature by a virtual touch. For reference, the information stored in the electronic device information DB 150 may be frequently updated by the user after purchase of the electronic device, and such update may be performed by, e.g., accessing a website for the electronic device to download information on the electronic device or downloading information on the electronic device through, e.g., a transmitter included in the electronic device. Various embodiments may be possible within the technical spirit of the present invention.
Accordingly, as a dominant eye setting is performed using the camera through the virtual touch setting unit 140, the position of the eye and finger in line with the extension line and varied may be easily computed, and thus, the motion (gesture) of the user's finger tip may be detected more rapidly and accurately.
Upon input of the command code generated by the virtual touch setting unit 140, the electronic device operation controller 150 converts the command code into a control command signal for controlling the operation of the electronic device 200 and sends the control command signal to the electronic device 200. The control command signal may be preferably sent as an infra red (IR) signal as by a conventional remote controller, but is not limited thereto. For example, the control command, signal may be sent through wireless fidelity (Wi-Fi), Bluetooth, Zigbee, or other short-range wireless communication schemes.
The electronic device 200 which is under the remote control according to the present invention may include various electronic devices configuring a home network, such as a lamp, speaker, air conditioner, electric fan, blind, window, curtain, heater, gas oven, or microwave oven, video player, camera, or CCTV.
As such, the remote control apparatus 100 is separately installed from an electronic device that is subjected to remote control, and thus, even when the electronic device is on the move, computation of varied 3D spatial coordinates of the electronic device may be seamlessly done and may be applied to various mobile electronic devices.
Operation of a remote control apparatus using a camera-based virtual touch according to the present invention as configured above is described below in detail with reference to the accompanying drawings. The same reference numeral as that shown in
Referring to
Subsequently, the matching identifier 130 is used to identify whether the position of the user's body portion, e.g., a finger, identified based on the coordinate data produced by the position calculator 120 with respect to the camera lens 30, matches a valid area with respect to the user's eye (S20). The matching may include when the user's eye 10 is in line with the user's hand 20 or when the user's hand 20 is positioned adjacent to the user's eye 20 as shown in
If the matching is identified to be achieved (820), a variation in the coordinate data of the eye and finger image-captured by a camera, which is the image obtaining unit 110, is detected to detect the user's finger motion (gesture) (S30).
It is determined whether the detected finger motion is a gesture for dominant eye setting or a gesture for controlling an electronic device (S40). In step S40, only whether the finger motion is a gesture for dominant eye setting or not may be determined excluding the determination as to whether the finger motion is a gesture for controlling an electronic device. Here, the gesture for dominant eye setting includes a case where the user makes no finger motion.
If the finger motion is a gesture for dominant eye setting, a dominant eye setting is performed (S50). As such, according to the present invention, matching or not is identified through an extension line of the eye and finger image-captured by the camera, and a dominant eye setting is carried out. Thus, unlike in the conventional method where a finger motion (gesture) is detected through 3D coordinates to perform a dominant eye setting in identifying matching or not through an obtained image, easier or clearer analysis of captured images may be achieved along with a quick dominant eye setting. As such, when the present invention is used for the purpose of dominant eye setting for controlling various electronic devices, the whole process may end in step S50.
Subsequently, comparison may be made with electronic device information previously stored in the electronic device information DB 145 based on the gesture detected in an operation subsequent to performing a dominant eye setting to thereby detect a corresponding electronic device (S60), and an electronic device 200 is set whose operation is to be controlled through a virtual touch (S70).
Set is a control signal as per control of the operation of the electronic device detected based on the detected gesture (S80). There may be various embodiments of methods for controlling the operation of the electronic device according to the detected gesture.
According to a first embodiment, an example may be taken in which the user's finger moves up, down, left, or right with respect to the extension line of the camera lens 30, eye 10, and finger 20, as shown in
In the first embodiment, as shown in
According to a second embodiment, an example may be taken in which the user's finger moves left to right (or right to left) along a straight line with respect to the extension line of the camera lens 30, eye 10, and finger 20, as shown in
In the second embodiment, as shown in
According to a third embodiment, an example may be taken in which the user's finger moves down to up (or up to down) along a straight line with respect to the extension line of the camera lens 30, eye 10, and finger 20, as shown in
In the third embodiment, as shown in
According to a fourth embodiment, an example may be taken in which the user's finger moves down to up (or up to down) along a curved line with respect to the extension line of the camera lens 30, eye 10, and finger 20, as shown in
In the fourth embodiment, as shown in
Use of the first to fourth embodiments enables selection or control of a particular icon displayed on a display of an electronic device, such as a TV, as shown in
As such, the operation of an electronic device may be easily remote-controlled using a virtual touch through the motion (gesture) of the user's finger tip based on 2D or 3D coordinate data in the position where the user's finger is viewed from the camera lens of the image obtaining unit 110 with respect to the extension line of the camera lens 30, eye 10, and finger 20.
Conversion into a control command signal for controlling the operation of the electronic device is made based, on a control signal set through the motion (gesture) of the user's finger tip based on the 2D or 3D coordinate data, thereby generating a control command signal for a corresponding electronic device (S90).
The generated command code is converted into a control command signal for controlling the operation of the corresponding electronic device and is sent to the electronic device (S100). The control command signal may be preferably sent as an infra red (IR) signal as by a conventional remote controller, but is not limited thereto. For example, the control command signal may be sent through wireless fidelity (Wi-Fi), Bluetooth, Zigbee, or other short-range wireless communication schemes.
It should be noted that although the technical spirit of the present invention has been described in connection with preferred embodiments thereof, this is merely for description purposes and should not be interpreted as limiting. It should be appreciated by one of ordinary skill in the art that various changes may be made thereto without departing from the technical spirit of the present invention. Accordingly, the scope of the present invention should be determined by the technical spirit of the invention as claimed in the claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 10-2014-0020985 | Feb 2014 | KR | national |
This application is a continuation of U.S. application Ser. No. 15/120,664 filed on Aug. 22, 2016, which is a U.S. national phase of PCT/KR2015/001703 filed on Feb. 23, 2015 which claims priority to Korean Patent Application No. 10-2014-0020985 filed on Feb. 22, 2014, the entire contents of which are herein incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| Parent | 15120664 | Aug 2016 | US |
| Child | 16260661 | US |
