The present invention relates to a user interface display device which changes a spatial image in bidirectional relation to the motion of a hand (interactively) by moving the hand disposed around the spatial image.
Known schemes for displaying video pictures in space include a two-eye scheme, a multi-eye scheme, a spatial image scheme, a volume display scheme, a hologram scheme and the like. In recent years, there has been proposed a display device for displaying video pictures which allows a user to intuitively manipulate a two-dimensional video picture or a three-dimensional video picture (a spatial image) displayed in space with his or her hand, finger and the like, thereby achieving an interaction with the spatial image.
As a recognition input means (user interface) for a hand, finger and the like in such a display device, there has been proposed a system which forms a lattice of vertical and horizontal light beams in a sensing region (plane) by using a multiplicity of LEDs, lamps and the like to sense an input body that intercepts the lattice of light beams by means of a light receiving element and the like, thereby detecting the position or coordinates of the input body (hand) (with reference to Patent Literatures 1 and 2, for example).
However, the display device having the user interface which senses the interception of the lattice of light beams formed in the sensing region (plane) to detect the position or coordinates of the input body as described above has a frame used for installation of the aforementioned LEDs and the light receiving element. This frame is always disposed in a near position (closer to an operator) relative to the spatial image to come into the field of view of the operator. This makes the operator conscious of the frame as an obstacle, resulting in unnatural or unsmooth motion of the hand of the operator in some cases.
In view of the foregoing, it is therefore an object of the present invention to provide a user interface display device which does not include any structure serving as an obstacle to manipulation around a spatial image projected in space to achieve an interaction with the spatial image by using a hand of an operator in a natural manner.
To accomplish the aforementioned object, a user interface display device according to the present invention is a user interface display device for causing a video picture appearing on a display surface of a flat panel display to be image-formed in a spatial position spaced a predetermined distance apart therefrom by means of an optical panel having an image-forming function, thereby interactively controlling the video picture on the flat panel display in association with the motion of a hand positioned around this spatial image, wherein the optical panel is disposed in parallel with a virtual horizontal plane based on an operator so that the optical axis of the optical panel is orthogonal to the virtual horizontal plane, wherein the flat panel display is disposed in offset relation below the optical panel in such an attitude that the display surface is inclined at a predetermined angle with respect to the virtual horizontal plane and is positioned to face upward, and wherein a light source for projecting light toward the hand and one optical imaging means for imaging the reflection of the light from the hand are provided in a pair below or above the spatial image image-formed above the optical panel.
The present inventor has diligently made studies to solve the aforementioned problem, and has hit upon the idea of shooting a hand with a small number of cameras distant from a spatial image for the purpose of reducing psychological burdens on an operator during an input operation using the hand. The present inventor has focused attention on the motion (image) of the hand during the shooting with the cameras, and has made further studies. As a result, the present inventor has found that the motion of the hand serving as an input body is sufficiently detected with a simple configuration having a single camera by placing a display and an optical panel for image-forming the display on the display in a predetermined positional relationship to project the display (spatial image) appearing on the display in space above the aforementioned optical panel and by shooting the hand inserted around the aforementioned spatial image with an optical imaging means such as a camera disposed below or above the spatial image to identify the position or coordinates of the aforementioned hand based on this image. Hence, the present inventor has attained the present invention.
The present invention has been made based on the aforementioned findings. The user interface display device according to the present invention includes a flat panel display for displaying a video picture, and an optical panel such as a lens for projecting a video picture in space. The aforementioned optical panel is disposed in parallel with a virtual horizontal plane based on an operator so that the optical axis of the optical panel is orthogonal to the virtual horizontal plane. The aforementioned flat panel display is disposed below the aforementioned optical panel in such an attitude that the display surface thereof is inclined and is positioned to face upward. A light source and one optical imaging means are provided in a pair below or above the aforementioned optical panel. Thus, the user interface display device according to the present invention is a user-friendly display device which allows the operator to perform an interaction with the aforementioned spatial image by using the hand in a natural manner without being conscious of the system which detects the position or coordinates of the input body.
Further, the user interface display device according to the present invention, in which the single optical imaging means is sufficient, is advantageous in that the user interface display device for detecting the motion of the hand is provided with simple facilities at low costs. Further, the flexibility of the placement of the aforementioned optical imaging means (camera or the like) is improved, so that the camera or the like may be provided (hidden) in a position of which an operator is unconscious.
In particular, in the user interface display device according to the present invention wherein the light source and the optical imaging means are disposed in adjacent relation around the optical panel and wherein this optical imaging means images the reflection of light from the hand positioned above the optical panel, the aforementioned optical parts may be unitized together. This improves the flexibility of the placement of the aforementioned optical parts, and makes the user interface display device more simplified in configuration and lower in costs.
In particular, the user interface display device according to the present invention preferably comprises: a control means for controlling the light source, the optical imaging means and the flat panel display; a shape recognition means for acquiring the reflection of light projected from the light source toward the hand as a two-dimensional image to binarize the two-dimensional image by computation, thereby recognizing the shape of the hand; and a display updating means for comparing the positions of the hand before and after a predetermined time interval to update the video picture on the flat panel display to a video picture corresponding to the motion of the hand, based on the motion of the hand. Thus, the user interface display device according to the pre sent invention uses only the one optical imaging means to be able to detect the motion of a human hand with high sensitivity from the image analysis of the one optical imaging means. Also, an interaction between the spatial image and the hand of the operator is achieved by updating (changing) a video picture on the aforementioned flat panel display to a video picture corresponding to the motion of the aforementioned hand, based on the aforementioned detection.
Next, embodiments according to the present invention will now be described in detail with reference to the drawings. It should be noted that the present invention is not limited to the embodiments.
The user interface display device according to the present invention projects and displays a video picture appearing on a flat panel display D as a two-dimensional spatial image I′ before the eyes of an operator (not shown) positioned behind a hand H. The user interface display device according to the present invention includes an optical panel O disposed in parallel with a virtual horizontal plane P based on (the sensibility of) the aforementioned operator, and the flat panel display D disposed below a position distant from this optical panel O and having a display surface Da inclined at a predetermined angle θ and positioned to face upward. The aforementioned user interface display device further includes at least one light source L for projecting light toward the aforementioned hand H, and an optical imaging means (camera C) for imaging reflected light from the hand H. The at least one light source L and the optical imaging means (camera C) are disposed in a pair below the spatial image I′ projected by the aforementioned optical panel O. This is a characteristic of the user interface display device according to the present invention.
The configuration of the aforementioned user interface display device will be described in further detail. An optical part (image-forming optical element) capable of optically image-forming an image such as a lens including a Fresnel lens, a lenticular lens, a fly-eye lens and the like, a lens array, a mirror, a micromirror array, and a prism are used for the aforementioned optical panel O. Of these components, a micromirror array capable of forming a sharp spatial image I′ is preferably employed in the present embodiment. It should be noted that this optical panel O is disposed so that an optical axis Q thereof is orthogonal to the virtual horizontal plane P based on the operator, i.e. so that the front surface or the back surface of the panel O is parallel with the aforementioned virtual horizontal plane P.
A flat-shaped, self light emitting display such as a liquid crystal display (LCD), an organic EL display and a plasma display (PDP) is preferably employed as the aforementioned flat panel display D. This flat panel display D is disposed below a position distant from the optical panel O in such an attitude that the display surface Da thereof is inclined at the predetermined angle θ with respect to the aforementioned virtual horizontal plane P and is positioned to face upward. The angle θ of the aforementioned flat panel display D with respect to the virtual horizontal plane P is set at 10 to 85 degrees. A display which produces colors using reflected light by means of an external light source, and a cathode ray tube display may be also used as the aforementioned flat panel display D.
The single camera C described above includes a CMOS or CCD image sensor, and is disposed below the aforementioned spatial image I′, with its shooting direction oriented upward. The light source L is disposed on the same side of (in this example, below) the aforementioned spatial image I′ as the aforementioned camera C. Examples of the light source L used herein include an illuminator or a lamp which emits light having a range other than that of visible light (e.g., infrared light having a wavelength on the order of 700 to 1000 nm) so as not to hinder the field of vision of an operator who performs an input operation, such as an LED and a semiconductor laser (VCSEL). The aforementioned camera C and the light source L may be disposed in a pair (as a set) above the spatial image I′ (hand H). Examples of the optical imaging means for use in the user interface display device according to the present invention include various optical sensors including a photoelectric conversion element such as a photodiode, a phototransistor, a photo IC, a photo reflector and CdS, in addition to the camera C including the aforementioned CMOS image sensor or the CCD image sensor.
Next, a more specific embodiment of the user interface display device according to the present invention will be described.
In the user interface display device according to this embodiment, two plano-convex Fresnel lenses (an outside shape of 170 mm square, and a focal length of 305 mm) laid one on top of the other are used as the optical panel 1. A ¼-inch CMOS camera (NCM03-S manufactured by Asahi Electronics Laboratory Co., Ltd.) is used as a camera 2. Infrared LEDs (having a wavelength of 850 nm, and an output of 8 mW; LED851W manufactured by Thorlabs, Inc.) are used as light sources 3. A liquid crystal display (a 12-inch TFT display manufactured by Panasonic Corporation) is used as the flat panel display D.
Although not shown, a computer is provided in the aforementioned user interface display device. The computer has the functions of: a control means for controlling the aforementioned light sources 3, the camera 2 and the flat panel display D; a shape recognition means for acquiring the reflection of light projected from the aforementioned light sources 3 toward the hand H as a two-dimensional image (H′) to binarize this two-dimensional image by computation (H″), thereby recognizing the shape of the hand H; and a display updating means for comparing the positions of the aforementioned hand H before and after a predetermined time interval to update a video picture appearing on the aforementioned flat panel display D to a video picture corresponding to the motion of the aforementioned hand H, based on the motion of the hand H. The angle (angle of the display surface Da) θ of the aforementioned flat panel display D with respect to the optical panel 1 (virtual horizontal plane P) is set at 45 degrees in this example.
Next, a method for specifying the position of the hand H inserted around the spatial image I′ (into a sensing region) of the aforementioned user interface display device and for detecting the motion of the hand H will be described in a step-by-step manner.
For the specification of the position (coordinates) of the aforementioned hand H, light is initially projected from the light sources 3 disposed below the hand H toward the hand H, as shown in
Next, the aforementioned acquired two-dimensional image H′ is binarized, based on a threshold value. Thereafter, as shown in
The process of detecting the motion of the aforementioned hand H employs the aforementioned specified fingertip coordinates T. In the method therefor, the step [light projecting step] of projecting the aforementioned light, the step [imaging step] of acquiring the two-dimensional image and the step [coordinate specifying step] of calculating the fingertip coordinates T are initially repeated at determined time intervals. The fingertip coordinates T after the repetition are measured again [measuring step].
The distance and direction of the movement of the aforementioned fingertip coordinates T are calculated using the values of the fingertip coordinates T(Xm,Yn) before and after the lapse of the aforementioned repetition. Based on the result of calculation, a video picture on the flat panel display D, i.e. the spatial image I′, is updated to a video picture corresponding to the motion of the aforementioned hand H [display updating step].
For example, when the hand (input body) makes a horizontally sliding movement (H0→H1) as shown in
For the detection of the motion of the aforementioned hand H, an identification region in which the motion (T0→T2) of the aforementioned fingertip coordinates T is allocated on an area-by-area basis to four directions [X(+), X(−), Y(+) and Y(−)] may be defined on the virtual imaging plane P′ having the coordinate axes extending in the X and Y directions, as shown in
As described above, the user interface display device according to the first embodiment of the present invention with a simple and less costly configuration is capable of specifying the position or coordinates of the hand H. In addition, this user interface display device does not have any structure serving as an obstacle to manipulation around the spatial image I′ projected in space to achieve an interaction with the spatial image I′ by using the hand H of an operator in a natural manner.
Next, the user interface display device according to a second embodiment of the present invention will be described.
The user interface display device according to the present embodiment uses an optical panel (micromirror array 10) having an image-forming function to cause a video picture (image I) appearing on the display surface Da of the flat panel display D to be image-formed (spatial image I′) in a spatial position above the panel. The aforementioned flat panel display D is disposed in offset relation below the aforementioned micromirror array 10 in such an attitude that the display surface Da thereof is inclined at the predetermined angle θ with respect to the virtual horizontal plane P based on the operator and is positioned to face upward. The light sources 3 for projecting light toward the hand H of the operator and the optical imaging means (PSD designated by the reference numeral 4) for imaging the reflection of light from the hand H are disposed in a pair below (
The configuration of the user interface display device according to the aforementioned second embodiment differs from that of the user interface display device according to the first embodiment in that the micromirror array 10 having a multiplicity of protruding corner reflectors (unit optical elements) is used as the image-forming optical element capable of optically image-forming an image, and in that the PSD (Position Sensitive Detector) is used as the optical imaging means for imaging the reflection of light from the hand H.
The aforementioned micromirror array (protruding corner reflector array) 10 will be described in detail. As shown in
As shown in cross section in
The pair of light reflecting surfaces (first side surface 12a and the second side surface 12b) which form an edge 12c of each of the unit optical elements 12 are designed to face toward the eyepoint of the operator (toward the base of the hand H as seen in
As shown in
The light sources 3 and the PSD (4) are provided in positions which are below the spatial image I′ and around the micromirror array 10 in the examples of
A flat-shaped, self light emitting display such as a liquid crystal display (LCD), an organic EL display and a plasma display (PDP) is preferably employed as the aforementioned flat panel display D, as in the first embodiment. The flat panel display D is disposed below the micromirror array 10 in such an attitude that the display surface Da thereof is inclined at the predetermined angle θ (in this example, 10 to 85 degrees) with respect to the aforementioned virtual horizontal plane P and is positioned to face upward.
Examples of the light sources 3 used herein include illuminators or lamps which emit light having a range other than that of visible light (e.g., infrared light having a wavelength on the order of 700 to 1000 nm) so as not to hinder the field of vision of an operator who performs an input operation, such as LEDs and semiconductor lasers (VCSELs).
The method for specifying the position of the hand H inserted around the spatial image I′ (into the sensing region) and for detecting the motion of the hand H in the user interface display device having the aforementioned configuration according to the second embodiment is performed by steps similar to those of the first embodiment (with reference to
The user interface display device according to the aforementioned second embodiment with a simple and less costly configuration is capable of specifying the position or coordinates of the hand H. In addition, this user interface display device does not have any structure serving as an obstacle to manipulation around the spatial image I′ projected in space to achieve an interaction with the spatial image I′ by using the hand H of an operator in a natural manner.
Next, the user interface display device according to a third embodiment of the present invention will be described.
The user interface display device according to the present embodiment uses an optical panel (micromirror arrays 20, 30, 40 and 50) having an image-forming function to cause a video picture (image I) appearing on the display surface Da of the flat panel display D to be image-formed (spatial image I′) in a spatial position above the panel. The aforementioned flat panel display D is disposed in offset relation below the micromirror array 20 (30, 40 and 50) in such an attitude that the display surface Da thereof is inclined at the predetermined angle θ with respect to the virtual horizontal plane P based on the operator and is positioned to face upward. The light sources 3 for projecting light toward the hand H of the operator and the optical imaging means (PSD designated by the reference numeral 4) for imaging the reflection of light from the hand H are disposed in a pair below (
The configuration of the user interface display device according to the aforementioned third embodiment differs from that of the user interface display device according to the aforementioned second embodiment in that one of the micromirror arrays 20, 30, 40 and 50 including one or two optical elements obtained by forming a plurality of parallel linear grooves spaced at predetermined intervals by dicing using a rotary blade on a surface of a flat-shaped transparent substrate is used as the image-forming optical element (optical panel) capable of optically image-forming an image.
In these micromirror arrays 20, 30, 40 and 50, the two optical elements (substrates) having the plurality of parallel grooves formed on the front surfaces thereof are laid one on top of the other, with one of the optical elements rotated 90 degrees (
The light-reflective wall surfaces of the first group of parallel grooves of the substrate and the light-reflective wall surfaces of the second group of parallel grooves of the substrate which constitute the aforementioned corner reflectors are what is called in “skew” relation as seen three-dimensionally. It is also advantageous that the adjustment of the optical performance of the optical elements, such as an increase in aspect ratio [height (length as measured in the direction of the thickness of the substrate)/width (width as measured in a horizontal direction of the substrate)] of the light reflecting surfaces of the aforementioned corner reflectors, is made relatively easily because the aforementioned parallel grooves and the light-reflective wall surfaces thereof are formed by dicing using a rotary blade.
The structures of the aforementioned respective micromirror arrays will be described individually in further detail. Optical elements (21 and 21′) constituting the micromirror array 20 shown in
Similarly, the micromirror array 30 shown in
Further, the micromirror array 40 shown in
The micromirror array 50 shown in
The configurations and arrangement of the light sources 3, the PSD (4), the flat panel display D and the like applied in the user interface display device according to the third embodiment including the aforementioned micromirror arrays 20, 30, 40 and 50 are similar to those of the aforementioned second embodiment. The method for specifying the position of the hand H inserted around the spatial image I′ (into the sensing region) and for detecting the motion of the hand H in the third embodiment is performed by steps similar to those of the first embodiment (with reference to
The user interface display device according to the aforementioned third embodiment with a simple and less costly configuration is capable of specifying the position or coordinates of the hand H. In addition, this user interface display device does not have any structure serving as an obstacle to manipulation around the spatial image I′ projected in space to produce the effect of achieving an interaction with the spatial image I′ by using the hand H of an operator in a natural manner. Further, the user interface display device according to the aforementioned third embodiment is advantageous in that the costs of the entire device are reduced because the micromirror arrays (20, 30, 40 and 50) used therein are less costly.
Although specific forms in the present invention have been described in the aforementioned examples, the aforementioned examples should be considered as merely illustrative and not restrictive. It is contemplated that various modifications evident to those skilled in the art could be made without departing from the scope of the present invention.
The user interface display device according to the present invention is capable of remotely recognizing and detecting the position or coordinates of a human hand by means of the single optical imaging means. This allows an operator to intuitively manipulate a spatial image without being conscious of the presence of an input system.
Number | Date | Country | Kind |
---|---|---|---|
2011-194937 | Sep 2011 | JP | national |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/JP2012/071455 | 8/24/2012 | WO | 00 | 3/5/2014 |