Patent Grant
11582441
This application is the U.S. National Phase under 35 U.S.C. § 371 of International Application No. PCT/JP2018/044569, filed on Dec. 4, 2018, the entire contents are hereby incorporated by reference.
The present invention relates to a head mounted display apparatus, and particularly relates to a technology effective for expressing the faithful occlusion in the AR (Augmented Reality) display.
In recent years, various services using AR have been proposed. The AR is a technology for displaying an image created by a computer, a portable computer, or the like so as to be superimposed on an entire screen or a part of a real image.
In a three-dimensional space, there is a front-back relationship in addition to an up-down relationship and a left-right relationship. Therefore, a state in which an object on the far side is hidden behind an object on the near side to be invisible, that is, the occlusion occurs.
In the AR display, in order to provide the user with an AR image that does not give a sense of discomfort, it is important to faithfully express the above-mentioned occlusion by processing a three-dimensional video or the like.
As an AR image processing technology that takes this kind of occlusion into consideration, for example, a technology in which a CG (Computer Graphics) image region hidden behind a real object when expressing the occlusion is cut out in an elliptical shape to perform occlusion processing has been known (see, for example, Patent Document 1).
Patent Document 1: US Patent Application Publication No. 2012/206452
In the occlusion processing described above, for example, in the case of single vision that displays the AR on a smartphone or tablet, it is only necessary to simply express the occlusion in accordance with the distance relationship between the real object and the CG image.
However, in the case of a head mounted display apparatus using binocular vision such as AR glasses, in order to faithfully express the occlusion in AR display, the parallax and convergence angle of both eyes need to be taken into consideration, and there is a problem that the AR image becomes unnatural if only the occlusion processing for single vision is simply applied.
Further, in the processing technology for the occlusion in Patent Document 1 described above, since the processing in which the difference in vision due to the binocular parallax, convergence angle, and the like of the user is taken into consideration is not performed and the CG image region is cut out in an elliptical shape instead of cutting out the CG image region hidden behind the real object along the outline of the real object, there is a problem that the display in the border between the real object and the CG image becomes unnatural.
An object of the present invention is to provide a technology capable of faithfully expressing the occlusion even in the binocular vision in the AR display by a head mounted display apparatus or the like.
The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.
The outline of the typical invention disclosed in this application will be briefly described as follows.
Namely, a typical head mounted display apparatus includes a first lens, a second lens, a first camera, a second camera, and an information processor. The first lens displays a CG image for a right eye. The second lens displays a CG image for a left eye. The first camera captures an image for the right eye. The second camera captures an image for the left eye.
The information processor generates the CG image for the right eye in which occlusion at the time of seeing by the right eye is expressed and the CG image for the left eye in which occlusion at the time of seeing by the left eye is expressed, based on the images captured by the first camera and the second camera, projects the generated CG image for the right eye onto the first lens, and projects the generated CG image for the left eye onto the second lens.
Further, a center of a lens of the first camera is provided at the same position as a center of the first lens (center of a pupil of a wearer). A center of a lens of the second camera is provided at the same position as a center of the second lens (center of a pupil of a wearer).
In particular, the information processor includes a first information generator, a second information generator, a first shielded region calculator, a second shielded region calculator, an image generator, and a display unit.
The first information generator generates occlusion information indicating a shielding relationship between the CG image to be displayed on the first lens and a real environment based on the images captured by the first camera and the second camera. The second information generator generates occlusion information indicating a shielding relationship between the CG image to be displayed on the second lens and the real environment based on the images captured by the first camera and the second camera.
The first shielded region calculator calculates a shielded region in which the CG image displayed on the first lens is shielded by an object when the CG image is seen by the right eye, based on the occlusion information generated by the first information generator. The second shielded region calculator calculates a shielded region in which the CG image displayed on the second lens is shielded by the object when the CG image is seen by the left eye, based on the occlusion information generated by the second information generator.
The image generator generates the CG image for the right eye in which the shielded region calculated by the first shielded region calculator is not displayed and the CG image for the left eye in which the shielded region calculated by the second shielded region calculator is not displayed, based on CG image generation data for generating the CG image. The display unit projects the CG image for the right eye and the CG image for the left eye generated by the image generator onto the first lens and the second lens, respectively.
The effect obtained by the typical invention disclosed in this application will be briefly described as follows.
It is possible to faithfully express the occlusion in the AR display of the binocular vision.
The same members are denoted by the same reference signs in principle throughout the drawings for describing the embodiment, and the repetitive description thereof will be omitted.
The embodiment will be described in detail below.
<Configuration Example of Head Mounted Display Apparatus>
As shown in
The lens 12 which is the first lens and the lens 13 which is the second lens are each fixed to a rim of the chassis 11. The lenses 12 and 13 fixed to the rim are provided so as to respectively correspond to the left and right eyes of the user who uses the head mounted display apparatus 10.
The camera 14 which is the first camera is provided on the upper part of the rim which fixes the lens 12, and the camera 15 which is the second camera is provided on the upper part of the rim which fixes the lens 13. These cameras 14 and 15 are, for example, stereo cameras. The stereo cameras capture an object by using the parallax of two camera.
The camera 14 is a camera corresponding to the right eye of the user, and the camera 15 is a camera corresponding to the left eye of the user. These cameras 14 and 15 take images, respectively.
Also, the center of the lens of the camera 14 is provided at a position substantially the same as the center of the lens 12. In other words, the center of the lens of the camera 14 is substantially the same as the center of the pupil of the right eye of the user.
Similarly, the center of the lens of the camera 15 is provided at a position substantially the same as the center of the lens 13. In other words, the center of the lens of the camera 15 is substantially the same as the center of the pupil of the right eye of the user.
As shown in
The operation input unit 17 is a user interface including, for example, a touch sensor. The touch sensor is composed of, for example, a capacitive touch panel that electrostatically detects the position of a user's finger or the like in contact with the touch sensor.
The controller 18 is responsible for the control in the head mounted display apparatus 10. The information processor 19 generates a CG image and displays it on the lenses 12 and 13. The information processor 19 is provided in, for example, a bridge portion of the chassis 11. The arrangement of the information processor 16 is not particularly limited, and the information processor 16 may be provided in, for example, a temple portion of the chassis 11.
The CG image generated by the information processor 16 is projected onto the lenses 12 and 13. The projected CG images are magnified and displayed by the lenses 12 and 13, respectively. Then, the CG images displayed on the lenses 12 and 13 are superimposed on the real image to form an AR image.
The communication interface 30 performs wireless communication of the information or the like by, for example, Bluetooth (registered trademark) or an Internet line.
<Configuration Example of Information Processor>
As shown in
The depth information generator 20 that constitutes a first information generator and a second information generator calculates and stores the distance to the object, the convergence angle, and the like. The depth information generator 20 includes a parallax matching unit 25 and a depth information storage 26.
The parallax matching unit 25 calculates the object distance and the convergence angle of all the objects in the images taken by the cameras 14 and 15 in
The object distance is the distance from the cameras 14 and 15 to the object, in other words, the depth information of the object. The convergence angle is an angle formed by the sight lines from both eyes at the object to be seen. This convergence angle can be calculated from, for example, the object distance and the position information of the object on the image.
The depth information storage 26 is composed of a semiconductor memory exemplified by, for example, a flash memory, and stores depth information for a right eye (right-eye depth information) and depth information for a left eye (left-eye depth information). The right-eye depth information is the information obtained by adding the object distance and the convergence angle calculated by the parallax matching unit 25 to the image taken by the camera 14. The left-eye depth information is the information obtained by adding the object distance and the convergence angle calculated by the parallax matching unit 25 to the image taken by the camera 15.
The shielded region calculator 21 that constitutes a first shielded region calculator and a second shielded region calculator calculates a shielded region of the CG image for a right eye (right-eye CG image) and a shielded region of the CG image for a left eye (left-eye CG image) based on the right-eye depth information and the left-eye depth information stored in the depth information storage 26 and the distance information of the CG image to be displayed. The shielded region is the region in which a part of the CG image is not displayed in order to express the occlusion.
The AR image generator 22 which is an image generator generates the right-eye CG image and the left-eye CG image based on the shielded region of the right-eye CG image and the shielded region of the left-eye CG image calculated by the shielded region calculator 21 and the image generation data.
The distance information of the CG image described above is the information indicating the distance and position of the CG image to be displayed. The image generation data is the data necessary for generating the CG image to be displayed, for example, data such as the shape and color scheme of the CG image.
The distance information and image generation data are provided by, for example, an application input from the outside. The application is acquired through, for example, the communication interface 30 in the control processor 16. Alternatively, it is also possible to provide a memory (not shown) in the information processor 19 and store the application input from the outside in the memory in advance.
The AR display unit 23 which is the display is an optical system that projects and draws the right-eye CG image and the left-eye CG image generated by the AR image generator 22 on the lenses 12 and 13, respectively.
<Operation Example of Head Mounted Display Apparatus>
Next, the operation of the head mounted display apparatus 10 will be described.
First, the cameras 14 and 15 take images. The images taken by the cameras 14 and 15 are input to the parallax matching unit 25 in the depth information generator 20. The parallax matching unit 25 calculates the object distance and the convergence angle from the images taken by the cameras 14 and 15.
The object distance is obtained by calculating the difference in position between the images taken by the cameras 14 and 15 by means of triangulation or the like by the parallax matching unit 25. Alternatively, instead of calculating the distance to the object based on the images of the cameras 14 and 15 constituting the stereo camera, for example, it is also possible to newly provide a distance sensor (not shown) or the like in the information processor 19, and measure the distance to the object by the distance sensor.
The convergence angle is calculated by the parallax matching unit 25 based on the calculated object distance and the position of the object. These calculations are performed for each object in the images taken by the cameras 14 and 15.
Thereafter, the parallax matching unit 25 adds the object distance and the convergence angle calculated from the image of the camera 14 to the image taken by the camera 14. Similarly, the parallax matching unit 25 adds the object distance and the convergence angle calculated from the image of the camera 15 to the image taken by the camera 15.
Then, the parallax matching unit 25 stores the image of the camera 14 to which the distance and the convergence angle have been added in the depth information storage 26 as the right-eye depth information described above, and stores the image of the camera 15 to which the distance and the convergence angle have been added in the depth information storage 26 as the left-eye depth information described above.
The shielded region calculator 21 acquires the distance to the object and the convergence angle from the right-eye depth information and the left-eye depth information stored in the depth information storage 26. Thereafter, based on the acquired distance to the object, the convergence angle, and the distance information of the CG image provided from the application, the shielded region calculator 21 calculates the region where the CG image displayed for the right eye is shielded by the real object and the region where the CG image displayed for the left eye is shielded by the real object.
The information of the shielded region calculated by the shielded region calculator 21 is output to the AR image generator 22. The AR image generator 22 generates the CG image to be superimposed on a real scene, based on the image generation data provided from the application.
At that time, the AR image generator 22 generates the CG image in which the region expressing the occlusion is not displayed, based on the information of the shield region calculated by the shielded region calculator 21.
The right-eye CG image and the left-eye CG image generated by the AR image generator 22 are output to the AR display unit 23. The AR display unit 23 projects the input right-eye CG image onto the lens 12 and the input left-eye CG onto the lens 13, respectively.
Consequently, the right-eye CG image in which the occlusion optimum for the field of view of the right eye is expressed and the left-eye CG image in which the occlusion optimum for the field of view of the left eye is expressed are displayed on the lenses 12 and 13, respectively. As a result, it is possible to provide the AR image in which faithful occlusion is expressed.
<Display of CG Image in Consideration of Binocular Vision>
The right eye and the left eye of the human are separated by about 6 cm, and thus the sceneries captured by the left and right eyes are slightly different from each other. At this time, the amount of deviation differs depending on the distance of the object. Therefore, when performing the AR display in consideration of binocular vision, it is necessary to grasp the visions of the real object captured by the left and right eyes of the user.
Further, as shown in
Thus, as described above, the images are taken by the camera 14 provided such that the center of the lens is substantially the same as the center of the user's right eye and the camera 15 provided such that the center of the lens is substantially the same as the center of the user's left eye. Consequently, it is possible to capture the images that are almost the same as the visions of the real object by the left and right eyes of the user.
As described above, when the user sees the real objects 200 and 201 in
For example, when the right eye of the human captures the real objects 200 and 201 in
The images taken by the cameras 14 and 15 are also the same as the images captured by the left and right eyes of the human, and as shown in the upper part of
Then, as shown in the lower part of
In this way, by respectively generating and displaying the right-eye CG image and the left-eye CG image in consideration of the shielding relationship in the left and right visual fields, the user can view the AR image in which the faithful occlusion is expressed.
As described above, it is possible to provide a natural AR image that does not give a sense of discomfort to the user.
<Another Example of CG Image Generation>
Further, the CG images may be displayed on the lenses 12 and 13 after the resolution, in other words, the blur amount of the CG image is adjusted in accordance with the distance information of the CG image provided from the application. This makes it possible to display a more natural CG image.
In this case, the depth information storage 26 stores the blur amount information including the distance of the CG image and the blur amount associated with the distance. The AR image generator 22 acquires the distance of the CG image based on the distance information of the CG image provided from the application. Then, the AR image generator 22 searches for the blur amount information stored in the depth information storage 26, and extracts the blur amount that matches or is close to the blur amount corresponding to the acquired distance.
Thereafter, the AR image generator 22 performs the blurring process to the CG image based on the extracted blur amount. The blur information indicates, for example, the degree of blur of the outline of the CG image, and the process of blurring the outline of the CG image is performed based on the blur information.
In this manner, since the CG image can be blurred in the same way as in the distant view, the sense of discomfort such as the CG image being clearly displayed even though it is a CG image displayed at the distance can be eliminated, so that a more natural AR image can be provided to the user.
Further, by generating the CG image using the blur amount information in which the size of the CG image and the like are taken into consideration instead of determining the blur amount using only the distance information of the CG image, it is possible to provide an AR image having a natural blur that fits in a distant view as compared with the case of using only the distance information.
In this case, the blur amount information is the information having the display distance and size of the CG image and the blur amount associated with the distance and size of the CG image. The AR image generator 22 acquires the distance of the CG image based on the distance information of the CG image provided from the application, and similarly acquires the size information of the CG image based on the image generation data of the CG image provided from the application.
Then, the AR image generator 22 searches for the blur amount information stored in the depth information storage 26, and extracts the blur amount that matches or is close to the blur amount corresponding to the acquired distance and size. Then, based on the extracted blur information, the AR image generator 22 performs a process of blending the outline of the CG image.
In the foregoing, the present invention has been specifically described based the embodiment, but it is needless to say that the present invention is not limited to the embodiment described above and can be variously modified within the range not departing from the gist thereof.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2018/044569 | 12/4/2018 | WO |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2020/115815 | 6/11/2020 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 6045229 | Tachi et al. | Apr 2000 | A |
| 6084557 | Ishida | Jul 2000 | A |
| 10306217 | Sato | May 2019 | B2 |
| 10623628 | Hanover | Apr 2020 | B2 |
| 20060170652 | Bannai et al. | Aug 2006 | A1 |
| 20120206452 | Geisner et al. | Aug 2012 | A1 |
| 20130011045 | Lee | Jan 2013 | A1 |
| 20130141419 | Mount et al. | Jun 2013 | A1 |
| 20140184496 | Gribetz | Jul 2014 | A1 |
| 20140333665 | Sylvan | Nov 2014 | A1 |
| 20150156481 | Varga | Jun 2015 | A1 |
| 20150356785 | Takemoto | Dec 2015 | A1 |
| 20180275410 | Yeoh et al. | Sep 2018 | A1 |
| 20200368616 | Delamont | Nov 2020 | A1 |
| Number | Date | Country |
|---|---|---|
| 10-112831 | Apr 1998 | JP |
| 10-327433 | Dec 1998 | JP |
| 2006-209664 | Aug 2006 | JP |
| 2015-230695 | Dec 2015 | JP |
| Entry |
|---|
| International Search Report issued in corresponding International Patent Application No. PCT/JP2018/044569, dated Feb. 26, 2019, with English translation. |
| Number | Date | Country | |
|---|---|---|---|
| 20220060680 A1 | Feb 2022 | US |
