Patent Grant
10503251
The present application is the U.S. national phase entry of the international application PCT/CN2017/079692, with an international filing date of Apr. 7, 2017, which claims the benefit of Chinese Patent Application No. 201610581791.0, filed on Jul. 22, 2016, the entire disclosures of which are incorporated herein by reference.
The present invention relates to the field of display technology, and more particularly to a display system and a display method.
Holographic display technology can reproduce the amplitude and phase information of the recorded scene, provide the image and focus depth information for the eyes of the observer, solve the visual fatigue problems caused by the inconsistency of convergence focusing in the existing stereoscopic display technology, and provide the most realistic effect of stereoscopic display for the observer. The key for realizing real-time holographic display is the high-resolution spatial light modulator. Liquid crystal on silicon (LCoS) device is currently costly and difficult to popularize. As a spatial light modulator, the resolution of the existing liquid crystal display (LCD) is inadequate. Seereal Corporation provided a window holography technology for LCD. The window holographic technology compresses the existing LCD in space by an imaging optical system, thereby providing a high density image modulation wavefront in a small window to realize holographic display. However, due to the limitation of small field of view, the window holographic technology should cooperate with the eye tracking system in naked eye display to enable watching when the eye moves.
The inventors have found that the existing eye tracking optical system requires a separate configuration of the imaging lens outside the display system to track the eyeball position. On the one hand, in order not to cause occlusion of display, the tracking imaging lens should be bias placed, which is not conducive to the frontal collection for the eye image. On the other hand, it is necessary to firstly locate the eye position in the three-dimensional directions accurately to calculate the offset of the tracking optical path, the requirements on the tracking accuracy and calculation are thus relatively stringent.
Therefore, it is desired to provide a display system and a display method that simplifies the optical design and obtains a compact and efficient optical system.
According to an aspect of the invention, an embodiment of the invention provides a display system. The display system includes a projection unit and an eye tracking unit. The projection unit includes a projection light source, a projection lens and a spatial light modulator. The projection lens is located between the projection light source and the spatial light modulator. In a direction of an optical axis of the projection lens, the projection lens includes a first lens portion overlapping with the spatial light modulator and a second lens portion not overlapping with the spatial light modulator. The eye tracking unit includes a camera. An imaging optical path of the camera passes through the second lens portion.
The display system provided by the embodiment of the present invention uses the peripheral portion of the projection lens as an imaging lens for eye tracking. The image space of the projection unit and the object space of the eye tracking unit are coincident so that the eye can be imaged in the optical axis direction of the projection lens, and the high pass portion of the projection lens can be effectively utilized, so as to realize accurate real-time eye tracking. The display system of the embodiment of the present invention can be advantageously used in the display field including holographic display technology, simplifying the optical design, providing a compact and efficient optical system.
Optionally, the display system further includes a beam splitter located between the projection lens and the projection light source. The beam splitter is configured to reflect light from the second lens portion to the camera.
By applying the beam splitter, normal projection imaging can be realized without changing the projection optical path, the object such as the user's eye can also be imaged with the second lens portion.
Optionally, the beam splitter is a beam splitting plate or a beam splitting prism.
With the beam splitting plate or beam splitting prism, there is no need to consider the polarization direction of the light beams, and the setup for the projection unit and eye tracking unit is simple.
Optionally, the beam splitter is a polarization splitting prism.
With the polarization splitting prism, the polarization direction of the light beams are perpendicular to each other, thereby completely separating the light beam of the projection unit and the light beam of the eye tracking unit, avoiding crosstalk between the projection unit and the eye tracking unit.
Optionally, the eye tracking unit further includes a lighting source. The lighting source is configured to provide illumination to an eye of an observer through the second lens portion.
With the above mentioned configuration, even in a dark environment, accurate eye tracking can be realized by applying active illumination.
Optionally, the lighting source includes an infrared light source.
The light beam emitted by the lighting source is generally used to illuminate the user's eye, therefore applying an infrared light source as a lighting source can reduce the irritation or interference to the user's eye.
Optionally, the projection lens has only one lens.
In order to simplify the system structure, projection can be realized with only one lens.
Optionally, the projection lens includes a plurality of lenses.
By applying the projection lens group (i.e., multiple lenses) for projection, the projection can be flexibly adjusted to meet a variety of design requirements.
Optionally, the projection lens has a circular shape, and the spatial light modulator has a rectangular shape.
With the above mentioned arrangement, the first lens portion corresponding to the spatial light modulator has a rectangular shape, and the peripheral portion of the projection lens (i.e., the second lens portion) still has an imaging function, thereby realizing imaging of the user's eye. Similarly, the projection lens and the spatial light modulator can also have other shapes respectively, as long as the projection lens includes a first lens portion overlapping with the spatial light modulator and a second lens portion not overlapping with the spatial light modulator in the direction of the optical axis of the projection lens.
Optionally, the projection light source includes a laser. Optionally, the projection unit further includes a beam expander, a micro objective lens, a pinhole filter and a rectangular diaphragm sequentially disposed between the projection light source and the projection lens.
By applying the pinhole filter, the projection light source provides illumination of an approximate point source that does not allow light with other spatial frequencies to pass through, eliminating stray light. With the laser light source, the laser beam is modulated by the spatial light modulator, holographic display can thus be realized.
Optionally, the second lens portion is located on the periphery of the first lens portion.
By applying the second lens portion located on the periphery of the first lens portion, the high pass portion of the projection lens can be effectively utilized, so as to realize accurate real-time eye tracking.
According to another aspect of the present invention, an embodiment of the present invention provides a display method. The display method includes the following steps: performing projection by using a projection light source, a projection lens and a spatial light modulator; in a direction of an optical axis of the projection lens, the projection lens including a first lens portion overlapping with the spatial light modulator and a second lens portion not overlapping with the spatial light modulator; and performing eye tracking by using a camera, the camera performing imaging with the second lens portion.
The display method provided by the embodiment of the present invention uses the peripheral portion of the projection lens as an imaging lens for eye tracking. The image space of the projection unit and the object space of the eye tracking unit are coincident so that the eye can be imaged in the optical axis direction of the projection lens, and the high pass portion of the projection lens can be effectively utilized so as to realize accurate real-time eye tracking. The display method of the embodiment of the present invention can be advantageously used in the display field including holographic display technology, simplifying the optical design, providing a compact and efficient optical system.
Optionally, the method further includes reflecting light from the second lens portion to the camera by using a beam splitter located between the projection lens and the projection light source.
By applying the beam splitter, normal projection imaging can be realized without changing the projection optical path, the object such as the user's eye can also be imaged with the second lens portion.
Optionally, the method further includes providing illumination to an eye of an observer through the second lens portion.
With the above mentioned configuration, even in a dark environment, accurate eye tracking can be realized by applying active illumination.
Optionally, the step of providing illumination to an eye of an observer through the second lens portion includes: providing infrared illumination to an eye of an observer through the second lens portion.
The light beam emitted by the lighting source is generally used to illuminate the user's eye, therefore applying an infrared light source as a lighting source can reduce the irritation or interference to the user's eye.
Optionally, the step of performing eye tracking by using a camera includes: determining an azimuth angle of the eye based on a boundary between iris and sclera.
Based on image processing, the azimuth angle of the eye can be easily determined based on the boundary between the iris and the sclera. For example, eyes with different azimuth angles can be directly imaged at different positions on the camera. After the azimuth angle and imaging position are calibrated, the corresponding azimuth angle can be calculated by simply determining the coordinate position of the eye in the image received by the camera.
Optionally, the step of performing eye tracking by using a camera includes: determining a distance between the eye and the projection lens based on clarity of an eye feature.
In holographic display, the distance between the eye and the display system is a necessary parameter for hologram calculation. If the distance does not vary greatly, it is possible to consider only the azimuth angle. If the distance varies greatly, the change in distance should also be taken into account. When the object is clearly imaged, there is a corresponding relationship between the imaging parameters of the camera (such as but not limited to: the focal length of the camera's zoom lens group) and the object distance (i.e., the distance between the eye and the camera). By reading the focal length of the zoom lens group, the distance between the eye and the camera can be obtained, thereby calculating the distance between the eye and the projection lens.
In order to more clearly illustrate the technical solutions in embodiments of the invention or in the prior art, the appended drawings needed to be used in the description of the embodiments or the prior art will be introduced briefly in the following. Obviously, the drawings in the following description are only some embodiments of the invention, and for those of ordinary skills in the art, other drawings may be obtained according to these drawings under the premise of not paying out creative work.
In the following, the technical solutions in embodiments of the invention will be described clearly and completely in connection with the drawings in the embodiments of the invention. Obviously, the described embodiments are only part of the embodiments of the invention, and not all of the embodiments. Based on the embodiments in the invention, all other embodiments obtained by those of ordinary skills in the art under the premise of not paying out creative work pertain to the protection scope of the invention.
According to an aspect of the invention, an embodiment of the invention provides a display system. As shown in
The display system provided by the embodiment of the present invention uses the peripheral portion of the projection lens 1011 as an imaging lens for eye tracking. The image space of the projection unit 101 and the object space of the eye tracking unit 102 are coincident so that the eye 103 can be imaged in the optical axis direction of the projection lens 1011, and the high pass portion of the projection lens 1011 can be effectively utilized, so as to realize accurate real-time eye tracking. The display system of the embodiment of the present invention can be advantageously used in the display field including holographic display technology, simplifying the optical design, providing a compact and efficient optical system.
In the context of the invention, each “unit” in the embodiments can be realized by a computer (e.g. personal computer) or a combination of a computer and a suitable sensor; the processing of each “unit” can be realized e.g. by a processor in the computer.
In the imaging optical system, the increase in the aperture of the lens is helpful to improve the imaging luminous flux, thus enhancing the adaptability of the imaging detection system in the dark environment. Moreover, the aperture of the lens is one of the determinants for the optical imaging resolution, and the imaging resolution directly affects the accuracy of eye detection. Therefore, imaging lenses with large aperture are extremely advantageous for eye tracking.
Optionally, as shown in
By applying the beam splitter 1015, normal projection imaging can be realized without changing the projection optical path, the object such as the user's eye can also be imaged with the second lens portion 1014.
Optionally, the beam splitter 1015 is a beam splitting plate or a beam splitting prism.
With the beam splitting plate or beam splitting prism, there is no need to consider the polarization direction of the light beams, and the setup for the projection unit and eye tracking unit is simple.
Optionally, the beam splitter 1015 is a polarization splitting prism.
With the polarization splitting prism, the polarization direction of the light beams are perpendicular to each other, thereby completely separating the light beam of the projection unit and the light beam of the eye tracking unit, avoiding crosstalk between the projection unit and the eye tracking unit.
Optionally, the eye tracking unit 102 further includes a lighting source 1022. The lighting source 1022 is configured to provide illumination to an eye of an observer through the second lens portion 1014.
With the above mentioned configuration, even in a dark environment, accurate eye tracking can be realized by applying active illumination. In particular, due to the reversibility of the optical path, the lighting source 1022 can be arranged at, for example, the position of the camera 1021, thereby providing active illumination.
Optionally, the lighting source 1022 includes an infrared light source.
The light beam emitted by the lighting source is generally used to illuminate the user's eye, therefore applying an infrared light source as a lighting source can reduce the irritation or interference to the user's eye.
In the above mentioned embodiment, the projection lens 1011 is used for projection. However, it can be understood by those skilled in the art that the projection lens 1011 can also be replaced with a projection lens group (i.e., a plurality of lenses). As shown in
Optionally, as shown in
In order to simplify the system structure, projection can be realized with only one lens.
Optionally, as shown in
By applying the projection lens group (i.e., multiple lenses) for projection, the projection can be flexibly adjusted to meet a variety of design requirements.
Optionally, as shown in
With the above mentioned arrangement, the first lens portion 1013 corresponding to the spatial light modulator has a rectangular shape, and the peripheral portion of the projection lens (i.e., the second lens portion 1014) still has an imaging function, thereby realizing imaging of the user's eye. Similarly, the projection lens and the spatial light modulator can also have other shapes respectively, as long as the projection lens includes a first lens portion overlapping with the spatial light modulator and a second lens portion not overlapping with the spatial light modulator in the direction of the optical axis of the projection lens.
Optionally, the projection light source 10 includes a laser 1017. Optionally, the projection unit 101 further includes a beam expander 1018, a micro objective lens 1019, a pinhole filter 1016 and a rectangular diaphragm (not shown) sequentially disposed between the projection light source 10 and the projection lens 1011.
By applying the pinhole filter 1016, the projection light source 10 provides illumination of an approximate point source that does not allow light with other spatial frequencies to pass through, eliminating stray light. With the laser 1017, the laser beam is modulated by the spatial light modulator 1012, holographic display can thus be realized.
Optionally, as shown in
By applying the second lens portion located on the periphery of the first lens portion, the high pass portion of the projection lens can be effectively utilized, so as to realize accurate real-time eye tracking.
In some embodiments, as shown in
According to another aspect of the present invention, an embodiment of the present invention provides a display method. As shown in
The display method provided by the embodiment of the present invention uses the peripheral portion of the projection lens as an imaging lens for eye tracking. The image space of the projection unit and the object space of the eye tracking unit are coincident so that the eye can be imaged in the optical axis direction of the projection lens, and the high pass portion of the projection lens can be effectively utilized so as to realize accurate real-time eye tracking. The display method of the embodiment of the present invention can be advantageously used in the display field including holographic display technology, simplifying the optical design, providing a compact and efficient optical system.
Optionally, as shown in
By applying the beam splitter, normal projection imaging can be realized without changing the projection optical path, the object such as the user's eye can also be imaged with the second lens portion.
Optionally, as shown in
With the above mentioned configuration, even in a dark environment, accurate eye tracking can be realized by applying active illumination.
Optionally, the step of providing illumination to an eye of an observer through the second lens portion includes: providing infrared illumination to an eye of an observer through the second lens portion.
The light beam emitted by the lighting source is generally used to illuminate the user's eye, therefore applying an infrared light source as a lighting source can reduce the irritation or interference to the user's eye.
Optionally, the step of performing eye tracking by using a camera includes: determining an azimuth angle of the eye based on a boundary between iris and sclera.
Based on image processing, the azimuth angle of the eye can be easily determined based on the boundary between the iris and the sclera. For example, eyes with different azimuth angles can be directly imaged at different positions on the camera. After the azimuth angle and imaging position are calibrated, the corresponding azimuth angle can be calculated by simply determining the coordinate position of the eye in the image received by the camera.
Optionally, the step of performing eye tracking by using a camera includes: determining a distance between the eye and the projection lens based on clarity of an eye feature.
In holographic display, the distance between the eye and the display system is a necessary parameter for hologram calculation. If the distance does not vary greatly, it is possible to consider only the azimuth angle. If the distance varies greatly, the change in distance should also be taken into account. When the object is clearly imaged, there is a corresponding relationship between the imaging parameters of the camera (such as but not limited to: the focal length of the camera's zoom lens group) and the object distance (i.e., the distance between the eye and the camera). By reading the focal length of the zoom lens group, the distance between the eye and the camera can be obtained, thereby calculating the distance between the eye and the projection lens.
The display system and display method provided by the embodiment of the present invention uses the peripheral portion of the projection lens as an imaging lens for eye tracking. The image space of the projection unit and the object space of the eye tracking unit are coincident so that the eye can be imaged in the optical axis direction of the projection lens, and the high pass portion of the projection lens can be effectively utilized so as to realize accurate real-time eye tracking. The display system and display method of the embodiment of the present invention can be advantageously used in the display field including holographic display technology, simplifying the optical design, providing a compact and efficient optical system.
Apparently, the person skilled in the art may make various alterations and variations to the invention without departing the spirit and scope of the invention. As such, provided that these modifications and variations of the invention pertain to the scope of the claims of the invention and their equivalents, the invention is intended to embrace these alterations and variations.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2016 1 0581791 | Jul 2016 | CN | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/CN2017/079692 | 4/7/2017 | WO | 00 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2018/014595 | 1/25/2018 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 6433760 | Vaissie | Aug 2002 | B1 |
| 7522344 | Curatu | Apr 2009 | B1 |
| 8998414 | Bohn | Apr 2015 | B2 |
| 20050122549 | Goulanian | Jun 2005 | A1 |
| 20050213182 | Cossairt | Sep 2005 | A1 |
| 20080198431 | Schwerdtner | Aug 2008 | A1 |
| 20100045781 | Flon | Feb 2010 | A1 |
| 20100290120 | Ditto | Nov 2010 | A1 |
| 20130089240 | Northcott | Apr 2013 | A1 |
| 20140361990 | Leister | Dec 2014 | A1 |
| 20140375790 | Robbins | Dec 2014 | A1 |
| 20150192775 | Suzuki | Jul 2015 | A1 |
| 20150205259 | Kim | Jul 2015 | A1 |
| 20160026253 | Bradski | Jan 2016 | A1 |
| 20160091722 | Liu et al. | Mar 2016 | A1 |
| 20160106581 | Gonzalez | Apr 2016 | A1 |
| 20160262608 | Krueger | Sep 2016 | A1 |
| 20160335475 | Krenzer | Nov 2016 | A1 |
| 20170032173 | Lin | Feb 2017 | A1 |
| 20170108528 | Atlas | Apr 2017 | A1 |
| 20170109580 | Kaehler | Apr 2017 | A1 |
| 20170131765 | Perek | May 2017 | A1 |
| 20170147859 | Zhang | May 2017 | A1 |
| 20170185037 | Lee | Jun 2017 | A1 |
| 20170255012 | Tam | Sep 2017 | A1 |
| 20170255013 | Tam | Sep 2017 | A1 |
| 20180011022 | Tahara | Jan 2018 | A1 |
| 20180031944 | McKnight | Feb 2018 | A1 |
| 20180122143 | Ellwood, Jr. | May 2018 | A1 |
| 20190049731 | Knuettel | Feb 2019 | A1 |
| 20190086598 | Futterer | Mar 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 101176043 | May 2008 | CN |
| 103248905 | Aug 2013 | CN |
| 104407440 | Mar 2015 | CN |
| 104407440 | Mar 2015 | CN |
| 105652444 | Jun 2016 | CN |
| 105785571 | Jul 2016 | CN |
| 105954992 | Sep 2016 | CN |
| 20130088666 | Aug 2013 | KR |
| 2008028829 | Mar 2008 | WO |
| Entry |
|---|
| David et al., “Design of integrated eye trackerdisplay device for head mounted systems” Proc. SPIE 7419, Infrared Systems and Photoelectronic Technology IV, 741910 (Aug. 27, 2009). |
| First Office Action for Chinese Patent Application No. 201610581791.0 dated Jan. 25, 2018. |
| Search Report for International Patent Application No. PCT/CN2017/079692 dated Jun. 30, 2017. |
| Number | Date | Country | |
|---|---|---|---|
| 20180292898 A1 | Oct 2018 | US |
