Patent Grant
10048750
The present application is a U.S. National Stage filing under 35 U.S.C. § 371 of international patent cooperation treaty (PCT) application No. PCT/CN2013/088554, filed Dec. 4, 2013, and entitled “CONTENT PROJECTION SYSTEM AND CONTENT PROJECTION METHOD,”which claims priority to Chinese Patent Application No. 201310390652.6, entitled “CONTENT PROJECTION SYSTEM AND METHOD”, filed on Aug. 30, 2013, which applications are hereby incorporated herein by reference in their respective entireties.
The present application relates to the field of virtual reality technologies, and in particular, to a content projection system and method.
In a mixed reality environment, a system often displays virtual graphical information, but currently, fused display of virtual graphic information and reality information is not yet implemented desirably. For example, when a user gazes somewhere (an actual object or a virtual graph), due to an incorrect depth or location of a virtual graph, the user fails to clearly see virtual graphic information needing to be seen; or, when a user gazes somewhere, virtual graphic information located at other depths and locations is also clearly displayed, and no proper bokeh effect is produced, which affects a real sense of immersion of the user. For example, currently, Google glasses can only display virtual content on a virtual screen at a specific distance, and as a result, the content cannot be well fused with the reality; therefore, the virtual screen can only be placed in angled upper front of the glasses, and a user needs to deliberately look upwards to view virtual graphic information, which seriously affects visual experience of the user. However, in a mixed reality system with stereoscopic visions for both eyes, information of stereoscopic depth is further added, and consequently, it is harder to reconcile a virtual system and a reality system, and setting the location of the virtual graphic information incorrectly further causes trouble such as diminution of vision, dizziness, and headache to a user.
The following presents a simplified summary in order to provide a basic understanding of some example embodiments disclosed herein. This summary is not an extensive overview. It is intended to neither identify key or critical elements nor delineate the scope of the example embodiments disclosed. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.
An example objective of the present application is to provide a content projection system and method, so that virtual graphic information is better fused with reality information, thereby improving visual experience of a user.
To achieve the foregoing objective, in a first example embodiment, this application provides a content projection system, comprising:
In a second example embodiment, the present application further provides a content projection method, comprising:
According to the system and method in the embodiments of the present application, a gaze point location of a user is detected in real time, an imaging parameter corresponding to visual information is obtained with respect to the gaze point location, and therefore, a virtual image is generated and projected to the fundus, so that virtual graphic information is better fused with reality information, the user obtains a better sense of immersion, and visual experience of the user is improved.
The method and apparatus of the present application are described below in detail with reference to the accompanying drawings and the embodiments.
In occasions of virtual reality, mixed reality, enhanced reality, enhanced virtuality, and the like in which virtual images need to be projected, to obtain a better fusion effect between a real scenario and a virtual scenario, or between virtual scenarios, and to bring a better sense of immersion to a user, a virtual image needing to be projected needs to be formed at a current gaze point location of the user. Therefore, as shown in
In a possible implementation manner of the embodiment of the present application, the content projection system 100 may be a portable and easy-to-use apparatus such as a pair of glasses (comprising frame glasses, contact lenses, goggles, and the like) having a gaze point detection apparatus 110 and a virtual image projection apparatus 130. In particular, for a user who originally has an eye disease such as a refractive error and needs to wear glasses for refractive correction, the system of the present application may be directly implemented on the glasses for refractive correction; or if a corresponding virtual scenario needs to be presented when a user watches a film or plays a game, the functions of the system may further be integrated on a device such as 3D glasses used when the user watches a film or plays a game, which brings no extra burden to the user.
Definitely, in other possible implementation manners of the embodiment of the present application, the content projection system may further be: for example, other optical devices used in coordination with eyes of the user, such as helmet eye lenses and a front windscreen for driving.
As shown in
The visual information described herein may be virtual graphic information (for example, a virtual dialog box and a virtual ball), image information (for example, picture or video information made in advance), or comprise both the graphic information and the image information.
The imaging parameter of the visual information described herein comprises: a location at which the visual information should be presented, a parameter of the visual information such as the size of the visual information, and a rendering parameter of the visual information, for example, a parameter of a virtual camera (such as a location, an orientation, and an angle of view), and a parameter such as light and special effects.
When a user views real objects with eyes, usually, the user can only clearly see objects within a certain range of depth of field near a gaze point, while a vague image of objects out of this range is formed at the fundus because these objects are out of focus. In a possible implementation manner of the embodiment of the present application, to make a generated virtual image better conform to visual effects of human eyes, and to bring more real experience to a user, the imaging parameter of the visual information further comprises depth-of-field information. The depth-of-field information may be used as a depth-of-field parameter of the virtual camera.
Therefore, in a possible implementation manner of the embodiment of the present application, the gaze point detection apparatus further comprises:
A picture of the pupil is usually obtained by using an image collection module (such as a micro camera) capable of shooting an image of the surface of the eye, and then, the size of the pupil of the eye is calculated by means of image processing. The technology for detecting the size of an eye pupil is an existing technology, and is not described in this embodiment again.
In this implementation manner, the visual information parameter determining module generates the depth-of-field information according to the size of the pupil of the eye.
Obtaining depth-of-field information according to the size of an eye pupil is an existing technology, and is not further described in this embodiment again. Generally speaking, a larger pupil indicates a smaller depth of field, and a smaller pupil indicates a greater depth of field.
In a possible implementation manner of the embodiment of the present application, the image rendering module comprises:
Herein, after the bokeh imaging processing is performed on the visual information according to the depth-of-field information, visual information out of the range of the depth of field is blurred, bringing real visual experience to a user.
As shown in
Taking the gaze point of the user being at the flowerpot 310 as an example, the gaze point detection apparatus of the content projection system according to the embodiment of the present application detects that a current gaze point of an eye is at the flowerpot 310.
The image generating apparatus generates a corresponding virtual image according to the first virtual information 320, the second virtual information 340, and locations (separately near the flowerpot 310 and the vase 330) at which the first virtual information 320 and the second virtual information 340 need to be presented. For example, in this implementation manner, the virtual image is an image comprising the first virtual information 320 and the second virtual information 340 which are located at specific locations and have corresponding sizes (for example, an object farther away is smaller than it is at a near place). If the virtual image is further required to have an out-of-focus imaging effect corresponding to the eye, processing such as bokeh may further be performed on the second virtual information 340 according to current depth-of-field information of the eye, and then the second virtual information 340 and the first virtual information 320 together form the virtual image.
Finally, the virtual image projection apparatus projects the virtual image at the fundus, so that the eye views an image in which the virtual image and the real scenario are mixed, as shown in
As shown in
In this implementation manner, the projection module 131 may be a micro projector; and
The at least one projection imaging parameter of the adjustable imaging module 132 described herein comprises: a focal length of the adjustable imaging module 132, a location of the adjustable imaging module 132 on the optical path, and the like. The adjustable imaging module 132 may change a light transmission manner by changing a structure or a location of the adjustable imaging module 132; and the adjustable imaging module 132 may be formed by a single lens with an adjustable imaging parameter, or formed by a lens assembly formed by multiple lenses, or may be an optical system comprising lenses and other optical components.
Preferably, in a possible implementation manner of the embodiment of the present application, the adjustable imaging module 132 may comprise an electronic adjustable lens, such as a liquid or liquid crystal lens disclosed in the U.S. Patent Publication No. US20070211207A1 and U.S. Pat. No. 4,572,616A; by controlling liquid or liquid crystal in the lens, an imaging parameter such as a shape and a refractive index of the lens is changed quickly. If the system of the present application is applied to a portable and wearable device such as glasses, using a single electronic adjustable lens as the adjustable lens module 132 can make the system smaller, lighter, and portable; moreover, a manner of applying the electronic adjustable lens to glasses has commercial applications, such as the Empower electronic varifocal glasses launched by Pixeloptics.
In addition to the foregoing electronic adjustable lens, a lens assembly formed by multiple lenses may further be used to form the adjustable lens module 132. For example, the imaging parameter of the adjustable lens module 132 is adjusted by changing locations of the multiple lenses and angles of optical axes of the lenses, and eccentrically setting the optical axes of the multiple lenses, where a part of or all of the multiple lenses are adjustable lenses.
In a possible implementation manner of the embodiment of the present application, the adjustable image module further comprises:
With the beam splitting unit, when viewing a real object or a virtual scenario in front, the user is not disturbed by the projection of the virtual image to the fundus of the eye of the user.
In the embodiment of the present application, the gaze point detection apparatus 110 may be in one of the following forms:
Definitely, persons skilled in the art may know that, in addition to the gaze point detection apparatuses 110 in the foregoing forms, other apparatuses that can be used for detecting a gaze point of an eye of a user may also be applied in the system of the embodiment of the present application.
As shown in
The gaze point detection apparatus 500 obtains, by analyzing and processing the image at the fundus of the eye, an optical parameter of the eye when the image collection device obtains the clearest image, and the current gaze point location of the eye can be calculated, which provides a basis for further eye self-adaptive operations.
The image presented at the “fundus” herein is mainly an image presented on the retina, and may be an image of the fundus, or an image of another object projected to the fundus. The eye herein may be a human eye, and may also be an eye of another animal.
As shown in
In a possible implementation manner of the embodiment of the present application, the adjustable imaging device 520 comprises: an adjustable lens unit 521, located on an optical path between the eye and the image collection device 510, where a focal length of the adjustable lens unit 521 is adjustable and/or a location of the adjustable lens unit 521 on the optical path is adjustable. With the adjustable lens unit 521, a system equivalent focal length between the eye and the image collection device 510 becomes adjustable. By means of the adjustment of the adjustable lens unit 521, the image collection device 510 obtains the clearest image of the fundus at a certain location or in a certain state of the adjustable lens unit 521. In this implementation manner, the adjustable lens unit 521 performs continuous and real-time adjustment in a detection process.
Preferably, in a possible implementation manner of the embodiment of the present application, the adjustable lens unit 521 is a focal length adjustable lens, configured to adjust the focal length of the focal length adjustable lens 521 by adjusting the refractive index and/or the shape of the focal length adjustable lens 521. Specifically: 1) the focal length is adjusted by adjusting the curvature of at least one surface of the focal length adjustable lens, for example, the curvature of the focal length adjustable lens is adjusted by adding or reducing a liquid medium in a cavity formed by two transparent layers; and 2) the focal length is adjusted by adjusting the refractive index of the focal length adjustable lens, for example, the focal length adjustable lens is filled with a specific liquid crystal medium, and an arrangement manner of the liquid crystal medium is adjusted by adjusting a voltage of an electrode corresponding to the liquid crystal medium, so as to change the refractive index of the focal length adjustable lens.
In another possible implementation manner of the embodiment of the present application, the adjustable lens unit 521 comprises: a lens assembly formed by multiple lenses, configured to adjust relative locations of the lenses in the lens assembly to adjust the focal length of the lens assembly. The lens assembly may also comprise a lens of which an imaging parameter such as the focal length is adjustable.
In addition to the foregoing two manners of changing the optical path parameter of the system by adjusting characteristics of the adjustable lens unit 521, the optical path parameter of the system may also be changed by adjusting the location of the adjustable lens unit 521 on the optical path.
Preferably, in a possible implementation manner of the embodiment of the present application, to avoid affecting user's experience of viewing an observed object, and to make the system portably applied to a wearable device, the adjustable imaging device 520 further comprises: a beam splitting unit 522, configured to form a light transmission path between the eye and the observed object and a light transmission path between the eye and the image collection device 510; in this case, the optical path can be folded, which reduces the volume of the system, and other visual experience of the user is not affected as far as possible.
Preferably, in this implementation manner, the beam splitting unit comprises: a first bean splitting unit, located between the eye and the observed object, and configured to transmit light from the observed object to the eye, and transmit light from the eye to the image collection device.
The first beam splitting unit may be a beam splitter, a beam splitting optical waveguide (comprising an optical fiber) or other proper beam splitting devices.
In a possible implementation manner of the embodiment of the present application, the image processing device 530 of the system comprises an optical path calibration module, configured to calibrate the optical path of the system, for example, align and calibrate the optical axis of the optical path, so as to ensure the measurement precision.
In a possible implementation manner of the embodiment of the present application, the image processing device 530 comprises:
In this implementation manner, the image collection device 510 can obtain the clearest image by using the adjustable imaging device 520; however, the clearest image needs to be found by using the image analyzing module 531; then, the optical parameter of the eye can be calculated according to the clearest image and the known optical path parameter of the system. Herein, the optical parameter of the eye may comprise the optical axis direction of the eye.
In a possible implementation manner of the embodiment of the present application, preferably, the system further comprises: a projection device 540, configured to project a light spot to the fundus. In a possible implementation manner, the function of the projection device may be implemented by means of a micro projector.
Herein, the projected light spot may have no specific pattern and be merely used for illuminating the fundus.
In a preferable implementation manner of the embodiment of the present application, the projected light spot comprises a pattern with abundant features. The abundant features of the pattern facilitate detection and improve the detection precision.
In order to avoid affecting the normal viewing of the eye, preferably, the light spot is an infrared light spot invisible to the eye.
In this case, to reduce interference of other spectra:
Preferably, in a possible implementation manner of the embodiment of the present application, the image processing device 530 further comprises:
For example, the projection control module 534 may self-adaptively adjust the luminance according to characteristics of an image obtained by the image collection device 510. Herein, the characteristics of the image comprise the contrast of image features, texture features, and the like.
Herein, a special circumstance of controlling the luminance of the light spot projected by the projection device is to turn on or turn off the projection device, for example, when the user continuously gazes at one point, the projection device can be turned off periodically; and when the fundus of the user is bright enough, a light emitting source can be turned off and a distance from the current gaze point of the sight line of the eye to the eye can be detected only using the information about the fundus.
In addition, the projection control module 534 may further control the luminance of the light spot projected by the projection device according to ambient light.
Preferably, in a possible implementation manner of the embodiment of the present application, the image processing device 530 further comprises: an image calibration module 533, configured to calibrate a fundus image to obtain at least one reference image corresponding to the image presented at the fundus.
The image analyzing module 531 compares images obtained by the image collection device 530 and the reference image, and performs calculation, to obtain the clearest image. Herein, the clearest image may be an obtained image having a minimum difference with the reference image. In this implementation manner, the difference between the currently obtained image and the reference image is calculated by using an existing image processing algorithm, for example, using a classical phase difference automatic focusing algorithm.
Preferably, in a possible implementation manner of the embodiment of the present application, the parameter calculation module 532 comprises:
The features of the eye herein may be obtained from the clearest image, and may also be obtained otherwise. The optical axis direction of the eye indicates a gaze direction of the sight line of the eye.
Preferably, in a possible implementation manner of the embodiment of the present application, the eye optical axis direction determining unit 5321 comprises: a first determining subunit, configured to obtain the optical axis direction of the eye according to the features of the fundus when the clearest image is obtained. Compared with obtaining the optical axis direction of the eye according to the features of the pupil and the eyeball surface, the accuracy of determining the optical axis direction of the eye according to the features of the fundus is higher.
When a light spot pattern is projected to the fundus, the size of the light spot pattern may be larger than a visible region of the fundus or smaller than a visible region of the fundus, where,
In another possible implementation manner of the embodiment of the present application, the eye optical axis direction determining unit 5321 comprises: a second determining subunit, configured to obtain the optical axis direction of the eye according to the features of the pupil of the eye when the clearest image is obtained. Herein, the features of the pupil of the eye may be obtained from the clearest image, and may also be obtained otherwise. Obtaining the optical axis direction of the eye according to the features of the pupil of the eye is an existing technology, and is not described herein again.
Preferably, in a possible implementation manner of the embodiment of the present application, the image processing device 530 further comprises: an eye optical axis direction calibration module 535, configured calibrate the optical axis direction of the eye, so as to determine the optical axis direction of the eye more accurately.
In this implementation manner, the known imaging parameter of the system comprises a fixed imaging parameter and a real-time imaging parameter, where the real-time imaging parameter is parameter information about the adjustable lens unit when the clearest image is obtained, and the parameter information may be recorded in real time when the clearest image is obtained.
After the current optical parameter of the eye is obtained, a distance between the gaze point of the eye and the eye can be calculated, which is specifically as follows:
According to (1) and (2), the distance do from the currently observed object 5010 (the gaze point of the eye) to the eye equivalent lens 5030 can be obtained as shown in formula (3):
According to the distance from the observed object 5010 to the eye obtained by the calculation described above, and the optical axis direction of the eye which can be obtained as a result of the above description, the gaze point location of the eye can be easily obtained, providing a basis for subsequent further interaction associated with the eye.
In this implementation manner, the image processing device is not shown in
Because the fundus is not bright enough under normal circumstances, it is better to illuminate the fundus; in this implementation manner, the fundus is illuminated by a light emitting source 640. In order avoid affecting user experience, preferably, the light emitting source 640 herein is an eye-invisible light emitting source, and is preferably a near-infrared light emitting source which has little impact on the eye 200 and to which the camera 610 is relatively sensitive.
In this implementation manner, the light emitting source 640 is located at an outer side of a spectacle frame on the right; therefore, a second beam splitter 650 together with the first beam splitter 620 is needed to complete the transmission of light emitted by the light emitting source 640 to the fundus. In this implementation manner, the second beam splitter 650 is also located in front of an incident surface of the camera 610, and therefore, the second beam splitter 650 further needs to transmit light from the fundus to the second beam splitter 650.
It can be seen that, in this implementation manner, to improve user experience and the collection definition of the camera 610, the first beam splitter 620 can preferably have characteristics of high reflectivity to infrared and high transmissivity to visible light. For example, an infrared reflective film may be set at one side of the first beam splitter 620 facing towards the eye 200 to implement the foregoing characteristics.
It can be seen from
In other implementation manners of the embodiment of the present application, the eye gaze point detection apparatus 600 may be located at one side, which is close to the eye 200, of the lens of the glasses 400; in this case, optical characteristic parameters of the lens need to be obtained in advance, and when the distance from the gaze point to the eye is calculated, influence factors of the lens are taken into consideration.
Light emitted by the light emitting source is reflected by the second beam splitter 650, projected by the focal length adjustable lens 630, and reflected by the first beam splitter 620, then enters the eye of the user through the lens of the glasses 400, and finally reaches the retina of the fundus; and the camera 610 shoots an image of the fundus through the pupil of the eye 200 via an optical path formed by the first beam splitter 620, the focal length adjustable lens 630, and the second beam splitter 650.
In a possible implementation manner of the embodiment of the present application, because the gaze point detection apparatus and the virtual image projection apparatus may each comprise: a device having a projection function (for example, the projection module of the virtual image projection apparatus and the projection device of the gaze point detection apparatus that are described above); and an imaging device with an adjustable imaging parameter (for example, the adjustable imaging module of the virtual image projection apparatus and the adjustable imaging device of the gaze point detection apparatus that are described above), the functions of the gaze point detection apparatus and the virtual image projection apparatus may be implemented by a same device.
As shown in
In a possible implementation manner of the embodiment of the present application, besides being used as adjustable imaging modules of the virtual image projection apparatus, the first beam splitter 620, the second beam splitter 650, and the focal length adjustable lens 630 may further be used as adjustable imaging devices of the gaze point detection apparatus. Herein, in a possible implementation manner, the focal length of the focal length adjustable lens 630 can be adjusted in different ranges, different ranges are separately corresponding to the gaze point detection apparatus and the virtual image projection apparatus, and the focal length may also be different. Alternatively, the focal length of the focal length adjustable lens 630 is adjusted in an integrated manner, but other optical components are further disposed at the front end of a photosensitive unit (such as a CCD) of the micro camera 610 of the gaze point detection apparatus, and are configured to perform auxiliary adjustment on the imaging parameter of the gaze point detection apparatus. In addition, in another possible implementation manner, configuration is performed such that an optical path from a light emitting surface (that is, a location for projecting the virtual image) of the light emitting source 640 to the eye is the same as an optical path from the eye to the micro camera 610, and when the micro camera 610 receives a clearest fundus image through the adjustment of the focal length adjustable lens 630, the virtual image projected by the light emitting source 640 is clearly presented right at the fundus. It can be seen from the above that, the eye gaze point detection apparatus and the virtual image projection apparatus in the embodiment of the present application may be implemented by using a set of devices, so that the whole system is simple in structure, small in volume, and portable.
Herein, the curved surface beam splitter 750 is configured to separately correspond to locations of the pupil associated with different optical axis directions of the eye and transmit an image presented at the fundus to an image collection device. In this way, a camera can shoot mixed and superposed images of various angles of the eyeball; however, because only a fundus part passing through the pupil can form a clear image on the camera, while other parts may be out of focus and cannot be clearly imaged, the imaging of the fundus part will not be interfered severely, and features of the fundus part can still be detected. Therefore, compared with the implementation manner shown in
In a possible implementation manner of the embodiment of the present application, the gaze point detection apparatus and the virtual image projection apparatus may also be reused. Similar to the embodiment shown in
In this case, the second beam splitter 750 is further configured to perform optical path transmission between the projection module and the fundus separately corresponding to the locations of the pupil associated with different optical axis directions of the eye. Because the virtual image projected by the projection device 740 is deformed after passing through the curved second beam splitter 750, in this implementation manner, the virtual image projection apparatus comprises:
In a possible implementation manner of the embodiment of the present application, to implement three-dimensional display, the visual information is separately corresponding to two eyes of the user.
In a possible implementation manner, the visual information corresponding to the two eyes of the user may be the same, but the imaging parameters of the visual information are separately corresponding to the two eyes of the user. For example, locations at which the same visual information separately corresponding to the two eyes needs to be presented are slightly different, and sizes of the locations may also be slightly different; in addition, because the locations of the two eyes are different, virtual camera parameters corresponding to the rendering parameter may also be slightly different. Therefore, two virtual images are generated according to the visual information separately corresponding to the two eyes and/or the imaging parameters of the visual information, and the virtual images are separately corresponding to the two eyes.
For example, a virtual image corresponding to a left eye is generated according to an imaging parameter of visual information corresponding to the left eye and corresponding visual information; and a virtual image corresponding to a right eye is generated according to an imaging parameter of visual information corresponding to the right eye and corresponding visual information. When the two eyes of the user correspondingly receive the two virtual images separately, a three-dimensional visual effect will be generated.
Besides implementing projection of three-dimensional virtual content by using the foregoing method, in a possible implementation manner of the embodiment of the present application, a same virtual image is projected to the two eyes of the user, but by separately projecting the virtual image to the two eyes with a certain deviation, the three-dimensional projection can also be implemented (for example, adjusting an optical axis parameter of an adjustable imaging module in a virtual image projection apparatus).
As shown in
A structure of the second virtual image projection apparatus is similar to a structure integrating the functions of the gaze point detection apparatus described in the embodiment in
A structure of the first virtual image projection apparatus is similar to the structure of the second virtual image projection apparatus 820 (the imaging parameter generating module of the virtual image projection apparatus is not shown in
Specifically, the program 932 may comprise program code, and the program code comprises a computer operation instruction.
The processor 910 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or is configured as one or more integrated circuits for implementing the embodiment of the present application.
The memory 930 is configured to store the program 932. The memory 930 may comprise a high-speed RAM memory, or may further comprise a non-volatile memory, for example, at least one magnetic memory.
It is clear to persons skilled in the art that, to make the description easy and brief, reference may be made to corresponding descriptions in the foregoing apparatus embodiments for specific operating processes of the devices and modules described above, descriptions are not repeated herein.
The foregoing content projection system according to the embodiment of the present application can automatically detect a gaze point location of a user in real time and self-adaptively process, corresponding to the gaze point location, virtual content needing to be projected, so that the virtual content and real or virtual scenarios being viewed by a user are fused with a better effect, and visual experience of the user is improved.
As shown in
In the embodiment of the present application, a gaze point location of an eye is detected in real time, and a corresponding virtual image is generated according to the current gaze point location, so that a virtual image viewed by a user changes as the gaze point changes, the virtual image can be better fused with a current real scenario or virtual scenario, and the user has a more real feeling, which brings a better sense of immersion.
Preferably, in a possible implementation manner of the embodiment of the present application, the visual information comprises: graphic information and/or image information.
Preferably, in a possible implementation manner of the embodiment of the present application, the image generating step comprises:
Preferably, in a possible implementation manner of the embodiment of the present application, the image generating step further comprises:
Preferably, in a possible implementation manner of the embodiment of the present application, the visual information is separately corresponding to two eyes of the user.
Preferably, in a possible implementation manner of the embodiment of the present application, the imaging parameters of the visual information are separately corresponding to the two eyes of the user.
Preferably, in a possible implementation manner of the embodiment of the present application, the virtual image projection step comprises:
Preferably, in a possible implementation manner of the embodiment of the present application, the virtual image projection step further comprises:
Preferably, in a possible implementation manner of the embodiment of the present application, the virtual image projection step comprises:
Preferably, in a possible implementation manner of the embodiment of the present application, the method further comprises:
Preferably, in a possible implementation manner of the embodiment of the present application, the imaging parameter of the visual information comprises depth-of-field information.
Preferably, in a possible implementation manner of the embodiment of the present application, the visual information parameter determining step further comprises generating the depth-of-field information according to the size of the pupil of the eye.
Preferably, in a possible implementation manner of the embodiment of the present application, the image rendering step comprises:
Preferably, in a possible implementation manner of the embodiment of the present application, the gaze point detection step comprises:
Preferably, in a possible implementation manner of the embodiment of the present application, the adjusting an imaging parameter of an optical path between an image collection location and the eye comprises:
Preferably, in a possible implementation manner of the embodiment of the present application, the gaze point detection step further comprises:
Preferably, in a possible implementation manner of the embodiment of the present application, the gaze point detection step further comprises:
The implementation manners of the foregoing various steps are the same as those in the descriptions of functions of corresponding modules or units in the foregoing apparatus embodiments, and are not described herein again.
Persons skilled in the art may understand that, in the foregoing methods of the specific implementation manners of the present application, serial numbers of the steps do not mean an execution sequence. The execution sequence of the steps should be determined according to the functions and internal logic of the steps, and should not constitute any limitation on the implementation processes of the specific implementation manners of the present application.
Persons of ordinary skill in the art can understand that, units and method steps of examples described with reference to the embodiments disclosed in the present application can be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in a manner of hardware or software depends on a specific application and a design constraint condition of the technical solution. Persons skilled in the art can implement the described functions for each specific application by using a different method, but such implementation shall not be considered beyond the scope of the present application.
If the functions are implemented in a form of a software function unit and are sold or used as independent products, the functions can be stored in a computer readable storage medium. Based on this, the above technical solution of the present application essentially or a part that contributes to the prior art or a part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, and comprises several instructions to instruct computer equipment (for example, a personal computer, a server, or network equipment) to perform all or a part of steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: any medium that can store program code, such as a USB flash drive, a removable hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc.
The foregoing implementation manners are merely used for explaining the present application, and are not intended to limit the present application. Persons of ordinary skill in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the present application, and therefore, all equivalent technical solutions also belong to the scope of the present application. The patent protection scope of the present application shall be subject to the claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2013 1 0390652 | Aug 2013 | CN | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/CN2013/088554 | 12/4/2013 | WO | 00 |
| Publishing Document | Publishing Date | Country | Kind |
|---|---|---|---|
| WO2015/027599 | 3/5/2015 | WO | A |
| Number | Name | Date | Kind |
|---|---|---|---|
| 4264154 | Petersen | Apr 1981 | A |
| 4572616 | Kowel et al. | Feb 1986 | A |
| 5182585 | Stoner | Jan 1993 | A |
| 5537163 | Ueno | Jul 1996 | A |
| 6072443 | Nasserbakht et al. | Jun 2000 | A |
| 6111597 | Tabata | Aug 2000 | A |
| 6151061 | Tokuhashi | Nov 2000 | A |
| 6152563 | Hutchison et al. | Nov 2000 | A |
| 6325513 | Bergner et al. | Dec 2001 | B1 |
| 7001020 | Yancey et al. | Feb 2006 | B2 |
| 7298414 | Stavely et al. | Nov 2007 | B2 |
| 7334892 | Goodall et al. | Feb 2008 | B2 |
| 7486988 | Goodall et al. | Feb 2009 | B2 |
| 7764433 | Kam et al. | Jul 2010 | B2 |
| 7766479 | Ebisawa | Aug 2010 | B2 |
| 8104892 | Hillis et al. | Jan 2012 | B2 |
| 8109632 | Hillis et al. | Feb 2012 | B2 |
| 8282212 | Hillis et al. | Oct 2012 | B2 |
| 8384999 | Crosby et al. | Feb 2013 | B1 |
| 8896632 | MacDougall et al. | Nov 2014 | B2 |
| 20020101568 | Eberl | Aug 2002 | A1 |
| 20020113943 | Trajkovic et al. | Aug 2002 | A1 |
| 20030043303 | Karuta et al. | Mar 2003 | A1 |
| 20030125638 | Husar et al. | Jul 2003 | A1 |
| 20050003043 | Sewal et al. | Jan 2005 | A1 |
| 20050014092 | Hasegawa et al. | Jan 2005 | A1 |
| 20050030438 | Nishioka | Feb 2005 | A1 |
| 20060016459 | Mcfarlane et al. | Jan 2006 | A1 |
| 20060103808 | Horie | May 2006 | A1 |
| 20060122530 | Goodall et al. | Jun 2006 | A1 |
| 20060146281 | Goodall et al. | Jul 2006 | A1 |
| 20060164593 | Peyghambarian et al. | Jul 2006 | A1 |
| 20060122531 | Goodall et al. | Aug 2006 | A1 |
| 20070019157 | Hills et al. | Jan 2007 | A1 |
| 20070211207 | Lo et al. | Sep 2007 | A1 |
| 20080002262 | Chirieleison | Jan 2008 | A1 |
| 20080106633 | Blum et al. | May 2008 | A1 |
| 20090066915 | Lai | Mar 2009 | A1 |
| 20090189974 | Deering | Jul 2009 | A1 |
| 20090279046 | Dreher et al. | Nov 2009 | A1 |
| 20090303212 | Akutsu et al. | Dec 2009 | A1 |
| 20110018903 | Lapstun et al. | Jan 2011 | A1 |
| 20110019258 | Levola | Jan 2011 | A1 |
| 20110213462 | Holladay | Jan 2011 | A1 |
| 20110051087 | Inoue et al. | Mar 2011 | A1 |
| 20110242277 | Do et al. | Oct 2011 | A1 |
| 20110279277 | Li-Chung | Nov 2011 | A1 |
| 20120007959 | Kwon et al. | Jan 2012 | A1 |
| 20120013389 | Thomas et al. | Jan 2012 | A1 |
| 20120038549 | Mandella et al. | Feb 2012 | A1 |
| 20120092618 | Yoo et al. | Apr 2012 | A1 |
| 20120113235 | Shintani | May 2012 | A1 |
| 20120127062 | Bar-Zeev et al. | May 2012 | A1 |
| 20120127422 | Tian et al. | May 2012 | A1 |
| 20120133891 | Jiang | May 2012 | A1 |
| 20120140044 | Galstian et al. | Jun 2012 | A1 |
| 20120154277 | Bar-Zeev et al. | Jun 2012 | A1 |
| 20120169730 | Inoue | Jul 2012 | A1 |
| 20120206485 | Osterhout et al. | Aug 2012 | A1 |
| 20120212499 | Haddick et al. | Aug 2012 | A1 |
| 20120212508 | Kimball | Aug 2012 | A1 |
| 20120242698 | Haddick et al. | Sep 2012 | A1 |
| 20120290401 | Neven | Nov 2012 | A1 |
| 20120293773 | Publicover et al. | Nov 2012 | A1 |
| 20120307208 | Trousdale | Dec 2012 | A1 |
| 20130044042 | Olsson et al. | Feb 2013 | A1 |
| 20130050646 | Nanbara | Feb 2013 | A1 |
| 20130072828 | Sweis et al. | Mar 2013 | A1 |
| 20130107066 | Venkatraman et al. | May 2013 | A1 |
| 20130127980 | Haddick et al. | May 2013 | A1 |
| 20130135203 | Croughwell | May 2013 | A1 |
| 20130147836 | Small et al. | Jun 2013 | A1 |
| 20130215504 | Kim et al. | Aug 2013 | A1 |
| 20130241805 | Gomez | Sep 2013 | A1 |
| 20130241927 | Vardi | Sep 2013 | A1 |
| 20130278631 | Border et al. | Oct 2013 | A1 |
| 20130335301 | Wong et al. | Dec 2013 | A1 |
| 20130335404 | Westerinen | Dec 2013 | A1 |
| 20130342572 | Poulos et al. | Dec 2013 | A1 |
| 20140078175 | Forutanpour et al. | Mar 2014 | A1 |
| 20140160157 | Poulos et al. | Jun 2014 | A1 |
| 20140225915 | Theimer et al. | Aug 2014 | A1 |
| 20140225918 | Mittal et al. | Aug 2014 | A1 |
| 20140232746 | Ro et al. | Aug 2014 | A1 |
| 20140240351 | Scavezze et al. | Aug 2014 | A1 |
| 20140267400 | Mabbutt et al. | Sep 2014 | A1 |
| 20140267420 | Schowengerdt et al. | Sep 2014 | A1 |
| 20140282224 | Pedley | Sep 2014 | A1 |
| 20140327875 | Blum et al. | Nov 2014 | A1 |
| 20140375680 | Ackerman et al. | Dec 2014 | A1 |
| 20150002542 | Chan et al. | Jan 2015 | A1 |
| 20150035861 | Salter et al. | Feb 2015 | A1 |
| 20150234184 | Schowengerdt et al. | Aug 2015 | A1 |
| 20150235632 | Liu et al. | Aug 2015 | A1 |
| 20150070391 | Nishimaki et al. | Sep 2015 | A1 |
| 20160034032 | Jeong | Feb 2016 | A1 |
| 20160035139 | Fuchs et al. | Feb 2016 | A1 |
| 20160062454 | Wang et al. | Mar 2016 | A1 |
| 20160171772 | Ryznar et al. | Jun 2016 | A1 |
| 20160189432 | Bar-Zeev et al. | Jun 2016 | A1 |
| 20160196603 | Perez et al. | Jul 2016 | A1 |
| 20160299360 | Fonte et al. | Oct 2016 | A1 |
| 20160370605 | Ain-Kedem | Dec 2016 | A1 |
| 20170092235 | Osman et al. | Mar 2017 | A1 |
| Number | Date | Country |
|---|---|---|
| 1372650 | Oct 2002 | CN |
| 1470227 | Jan 2004 | CN |
| 1141602 | Mar 2004 | CN |
| 1527126 | Sep 2004 | CN |
| 1604014 | Apr 2005 | CN |
| 1645244 | Jul 2005 | CN |
| 1653374 | Aug 2005 | CN |
| 1901833 | Jan 2007 | CN |
| 1912672 | Feb 2007 | CN |
| 2868183 | Feb 2007 | CN |
| 1951314 | Apr 2007 | CN |
| 101069106 | Nov 2007 | CN |
| 101072534 | Nov 2007 | CN |
| 101097293 | Jan 2008 | CN |
| 101103902 | Jan 2008 | CN |
| 201005945 | Jan 2008 | CN |
| 101116609 | Feb 2008 | CN |
| 101155258 | Apr 2008 | CN |
| 101194198 | Jun 2008 | CN |
| 101430429 | May 2009 | CN |
| 201360319 | Sep 2009 | CN |
| 201352278 | Nov 2009 | CN |
| 101900927 | Jan 2010 | CN |
| 101662696 | Mar 2010 | CN |
| 201464738 | May 2010 | CN |
| 101782685 | Jul 2010 | CN |
| 101819331 | Sep 2010 | CN |
| 101819334 | Sep 2010 | CN |
| 201637953 | Nov 2010 | CN |
| 101917638 | Dec 2010 | CN |
| 201754203 | Mar 2011 | CN |
| 102008288 | Apr 2011 | CN |
| 102083390 | Jun 2011 | CN |
| 102203850 | Sep 2011 | CN |
| 102292017 | Dec 2011 | CN |
| 102419631 | Apr 2012 | CN |
| 102481097 | May 2012 | CN |
| 101149254 | Jun 2012 | CN |
| 102487393 | Jun 2012 | CN |
| 202267785 | Jun 2012 | CN |
| 102572483 | Jul 2012 | CN |
| 102576154 | Jul 2012 | CN |
| 202383380 | Aug 2012 | CN |
| 102918444 | Feb 2013 | CN |
| 102939557 | Feb 2013 | CN |
| 102981270 | Mar 2013 | CN |
| 103054695 | Apr 2013 | CN |
| 103065605 | Apr 2013 | CN |
| 103150013 | Jun 2013 | CN |
| 103190883 | Jul 2013 | CN |
| 103197757 | Jul 2013 | CN |
| 103280175 | Sep 2013 | CN |
| 103297735 | Sep 2013 | CN |
| 103353663 | Oct 2013 | CN |
| 103353667 | Oct 2013 | CN |
| 103353677 | Oct 2013 | CN |
| 103558909 | Feb 2014 | CN |
| 19959379 | Jul 2000 | DE |
| 2646859 | Oct 2013 | EP |
| 03023431 | Jan 1991 | JP |
| 2676870 | Nov 1997 | JP |
| -109289973 | Nov 1997 | JP |
| 3383228 | Mar 2003 | JP |
| 2003307466 | Oct 2003 | JP |
| 2005058399 | Mar 2005 | JP |
| 2007129587 | May 2007 | JP |
| 201143876 | Mar 2011 | JP |
| 2012199621 | Oct 2012 | JP |
| 2012247449 | Dec 2012 | JP |
| 201012448 | Apr 2010 | TW |
| 2004023167 | Mar 2004 | WO |
| 2005077258 | Aug 2005 | WO |
| 2012075218 | Jun 2012 | WO |
| 2012083415 | Jun 2012 | WO |
| 2013074851 | May 2013 | WO |
| Entry |
|---|
| Gao et al., “Measuring Directionality of the Retinal Reflection with a Shack-Hartmann Wavefront Sensor,” 2009, Optical Society of America. |
| Lee et al., “A robust eye gaze tracking method based on a virtual eyeball model,” Springer-Verlag, 2008. |
| International Search Report dated Mar. 6, 2014 for PCT Application No. PCT/CN2013/088540, 8 pages. |
| Jeong, et al. “Tunable microdoublet lens array”, Optics Express, vol. 12, Issue 11, May 2004, pp. 2494-2500. |
| International Search Report dated Apr. 3, 2014 for PCT Application No. PCT/CN2013/088531, 10 pages. |
| International Search Report dated Feb. 27, 2014 for PCT Application No. PCT/CN2013/088522, 6 pages. |
| International Search Report dated May 8, 2014 for PCT Application No. PCT/CN2013/088547, 4 pages. |
| Kim et al., “A 200 s Processing Time Smart Image Sensor for an Eye Tracker using pixel-level analog image processing”, IEEE Journal of Solid-State Circuits, vol. 44, No. 9, Sep. 2009, 10 pages. |
| Hansen et al., “In the eye of the beholder: a survey of models for eyes and gaze”, IEEE Transactions on pattern analysis and machine intelligence, vol. 32, No. 3, Mar. 2010, 23 pages. |
| International Search Report dated May 28, 2014 for PCT Application No. PCT/CN2013/088553, 6 pages. |
| International Search Report dated May 28, 2014 for PCT Application No. PCT/CN2013/088545, 4 pages. |
| Office Action dated Feb. 27, 2017 for U.S. Appl. No. 14/783,495, 39 pages. |
| Office Action dated Apr. 21, 2017 for U.S. Appl. No. 14/781,581, 19 pages. |
| Office Action dated Apr. 20, 2017 for U.S. Appl. No. 14/781,578, 77 pages. |
| Office Action dated Mar. 30, 2017 for U.S. Appl. No. 15/028,019, 36 pages. |
| Office Action dated May 3, 2017 for U.S. Appl. No. 14/781,306, 46 pages. |
| International Search report dated Jun. 12, 2014 for PCT Application No. PCT/CN2013/088554, 4 pages. |
| Ji et al., “Real-Time Eye, Gaze and Face Pose Tracking for Monitoring Driver Vigilance”, Real-Time Imaging 8, 357-377 (2002) available online at http://www.idealibrary.com, 21 pages. |
| International Search Report dated Jan. 8, 2015 for PCT Application No. PCT/CN2014/088242, 2 pages. |
| International Search Report dated May 5, 2014 for PCT Application No. PCT/CN2013/088544, 4 pages. |
| International Search Report dated Jun. 5, 2014 for PCT Application No. PCT/CN2013/088549, 4 pages. |
| Smith, et al., “Determining Driver Visual Attention With One Camera”, IEEE Transactions on Intelligent Transportation Systems, vol. 4, No. 4, Dec. 2003, 14 Pages. |
| Singh, et al., “Human Eye Tracking and Related Issues: A Review”, International Journal of Scientific and Research Publications, vol. 2, Issue 9, Sep. 2012, ISSN 2250-3153, 9 pages. |
| Office Action dated Dec. 29, 2016 for U.S. Appl. No. 14/780,519, 25 pages. |
| Office Action dated Jun. 29, 2017 for U.S. Appl. No. 14/783,495, 50 pages. |
| Office Action dated Jul. 12, 2017 for U.S. Appl. No. 14/780,519, 45 pages. |
| Office Action dated Jun. 8, 2017 for U.S. Appl. No. 14/779, 968, 79 pages. |
| Office Action dated Oct. 4, 2017 for U.S. Appl. No. 14/781,584, 95 pages. |
| Office Action dated Nov. 9, 2017 for U.S. Appl. No. 14/781,578, 64 pages. |
| Office Action dated Nov. 9, 2017 for U.S. Appl. No. 14/780,519, 24 pages. |
| Office Action dated Nov. 17, 2017 for U.S. Appl. No. 14/783,495, 32 pages. |
| Office Action dated Dec. 14, 2017 for U.S. Appl. No. 14/779,321, 82 pages. |
| Office Action dated Dec. 15, 2017 for U.S. Appl. No. 14/779,968, 67 pages. |
| Office Action dated Feb. 5, 2018 for U.S. Appl. No. 14/779,321, 38 pages. |
| Office Action dated Jun. 29, 2018 for U.S. Appl. No. 14/781,578, 67 pages. |
| Office Action dated Jun. 25, 2018 for U.S. Appl. No. 14/779,321, 43 pages. |
| Office Action dated Jun. 14, 2018 for U.S. Appl. No. 14/780,519, 29 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20160179193 A1 | Jun 2016 | US |
