Multi-mode display device

Abstract

The present invention relates generally to various arrangements of optical and electronic components to form a high-resolution helmet mounted display (HMD) or other compact display device. In particular, the current invention is designed in such a way as to allow it to operate utilizing several different kinds of image generation device and to incorporate many different features including the option of “eye tracking”.

Description

BACKGROUND AND SUMMARY OF THE INVENTION

[0002] The present invention relates generally to various arrangements of optical and electronic components to form a high-resolution helmet mounted display (HMD) or other compact display device. In particular, the current invention is designed in such a way as to allow it to operate utilizing several different kinds of image generation device and to incorporate many different features including the option of “eye tracking”.

[0003] Recently there have been several major breakthroughs in display generation technology. These breakthroughs have occurred in the fields of AMLCD, Ferro-Electric Display technology and a new technology known as LCD on silicon. Display devices will soon be the emerging with XGA and SXGA performance. At this time, it is uncertain which technology will prevail in the HMD market. This HMD design works equally well with both transmissive and reflective technologies. When used with a transmissive AMLCD type display device, the HMD design allows the generation of a full stereoscopic (separate image to each eye) image from a single display device. This is achieved by “time-multiplexing” the use of the display device between the two eyes. When used with a reflective display technology such as FED or LCD on silicon device, a separate display chip is required for each eye. In addition to the versatility of this display configuration to utilize either of these display technologies, it is also possible to utilize both of these display technologies simultaneously. In such a situation, the images produced would consist of an “optical overlay” of the images generated by the FED and AMLCD devices. Under certain circumstances this configuration can yield great benefits. For example, it would be possible to provide a large panoramic field of view in relatively low resolution (perhaps in monochrome) and use the reflective displays to provide a smaller high-quality field of view as a subset of this larger field image. It is also possible utilizing this unique HMD design to combine multiple fields of view utilizing pairs of similar display technology. For example, utilizing a pair of transmissive AMLCD displays (2 displays for each eye) or a pair of FED displays for each eye.

[0004] With the growing interest in “augmented reality” and virtual reality based CAD there is a growing demand for new and more efficient man-machine interface tools. One of the most sophisticated user interface tools is that of the “eye cursor” or “eye tracking device.” Such “eye cursor” technology calculates the precise direction that each eye is pointing and mathematically calculates the “target” that is being viewed. This technology is inherently three-dimensional and represents an ideal interface tool for 3-D virtual worlds and augmented reality. The unique design of this HMD allows the incorporation of eye tracking in addition to the display tasks normally associated with an HMD. In addition, the imaging technology that comprises the eye tracking system can be used to overcome one of the major problems with lens based HMD designs which is the ability to automatically accommodate viewers with different inter-ocular spacing (spacing between the eyes). Most off-the-shelf HMD's which are utilizing a lens based design require manual inter-ocular adjustment for each viewer. This adjustment is often difficult and time-consuming. Both of which, are factors that make such HMD's inappropriate for public use. By utilizing the imaging hardware of the eye tracking system as feedback an automatic or servo controlled inter-ocular adjustment can be readily achieved. Interestingly, such precise inter-ocular adjustments are an essential requirement for the correct operation of most eye tracking hardware. In this way, both hardware requirements are met and are mutually symbiotic.

[0005] In certain situations, a “see-through” augmented reality HMD design is preferable to a fully enclosed virtual reality configuration. In such a situation it is possible to configure the HMD design described in this specification to “see-through-mode” by replacing the eye tracking video cameras with corrective optics. This will allow the user to see virtual objects superimposed upon the real world images at the cost of losing the eye tracking and automatic inter-ocular adjustment features of the design. An alternative configuration which achieves many of the same objectives is simply to include two miniature video cameras as part of the HMD design. These cameras would provide “real-time” video feeds which can be digitally combined with the “virtual objects” displayed in the HMD. It would of course be possible to provide the same monoscopic video signal from a single miniature video camera to both eyes of the HMD however this would preclude the use of eye tracking for all but the virtual objects displayed in the system because eye-tracking inherently requires stereoscopic images to provide the required three-dimensional information.

[0006] In many ways, this electronic equivalent of “see-through mode” is actually superior to the simplistic optical approach as it inherently avoids the problems of “transport lag” and “positional inaccuracies” associated with optical superimposing of virtual objects. This is possible because the same “live video feeds” are being used to determine the spatial points of reference for the projection of the virtual objects therefore, any “transport lag” and/or “positional inaccuracies between the “real world” and the “virtual world” are not perceived by the user because they are common to both the background and the virtual objects projected upon it. In effect, the user is working entirely within a “virtual environment”.

PREFERRED EMBODIMENT

[0007] The preferred embodiment of this design the shown in FIGS. 1,2. The HMD consists of two separate optical assemblies and one optional common optical device. Optical assembly 1 consists of mirror M1, beamsplitter BS1, beamsplitter BS3, eyepiece E1, light source LS3 and optional components consisting of reflective display device RD1, collimating lens CL1, infrared filter F1, camera ETC1, and infrared light sources LS1. Optical assembly 2 is basically identical and consists of mirror M2, beamsplitter BS3, beamsplitter BS4, eyepiece E2, light source LS2 and optional components consisting of reflective display device RD2, collimating lens CL2, infrared filter F2, camera ETC2, and infrared light sources LS2. The common optical component is transmissive display device TD.

[0008] Mode 1:

[0009] (Reflective Mode—Virtual Reality)

[0010] In this mode two reflective display devices (such as FED or reflective LCD) are used to display a separate image to each eye. The video information delivered to each display device may be purely synthetic or be a combination of virtual imagery and real-time video feeds. Although different configurations are possible, beamsplitter's BS1 and BS2 would generally be of non-polarized type and beamsplitter's BS3 and BS4 would be of broadband polarizing design. Light sources LS3 and LS4 can be of any appropriate type but would most probably be of an LED design. In this configuration, light from light source LS3 would pass through beamsplitter BS3. Only light with one plane of polarization will be allowed through beamsplitter BS3. The other plane of polarization being reflected from the hypotenuse of the beamsplitter. In this particular configuration the optional transmissive display element TD would simply be replaced by a non-reflective baffle. Light being reflected towards this baffle would simply be absorbed. The reminder of the plane polarized light incident upon reflective display device RD1 will then selectively undergo rotation of the plane of polarization (phase retardation) depending upon the state of each individual pixel. The light that has undergone this rotation (pixels which are on) will then be reflected from the hypotenuse of beamsplitter BS3 towards beamsplitter BS1. As beamsplitter BS1 is non-polarizing, half of the light incident upon it will pass through the beamsplitter towards mirror M1 the other half of the light will reflect off the hypotenuse of beamsplitter BS1 towards Infrared filter F1. As light source L3 contains no infrared component, this light will be absorbed by F1. The light that strikes concave mirror M1 will form a real image at some distance from M1 such that the light from this image will once again be partially reflected from the hypotenuse of beamsplitter BS1. This reflected light will enter E1 and form an image which is viewable by the user of the HMD.

[0011] Ideally, beamsplitter's BS3 and BS1 should be optically bonded to reduce reflections. BS1 and BS2 could be of polarizing design if an additional phase retarder is placed between beamsplitter BS1 and mirror M1 (and between beamsplitter BS2 and mirror M2) as shown in FIG. 7. In fact, such a configuration will yield significantly higher optical performance. The reason why this is not considered the preferred embodiment is that polarizing beamsplitter's are relatively expensive devices. It is also possible (at the cost of further reducing system optical efficiency) to replace beamsplitter's BS3,BS4 with simple non polarizing beamsplitter's and by adding additional polarizers P1-P4 as shown in FIG. 5. which will operate in an optically similar manner. Equally valid is a somewhat simpler configuration utilizing single polarizing elements P1, P2 and shown in FIG. 5. These configurations may be preferable in a production model given that in a further modification, the cube beamsplitter's BS1-BS4 could potentially be replaced by simple partially silvered mirrors which are inexpensive. Reducing the overall optical efficiency may not pose a significant problem if the light sources LS3 and LS4 are sufficiently powerful. The replacement of Cube beamsplitter's with partially silvered mirror style beamsplitter's does however have the disadvantage of double reflections.

[0012] The miniature video cameras ETC1,ETC2 work in conjunction with eyepiece lenses E1, E2 to form images of the user's eyes. Infrared filters F1, F2 are used to block stray light from the display chips from entering the cameras directly. Light sources LS1, LS2 can be of a variety of different sources and/or different number of light's. However, a configuration of four infrared LED's is probably preferred. A simple configuration of 4 LED's in a square can be used to achieve both eye tracking and eye positioning (for inter-ocular adjustment). In such a configuration the LED's will produce four distinct reflections from the cornea of the viewer's eyes. This information, in addition to the relative position of the pupil of the eye can be used to determine the direction that the eye is pointing in 3-D (i.e. a line of sight can be determined). The two separate lines of sight (one for each eye) can be used to locate a specific point in 3-D space. Infrared light (or near infrared) is used in preference to visible light, firstly because it allows the stray light from the image projection to be easily eliminated and secondly because the use of visible light would be a constant source of irritation to the user.

[0013] In this embodiment of the HMD, eyepiece E1, beamsplitter's BS1, BS3 light source LS3 and reflective display device RD1, collimating lens CL1 and camera assembly consisting of infrared filter F1, light source or sources LS1 and camera ETC1 would be physically joined together and would be free to move laterally as shown in FIGS. 1,2. Mirror M1 is free to move laterally on the same axes as this assembly and may or may not be physically connected to the said assembly. The lateral adjustment of the aforementioned optical assembly (and its counterpart) allows for automatic adjustment of different eye spacing (inter-ocular adjustment). This adjustment would ideally be performed by some form of an electromechanical means such as a servo system. The independent adjustment of the position of mirror M1 will provide focus or diopter adjustment. To reduce costs, the adjustment of mirrors M1, M2 may be done manually, or alternatively the position of M1,M2 may be fixed with respect to the separate optical assemblies and focus/diopter adjustment may be made directly at the eyepieces E1, E2.

[0014] Mode 2:

[0015] (Transmissive Mode—Virtual Reality)

[0016] In an alternative configuration, the reflective display devices RD1,RD2 can be replaced by a single transmissive display device (such as an AMLCD). In this configuration stereoscopic images (a separate image to each eye) can be achieved with a single display element by “time multiplexing” the two images (i.e. by alternating quickly between each light source). This “time multiplexing” is achieved by utilizing two separate light sources LS4, LS3 for the illumination of the left and right eye images respectively. It is of course also possible to provide the same image to the left and right eyes simultaneously, simply by switching light sources LS3, LS4 on at the same time. In many ways the transmissive configuration is more efficient because only a single display element is required for the generation of two separate images. However, at the present time, reflective display technology is significantly more advanced than similar transmissive display technology. One of the major differences in the hardware configuration when a transmissive element is utilized is that there can no longer be a physical connection between beamsplitter's BS1, BS3 and BS2, BS4. In this configuration beamsplitter's BS3, BS4, light sources LS3, LS4 and transmissive display device TD form a single optical assembly. This leaves optical assembly-1 consisting of beamsplitter BS1, eyepiece E1 and optional components infrared filter F1, light sources LS1 and miniature camera ETC1. Optical assembly-2 consists of beamsplitter BS2, eyepiece E2 and optional components infrared filter F2, light sources LS2 and miniature camera ETC2. The operation of this modified optical assembly will be similar to that previously described, with the exception that it is now an absolute requirement that mirrors M1,M2 are adjustable independently from optical assemblies 1,2 in order to maintain the correct focal position.

[0017] Additional Enhancements:

[0018] With the reflective design, the inclusion of collimating optics CL1, CL2 (as shown in FIGS. 1-7) can greatly improve the overall optical efficiency of the system without seriously degrading the image quality.

[0019] Mode 3:

[0020] Reflective focusing optics M1, M2 can be replaced by refractive focusing optics consisting of lenses FL1, FL2 as shown in FIG. 6. Although theoretically, the mirror elements could be now be eliminated from the design by simply orienting beamsplitter's BS1, BS2 to reflect the light directly into eyepieces E1, E2 respectively, in practice, this would require the focusing optics FL1, FL2 to have an excessively short focal length. A more practical solution to this configuration is to replace concave mirrors M1, M2 with plain mirrors, leaving the beamsplitter's BS1, BS2 as in the previous configuration. This variation in the design greatly increases the required focal length of optics FL1, FL2 which ease's the design requirements.

[0021] Color Generation:

[0022] The generation of color in this HMD design can be achieved with either spatial color (red/green/blue picture elements) or by temporal color (display of rapid succession of red green and blue image fields usually by changing the colour of the light incident on the display device) and is dependent upon the display devices utilized.

[0023] Alternative Configurations:

[0024] The alternative configurations of this HMD as shown in FIGS. 1-7 simply highlight alternative configurations for a specific component of the overall system. It is of course possible to combine any of the configurations shown in FIGS. 1-7 to form an overall system with the desired characteristics.

[0025] The rotational orientation of the sub assembly consisting of light source LS3, beamsplitter BS3 and optional components RD1, CL1 and its counterpart consisting of light source LS4, beamsplitter BS4 and optional components RD2, CL2 is not a factor defining the intellectual property and the illustrations showing a particular configuration should not be considered a reduction in the generality of this specification.

Claims

1. A compact display device that is capable of utilizing either reflective or transmissive display technologies for the generation of images for a helmet mounted display device (HMD) or other compact display device.

2. A compact display device that is capable of utilizing a combination of reflective and transmissive display technologies for the generation of images for a helmet mounted display device (HMD) or other compact display device.

3. A display device as described in claims 1,2 which generate two separate (stereoscopic) images.

4. A display device as described in claims 1,2 which generates a single monoscopic image.

5. An optical assembly as shown in FIGS. 1,2 which consists of mirror M1, non-polarized beamsplitter BS1, polarized beamsplitter BS3, eyepiece E1, light source LS3, reflective display device RD1, collimating lens CL1 and optional components consisting of infrared filter F1, camera ETC1, and infrared light sources LS1.

6. An optical assembly (assembly-1) as described in claim 5 and shown in FIGS. 1,2, in which light from light source LS3 passes through polarizing beamsplitter BS3 and in doing so becomes plane polarized. This plane polarized light is then incident upon collimating lens CL1. This light passes through collimating lens CL1, becoming partially collimated as it does so. This partially collimated light is reflected from the surface of reflective display device RD1 and in doing so alters the plane of polarization of the reflected light according to the state of each individual pixel on the display device. As this reflected light passes once again through collimating lens CL1 it becomes substantially collimated. The reflected light which has undergone a rotation in the plane of polarization then reflects from the hypotenuse of polarizing beamsplitter BS3 and is directed towards beamsplitter BS1. As beamsplitter BS1 is non-polarizing, a portion of the light incident upon it will pass through the beamsplitter towards mirror M1 the other portion of the light will reflect off the hypotenuse of beamsplitter BS1 towards Infrared filter F1. As light source L3 contains no infrared component, this light will be absorbed by F1. The light that strikes concave mirror M1 will form a real image at some distance from M1 such that the light from this image will once again be partially reflected from the hypotenuse of beamsplitter BS1. This reflected light will enter E1 and form an image which is viewable by the user of the HMD.

7. An optical assembly as described in claim 5 and shown in FIG. 1, in which camera ETC1 generates an image of the user's eye which is interpreted by digital processing electronics to determine the position and orientation of the user's eye with respect to the eyepiece E1. This information can then be used to determine both the correct inter-ocular spacing for the eyepieces and the line of sight for the left eye.

8. An optical assembly as described in claim 5 and shown in FIG. 3, in which camera ETC1, infrared filter F1 and light sources LS1 are replaced with correctional optics assembly CO1. This optics assembly which consists of a lens or group of lenses is designed to generate a real image of the “real world” which will then be optically combined “superimposed” with the projected image from optical assembly 1 by virtue of beamsplitter BS1.

9. An optical assembly as described in claims 5-6 and shown in FIG. 4 such that polarizing beamsplitter BS3 is replaced with a non-polarizing beamsplitter and additional plane polarizers P1, P3 are placed between light source LS3 and the beamsplitter BS3 and between the beamsplitter BS3 and beamsplitter BS1 respectively, to achieve a similar optical characteristic.

10. An optical assembly as described in claims 5-6 and shown in FIG. 5 such that polarizing beamsplitter BS3 is replaced with a non-polarizing beamsplitter and an additional plane polarizer P1 is placed between beamsplitter BS3 and the reflective display device RD1 to achieve a similar optical characteristic.

11. An optical assembly as described in claims 5-6 and shown in FIG. 6 such that concave mirror M1 is replaced by a combination of lens or lens elements FL1 and plane mirror M1 to achieve a similar optical characteristic.

12. An optical assembly as described in claims 5-6 and shown in FIG. 7 such that non-polarizing beamsplitter BS1 is replaced with a polarizing beamsplitter and an additional phase retarder PR1 is placed between beamsplitter BS1 and mirror M1 to achieve a similar optical characteristic.

13. An optical assembly as shown in FIGS. 1,2 which consists of mirror M2, non-polarized beamsplitter BS2, polarized beamsplitter BS4, eyepiece E2, light source LS2, reflective display device RD2, collimating lens CL2 and optional components consisting of infrared filter F2, camera ETC2, and infrared light sources LS2.

14. An optical assembly (assembly-2) as described in claim 13 and shown in FIGS. 1,2, in which light from light source LS4 passes through polarizing beamsplitter BS4 and in doing so becomes plane polarized. This plane polarized light is then incident upon collimating lens CL2. This light passes through collimating lens CL2, becoming partially collimating as it does so. This partially collimated light is reflected from the surface of reflective display device RD2 and in doing so alters the plane of polarization of the reflected light according to the state of each individual pixel on the display device. As this reflected light passes once again through collimating lens CL2 it becomes substantially collimated. The reflected light which has undergone a rotation in the plane of polarization then reflects from the hypotenuse of polarizing beamsplitter BS4 and is directed towards beamsplitter BS2. As beamsplitter BS2 is non-polarizing, a portion of the light incident upon it will pass through the beamsplitter towards mirror M2 the other portion of the light will reflect off the hypotenuse of beamsplitter BS2 towards Infrared filter F2. As light source L4 contains no infrared component, this light will be absorbed by F2. The light that strikes concave mirror M2 will form a real image at some distance from M2 such that the light from this image will once again be partially reflected from the hypotenuse of beamsplitter BS2. This reflected light will enter E2 and form an image which is viewable by the user of the HMD.

15. An optical assembly as described in claim 13 and shown in FIG. 1, in which camera ETC2 generates an image of the user's eye which is interpreted by digital processing electronics to determine the position and orientation of the user's eye with respect to the eyepiece E2. This information can then be used to determine both the correct inter-ocular spacing for the eyepieces and the line of sight for the right eye.

16. An optical assembly as described in claim 13 and shown in FIG. 3, in which camera ETC2, infrared filter F2 and light sources LS2 are replaced with correctional optics assembly CO2. This optics assembly which consists of a lens or group of lenses is designed to generate a real image of the “real world” which will then be optically combined “superimposed” with the projected image from optical assembly-2 by virtue of beamsplitter BS2.

17. An optical assembly as described in claims 13,14 and shown in FIG. 4 such that polarizing beamsplitter BS4 is replaced with a non-polarizing beamsplitter and additional plane polarizers P2, P4 are placed between light source LS4 and the beamsplitter BS2 and between the beamsplitter BS4 and beamsplitter BS2 respectively, to achieve a similar optical characteristic.

18. An optical assembly as described in claims 13,14 and shown in FIG. 5 such that polarizing beamsplitter BS4 is replaced with a non-polarizing beamsplitter and an additional plane polarizer P2 is placed between beamsplitter BS4 and the reflective display device RD1 to achieve a similar optical characteristic.

19. An optical assembly as described in claims 13,14 and shown in FIG. 6 such that concave mirror M2 is replaced by a combination of lens or lens elements FL2 and plane mirror M2 to achieve a similar optical characteristic.

20. An optical assembly as described in claims 13,14 and shown in FIG. 7 such that non-polarizing beamsplitter BS2 is replaced with a polarizing beamsplitter and an additional phase retarder PR2 is placed between beamsplitter BS2 and mirror M2 to achieve a similar optical characteristic.

21. A display device as described in claims 1-3 which consists of two separate optical assemblies as described in claims 5-20 which are used to deliver a separate image to each eye of the user.

22. A display device as described in claims 1-2,4 which consists of a single optical assembly as described in claims 5-20 which is used to deliver an image to one eye of the user.

23. A display device as described in claims 1-3,21 that uses the eye position and direction information from both the left and right eyes to determine the 3-D spatial coordinates of the user's gaze.

24. A display device as described in claims 1-4,21-22 that uses the eye position and direction information to determine the “line-of-sight” of the user's gaze.

25. A display device as described in claims 1-3,21,23,24 that uses the eye position information to automatically adjust the lateral position of optical assemblies 1,2 (as described in claims 5-20) such that the said optical assemblies will be in the optimal position for the display of images to the user.

26. A display device as described in claims 1-2,4,22,24 that uses the eye position information to automatically adjust the lateral position of a single optical assembly as described in claims 5-20 such that the said optical assembly will be in the optimal position for the display of images to the user.

27. A display device as described in claims 1-3,21,23-25 and shown in FIGS. 1-7 that allows for the independent adjustment of the lateral position of mirrors M1, M2 in addition to the lateral adjustment of optical assemblies 1,2 (as described in claims 5-20) such that the said optical assemblies will be in the optimal position for the display of images to the user.

28. A display device as described in claims 1-2,4,22,24,26 and shown in FIGS. 1-7 that allows for the independent adjustment of the lateral position of mirrors M1, M2 in addition to the lateral adjustment of optical assemblies 1,2 (as described in claims 5-20) such that the said optical assemblies will be in the optimal position for the display of images to the user.

29. A display device as described in claims 1-3,21,23-25,27 and shown in FIGS. 1-7 that incorporates some form of electromechanical means for the adjustment of optical assemblies 1,2 (as described in claims 5-20) and/or the independent adjustment of mirrors M1, M2.

30. A display device as described in claims 1-2,4,22,24,26,28 and shown in FIGS. 1-7 that incorporates some form of electromechanical means for the adjustment of a single optical assembly (as described in claims 5-20) and/or the independent adjustment of mirrors M1 or M2.

31. A display device as described in claims 1-3,21,23-25,27,29 and shown in FIGS. 1-7 which consists of two separate optical assemblies as described in claims 5-20 which utilize a single transmissive type display device (such as an AMLCD) positioned between beamsplitter's BS3 and BS4 to deliver a separate image to each eye of the user and which may or may not include the reflective display devices RD1, RD2.

32. A display device as described in claims 1-2,4,22,24,26,28,30 and shown in FIGS. 1-7 which consists of a single optical assembly as described in claims 5-20 which utilizes a single transmissive type display device (such as an AMLCD) positioned between beamsplitter's BS3 and BS4 to deliver a separate image to each eye of the user and which may or may not include the reflective display devices RD1 or RD2 and which may have the light source LS3 or LS4 projecting light directly through the transmissive display device in a “back lit” mode of operation as an alternative to utilizing the beamsplitter BS3 or BS4 to reflect the light onto the transmissive display device.

33. A display device as described in claims 1-3,21,23-25,27,29 and shown in FIGS. 1-7 which consists of two separate optical assemblies as described in claims 5-20 which utilize a single transmissive type display device (such as an AMLCD) positioned between beamsplitter's BS3 and BS4 to deliver the same image to each eye of the user and which may or may not include the reflective display devices RD1, RD2.

34. A display device as described in claims 1-6,8-14,16-22,27-33 which does not incorporate the optional components ETC1, F1, LS1, ETC2, F2, LS2.

35. A display device as described in claims 1-7,9-15,17-33 which utilizes one or more light sources LS1, LS2 as shown in FIG. 1, such that the said light sources will produce a strong reflection or reflections from the cornea of the user's eye in addition to providing other visual information such as the relative position of the user's pupil and/or iris for the purpose of providing the visual information for determining the position and orientation of the user's eye.

36. An eye tracking module which utilizes the optical configuration as described in claims, 5-7,13-15,35 and shown in FIG. 1 which can determine both the position and orientation of an individual eye of the user and from this information can calculate both the mis-alignment between the eyepiece optics E1,E2 and the viewer's eye and the “line-of-sight” for that particular eye

37. An eye tracking system which utilizes two separate “eye-tracking” modules as described in claim 36 which can calculate both the inter-ocular spacing of the particular user wearing the display system and can determine the 3-D spatial coordinates of the user's gaze.