Patent Application
20050083516
The present invention relates to display technology. Advances in display technology have made high resolution displays widely available. To achieve this level of wide spread availability, advances have been made in the materials used to construct the display devices as well as the drivers and graphics processors used to render images on the devices. Display devices capable of delivering high resolution images are not limited to traditional, cathode-ray tub (CRT) but also include liquid crystal display (LCD), plasma screens and others. New display technologies also take advantage of high speed processors and advanced image processing methods to render images.
In addition to traditional display, these advanced materials and technologies have been applied to render images so that they appear three-dimensional rather than flat or two dimensional on the surface of the screen. There are two principal stereo or three-dimensional technologies for detached (i.e., not head-mounted) display. The first display technology presents two images on a screen and requires the observer or user to wear glasses that block one image or the other during viewing. The glasses include polarized glasses, colored glasses and shuttered glasses that selectively let different images pass through the glasses to each eye.
Autostereoscopic is a second type of technology that renders stereo or three-dimensional images without glasses. The autostereoscopic displays perform the three-dimensional imaging within the display without relying on other processing or peripheral equipment like glasses or eyewear. Instead, the displays are outfitted with an objective containing numerous refractory elements that direct different images to each eye of the observer. These autostereoscopic displays currently require the user to remain relatively still as the images are directed to different view zones corresponding to each eye. Further, the quality or resolution of the image is controlled by the resolution of the underlying display technology and the optical characteristics of the objective overlaying the surface of the display.
Unfortunately, the objectives used to perform autostereoscopic displays may have inherent defects, aberrations or impurities that degrade the images they produce in the autostereoscopic display. This not only affects the quality of the images but more importantly degrades autostereoscopic display as it is more difficult to control the direction in which the emitted energy moves through the objectives and into one or more view zones. Despite the inherent imperfections and drawbacks, conventional autostereoscopic displays continue to use these objectives having these inherent defects.
The relatively few people using these autostereoscopic display devices in various niche applications tolerate the relative expense, limitations of acuity and resolution the autostereoscopic devices can provide in exchange for stereo imaging. This will change as display resolution increases and the number of people using autostereoscopic systems grows. Eventually, the effect of these inherent defects will become more apparent and less tolerable as use of autostereoscopic displays becomes more commonplace. Indeed, more cost-effective design and production methods are needed if usage of autostereoscopic displays is to become widespread.
Features of the present invention and the manner of attaining them, and the invention itself, can be understood by reference to the following detailed description of embodiments of the invention, taken in conjunction with the accompanying drawings and schematics, wherein:
Like reference numbers and designations in the various drawings indicate like elements.
One aspect of the present invention features a method of calibrating an objective. The calibration includes receiving the objective over a raster-organized surface having both image display and image acquisition modalities, positioning a calibration model one or more positions position before the objective and the raster-organized surface in preparation for acquiring images of the calibration model, receiving images of the calibration model through the objective and onto raster-organized surface in an acquisition mode, identifying optical characteristics of objective through a comparison of received images of the calibration model.
Another aspect of the invention includes a method of displaying images using an objective. The displaying of images includes receiving the objective over a raster-organized surface having both image display and image acquisition modalities, loading a calibration vector corresponding to the objective that compensates for optical characteristics of the objective when used in both a display mode and an acquisition mode and displaying images through the raster organized surface and objective compensated in accordance with the calibration vector for the objective.
Aspects of the present invention are advantageous in at least one or more of the following ways. An optical objective (hereinafter an objective) used with a display device can be readily calibrated for improved resolution and reduced distortion caused by imperfections or aberrations in the objective. In autostereoscopic applications, the objective containing many micro refractive elements is placed over a raster organized surface to display images in three-dimensions. Calibration of the objective results in three dimensional images that have less distortion, appear more realistic and potentially cause less eye strain. In general, calibration of the objective is important as resolution of autostereoscopic and other display modes increase dramatically making imperfections more apparent and detrimental to the quality of the images being projected.
Calibrating the objective used in autostereoscopic and other displays also facilitates lower costs/unit as digital processing can compensate for imperfections in the objectives. Instead of discarding flawed objectives used in autostereoscopic and other applications, implementations of the present invention can be applied to compensate and accommodate for inherent aberrations or imperfections in the refractive portions of the objective. This effectively increases the yield of manufacturing these objectives without significant retooling or reconfiguring of existing manufacturing facilities.
Yet another benefit, implementations of the present invention calibrates an objective using a raster organized surface capable of both image display and image capture modalities. To calibrate the objective, the raster organized surface receives an object's image through the objective in image capture mode and then compares the results with other observations of the object. Inherent imperfections in the objective are noted and compensated in subsequent image capture and image display operations. This is advantageous as calibration can be achieved without complex additional cameras and feedback loops. Instead, the display having the dual mode capture and display modalities is able to measure the variation in the objective directly.
A first application includes a three-dimensional video system achieved by switching between image display and image acquisition modalities using displays nearest first eyes 102 and second eyes 106 respectively. The autostereoscopic implementation of the present invention allows first eyes 102 and second eyes 106 to be repositioned relative to their respective display and assume different viewpoints. People in video conference meetings with viewpoints of first eyes 102 or second eyes 106 may obtain different angles of facial expressions, gestures or other cues by moving eye location relative to the corresponding display rather than switching to cameras with different fixed perspectives. Calibrating the objectives in accordance with the present invention is important to presenting a consistent three-dimensional view even though nearby lenslets or lenticules used during operation have different inherent inconsistencies or aberrations. By calibrating and adjusting the image, stereoscopic images appear more consistent and realistic to parties viewing an autostereoscopic display.
Calibrating the refractive elements in accordance with the present invention also provides higher quality images than otherwise possible as inherent imperfections in the lenses are corrected in real-time as the images are acquired or displayed. Implementations of the present invention can also use eye tracking operations to identify areas of a display being viewed and increase resolution, reduce distortion, or suppress transmission outside of the localized area, rather than putting resources into all areas of the display uniformly.
Other applications 110 also benefit from calibration operations of the present invention as implemented in autostereoscopic displays and other displays using an objective. These other applications 110 include medical 114 (surgical simulation/training, surgical imaging and general imaging) hazardous materials 116 (managing relocation and disposal of hazardous waste), business 118 (presentations, financial 3D modeling, trade shows, reception areas/lobbies and industrial inspection), military 120 (simulations, training, heads-up displays, cockpit controls, satellite imaging, data reconnaissance and analysis), real estate 122 (virtual real estate tours, sales presentations, virtual property inspections and walk-through) and entertainment 124 (movie theatres, personal gaming devices and personal computer games).
In one or more of these applications, data gathering cameras and equipment collect information for three-dimensional representation which is then displayed in three dimensions on first calibrated display 104, second calibrated display 108 for eyes 102 or eyes 106 respectively and any other number of displays and viewing eyes. Of course, many other applications not specifically described herein that benefit from higher resolution images and three-dimensional viewing could also benefit by implementations of the present invention.
In one implementation, display material 202 is composed from one or more different raster organized surface capable of operating in both an image display and an image acquisition modality. The raster organized surface can be constructed from adjacent emitting elements and sensing elements to perform the image display and image acquisition modalities respectively. For example, the emitting elements could be implemented using liquid crystal display (LCD) and light emitting diode (LED) components while the sensing elements could be constructed from photoreceptor materials.
Alternatively, the raster oriented surface could also be created using dual-purpose elements configured to perform both image display and image acquisition modalities under a control signal. In one implementation, the dual-purpose elements could be configured from an organic light emitting device (OLED) material that emits energy to perform image display under control of a first control signal and senses energy to perform image acquisition when under control from a second control signal.
In either of these or other implementations, objective 204 is a type of lens mounted on a raster display surface 202 that facilitates directing visual information to view zones 206 suitable for viewing eyes 208. To provide autostereoscopic viewing by viewing eyes 208, different colors or visual information are directed to different eyes thereby giving a three-dimensional effect to a person positioned at viewing eyes 208.
Objective 204 can be an array composed of many lenslet type refractive elements or many lenticule type refractive elements also arranged in an array structure. The lenslet shape corresponds to a slice of sphere and provides control over the refraction of light in two dimensions. In contrast, an objective having lenticules is also composed of an array or matrix of refractive elements but provides control over the refraction of light in essentially only one direction rather than two.
The number of view zones depends on the resolution of the underlying raster organized surface 202 and the ability for objective 204 to project regions in space, described as view zones, and into each eye of viewing eyes 208. In one implementation, the number of view zones projected corresponds to the number of subpixels in the set of subpixels corresponding to each lens of the objective. For example, the n view zones 206 corresponds to n subpixels for each lenslet or lenticule of objective 204.
Moreover, the number of view zones needed for autostereoscopic display also depends on the intraocular distance i between viewing eyes 208 and the ability for an autostereoscopic display to keep the correct view zone over the corresponding left or right eye as may be appropriate. On conventional systems, this would generally require that viewing eyes 208 remain fixed or not move beyond a very small range of a particular view zone, depending on the correctness of the optical elements.
However, an objective calibrated in accordance with the present invention is able to keep images projected into these view zones 206 positioned most appropriately for the viewer. Calibrated objectives of the present invention are able to better direct images to the correct eye of viewing eyes 208 even if viewing eyes 208 inadvertently move or shift to different pairs of view zones 206 or positions within view zones. With calibration of the present invention, more view zones seem present as the effective acuity in the objective is increased. Accordingly, viewing eyes 208 are less likely to detect a switch between lenslets or lenticules as the calibration reduces or eliminates imperfections or differences between the refractory elements and the images they produce between the view zones.
Further, combining calibration of the present invention with eye tracking features facilitates movement of viewing eyes 208 from one view zone to another without a characteristic jump, “black-out” or “null” area as the transition between view zones 206 takes place. By tracking eye location, implementations of the present invention can dynamically increase resolution or even the number of effective view zones 206 near or adjacent to the area being seen by viewing eyes 208. In effect, this allows individual view zones to take on the character of multiple view zones, with varying display dependent upon the position of the eye within the zone.
In one implementation, semiconductor fabrication techniques are used to form pixel 212 and other pixels in the sequence upon bottom plate 210. In response to a bias signal or other control signal, one or more subpixels in pixel 212 direct different color values into areas of space. In this example, pixel 212 and other pixels have only three subpixels as indicated by the triplet of values (i.e., 123) however alternate implementations can have greater or fewer subpixels as needed depending on the number of view zones implemented.
To facilitate fabrication, objective 214 is a monolithic array of lenslets placed over the raster organized surface and calibrated in accordance with the present invention. Depending on the application, lenslets are manufactured and grouped into square, hexagonal or random shapes and secured to upper plate 211. Alternative implementations may fabricate the lenslet or lenticular array along with the semiconductor layer that lies beneath it. In this example, each lenslet in objective 214 projects three different view zones depending on the location and position of eyes in the view zone in the space over objective 214 as exemplified by first lenslet 216 or second lenslet 218. The light rays from first lenslet 216 provide colors from the 2nd subpixel into a second view zone while the light rays passing through lenslet 218 provide colors from the 1st subpixel into a first view zone. Other lenslets in objective 214 also contribute to the three different view zones but their contribution has been omitted in this example for purposes of explanation. In an alternative implementation, objective 214 is made up of one or more micro lenticules organized in a monolithic columnar array that refracts light in a single dimension.
Lenslets 302 are mounted securely over a raster organized surface mount 306 by way of a frame and optionally an adhesive material if the mounting is to be permanent. Raster organized surface mount 306 has a communication port 307 to transmit calibration data to calibration storage area 304 of lenslets 302 by way of objective calibration processing component 305. In one implementation, raster organized surface mount 306 has X×Y raster elements grouped in pixels and subpixels to match lenslets 302 in a desired ratio between lenslets and pixels. Many different dimensional arrays of subpixels can be positioned beneath a variety of different lenslets or lenticules in an objective depending on the numerosity and type of view zones to be produced.
Objective and raster compound 308 are put in calibration mode in accordance with the present invention and presented with a calibration model 310 and light source 314. Raster elements are biased or otherwise given a control signal to record image of calibration model 310. Calibration model 310 is positioned at a variety of locations before the display and multiple views of are acquired. Objective calibration processing component 305 processes these views to determine the deviation of the objective's individual lenses from a perfect lens and generates a calibration vector reflecting this information. The calibration vector can be stored directly in calibration storage area 304 of lenslets 302 or externally in a CD-ROM 316, diskette or other storage to be shipped with the objective by the manufacturer. Once again, this calibration vector is loaded from storage by a device driver in a computer at a subsequent time period to compensate for the inherent errors of the objective used in an autostereoscopic or other type of display device.
In one implementation, an objective in objective and raster compound 308 is removed during manufacturing to make room for calibrating other objectives on raster organized surface mount 306. A first objective is calibrated and then distributed with calibration information for mounting on one or more different types of raster organized surfaces including LCD or LED surfaces as previously described. Alternatively, objective and raster compound 308 are not separated but sold as an integral unit for one or more different types of displays requiring calibrated objectives for higher resolution, quality imaging applications and improved autostereoscopic imaging.
Objective and raster compound 308 also has the ability to act as both an image display and image acquisition device. On occasion, a control signal sent to objective and raster compound 308 causes the raster surface to enter into image acquisition mode rather than an image display mode. Multiple images are obtained by the various pixels and subpixels under each lenslet or lenticules of the objective acting as small cameras.
The multiple images obtained have many uses. First, the multiple images can be superimposed to create images with super-resolution results and improve calibration operations in accordance with the present invention. Further, the super-resolution images can be used for highly accurate eye tracking of viewing eyes 320. Typical images can be greatly enhanced in accordance with the present invention by interlacing image acquisition for eye tracking with image display. For example, the eye location information is used to increase resolution of the autostereoscopic display especially in the areas that a viewer is observing the display. Even without super-resolution processing, the multiple different images can be transmitted to another person and viewing eyes receiving three-dimensional images over an autostereoscopic three dimensional video conferencing system or any other application that could use the multiple images acquired.
Next, implementations of the present invention then position a calibration model before the raster-organized display surface and objective (404). The calibration model is selected to best bring out imperfections or aberrations of the objective in accordance with the present invention. For example, the calibration model can be a Macbeth color chart or other calibration images well known in the art. Calibration may occur during manufacture or in other controlled environments to ensure the calibration is performed adequately and consistently. Alternatively, a user may perform calibration subsequent to manufacturing while installing the objective and display device onto a computer system. In either case, lighting as well as other conditions on the calibration model may need to be carefully maintained.
Once the calibration model is positioned, implementations of the present invention receive images of the calibration model through objective and onto raster-organized surface in acquisition mode (406). Multiple versions of the image are captured as images are received by the raster organized surface. Calibration comes from comparison of elements from each of the multiple image to all others (408). The calibration model is moved around before the system so as to cover as much of the field of view of the imagers as possible. Images collected are compared and contrasted with each other and processed during the calibration process to increase the accuracy of the measurements. These calculations are used to determine the calibration vector for all components of the objective.
For later use, implementations of the present invention record a calibration vector to compensate for optical characteristics of objective during both display and acquisition modes (410). The calibration vector includes information to compensate for specific lenslets or lenticules in the objective being measured and calibrated. For example, the calibration vector may identify lens distortion, focal length, and related optical and geometric properties of the lens system. This calibration vector can be multidimensional or even provide references to another database or other external reference with more information about the objective calibrated. As previously described, the calibration vector can be stored on the objective directly in a storage area or externally using a CD-ROM, floppy-diskette or other type of removable media.
In either of these or other implementations, the calibration operation loads a calibration vector corresponding to the objective to compensate for optical characteristics of the objective (504). As previously described the calibration vector stores particular optical characteristics of individual objectives and can take many different types of storage format. The vector can be a simply array, a matrix of values, a compact relational database or may even be an object within an hierarchical class structure for a complex object oriented driven database system.
In accordance with the present invention, the display of images through raster organized surface and objective are compensated in accordance with the calibration vector (506). For example, the images provided in an autostereoscopic display are altered in real-time as still or video images are presented and viewed by a pair of eyes from an observer. The effect is improved acuity of the still and/or video images as they are presented to and observed by one or more of the observers properly positioned in the several different view zones. Areas compensated for during calibration accommodate for variations in focal length, aspects ratios, skew characteristics, principal point and center of distortion, relative position and orientation in space as well as lens distortion, color quality, and any other deleterious influences on the image formation process.
Though often used in image display mode, a raster organized surface of the present invention on occasion switches to an image acquisition mode for various purposes. In a video conference system, image acquisition mode allows the raster organized surface to act as a camera for a moment and thereby allow parties to exchange visual information with little change in perspective—especially when compared with a remotely mounted and separate camera device; this also allows the video conference system to operate with a minimal amount of space and equipment. Additionally, the images acquired can be used to provide features related to multiviewpoint and eye tracking driven enhancements.
Other enhancements are also possible by tracking eye movement and displaying information based on the location of the eyes relative to the view zone in an autostereoscopic or other type of display. Accordingly,
Eye tracking results are used to then enhance portions of the image being generated. In one implementation, the view zone is adjusted and displayed by raster-organized surface according to the location of eyes (610). The content of view zones used in an autostereoscopic display can be adjusted to tailor the display to the locations of eyes viewing the screen. Colors and other images can be presented precisely where the eyes appear in the view zone based on the eye tracking information. This is an improvement over systems that present visual information at an average position in a view zone, with content that is unchanging throughout that zone.
Further, implementations of the present invention may also increase resolution on raster-organized surface in display mode based on the eye location information (612). This enhancement would increase the movement-observed resolution of view zones nearer to the detected eyes and give the effect of an increased number of view zones. Using the present invention to track the viewer's eyes reduces the likelihood that the viewer's eyes see discontinuous content at transitions between view zones and experience jerky percepts at those transitions. Because of the improved knowledge of both viewer eye position and the geometric character of the display function, the percept received more closely reflects the reality that is meant to be conveyed—inaccuracies or imprecisions in the three-dimensional nature of the display will be minimized.
In one implementation, memory 702 includes an objective calibration processing component 718, objective calibration vector generation component 720, a multiple sample image reconstruction component 722, and a run-time module 726 that manages system resources used when processing one or more of the above components on system.
Objective calibration processing component 718 calibrates objectives in accordance with the present invention and as previously described. In operation, objective calibration processing component 720 analyzes the data and determines the calibration information to be associated with the objective. Multiple sample image reconstruction component 722 creates a single high quality image reconstructed or integrated from multiple slightly different copies of the image being received. This can be used for super-resolution as previously described.
While examples and implementations have been described, they should not serve to limit any aspect of the present invention. Accordingly, implementations of the invention can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Apparatus of the invention can be implemented in a computer program product tangibly embodied in a machine-readable storage device for execution by a programmable processor; and method steps of the invention can be performed by a programmable processor executing a program of instructions to perform functions of the invention by operating on input data and generating output. The invention can be implemented advantageously in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. Each computer program can be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language if desired; and in any case, the language can be a compiled or interpreted language. Suitable processors include, by way of example, both general and special purpose microprocessors. Generally, a processor will receive instructions and data from a read-only memory and/or a random access memory. Generally, a computer will include one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM disks. Any of the foregoing can be supplemented by, or incorporated in, ASICs.
While specific embodiments have been described herein for purposes of illustration, various modifications may be made without departing from the spirit and scope of the invention. Accordingly, the invention is not limited to the above-described implementations, but instead is defined by the appended claims in light of their full scope of equivalents.
