Patent Application
20160086370
This invention relates to digital video display systems, and more particularly to operating systems for large-scale displays composed of a plurality of individual display devices.
Digital imaging systems display visual information utilizing a control source and a display. The control source, such as a computer system, may utilize mapping functions to enable applications such as combining multiple projected images into one image to make a super resolution display. Displays include any devices that are capable of visually showing information, still pictures, or video (moving pictures). A display generally consists of one of more light sources, and a method of spatially modulating the light can be divided into (1) direct view displays, and (2) projection displays. Direct view displays include flat panels such as liquid crystal displays (LCDs) and plasma display panels (PDPs). Projection displays include technologies that use microdisplay panels such as (1) digital light processing (DLP) which are based on digital micromirror devices (DMDs), (2) transmissive LCD panels, and (3) liquid crystal on silicon (LCOS). The quality of a digital imaging system is determined by the expectations of the viewer. Quality factors include resolution, contrast, geometric accuracy, color accuracy, freedom from distracting artifacts and other performance properties which contribute to the generally pleasing nature of the image or allow the displayed digital image to accurately represent the original digital image or an image found in nature. It is desired to correct for certain inaccuracies in the image produced by a display by applying corrective measures to image information.
Three-dimensional imaging may refer to an image in the plane of the display that is rendered to appear with perspective. Three-dimensional imaging is also used to refer to stereoscopic digital imaging where the image comes out of the plane of the display to fill three-dimensional space. Stereoscopic digital imaging presents a different image to the left eye and the right eye in order to simulate the effect of viewing natural images in three-dimensional space. Stereoscopic imaging may contribute to the perceived quality of digital imaging systems if the three-dimensional images are properly presented to each eye without causing eyestrain. It is generally desired to provide a mechanism to support such image creation within a display system and improve overall display quality.
The present invention is a method and system for improving display quality by injecting a portion of computer code into an existing compositor operatively arranged and configured on a computing system, using the portion of computer code to apply a mapping function to a first digital image, and forming a second digital image based upon the first digital image as adapted by applying the applied mapping function. The second digital image is then displayed to a viewer via one or more displays wherein each display forms a part of the displayed digital image. The contribution of each display can be such that there is varying overlap, from partial overlap to substantially complete overlap. Substantially complete overlap enables the display of stereoscopic images. The mapping function can be used for modification of geometry, creation of blended images or correction of a characteristic (such as color, intensity, etc.) of the display system. The characteristic can be sensed using a detector such as a camera directed at the displayed image, or by internal sensing of the image. The portion of code can be injected into a graphics driver controlling hardware generating a portion of a displayed digital image. The overall display system may be based upon one or more computer systems each with one or more graphics cards connected to one or more display devices such projector.
The invention description below refers to the accompanying drawings, of which:
A detailed description of super resolution displays, in which the teachings of illustrative embodiments here can be based, may be found in U.S. Pat. No. 6,456,339, the teachings of which are entirely incorporated herein by reference as useful background information. Digital imaging systems may achieve high-resolution displays by combining more than one display such that the displays add their pixels together into one displayed digital image composed of multiple parts. Each part of the displayed digital image is composed primarily of pixels from an individual projector, but also has overlap regions from the adjacent displays. The parts of the displayed digital image are partially overlapping in order to perform gradual blending between the parts. The partial overlap regions can extend over approximately 10% to 30% of the area of the displayed digital image. A mapping function can be applied to the original digital image to split it into parts and to reassemble the parts for display. The mapping function is defined as a mathematical transformation which modifies a digital image to produce a second digital image which is changed in some way. A correction function is a specific type of mapping function which applies a desired correction to a digital image. The mapping function can include such transformations as warping, color matching, lateral color correction, or any other desired changes to the digital image. In some cases, the mapping function may degrade the image quality as measured in the traditional sense in order to achieve a specific desired effect.
An alternate embodiment of the system configuration makes use of substantially complete overlap of digital images from multiple displays (the displays often being arranged as a “multiplicity” of two or more, and at least a “plurality” of two or more in various embodiments). Substantially complete overlap is overlap of at least 90%. In one embodiment, the substantially complete overlap is aligned so that the parts from multiple displays are exactly aligned with accuracy down to the pixel or sub-pixel level. This configuration can be used to increase the amount of light by stacking multiple displays. In another embodiment, the system provides substantially complete overlap, but the parts consist of left eye and right eye images to enable stereoscopic display of three-dimensional images. Exact pixel alignment may not necessary in the case of stereoscopic projection.
A special case of stacking occurs when there is only a single projector. In this case, a lens can be used to wrap the image in such as way that one side wraps to meet the other side of the image.
Another alternate embodiment of the system configuration makes use of digital images that do not overlap. In this case, the parts may be joined at the edges or may have a gap region between the edges. In the case of gap between the edges, the image can be shifted so that important features of the displayed digital image do not fall in the gap region.
Another alternate embodiment of the system configuration employs a third and fourth display that partially overlaps when the first and second display have substantially complete overlap as a stereoscopic image.
Reference is now made to
Illustratively, the compositor 104 is the part of the computer operating system 102 that arranges various parts of the digital image that will be shown by the computer. In conventional computer operation, the compositor combines the windows displayed from various applications and the operating system. It also handles translucency of windows in certain operating systems, such as Microsoft's Windows Vista® operating system, which include a translucency capability.
The digital image can comprise a still image or a series of still images that make a moving image (also called video). The digital image may be generated in the computer or it may be generated externally from the computer. The digital image may be captured by the computer with an image capture card in the computer.
Projectors 114, 116, and 118 project modulated light beams 120, 122, and 124 respectively. Light beams 120, 122, and 124 form projected digital image 150 on screen 126. Sensor 112 senses the parameters of projected digital image 150 and sends the parameters to mapping software 160 which runs on mapping hardware 108. Mapping software 160 uses a mapping function to calculate mapping signals and sends the mapping signals to projectors 114, 116, and 118. In various embodiments, the mapping function is adapted to fit the graphics driver software and display. Projectors 114, 116, and 118 use the mapping signals to apply the mapping function to the digital image from graphics driver 130. Mapping hardware 108 can be computer hardware, such as a computer card that runs on a conventional computer, or mapping hardware 108 can be specialized hardware that is custom designed to perform mapping calculations. In an embodiment where mapping hardware 108 is computer hardware, it can comprise a card that is installed into computer 100, or it can be installed into a separate computer dedicated to performing the mapping calculations. In another embodiment where mapping hardware 108 is a computer hardware, it can comprise a processing unit and a digital communication link configured to receive an image data stream from a sensor. The compositor software is shown as being part of the operating system in
More particularly, in an illustrative embodiment, the interface to software such as a driver or operating system is accomplished via an application programming interface (API) 360, 362, or equivalent mechanism know to those ordinarily skilled in software development. APIs 360, 362 can respectively reside within the compositor software 304, the graphics driver software 330 and/or within another application as appropriate. The API provides a communications mechanism between software modules (where such modules can be developed by different sources or entities at different time) in order to support the building of applications. Illustratively, calls through the API by another program or process cause the execution of code that depends upon the information in the call. It is contemplated to modify a stream of execution by intercepting calls to the API, inserting replacement and/or different code to execute, and then either (a) continuing the call as normal or (b) returning to the caller. This is typically accomplished by modifying the existing API interface by copying one or more of the existing API pointers to the existing application code and replacing them with new pointers (depicted in API blocks 360, 362) pointing to new code for execution. That new code can then either (a) return to the code that called the API, or (b) continue the call through the old code via the copied API pointers which then return as needed. In an illustrative embodiment, the graphics driver 330 provides an API 362 that allows programs (for example the operating system 302) to make calls that cause code at the in the graphics driver 330 to be executed (for instance, to paint a pattern at a particular location). The present invention can intercept the API calls to the graphics driver 330 by copying the original API pointer and inserting a new pointer to point to the mapping code (308). Thus, during execution, if a particular API call is made, the mapping code can modify the data in the API call and then pass the stream of code execution to the pre-existing API code via the copied API pointer.
By way of operational example, if data is initially formatted for display on a certain type/model of display, such as a flat screen, then the mapping code can be adapted to intercept the data at the graphics driver level so as to modify for display on a different type/model of display, such as a screen with a different geometry. The mapping code can then be passed along for display via the original graphics driver. While not shown, the depicted API and pointer arrangement, or an appropriate variation thereof, is applicable to the embodiments of
In the arrangements of
Projectors can be embedded in a variety of electronic devices such as cell phones, handheld game systems, laptop computers and/or other personal electronic devices. In an illustrative implementation personal electronic device with a single projector can employ a mapping function in accordance with the embodiments herein for such purposes as warping menus or combining images with other personal electronic devices.
Also, in the exemplary arrangements of
It should be apparent that one advantage of injection-based digital imaging systems such as those shown in
Another advantage of injection-based digital imaging systems is that they may be used with conventional off-the-shelf (COTS) projectors and COTS graphics cards. The digital imaging system shown in
Yet another advantage of injection-based digital imaging systems is that they can make use of the inherent hardware acceleration in various commercially available computer graphics hardware. A system with separate hardware and software for mapping would generally be unable to utilize this hardware acceleration.
Illustratively, the use of mapping functions can be divided into two types: (1) application-dependent and (2) application-independent. Injection enables application-independent mapping with COTS graphics cards without the need for specialized hardware.
Some further considerations with respect to the mapping functions and alternate embodiments of the system and method are now described.
In an illustrative embodiment, the calculation of the mapping function generally includes sampling pixels in the digital image, calculating a new position for each pixel, and then assembling a mapped digital image with the pixels in the new position, also called re-sampling. The mapping function can also include other transformations of the pixels such as intensity or color changes. Intensity mapping of pixels is also referred to as shading. Interpolation between pixels is desired during the re-sampling. The interpolation can be performed using closest neighbor, linear, or nonlinear calculations according to conventional algorithms thereof. It is contemplated that a spatially varying filter can be included in the mapping function in order to change white level (contrast), black level (brightness), or other image parameters dependent on the spatial position and content in the digital image. The spatially varying filter can be used to control blooming or other secondary effects introduced by the display.
A variety of mapping functions can be performed depending on the requirements of the digital imaging system. The mapping functions can include warping, blending, and color matching. Warping is used, for example, to compensate for different angles of projection from multiple projectors, or to make digital images fit onto curved screens so that minimal distortion is apparent when viewing the projected digital image. Twisting and zooming may be considered two types of warping used to align multiple parts of an image to fit together when forming one projected digital image from multiple parts. Blending makes the edges of the multiple parts fit together when forming one projected digital image from multiple parts. Color matching is used to make multiple parts of the projected digital image match each other so that they appear to come from one projector. Color matching is particularly desirable when multiple projectors exhibit differing color output. The least expensive COTS projectors frequently benefit from color matching because loose manufacturing tolerances can result in large color differences between projectors of the same model. The mapping functions allow multiple projectors to be placed in position with only approximate alignment. The mapping function is then calculated to stitch together the parts of the image in order to form one digital projected image out of the multiple parts projected by each projector.
The mapping function can be a fixed transformation or it may be a temporally-varying transformation. In the temporally-varying case, the transformation may be calculated from sensor feedback or it may be calculated based on other external parameters. The calculation may take place in real time or in near real time.
The mapping function can be entirely determined by computer calculation (automatically), or it can be partially or completely supplied by user input (manually). By way of example, the user may wish to directly control the position of certain elements in the digital image with means to manipulate the image directly such as a computer mouse or other user input device.
Injection-based digital imaging systems can also be used to provide visual information such as messages to viewers. The visual information may consist of elements generated by the digital imaging system, such as menu information, device control information, notifications, or system status information. Visual information may further be created by other software operatively running on the computer hardware system, such as, but not limited to, presentation application, gaming applications, or various application GUI widgets. The visual information may further consist of content external to the digital imaging system that is operatively communicated to the digital imaging system through a communication or network link. By way of example, external content information can be communicated from a server housing a database of information, or a server hosting website information. The information can be in the form of video, image, or other data forms. By way of further example, the communication system employed can be, but not limited to, digital system such as, Wi-fi, Wimax, cellular system, and other various digital communication system. The communication system can further be configured to receive analog signal from cable operator, from analog antennae, etc. Video content can further be streamed using conventional mechanisms apparent to those skilled in the art. The information contained in the message may be inserted by the application software or at any other point in the software chain prior to and including the point of injection. The visual information may allow the user to control parameters of the projected digital image or the equipment used for projection. These parameters may include the color of the white point or other image characteristics. A graphical user interface associated with, for example the above-described computer and operating system can be utilized for toggling the injection on and off.
The injection can be toggled on or off automatically under software control. For example, if an application circumvents the compositor, the injection may be turned off to prevent unwanted changes to the image. Alternately, the injection can add a portion of computer code to an application rather than the compositor.
In an embodiment, the portion of computer code injected into the compositor can be constructed and arranged to communicate with another portion of mapping software which is located outside of the computer, such as in external mapping hardware.
The operating system can also be adapted to accept the injection of the portion of computer code used for mapping. For example, the operating system may be designed to make the injection work smoothly by passing variables to the portion of computer code.
According to an embodiment, patterns can be injected into the projected digital images for test, alignment, and/or feedback purposes. Watermarking patterns with invisibly small changes or other invisible test patterns such as individual frames may be used. The sensor can be synchronized to the individual frames. A three-dimensional user interface can be injected into the projected digital image for purposes (that should be apparent to those of skill in the art) such as controlling the computer, manipulating data, interacting with the environment in the immediate proximity to the user, or interacting with an environment remotely located from the user.
Head tracking, eye tracking, or other external parameters can be used to change the injected mapping function according to the viewer's field of view or other conditions of the viewer. In the example of warping, head tracking can be used to position the projected digital image with proper warping on a dome or other immersive screen for simulation purposes. The appropriate geometric parameters to accomplish the exemplary warp are understood by those of skill in the art.
Injection-based digital imaging systems can also be used to modify the projected digital images as a function of time. For example, if the projector alignment is only acceptable for a certain period of time (because of possible physical misalignment over time, etc.), a visible warning can be injected after the time period has expired. Alternately, the mapping function can be automatically changed after a certain period of time to compensate for a predetermined pattern of alignment drift such as thermal warming effects. In another embodiment, the mapping can be changed based on the time of day to compensate for increased ambient light during day hours of operation, awhile the compensation is removed during night hours of operation. In another embodiment, the mapping can depend on external factors such room temperature or humidity. The mapping can be predetermined in a look-up table, or can be calculated in real time or near-real time.
Sensors used for feedback in various implementations of the system can include still cameras, video cameras illustratively operating at 30 frames per second or another frame rate, ambient light sensors, or any other device that senses a characteristic of the projected digital image or the surrounding environment. In the example of still cameras, the computer may be used to trigger the camera to take pictures as required. Sensors may also be used to determine operational characteristics of the projectors such as their temperature or operation mode.
U.S. published patent application No. 20080/246781 A1, entitled SYSTEM AND METHOD FOR PROVIDING IMPROVED DISPLAY QUALITY BY DISPLAY ADJUSTMENT AND IMAGE PROCESSING USING OPTICAL FEEDBACK, by Rajeev J. Surati, et al., the teachings of which are expressly incorporated herein by reference as useful background information, describes the optical mechanical thermal and electronic (OMTE) parameters of a display. The OMTE parameters may be modified under computer or manual control and then the mapping function may make transformations to the image that work together with the OMTE changes.
A specific example of mapping function injection for a digital imaging system is based on the following three conditions: (1) There is a mapping function generated that will be injected into the digital imaging system; (2) the operating system has compositing software such as that instantiated in Microsoft Windows Vista®, wherein this compositing software can include or comprise the Desktop Window Manager (dwm.exe) in Microsoft Windows®, the equivalent windows manager in the Apple Leopard® operating system, or Compiz™ in the Linux operating system; and (3) the compositing software is enacted by a common library such as DirectX®, OpenGL™, or DDraw™. Also, in order to enable the mapping function injection, the functions that are called that display to the screen such as Present in DirectX or GLSwap in OpenGL (or a lower level functionality in an underlying library) should be known by the process. In an embodiment, the portion of computer code that performs the injection overwrites the functions used by the compositing software that are relevant for initializing. The portion of computer code further detects certain relevant states with functions that call from the original function. Prior to those calls, the portion of computer code initializes, inspects or sets states, or enacts the mapping as appropriate. In the example of Direct3D, this can occur during the operation of CreateDevice for initialization or during the operation of Present. The enactment usually initializes and utilizes a set of shaders upon a mesh, or shaders on vertices of the image. In OpenGL that would include the operation of GLSwap. In an embodiment, the portion of computer code can be injected whenever the compositing software or application software is started or restarted.
Note also that specific hardware used for implementing injection into a digital imaging system can desirably make use of commercially available computer graphics boards such as the Quadro™ series of graphics cards made by NVIDIA (Santa Clara, Calif.) in various embodiments. In various embodiments, other implementations of injection of a mapping function include corrections for dithering, corrections for dithering before a warping correction, correction of mura defects, shading correction for transmissive LCD projectors, mapping for cell phones with embedded projectors and cameras, warping of multiplayer simulations such as Microsoft Flight Simulator X (Microsoft Corporation, Redmond, Wash.), warping the menus of the Microsoft Windows CE operating system running on projector, radial distortion correction of a single display, adjusting for display motion of a single display, showing only a portion of a larger display such as showing a circular portion of a rectangular display, and projecting on a corner of a room and correcting for the condition where one wall or the ceiling has a different color or other surface property. The specific parameters to be injected are computed in a conventional manner that should be apparent to those of skill in the art.
In various embodiments, a multiple computer platform is employed. By way of example, a first and a second computer system is configured with a first and a second compositor, respectively. The first and second computers are configured to communicate between each other over a conventional communication link in a manner that should be apparent to those skilled in the art. Computer codes are injected into the first and second computer, respectively. During operation, the first computer applies a first mapping function to output display signals to a first display which forms a displayed digital image. The display signals are based partially on the information communicated with the second computer. Similarly, the second computer applies a second mapping function to output display signals to a second display which forms a displayed digital image. The display signals are based partially on the information communicated with the first computer.
In various alternate embodiments, the mapping function is stored on a computer-readable medium. The computer code is operatively configured to control the computer hardware upon reading of the computer-readable medium by the computer, as apparent to those of ordinary skill in the art. Upon initialization by the computer system, the computer code would function as operatively described in the specification herein.
The foregoing has been a detailed description of illustrative embodiments of the invention. Various modifications and additions can be made without departing from the spirit and scope of this invention. Each of the various embodiments described above may be combined with other described embodiments in order to provide multiple features. Furthermore, while the foregoing describes a number of separate embodiments of the apparatus and method of the present invention, what has been described herein is merely illustrative of the application of the principles of the present invention. For example, there may be provided one computer in the image display system as shown in
This application is a continuation of co-pending U.S. patent application Ser. No. 12/818,102, filed Jun. 17, 2010, entitled SYSTEM AND METHOD FOR INJECTION OF MAPPING FUNCTIONS, which application claims the benefit of U.S. Provisional Application Ser. No. 61/218,397, filed Jun. 18, 2009, entitled SYSTEM AND METHOD FOR INJECTION OF MAPPING FUNCTIONS, the entire disclosures of each are herein incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| 61218397 | Jun 2009 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 12818102 | Jun 2010 | US |
| Child | 14863216 | US |
