Patent Grant
6985168
1. Field of Invention
The present invention relates to an improved method and system for producing stereoscopically-multiplexed images from stereoscopic image-pairs and displaying the same stereoscopically, in an interactive manner that allows viewers to perceive displayed imagery with a sense of realism commensurate with natural viewing of physical reality.
2. Brief Description of State of the Art
In the contemporary period, stereoscopic display systems are widely used in diverse image display environments, including virtual-reality applications. The value of such image display systems resides in the fact that viewers can view objects with depth perception in three-dimensional space.
In general, stereoscopic image display systems display pairs of stereoscopic images (i.e. stereoscopic image-pairs) to the eyes of human viewers. In principle, there are two ways in which to produce stereoscopic image-pairs for use in stereoscopic display processes. The first technique involves using a “real” stereoscopic-camera, positioned with respect to a real 3-D object or scene, in order to acquire each pair of stereoscopic images thereof. The second techniques involves using a computer-based 3-D modeling system to implement a “virtual” stereoscopic-camera, positioned with respect to a (geometric) model of a 3-D object or scene, both represented within the 3-D modeling system. In the first technique, it is necessary to characterize the real-image acquisition process by specifying the camera-parameters of the real stereoscopic-camera used during the image acquisition process. In the second technique, it is necessary to characterize the virtual-image acquisition process by specifying the “camera-parameters” of the virtual stereoscopic-camera used during the image acquisition process. In either case, the particular selection of camera parameters for either the real or virtual stereoscopic-camera necessarily characterizes important properties in the stereoscopic image-pairs, which are ultimately stereoscopically-multiplexed, using one or another format, prior to display.
Presently, there are several known techniques for producing “spectrally-multiplexed images”, i.e. producing temporal-multiplexing, spatial-multiplexing and spectral-multiplexing.
Presently, there exist a large number of prior art stereoscopic display systems which use the first technique described above in order to produce stereoscopically-multiplexed images for display on the display surfaces of such systems. In such prior art systems, the viewer desires to view stereoscopically, real 3-D objects existing in physical reality. Such systems are useful in laprascopic and endoscopic surgery, telerobotics, and the like. During the stereoscopic display process, complementary stereoscopic-demultiplexing techniques are used in order to provide to the left and right eyes of the viewer, the left and right images in the produced stereoscopic image-pairs, and thus permit the viewer to perceive full depth sensation. However, the selection of camera parameters used to produce the displayed stereoscopic image-pairs rarely, if ever, correspond adequately with the “viewing parameters” of the viewer's, human vision system, which ultimately views the displayed stereoscopic image-pairs on the display surface before which the viewer resides.
Also, there exist a large number of prior art stereoscopic display systems which use the second technique described above in order to produce stereoscopically-multiplexed images for display on the display surfaces of such systems. In such systems, the viewer desires to view stereoscopically, synthetic 3-D objects existing only in virtual reality. Such systems are useful in flight simulation and training, virtual surgery, video-gaming applications and the like. During the stereoscopic display process, complementary stereoscopic-demultiplexing techniques are also used to provide to the left and right eyes of the viewer, the left and right images in the produced stereoscopic image-pair. However, the selection of camera parameters used to produce the displayed stereoscopic image-pairs in such systems rarely, if ever, correspond adequately with the viewing parameters of the viewer's human vision system, which who ultimately views the displayed stereoscopic image-pairs on the display surface before which the viewer resides.
Consequently, stereoscopic viewing of either real or synthetic 3-D objects in virtual reality environments, using prior art stereoscopic image production and display systems, have generally lacked the sense of realism otherwise experienced when directly viewing real 3-D scenery or objects in physical reality environments.
Thus there is a great need in the art for a stereoscopic image production and display system having the functionalities required in high performance virtual-reality based applications, while avoiding the shortcomings and drawbacks associated with prior art systems and methodologies.
Accordingly, it is a primary object of the present invention to provide an interactive-based system for producing and displaying stereoscopically-multiplexed images of either real or synthetic 3-D objects that permits realistic stereoscopic viewing thereof, while avoiding the shortcomings and drawbacks of prior art systems and methodologies.
Another object of the present invention is to provide a such a system, in which the true viewing parameters of the viewer, including head/eye position and orientation, are continuously acquired relative to the display surface of the stereoscopic display subsystem and used during the producing of stereoscopically-multiplexed images of synthetic 3-D objects being stereoscopically viewed by the viewer in a virtual reality (VR) viewing environment, such as presented in flight simulation and training, virtual surgery, video-gaming and like applications.
A further object of the present invention is to provide a such a system, in which the true viewing parameters of the viewer, including head/eye position and orientation, are continuously acquired relative to the display surface of the stereoscopic display subsystem and used during the producing of stereoscopically-multiplexed images of real 3-D objects being stereoscopically viewed by the viewer in a virtual reality (VR) viewing environment, such as presented in laprascopic and endoscopic surgery, telerobotic and like applications.
Another object of the present invention is to provide such a system, in which the stereoscopically-multiplexed images are spatially-mulitplexed images (SMIs) of either real or synthetic 3-D objects or scenery.
Another object of the present invention is to provide a process for producing and displaying, in real-time, spatially-mulitplexed images (SMIs) of either real or synthetic 3-D objects or scenery, wherein the true viewing parameters of the viewer, including head/eye position and orientation, are continuously acquired relative to the display surface of the stereoscopic display subsystem and used during the producing of stereoscopically-multiplexed images of either the real or synthetic 3-D objects being stereoscopically viewed by the viewer in a virtual reality (VR) viewing environment.
Another object of the present invention is to provide a stereoscopic camera system which is capable of acquiring., on a real-time basis, stereoscopic image-pairs of real 3-D objects and scenery using camera parameters that correspond to the range of viewing parameters that characterize the stereoscopic vision system of typical human viewers.
Another object of the present invention is to provide a system of compact construction, such as notebook computer, for producing and displaying, in real-time, micropolarized spatially-mulitplexed images (SMIs) of either real or synthetic 3-D objects or scenery, wherein the true viewing parameters of the viewer, including head/eye position and orientation, are continuously acquired relative to the display surface of the portable computer system and used during the production of spatially-multiplexed images of either the real or synthetic 3-D objects being stereoscopically viewed by the viewer in a virtual reality (VR) viewing environment, wearing a pair of electrically-passive polarizing eye-glasses.
Another object of the present invention is to provide a such a system in the form of desktop computer graphics workstation, particularly adapted for use in virtual reality applications.
It is yet a further object of the present invention to provide such a system and method that can be carried out using other stereoscopic-multiplexing techniques, such as time-sequential (i.e. field-sequential) multiplexing or spectral-multiplexing techniques.
Another object of the present invention is to provide a stereoscopic display system as described above, using either direct or projection viewing techniques, and which can be easily mounted onto a moveable support platform and thus be utilizable is flight-simulators, virtual-reality games and the like.
A further object of the present invention is to provide a stereoscopic-multiplexing image production and display system which is particularly adapted for use in scientific visualization of diverse data sets, involving the interactive exploration of the visual nature and character thereof.
These and other objects of the present invention will become apparent hereinafter and in the claims to invention.
For a more complete understanding of the Objects of the Present Invention, the Detailed Description of the Illustrated Embodiments should be read in conjunction with the accompanying Drawings, in which:
Referring to
The system and method of the present invention may utilize any one or a number of available stereoscopically-multiplexing techniques, such temporal-multiplexing (i.e. field-sequential-mulitplexing), spatial-multipiplexing or spectral-multiplexing. For purposes of illustration, spatial-multiplexing will be described. It is understood, that when using other stereoscopic display techniques to practice the system and method of the present invention, various modifications will need to be made. However, after having read the teachings of the present disclosure, such modifications will be within the knowledge of one of ordinary skill in the art.
As shown in
It is understood there will be various embodiments of the system of the present invention depending on whether stereoscopic image-pairs (comprising pixels selected from left and right perspective images) are to be produced from either (i) real 3-D objects or scenery existing in physical reality or (ii) virtual (synthetic) 3-D objects or scenery existing in virtuality. In either event, the real or synthetic 3-D object will be referenced to a three-dimensional coordinate system PM. As used hereinafter, all processes relating to the production of stereoscopic image-pairs shall be deemed to occur within “3-D Image-Pair Production Space (RA)” indicated in
In general, stereoscopically-multiplexed image production subsystem 2 may include a stereoscopic-image pair generating (i.e. computing) subsystem 7 illustrated in
In general, computer models of synthetic objects in 3-D Image Production Space (RA) may be represented using conventional display-list graphics techniques (i.e. using lists of 3-D geometric equations and parameters) or voxel-based techniques. As illustrated in
Alternatively, voxel-based models (m) of synthetic objects may be created using a parallel computing system of the type disclosed in U.S. Pat. No. 5,361,385 to Bakalash entitled “Parallel Computing System For Volumetric Modeling, Data Processing and Visualization” incorporated herein by reference in its entirety. When using voxel-based models of real or synthetic 3-D objects, suitable modifications will be made to the mapping processes mmci and mmcr generally illustrated in FIG. 3A.
As illustrated in
As shown in
The right and left quantization mapping processors 24 and 25 receive as input, the eye/head/display tracking information (i.e. transformations Twd, Tdv, Tvel, and Tver), which are uses to define the right and left quantization mappings, mcpr and mcpl, respectively. Not only do these two quantization mappings define the quantization of the images Icr and Icl, respectively, they also define the size and location of the images Icr and Icl, respectively, on the surfaces scr and scl, respectively. The size and location of the images Icr and Icl, within the surfaces scr and scl, defines which portions of the physical object, M, imaged on the quantization samplers are represented by the output images Ipr and Ipl. This flexibility of choosing the size and location of the images Icr and Icl allows each channel, right and left, of the stereoscopic image-acquisition subsystem 8 to independently “look at” or acquire different portions, along a viewing direction, of the imaged object M the above-described eye/head/display tracking information. In the illustrated embodiment, the mappings mcpr and mcpl, the right and left pixel scanner/processors 24 and 25, and the right and left quantization mapping processors 21 and 22 are each implemented by non-mechanical means, thus making it possible to change the “looking” direction of Stereoscopic image acquisition subsystem speeds comparable or better that of the human visual system. The rotation and position processor 17 controls the 3 (i) axis “rotation” mechanism 16 (which is responsible for aiming the wide angle optics and the quantization samplers), and (ii) the 3 axis “translation” mechanism (which is responsible for moving the wide angle optics and the quantization samplers to different locations in the image acquisition space RA).
As illustrated in
In the illustrative embodiment shown in
As illustrated in
As illustrated in
The display position and orientation parameters, required for display viewing transform Twd, are transmitted to stereoscopic image production subsystem 2 preferably by way of electro-magnetic position/orientation signals, which are received at subsystem 2 using base transceiver 46 well known in the art.
As illustrated in
Having described the structure and function of the stereoscopically-muitiplexed image production and display system of the present invention, it is appropriate at this juncture to now describe the generalized and particular processes of the present invention which, when carried out on the above-described system in a real-time manner, support realistic stereoscopic viewing of real and/or synthetic 3-D objects by viewers who “visually interact” with the stereoscopic image display subsystem 6 thereof.
In
The object representation subsystem comprises means for dynamically changing or static object or objects, M, which can be either real physical objects, Mr, or synthetic objects, Ms. These objects, M, are referenced to the coordinate frame pm in the image acquisition space RA.
As illustrated in
The right object mapping, mmcr, creates an image representation, Icr, of M, onto the surface scr. In a similar manner, the left object mapping, mmcl, creates an image representation, Icl, of M, onto the surface scl. Both surfaces scr and scl are referenced with respect to coordinate frame pq by transformations Tqcr and Tqcl, respectively. The object mappings mmcr and mmcl can be either physical optical imaging mappings or virtual geometric mappings implemented with software or hardware. Images Icr and Icl (on surfaces scr and scl, respectively) can be represented by physical optical images or geometric representations (hardware or software). The images Icr and Icl taken together form the beginnings of a stereoscopic image-pair which represent a portion of the object(s) M. The transforms Tmq, Tqcr, and Tqcl and the mappings mmcr and mmcl are defined such that the resulting images Icr and Icl will lead to the creation of a realistic stereoscopic image-pair in later steps of this process.
The right quantization mapping, mcpr, describes the conversion of the object image Icr, into a pixelized image Ipr, on the synthetic surface, spr. The image Ipr can be modified with the supplemental pixel information dr. In a similar manner, the left quantization mapping, mcpl, describes the conversion of the object image Icl, into a pixelized image Ipl, on the synthetic surface, spl. The image Ipl can be modified with the supplemental pixel information dl. The pixelized images Ipr and Ipl are represented by a 2-D array, where each element in the array represents a spatial pixel in the image and contains spectral data about the pixel. The quantization mappings, mcpr and mcpl, basically indicate how an arbitrary region of Icr and Icl (respectively) are mapped into the pixelized images Ipr and Ipl (as shown in
As illustrated in
As illustrated in
As illustrated in
The object representation process operates on an object representation of either a real physical object(s), Mr, or synthetic object representations, Ms. The synthetic object representations, Ms, can be represented in any convenient form such as the common geometric representations used in polygonal modeling systems (vertices, faces, edges, surfaces, and textures) or parametric function representations used in solids modeling systems (set of equations) which is well known in the art. The result of the object representation process steps is the creation of an object representation M which is further processed by the stereoscopic image-pair production process steps.
The stereoscopic image-pair generation process steps operate on the object representation, M and produces the right and left pixelized stereoscopic image-pairs Ipr and Ipl, respectively. The steps of the stereoscopic image-pair generation process use the transformations Tdv (acquired by the head position and orientation tracking subsystem), Tvel and Tver (acquired by the eye position and orientation tracking subsystem), and Twd (acquired by the display position and orientation tracking subsystem) and the acquisition of the display parameters msd, ss, and sd to compute various transformations and mappings as will be described next.
The transformation Tmq describes the position and orientation placement of the right and left stereoscopic image-pair acquisition surfaces scr and scl. Tmq is computed by the function fTmq which accepts as parameters Twd, Tdv, Tvel, Tver, and pm. fTmq computes Tmq such that the images Icr and Icl taken together form the beginnings of a stereoscopic image-pair which represents a portion of the object(s) M.
The transformation Tqcr describes the position and orientation placement of the right stereoscopic image-pair acquisition surface Icr with respect to pq. Tqcr is computed by the function fTqcr which accepts as parameters Tmq, Tdv, Tvel, and Tver. fTqcr computes Tqcr such that the image Icr from the surface scr forms the beginnings of a realistic stereoscopic image-pair. In a similar manor, the transformation Tqcl describes the position and orientation placement of the left stereoscopic image-pair acquisition surface Icl with respect to pq. Tqcl is computed by the function fTqcl which accepts as parameters Tmq, Tdv, Tvel, and Tver. fTqcl computes Tqcl such that the image Icl from the surface scl forms the beginnings of a realistic stereoscopic image-pair.
The right object mapping, mmcr, creates an image representation, Icr, of M, onto the surface scr. mmcr is computed by the function fmmcr which accepts as parameters Tmq, Tqcr, Tdv, Tver, sd, and msd. mmcr represents either a real optical imaging process in the case of a real object Mr or a synthetic rendering process (well known in the art) in the case of a synthetic object Ms. In a similar manor, the left object mapping, mmcl, creates an image representation, Icl, of M, onto the surface scl. mmcl is computed by the function fmmcl which accepts as parameters Tmq, Tqcl, Tdv, Tvel, sd, and msd. mmcl represents either a real optical imaging process in the case of a real object Mr or a synthetic rendering process (well known in the art) in the case of a synthetic object Ms.
The image representation Icr on surface scr is formed by the function fIcr which accepts as parameters mmcr and M. In a similar manor, the image representation Icl on surface scl is formed by the function fIcl which accepts as parameters mmcl and M. Mappings mmcr and mmcl are defined in such a way that the images Icr and Icl taken together form the beginnings of a realistic right and left, respectively, stereoscopic image-pair which represents a portion of the object(s) M.
The right quantization mapping, mcpr, describes the conversion of the object image Icr, into a pixelized image Ipr, on the synthetic surface, spr. mcpr is computed by the function fmcpr which accepts as parameters Tdv, Tvel, Tver, sl, ss, and msd. The quantization mapping, mcpr indicates how an arbitrary region of Icr is mapped into the pixelized image Ipr. In a similar manor, the left quantization mapping, mcpl, describes the conversion of the object image Icl, into a pixelized image Ipl, on the synthetic surface, spl. mcpl is computed by the function fmcpl which accepts as parameters Tdv, Tvel, Tver, sd, ss, and msd. The quantization mapping, mcpl indicates how an arbitrary region of Icl is mapped into the pixelized image Ipr.
The right pixelized image Ipr on surface spr is formed by the function fIpr which accepts as parameters Icr, mcpr, and dl, where dl represents supplemental pixel information. In a similar manor, the left pixelized image Ipl on surface spl is formed by the function fIpl which accepts as parameters Icl, mcpl, and dr, where dr represents supplemental pixel information. Mappings mcpr and mcpl are defined in such a way that the resulting images Ipr and Ipl will lead to the creation of a realistic stereoscopic image-pair in later steps of this process. Mappings mcpr and mcpl can also be used to correct for limitations of an implemented system for performing the mappings mmcr and mmcl described above.
The stereoscopic image multiplexing process steps operate on the right and left pixelized images Ipr and Ipl respectively and produces the right and left multiplexed stereoscopic image representations Isl and Isr. The steps of the stereoscopic image-pair multiplexing process use the transformations Tdv (acquired by the head position and orientation tracking subsystem), and Tvel and Tver (acquired by the eye position and orientation tracking subsystem), and the acquisition of the display parameters msd, ss, and sd to compute various transformations and mappings as will be described next.
The right multiplexing mapping, mpsr, defines the mapping of pixels in Ipr to pixels in Isr. mpsr is computed by the function fmpsr which accepts as parameters Tdv, Tvel, Tver, sd, ss, and msd. In a similar manor, the left multiplexing mapping, mpsl, defines the mapping of pixels in Ipl to pixels in Isl. mpsl is computed by the function fmpsl which accepts as parameters Tdv, Tvel, Tver, sd, ss, and msd.
The right multiplexed image Isr, on surface ss, is formed by the function fIsr which accepts as parameters Ipr and mpsr. likewise, the left multiplexed image Isl, on surface ss, is formed by the function fIsl which accepts as parameters Ipl and mpsl. Isr and Isl are formed by the mappings mpsr and mpsl in such a manor as to be compatible with the stereoscopic image-pair display subsystem. The composite multiplexed stereoscopic image, Is, is formed from the compositing of Isr and Isl.
The stereoscopic image-pair display process steps operate on the right and left stereoscopic images Isr and Isl, respectively, using the display mapping msd, to display the right and left stereoscopic image display pairs Idr and Idl. The mapping msd can be an electronics mapping to pixels on a display or projection optics to image onto a screen.
The right stereoscopic display image Idr, on surface sd, is formed by the function/process fIdr which accepts as parameters Isr and msd. likewise, the left stereoscopic display image Idl, on surface sd, is formed by the function/process fIdl which accepts as parameters Isl and msd. The function/processes fIdr and fIdl form the stereoscopic encoding process which encodes the right and left stereoscopic display images, Idr and Idl in a form which can be viewed in a stereoscopic viewing mode by the stereoscopic image-pair viewing process or precesses. The composite multiplexed stereoscopic display image, Id, is formed from the compositing of Idr and Idl.
The stereoscopic display surface, sd, has imbedded coordinates pd which are related to pw by the transformation Twd. The display position and orientation tracking process tracks the interaction of the display with the virtual environment M′ and acquires the transformation Twd.
The stereoscopic image-pair viewing process steps represent the viewing decoding of the right and left stereoscopic display images, Idr and Idl, through the decoding mappings mder and mdel, respectively, to produce the right and left viewer images, Ier and Iel, respectively. The right and left viewer Images Ier and Iel are formed on the right and left viewing surfaces ser and sel, respectively. The right viewing surface, ser, has imbedded coordinate frame per. Coordinate frame per is related to frame pv by the transformation Tver. Likewise, the left viewing surface, sel, has imbedded coordinate frame pel. Coordinate frame pel is related to frame pv by the transformation Tvel. The function/process fIpr accepts parameters Idr, and mder and performs the actual decoding of the image Idr to form the image Ier. Likewise, the function/process fIpl accepts parameters Idl, and mdel and performs the actual decoding of the image Idl to form the image Iel. The combination of images Ier and Iel in the visual processing center, B, forms the image Ib. Ib represents the perceived stereoscopic image M′ as represented in the visual processing center B through the use of the function/process fIb.
Coordinate frame pv represents the imbedded coordinate frame of the combined right and left viewing surfaces ser and sel, respectively. pv is related to the stereoscopic image-pair display coordinates system, pd, by the transformation Tdv.
The head position and orientation tracking process tracks the interaction of the combined right and left viewing surfaces, ser and sel, with the display surface, sd, and acquires the transformation Tdv to describe this interaction. The eye position and orientation tracking process tracks the interaction of each individual right and left viewing surface, ser and sel, with respect to the coordinate frame pv, and acquires the right and left viewing surface transformations, Tver and Tvel.
The overall process steps set forth in the process groups A through E in
Referring to
vo(7,3)=a v(6,2)+b v(7,2)+c v(8,2)+d v(6,3)+e v(7,3)+f v(8,3)+g v(6,4)+h v(7,4)+i v(8,4)
The quantized images, Ipl and Ipr, can be represented by a 2-D array of pixels, Ipl(x,y,t) and Ipr(x,y,t), where each x,y pixel entry represents a quantized area of the images Icl and Icr at quantized time t. The time value, t, in the Ipl and Ipr images, represents the time component of the image. The time value, image time, increments by 1 each time the Ipl and Ipr images change. Each image pixel is represented by a spectral vector, v(x,y,t), of size N×1 or P×1 (N for images Ipl and Ipr and P for Isl and Isr). A typical representation would be a 3×1 vector representing the red, blue, and green components of the pixel. Individual values in the intensity vector are represented as v(x,y,i) where i is the particular intensity element.
In essence, the general stereoscopically-multiplexed image process maps the pixels of Ipl into the pixels of Isl using the mapping mpsl and mapping the pixels of Ipr into the pixels of Isr using the mapping mpsr. The images Isl and Isr can be combined into the composite image Is or can be left separate. The general stereoscopically-multiplexed image process can be described by the following equations:
Isl(x,y,t)=Rl(x,y,t,Ipl(t)) Kl(x,y,t){Ipl(t)} 1.
Isr(x,y,t)=Rr(x,y,t,Ipr(t)) Kr(c,y,t){Ipr(t)} 2.
Is(x,y,t)=cl(t) Isl(x,y,t)+cr(t)Isr(x,y,t) 3.
where:
The operations represented by Equations 1 and 2 above are evaluated for each x,y pixel in Isl and Isr (this process can be performed in parallel for each pixel or for groups of pixels). The operations are performed in two steps. First, the spatial kernel Kl(x,y,t) is applied to the image Ipl(t) which forms a linear combination of the neighboring pixels of Ipl(x,y,t) to produce a spectral vector vl(x,y,t) at (x,y) at time t. Second, this spectral vector vl(x,y,t) is multiplied by the spectral transformation matrix, Rl(x,y,t,Ipl), to produce a modified spectral vector which is stored in Isl(x,y,t). This process is carried out for each pixel, (x,y), in Isl(t). The same process is carried out for each pixel in Isr(x,y,t). The resulting images, Isl(t) and Isr(t) can be composited into a single channel Is(t) by equation 3 above by a simple linear combination using weights cl(t)and cr(t)which are functions of time t. Typically, cl(t)=cr(t)=1. Advantageously, Equations 1, 2, and 3 can be evaluated in a massively parallel manner
The spatial kernels, Kl and Kr, are N×N matrices where N is odd, and each entry in the matrix represents a linear weighting factor used to compute a new pixel value based on its neighbors (any number of them). A spatial kernel transforms a group of pixels in an input image into a single pixel in an output image. Each pixel in the output image (x,y) at a particular time (t) has a kernel K(x,y,t) associated with it. This kernel defines the weights of the neighboring pixels to combine to form the output image and is an N×N matrix where N is odd. The center element in the kernel matrix is the place holder and is used to define the relative position of the neighboring pixels based on the place holder pixel which represents x,y.
Notably, using massively parallel computers and a real-time adapting electronically adapting micorpolarization panels 41, it is possible to adaptive encoding system which changes the micropolarization pattern (e.g. P1, P2, P1, P2, etc.) upon display surface 40, as required by the symmetries of the image at time (t).
Below are three examples of possible kernel functions that may be used with the system of the present invention to produce and display (1) spatially-multiplexed images (SMI) using a 1-D spatial modulation function, (2) temporally-multiplexed images, and (3) a SMI using a 2-D spatial modulation function. Each of these examples will be considered in their respective order below.
In the first SMI example, the kernel has the form:
This happens to be the left image Kernel for the 1-D spatial multiplexing format. Note that the above kernel is not a function of time and it therefor a spatial multiplexing technique. The center entry defines the pixel in question and the surrounding entries define the weights of the neighboring pixels to combine. Each corresponding pixel value is multiplied by the weighting value and the collection is summed. The result is the spectral vector for the Is image. In the above case, we are averaging the current pixel with the pixel above it on odd lines and doing nothing for even lines. The corresponding Kr for the SMI format is given below:
In the second example, the kernel functions for the field sequential (temporal multiplexing) technique are provided by the following expressions:
Kl(x,y,t)=[1] for frac(t)<0.5
Kl(x,y,t)=[0] for frac(t)>0.5
Kr(x,y,t)=[1] for frac(t)>0.5
Kr(x,y,t)=[0] for frac(t)<0.5
where, frac(t) returns the fractional portion of the time value t. These expressions state that left pixels are only “seen” for the first half of the time interval frac(t) and right pixels for the second half of the time interval frac(t).
The third example might be used when a pixel may be mapped into more than one pixel in the output image, as in a 2-D a checker board micropolarization pattern. An example kernel function, Kr, for a checkerboard polarization pattern might look like the following:
Having described these types of possible kernels that may be used in the stereoscopic multiplexing process, attention is now turned to the spectral transformation matrices, Rl and Rr, which addresses the stereoscopic-multiplexing of the spectral components of left and right quantized perspective images.
The spectral transformation matrices, Rl and Rr, define a mapping of the spectral vectors produced by the kernel operation above to the spectral vectors in the output images, Isl and Isr. The spectral vector representation used by the input images, Ipl and Ipr, do not need to match the spectral vector representation used by the output images, Isl and Isr. For example, Ipl and Ipr could be rendered in full color and Isl and Isr could be generated in gray scale. The elements in the spectral vector could also be quantized to discrete levels. The spectral transformation matrices are a function of the x,y pixel in question at time t and also of the entire input image Ip. The parameterization of Rl and Rr on Ipl and Ipr (respectively) allows the spectral transformation to be a function of the color of the pixel (and optionally neighboring pixels) in question. By definition, a spectral transformation matrix is a P×N matrix where N is the number of elements in the spectral vectors of the input image and P is the number of elements in the spectral vectors of the output image. For example, if the input image had a 3×1 red, green, blue spectral vector and the output image was gray scale, 1×1 spectral vector, a spectral transform matrix which would convert the input image into a b/w image might look like, Rl(x,y,t)=[0.3 0.45 0.25] which forms the gray scale pixel by summing 0.3 times the red component, 0.45 times the green component, and 0.25 times the blue component. A color multiplexing system with a red, green, and blue 3×1 spectral vector might look like this:
Rl(x,y,t)=diag(1 0 1) for frac(t)<0.5
Rl(x,y,t)=diag(0 1 0) for frac(t)>0.5
Rr(x,y,t)=diag(0 1 0) for frac(t)<0.5
Rr(x,y,t)=diag(1 0 1) for frac(t)>0.5
Where diag(a,b,c) is the 3×3 diagonal matrix with diagonal elements a, b, c. A spectral transformation matrix to create a red/green anaglyph stereoscopic image could be:
Note, in the above cases, when the spectral kernels are not specified they is assumed to be Kl=[1] and Kr=[1].
In
The object representation subsystem is comprises the dynamically changing or static object or objects, M, which are synthetic objects, Ms. These objects, M, are referenced to the coordinate frame pm in the image acquisition space RA.
The stereoscopic image-pair production subsystem is comprised of the right and left object surfaces scr and scl (with imbedded coordinate frames pcr and pcl), the right and left pixel surfaces spr and spl, the right and left object mappings mmcr and mmcl, the right and left quanitization mappings mcpr and mcpl, the coordinate frame pq, and the supplemental right and left pixel information dr and dl. Coordinate frame pq is referenced with respect to the coordinate frame pm by the transformation Tmq.
The right object mapping, mmcr, creates an image representation, Icr, of M, onto the surface scr. In a similar manor, the left object mapping, mmcl, creates an image representation, Icl, of M, onto the surface scl. Both surfaces scr and scl are referenced with respect to coordinate frame pq by transformations Tqcr and Tqcl, respectively. The object mappings mmcr and mmcl are virtual geometric mappings implemented with software or hardware. Images Icr and Icl (on surfaces scr and scl, respectively) are represented by geometric representations (hardware or software). The images Icr and Icl taken together form the beginnings of a stereoscopic image-pair which represent a portion of the virtual object(s) M. The transforms Tmq, Tqcr, and Tqcl and the mappings mmcr and mmcl are defined such that the resulting images Icr and Icl will lead to the creation of a realistic stereoscopic image-pair in later steps of this process.
The right quantization mapping, mcpr, describes the conversion of the object image Icr, into a pixelized image Ipr, on the synthetic surface, spr. The image Ipr can be modified with the supplemental pixel information dr. In a similar manor, the left quantization mapping, mcpl, describes the conversion of the object image Icl, into a pixelized image Ipl, on the synthetic surface, spl. The image Ipl can be modified with the supplemental pixel information dl. The pixelized images Ipr and Ipl are represented by a 2-D array, where each element in the array represents a spatial pixel in the image and contains spectral data about the pixel. The quantization mappings, mcpr and mcpl, indicate how an arbitrary region of Icr and Icl (respectively) are mapped into the pixelized images Ipr and Ipl.
The stereoscopic image multiplexing subsystem performs the right and left multiplexing mappings mpsr and mpsl, the stereoscopic image surface ss, the right multiplexing image Isr, and left multiplexed image Isl, and the composite multiplexed image Is. The right multiplexing mapping mpsr defines the spatial mapping of pixels in Ipr to pixels in Isr. Similarly, the left multiplexing mapping mpsl defines the spatial mapping of pixels in Ipl to pixels in Isl. The images Isr and Isl represent the right and left eye stereoscopic perspectives of the object(s) M. Isr and Isl are formed by the mappings mpsr and mpsl in such a manor as to be compatible with the micro-polarizer based stereoscopic image-pair display subsystem. Is is formed from Isr and Isl as will be described later.
The stereoscopic image-pair display subsystem performs the mapping msd, the display surface sd, the right stereoscopic spatial multiplexed display image Idr, the left stereoscopic spatial multiplexed display image Idl, and the composite stereoscopic spatial multiplexed display image Id, and the coordinate frame pd. The mapping msd defines the mappings of the pixels of Is onto the display pixels as represented by Id. The mapping msd represents an optical projection subsystem and can include some scaling factors between the image acquisition space RA and the image display space RB. The images Idr and Idl form a realistic spatially multiplexed stereoscopic display image-pair which, when viewed by the stereoscopic image-pair viewing subsystem, form a realistic representation, M′, of the object(s) M The virtual object M′, is represented in the image display space, RB, which is referenced to coordinate frame pw. The display surface sd, contains a micro-polarizer array which performs polarization encoding of the images Idr and Idl. Surface sd has imbedded coordinates pd which are referenced to the image display space coordinates pw by the transformation Twd.
The stereoscopic image-pair viewing subsystem performs the right and left optical imaging mappings mder and mdel, the right viewing surface ser with imbedded coordinate system per, the left viewing surface sel with imbedded coordinate system pel, the right and left viewed images Ier and Iel, the viewing coordinate system pv, and the visual processing subsystem B. The right viewing image Ier is formed on the right viewing surface ser by the right optical imaging mapping mder which performs a polarization decoding process. The left viewing image Iel is formed on the left viewing surface sel by the left optical imaging mapping mdel which performs a polarization decoding process. The relationship between the right an left viewing surfaces and the viewing coordinate system, pv, is given by the transformations Tver and Tvel respectively. The relationship between the viewing coordinate system, pv, and the display surface coordinate system pd is given by the transformation Tdv. The transformations Tdv, Tver, and Tvel describe the position and orientation of the right and left viewing surfaces with respect to the display surface sd.
b shows the process steps to be carried out when computing the transformations and mappings to implement the system shown in FIG. 5A. The process steps are organized into five process groups labeled A through E in
The object representation process steps operate on an object representation of a synthetic object, Ms. The synthetic object representations, Ms, can be represented in any convenient form such as the common geometric representations used in polygonal modeling systems (vertices, faces, edges, surfaces, and textures) or parametric function representations used in solids modeling systems (set of equations) which is well known in the art. The result of the object representation process steps is the creation of an object representation M which is further processed by the stereoscopic image-pair generation process steps.
The stereoscopic image-pair generation process steps operate on the object representation, M and produces the right and left pixelized stereoscopic image-pairs Ipr and Ipl, respectively. The steps of the stereoscopic image-pair generation process use the transformations Tdv (acquired by the head position and orientation tracking subsystem), Tvel and Tver (acquired by the eye position and orientation tracking subsystem), and Twd (acquired by the display position and orientation tracking subsystem) and the acquisition of the display parameters msd, ss, and sd to compute various transformations and mappings as will be described next.
The transformation Tmq describes the position and orientation placement of the right and left stereoscopic image-pair generation surfaces scr and scl. Tmq is computed by the function fTmq which accepts as parameters Twd, Tdv, Tvel, Tver, and pm. fTmq computes Tmq such that the images Icr and Icl taken together form the beginnings of a stereoscopic image-pair which represents a portion of the object(s) M.
The transformation Tqcr describes the position and orientation placement of the right stereoscopic image-pair acquisition surface Icr with respect to pq. Tqcr is computed by the function fTqcr which accepts as parameters Tmq, Tdv, Tvel, and Tver. fTqcr computes Tqcr such that the image Icr from the surface scr forms the beginnings of a realistic stereoscopic image-pair. In a similar manor, the transformation Tqcl describes the position and orientation placement of the left stereoscopic image-pair acquisition surface Icl with respect to pq. Tqcl is computed by the function fTqcl which accepts as parameters Tmq, Tdv, Tvel, and Tver. fTqcl computes Tqcl such that the image Icl from the surface scl forms the beginnings of a realistic stereoscopic image-pair.
The right object mapping, mmcr, creates an image representation, Icr, of M, onto the surface scr. mmcr is computed by the function fmmcr which accepts as parameters Tmq, Tqcr, Tdv, Tver, sd, and msd. mmcr represents a synthetic rendering process (well known in the art). In a similar manor, the left object mapping, mmcl, creates an image representation, Icl, of M, onto the surface scl. mmcl is computed by the function fmmcl which accepts as parameters Tmq, Tqcl, Tdv, Tvel, sd, and msd. mmcl represents a geometric rendering process (well known in the art).
The image representation Icr on surface scr is formed by the function fIcr which accepts as parameters mmcr and M. In a similar manor, the image representation Icl on surface scl is formed by the function Icl which accepts as parameters mmcl and M. Mappings mmcr and mmcl are defined in such a way that the images Icr and Icl taken together form the beginnings of a realistic right and left, respectively, stereoscopic image-pair which represents a portion of the object(s) M.
The right quantization mapping, mcpr, describes the conversion of the object image Icr, into a pixelized image Ipr, on the synthetic surface, spr. mcpr is computed by the function fmcpr which accepts as parameters Tdv, Tvel, Tver, sd, ss, and msd. The quantization mapping, mcpr indicates how an arbitrary region of Icr is mapped into the pixelized image Ipr. In a similar manor, the left quantization mapping, mcpl, describes the conversion of the object image Icl, into a pixelized image Ipl, on the synthetic surface, spl. mcpl is computed by the function fmcpl which accepts as parameters Tdv, Tvel, Tver, sd, ss, and msd. The quantization mapping, mcpl indicates how an arbitrary region of Icl is mapped into the pixelized image Ipr.
The right pixelized image Ipr on surface spr is formed by the function fIpr which accepts as parameters Icr, mcpr, and dl, where dl represents supplemental pixel information. In a similar manor, the left pixelized image Ipl on surface spl is formed by the function fIpl which accepts as parameters Icl, mcpl, and dr, where dr represents supplemental pixel information. Mappings mcpr and mcpl are defined in such a way that the resulting images Ipr and Ipl will lead to the creation of a realistic stereoscopic image-pair in later steps of this process. Mappings mcpr and mcpl can also be used to correct for limitations of an implemented system for performing the mappings mmcr and mmcl described above.
The stereoscopic image spatial multiplexing process steps operate on the right and left pixelized images Ipr and Ipl respectively and produces the right and left spatially multiplexed stereoscopic image representations Isl and Isr. The steps of the stereoscopic image-pair spatial multiplexing process use the transformations Tdv (acquired by the head position and orientation tracking subsystem), and Tvel and Tver (acquired by the eye position and orientation tracking subsystem), and the acquisition of the display parameters msd, ss, and sd to compute various transformations and mappings as will be described next.
The right multiplexing mapping, mpsr, defines the mapping of pixels in Ipr to pixels in Isr. mpsr is computed by the function fmpsr which accepts as parameters Tdv, Tvel, Tver, sd, ss, and msd. In a similar manor, the left multiplexing mapping, mpsl, defines the mapping of pixels in Ipl to pixels in Isl. mpsl is computed by the function fmpsl which accepts as parameters Tdv, Tvel, Tver, sd, ss, and msd.
The right multiplexed image Isr, on surface ss, is formed by the function fIsr which accepts as parameters Ipr and mpsr. likewise, the left multiplexed image Isl, on surface ss, is formed by the function fIsl which accepts as parameters Ipl and mpsl. Isr and Isl are formed by the mappings mpsr and mpsl to be compatible with the micro-polarizing filter based stereoscopic image-pair display subsystem. The composite multiplexed stereoscopic image, Is, is formed from the compositing of Isr and Isl.
The stereoscopic image-pair display process steps operate on the right and left stereoscopic images Isr and Isl, respectively, using the display mapping msd, to display the right and left stereoscopic image display pairs Idr and Idl on the micro-polarizer based display surface, sd. The mapping msd represent projection optics.
The right stereoscopic display image Idr, on surface sd, is formed by the function/process fIdr which accepts as parameters Isr and msd. likewise, the left stereoscopic display image Idl, on surface sd, is formed by the function/process fIdl which accepts as parameters Isl and msd. The function/processes Idr and fIdl form the stereoscopic encoding process which encodes the right and left stereoscopic display images, Idr and Idl, using polarized light (via the application of a micro-polarizer to the display surface sd) so at to be viewed in a stereoscopic viewing mode by the stereoscopic image-pair viewing process or precesses. The composite multiplexed stereoscopic display image, Id, is formed from the compositing of Idr and Idl.
The stereoscopic display surface, sd, has imbedded coordinates pd which are related to pw by the transformation Twd. The display position and orientation tracking process tracks the interaction of the display with the virtual environment M′ and acquires the transformation Twd.
The stereoscopic image-pair viewing process steps represent the viewing decoding of the right and left stereoscopic display images, Idr and Idl, through the decoding mappings mder and mdel, respectively, to produce the right and left viewer images, Ier and Iel, respectively. The right and left viewer Images Ier and Iel are formed on the right and left viewing surfaces ser and sel, respectively. The right viewing surface, ser, has imbedded coordinate frame per. Coordinate frame per is related to frame pv by the transformation Tver. Likewise, the left viewing surface, sel, has imbedded coordinate frame pel. Coordinate frame pel is related to frame pv by the transformation Tvel. The function/process fIpr accepts parameters Idr, and mder and performs the actual decoding of the image Idr to form the image Ier using a polarizing filter decoder. Likewise, the function/process fIpl accepts parameters Idl, and mdel and performs the actual decoding of the image Idl to form the image Iel using a polarizing filter decoder. The combination of images Ier and Iel in the visual processing center, B, forms the image Ib. Ib represents the perceived stereoscopic image M′ as represented in the visual processing center B through the use of the function/process fIb.
Coordinate frame pv represents the imbedded coordinate frame of the combined right and left viewing surfaces ser and sel, respectively. pv is related to the stereoscopic image-pair display coordinates system, pd, by the transformation Tdv.
The head position and orientation tracking process tracks the interaction of the combined right and left viewing surfaces, ser and sel, with the display surface, sd, and acquires the transformation Tdv to describe this interaction. The eye position and orientation tracking process tracks the interaction of each individual right and left viewing surface, ser and sel, with respect to the coordinate frame pv, and acquires the right and left viewing surface transformations, Tver and Tvel.
The overall process steps defined by the process groups A through E in
In this embodiment, images are dynamically changing or static object or objects, M, composed of real objects, Mr, are imaged by subsystem 8. These objects, M, are referenced to the coordinate frame pm in the image acquisition space RA.
The stereoscopic image-pair acquisition subsystem performs the right and left object surfaces scr and scl (with imbedded coordinate frames pcr and pcl), the right and left pixel surfaces spr and spl, the right and left object mappings mmcr and mmcl, the right and left quanitization mappings mcpr and mcpl, the coordinate frame pq, and the supplemental right and left pixel information dr and dl. Coordinate frame pq is referenced with respect to the coordinate frame pm by the transformation Tmq.
The right object mapping, mmcr, creates an image representation, Icr, of M, onto the surface scr. In a similar manner, the left object mapping, mmcl, creates an image representation, Icl, of M, onto the surface scl Both surfaces scr and scl are referenced with respect to coordinate frame pq by transformations Tqcr and Tqcl, respectively. The object mappings mmcr and mmcl are optical imaging process. Images Icr and Icl (on surfaces scr and scl, respectively) are represented by physical optical images. The images Icr and Icl taken together form the beginnings of a stereoscopic image-pair which represent a portion of the virtual object(s) M. The transforms Tmq, Tqcr, and Tqcl and the mappings mmcr and mmcl are defined such that the resulting images Icr and Icl will lead to the creation of a realistic stereoscopic image-pair in later steps of this process.
The right quantization mapping, mcpr, describes the conversion of the object image Icr, into a pixelized image Ipr, on the synthetic surface, spr. The image Ipr can be modified with the supplemental pixel information dr. In a similar manor, the left quantization mapping, mcpl, describes the conversion of the object image Icl, into a pixelized image Ipl, on the synthetic surface, spl. The image Ipl can be modified with the supplemental pixel information dl. The pixelized images Ipr and Ipl are represented by a 2-D array, where each element in the array represents a spatial pixel in the image and contains spectral data about the pixel. The quantization mappings, mcpr and mcpl, indicate how an arbitrary region of Icr and Icl (respectively) are mapped into the pixelized images Ipr and Ipl.
The stereoscopic image multiplexing subsystem performs the right and left multiplexing mappings mpsr and mpsl, the stereoscopic image surface ss, the right multiplexing image Isr, and left multiplexed image Isl, and the composite multiplexed image Is. The right multiplexing mapping mpsr defines the spatial mapping of pixels in Ipr to pixels in Isr. Similarly, the left multiplexing mapping mpsl defines the spatial mapping of pixels in Ipl to pixels in Isl. The images Isr and Isl represent the right and left eye stereoscopic perspectives of the object(s) M. Isr and Isl are formed by the mappings mpsr and mpsl in such a manor as to be compatible with the micro-polarizer based stereoscopic image-pair display subsystem. Is is formed from Isr and Isl as will be described later.
The stereoscopic image-pair display subsystem performs the mapping msd, the display surface sd, the right stereoscopic spatial multiplexed display image Idr, the left stereoscopic spatial multiplexed display image Idl, and the composite stereoscopic spatial multiplexed display image Id, and the coordinate frame pd. The mapping msd defines the mappings of the pixels of Is onto the display pixels as represented by Id. The mapping msd represents an optical projection subsystem and can include some scaling factors between the image acquisition space RA and the image display space RB. The images Idr and Idl form a realistic spatially multiplexed stereoscopic display image-pair which, when viewed by the stereoscopic image-pair viewing subsystem, form a realistic representation, M′, of the object(s) M. The virtual object M′, is represented in the image display space, RB, which is referenced to coordinate frame pw. The display surface sd, contains a micro-polarizer array which performs polarization encoding of the images Idr and Idl. Surface sd has imbedded coordinates pd which are referenced to the image display space coordinates pw by the transformation Twd.
The stereoscopic image-pair viewing subsystem performs the right and left optical imaging mappings mder and mdel, the right viewing surface ser with imbedded coordinate system per, the left viewing surface sel with imbedded coordinate system pel, the right and left viewed images Ier and Iel, the viewing coordinate system pv, and the visual processing subsystem B. The right viewing image Ier is formed on the right viewing surface ser by the right optical imaging mapping mder which performs a polarization decoding process. The left viewing image Iel is formed on the left viewing surface sel by the left optical imaging mapping mdel which performs a polarization decoding process. The relationship between the right an left viewing surfaces and the viewing coordinate system, pv, is given by the transformations Tver and Tvel respectively. The relationship between the viewing coordinate system, pv, and the display surface coordinate system pd is given by the transformation Tdv. The transformations Tdv, Tver, and Tvel describe the position and orientation of the right and left viewing surfaces with respect to the display surface sd.
The object representation process steps operate on real physical object, Mr. The result of the object representation process steps is the creation of an object representation M which is further processed by the stereoscopic image-pair acquisition process steps.
The stereoscopic image-pair generation process steps operate on the object representation, M and produces the right and left pixelized stereoscopic image-pairs Ipr and Ipl, respectively. The steps of the stereoscopic image-pair generation process use the transformations Tdv (acquired by the head position and orientation tracking subsystem), Tvel and Tver (acquired by the eye position and orientation tracking subsystem), and Twd (acquired by the display position and orientation tracking subsystem) and the acquisition of the display parameters msd, ss, and sd to compute various transformations and mappings as will be described next.
The transformation Tmq describes the position and orientation placement of the right and left stereoscopic image-pair acquisition surfaces scr and scl. Tmq is computed by the function fTmq which accepts as parameters Twd, Tdv, Tvel, Tver, and pm. fTmq computes Tmq such that the images Icr and Icl taken together form the beginnings of a stereoscopic image-pair which represents a portion of the object(s) M.
The transformation Tqcr describes the position and orientation placement of the right stereoscopic image-pair acquisition surface Icr with respect to pq. Tqcr is computed by the function fTqcr which accepts as parameters Tmq, Tdv, Tvel, and Tver. fTqcr computes Tqcr such that the image Icr from the surface scr forms the beginnings of a realistic stereoscopic image-pair. In a similar manor, the transformation Tqcl describes the position and orientation placement of the left stereoscopic image-pair acquisition surface Icl with respect to pq. Tqcl is computed by the function fTqcl which accepts as parameters Tmq, Tdv, Tvel, and Tver. fTqcl computes Tqcl such that the image Icl from the surface scl forms the beginnings of a realistic stereoscopic image-pair.
The right object mapping, mmcr, creates an image representation, Icr, of M, onto the surface scr. mmcr is computed by the function fmmcr which accepts as parameters Tmq, Tqcr, Tdv, Tver, sd, and msd. mmcr represents an optical imaging process (well known in the art). In a similar manor, the left object mapping, mmcl, creates an image representation, Icl, of M, onto the surface scl. mmcl is computed by the function fmmcl which accepts as parameters Tmq, Tqcl, Tdv, Tvel, sd, and msd. mmcl represents an optical imaging process (well known in the art).
The image representation Icr on surface scr is formed by the function fIcr which accepts as parameters mmcr and M. In a similar manor, the image representation Icl on surface scl is formed by the function fIcl which accepts as parameters mmcl and M. Mappings mmcr and mmcl are defined in such a way that the images Icr and Icl taken together form the beginnings of a realistic right and left, respectively, stereoscopic image-pair which represents a portion of the object(s) M.
The right quantization mapping, mcpr, describes the conversion of the object image Icr, into a pixelized image Ipr, on the synthetic surface, spr. mcpr is computed by the function fmcpr which accepts as parameters Tdv, Tvel, Tver, sd, ss, and msd. The quantization mapping, mcpr indicates how an arbitrary region of Icr is mapped into the pixelized image Ipr. In a similar manor, the left quantization mapping, mcpl, describes the conversion of the object image Icl, into a pixelized image Ipl, on the synthetic surface, spl. mcpl is computed by the function fmcpl which accepts as parameters Tdv, Tvel, Tver, sd, ss, and msd. The quantization mapping, mcpl indicates how an arbitrary region of Icl is mapped into the pixelized image Ipr.
The right pixelized image Ipr on surface spr is formed by the function Ipr which accepts as parameters Icr, mcpr, and dl, where dl represents supplemental pixel information. In a similar manor, the left pixelized image Ipl on surface spl is formed by the function fIpl which accepts as parameters Icl, mcpl, and dr, where dr represents supplemental pixel information. Mappings mcpr and mcpl are defined in such a way that the resulting images Ipr and Ipl will lead to the creation of a realistic stereoscopic image-pair in later steps of this process. Mappings mcpr and mcpl can also be used to correct for limitations of an implemented system for performing the mappings mmcr and mmcl described above.
The stereoscopic image spatial multiplexing process steps operate on the right and left pixelized images Ipr and Ipl respectively and produces the right and left spatially multiplexed stereoscopic image representations Isl and Isr. The steps of the stereoscopic image-pair spatial multiplexing process use the transformations Tdv (acquired by the head position and orientation tracking subsystem), and Tvel and Tver (acquired by the eye position and orientation tracking subsystem), and the acquisition of the display parameters msd, ss, and sd to compute various transformations and mappings as will be described next.
The right multiplexing mapping, mpsr, defines the mapping of pixels in Ipr to pixels in Isr. mpsr is computed by the function fmpsr which accepts as parameters Tdv, Tvel, Tver, sd, ss, and msd. In a similar manor, the left multiplexing mapping, mpsl, defines the mapping of pixels in Ipl to pixels in Isl. mpsl is computed by the function fmpsl which accepts as parameters Tdv, Tvel, Tver, sd, ss, and msd.
The right multiplexed image Isr, on surface ss, is formed by the function fIsr which accepts as parameters Ipr and mpsr. likewise, the left multiplexed image Isl, on surface ss, is formed by the function fIsl which accepts as parameters Ipl and mpsl. Isr and Isl are formed by the mappings mpsr and mpsl to be compatible with the micro-polarizing filter based stereoscopic image-pair display subsystem. The composite multiplexed stereoscopic image, Is, is formed from the compositing of Isr and Isl.
The stereoscopic image-pair display process steps operate on the right and left stereoscopic images Isr and Isl, respectively, using the display mapping msd, to display the right and left stereoscopic image display pairs Idr and Idl on the micro-polarizer based display surface, sd. The mapping msd represent projection optics.
The right stereoscopic display image Idr, on surface sd, is formed by the function/process fIdr which accepts as parameters Isr and msd. likewise, the left stereoscopic display image Idl, on surface sd, is formed by the function/process fIdl which accepts as parameters Isl and msd. The function/processes fIdr and fIdl form the stereoscopic encoding process which encodes the right and left stereoscopic display images, Idr and Idl, using polarized light (via the application of a micro-polarization panel 41 to the display surface sd) so at to be viewed in a stereoscopic viewing mode by the stereoscopic image-pair viewing process or precesses. The composite multiplexed stereoscopic display image, Id, is formed from the compositing of Idr and Idl.
The stereoscopic display surface, sd, has imbedded coordinates pd which are related to pw by the transformation Twd. The display position and orientation tracking process tracks the interaction of the display with the virtual environment M′ and acquires the transformation Twd.
The stereoscopic image-pair viewing process steps represent the viewing decoding of the right and left stereoscopic display images, Idr and Idl, through the decoding mappings mder and mdel, respectively, to produce the right and left viewer images, Ier and Iel, respectively. The right and left viewer Images Ier and Iel are formed on the right and left viewing surfaces ser and sel, respectively. The right viewing surface, ser, has imbedded coordinate frame per. Coordinate frame per is related to frame pv by the transformation Tver. Likewise, the left viewing surface, sel, has imbedded coordinate frame pel. Coordinate frame pel is related to frame pv by the transformation Tvel. The function/process fIpr accepts parameters Idr, and mder and performs the actual decoding of the image Idr to form the image Ier using a polarizing filter decoder. Likewise, the function/process fIpl accepts parameters Idl, and mdel and performs the actual decoding of the image Idl to form the image Iel using a polarizing filter decoder. The combination of images Ier and Iel in the visual processing center, B, forms the image Ib. Ib represents the perceived stereoscopic image M′ as represented in the visual processing center B through the use of the function/process fIb.
Coordinate frame pv represents the imbedded coordinate frame of the combined right and left viewing surfaces ser and sel, respectively. pv is related to the stereoscopic image-pair display coordinates system, pd, by the transformation Tdv.
The head position and orientation tracking process tracks the interaction of the combined right and left viewing surfaces, ser and sel, with the display surface, sd, and acquires the transformation Tdv to describe this interaction. The eye position and orientation tracking process tracks the interaction of each individual right and left viewing surface, ser and sel, with respect to the coordinate frame pv, and acquires the right and left viewing surface transformations, Tver and Tvel.
The overall process steps defined by the process groups A through E in
Having described the illustrative embodiments of the present invention, several modifications readily come to mind.
In particular, the display subsystem 6 and display surface 40 maybe realized using the 3-D projection display system and surface disclosed in Applicants copending application Ser. No. 08-339,986.
The system and method of the present invention have been described in great detail with reference to the above illustrative embodiments. However, it is understood that other modifications to the illustrative embodiments will readily occur to persons with ordinary skill in the art. All such modifications and variations are deemed to be within the scope and spirit of the present invention as defined by the accompanying Claims to Invention.
This application is a CON of Ser. No. 09/451,012 Nov. 29, 1999 U.S. Pat. No. 6,556,236 which is a CON of Ser. No. 08/375,905 Jan. 20, 1995 U.S. Pat. No. 6,011,581. This Patent Application is a Continuation-in-Part of patent application Ser. No. 08/339,986 now U.S. Pat. No. 5,502,481 entitled “Desktop-Based Projection Display System For Stereoscopic Viewing of Displayed Imagery Over A Wide Field Of View” filed Nov. 14, 1994 by Dentinger, et al.; co-pending patent application Ser. No. 08/126,077 entitled “A System for Producing 3-D Stereo Images” filed Sep. 23, 1993 by Sadeg M. Faris; co-pending patent application Ser. No. 08/269,202 entitled “Methods for Manufacturing Micro-Polarizers” filed on Jun. 30, 1994 by Sadeg M. Faris; and co-pending patent application Ser. No. 07/976,518 entitled “Method and Apparatus for Producing and Recording Spatially-Multiplexed Images for Use in 3-D Stereoscopic Viewing Thereof” filed Nov. 16, 1992 by Sadeg M. Faris. Each of these copending Patent Applications is incorporated herein by reference in its entirety.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4987487 | Ichinose et al. | Jan 1991 | A |
| 6608622 | Katayama et al. | Aug 2003 | B1 |
| Number | Date | Country | |
|---|---|---|---|
| 20040036763 A1 | Feb 2004 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 09451012 | Nov 1999 | US |
| Child | 10423488 | US | |
| Parent | 08375905 | Jan 1995 | US |
| Child | 09451012 | US | |
| Parent | 10423488 | US | |
| Child | 09451012 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 08339986 | Nov 1994 | US |
| Child | 10423488 | US |
