The invention relates to a method for visualisation of a 3-dimensional (3-D) scene model of a 3-D image, with a 3-D display plane comprising 3-D pixels by emitting and/or transmitting light into certain directions by said 3-D pixels, thus visualising 3-D scene points.
The invention further relates to a 3-D display device comprising a 3-D display plane with 3-D pixels.
Three dimensional television (3-DTV) is a major goal in broadcast television systems. By providing 3-DTV, the user is provided with a visual impression that is as close as possible to the impression given by the original scene. There are three different methods for providing a 3-dimensional impression which are accommodation, which means that the eyelens adapts to the depth of the scene, stereo, which means that both eyes see a slightly different view on the scene, and motion parallax, which means that moving the head will give a new and possibly very different view on the scene.
One approach for providing a good impressing of a 3-D image is to record a scene by a high number of cameras. Each camera capturing the scene from a different viewpoint. For displaying the captured images, all of these images have to be displayed in viewing directions corresponding to the camera positions. During acquisition, transmission, and display occur many problems, as many cameras need much room and have to be placed very close to each other, the images from the cameras require high bandwidth to be transmitted, and also an enormous amount of signal processing for compression, decompression is needed and finally, many images have to be shown simultaneously.
From document WO 99/05559 a method for providing an N-view autostereoscopic display is disclosed, using a lenticular screen. By using the lenticular screen, each pixel may direct its light into a different direction, where the lightbeam of one lenticule is a parallel lightbeam. By providing this method, it is possible to display various views and thus providing a stereo impression for the viewer. The method disclosed therein needs the calculation of information about the direction of emission of light for each pixel outside each pixel.
Due to the deficiencies in the prior art method, it is an object of the invention to provide a method and a display device which allows bandwidth reduction between the display device and a control device. It is a further object of the invention to allow easy manufacturing of display devices. It is yet a further object of the invention to provide for a fully correct representation of the 3-D geometry of a 3-D scene.
These objects of the invention are solved by a method which is characterized in that said 3-D scene model is converted into a plurality of 3-D scene points, said 3-D scene points are fed at least partially to at least one of said 3-D pixels, and said at least one 3-D pixel calculates its contribution to the visualisation of a 3-D scene point. The calculation of the contribution of a 3-D pixel to a 3-D scene point within the 3-D pixel itself allows for high speed calculation of images. Also an enormous amount of images can be rendered without having to transmit these images from a separate unit to the display.
A 2-D pixel may be a device that can modulate the emission or transmission of light. A spatial light modulator may be a grid of NxxNy 2-D pixels. A 3-D pixel may be a device comprising a spatial light modulator that can direct light of different intensities in different directions. It may contain light sources, lenses, spatial light modulators and a control unit. A 3-D display plane may be a 2-D plane comprising an MxxMy grid of 3-D pixels. A 3-D display is the entire device for displaying images.
A voxel may be a small 3-D volume with the size Dx, Dy, Dz, located near the 3-D display plane. A 3-D voxel matrix may be a large volume with width and height equal to those of the 3-D display plane, and some depth. The 3-D voxel matrix may comprise Mx*My*Mz voxels. The 3-D display resolution may be understood as the size of a voxel. A 3-D scene may be understood as an original scene with objects.
A 3-D scene model may be understood as a digital representation in any format containing visual information about the 3-D scene. Such a model may contain information about a plurality of scene points. Some models may have surfaces as elements (VRML) which implicitly represent points. A cloud of points model may explicitly represent points. A 3-D scene point is one point within a 3-D scene model. A control unit may be a rendering processor that has a 3-D scene point as input and provides data for a spatial light modulator in 3-D pixels.
A 3-D scene always consists of a number of 3-D scene points, which may be retrieved from a 3-D model of a 3-D image. These 3-D scene points are positioned within a 3-D voxel matrix in and outside the display plane. Whenever a 3-D scene point is placed within the display plane, all 2-D pixels within one 3-D pixel co-operate, emitting light in all directions, defining the maximum viewing angle. By emitting light in all directions, the user sees this 3-D scene point within the display plane. Whenever a number of 2-D pixels from different 3-D pixels co-operate, they may visualise scene points positioned within a 3-D voxel matrix.
The human visual system observes the visual scene points at those spatial locations, where the bundle of light rays is “thinnest”. For each scene point, the internal structure of the light that is “emitted” depends on the depth of the scene point. Light that emerges in different directions from it, originates from different locations, different 2-D pixels, within the scene point, but this is perceptually not visible as long as the structure is below the eye resolution. That means that a minimum viewing distance should be kept from the display, similar to any conventional display. By emitting light within each 3-D pixel into a certain direction, all emitted light rays of all 3-D pixels interact, and their bundle of light rays is “thinnest” at different locations. The light rays interact at voxels within a 3-D voxel matrix. Each voxel may represent different 3-D scene points.
Each 3-D pixel may decipher whether or not to contribute to the 3-D 20 displaying of a particular 3-D scene point. This is a so called “rendering process” of one 3-D pixel. Rendering in the entire display is enabled by deciphering all 3-D scene points from one 3-D scene for or by all 3-D pixels.
A method according to claim 2 is preferred. 2-D pixels of one 3-D pixel contribute light to one 3-D scene point. Depending on the spatial position of a 3-D scene point, 2-D pixels from different 3-D pixels emit light so that the impression on a viewer's side is that the 3-D scene point is exactly at its spatial position as in the 3-D scene.
To provide a method which is resilient to errors within 3-D pixels, a method according to claim 3 is provided. By redistributing the 3-D scene points, errors in single 3-D pixels may be circumvented. The other 3-D pixels still provide light for the display of a 3-D scene point. Further, as missing 3-D pixels are similar to bad 3-D pixels, a square and a flat panel display can then be cut into an arbitrary shaped plane. Also, multiple display planes can be combined into one plane by only connecting their 3-D pixels. The resulting plane will still show the complete 3-D scene, only the shape of the plane will prohibit viewing the scene from some specific angles.
Parallel to redistributing the 3-D scene points within all 3-D pixels a distribution according to claim 4 is preferred. In this so called “load” mode, all images are actually acquired or rendered outside the 3-D pixels. After that they are loaded into the 3-D pixels. This may be interesting for displaying still images.
Rather than performing rendering in parallel within every 3-D pixel, a method according to claim 5 is proposed. A rendering process, e.g. the decision which 2-D pixel contributes light to displaying a 3-D scene point, can be done partly non-parallel by connecting several 3-D pixels to one rendering processor or to comprise a rendering processor within “master” pixels. An example is, to provide all rows of 3-D pixels of the display with one dedicated 3-D pixel comprising a rendering processor. In that case an outermost column of 3-D pixels may act as “master” pixel for that row, while the other pixels of that row may serve as “slave” pixels. The rendering is done in parallel by dedicated processors for all rows, but sequential within each row.
A method according to claim 6 is further preferred. All 3-D scene points within a 3-D model are offered to one or more 3-D pixels. Each 3-D pixel redistributes all 3-D scene points from its input to one or more neighbours. Effectively, all scene points are transmitted to all 3-D pixels. A 3-D scene point is a data-set, with information about position, luminance, colour, and further relevant data.
Each 3-D scene point has co-ordinates x, z, y and a luminance value I. The 3-D size of a 3-D scene point is determined by the 3-D resolution of the display which may be the size of the voxel of the 3-D voxel matrix. All of the 3-D scene points are sequentially, or in parallel, offered to substantially all 3-D pixels.
In general, each 3-D pixel has to know its relative position within the display plane grid to allow a correct calculation of the 2-D pixels contributing light to a certain 3-D scene point. However, a method according to claim 7 solves this problem. Each 3-D pixel may then change the co-ordinates of 3-D scene points slightly before transmitting them to its neighbours. This can be used to account for the relative difference in position between two 3-D pixels. In that case, no global position information needs to be stored within 3-D pixels, and the inner structure of all 3-D pixels can be the same over the entire display.
A so called “z-buffer” mechanism is provided according to claim 8. As a 3-D pixel receives a stream of all 3-D scene points, it may happen that more than one 3-D scene point needs the contribution of the same 2-D pixel. In case two 3-D scene points need for their visualisation the contribution of one 2-D pixel which is located within one 3-D pixel, it has to be decided which 3-D scene point “claims” this particular 2-D pixel. This decision is done by occlusion semantics, which means that the point that is closest to the viewer should be visible, as that point might occlude other scene points from his viewpoint.
As horizontal parallax is far more important than vertical parallax, a method according to claim 9 is provided. If horizontal parallax is incorporated, the number of 2-D pixels required for displaying a 3-D scene is reduced. A 3-D pixel with only one row of 2-D pixels might be sufficient for creating horizontal parallax.
To incorporate colour, a method according to claim 10 is provided. Within a 3-D pixel, more than one light source may be multiplexed spatially or temporally. It is also possible to have 3-D pixels for each basic colour, e.g. RGB. It should be noted that a triplet of three 3-D pixels may be incorporated as one 3-D pixel.
A further aspect of the invention is a display device, in particular for a pre-described method, where said 3-D pixels comprise an input port and an output port for receiving and putting out 3-D scene points of a 3-D scene, and said 3-D pixel at least partially comprise a control unit for calculating their contribution to the visualisation of a 3-D scene point representing said 3-D scene.
To enable transmission of 3-D scene points between 3-D pixels, a display device according to claim 12 is proposed.
A grid of 3-D pixels and a grid of 2-D pixels may also be provided. When the display is viewed at the correct minimum viewing distance, the grid of the 3-D pixels is below the eye resolution. Voxels will be observed with the same size. This size equals horizontally and vertically the size of the 3-D pixels. The size of a voxel in depth direction equals its horizontal size divided by tan (½α). Where a is the maximum viewing angle of each 3-D pixel, which also equals the total viewing angle of the display. For α=90°, the resolution is isotropic in all directions. The size of 3-D scene points grows linearly with depth, with a factor of 1+2|z|/N. This forms a restriction on how far scene points can be shown well in free space outside the display. At the depth position z=±½N scene points, the original resolution is divided in half in all directions, which can be taken as a maximum viewing bound.
A spatial light modulator according to claim 13 is preferred.
A display device according to claim 14 is also preferred, as by using a point light source, each 2-D pixel emits light into a very specific direction, all 2-D pixels of a 3-D pixel covering the maximum viewing angle.
During rendering, the display shows the previously rendered image. Only when an “end” signal is received, the entire display shows the newly rendered image. Therefore, buffering is needed as is provided by a display device according to claim 15. By using a so called “double buffering”, flickering during rendering may be avoided.
These and other aspects of the invention will be apparent from and elucidated with reference to the following figures. In the figures show:
a-c show top-views of 2-D pixels 6. In
b shows a collimated back-light for the entire display and a thick lens 9a This allows transmission of light in the whole viewing direction.
In
The 3-D scene which the display shows, always consists of a number of 3-D scene points. Whenever the scene point is within the display plane, all 2-D pixels 6 within the same 3-D pixel co-operate, as depicted by voxel A, which means that light from said point light source 5 is directed in all directions, emerging from this 3-D pixel 4. The user sees the 3-D scene point within voxel A.
Whenever a number of 2-D pixels 6 from different 3-D pixels 4 co-operate, they may visualise scene points at positions within the 3-D voxel matrix of the display plane as can be seen with voxel B.
The ray of lights emitted from the various 3-D pixels 4 co-operate and their bundle of lights is “thinnest” at the position of a 3-D scene point represented by voxel B. By deciding which 2-D pixels 6 contribute light to which 3-D scene point, a 3-D scene may be displayed within the display range of the display 2. When the display is viewed at the correct distance, the 2-D voxel matrix resolution is below the eye resolution.
As can be seen in
To create a view of 3-D scene point within voxel B, two dedicated points P and Q within the voxel B are selected as indicated. From these points P, Q, lines are drawn towards the point light sources 5 within the 3-D pixels 4. For the 3-D pixel 4 on the left, this results in the intersections Sx and Tx. All 2-D pixels that have their middle in between these two intersections Sx and Tx should contribute to the visualisation of the 3-D scene point bounded by said points P and Q. The distance between the intersections Tx and Sx is the distance Sz.
Transformed co-ordinates with the values Sz, Sx, Sy,Tx and Ty may be found for simplification of the implementation of the signal processing in the control units as
The values Sx, Sy and Sz are transformed co-ordinates. Their value is in units of the x2D and y2D axes, and can be fractional (implementation by floating point or fixed point numbers). When Z3D is zero, it can safely be set to a small non-zero value as e.g. Z3D=±½, to avoid infinity in
this has no visible effect.
For the right-neighbouring 3-D pixel, the above identified values are transformed by every 3-D pixel prior to transmitting it to its neighbours, which means that a 3-D pixel needs no information about its own location within the display and are practically the same:
Sz′=Sz
Sx′=Tx
Tx′=Sx′+Sz′
ty′=Sy′+Sz′.
A similar relation holds for neighbouring 3-D pixels in the vertical direction (not depicted in
An error resilient implementation of 3-D pixels is depicted in
The implementation of a 3-D pixel 4 is depicted in
Within the 3-D pixel 4, two additions are performed to obtain Tx and Ty, after which the transformed data set is sent to horizontal and vertical neighbouring 3-D pixels 4. The output is checked by block 4f. If the 3-D pixel 4 decides that it is not functioning correctly itself, via a self-check, it does not send its clock signal CLK to its neighbours, so that those 3-D pixels 4 will receive only data from other, correctly functioning neighbouring 3-D pixels 4. The additions performed in 3-D pixel 4 are Sx+Sz as well as Sy+Sz.
The rendering process is carried out within a 3-D pixel 4. To control the rendering process, global signals “start” and “end” are sent to all 3-D pixels within the entire display. Upon the reception of a “start” signal, all 3-D pixels are reset and all 3-D scene points to be rendered are sent to the display. As all 3-D scene points have to be provided to all 3-D pixels, some clock cycles have to be waited to ensure that the last 3-D scene point has been received by all 3-D pixels in the display. After that, the “end” signal is sent to all 3-D pixels of the display.
During the rendering period the display shows the previously rendered image. Only after reception of the “end” signal, the entire display shows the newly rendered image. This is a technique called “double buffering”. It avoids that viewers observe flickering. This might otherwise occur, as during rendering the luminance of 2-D pixels may change several times, e.g. due to “z-buffering”, since a new 3-D scene point may occlude a previous 3-D scene point.
The rendering within a 3-D pixel 4 is depicted in
The comparators in horizontal direction decide, whether the co-ordinates Sx and Tx lie within a 2-D pixel 0 to N-1 in horizontal direction. The comparators in vertical direction decide, whether the co-ordinates Sy and Ty lie within a 2-D pixel 0 to N-1 in vertical direction. If the co-ordinates lie between two 2-D pixels, the output of one of the comparators is HIGH and the output of the XOR box is also HIGH.
Within one 3-D pixel, Nx*Ny 2-D pixels are provided, with indexes 0<=ij<=N-1. Each 2-D pixel ij has registers, one for luminance Iij, one for transformed depth Sz,ij of the voxel to which this 2-D pixel is contributed at a particular moment during rendering, and one Rij coupled to the spatial light modulator of the 2-D pixel (not depicted). The luminance value for each pixel is determined by the variables ri and cj and the depth variable zij, which denotes the depth of the contributed voxel. The zij value is a boolean variable from the comparator COMP, that compares the current transformed depth Sz with the transformed depth Sz,ji.
Whether the contribution of a 2-D pixel to a past 3-D scene point should change to the 3-D scene point currently provided at the input depends on three necessary requirements:
a) the intersection requirement is met horizontally (ci=1);
b) the intersection requirement is met vertically (rj=1);
c) the current 3-D scene point lies closer to the viewer than the past 3-D scene point (zij=1).
The control signal “start” resets all registers. The register Iij is set to “black” and Szij to a value representing z=minus infinity. After that, all 3-D scene points are provided to all 3-D pixels. For each 3-D scene point, the luminance values for all 2-D pixels are determined. In case, a 2-D pixel lies between intersection S and T, which means ri=cj=1, a “z-buffer” mechanism decides whether the new 3-D scene point lies closer to the viewer than a previously rendered one. When this is the case, the 3-D pixel decides that the 2-D pixel should contribute to the visualisation of the current 3-D scene point. The 3-D pixel then copies the 3-D scene point luminance information into its register Iij and the 3-D scene point depth information into register Szij.
When the “end” signal is received, the luminance register Iij value is copied to the register Rij for determining the luminance of each 2-D pixel for displaying the 3-D image.
By providing the described method, any number of viewers can simultaneously view the display, no eye-wear is needed, stereo and motion parallax is provided for all viewers and the scene is displayed in fully correct 3-D geometry.
Number | Date | Country | Kind |
---|---|---|---|
02079580.3 | Nov 2002 | EP | regional |
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/IB03/04437 | 10/8/2003 | WO | 4/27/2005 |