Not Applicable.
Not Applicable.
The present invention relates generally to the display of holographic images. More particularly, the invention relates to a system and method for the production and display of pseudo holographic images generated from stereoscopic natural and synthetic images, or from photographs taken with multiple cameras set up to point concentrically towards the subject photographed, on viewer position aware devices such as cell phones, PDA's, computer monitors etc.
The three dimensional (3D) display of images using the technique of stereoscopy is well known in the art. The stereoscopic viewing of 3D images requires cumbersome use of special eye wear and display equipment. For example, U.S. Pat. No. RE39342 (Starks et. al.) discusses a method for converting source images to synthesized stereoscopic images where at least one of the images has been modified relative to the source image and the modified stereo images are viewed through stereo viewing glasses.
Auto-stereoscopic 3D images are an improvement over conventional stereoscopic 3D images. Unlike stereoscopic 3D images, no special glasses are needed to view these images to experience the illusion of depth in the scene presented. The 3D images generated by autostereoscopy are the result of a simulated parallax effect produced by a range of multiple available viewing angles. U.S. Pat. No. 7,446,733 (Hirimai) provides for the projection of a three dimensional still or dynamic image without any need for dedicated eye glasses or coherent light.
Generation of stereoscopic images from 2D images has been achieved by various methods and processes. For example, Intl. Apl. No. PCTIKR2008/000015 (Kim) discloses a method that generates stereoscopic images from 2D images using a mesh map. 3D images for display have also been generated from 2D images using the technique of integral photography. For example, U.S. Pat. No. 7,142,232 (Kremen) describes a method and apparatus for reducing a three-dimensional scene to an integral photograph or fabricating holograms of three-dimensional scenes and thereafter magnifying the integral photograph or holograms to project a three-dimensional image for use in home entertainment and computer technology.
Conversion of 2D images to 3D images is also well known in the art. U.S. Pat. No. 6,686,926 (Kaye) describes a dimensional process for the conversion of 2D images to 3D images which involves scanning images into a computer based system and with the use of graphic image software, creating a three-dimensional image that can be used for viewing and for re-recording for three-dimensional viewing. The process described in this patent converts only a single image to a 3D image.
3D still and dynamic images have also been generated for display on electronic devices. The auto-stereoscopic 3D display of images is gaining in popularity among users of handheld devices such as cell phones, and Personal Digital Assistant device (PDA)'s as well as laptop and desktop computers. Intl. Apl. No. PCT/US2008/050025 provides visualization of 3D video images on a stereo enabled display of a low power device, such as a mobile phone, a computer, a video game platform, or PDA. Computer implemented methods for rendering images on a two dimensional display, such as a computer, handheld device, or television screen so that the image appears to be positioned in a three dimensional space is disclosed in Intl. Apl. No. PCT/US2006/032093.
Imparting three-dimensional characteristics in a two dimensional space to create the perception of depth to images is described in Intl. Apl. No. PCT/US2008/068072. Intl. Apl. No. PCT/US2006/032093, discloses a system and computer implemented method for rendering images on a two dimensional display such as a computer, handheld device, or television screen so that the images appear to be positioned in a three-dimensional space.
Integration and display of a 3D image and a camera image in a mobile device is disclosed in U.S. App. No. 20090195539 (Kim). In this application, the processing method for the integration between the two images involves recognition of a specific image pattern defined by the user, tracing of that pattern within an image, and the interfacing of a camera image and a 3D image based on the tracing results. The 3D object is animated and rendered using a 3D graphic engine and the rendered image of the 3D object and the camera image are integrated and displayed.
Holographic displays unlike stereoscopic and auto-stereoscopic displays provide a more realistic three dimensional view of the principal objects and scenery of a captured image. 3D holographic images are not well known in the art. Currently, 3D images for handheld devices such as smart phones involve synthetic creation of dynamic images that require rotation of the phone in all directions in order to observe the movement of the image.
Systems and methods for creating, editing, and distributing stereoscopic images have also been disclosed in prior art. For example, Intl. Apl. No. W00180548 describes a system and method for the registration and distribution of stereoscopically complementary pairs of images suitable for a variety of different viewing formats.
The 3D displays currently in vogue generated on flat panel screens of handheld devices and computer monitors, use lenticular lenses or a parallax barrier. Other displays use eye tracking systems to automatically adjust the two displayed images to follow the viewer's eyes as they move their head (http://en.wikipedia.org/wiki/Autostereoscopy).
The present invention is capable of generating pseudo holographic images from real pictures that provide depth and volume perception with either a left to right movement of the smart phone device or when held at other angles. For devices that cannot move the relative position of the viewer to the display, the invention provides the depth and volume perception. The invention uses a process of matching elements in a pair of stereoscopic images, editing the images using the techniques of morphing, tweening and image warping among others to extrapolate an intermediary image which then further undergoes a post production phase of color correction, resizing etc. before being displayed as pseudo holographic image on a mobile device such as a smart phone, PDA, laptop computer, or stationary computer monitor.
The present invention provides for the creation of pseudo holographic images that can be viewed on handheld electronic devices or on computers without the need for specialized stereoscopic glasses or expensive accessories. The 3D virtual image is created and visualized by rotating the handheld device left to right or holding the device at any another angle. The angle of the phone or handheld device will determine the picture selected to be displayed from among the actual and intermediary generated pictures. The angle at which the viewer sees the computer displays will determine the picture selected to be displayed from among the actual and/or intermediary generated pictures.
The pseudo hologram as defined by the present invention contains stereo images (or a set of images), each containing one or more layers of subjects and backgrounds. The layers are generated from different camera shots or from different image sources if the final pseudo hologram is made of mixed media. The present invention uses tools to create a pseudo hologram from a pair of stereo images or a set of images (more than one pair of stereo images, concentric shots), exports a set of intermediary images, and/or displays them natively. The process used for the generation of the holographic images and their display on the screens of the electronic devices is initiated by the acquisition and importation of a set of natural or synthetic stereoscopic images, or a set of pictures from concentric cameras.
The type of stereoscopic images that can be imported and used to generate the holographic image may be, a pair of parallel stereoscopic photographs, a pair of toe-in stereoscopic photographs, two pairs of toe-in stereoscopic photographs, frames from 3D movies, stereo images created with computer software, drawings, or any other hand or computer generated stereoscopic images. Once the images have been acquired, the acquired images undergo specific alterations to produce the desired final images. The processing procedures applied to the images may include standard photography post-production procedures such as smoothing the skin of a person, adjusting colors and contrasts, removing or adding elements to the image etc. The left and right stereoscopic images preferably undergo the same processing procedures. In stereo-photography, a standard processing procedure is to correct color, misalignment, optical distortion, or tilt. When more than one pair of stereo images or when concentric images are captured, misalignment, distortion and tilt may occur as well.
When intermediary images are extrapolated, the next phase involves matching key corresponding elements on both left and right source images. This part of the process can be done manually or through automation using pattern detection software. The elements of the main subject and back grounds as well as the boundaries between the two can be grouped separately to undergo their own transformation.
The generation of intermediary images can be achieved through a variety of processes known in the art such as morphing, tweening, image warping and the like. The idea is to extrapolate an image of the subject from a viewing angle from which the subject was not actually photographed. The intermediary images can be generated in real time or pre-generated for viewing according to the convenience and desire of the viewer. Pre-generated intermediary images would be better suited in the case of a mobile device which does not have sufficient computing power to generate the intermediary pictures in real time.
In the second post production phase certain modifications and effects are applied to images that are pre-generated, to fix errors in the generation of the intermediary images, such as transformation of some of the elements in the images to make them appear unreal. Some of the modifications and effects include, watermarking (adding text or logo), resizing, color correction, adding frames around certain images etc.
The final step is the display of the pseudo holographic image generated from the original images and processed intermediary images. At this stage, the volume and depth of the image will be apparent to the viewer of the screen in which the image is displayed and will appear as a pseudo holographic image. The holographic appearance of the image can be visualized by just the left to right movement of a hand held device or by holding the device at any angle to produce the desired holographic experience.
Desk top computer and other fixed monitors can also be used to display the pseudo holographic images generated by the present invention. In that instance, the image to be displayed would be selected according to the mouse position or the viewer's position. The viewer position can be deduced by the software processing the pictures of a webcam placed near a monitor. The software used would recognize movement and its relative position to choose the image to be displayed. In that instance, the image would appear to be tracing the viewer and would point towards the viewer.
a) is a flowchart of a method of producing a set of images for a pseudo holographic display from stereoscopic images formed from cameras positioned at different viewing angles in accordance with an embodiment of the present invention;
b) is a flowchart of a method of producing a set of images for a pseudo holographic display only from actual images formed from cameras positioned at different viewing angles in accordance with an embodiment of the present invention;
a-d) are diagrams of an exemplary camera arrangements or positioning for producing a pseudo holographic display from stereoscopic images in accordance with various embodiments of the present invention;
The present invention uses a set of actual, or a set of actual and extrapolated images, to produce a pseudo holographic display. Referring now to
Once the first post production phase is completed, the method proceeds to step 6 wherein the corresponding key elements in the left and right side image are matched. Corresponding elements in the two images, which can be in the left and right images for a stereo pair or in simply two adjacent view points, are identified so that the images can be aligned with respect to the subject matter of the images. These matching points should correspond to readily identifiable features of the image such as the eyes of a subject or the boundaries of an object. The matching process is discussed in more detail below with respect to
The intermediary images then undergo a second post production phase 10 wherein the intermediary images are subjected to standard image processing procedures such as smoothing the skin of a person and adjusting colors and contrasts in a manner similar to that applied to the original images in the first post production phase in step 4. The processed set of images 12 can then be used to create a pseudo holographic display as described in
b) is a flowchart of a method of producing a set of images for a pseudo holographic display only from actual images formed from cameras positioned at different viewing angles in accordance with an embodiment of the present invention.
Referring now to
Once the viewing angle is determined, an image corresponding to that viewing angle can be selected or calculated as shown in step 32. Depending upon the processing power of the device that will be used to produce the pseudo holographic display, the intermediary images may be pre-determined prior to any displaying of the images and stored in the device's memory or, if the device is quick enough, the intermediary images may be created on the fly as they are needed. Once the proper image is selected, the image is displayed in step 34 and the method returns to step 30 whereby the viewing angle of the display is again determined and another corresponding image selected. By doing so, the image will appear to the viewer to be tracing the viewer and will always point towards the original direction of the screen or viewer. Thus, when the device is turned, the orientation of the image appears to remain fixed thereby producing a pseudo holographic display.
As an example of the method of
Referring now to
Separate background images and subject images may be created and processed to enhance the pseudo holographic effect. When shooting a subject/person in a real environment, there may be details of the background that are obscured behind the subject/model. When the final composite image is turned, areas that were obscured by the subject in one view will move into a position whereby they should be visible to the viewer. However, if separate background and subject layers are not used, these background details are not available during creation of the intermediary images and processing. Therefore, when creating the pseudo holographic effect of the present invention with an image that has a main subject and a background, separate background and subject layers are preferably created. If the background is not shot separately from the subject, the background details in the intermediary images are extrapolated from the original actual images. The background layer is re-created by cutting out the subject. The empty spaces due to subject cut out can fall into two categories: space that appear in one of the actual images and disappear in the other actual image. These spaces or elements get warped from the actual position in the actual image they appear in to the position and size they would appear at if the subject was not in the actual image.
The basic process to add layers is to take a photo of the environment before the subject is put into place so that two pseudo holographic images can be created: one of the background/environment and one of just the subject. Each viewpoint image has a background layer (the corresponding viewpoint of the environment) and the subject layer (the corresponding viewpoint of subject) which is transparent around the subject so that the user sees through to the background layer. Alternatively, the subject is cutout and put in a layer in front of the background layer.
Referring now to
Once the initial subject images and background images are acquired, matching elements in each of the layers are then determined so that intermediary images can be created for both the subject and background layers of the images. A separate set of intermediary images are created from both the background images 66 and 68 and subject images 70 and 72 to produce separate sets of background images 76 and subject images 74 for use in producing the pseudo holographic effect. Images 74 and 76 are intermediary images composed of intermediary subject images from generated 70 and 72 combined with intermediary background images generated from 66 and 68. When movement of the device or user is detected, both a background image and a subject image corresponding to the change in viewing angle are selected for display. When the images are displayed, the subject images 70 and 72 and corresponding intermediary generated images between 70 and 72 are layered or placed in front of the background images 66 and 68 and corresponding intermediary generated images between 66 and 68. The pseudo holographic images can then be displayed such that both the background and the subject appear to follow the viewer. A green screen can be used to shoot the subject and the background added in post production with the viewing angle carefully matched to the subject. The background can be a real scene, or a calculated 3D environment, or a combination.
Referring now to
a) is a diagram of an exemplary stereoscopic camera arrangement for producing a pseudo holographic display from stereoscopic images in accordance with an embodiment of the present invention. The cameras 90 and 92 are positioned so that each camera has a different viewing angle with respect to the subjects 94 and 96 to be displayed.
b) is a diagram of an exemplary linear camera arrangement using three cameras for producing a pseudo holographic display from images in accordance with an embodiment of the present invention. The cameras 91, 93 and 95 are positioned so that each camera has a different viewing angle with respect to the subjects 97, 99 and 101 to be displayed.
c) is a diagram of an exemplary linear camera arrangement for producing a pseudo holographic display from images in accordance with an embodiment of the present invention. The five cameras 105, 107, 109, 111 and 113 are positioned so that each camera is pointed in a slight different direction so they have a different viewing angle with respect to the subjects 115, 117 and 119 to be displayed.
d) is a diagram of an exemplary linear camera arrangement for producing a pseudo holographic display from images in accordance with an embodiment of the present invention. The five cameras 105, 107, 109, 111 and 113 are positioned so that each camera is pointed in the same different but in a different position so they have a different viewing angle with respect to the subjects 115, 117 and 119 to be displayed.
A method of creating a pseudo holographic display with a series of cameras arranged concentrically or linearly begins with acquiring of the images in step 150. Similar post production processing is then applied to the images in step 152. Matching elements in the processed images are defined in step 154. Intermediary images are then extrapolated from the images in step 156 and post production processing applied to all of the images in step 158. In step 160, a software algorithm in the device determines the orientation of the device and selects an image for display such that an intelligent pseudo holographic display is created.
A method of creating a pseudo holographic display with a series of cameras arranged concentrically or linearly from only actual camera images begins with acquiring of the images in step 170. Similar post production processing is then applied to the images in steps 172 and 174. In step 176, a software algorithm in the device determines the orientation of the device and selects an image for display such that an intelligent pseudo holographic display is created.
A method of creating a pseudo holographic display with both a subject layer and a background layer begins with acquiring of the subject images in step 180 and background images in step 182. Similar post production processing is then applied to the images of the subject in step 184 and similar post production processing is then applied to the images of the background in step 186. Matching elements in the processed subject and background images are defined in steps 188 and 190. Intermediary images are then extrapolated from the images in steps 192 and 194. The subject and background images are then combined in step 196. In step 198, a software algorithm in the device determines the orientation of the device and selects the appropriate subject and background image for display such that an intelligent pseudo holographic display is created.
Additional cameras can be used to further refine the pseudo holographic effect.
The foregoing description and summary of the invention with the preferred embodiments accompanied by the drawings should not be construed to limit the scope of the invention. It should be understood and obvious to one skilled in the art that the embodiments of the invention thus described may be further modified without departing from the spirit and scope of the invention.
The present application claims priority from co-pending U.S. Provisional Patent Application Ser. No. 61/240,394 filed Sep. 8, 2009.
Number | Date | Country | |
---|---|---|---|
61240394 | Sep 2009 | US |