From a user's point of view, motion parallax viewing techniques provide 3D perception of a 3D scene without requiring special viewing devices such as stereoscopic display devices, shuttle glasses, polarized glasses, and the like. Because the user's experience is equivalent to looking at the scene in a mirror or through the window, motion parallax viewing tends to not cause effects such as eye strain that are commonly associated with using special viewing devices. To date, the motion parallax effect has been used only for viewing 3D virtual content generated by computer graphics, but has not been employed for viewing 2D photo and/or video content captured by cameras. Employing motion parallax effect for viewing 2D photos and videos involves the extraction of 3D information from a real life scene during anchor after image capture.
The material described herein is illustrated by way of example and not by way of limitation in the accompanying figures. For simplicity and clarity of illustration, elements illustrated in the figures are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference labels have been repeated among the figures to indicate corresponding or analogous elements. In the figures:
One or more embodiments or implementations are now described with reference to the enclosed figures. While specific configurations and arrangements are discussed, it should be understood that this is done for illustrative purposes only. Persons skilled in the relevant art will recognize that other configurations and arrangements may be employed without departing from the spirit and scope of the description. It will be apparent to those skilled in the relevant art that techniques and/or arrangements described herein may also be employed in a variety of other systems and applications other than what is described herein.
While the following description sets forth various implementations that may be manifested in architectures such as system-on-a-chip (SoC) architectures for example, implementation of the techniques and/or arrangements described herein are not restricted to particular architectures and/or computing systems and may be implemented by any architecture and/or computing system for similar purposes. For instance, various architectures employing, for example, multiple integrated circuit (IC) chips and/or packages, and/or various computing devices and/or consumer electronic (CE) devices such as set top boxes, smart phones, etc., may implement the techniques and/or arrangements described herein. Further, while the following description may set forth numerous specific details such as logic implementations, types and interrelationships of system components, logic partitioning/integration choices, etc., claimed subject matter may be practiced without such specific details. In other instances, some material such as, for example, control structures and full software instruction sequences, may not be shown in detail in order not to obscure the material disclosed herein.
The material disclosed herein may be implemented in hardware, firmware, software, or any combination thereof. The material disclosed herein may also be implemented as instructions stored on a machine-readable medium, which may be read and executed by one or more processors. A machine-readable medium may include any medium and/or mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a machine-readable medium may include read only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; electrical, optical, acoustical or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), and others.
References in the specification to “one implementation”, “an implementation”, “an example implementation”, etc., indicate that the implementation described may include a particular feature, structure, or characteristic, but every implementation may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same implementation. Further, when a particular feature, structure, or characteristic is described in connection with an implementation, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other implementations whether or not explicitly described herein.
As shown in
System 100 also includes a motion parallax viewing engine 106, a database 108 and a display engine 110, all communicatively coupled to each other directly or via network 104. In various implementations, as will be explained in greater detail below, parallax viewing engine 106 may receive source images 107 via network 104 and may perform various processes on those images to obtain 3D information such as view angles associated with the various images. Parallax viewing engine 106 may store the 3D information associated with the source images 107 in database 108. In various implementations, as will be explained in greater detail below, display engine 110 may receive source images 107 and associated 3D information from the imaging device 102 directly or via network 104 and may undertake various processes to provide images for presentation on a display 112 that depend on a user's viewing angle with respect to display 112.
As shown in
In various implementations, the two imaging devices 202 and 204 may be similar devices. For instance, in some implementations, devices 202 and 204 may be similar high-resolution color cameras. In yet other implementations, devices 202 and 204 may be similar color-depth cameras such as structured light cameras or time-of-flight cameras. In yet further implementations, the two imaging devices 202 and 204 may be dissimilar devices. For instance, in some implementations, device 202 may be a high-resolution color camera while device 204 may be a wide field-of-view camera equipped, for example, with a fisheye lens.
System 200 also includes parallax viewing engine 106, database 108 and display engine 110, all communicatively coupled to network 104 and to each other via network 104. In various implementations, as will be explained in further detail below, parallax viewing engine 106 may receive source images 206 and 208 via network 104 and may perform various processes such as stereo reconstruction on those images to obtain 3D information associated with scene 105. Parallax viewing engine 106 may store the 3D information in database 108. In various implementations, as will be explained in greater detail below, display engine 110 may receive the 3D information via network 104 and may undertake various processes to provide synthesized images of scene 105 that depend on a user's viewing angle with respect to display 112.
While
Process 300 may begin at block 302 where multiple source video images 301 may be received. For example, referring to system 100, block 302 may involve parallax viewing engine 106 receiving source images 107 via network 104. In some implementations, the source images may be received from database 108 at block 302.
View angles of the source images may then be determined at block 304. In various implementations, block 304 may involve parallax viewing engine 106 using known techniques (see, e.g., M. Goesele et al., “Multi-View Stereo for Community Photo Collections,” IEEE 11th International Conference on Computer Vision (2007)) to determine the view angle of each image received at block 302. For example,
Returning to discussion of process 300, the view angles determined at block 304 may be stored as metadata associated with the corresponding source images (block 306). For instance, in various implementations, having determined view angles at block 304, parallax viewing engine 106 may undertake block 306 by storing view angle metadata in database 108 in such a manner that the view angle metadata is associated with the corresponding source images in database 108.
At block 308, a user viewing angle may be determined. In various implementations, block 308 may involve a mechanism associated with a display, such as a front-facing camera and associated logic, determining the angle of a user with respect to the display where the display is to be used to present images of scene 105 to the user. For example,
Returning to discussion of process 300, a best-matched source image having a view angle closest to the user viewing angle may be determined (block 310). In various implementations, block 308 may involve display engine 110 accessing the view angle metadata resulting from block 306 and comparing the corresponding view angles to the user viewing angle determined at block 308 to determine a best-matched source image corresponding to an image view angle closest in value to the user viewing angle. In undertaking block 310, display engine 110 may access view angle metadata stored in database 108.
At block 312, the best-matched source image may be displayed. For example, having determined the best-matched source image at block 310, display engine 110 may present that source image on display 112. In undertaking block 312, display engine 110 may retrieve the corresponding source image from database 108.
At block 314, a determination may be made as to whether the user viewing angle has changed. For example, referring also to
While block 308, as described above, employs a front-facing camera to determine a user viewing angle, the present disclosure is not limited to particular methods for determining a user viewing angle. For instance, other techniques that may be employed to determine user viewing angle include using well known mouse, keyboard, and/or touch screen user control techniques. For instance, a user viewing angle determination may be made as a result of a user's interaction with a touch screen computing system. For example, a user viewing angle may be indicated by a user touching a particular location on a touch screen. Further, the user touching the screen and then sliding their finger in a particular direction and so forth may indicate a change in user viewing angle.
Process 600 may begin at block 602 where at least a pair of source images may be received. For example, referring to system 200, block 602 may involve parallax viewing engine 106 receiving first and second source images 206 and 208 via network 104. In some implementations, the source images may be received from database 108 at block 602.
As noted above in the discussion of
By contrast, in implementations where imaging devices 202 and 204 are dissimilar, source images 206 and 208 may likewise be dissimilar. For instance, in implementations where device 202 is a high-resolution color camera and device 204 is a wide field-of-view camera, source image 206 may be a high-resolution color image while source, image 208 may be a lower resolution, wide field-of-view color image. In such implementations, images 206 and 208 may have similar aspect ratios but may capture different portions or aspects of scene 105. For instance, image 206 may be a high-resolution color image that provides high-resolution visual details in the middle of the field-of-view of scene 105 while fisheye image 208 may provide a lower resolution peripheral view of scene 105.
At block 604, the source images may be analyzed to obtain 3D information of scene 105. In various implementations, where the source images are obtained from similar, calibrated imaging devices, block 604 may include extracting 3D information of scene 105 and estimating camera motion such as rotation and translation between the source images using known stereo reconstruction techniques (see, e.g., Seitz et al., “A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms,” in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (2006)). In various implementations, the 3D information generated at block 604 and associated with the source images received at block 602 may include 3D coordinates of the scene (e.g., for scene feature points in the scene's world coordinate system) as well as camera pose information associated with the two source images.
In undertaking block 604, camera view angles of the two source images 206 and 208 may be used as left-most and right-most reference view angles. In some implementations, where, for example, the source images are obtained from color-depth imaging devices, depth data in the source images may also be employed to aid in the extraction of 3D information from texture-less scenes or in implementations where the baseline between the imaging devices is large enough to preclude reliable stereo reconstruction of the scene. At block 606, the 3D information may be stored as metadata associated with the source images. For example, 3D information may be stored as metadata in database 108 of system 200. In various implementations, blocks 602-606 of process 600 may be undertaken by parallax viewing engine 106.
At block 608, a user viewing angle may be determined. For example, block 608 may be undertaken in a manner similar to that described herein with respect to block 308 of process 300. As noted previously with respect to block 308, a user viewing angle may be determined using a front-facing camera on display 112 or in response to user manipulation of a mouse, keyboard, touch screen or the like.
At block 610, an image may be synthesized based, at least in part, on the 3D information determined at block 604 and the user viewing angle determined at block 608, in various implementations, block 610 may include using known techniques to project the 3D information to generate an image of scene 105 having a perspective corresponding to the user's viewing angle with respect to display 112. The resulting synthesized image may then be displayed at block 612. For example, the synthesized image may be rendered or presented on display 112.
At block 614, a determination may be made as to whether the user viewing angle has changed. For example, referring again to
While the implementation of example processes 300 and 600, as illustrated in
In addition, any one or more of the processes and/or blocks of
System 700 includes a processor 702 having one or more processor cores 704. Processor cores 704 may be any type of processor logic capable at least in part of executing software and/or processing data signals. In various examples, processor cores 704 may include CISC processor cores, RISC microprocessor cores, VLIW microprocessor cores, and/or any number of processor cores implementing any combination of instruction sets, or any other processor devices, such as a digital signal processor or microcontroller.
Processor 702 also includes a decoder 706 drat may be used for decoding instructions received by, e.g., a display processor 708 and/or a graphics processor 710, into control signals and/or microcode entry points. While illustrated in system 700 as components distinct from core(s) 704, those of skill in the art may recognize that one or more of core(s) 704 may implement decoder 706, display processor 708 and/or graphics processor 710. In some implementations, processor 702 may be configured to undertake any of the processes described herein including the example processes described with respect to
Processing core(s) 704, decoder 706, display processor 708 and/or graphics processor 710 may be communicatively and/or operably coupled through a system interconnect 716 with each other and/or with various other system devices, which may include but are not limited to, for example, a memory controller 714, an audio controller 718 and/or peripherals 720. Peripherals 720 may include, for example, a unified serial bus (USB) host port, a Peripheral Component Interconnect (PCI) Express port, a Serial Peripheral Interface (SPI) interface, an expansion bus, and/or other peripherals. While
In some implementations, system 700 may communicate with various I/O devices not shown in
System 700 may further include memory 712. Memory 712 may be one or More discrete memory components such as a dynamic random access memory (DRAM) device, a static random access memory (SRAM) device, flash memory device, or other memory devices. While
The devices and/or systems described herein, such as example systems 100, 200 and/or 700 represent several of many possible device configurations, architectures or systems in accordance with the present disclosure. Numerous variations of systems such as variations of example systems 100, 200 and/or 700 are possible consistent with the present disclosure.
Process 800 may begin at block 802 where multiple 2D images 801 of a scene may be received as described herein. At block 804, 3D information associated with the scene may be determined. For example, referring to processes 300 and 600, block 804 may include undertaking blocks 304 or 604, respectively, as described herein. The 3D information may then be stored as metadata (block 806) as described herein, and, at block 808, a user viewing angle with respect to a display may be determined as also described herein. At block 810, an image may be generated using, at least in part, the 3D information associated with the scene and the user viewing angle. For example, referring to processes 300 and 600, block 810 may include undertaking blocks 310 or 610, respectively, as described herein. At block 811, the generated image may be displayed. Finally, at block 812, a determination may be made as to whether the user viewing angle has changed. If so, blocks 808 and 810 may be repeated; otherwise, process 800 returns to block 811 to continue displaying the current generated image. In this manner, process 800 may provide for a user-steerable 3D perception or viewing experience.
The systems described above, and the processing performed by them as described herein, may be implemented in hardware, firmware, or software, or any combination thereof. In addition, any one or more features disclosed herein may be implemented in hardware, software, firmware, and combinations thereof, including discrete and integrated circuit logic, application specific integrated circuit (ASIC) logic, and microcontrollers, and may be implemented as part of a domain-specific integrated circuit package, or a combination of integrated circuit packages. The term software, as used herein, refers to a computer program product including a computer readable medium having computer program logic stored therein to cause a computer system to perform one or more features and/or combinations of features disclosed herein.
While certain features set forth herein have been described with reference to various implementations, this description is not intended to be construed in a limiting sense. Hence, various modifications of the implementations described herein, as well as other implementations, which are apparent to persons skilled in the art to which the present disclosure pertains are deemed to lie within the spirit and scope of the present disclosure.
Filing Document | Filing Date | Country | Kind | 371c Date |
---|---|---|---|---|
PCT/US11/51197 | 9/12/2011 | WO | 00 | 5/12/2014 |