Patent Application
20140132736
The present disclosure relates to technology for capturing perspective images for use in three-dimensional image display and multi-view two-dimensional image display.
Recent developments in stereo display technologies can enable viewers to view objects in three-dimensions or multi-view in two-dimensions. An array of cameras can be used to capture multiple perspective views of a scene to be later displayed, for example, by projection onto a screen. The dimensional size of cameras can limit the number of cameras that can be packed in such an array. An image capturing system is disclosed that facilitates use of a reduced number of cameras for capturing images of a scene for three-dimensional image display and/or multi-view two-dimensional image display.
In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present systems and methods. It will be apparent, however, to one skilled in the art that the present systems and methods may be practiced without these specific details. Reference in the specification to “an embodiment,” “an example” or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least that one example, but not necessarily in other examples. The various instances of the phrase “in one embodiment” or similar phrases in various places in the specification are not necessarily all referring to the same embodiment.
As used herein, the term “includes” means includes but not limited to, the term “including” means including but not limited to. The term “based on” means based at least in part on.
Image capture systems provided herein can be used to capture different perspective views of objects in scenes. These captured images can be displayed, for example being projected using projection display systems, to provide a three-dimensional image display and/or multi-view two-dimensional image display. Multiple image capture devices, each placed at a different orientation and/or position relative to the objects in a scene, facilitate the capture of multiple views of the scene. Increasing the number of image capture devices for capturing the multiple images of that scene can facilitate three-dimensional image viewing when these multiple images are displayed, for example by projection at a screen. For example, using these multiple captured images, a viewer can view stationary and/or moving three-dimensional imagery or multi-view two-dimensional imagery with correct perspective if the projection of the multiple captured images is properly coordinated and synchronized. Enhancement of the captured image quality can be obtained by reducing the spacing between the image capture devices used to capture the multiple images. For example, the quality of a continuous 3D imagery can be enhanced if the spacing between image capture devices used to capture the multiple images is about one (1) per centimeter. A spacing and packing of one (1) image capture device per centimeter may be obtained if small image capture device are used. However, small image capture devices can be inferior in image capture quality. The reduction of the spacing of image capture device also may require an increase in the number of image capture devices used, which can be costly and impractical. Also, the variability in reliability of the increased number of image capture devices can affect the overall performance of the image capture system.
Described herein are systems and methods that can be used to capture successive views of objects in a scene using a reduced number of image capture devices. The scene can be a static scene or a moving scene. At least two light-deflecting devices are associated with each image capture device. The at least two light-deflecting devices are positioned between the respective image capture device and the objects in the scene. At least one of the at least two light-deflecting devices is moved so that the at least two light-deflecting devices are oriented at different orientations. In combination with the at least two light-deflecting devices in the different orientations, a single image capture device can be used to capture two or more perspective views of objects in a scene at angles and in positions that replicate image capture capability of additional image capture devices. Thus, the systems and methods disclosed herein facilitate image capture device replication by using light-deflecting devices to reduce the number of image capture devices used in an image capture array. The image capture devices in combination with the at least two light-deflecting devices can be used to capture images of different perspective views of objects in a scene with sufficient image quality for display, such as at a screen using three-dimensional and/or two-dimensional multiview image projection systems. Non-limiting examples of screens include continuous corridors, a wall, the screens of movie theaters, etc. In an example, the length of the screen can be extended in the horizontal direction and made conformal to the contour of a real wall or some other surface with features such as twist and turns.
Examples of the light-deflecting devices that are applicable to any of the examples described herein, and according to the principles described herein, include mirrors, micromirrors, and any other device that can be operated as described herein to deflect the path of light rays for capturing successive views of objects in a scene.
Various examples of the present disclosure are directed to image capture systems that include at least one an image capture device and at least two light-deflecting devices associated with each of the an image capture devices. The at least two light-deflecting devices are positioned between the respective image capture device and the scene. The at least two light-deflecting devices are oriented in at least two different orientations to re-direct the path of light rays from the objects in the scene to the respective image capture device such that the image capture device captures at least two different perspective views of objects in a scene when the light-deflecting devices are oriented in the at least two different relative orientations. In this arrangement, each of the image capture devices are “replicated” many times (e.g., 1-100 times) through the use of the light-deflecting device mechanisms described herein to scan the light rays from the objects in the scene across the image capture devices. At least one actuation system is operably connected to at least one of the light-deflecting devices to cause the motion and rotation of the respective light-deflecting device to change its orientation according to the principles described herein. Examples of the actuation system include a motor or other type of actuator. Another example of an actuation system is an electromechanical servo system.
In an example, the at least two light-deflecting devices 108 can be positioned within the same housing as the associated image capture device 102. In an example, the at least two light-deflecting devices 108 can be positioned external to the housing of the associated image capture device 102. Examples of image capture device 102 include any device that captures an image by gathering light through its aperture, including a digital camera, a video camera, video recorder, a still image capture device, just to name a few. The image capture device can be a multiple-lens camera. The image processing system 104 can include a computer-readable medium and one or more processors for storing, processing, transmitting image data, and controlling the image capture device 102. The digital processing system 106 is a computing device that includes machine readable instructions, including firmware or software, that coordinate the operation of the image capture device 102 and its at least two associated light-deflecting devices 108 to capture the different perspective images, as described herein in various examples. In an example where more than one image capture device 102 is used, each with at least two associated light-deflecting devices 108, digital processing system 106 includes machine readable instructions, including firmware or software, that can be used to coordinate the operation of the image capture devices 102 to capture the different perspective images, as described herein in various examples.
In the example illustration of
In the example of
In another non-limiting example, the light-deflecting devices can be oriented at different angles to capture nine (9) different perspective views of the object: ic−4φ, ic−3φ, ic−2φ, ic−φ, ic, ic+φ, ic+2φ, ic+3φ, ic+4φ. In this example, an image capture device and associated light-deflecting devices is used to provide the capabilities of an array of nine image capture devices. At each different position and orientation of the light-deflecting devices, the image capture device captures a different perspective view of objects in a scene as if it is a different image capture device in an array of image capture device. In a general example, the light-deflecting devices can be oriented at different angles to capture a number (n) different perspective views of the objects in a scene: ic±nφ (where n=0,1,2, . . . ).
In other examples similar to
In an example, the image capture devices can be configured to synchronize with the orientation of the light-deflecting devices so that a different perspective view of objects in a scene can be captured at each of the different positions: ic±nφ (where n=0,1,2, . . . ). Furthermore, the different perspective view of objects in a scene can be captured during a time interval that is shorter than the resolution of the human eye. For example, the different perspective images can all be captured in about 1/1000th of a second (an effective rate of 1000 frames per second). A frame includes several different perspective views of the objects in the scene. Each different perspective is captured in a time interval of 1/N of the number of image capture devices (N) that a single physical image capture device is emulating. In the example configuration shown in
The image capture devices are configured to capture the different perspective images, and the rotational positioning of the associated light-deflecting devices are coordinated and synchronized to re-direct the light rays from the different portions of the objects, so that, when displayed, such as by being projected on a screen, a viewer sees stationary or moving three-dimensional imagery with correct perspective on the screen. In the example configuration shown in
Several image capture devices, each coupled with its associated light-deflecting devices, can be used to replace an entire array of image capture devices. A set of different perspective views are captured at each of the image capture devices in a time synchronized manner that mimics the operation of the eliminated neighboring image capture devices.
The different perspective images captured by an image capture device, and the orientation of the associated light-deflecting devices, can be synchronized among the different image capture devices so that the different perspective views are captured in a time-multiplexed manner. An example of multiplexed operation of image capture devices and associated light-deflecting devices is described in connection with an example system where each image capture devices is used with associated light-deflecting devices to capture perspective views from five (5) different portions of objects in a scene. For example, referring to
In this example multiplexed timing sequence, at time slot T1, image capture device I4 captures a perspective view i2, image capture device I9 captures a perspective view i7, and image capture device I14 captures a perspective view i12; at time slot T2, image capture device I4 captures a perspective view i3, image capture device I9 captures a perspective view i8, and image capture device I14 captures a perspective view i13; and so forth. This example sequence can be repeated in order with each repeated image capture sequence (1,2,3,4,5), or the sequence can be inverted (5,4,3,2,1). The capture sequence could also be a combination of the forward and inverted sequences. In other examples, other multiplexed image capture and timing sequence are applicable that can be used to capture different perspective views for later display, including by projection, as stationary or moving three-dimensional imagery or multi-view two-dimensional imagery with correct perspective on the screen. As described above, a frame rate of about 100 frames per second or less can be used. In another example, a frame rate of about 30 frames per second can be used. In this example, the physical image capture devices and associated light-deflecting devices operate at a frame rate five (5) times faster since each physical image capture device emulates five (5) image capture devices.
The movements of the light-deflecting devices can be time synchronized and the magnitude of their deflection and orientation can be coordinated to capture successive views of objects in the scene.
The operation of the image capture device and associated light-deflecting devices described in connection with
As illustrated in relation with the example of
In accordance with the principles of
In an example, the image capture devices of
Several image capture devices, each coupled with its associated light-deflecting devices, can be used according to the principles of
The operation of the image capture devices and associated light-deflecting devices according to the principles of
The light-deflecting devices according to the principles described herein are oriented at different angles relative to the plane of their respective image capture device, and in some examples, oriented at different angles relative to each other, to facilitate capture of the different translated perspective views (illustrated in the non-limiting examples of
According to the principles described herein, several image capture devices, each coupled with its associated light-deflecting devices, can be used to replace an entire array of image capture devices. The number of image capture device in the array is reduced by a factor N, where N represents the number of image capture devices that each physical image capture device and associated light-deflecting devices is emulating. The image capture devices are configured to capture the different perspective views of the objects in a scene, and the angle of deflection of the light-deflecting devices are coordinated and synchronized to re-direct the light rays to capture the different perspective views of the objects at the different angular orientations. Each of the image capture devices and respective associated light-deflecting devices are positioned relative to the objects in the scene, and separated from each other, so that the multiple different perspective views captured by each image capture devices can be brought together and displayed to a viewer as a stationary or moving three-dimensional imagery, or multiview two-dimensional imagery, with correct perspective on a screen.
Example projection display systems disclosed in International Application No. PCT/US2010/055004, filed Nov. 1, 2010 include at least two light-deflecting devices associated with each projector. In combination with at least two light-deflecting devices in the different orientations, a single projector is used to project two or more perspective views of images at a screen at angles and in positions that replicate the projections from multiple projectors. The image display systems of International Application No. PCT/US2010/055004 can be used with screens of different shapes, including flat, spherical, and a paraboloid screen. Example screens include continuous corridors, a wall, the screens of movie theaters, etc.
In the example of
The images captured according to the principles described herein also can be displayed using a traditional stereoscopic or binocular three-dimensional display.
Using the image capture systems and principles described herein, display systems can be used to allow viewers to experience three-dimensional without having to wear glasses or goggles or multiview two-dimensional imagery. Viewers can see three-dimensional and two-dimensional imagery with correct perspective views. In an example, when the spacing between the perspective views is larger than the spacing between a viewer's eyes, the viewer is presented with multiple two-dimensional perspective views separated by three-dimensional perspective views.
Reference is made to
Different perspective views of objects in a scene, captured by an image capture array that includes at least two of the image capture systems described herein, can be projected onto a screen such that a viewer looking at the screen from a viewing zone receives a first perspective view in the viewer's left eye and a second perspective view in the viewer's right eye.
The different perspective views captures by the image capture devices and associated light-deflecting devices according to the principles described herein can be projected using front or rear projection environments.
The operation of the image capture devices and respective associated light-deflecting devices described herein provide unique arrangement of image capture devices that can facilitate capture of successive views for use in continuous three-dimensional displays and multi-view two-dimensional display. In each of the arrangements described herein, each one of the image capture devices can be replicated many times, up to 100 times or more, through the use of a set of synchronized moving light-deflecting devices. The replication is accomplished with the unique arrangements of associated light-deflecting devices as described. The movements of the light-deflecting devices are time-synchronized and magnitude-coordinated to capture successive views of objects in a scene. This creates a flexible and versatile image capture environment that is customizable to various applications and is efficient in both hardware and software resources. Non-limiting examples of applications to which the differing perspective images of objects captured herein are applicable are immersive three-dimensional display for teleconferencing and personal gaming, scientific and industrial visual representations and trainings, and entertainment.
Although examples are described herein relative to image capture devices and respective associated light-deflecting devices arranged in a row or a curve, in other examples, the principles describe herein are applicable also to stacked arrangements of the image capture devices and associated light-deflecting devices. Furthermore, the principles describe herein are applicable also to two-dimensional arrangements of the image capture devices and associated light-deflecting devices on a plane, to three-dimensional arrangements of the image capture devices and associated light-deflecting devices (two-dimensional arrangements in several stacked planes), or to any other geometrical arrangement of the image capture devices and associated light-deflecting devices.
The image capture devices and associated light-deflecting devices according to the principles described herein provide several advantages over an array of physical image capture devices. As previously described, the number of image capture devices used can be reduced. Also, the physical spacing between the image capture devices is increased, which allows the use of higher resolution, more sophisticated image capture devices. Such higher resolution image capture devices can be bulkier than the mini-sized image capture devices or pico-sized image capture devices that would be used in view of the spacing restrictions in an array. Since there are fewer image capture devices, then fewer data streams are used to transmit signals to the fewer image capture devices and they are easier to synchronize. For an example system architecture, it may be easier to manage one data stream instead of, for example, ten data streams. There may be an increase in the data rate per image capture device. An image capture system according to the principles described herein may exhibit greatly improved reliability over other image capture systems.
In an example, the light-deflecting devices could be stepped at discrete locations to minimize motion blur during capture. Alternatively, the light-deflecting devices could be smoothly rotating at a known or constant velocity. In this example, the captured images may have the appearance of motion blur with a known motion. With the known light-deflecting device motion, deconvolution and deblurring techniques can be applied to improve the quality of the final captured images. These techniques also may be applied to interpolate and sharpen individual images for perspectives views captured. Noise reduction techniques may also be applied. Constrained and redundant representations, including epipolar-plane images, can be leveraged, thereby simplifying three-dimensional modeling.
For time varying scenes, the captured perspective views at each time instance also may be reconstructed by interpolating across the deblurred images from the image capture device operating as a virtual device, as well as synchronizing the other image capture devices in the array that are operating as virtual devices.
The systems and principles described herein are also applicable to mobile devices that include image capture devices.
In certain examples, the captured perspective views can be displayed, including being projected onto a screen, in separate but approximately equal time slots using time-division multiplexing, as described above. In other examples, the images can be stereo image pairs, each image pair representing a different three-dimensional perspective view of the objects or the scene, as described above.
The method can include dynamically orienting the light-deflecting devices to re-direct the path of light rays from the objects in the scene to provide a perspective view with a translational shift at the image capture device, as described herein. At least one actuation system operably connected to at least one of the light-deflecting devices can be used to orient the light-deflecting device relative to the path of the light rays to produce the translational shift. In another example, the method can include dynamically orienting the light-deflecting devices to re-direct the path of light rays from the objects to the image capture device to capture different perspective views of differing angles. At least one actuation system operably connected to at least one of the light-deflecting devices can be used to orient the light-deflecting device relative to the path of the light rays to produce the differing angles.
The different perspective views captured according to the principles described herein can be displayed, including being projected onto a screen, such that a viewer looking at the screen from a viewing zone receives a first perspective view in the viewer's left eye and a second perspective view in the viewer's right eye. The first perspective view in the viewer's left eye and the second perspective view in the viewer's right eye can form a stereo image pair, providing the viewer with a three-dimensional, perspective view image of a displayed scene (such as one projected onto the screen). The first perspective view in the viewer's left eye and the second perspective view in the viewer's right eye can form a two-dimensional, perspective view image of a displayed scene (including one projected onto the screen).
The foregoing description, for purposes of explanation, used specific nomenclature to provide a thorough understanding of the disclosure. However, it will be apparent to one skilled in the art that the specific details are not required in order to practice the systems and method disclosed herein. The foregoing descriptions of specific examples are presented for purposes of illustration and description. They are not intended to be exhaustive of or to limit to the precise forms disclosed. Obviously, many modifications and variations are possible in view of the above teachings. The examples are shown and described in order to best explain the principles of the disclosure and its practical applications, to thereby enable others skilled in the art to best utilize the disclosure and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the scope of the disclosure be defined by the following claims and their equivalents.
| Number | Date | Country | Kind |
|---|---|---|---|
| PCT/US2010/055004 | Nov 2010 | US | national |
This patent application claims priority to International Application No. PCT/US2010/055004, filed Nov. 1, 2010, the disclosure of which is incorporated by reference in its entirety for the disclosed subject matter as though fully set forth herein.
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/US11/30485 | 3/30/2011 | WO | 00 | 9/30/2013 |
