Patent Application
20160140733
The present disclosure relates generally to multiple view point high speed video applications.
Capturing video of object motion (e.g., a vehicle, person, or other object moving, etc.) can be useful in a variety of applications. Typically, images of the object motion captured at a high speed or frame rate are slowed down during presentation of the images enabling examination of characteristics and features that would likely go undetected otherwise. Having multiple high speed video streams captured from multiple viewpoints is even more helpful in various applications. However, conventional approaches to capturing high speed video are usually expensive and resource intensive.
There are a variety of applications that attempt to use high speed video. In auto industry car crash testing, being able to view a high-speed crash process in slow motion from multiple angles can be very helpful in identifying potential design defects and improve safety. In scientific research fields such as biomechanics, it is often important to clearly observe rapid experimental processes such as high-speed animal movement from multiple viewpoints. In TV broadcasting events (e.g., such as sports, etc.), attempts are made to capture high-speed motion from multiple viewpoints in order to achieve free viewpoint video and try to create an immersive experience. In the movie and computer game industries, attempts at capturing the performance of actors/actresses in high-speed from multiple viewpoints is often utilized for attempts at many further video post-processing tasks such as relighting and matting.
In order to capture high-speed motion from multiple viewpoints, one traditional approach is to use multiple, synchronized high-speed cameras capturing images from different viewpoints. This approach, however, usually has very high costs (e.g., high bandwidth cost, hardware cost, image quality cost, etc.). In terms of bandwidth cost, conventional attempts usually involve extremely large bandwidth transfers and storage of multiple high-speed video streams simultaneously. For example, a single 1000 fps HD video camera records about 6 GBytes of raw data per second, which is over 30 times more than comparable data typically recorded by a regular-speed camera. With respect to hardware cost, a typical high speed video camera (e.g., 1000 fps, etc.) can cost significantly more than the cost of a regular video camera of the same resolution (e.g., 30 fps, etc.). Furthermore, additional expensive hardware is often required for accurate synchronization among multiple high-speed cameras. There are also typically impacts to image quality given that high-speed cameras often have very limited exposure time (e.g., less than 1/1000 second for a 1000 fps camera), and thus usually require much stronger light to ensure high image quality compared to regular-speed video cameras. As a result, many high-speed cameras are only monotonic and cannot capture color images. As a practical matter, these high costs typically associated with high speed cameras often result in significantly limiting the number of captured viewpoints about the target scene.
There are a few traditional approaches that attempt to use an array of regular-speed cameras (e.g., that are typically available at much lower costs than high speed cameras) to emulate high-speed cameras, however these conventional attempts are often problematic. One popular approach attempts to use multiple regular-speed cameras in a stagger fashion. Conventional attempts at using multiple regular-speed cameras in a stagger fashion have various limitations. In such methods, the regular-speed cameras often start recording with a different time offset and the conventional attempts usually require special hardware control support (e.g., involves pixel-wise exposure control that is typically quite expensive itself, etc.) to enable the cameras to start recording with millisecond-level accuracy. Moreover, these methods are usually limited to emulation of a single viewpoint high-speed camera, which often severely limits their usefulness. Some approaches further require that the captured images be deblurred, however image deblurring itself is generally considered an ill-posed open problem and usually very computation intensive.
Systems and methods that facilitate an efficient and effective multi-view hybrid system for high-speed motion analysis are presented. In one embodiment, a device includes a processing circuit that performs multi-view processing and a memory that stores information for the processing circuit. The processing circuit receives input associated with a 3D scene, wherein the input includes information associated with a first frame sequence at a first frame rate corresponding to a first camera and information associated with a second frame sequence at a second frame rate corresponding to a second camera; the first frame rate is faster than the second frame rate. Temporally synchronized frames recorded in the first frame sequence and the second frame sequence at given timestamps are aligned, a three dimensional (3D) model for a dynamic scene at the first frame rate is created, and synthesized image frame sequences associated with a plurality of additional viewpoints are generated at the first frame rate. The first frame sequence and the second frame sequence can be calibrated. A calibration pre-process is conducted to determine intrinsic parameters and extrinsic parameters associated with the first camera and the second camera.
In one exemplary implementation, the 3D model is based upon 3D key frame reconstruction, constrained 3D motion estimation, and 3D key frame interpolation. The 3D key frame reconstruction can include a visual-hull based method to estimate a 3D mesh model of a target object as the 3D model of a target scene. A kinematic-based 3D motion model is used to estimate a set of motion parameters from the 3D key frames. Approximate optimal kinematic-based 3D motion parameters are used to establish an intermediate interpolated 3D frame sequence based on consecutive 3D key frame sequences and the interpolated 3D frame sequence gives a dynamic 3D model at the first rate for a dynamic scene from a virtual camera viewpoint at the first rate. Calibrated frame sequences captured from multiple viewpoints can be used to enhance video quality including denoising and correlation among multiple views. Spatial-temporal coherence within and among multiple video streams can be used to increase signal-to-noise ratio.
A multi-view method can include: calibrating a first video frame sequence at a first frame rate from a first camera and a second video frame sequence at a second frame rate from a second camera, wherein the first frame rate is faster than the second frame rate; creating a three dimension (3D) model based upon 3D key frame reconstruction, constrained 3D motion estimation, and 3D key frame interpolation; and generating a plurality of synthesized image frame sequences at the first frame rate, wherein the plurality of synthesized image frame sequences are associated with a plurality of viewpoints different from a viewpoint associated with the first camera. The calibrating can use a plurality of intrinsic parameters and a plurality of extrinsic parameters to produce a calibrated first frame sequence and a calibrated second frame sequence. The calibrating can also include at least one of the following: correcting lens distortion for the first video frame sequence and the second video frame sequence based upon the plurality of intrinsic parameters; and enabling geometric calibration by warping the first video frame sequence and the second video frame sequence to align with a reference sequence based upon the plurality of intrinsic parameters and the plurality of extrinsic parameters. The plurality of intrinsic parameters and the plurality of extrinsic parameters include at least one of the following: an intrinsic camera parameter for the first camera and an intrinsic camera parameter for the second camera; lens distortion coefficients for the first camera and the second camera; and an extrinsic 6 degree-of-freedom position and pose for the first camera and the second camera.
In one exemplary implementation, the 3D key frame reconstruction further includes: reconstructing a sequence of 3D key frames based upon a calibrated second video frame sequence; ascertaining a kinematic motion parameter based upon the sequence of 3D key frames, an intrinsic parameter, and an extrinsic parameter; and establishing a sequence of interpolated 3D frames for a virtual camera viewpoint at a first frame rate based upon the sequence of 3D key frames and the kinematic motion parameter. Generating a plurality of synthesized image frame sequences further includes: creating a textured 3D model sequence by texture mapping a calibrated regular frame rate image sequence to a high frame rate dynamic 3D model; and constructing a synthesized image frame sequence for a virtual high-speed camera viewpoint based upon the textured 3D model sequence. Generating a synthesized image frame sequence can include utilizing spatial-temporal coherence in multi-view video denoising.
In one embodiment, a system includes: a first camera that captures images at a first speed; a second camera that captures images at a second speed, the second speed is slower than the first speed; and a multi-view system for simulating a plurality of viewpoints at the first speed in addition to a viewpoint of the first camera. The image results of the first camera capture and the second camera capture are utilized to sample respective temporal appearance discrepancies. Spatial-temporal coherence in the sampled image results are utilized to synthesis video streams at the first speed from a viewpoint different than the viewpoint of the first camera. The first camera can be one of a first plurality of cameras that capture images at the first rate and the second camera can be one of a second plurality of cameras that capture images at the second rate, wherein the number of cameras included in the first plurality of cameras is less than the number of cameras included in the second plurality of cameras. In an alternative embodiment, the number of cameras included in the first plurality of cameras is the same as or more than the number of cameras included in the second plurality of cameras. The first frame rate is in or above the range of hundreds of frames per second the second rate is in the range of tens of frames per second. Positions and poses of the first camera and second camera can change with respect to a scene but remain fixed relative to one another during a scene capture. The positions and poses of the first camera and second camera can also change with respect to one another after a scene capture and before a second scene capture
The accompanying drawings, which are incorporated in and form a part of this specification, are included for exemplary illustration of the principles of the present invention and not intended to limit the present invention to the particular implementations illustrated therein. The drawings are not to scale unless otherwise specifically indicated.
Reference will now be made in detail to the various embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. While described in conjunction with these embodiments, it will be understood that they are not intended to limit the disclosure to these embodiments. On the contrary, the disclosure is intended to cover alternatives, modifications and equivalents, which may be included within the spirit and scope of the disclosure as defined by the appended claims. Furthermore, in the following detailed description of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be understood that the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present disclosure.
Systems and methods that facilitate an efficient and effective multi-view hybrid system for high-speed motion analysis are presented. The multi-view hybrid system can include a mix of relatively few high-speed cameras (e.g., 1000 fps, etc.) and a greater number of regular-speed cameras (e.g., 30 fps, etc.) that are utilized to provide more high speed views than high speed cameras included in the system. The new multi-view hybrid approach exploits spatial-temporal coherence in sampled images from both the regular-speed cameras and high-speed cameras to increase the number of high speed motion viewpoints, unlike limited previous attempts (e.g., that were directed to static images, that did not exploit the spatial-temporal coherence in multi-view video denoising, etc.). In one embodiment, the high-speed motion captured from one or a few high-speed cameras is “transferred” to other viewpoints to create a set of virtual high-speed camera viewpoints that provide the ability to obtain high-speed motion information from multiple corresponding viewpoints.
In one embodiment, received high speed video streams and regular speed video streams are calibrated. The calibrated regular speed video streams associated with the regular speed cameras are used to reconstruct 3D key frames at the regular frame rate. These reconstructed 3D key frames, together with the 2D high frame rate images captured by the high-speed cameras, are used to estimate optimal kinematic model parameters that control the motion between two neighboring 3D key frames. The kinematic model parameters are utilized to interpolate intermediate 3D frames at a high frame rate and the interpolated 3D frames are then used to synthesize 2D images in synthesized image frame sequences from other viewpoints at the high frame rate. The multiple image sequences captured from multiple viewpoints can be processed together to enhance the image quality. The processed images can be texture mapped to the above reconstructed 3D models for synthesizing 2D images at virtual high-speed camera viewpoints. Thus, the virtual high-speed camera viewpoints results are realized in a manner that facilitates efficient bandwidth use and low hardware costs by utilizing a single or few high-speed cameras jointly with multiple regular-speed cameras in the hybrid system.
In block 110, a video camera process is performed. The video camera process includes capturing a scene by multiple cameras and generating a plurality of video frame sequences associated with a plurality of view points corresponding to the multiple cameras respectively. In one embodiment, the plurality of video frame sequences include a first video frame sequence at a first speed or frame rate and a second video frame sequence at a second speed or frame rate that is slower than the first speed or frame rate. The first speed or frame rate is faster than the second speed or frame rate. The first video frame sequence and the second video frame sequence can include images of a dynamic scene (e.g., a person running, a car test crash, etc.) captured at approximately the same time.
It is appreciated that new multi-view hybrid approaches are compatible with a variety of implementations. The first frame rate can be a relatively high-speed frame rate and the second frame rate can be a regular frame rate. The second camera can be one of a plurality of relatively low cost regular-speed cameras placed relatively densely (e.g., to measure spatial appearance discrepancy over multiple viewpoints, etc.) while the first camera is a high-speed cameras (e.g., used to sample temporal appearance discrepancy at one viewpoint, etc.).
In block 120, a multiple viewpoint video method or process is performed. The multiple viewpoint or multi-view process generates frame sequence information (e.g., a video stream or frame sequence, etc.) from additional viewpoints without physical cameras at the viewpoints. The first video frame sequences and the second video frame sequence are temporally synchronized. For a given timestamp, respective frames can be located in the first video sequence at the timestamp and in the second video sequence at the timestamp. In one embodiment, information associated with a first video frame sequence captured by a first camera at a first rate or speed is utilized to sample temporal appearance discrepancy while information associated with a second video frame sequence captured by a second camera at a second rate or speed is utilized to measure spatial appearance discrepancy. The spatial-temporal coherence in these sampled images is then utilized to create or generate a synthesized video sequence at the first frame rate or speed. The synthesized video sequence can be from the perspective of a viewpoint different than viewpoints of the first camera.
It is appreciated that the new multi-view hybrid imaging approach can be implemented at a variety of frame rates and configurations. In one embodiment, one frame rate is relatively faster than another. For ease of explanation, the following description is presented in terms of high speed cameras with high speed frame rates and regular speed cameras with regular speed frame rates.
It is appreciated that new multi-view approaches are compatible with a variety of configurations and implementations. In one embodiment, a target dynamic scene is captured through a plurality of M regular-speed cameras and N high-speed cameras where N>=1. The number of M regular-speed cameras can be larger than the number of N high-speed cameras. A plurality of M frame sequences at a regular frame rate and N frame sequences at a high frame rate are generated accordingly. The regular rate frame sequences come from regular-speed cameras recording the appearance of the target scene over time from a particular view at a regular frame rate (e.g. 24 fps, 30 fps, etc.). The high rate frame sequences come from high-speed cameras recording the appearance of the target scene over time from a particular view at a high frame rate (e.g., 500 fps, 1000 fps, etc.).
The plurality of M regular-speed cameras and N high-speed cameras are placed together into a hybrid imaging system (e.g., similar to system 200, etc.). In one embodiment, a regular-speed camera Li can be placed in the 3D real world arbitrarily, as long as there exists another regular-speed camera Li that has a large enough overlapping field of view (determined by a threshold Tr) with Li. A high-speed camera Hi can be placed in the 3D real world arbitrarily, as long as there exists another regular-speed camera Lj that has a large enough overlapping field of view (determined by a threshold Th) with Hi. The high-speed cameras can be placed relatively sparsely in the hybrid imaging system, while the regular-speed cameras can be placed more densely.
With respect to moving the positions of the regular-speed cameras and high-speed cameras, for a single dynamic scene, the relative positions and poses of the regular-speed cameras and high-speed cameras within the hybrid imaging system remain fixed during each recording. However, the position and pose of the hybrid imaging system itself can change during a recording (as long as the relative positions and poses remain substantially constant). In one exemplary implementation, the entire hybrid imaging system can be mounted on a moving platform for recording the scene and the platform can move but the positions of the cameras on the platform remain the same relative to one another. For capturing different dynamic scenes, the relative positions and poses of the regular-speed cameras and high-speed cameras can be reconfigured between scenes.
In block 310, an alignment process is performed. The alignment process can include a calibration operation on a camera or the results of the capture process (e.g., results similar to results of block 110). The alignment process may include multiple views and the calibration is performed on multiple cameras. A plurality of intrinsic and extrinsic parameters can be used to align a set of temporally synchronized image frames. The image frames can be recorded in at a first frame rate (e.g., a high speed frame rate, etc.) and a second frame rate (e.g., a regular frame rate, etc.) and synchronized at a given timestamp.
In block 320, a three dimension (3D) model process is performed. The 3D model process includes reconstructing a 3D model associated with a scene captured in a video capture process (e.g., similar to block 110, etc.). The 3D model process can include: a three dimension 3D key frame reconstruction, a constrained 3D motion estimation, and a 3D key frame interpolation. Additional explanation of 3D model processes is set forth in later portions of the detailed description.
In block 330, a synthesized image frame sequence process is performed. In one embodiment, multiple frame sequences are created or synthesized. The multiple synthesized image frame sequences can be synthesized at frames rates (e.g., a high speed frame rate, etc.) faster than the second frame rate (e.g., a regular frame rate, etc.). Additional explanation of synthesized image frame sequence processes is set forth in later portions of the detailed description.
In block 410, lens distortion is corrected for the first frame sequences and the second frame sequences.
In block 420, geometric correction is performed by geometrically aligning a frame in a first (e.g., high, faster, etc.) frame sequence and a frame in a second (e.g., regular, slower, etc.) frame sequence using extrinsic parameters and intrinsic parameters. There are a variety of ways to obtain the intrinsic and extrinsic parameters. In one embodiment, a calibration pre-process is conducted to compute the intrinsic and extrinsic parameters of regular-speed cameras and high-speed cameras. A calibration pre-process can be adopted to compute the intrinsic camera parameters, the lens distortion coefficients, and the extrinsic 6 degree-of-freedom position and pose for the regular-speed cameras and high-speed cameras. In other embodiments, some of the intrinsic parameters can be obtained through the camera manufacturer, and the remaining intrinsic and extrinsic parameters can be computed through the calibration pre-process.
In block 430, additional optional operations of geometric correction and radiometric correction can also be performed. For example, operations such as radiometrically correcting the appearance of the regular frame rate and high frame rate image frames can be performed.
In block 510, 3D key frame reconstruction is performed. Frames from a calibrated regular frame rate image sequence are used to reconstruct a 3D model sequence at the same regular frame rate of the regular-speed cameras (e.g., using standard multi-view stereo (MVS) methods, etc.). The 3D model sequence includes a sequence of 3D key frames. A calibrated regular frame rate image frame captured by the i-th regular-speed camera at timestamp t, where i=1, . . . , M, is denoted by IiL(t). A reconstructed 3D key frame at timestamp t is denoted by S({I1L(t), . . . , IML(t)}). The reconstructed 3D key frame operates as a 3D model S({I1L(t), . . . , IML(t)}) of a target scene reconstructed using the calibrated regular frame rate image frames I1L(t), . . . , IML(t). In one embodiment, a visual-hull based method described can be used to estimate a 3D mesh model of the target objects as the 3D model of the target scene, provided its high accuracy and near real-time performance. Other MVS approaches can also be used (e.g., L1 and L2 in
In block 520, constrained 3D motion estimation is performed. A kinematic-based 3D motion model can be used to estimate a set of motion parameters from the 3D key frames. The use of the kinematic-based 3D motion model is based on the assumption of continuous and smooth motion between consecutive key frames (which is typically reasonable because of the characteristics of real-world object motion). The motion model can be either parametric or non-parametric. The reconstructed 3D key frame at timestamp t is S({I1L(t), . . . , IML(t)}) where: IiL(t) denotes a calibrated regular frame rate image frame captured by the i-th regular-speed camera at timestamp t; and IiH(t) denotes a calibrated high frame rate image frame captured by the i-th high-speed camera at timestamp t, and i=1, . . . , N. In one exemplary implementation θ denotes the set of kinematic-based 3D motion parameters. Given θ and the intrinsic and extrinsic parameters of the high-speed cameras from the alignment process, a synthesized image frame sequence at a high frame rate at the timestamp t of the i-th high-speed camera can be generated or computed as Pi(S({I1L(t), . . . , IML(t)}), θ), where Pi is the projection operation from a 3D model to a 2D image. An optimal kinematic motion parameter θ′ can be approximated by minimizing the image discrepancy between the captured high frame rate image frames and the synthesized image frame sequence at a high frame rate. The minimization can be estimated by the following cost function:
where ƒ(I1, I2) is a function that measures the image discrepancy between image I1 and image I2. Examples of the image discrepancy measurement function include sum of squared differences (SSD), sum of absolute differences (SAD), image structural similarity (SSIM), and so on. The above image discrepancy is summed for the N high-speed cameras.
In block 530, 3D key frame interpolation is performed. For a given virtual high-speed camera view, the approximate optimal kinematic-based 3D motion parameters θ* are used to establish or compute an intermediate interpolated 3D frame sequence based on the consecutive 3D key frame sequences. This procedure is analogous to key framing and interpolation in 2D images in animation. The interpolated 3D frame sequence gives the high frame rate dynamic 3D model for the captured dynamic scene from the virtual high-speed camera viewpoint. After that, the interpolated 3D frame sequence is used as the input to the view synthesis, which generates synthesized image frame sequences from the virtual high-speed camera viewpoints.
In block 610, a multi-view video enhancement process is performed. The calibrated image frames captured from multiple viewpoints are used to enhance the video quality (e.g., especially for video denoising, etc.). This step is usually used for high-speed imaging because of the short exposure time for a frame. Adoption of various approaches can be utilized and extended for multi-view video enhancement. Specifically, the correlation among multiple views and the spatial-temporal coherence within and among multiple video streams are used to increase the signal-to-noise ratio.
It is appreciated that the new multi-view approaches are readily adaptable to various implementations. The new multi-view approaches can be modified in accordance with various trade-off decisions. In an alternative embodiment (e.g., similar to the workflow described in
In block 620, a 3D model texturing process is performed. The image frames of the enhanced video are geometrically mapped to the surface of the interpolated 3D frames of the target object to generate a textured 3D model sequence by using standard computer graphics techniques. In one exemplary implementation in which the multi-view video enhancement step is bypassed, the calibrated image frames captured from multiple viewpoints are directly used in the model texturing step to generate the textured 3D model sequence.
In block 630, a virtual high-speed camera view synthesized image frame sequence process is performed. For a given virtual high-speed camera viewpoint, a corresponding synthesized image sequence is generated based upon the textured 3D model sequence by using standard computer graphics methods.
It is appreciated that the reconstructed 3D model sequence with textures described herein can be used in many applications (e.g., video relighting, high-resolution video mosaic, virtual object insertion, free-view video, 3D tele-presence/avatar, e-commerce, etc.) The new hybrid approach facilitates cost-effective imaging for multi-view high-speed motion analysis. By jointly using regular-speed cameras (e.g., which have relatively low costs and can be placed relatively densely to measure spatial appearance discrepancy over multiple viewpoints, etc.) and one or a few high-speed cameras (e.g., used to sample temporal appearance discrepancy at one or a few viewpoints, etc.). The new approach is able to overcome various issues and limitations (e.g., bandwidth cost, hardware cost, image quality cost, limited number of high speed viewpoints, etc.) often associated with conventional multi-view high-speed motion capture attempts (e.g., expensive special hardware to start cameras at high-precision time offset, only handle a single viewpoint, require densely placed high-speed cameras, etc.). By jointly using multiple regular-speed cameras and one or a few high-speed cameras, the new multi-view approach is able to reconstruct the 3D key frames and estimate the motion of dynamics scenes. The 3D key frames and motion are used to efficiently generate intermediate interpolated 3D frames which enable the effective synthesis of image frame sequences at a high frame rate from a set of virtual viewpoints. Multi-view video enhancement with multiple regular frame and high frame rate image sequences. By jointly using multiple regular frame rate and one or a few high frame rate image sequences, the new approach is able to enhance the signal-to-noise ratio for each image sequence, which can be especially useful for high-speed imaging because of the limited exposure time.
Embodiments described herein may be discussed in the general context of computer-executable instructions, such as program modules, residing on some form of computer-readable storage medium executed by one or more computers or other devices. By way of example, and not limitation, computer-readable storage media may comprise non-transitory computer-readable storage media. Non-transitory computer-readable storage media includes all computer-readable media except for a transitory, propagating signal. Computer-readable storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules or other data. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or distributed as desired in various embodiments.
Although certain preferred embodiments and methods have been disclosed herein, it will be apparent from the foregoing disclosure to those skilled in the art that variations and modifications of such embodiments and methods may be made without departing from the spirit and scope of the invention. It is intended that the invention shall be limited only to the extent required by the appended claims and the rules and principles of applicable law.
