Patent Application
20180020167
Chromakey systems operate by obtaining a video of objects and/or persons, referred to as the subject, in front of a chromakey color background. That chromakey color background is changed to some other simulated background. The viewer of the video sees the subject overlaid over the simulated background. The chromakey subject is thus shown on a different background, as desired, by changing the chromakey background into some desired image or video using the image processing equipment. The people and objects can then be formed as virtual settings in the resulting video. Chromakey operations have limitations. The object cannot be formed of certain colors which might interfere with the chromakey operation. For example, when a green screen is used as the chromakey background color, the subject should not have any green in it, and cannot have certain other colors in it that would interfere with the green. If the subject had green, then the viewer would see the background in place of the parts of the subject that were green.
The present application describes a multiple image capture video system.
Embodiments as described herein describe the camera that obtains a video that is separated by an image processor. Adjacent in time frames of the video have different backgrounds. In one embodiment, there are two backgrounds, one for chromakey, and one for an actual background, e.g. a background formed of an image. The video was taken twice as fast as desired, so that there are two backgrounds for each real desired frame of the video. Then, the image processing system separates the two frames, so that a first video is obtained that has a chromakey background, and a second video is obtained that has an actual background. Both the videos have the same objects in them, and are in essence the same video, but with different backgrounds.
In the Drawings:
Digital technology enables an LED video display screen or other digital projection source to be used to actually display the background, as an alternative to the chromakey screen that is used behind the subject of a chromakey video. The LED screen is used to display the actual background which is desired to be seen behind the object, where the actual background is typically an image or video (series of images) displayed behind the objects. An advantage of this system is that the background emits illumination onto the subject, which makes the subject's lighting look more natural. Also, there would be no limitations on the color(s) that could be included in the subject, as there typically is with chromakey systems, which typically prevent the subject from wearing green or blue clothing or items.
However, the inventors also recognize that the chromakey system includes certain advantages. Among those advantages is that when a chromakey system is used, the video creator can freely make changes to the background at any later time. Also, there can be pixelation and other effects from displaying the background on an LED screen. The image processing device that is used as part of the chromakey system can substitute any desired resolution of image or video for the background. Also, the chromakey system enables the video producer to change aspects of the background as desired.
The present invention describes technology to address these issues. The overall system is shown in
A camera 120 records a video. The video is formed of individual frames that include the subject 100 in front of the screen 110, which may be an LED screen. A processing element 140 operates as described herein, to synchronize the camera 120 to take frames that are synchronized with the display on the screen of the alternating backgrounds.
According to an embodiment, the camera takes more images than are necessary in order to create the final video. In embodiments, the camera takes n frames, where n is 2 or greater, for each desired frame of the final video. One of the frames has an actual background, which in an embodiment may illuminate the objects. Another frame, which is adjacent in time to the first frame, has a chromakey background. The a chromakey background can be any of the chromakey frames as described herein. Where n is 2, there is typically one chromakey frame and one actual background frame. Where n is 3, there may be two chromakey frames of different colors or formats, and one background frame, or two different background frames having different scenes, and one chromakey frame.
In one simplified embodiment, the camera may take double the images which are necessary to obtain the video. In one embodiment, for example, a 24 frame per second video is desired. In this embodiment, 30 frames per second are actually taken, with each being 1/48 of a second receiving the chromakey backgrounded frame, and 1/48 of a second frames receiving ambient light from the actual image of the desired background image or scene on the video screen.
Alternate frames include different backgrounds on the screen 110, which are later separated by the image processor 150. This produces two videos, where both have the same “content”, i.e., the same objects in the same locations at the same times, but different backgrounds.
In this first embodiment, a first frame can be a chromakey background such as a green screen. During that first frame, the camera 120 is triggered at a time when the green screen is displayed in the background. This obtains frame number 0 in chromakey background.
A short time later, e.g. 1/24th of a second later, the screen 110 is controlled to display an desired background, for example the background of either the actual scene which is intended, or something approximating the actual screen which is intended. While the actual background is being displayed, the second frame is obtained, that frame being frame number 1 again, but this time with the desired background.
An image processing module 150 separates out the alternate frames into chromakey frames 151 and actual background frames 152.
In this way, from a single video acquisition, the video acquirer gets the opportunity to receive either a chromakey sequence of frames, or an desired/ actual background sequence of frames, or parts of both.
More complex operations can also be carried out using this system.
In one embodiment, the image from the actual background frames actually have more realistic lighting on the subject then the image from the chromakey frames. In this embodiment, the image processor 160 can be used for substituting the subject image from the actual background frames into the chromakey frames, so that the lighting of the subject in the chromakey frames is more natural looking by virtue of its being lit from the intended background “alternating” frames displayed from the screen 110.
The image capture device 120 can be any kind of camera, e.g. a film camera or a digital cinema camera or any other kind of camera. The camera is synchronized via a camera shutter pulse with the projection of light from the video display array 110. The camera is preferably a global shutter style camera, although a rolling shutter can be used also.
Imagers that use a global shutter operate by resetting the entire imager before each frame is taken. Each frame is comprised of multiple different pixels, each pixel being a sensor well in the imager device that integrates charge received via its photoreceptors. Before integration of any frame, the entire imager is reset to remove any residual signal in the sensor wells. The wells or pixels then integrate while exposed to the image frame, to accumulate charge for some period of time. In a global shutter of this type, the light collection for frame starts and ends at exactly the same time for all pixels. At the end of the time during which light is collected, or “integration period”, all charges are simultaneously transferred to light shielded areas of the sensor. The light shield prevents further accumulation of charge during the readout process. The charge is then shifted out of the light shielded areas of the sensor and read out, as the output of the camera.
This means that with a global shutter sensor, the scene is “frozen” in time with no motion blur. A preferred embodiment uses a digital imager with a global shutter, so that all pixels of the frame are acquired at the same moment.
Another embodiment, however, can use this system with a rolling shutter. With the rolling shutter, the photodiodes (pixels) do not collect light at the same time. All the pixels in one row of the imager collect light during exactly the same period of time, but the time that light collection starts and ends is slightly different for each row. The top row of the imager is the first one to start collecting the light and is the first one to finish collecting. The start and end of the light collection for each following row is slightly delayed. The total light collection time for each row is exactly the same, and the delay between rows in the rolling shutter sensor is constant. This means that with a rolling shutter sensor, the scene can have blur effects from motion. A rolling shutter sensor can still be used with this system, by syncing the background image/chromakey on the display screen 110 for the amount of time that the rolling shutter needs to “roll” from one end to the other, or by rolling the background image on the frame in sync with rolling of the shutter.
In one embodiment, the interleaved image sequence can be interleaved between an actual background, and a single color chromakey.
In another embodiment, the interleaved image sequence can be interleaved between the actual background, and multiple different color chromakeys. For example in this embodiment, the camera 120 can take four times as many images as are necessary for the actual video, e.g. 60 or 120 images per second. The first image may be actual background, while the second image is blue chromakey, the third image is green chromakey, and the fourth image is red chromakey. This may allow the video producer to have the maximum amount of flexibility, since they can use actual/desired background and/or any of multiple different chromakey colors, for each of the frames. When different objects with different colors are present in the different frames, different chromakey backgrounds can be selected.
Another embodiment can use the alternating frame system of the first embodiment, along with the special chromakey frame shown in
In one embodiment, the special frame of
Other embodiments are intended to be encompassed within the invention. Moreover, the invention is intended to include additional features such as processors, memory, and software.
This application claims priority from provisional application No. 62/334,324, filed May 10, 2016, the entire contents of which are herewith incorporated by reference.
| Number | Date | Country | |
|---|---|---|---|
| 62334324 | May 2016 | US |
