Various embodiments are illustrated by way of example, and not by way of limitation, in the accompanying drawings.
Illustrated in
Illustrated in
a and
One architectural configuration of a frame registrator is shown in
A graph for a set of horizontal projections for a base frame and a movement frame is illustrated in
A graph of a set of vertical projections for a base frame and a movement frame is illustrated in
The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment, configuration or design described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments or designs. In general, described herein, is a novel method and apparatus to reduce blurriness and/or noise in digital pictures by generating a global motion vector and using the global motion vector for frame registration.
In conventional camera devices, when a user takes a snapshot (currently done by pressing a button), mostly only one frame is used to generate a picture. Methods which employ using more than one frame to generate a picture often are not successful because they yield poor results. With conventional camera devices, the picture may be blurry due to movements produced by the user's own hand movements, these hand movements are known as hand jitter. Conventional camera devices also are challenged by the amount of time required to expose a picture. Under low lighting conditions, the exposure time is typically increased. Increasing the exposure time increases the amount of noise that a user may see due to low lighting conditions as well as increases the probability that hand jitter will produce a blurry picture. Currently, camera devices may contain small gyroscopes to compensate for the hand jitter produced by the user. However, there are many challenges faced when placing gyroscopes on mobile units. Even when these challenges are overcome, the digital hand jitter reduction techniques may be used in combination with devices that have gyroscopes. Current camera devices may also scale the gain under low lighting conditions. Unfortunately, simply increasing the gain amplifies the noise present as a result of the low light level. The result is often a picture of poor quality. Similarly, digital compensation for hand jitter does not always provide adequate results. However, with the techniques disclosed throughout this disclosure, it has been possible to reduce hand jitter, as well as reduce noise under lower light conditions.
Before a user presses the button to take a snapshot and produce a digital picture, a preview mode, may capture a series of frames produced by the image sensor 102. The whole frame or a sub-part of the frame is referred to as an image or interchangeably a picture. For illustrative purposes, it is convenient to discuss the images being processed as a series of frames. Although it should be recognized that not the entire frame need be processed when using a front-end image processing module 106. In addition, the sequence of frames is also known as a stream. The stream may be provided to a front-end image processing module 106 where they are de-mosaiced in order to obtain full RGB resolution as an input to the still image and video compressor 108. As the stream passes through the front-end image processing module 106, in the preview mode, statistics may be collected on frames that aid with the production of the digital picture. These statistics may be, but are not limited to, exposure metrics, white balance metrics, and focus metrics.
The front-end image processing module 106 may feed various signals, which help control the image sensor 102, back into the image sensor module 104. The still image and video compressor 108 may use JPEG compression, or any other suitable compression algorithm. An auto-exposure control module 110 may receive a value proportional to the light level being processed by the front-end image processing module 106, and compare it to a stored light target, in order to aid in at least one of the functions of the front-end image processing module 106. Images that are processed through the modules in front-end image processing module 106 are part of digital frames. The stream may also be sent to a view finder which may be located in display module 112. In the preview mode, a preview decision from the display module 112 may be used in the control of the auto-exposure.
The preview mode in a mobile unit having a digital camera may be used in either a normal mode or a hand jitter reduction (hjr) mode. The user may select the hjr mode (shown as hjr select in
After the color conversion module processes a frame, three color image-components (Y, Cb, and Cr) may be may be sent to hand jitter control module 212. The various parameters from the auto-exposure control module may be fed into hand jitter control module 212. Hand jitter control module 212 may serve multiple purposes. Hand jitter control module 212, may determine the image processing that takes place after the snapshot. Hand jitter control module 212 may detect the value of hjr select, and determine if hand jitter reduction (hjr) needs to be performed. Even though the user has selected hjr mode, hand jitter control module 212 may determine that image processing as is done in normal mode may take place. Hand jitter control module 212 may determine that image processing in hjr mode take place. Generating a digital picture image processing in hjr mode may include capturing a single frame or multiple frames. If hand jitter control module 212 determines that multiple frames will be captured, after passing through hjr control module, the frames may be sent to noise reduction/frame registration module 214, along with a parameter which indicates how many frames may be processed by noise reduction/ frame registration module 214. If a single frame is to be processed, noise reduction may take place on the single frame through the use of a noise reduction module 215. Noise reduction module may be a bayer filter, or other similar filter. If multiple frames are to be processed, noise reduction/ frame registration module 214 may buffer the number of frames, numf, specified by hand jitter control module 212, and perform frame registration on them. Depending on how many frames and the light level, the purpose of the multiple frame registration may serve the purpose of noise reduction and/or blur reduction. Multiple frame registration may be done by a frame registration module 216.
If hand jitter control module 212 determines that image processing takes place as in normal mode, noise reduction/ frame registration module 214 may not be used, and the output from color correction module 210, for example, may be used, even though the user selected hjr mode. Depending on what image processing (the one in normal node or the one in hjr mode) is determined by hand jitter control module 212, a signal (sel) may be used to select which multiplexer 217 output to send to post-process module 218. The output of post-process module 218 may be sent to still and image video compressor 108 and/or display module 112.
In addition to outputting a select signal (sel) and the number of frames to use for noise reduction and/or frame registration, hand jitter control module 212 may also output other parameters: new auto-exposure frame rate (ae fr_new), new auto-exposure gain (ae gain_new), new auto-exposure time (ae time_new), and the number of frames to be processed (numf). These parameters may be sent to image sensor module 104 to control image sensor 102. A digital gain may also be output by hand jitter control module 212 and may be applied at any module after the image sensor module 104. As an example, the digital gain may be applied during the white-balance/color correction module 206.
Those ordinarily skilled in the art will recognize that while pixels are normally described, sub-pixels, or multiple pixels may also be used as inputs into front-end image processing module 106a. Furthermore, a sub-set of these image-components or other forms: RGB, and spatial-frequency transformed pixels, may also be sent to a hand jitter control module, such as hand jitter control module 212.
As mentioned previously, the frame registration module 216 may used to reduce the amount of blurriness or reduce noise in a digital picture with efficient processing resources suitable for mobile applications. Currently, a normal exposure time for a picture may be around 150-300 milli-seconds (ms). Instead of capturing one picture (frame) in 150-300 ms, N frames may be captured and processed at reduced exposure times prior to frame registration. In order to reduce the amount of blurriness in a picture, frame registration module 216 compensates for the amount of rotational movement between any two frames amongst the N frames being processed at the reduced exposure times.
Typically in a frame registration module 216, N frames are processed by iteratively selecting a pair of frames at a time: a base frame and a movement frame. Compensation of the amount of horizontal and vertical movement, between the base frame and the movement frame, is accomplished by estimating a global motion vector during every iterative frame pair selection and “registering” the movement frame to the base frame. After computing an estimate of the global motion vector between the horizontal and vertical movement frame relative to the base frame, the movement frame is registered to the base frame by adding the estimated global motion vector to the base frame. The registered frame represents the compensation of the base frame due to the estimated horizontal and vertical movement between the base frame and the movement frame. The registered frame may be used as a new base frame or may be used as a new movement frame. The selection of how any frame, registered or not registered, depends on the parity of the number of frames being processed and may be configurable. The frame selection process is discussed in more detail in
The frame selection process may be implemented by a frame selector control 300, seen in
After the color conversion process, three color image-components (Y, Cb, and Cr) may be input into a frame registration module 216a. However, those ordinarily skilled in the art will recognize that a sub-set of these image-components or other forms may be used, such as RGB; and spatial-frequency transformed pixels of these image-components. Furthermore, while pixels are normally used, sub-pixels, or multiple pixels may also be used as inputs. Image-component Y may be routed to input Y frame buffer 302, image-component Cb may be routed to input/merged Cb frame buffer 304, and image-component Cr may be routed to input/merged Cr frame buffer 306. Frame registration may be carried out on all three image-components (Y, Cb, and Cr). Estimation on a global motion vector need only be performed on one of the image-components, although it may be performed on more than one components. As an example, the Y image-component may be used to estimate a global motion vector. As such, a registered frame may be routed to merged Y frame buffer 308. In addition, the frame registration process may be carried out on only part of a frame if desired.
Frame selector 300 may have up to five outputs, mux_sel, fsel_Cb, fsel_Cr, fsel_fY, and fsel_gY. From frames fa, fb, ga, or gb, mux_sel selects from mux 310 which two pair of frames may be used to estimate a global motion vector between a base frame and a movement frame. A base frame is designated by frame sfY_a, and a movement frame is designated by a movement frame sfY_b. Selection of frame fa and frame fb may be through signal fsel_fY, while selection of frame ga and frame gb may be through signal fsel_gY. Similarly, fsel_Cb and fsel_Cr may be used to select which Cb (sfCb) and Cr (sfCr) frames may be used for frame registration of the Cb and Cr image-components.
Frame sfY_a may be routed to a projection generator 314, which may generate I horizontal projections and M vertical projections. Interpolation or decimation of rows in a frame may also be implemented, i.e., the number of horizontal projections generated in a frame may be less (decimation) than I, or more (interpolation) than I. Hence, there may be D horizontal projections, where D represents the number of decimated rows, D is less than I. There may be L horizontal projections, where L represents the number interpolated rows, L is more than I. The horizontal projections, whether interpolated, decimated, or neither) may be low pass filtered via low pass filter 316. The set of horizontal projections (whether filtered or not) {H(0), H(1), . . . H(I-1)}, may be represented by a vector, and is denoted by H. Analogously, interpolation or decimation of columns in a frame may also be implemented. The vertical projections (whether interpolated, or neither) may be low pass filtered via low pass filter 318. The set of vertical projections (whether filtered or not) {V(0), V(1), . . . V(M-1)}, may be represented by a vector, and is denoted by V.
Similarly, frame sfY_b may be routed to a projection generator 314, which may generate horizontal and vertical projections. Interpolation or decimation of rows in a frame may also be implemented. The horizontal projections (whether interpolated, decimated, or neither) may be low pass filtered via low pass filter 320. The set of horizontal projections {H′(0), H′(1), . . . H′(I-1)}, may be represented by a vector, and is denoted by H′. Analogously, interpolation or decimation of columns in a frame may also be implemented. The vertical projections (whether interpolated, decimated, or neither) may be low pass filtered via low pass filter 322. The set of vertical projections (whether filtered or not) {V′(0), V′(1), . . . V′(M-1), may be represented by a vector, and is denoted by V′.
Global motion vector estimator 324 receives four sets of data, namely input vectors H, V, H′ and V′, and generates global motion vector estimate gmv 329, where gmv is (Δx, Δy). Global motion vector gmv 329 may be added to sfY_a, sfCb, and sfCr in frame registrator 330. The resulting registered frames, namely, sfY_a+gmv, sfCb+gmv, sfCr+gmv, may be stored in buffer memories merged_Y frame buffer, input/merged Cb frame buffer, and input/merged Cr frame buffer, respectively. After the first iteration of frame registration(s), frame ga may be available for the second iteration of frame registration(s). Early terminator 332 may determine if the global motion vector estimate is within a certain tolerance and cause an early exit condition and terminate the frame registration earlier than the processing of N frames.
Although frames may be processed simultaneously, through interleaving of rows between a base frame and a movement frame,
As in
The inputs (H, V, H′ and V′) and the output (global motion vector 329) to motion vector estimator 328, as well as the components and calculations that follow the estimate of global motion vector estimate 329, are as disclosed in
Selection of which frame is a base frame and which frame is a movement frame may be designated by frame flow-trees such as those illustrated in
Frame flow-tree 332b illustrated in
Frame flow-tree 332d illustrated in
Frame flow-trees 332e and 332f illustrated in
As the number of frames increases, the exposure times between frames decreases and the probability that there is a smaller linear displacement increases. That is, the estimated linear displacement between frames when the exposure time is smaller is likely to be smaller, thus, accuracy in the estimate used for compensating for the linear displacement better. Hence, the process illustrated in frame flow-tree 332f may be implemented in a device that takes digital pictures using hand jitter reduction, since the process is likely to be sufficient most of the time to the human eye. Other applications that may require higher resolutions and are not as concerned with computation time, may wish to implement a process where there are a higher number of total frame registrations, such as in frame flow-tree 332e.
Projections may be on columns or rows.
Similarly, projection correlator 390, may compute the difference between the two input vectors (V370y and V′ 370y′) and generates a projection correlation error (pce) vector at each shift position between the two input vectors. As discussed above, computing the difference of the input vectors for the set of shift positions between the input vectors generates a set of pce vectors. Computing the norm of each pce vector at each shift generates a pce value. Each pce value may be stored in memory 396. Minimum pce value index selector 398 selects the minimum pce value amongst the set of pce values stored in memory 396. The index selector outputs the shift position that corresponds to the minimum pce value, i.e., the index of the minimum pce value is the shift position, and is called the y-component of the global motion vector, Δy 328.
a illustrates more details of the global motion vector estimator 324. Input vectors, H360x and H′ 360x′, are input into projection correlator 380. Either input vector may be connected to a shifter 381 which shifts by Δy positions. The shifter 381 is used for shift aligning the vector H360x with the different vector H′ 360x′. Subtractor 382 computes the difference between the two input vectors and generates a projection correlation error (pce) vector at each shift position between the two input vectors. Computing the norm of each pce vector at each shift position generates a pce value. To compute the norm, abs block 383 computes the absolute value of the pce vector, and summer 384 sums all the elements of the pce vector. Thus, each pce value is a norm of a pce vector. Illustrated in
Analogously,
Mathematically, the set (for all values of Δy,) of pce values to estimate a vertical movement between frames is captured by equation 1 below:
Equation 1 above, may generate a vertical estimate that is between 0 and M-1 elements (e.g., pixels, sub-pixels). Equation 2 below, may also be used to capture the set of pce values. However, this may generate a vertical estimate that is between -M/2 and M/2-1 elements (e.g., pixels, sub-pixels).
Similarly, equations 3 and 4, may be used to estimate a horizontal movement between frames.
Equation 3 above, may generate a vertical estimate that is between 0 and I-1 elements (e.g., pixels, sub-pixels). Equation 4 below, may also be used to capture the set of pce values. However, this may generate a vertical estimate that is between -I/2 and I/2-1 elements (e.g., pixels, sub-pixels).
Subsequent to all N frames being processed on one image-component (e.g., Y), it may be possible to add all the estimated rotation angles to the appropriate frames on the other image-components (e.g., Cb and Cr). This may happen because projections need only be generated for one image-component and the frame registration sequence is known beforehand via a frame flow-tree. This possible architecture configuration of frame registrator 330 is not shown. One configuration of frame registrator 330, which is shown in
The setting of both sign-bits may trigger an early exit signal and if desired, stop the processing of the N unregistered frames.
A graph 472 for a set of horizontal projections for a base frame (labeled 1st image) and a movement frame (labeled 2nd image) is illustrated in
A graph 474 for a set of vertical projections for a base frame (labeled 1st image) and a movement frame (labeled 2nd image) is illustrated in
A number of different configurations and techniques have been described. The techniques may improve removing blurriness from images with longer exposure times. The techniques and configurations may also aid in the reduction of hand jitter for practically any digital device that takes pictures. The techniques and configurations may be implemented in hardware, software, firmware, or any combination thereof If implemented in software, the techniques and configurations may be directed to a computer-readable medium comprising computer- readable program code (also may be called computer-code), that when executed in a device that takes pictures, performs one or more of the methods mentioned above.
The computer-readable program code may be stored on memory in the form of computer readable instructions. In that case, a processor such as a DSP may execute instructions stored in memory in order to carry out one or more of the techniques described herein. In some cases, the techniques may be executed by a DSP that invokes various hardware components, such as projection correlation, to accelerate the frame registration process. The frame registration techniques and configurations disclosed may be implemented in one or more microprocessors, one or more application specific integrated circuits (ASICs), and one or more field programmable gate arrays (FPGAs), or some other hardware-software combination. These techniques and configurations are within the scope of the following claims.
This application claims the benefit of provisional U.S. Application Ser. No. 60/760,768, entitled “HAND JITTER REDUCTION SYSTEM DESIGN,” filed Jan. 19, 2006. This disclosure is related to co-pending patent applications “A hand jitter reduction system for cameras” (Attorney Docket No. 060270), co-filed with this application, and “Hand jitter reduction compensating for rotational motion” (Attorney Docket No. 060268) co-filed with this application.
Number | Date | Country | |
---|---|---|---|
60760768 | Jan 2006 | US |