The present disclosure generally relates to digital images, and more particularly to stabilization of digital images.
A digital imaging device, such as a digital camera, may be used to capture a variety of scenes. An image of a scene captured by the digital camera may exhibit a degree of blurriness. The blurriness is reflected in the image due to unwanted motion present in the image. The unwanted motion present in the image is caused either by a movement in the scene or by a movement of the digital camera while a user is capturing the scene using the digital camera. Either or both of these movements cause motion artifacts and blurriness in the image. A process of removing the blurriness and motion artifacts from the image is termed as image stabilization.
The present disclosure provides a method and a system to produce stabilized images with reduced blurriness and motion artifacts.
In one aspect, the present disclosure provides a method for processing a digital image, the method comprising: selecting a set of frames from a plurality of frames captured by a digital imaging device; identifying a set of pixel blocks from the set of frames; and integrating the set of pixel blocks to process the digital image.
In another aspect, the present disclosure provides a digital imaging device having an image processor for processing a digital image, the image processor comprises: a frame selecting module capable of selecting a set of frames from a plurality of frames captured by a digital imaging device; an identifying module capable of identifying a set of pixel blocks from the set of frames; and an integrating module capable of integrating the set of pixel blocks to generate the digital image.
In yet another aspect of the present disclosure, the present disclosure provides computer-implemented methods, computer systems and a computer readable medium containing a computer program product for processing a digital image by an image processor, the computer program product comprising: program code for selecting a set of frames from a plurality of frames captured by a digital imaging device: program code for identifying a set of pixel blocks from the set of frames; and program code for integrating the set of pixel blocks to process the digital image.
The accompanying drawings, where like reference numerals refer to identical or functionally similar elements throughout the separate views, together with the detailed description below, are incorporated in and form part of the specification, and serve to further illustrate embodiments of concepts that include the claimed disclosure, and explain various principles and advantages of those embodiments.
The method and system have been represented where appropriate by conventional symbols in the drawings, showing only those specific details that are pertinent to understanding the embodiments of the present disclosure so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.
Before describing in detail embodiments that are in accordance with the present disclosure, it should be observed that the embodiments reside primarily in combinations of method steps and system components related to processing a digital image.
As used herein, relational terms such as first and second, and the like may be used solely to distinguish one module or action from another module or action without necessarily requiring or implying any actual such relationship or order between such modules or actions. The terms “comprises,” “comprising,” or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements that does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by “comprises . . . a” does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.
Any embodiment described herein is not necessarily to be construed as preferred or advantageous over other embodiments. All of the embodiments described in this detailed description are illustrative, and provided to enable persons skilled in the art to make or use the disclosure and not to limit the scope of the disclosure, which is defined by the claims.
The present disclosure provides a method and a system for stabilizing a digital image. Specifically, the method and the system disclosed in the present disclosure reduce motion artifacts and blurriness from a digital image in a capture process in a digital camera. More specifically, in the capture process, a plurality of frames of a scene is captured by the digital camera and is integrated so as to aggregate sharpest pixel blocks of the plurality of frames. The integration of the plurality of frames results in a stabilized digital image which is free from motion artifacts and blurriness. Each frame of the plurality of frames is basically an image and is captured at a very short interval of the scene. In one embodiment, the digital camera may capture N frames of the scene in one second, where 2≦N≦16. In one embodiment of the present disclosure, the digital image is processed/generated/stabilized by selecting sharpest pixels blocks from the plurality of frames and subsequently integrating the sharpest pixels blocks to achieve the stabilized digital image, as will be explained in conjunction with
Referring to
In one embodiment of the present disclosure, the plurality of frames captured by the digital imaging device 100 in one second may be N, where 2≦N≦16. The plurality of frames may be stored in the image buffer 106 which may be a memory device capable of storing a large amount of data. The image buffer 106 may be coupled to the image processor 108. The image processor 108 is capable of reading the plurality of frames from the image buffer 106 for processing the plurality of the frames. In one embodiment, the digital imaging device 100 may be in the form of the digital camera, in which case, the digital imaging device 100 may include other components dictated by functions of the digital camera.
Referring now to
At a more detailed level, it is to be understood that the frame selecting module 200 is capable of selecting a set of frames K from the plurality of frames N stored in the image buffer 106. The set of frames may include one or more frames. In one embodiment, the set of frames may include frames 300 to 320 as shown in
In one embodiment, each frame of the plurality of frames may be assigned a sharpness parameter. In a preferred embodiment, the sharpness parameter for each frame of the plurality of frames may be calculated by using a smoothed version of local gradient information. Specifically, the sharpness parameter for each frame may be calculated using the following equation:
where, hi,j is an impulse response of a high pass filter for a pixel xm,n of a frame, and wm,n is a weight array. The term in parenthesis is a high pass version of the frame at pixel location (m, n) and hence reflects local sharpness or gradient information in the frame. The local gradient information is summed over the entire frame to give a sharpness parameter. The sum is weighted using a weight array so that it is possible to put emphasis in selected areas of the frame, e.g. putting higher emphasis on middle portions of the frame as compared to boundary portions.
Based on the sharpness parameter of each frame, the set of frames may be selected from the plurality of frames by the frame selecting module 200. Specifically, the frame selecting module 200 may select sharpest frames, which may constitute the set of frames, from the plurality of frames.
Subsequently, another sharpness parameter may be computed, using the method explained above, for each pixel block in each frame of the set of frames. Based on this sharpness parameter, the identifying module 202 may identify a set of pixel blocks from the set of frames. For example, the identifying module 202 may select a pixel block 1a from the frame F1 when the pixel block 1a has a highest sharpness parameter among corresponding pixel blocks in remaining frames of the set of frames. Similarly, the identifying module 202 may select sharpest pixel blocks, which may constitute the set of pixel blocks, from the set of frames.
After the set of pixel blocks is identified, the motion estimating module 204 may calculate motion vectors on a block by block basis for the set of frames. Subsequently, the motion estimating module 204 may compensate a motion between the set of frames based on the motion vectors. In a preferred embodiment, the motion vectors are calculated for block size of 16×16 with a search range of plus minus 8 in each direction. The motion vector for a 16×16 pixel block may be calculated using the following equation:
Motion vector=Global coarse motion vector+Local fine motion vector
Global coarse motion vector may be calculated using N×N regions spread uniformly throughout a frame. Motion estimations for each of the N×N regions are performed over a relatively large search range to obtain N×N motion vectors, one for each region. Then a classification method is applied to detect outliers among the N×N motion vectors, and linear interpolation among the motions vectors that are not classified as outliers is used to adjust outlier motion vector values. The global coarse motion vector is then calculated as the average of the N×N motion vectors after processing the outlier motion vector values. In a preferred embodiment, N=7, and the search range is plus minus 32. Even though this is a relatively large search range, a number of blocks that this search will be performed on are 49, which is small compared to all the 16×16 blocks in the entire frame.
After the global coarse motion vector has been determined, the local fine motion vector for each 16×16 block in the frame is determined using the global coarse motion vector as an offset. In this embodiment, the search range for the local fine motion vector can be significantly reduced. In a preferred embodiment, the search range of plus minus 8 pixels is used for local fine motion vector estimation.
Having the motion vectors and sharpness parameters for each pixel block of the set of frames, the integrating module 206 may integrate the set of pixel blocks to generate a stabilized digital image 322 as shown in
The epipolar line constraint means that all pixel blocks that share a same row should fall along a straight line at same angle after motion compensation. The ordering constraint means that if a pixel block m is on a left of a pixel block n before motion compensation, then a relative directional position of the two pixels blocks m and n should remain the same after motion compensation. The continuity constraint means that the motion vectors of neighboring pixel blocks should be smooth.
The motion vectors for each pixel block are stored in the frame and are then adjusted to maintain the three constraints; thereby compensating the motion and avoiding the artifacts. This may be done using an iterative procedure that is equivalent to low pass filtering the motion vectors so that a motion from block to block transitions smoothly and continuously.
Finally, the digital image 322 may be read by the post capture processing module 208 which uses a combination of two linear filters and a contrast mapping step (not shown). The two linear filters have a low pass and a high pass characteristic, respectively. Filtering of the digital image is controlled by an edge detector such as a Sobel edge detector. Using the edge information, non-edge pixels are low pass filtered whereas edge pixels are high pass filtered. This configuration serves to filter noise in the digital image as well as enhances the edges. The filtered result is put to a local contrast mapping step to enhance the local contrast. A pre-defined S-curve that is normalized to maximum and minimum pixels within each pixel neighborhood is used for mapping pixel data.
Referring now to
In one embodiment, after the sharpest frame F2 is mapped onto the image Y, each pixel block of the image Y is compared with corresponding pixel blocks of remaining frames of the set of frames. If a sharpness parameter of a pixel block in the image Y is less than a sharpness parameter of a corresponding pixel block in a frame of the remaining frames, then the pixel block in the image Y is replaced by the corresponding pixel block of the frame. For example, the sharpness parameter of the first pixel block 1b of the image Y may be compared with the sharpness parameters of corresponding pixel blocks 1a, 1c, 1d, to 1k of frames F1, F3, F4 to Fk, respectively. If the sharpness parameter of the pixel block 1b in the image Y is less than the sharpness parameter of any of the corresponding pixel blocks 1a, 1c, 1d, to 1k, then the pixel block 1b gets replaced in the image Y with the corresponding pixel block having a higher sharpness parameter. In this embodiment, the pixel block 1b is replaced by the corresponding pixel block 1a of frame F1 as the sharpness parameter of the corresponding pixel block 1a is higher than that of the pixel block 1b. Similarly, all the pixel blocks of the image Y are compared with corresponding pixel blocks of the remaining frames of the set of frames to generate a digital image 322 having sharpest pixel blocks of the set of frames.
In another embodiment of the present disclosure, after the sharpest frame is mapped onto the image Y, the sharpness parameter of each pixel block of the image Y is compared with corresponding pixel blocks of a next frame of the set of frames. A pixel block in the image Y is replaced by a corresponding pixel block of the next frame based on the sharpness parameter, to generate an improved image Y1 (not shown). Subsequently, each pixel block of the improved image Y1 is compared with a next frame to generate an improved image Y2 (not shown). This process may continue till the digital image 322 is generated having sharpest pixel blocks selected from the set of frames. To illustrate this with the help of an example, consider that the image Y and the frame F1 are integrated so as to generate an improved image Y1. Subsequently, the improved image Y1 is compared with frame F3 and same integration procedure is performed to generate an improved image Y2. This is continued until all the frames are considered on a block by block basis to generate the digital image 322.
Prior to integrating the image Y with the remaining frames, motion vectors are calculated on a block by block basis between the image Y and the remaining frames in a manner explained above. Further, the motion vectors are adjusted based on the dime constraints explained above to avoid artifacts. Furthermore, a motion between the image Y and the remaining frames is compensated based on the motion vectors to generate stabilized digital image 322.
Referring now to
Referring now to
It will be appreciated that embodiments of the disclosure described herein may comprise one or more conventional processors and unique stored program instructions that control the one or more processors to implement, in conjunction with certain non-processor circuits, some, most, or all functions of processing a sensor data. Alternatively, some or all functions of processing a sensor data could be implemented by a state machine that has not stored program instructions, or in one or more Application Specific Integrated Circuits (ASICs), in which each function or some combinations of certain of the functions are implemented as custom logic. Of course, a combination of the two approaches could be used. Thus, methods and means for these functions have been described herein. Further, it is expected that one of ordinary skill, notwithstanding possibly significant effort and many design choices motivated by, for example, available time, current technology, and economic considerations, when guided by the concepts and principles disclosed herein will be readily capable of generating such software instructions and programs and ICs with minimal experimentation.
As will be understood by those familiar with the art, the disclosure may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Likewise, the particular naming and division of the modules, agents, managers, functions, procedures, actions, methods, classes, objects, layers, features, attributes, methodologies and other aspects are not mandatory or significant, and the mechanisms that implement the disclosure or its features may have different names, divisions and/or formats. Furthermore, as will be apparent to one of ordinary skill in the relevant art, the modules, agents, managers, functions, procedures, actions, methods, classes, objects, layers, features, attributes, methodologies and other aspects of the disclosure can be implemented as software, hardware, firmware or any combination of the three. Of course, wherever a component of the present disclosure is implemented as software, the component can be implemented as a script, as a standalone program, as part of a larger program, as a plurality of separate scripts or programs, as a statically or dynamically linked library, as a kernel loadable module, as a device driver, and/or in every and any other way known now or in the future to those of skill in the art of computer programming. Additionally, the present disclosure is in no way limited to implementation in any specific programming language, or for any specific operating system or environment. Accordingly, the disclosure of the present disclosure is intended to be illustrative, but not limiting, of the scope of the disclosure, which is set forth in the following claims.