IMAGER-CREATED IMAGE SIGNAL-DISTORTION COMPENSATION METHOD, IMAGER-CREATED IMAGE SIGNAL-DISTORTION COMPENSATION APPARATUS, IMAGE TAKING METHOD AND IMAGE TAKING APPARATUS

Abstract

Disclosed herein is a method of compensating an imager-created image for a distortion. The method includes: a movement-vector detection process of receiving data of the imager-created image and detecting a movement vector; an optical-zoom portion vector computation process of computing an optical-zoom portion vector corresponding to an image movement; an optical-zoom portion vector subtraction process of subtracting the optical-zoom portion vector from the movement vector detected; a global hand-movement displacement vector computation process of computing a global hand-movement displacement vector for the imager-created image from the movement vector in the case of no optical zoom operation and computing a global hand-movement displacement vector for the imager-created image from a difference vector output in the case of an optical zoom operation; and a hand-movement compensation process of compensating the imager-created image for a distortion caused by a hand movement on the basis of the global hand-movement displacement vector.

Description

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram showing a typical configuration of a hand-movement displacement-vector detection unit employed in an image taking apparatus adopting a method for compensating an imager-created image for a distortion in accordance with a first embodiment;

FIG. 2 is a block diagram showing a typical configuration of the image taking apparatus adopting a method to compensate an imager-created image for a distortion in accordance with the first embodiment;

FIG. 3 is an explanatory diagram to be referred to in description of optical zoom control executed in the image taking apparatus;

FIG. 4 is an explanatory diagram to be referred to in description of an embodiment implementing a method to compensate an imager-created image for a distortion in accordance with an embodiment;

FIGS. 5A and 5B are each an explanatory diagram to be referred to in description of an embodiment implementing a method to compensate an imager-created image for a distortion in accordance with an embodiment;

FIGS. 6A and 6B are each an explanatory diagram to be referred to in description of an embodiment implementing a method to compensate an imager-created image for a distortion in accordance with an embodiment;

FIG. 7 is an explanatory diagram to be referred to in description of an embodiment implementing a method to compensate an imager-created image for a distortion in accordance with an embodiment;

FIG. 8 is an explanatory diagram to be referred to in description of a process to detect a movement vector by adoption of a block matching technique;

FIG. 9 is an explanatory diagram to be referred to in description of a process to detect a movement vector by adoption of the block matching technique;

FIG. 10 shows a flowchart to be referred to in explanation of a process to detect a movement vector by adoption of the block matching technique;

FIG. 11 is an explanatory diagram to be referred to in describing an outline of a process to detect a movement vector in accordance with an embodiment;

FIGS. 12A and 12B are each an explanatory diagram to be referred to in describing an outline of a process to detect a movement vector in accordance with an embodiment;

FIG. 13 is an explanatory diagram to be referred to in describing an outline of a process to detect a movement vector in accordance with an embodiment;

FIGS. 14A and 14B are each an explanatory diagram to be referred to in description of a typical process to detect an accurate movement vector by adoption of an image processing method according to an embodiment;

FIG. 15 is an explanatory diagram to be referred to in description of a typical process to detect an accurate movement vector by adoption of an image processing method according to an embodiment;

FIG. 16 is an explanatory diagram to be referred to in describing an outline of a process to detect a movement vector in accordance with an embodiment;

FIG. 17 is an explanatory diagram to be referred to in describing an outline of a process to detect a movement vector in accordance with an embodiment;

FIGS. 18A and 18B are each an explanatory diagram to be referred to in description of a typical process to detect an accurate movement vector in accordance with a first typical implementation of the movement-vector detection method according to the embodiment;

FIG. 19 is an explanatory diagram to be referred to in description of a process to detect an accurate movement vector in accordance with the first typical implementation of the movement-vector detection method according to the embodiment;

FIG. 20 is an explanatory diagram to be referred to in description of a process to detect an accurate movement vector in accordance with the first typical implementation of the movement-vector detection method according to the embodiment;

FIGS. 21A and 21B are each an explanatory diagram to be referred to in description of a process to detect an accurate movement vector in accordance with the first typical implementation of the movement-vector detection method according to the embodiment;

FIGS. 22A, 22B, 22C and 22D are each an explanatory diagram to be referred to in description of a process to detect an accurate movement vector in accordance with the first typical implementation of the movement-vector detection method according to the embodiment;

FIG. 23 is an explanatory diagram to be referred to in description of a process to detect an accurate movement vector in accordance with the first typical implementation of the movement-vector detection method according to the embodiment;

FIG. 24 is an explanatory diagram to be referred to in description of a process to detect an accurate movement vector in accordance with the first typical implementation of the movement-vector detection method according to the embodiment;

FIGS. 25A and 25B are each an explanatory diagram to be referred to in description of a process to detect an accurate movement vector in accordance with a second typical implementation of the movement-vector detection method according to the embodiment;

FIG. 26 is an explanatory diagram to be referred to in description of a process to detect an accurate movement vector in accordance with the second typical implementation of the movement-vector detection method according to the embodiment;

FIG. 27 is an explanatory diagram to be referred to in description of a process to detect an accurate movement vector in accordance with the second typical implementation of the movement-vector detection method according to the embodiment;

FIGS. 28A, 28B, 28C and 28D are each an explanatory diagram to be referred to in description of a process to detect an accurate movement vector in accordance with the second typical implementation of the movement-vector detection method according to the embodiment;

FIG. 29 is an explanatory diagram to be referred to in describing the performance of the movement-vector detection method according to an embodiment;

FIG. 30 is an explanatory diagram to be referred to in describing an outline of a method to detect movement vectors in accordance with an embodiment;

FIG. 31 is an explanatory diagram comparing the characteristic of the movement-vector detection method according to the embodiment with that of the method in the related art;

FIG. 32 is an explanatory diagram comparing the characteristic of the movement-vector detection method according to the embodiment with that of the method in the related art;

FIG. 33 is an explanatory diagram comparing the characteristic of the movement-vector detection method according to the embodiment with that of the method in the related art;

FIG. 34 shows a flowchart to be referred to in explanation of processing carried out to detect a movement vector in accordance with a first typical implementation in the image taking apparatus according to the first embodiment;

FIG. 35 shows the continuation of the flowchart to be referred to in explanation of the processing carried out to detect a movement vector in accordance with the first typical implementation in the image taking apparatus according to the first embodiment;

FIG. 36 shows a flowchart to be referred to in explanation of processing carried out to detect a movement vector in accordance with a second typical implementation in the image taking apparatus according to the first embodiment;

FIG. 37 shows the continuation of the flowchart to be referred to in explanation of the processing carried out to detect a movement vector in accordance with the second typical implementation in the image taking apparatus according to the first embodiment;

FIG. 38 shows a flowchart to be referred to in explanation of processing carried out to detect a movement vector in accordance with a third typical implementation in the image taking apparatus according to the first embodiment;

FIG. 39 shows a continuation of the flowchart to be referred to in explanation of the processing carried out to detect a movement vector in accordance with the third typical implementation in the image taking apparatus according to the first embodiment;

FIG. 40 shows another continuation of the flowchart to be referred to in explanation of the processing carried out to detect a movement vector in accordance with the third typical implementation in the image taking apparatus according to the first embodiment;

FIG. 41 shows a further continuation of the flowchart to be referred to in explanation of the processing carried out to detect a movement vector in accordance with the third typical implementation in the image taking apparatus according to the first embodiment;

FIG. 42 shows a flowchart to be referred to in explanation of entire processing operations carried out by an image taking apparatus according to the first embodiment;

FIG. 43 is a block diagram showing a typical configuration of an image taking apparatus adopting a method to compensate an image signal for distortions in accordance with a second embodiment;

FIG. 44 is an explanatory diagram to be referred to in description of processing to detect a movement vector in the image taking apparatus according to the second embodiment;

FIG. 45 is an explanatory diagram to be referred to in description of processing to detect a movement vector in the image taking apparatus according to the second embodiment;

FIG. 46 shows a flowchart to be referred to in explanation of the processing carried out to detect a movement vector in the image taking apparatus according to the second embodiment;

FIG. 47 shows the continuation of the flowchart to be referred to in explanation of the processing carried out to detect a movement vector in the image taking apparatus according to the second embodiment;

FIGS. 48A and 48B are each an explanatory diagram to be referred to in description of an effect caused by a hand movement as an effect on an imager-created image;

FIG. 49 is an explanatory diagram to be referred to in describing a movement of an imager-created image obtained as a result of an optical zoom operation; and

FIG. 50 is an explanatory diagram to be referred to in describing a movement of an imager-created image obtained as a result of an optical zoom operation accompanied by a hand movement.

Claims

1. A method of compensating an imager-created image for a distortion, said method comprising: a movement-vector detection process of receiving data of said imager-created image from an image taking device and detecting a movement vector for each screen of said imager-created image;an optical-zoom portion vector computation process of computing an optical-zoom portion vector corresponding to an image movement occurring on said imager-created image as a result of an optical zoom operation;an optical-zoom portion vector subtraction process of subtracting said optical-zoom portion vector computed in said optical-zoom portion vector computation process from said movement vector detected in said movement-vector detection process;a global hand-movement displacement vector computation process of computing a global hand-movement displacement vector for said imager-created image from said movement vector detected in said movement-vector detection process in the case of no optical zoom operation and computing a global hand-movement displacement vector for said imager-created image from a difference vector output from said optical-zoom portion vector subtraction process as a result of said vector subtraction in the case of an optical zoom operation; anda hand-movement compensation process of compensating said imager-created image for a distortion caused by a hand movement on the basis of said global hand-movement displacement vector computed in said global hand-movement displacement vector computation process.

2. A method of compensating an imager-created image for a distortion in accordance with claim 1, whereby in said movement-vector detection process, a plurality of movement vector is detected for each of a plurality of picture divisions obtained by dividing said imager-created image in horizontal and vertical directions;in said optical-zoom portion vector computation process, an optical-zoom portion vector is computed for each of said plurality of picture divisions; andin said optical-zoom portion vector subtraction process, each one of said optical-zoom portion vectors computed in said optical-zoom portion vector computation process is subtracted from one of said movement vectors each detected for one of said picture divisions in said movement-vector detection process.

3. A method of compensating an imager-created image for a distortion in accordance with claim 2, whereby said optical-zoom portion vector is computed in said optical-zoom portion vector computation process for any specific one of said picture divisions from at least a period of time, which is the length of time lapsing since the start of said optical zoom operation, a zoom velocity and a distance between the center position of said specific picture division and said imager-created image.

4. A method of compensating an imager-created image for a distortion in accordance with claim 2, wherein said movement-vector detection process comprises the steps of: detecting information on a difference in image between a particular one of said picture divisions on an observed screen serving as a reference screen and said particular picture division on an original screen used as a target screen leading ahead of said reference screen;setting at least a target block having a predetermined size equal to the size of a plurality of pixels at predetermined locations in each of said picture divisions on said original screen;setting a plurality of reference blocks each having a size equal to said predetermined size of said target block in a search range set in said observed screen;searching said plurality of search range for a reference block having a strongest correlation with said specific target block; anddetecting said movement vector representing the magnitude and direction of a shift over said reference screen from a position corresponding to said specific target block to the position of said detected reference block.

5. A method of compensating an imager-created image for a distortion in accordance with claim 4, wherein said step of detecting a movement vector in said movement-vector detection process for each of said picture divisions is executed by carrying out:a difference absolute value sum computation process for each specific one of said reference blocks as a process of finding a sum of the absolute values of differences in pixel value between all pixels in said specific reference block and all pixels at corresponding positions on said specific target block;a contracted reference-vector setting process of setting reference vectors each representing the magnitude and direction of a shift over said reference screen from a position corresponding to said specific target block to the position of one of said reference blocks and contracting said reference vectors at a predetermined contraction factor to result in contracted reference vectors;a sum-of-absolute-difference table creation process of creating a shrunk sum-of-absolute-difference table including table elements, the number of which is determined on the basis of the number of said contracted reference vectors and said contraction factor, and allocating each of said table elements as a location to be used for storing a value computed from sums each found for one of said reference blocks as said sum of the absolute values of said differences in pixel values; anda movement-vector computation process of computing said movement vector for each of said picture divisions as a movement vector between said reference screen and said original screen by making use of at least a contracted reference vector corresponding to a smallest one among values each stored in said shrunk sum-of-absolute-difference table as said sum of the absolute values of said differences in pixel values; andsaid sum-of-absolute-difference table creation process is carried out by execution ofa neighborhood reference vector detection sub-process of detecting a plurality of neighborhood reference vectors each having a vector quantity close to the vector quantity of a corresponding one of said contracted reference vectors obtained in said contracted reference-vector setting process,a sum-component computation sub-process of computing a plurality of sum components each associated with one of neighborhood reference vectors determined in said neighborhood reference vector detection sub-process as said neighborhood reference vectors corresponding to a specific one of said contracted reference vectors from a sum computed in said difference absolute value sum computation process for a reference block pointed to by a reference vector, from which said specific contracted reference vector is contracted, as said sum of the absolute values of said differences in pixel values, anda component-total computation sub-process of computing a total of sum components, which have been computed in said sum-component computation sub-process as said sum components each associated with a specific one of said neighborhood reference vectors, by cumulatively adding each of said sum components to a cumulative total obtained so far for said specific neighborhood reference vector.

6. A method of compensating an imager-created image for a distortion in accordance with claim 4, whereby said step of detecting a movement vector in said movement-vector detection process for each of said picture divisions is executed by carrying out the sub-steps of:setting at least a target block having a predetermined size equal to the size of a plurality of pixels at predetermined locations in each of said picture divisions on said original screen;setting a plurality of reference blocks each having a size equal to said predetermined size of said target block in a search range set in said reference screen;searching said search range for a reference block having a strongest correlation with said specific target block; anddetecting said movement vector representing the magnitude and direction of a shift over said reference screen from a position corresponding to said specific target block to the position of said detected reference block;said sub-step of detecting said movement vector is executed by carrying outa difference absolute value sum computation process for each specific one of said reference blocks as a process of finding a sum of the absolute values of differences in pixel value between all pixels in said specific reference block and all pixels at corresponding positions on said specific target block,a contracted reference-vector setting process of setting reference vectors each representing the magnitude and direction of a shift over said reference screen from a position corresponding to said specific target block to the position of one of said reference blocks and contracting said reference vectors at a predetermined contraction factor to result in contracted reference vectors,a sum-of-absolute-difference table creation process of creating a shrunk sum-of-absolute-difference table including table elements, the number of which is determined on the basis of the number of said contracted reference vectors and said contraction factor, and allocating each of said table elements as a location to be used for storing a value computed from sums each found for one of said reference blocks as said sum of the absolute values of said differences in pixel values, anda movement-vector computation process of computing said movement vector for each of said picture divisions as a movement vector between said reference screen and said original screen by making use of at least a contracted reference vector corresponding to a smallest one among values each stored in said shrunk sum-of-absolute-difference table as said sum of the absolute values of said differences in pixel values, andsaid sum-of-absolute-difference table creation process is carried out by execution ofa neighborhood reference vector detection sub-process of detecting a plurality of neighborhood reference vectors each having a vector quantity close to the vector quantity of a corresponding one of said contracted reference vectors obtained in said contracted reference-vector setting process,a sum-component computation sub-process of computing a plurality of sum components each associated with one of neighborhood reference vectors determined in said neighborhood reference vector detection sub-process as said neighborhood reference vectors corresponding to a specific one of said contracted reference vectors from a sum computed in said difference absolute value sum computation process for a reference block pointed to by a reference vector, from which said specific contracted reference vector is contracted, as said sum of the absolute values of said differences in pixel values, anda component-total computation sub-process of computing a total of sum components, which have been computed in said sum-component computation sub-process as said sum components each associated with a specific one of said neighborhood reference vectors, by cumulatively adding each of said sum components to a cumulative total obtained so far for said specific neighborhood reference vector.

7. An image taking method to be adopted by an image taking apparatus having an optical zoom operation unit to serve as a method of compensating an imager-created image for a distortion due to a positional change caused by a movement made by a hand of a photographer operating said image taking apparatus during a photographing operation as a positional change of an image taking device employed in said image taking apparatus as well as serve as a method of recording information of said compensated imager-created image onto a recording medium, said method comprising: a movement-vector detection process of driving movement-vector detection means employed in said image taking apparatus to receive data of said imager-created image from said image taking device and detect a movement vector for each screen of said imager-created image;a zoom-operation determination process of driving zoom-operation determination means employed in said image taking apparatus to produce a result of determination as to whether or not an optical zoom operation has been carried out by said photographer by operating an optical zoom operation unit employed in said image taking apparatus;an optical-zoom portion vector computation process of driving optical-zoom portion vector computation means employed in said image taking apparatus to compute an optical-zoom portion vector corresponding to a positional change of said image taking device creating said imager-created image as a result of said optical zoom operation;an optical-zoom portion vector subtraction process of driving optical-zoom portion vector subtraction means employed in said image taking apparatus to subtract said optical-zoom portion vector computed in said optical-zoom portion vector computation process from said movement vector detected in said movement-vector detection process if said determination result produced by said zoom-operation determination means indicates that an optical zoom operation has been carried out;a global hand-movement displacement vector computation process of driving global hand-movement displacement vector computation means employed in said image taking apparatus to compute a global hand-movement displacement vector for said imager-created image from said movement vector detected in said movement-vector detection process if said determination result produced by said zoom-operation determination means indicates that no optical zoom operation has been carried out and compute a global hand-movement displacement vector for said imager-created image from a difference vector output from said optical-zoom portion vector subtraction process as a result of said vector subtraction if said determination result produced by said zoom-operation determination means indicates that an optical zoom operation has been carried out;a hand-movement compensation process of driving hand-movement compensation means employed in said image taking apparatus to compensate said imager-created image for a distortion caused by a hand movement on the basis of said global hand-movement displacement vector computed in said global hand-movement displacement vector computation process; andan information recording process of driving information recording means employed in said image taking apparatus to record information of said imager-created image compensated in said hand-movement compensation process onto said recording medium.

8. A method of compensating an imager-created image for a distortion in accordance with claim 7, whereby in said movement-vector detection process, a movement vector is detected for each of a plurality of picture divisions obtained by dividing said imager-created image in horizontal and vertical directions;in said optical-zoom portion vector computation process, an optical-zoom portion vector is computed for each of said plurality of picture divisions in said movement-vector detection process; andin said optical-zoom portion vector subtraction process, each one of said optical-zoom portion vectors computed in said optical-zoom portion vector computation process is subtracted from one of said movement vectors each detected for one of said picture divisions in said movement-vector detection process.

9. A method of compensating an imager-created image for a distortion in accordance with claim 8, whereby said optical-zoom portion vector is computed in said optical-zoom portion vector computation process for any specific one of said picture divisions from at least a period of time, which is the length of time lapsing since the start of said optical zoom operation, a zoom velocity and a distance between the center position of said specific picture division and said imager-created image.

10. An apparatus for compensating an imager-created image for a distortion, said apparatus comprising: movement-vector detection means for receiving data of said imager-created image from an image taking device and detecting a movement vector for each screen of said imager-created image;optical-zoom portion vector computation means for computing an optical-zoom portion vector corresponding to an image movement occurring on said imager-created image as a result of an optical zoom operation;optical-zoom portion vector subtraction means for subtracting said optical-zoom portion vector computed by said optical-zoom portion vector computation means from said movement vector detected by said movement-vector detection means;global hand-movement displacement vector computation means for computing a global hand-movement displacement vector for said imager-created image from said movement vector detected by said movement-vector detection means in the case of no optical zoom operation and computing a global hand-movement displacement vector for said imager-created image from a difference vector output by said optical-zoom portion vector subtraction means as a result of said vector subtraction in the case of an optical zoom operation; andhand-movement compensation means for compensating said imager-created image for a distortion caused by a hand movement on the basis of said global hand-movement displacement vector computed by said global hand-movement displacement vector computation means.

11. An apparatus for compensating an imager-created image for a distortion in accordance with claim 10, wherein said movement-vector detection means detects said plurality of movement vector for each of a plurality of picture divisions obtained by dividing said imager-created image in horizontal and vertical directions;said optical-zoom portion vector computation means computes said optical-zoom portion vector for each of said plurality of picture divisions; andsaid optical-zoom portion vector subtraction means subtracts each one of said optical-zoom portion vectors computed by said optical-zoom portion vector computation means from one of said movement vectors each detected for one of said picture divisions by said movement-vector detection means.

12. An apparatus for compensating an imager-created image for a distortion in accordance with claim 11, wherein said optical-zoom portion vector computation means computes said optical-zoom portion vector for any specific one of said picture divisions from at least a period of time, which is the length of time lapsing since the start of said optical zoom operation, a zoom velocity and a distance between the center position of said specific picture division and said imager-created image.

13. An image taking apparatus comprising: an image taking device;an optical zoom operation unit;optical-zoom driving means for carrying out an optical zoom process by controlling an optical lens system in accordance with said optical zoom operation carried out on said optical zoom operation unit;movement-vector detection means for receiving data of said imager-created image from said image taking device and detecting a movement vector for each screen of said imager-created image;zoom-operation determination means for producing a result of determination as to whether or not an optical zoom operation has been carried out by said photographer by operating said optical zoom operation means;optical-zoom portion vector computation means for computing an optical-zoom portion vector corresponding to a positional change of said image taking device as a result of said optical zoom operation;optical-zoom portion vector subtraction means for subtracting said optical-zoom portion vector computed by said optical-zoom portion vector computation means from said movement vector detected by said movement-vector detection means if said determination result produced by said zoom-operation determination means indicates that an optical zoom operation has been carried out;global hand-movement displacement vector computation means for computing a global hand-movement displacement vector for said imager-created image from said movement vector detected by said movement-vector detection means if said determination result produced by said zoom-operation determination means indicates that no optical zoom operation has been carried out and computing a global hand-movement displacement vector for said imager-created image from a difference vector output by said optical-zoom portion vector subtraction means as a result of said vector subtraction if said determination result produced by said zoom-operation determination means indicates that an optical zoom operation has been carried out;hand-movement compensation means for compensating said imager-created image for a distortion caused by a hand movement on the basis of said global hand-movement displacement vector computed by said global hand-movement displacement vector computation means; andinformation recording means for recording information of said imager-created image compensated by said hand-movement compensation means onto said recording medium.

14. The image taking apparatus according to claim 13, wherein said movement-vector detection means detects a plurality of movement vector for each of a plurality of picture divisions obtained by dividing said imager-created image in horizontal and vertical directions;said optical-zoom portion vector computation means computes said optical-zoom portion vector for each of said plurality of picture divisions; andsaid optical-zoom portion vector subtraction means subtracts each one of said optical-zoom portion vectors computed by said optical-zoom portion vector computation means from one of said movement vectors each detected for one of said picture divisions by said movement-vector detection means.

15. The image taking apparatus according to claim 14, wherein said optical-zoom portion vector computation means computes said optical-zoom portion vector for any specific one of said picture divisions from at least a period of time, which is the length of time lapsing since the start of said optical zoom operation, a zoom velocity and a distance between the center position of said specific picture division and said imager-created image.

16. An apparatus for compensating an imager-created image for a distortion, said apparatus comprising: a movement-vector detection unit configured to receive data of said imager-created image from an image taking device and detect a movement vector for each screen of said imager-created image;an optical-zoom portion vector computation unit configured to compute an optical-zoom portion vector corresponding to an image movement occurring on said imager-created image as a result of an optical zoom operation;an optical-zoom portion vector subtraction unit configured to subtract said optical-zoom portion vector computed by said optical-zoom portion vector computation unit from said movement vector detected by said movement-vector detection unit;a global hand-movement displacement vector computation unit configured to compute a global hand-movement displacement vector for said imager-created image from said movement vector detected by said movement-vector detection unit in the case of no optical zoom operation and compute a global hand-movement displacement vector for said imager-created image from a difference vector output by said optical-zoom portion vector subtraction unit as a result of said vector subtraction in the case of an optical zoom operation; anda hand-movement compensation unit configured to compensate said imager-created image for a distortion caused by a hand movement on the basis of said global hand-movement displacement vector computed by said global hand-movement displacement vector computation unit.