1. Field of the Invention
The present invention relates to an image composer, and more specifically to an image composer for combining two or more images to produce a composite image.
2. Description of the Background Art
Conventionally, in order to produce a composite image, two or more photographs are taken, for example, by a camera supported on a tripod stand or the like and its self-timer or remote control function activated and then recorded in the form of original photographs to be combined together. As examples of method for producing a composite image, one method is of physically combining such two or more photographs with paste, and another method is of loading the image data of plural photographs on a personal computer to produce a single frame of composite image through manual operation on the display screen of the computer. In these methods using plural photographs obtained with a camera supported on a tripod, the background scenes such as a field view, buildings, etc., are generally not changed between the original photographs, which are therefore can be conveniently used to produce a composite image. It is, however, a burden or time-consuming to carry the tripod stand and attach the camera to the stand at a photographing site.
A recent development on digitalization of photographing makes it easier to produce composite photographs by using a personal computer. In U.S. patent application publication No. US 2002/0030634 A1 to Noda et al., for example, a method is disclosed for using on a personal computer digital images of, e.g. a background scene and a subject to selectively cut out a required part of a digital image and paste that part onto another digital image to thereby produce a resultant composite image.
However, the operation according to Noda et al., is potentially troublesome because the selection of trimming a range, i.e. cutting off a part, of the image and the determination of a position on which that part is to be pasted must be performed by manual operation. Further, when the images are different in scale or one image is rotated with respect to the other, the part needs to be enlarged or reduced, or rotated before pasted, the operation will be more troublesome.
It is an object of the present invention to provide an image composer that is capable of readily producing a composite image.
In accordance with the present invention, a partial region of an image not common to another image is extracted therefrom for producing a composite image. More specifically, an image composer according to the invention comprises: a storage for storing data of a first and a second images; a region classifier for comparing the first and second images with each other, and extracting as a non-common sub-region a partial region of one of the images which region is not similar in its pixel property to the corresponding region of the other image; and an image composing unit for superimposing the non-common sub-region of the image on the other image to produce a composite image.
Thus, the region classifier compares the first image and second image read out from the storage, thereby extracting as a non-common sub-region a partial region of the image which region is not similar in its pixel property to the corresponding region of the other image. The image composing unit is able to produce a composite image by superimposing the extracted non-common sub-region of the image on the other image.
First and second images preferably contain a common subject in a part of the region thereof. In such a case, the image composing unit decides, based on the position of the common subject, a position in the other image where the non-common sub-region will be superimposed.
Thus, the image composing unit is able to decide, based on the position of the common subject, a position in the other image where the non-common sub-region of the one image is superimposed, and produce a composite image.
According to the present invention, it is possible to produce a composite image readily by extracting a non-common sub-region from two or more images.
The objects and features of the present invention will become more apparent from consideration of the following detailed description taken in conjunction with the accompanying drawings in which:
Now, preferred embodiments of an image composer according to the present invention will be described in detail with reference to the accompanying drawings. For the purpose of making the present invention understood easier, the description will specifically be made on the case where two original images are combined with each other to form a resultant single image. In that case, one of the two original images which is used as a base for such a resultant single image will hereinafter be referred to as a base image, while the other image part of which not common to at least part of the base image is extracted to be superimposed or pasted onto the base image will hereinafter be referred to as a superimposition image. Further, the single image produced from the two original images will hereinafter be referred to as a composite image.
Referring initially to
The storage 2 is adapted for storing various information in the form of digital data, and may be implemented by, e.g. a read-only memory (ROM) and a hard-disk drive (HDD). Information to be stored in the storage 2 may be operational program sequences runnable on the processor 5 and various kinds of image data including base image data and superimposition image data, and so forth.
The display 3 is adapted for visualizing and displaying information, and may be implemented by a liquid crystal display or an electro-luminescence display, for example. A variety of image data and other pertinent data are displayed on the display 3 under the control of the processor 5.
The input unit 4 is adapted for entering information to the image composer 1, and may include a keyboard and a pointing device such as a mouse. The input unit 4 is used to feed information that was input by the user to the processor 5.
The processor 5 is adapted for controlling the entire operations that are to be performed in the image composer 1, and may include a central processing unit (CPU), a random access memory (RAM), for example. The processor 5 includes operational functions represented by several functional blocks such as an image input unit 51, a feature extractor 52, a feature checker 53, a region classifier 54, an image composing unit 55, and a display controller 56. Those units 51 to 56 will be described briefly here, and a detailed description of them will be given later with reference to
The image input unit 51 is adapted to be responsive to, for example, an image compositing instruction that was input through the input unit 4 by the user to read out data representative of a base image and a superimposition image 300 from the storage 2. Signals are designated with reference numerals designating connections on which they are conveyed.
The feature extractor 52 is adapted to receive the data of base and superimposition images from the image input unit 51, and then extract their feature points and feature patterns, i.e. pixel properties. In this embodiment, the term “feature point” is used to represent a specific part of an image such as edges and corners of subjects in the image, while the term “feature pattern” is used to represent properties of the feature points. A detailed description of them is to be described later.
The feature checker 53 is adapted to compare each of the feature points of a base image with the feature points of a superimposition image based on the feature pattern of that feature point. When similarity in feature pattern substantially equal to or over a predetermined threshold is found between a feature point of the base image and that of the superimposition image, both of the points thus found are determined as a pair of points corresponding with each other. The value of the threshold may be determined depending upon purposes or accuracy.
The region classifier 54 is adapted to sort or sub-divide the region of a superimposition image into a non-common sub-region and a common sub-region. The non-common sub-region, in the superimposition image, refers to a sub-region that is occupied by an object not present in the base image, i.e. a partial region having feature patterns whose similarity to those in the corresponding sub-region of the base image is lower than the predetermined threshold. The common sub-region, in the superimposition image, means a sub-region other than the non-common sub-region. The region classifier 54 may be adapted to decide the non-common sub-region, for instance, by a rectangle circumscribing a partial set of feature points not adopted as a pair of corresponding points such as to contain the set therein. A partial set of feature points denotes a set of plural feature points of one object, such as a person or an automobile, contained in an image. One superimposition image may contain one or more partial sets of features.
The image composing unit 55 is adapted to overlay or superimpose, i.e. paste, a non-common sub-region of the superimposition image onto the base image to produce a composite image in the form of digital data.
The display controller 56 is adapted to provide the display 3 with an instruction to display a composite image 302 produced by the composing unit 55. The display 3 may be of the type provided outside the composer 1, and in that case the display controller 56 may be adapted to control such an external display so as to cause the composite image 302 to be displayed on the external display.
Now, processing steps in the instant embodiment will be described by referring to
The present embodiment is specifically adapted to define the feature point of an image as, for example, the point detected as the position of the maxima of a Harris operation element of the image and defining the edge or boundary of an object in the image. For a more detailed discussion on the Harris operation element, see, for example, C. Harris et al., “A combined Corner and Edge Detector,” Proc., 4th Alvey Vision Conf., pp. 147-151, 1988, cited as merely teaching the general background art.
The instant embodiment is adapted to define a feature pattern as a differential luminance value, in the form of differential luminance vector, of the portion surrounding a feature point, the differential luminance value not being affected in its nature by the scale value and orientation of the image. For a more detailed discussion on the differential luminance value of the portion surrounding a feature point, see, for example, C. Schumid et al., “Local Greyvalue Invariants for Image Retriever,” IEEE Trans., PAMI, Vol. 19, No. 5, pp. 530-535, 1997, also cited here as merely teaching the general background art.
Besides, the feature pattern may be of scalar values, such as pixel values or the average value thereof, calculated from one or more pixels, other kinds of vector, e.g. luminance vectors, and so on.
Below, a description will be given of the processing steps in the image composer 1 that are executed when the user manipulates the input unit 4 to instruct the composer 1 to produce a composite image. In operation, the digital data representative of a base image and a superimposition image have been stored in the storage 2 in advance.
Initially, the image input unit 51 of the processor feeds the base image, i.e. the image A shown in
The image input unit 51 then feeds the processor 5 with the superimposition image, i.e. the image B shown in
Subsequently, the feature extractor 52 extracts feature points fa, which are represented by the crosses in
More specifically, the group Fa of the feature points fa can be represented as a set of plurality (m) of feature points fa1 to fam, where m is a natural number, by the following Expression (1):
Fa={fa1,fa2, . . . , fam} (1)
The kth feature fak can be represented by the following Expression (2), where k is a natural number not more than m:
fak={xak,yak,sak,θak,vak} (2)
in which x is an x-coordinate value; y is a y-coordinate value; s is a scale value, which represents the scale or reduction ratio of an image, a relative value to a predetermined reference value, for example; and θ is an angle of rotation with respect to a reference direction extending from a predetermined reference point. Note that a subscript, such as “ak”, etc., of a parameter indicates the correspondence with a feature point, such as “Fak”, etc.
The feature pattern va can be expressed as a p-dimensional vector, as follows:
vak={vak(1),vak(2), . . . , vak(p)} (3)
where p is also a natural number representing the dimension.
The feature extractor 52 also extracts feature points fb represented by the crosses in
Fb={fb1,fb2, . . . , fbn} (4)
fbk={xbk,ybk,sbk,θbk,vbk} (5)
vbk{vbk(1),vbk(2), . . . , vbk(p)} (6)
in which the number of the feature points fb is n.
Next, the feature checker 53 puts the individual feature points of the image A and those of the image B in correspondence (step S250). More specifically, the feature checker 53 determines whether or not there is a feature point {fai} of the base image A corresponding to a feature point {fbj} of the superimposition image B. An example of the method of determination is of calculating the square of the Euclidean distance between features by the following Expression (7).
With respect to a certain value j, if i is such a value that D(i, j) is within a predetermined threshold value and is the minimum value of D(i, j), it can be determined that the feature point {fai} corresponds to the feature point {fbj}. Also, with respect to a certain value j, when there is no value i satisfying the condition that D(i, j) exceeds the predetermined threshold value, the feature point {fbj} is determined as a point having no corresponding feature point on the base image. When there is no corresponding point pair, an object that is not present in a base image is included in a superimposition image. In both the case where there is a corresponding point pair and the case where there is no corresponding point pair, the feature checker 53 stores the result of determination in the storage 2.
Using the result obtained in step S250, the region classifier 54 sorts the region of the superimposition image B into a non-common sub-region and a common sub-region, which are referred to as NC2 and C2 in
On the other hand, the common sub-region is a sub-region considered to be the background for a composite image and can be decided, for example, as a sub-region other than the non-common sub-region.
Subsequently, the image composing unit 55 overlays or superimposes the non-common sub-region obtained in step S260 at its corresponding position in the base image to produce a composite image (step S270). Examples of image overlaying methods include a method of replacing a luminance value, a method of using the average luminance value between a superimposition image and a base image, and so forth.
The display controller 56 instructs the display 3 to display the composite image produced in step S270 (step S280). In response to the instruction, the display 3 displays the composite image on its display screen.
Thus, the image composer 1 of the present embodiment automatically decides a part of a superimposition image that will be overlaid or superimposed on a base image. Therefore the user is able to create a composite image readily by simply inputting base and superimposition images photographed at the same location and containing a subject common to both of them.
Now, an alternative embodiment of the present invention will be described in detail. This alternative embodiment does not need to use a base image and a superimposition image photographed at nearly the same orientation and scale value. More specifically, in the alternative embodiment, a common subject in a superimposition image may be different, i.e. shifted, in position, or in orientation, or in scale with respect to corresponding one in a base image. In addition, the alternative embodiment is adapted to transform, such as move, rotate, zoom or combine, a superimposition image rather than a base image. However, the system may be adapted to transform a base image without transforming a superimposition image fixed.
The transformation detector 531, based on the result of checking in the feature checker 53, is adapted to calculate image transformation parameters for transforming the superimposition image as required in such a way that corresponding feature points, in pair, of the base and superimposition images are overlaid with each other.
The image transformer 532 is adapted to use the image transformation parameters calculated by the transformation detector 531 to transform the superimposition image accordingly.
Now, processing steps in the alternative embodiment will be described with reference to
Well, a description will be given of the processing steps in the image composer 1a which are carried out when the user manipulates the input unit 4 to produce a composite image. In operation, the digital data of a base image and a superimposition image have already been stored in the storage 2.
As previously stated, unlike the case of the embodiment according to the image composer 1, the superimposition image may be shifted from the base image in its corresponding position, orientation and scale.
Initially, the image input unit 51 feeds the processor 5 with the digital data of the base image C or E shown in
Subsequent steps S630 to S650 are the same as the aforementioned steps S230 to S250,
After step S650, the transformation detector 531 calculates image transformation parameters employing the coordinate values of a corresponding point pair (step S651). More specifically, the transformation detector 531, based on the result of checking in step S650, calculates image transformation parameters for transforming the superimposition image in such a manner that corresponding points of the base and superimposition images are overlaid with each other.
For example, as shown in
Δx{Σr=1t(xbr−xar)}/t={Σr=1tΔxr}/t (8)
in which t represents the number of corresponding point pairs.
In
Likewise, when the superimposition image are shifted only in a direction of y-axis from its corresponding position in the base image, the transformation detector 531 substitutes into the following Expression (9) the information about the y coordinate points (yai and ybi) of the corresponding point pair to calculate an image transformation parameter, i.e. Δy in this case.
Δy={Σ
r=1
t(ybr−yar)}/t={Σr=1tΔyr}/t (9)
When the base and superimposition images are different only in scale, the transformation detector 531 substitutes into the following Expression (10) the information about the scale values (sai and sbi) of the corresponding point pair to calculate an image transformation parameter, i.e. Δs in this case.
Δs={Σ
r=1
t(sbr/sar)}/t={Σr=1tΔsr}/t (10)
When the base and superimposition images are different only in orientation, the transformation detector 531 substitutes into the following Expression (11) the information about the amount of orientation (θai and θbi) of the corresponding point pair to calculate an image transformation parameter, i.e. Δθ in this case.
Δθ={Σr=1tθbr−θar)}/t={Σr=1tΔθr}/t (11)
As with the example shown in
(i) creating disjoint categories of candidates of a target value in ΔX-ΔY-ΔΘ space, i.e. a value to be specified, which categories are known as bins,
(ii) calculating values (ΔXr, Δyr, Δsr, Δθr; r=1 . . . t) of pairs by the expressions such as Expressions (8) to (11),
(iii) classifying the values into bins which the values belong to, i.e. voting for the bins, and
(iv) specifying a bin to which the most values belong and adopt an appropriate value in this bin, e.g. median of the bin, as the target value. In a case where the voting process is applied to plural image transformation parameters, the values of the parameters may be calculated, for example, in such an order of scale difference (Δs), orientation difference (Δθ) and positional difference (Δx, Δy) as the accuracy becomes higher.
In
Referring to
The region classifier 54, as with step S260, classifies the region of the superimposition image into a non-common sub-region NC6 and a common sub-region, employing the result of determination obtained in step S650 (step S660), see
Subsequently, the image composing unit 55, as in step S270, overlays or pastes the non-common sub-region obtained in step S660 at the corresponding position in the base image to produce a composite image (step S670). By transforming the superimposition image, its non-common sub-region is shifted outside the corresponding region of the base image, and the non-common sub-region may be forced to be moved, or reduced, so that it is included in the corresponding region of the base image, or that effect may be given to the user by being displayed on the display 3.
The display controller 56 instructs the display 3 to display the composite image produced in step S670, corresponding to step S280. After receiving the instruction, the display 3 displays the composite image on its screen. For instance, as shown in
Thus, the image composer 1a in
While the image transformer 532 is arranged between the transformation detector 531 and the region classifier 54 with the instant alternative embodiment, it may alternatively be arranged between the region classifier 54 and the image composing unit 55, and even in this case, the same advantages are obtainable. In addition, although the entire superimposition image is transformed in this embodiment, the system may be adapted so that only its non-common sub-region may be transformed, and even that case is able to obtain the same composite image.
In the two illustrative embodiments described above, while the common subject is a house, or more broadly a building, to the base and superimposition images, the present invention is applicable to many types of common subjects so long as the image composer 1 or 1a is adapted to extract its feature points and its feature patterns. Examples are persons (or persons wearing the same clothe), vehicles such as automobiles and trains, or more broadly moving bodies, plants such as flowers and trees, animals, articles such as books and desks, and so on. Hereinafter, a description will be given of examples of common subjects other than buildings. The processing steps in the image composer 1 or 1a are practically the same as the two embodiments described above, and therefore a detailed description of the corresponding processing steps will not be given.
Referring to
The base image shown in
The base image contains an automobile C1 and a giraffe G, while the superimposition image contains an automobile C2 which is the same as the automobile C1, an elephant E1, and a mouse MS1. The image composer 1 or 1a can produce their composite image with respect to the same automobile C1, in which image there are the original giraffe G, a reduced elephant E2 and a reduced mouse MS2, simultaneously as shown in
Note that the above-described processing steps in the image composer 1 or 1a can be implemented by a program sequence for causing the central processing unit (CPU) of a computer to execute those steps.
While the present invention has been described with reference to the particular illustrative embodiments, it is not to be restricted by the embodiments. It is to be appreciated that those skilled in the art can change or modify the embodiments without departing from the scope and spirit of the present invention. For example, by performing all or part of each of the above-described processing steps more than once, the image composer 1 or 1a can produce a composite from three or more original images.
The image composer 1 or 1a, in addition to personal computers, can also be implemented by digital cameras, camera-built-in cellular phones, etc. This makes it possible to produce a composite image at the location where photographs were taken, and then confirm the contents thereof.
In the case where a building, which is common to more than two original images, have been photographed at slightly different angles, each superimposition image may be coordinate-transformed so that the feature points of the building in the base image coincide with those in the superimposition image, or the superimposition image may be coordinate-transformed into only a similar form. In this case, accompanying with the transformation of the common sub-region, the superimposition region, i.e. the non-common sub-region, may be coordinate-transformed into a dissimilar or similar form. If the superimposition region is transformed into similar form, even though the other region is coordinate-transformed into a dissimilar form, a person or another object in the superimposition region can be prevented from being undesirably transformed in accordance with the coordinate transformation.
It should be noted that superimposition regions are not limited to be rectangular in shape. Examples of the shapes may be triangular, pentagonal, hexagonal, circular, elliptical, starlike, heart-shaped, and so on. Finally, the hardware units, flowcharts, and other details given herein can be changed or modified without departing from the scope of the present invention hereinafter claimed.
The entire disclosure of Japanese patent application No. 2007-76035 filed on Mar. 23, 2007, including the specification, claims, accompanying drawings and abstract of the disclosure, is incorporated herein by reference in its entirety.
Number | Date | Country | Kind |
---|---|---|---|
2007-76035 | Mar 2007 | JP | national |