The present application claims foreign priority based on Japanese Patent Application No. 2009-273983, filed Dec. 1, 2009, the contents of which is incorporated herein by reference.
1. Field of the Invention
The present invention relates to an image processing technology for projectively transforming a multivalued image picked up by an imaging device to correct perspective distortion of the multivalued image.
2. Description of Related Art
Conventionally, in apparatuses for picking up an image of a test object with an imaging device to inspect the test object and detect a defect thereof by use of the picked-up multivalued image, a variety of perspective distortion has occurred in accordance with a positional relation between the test object and the imaging device, a configuration of a lens, and the like. In many apparatuses, the picked-up multivalued image is appropriately projectively transformed for preventing deterioration in accuracy of inspection on the test object, accuracy of defect detection, and the like, to correct perspective distortion included in the multivalued image. By appropriately performing projective transformation, perspective distortion that occurs in accordance with the positional relation between the test object and the imaging device can be corrected.
For example, in Japanese Unexamined Patent Publication No. 2006-074512, a rectangle being an external form of a pedestal whose shape is well known is regarded as a reference polygon, and the rectangle as the reference polygon is compared with a rectangle specified by a contour of a pedestal image, to obtain a projective correction parameter. Using the obtained projective correction parameter, a multivalued image picked up by a camera is projectively transformed for correction.
Generally, in the case of projectively transforming a multivalued image, a multivalued image before transformation and a multivalued image after transformation are different from each other in an area where pixels exist. Namely, even in an area where pixels exist in the multivalued image before transformation, pixels may not exist in the multivalued image after transformation, or the reverse case may also take place. Therefore, when an area where pixels do not exist in the multivalued image before transformation is projectively transformed, pixels of the corresponding area in the multivalued image after transformation are made black.
However, when the entire image after transformation is intended to be obtained in the conventional image processing apparatus, even in the case of the black pixels existing in the multivalued image after transformation, projective transformation itself is performed by conducting coordinate transformation calculation of every pixel point by point to find a portion of the multivalued image before transformation, to which each pixel in the multivalued image after transformation corresponds. There has thus been a problem in that it is difficult to reduce processing time, so as to complete image processing at high speed.
Further, in the case of a multivalued image with a distant background appearing therein other than the imaging object and large perspective distortion of the multivalued image of the imaging object, there has also been a problem in that an unnecessary multivalued image (virtual image), whose polarity based upon a predetermined condition for projective transformation is a reverse polarity, might be included other than an image as an object to be transformed by projective transformation.
The present invention relates to an image processing apparatus capable of reducing processing time and also suppressing appearance of a virtual image in the case of projectively transforming a multivalued image, an image processing method used in the image processing apparatus, and a computer program for causing a computer to execute processing in the image processing method.
According to one embodiment of the present invention, there is provided an image processing apparatus, which executes image processing on a multivalued image obtained by picking up an image of an imaging object with an imaging device, the apparatus including: a projective transformation parameter calculating device for calculating a predetermined projective transformation parameter for projectively transforming a multivalued image before image processing to correct perspective distortion; a projected area calculating device for calculating a projected area, obtained by projectively transforming an area where pixels of the multivalued image before image processing exist to an output area where pixels of a multivalued image after image processing exist, based upon the projective transformation parameter calculated by the projective transformation parameter calculating device; an effective area specifying device for specifying as an effective area an area in which the projected area calculated by the projected area calculating device is overlapped with the output area where the pixels of the multivalued image after image processing exist; and an image transforming device for performing coordinate transformation based upon the effective area specified by the effective area specifying device and the projective transformation parameter calculated by the projective transformation parameter calculating device, to generate the multivalued image after image processing from the multivalued image before image processing.
Further, according to another embodiment of the present invention, the image processing apparatus includes a shape information input accepting device for accepting an input of shape information that specifies shapes of the imaging object before image processing and after image processing on the multivalued image, wherein the projective transformation parameter calculating device calculates the projective transformation parameter based upon the shape information the input of which has been accepted by the shape information input accepting device.
Further, according to another embodiment of the present invention, the image processing apparatus includes an image displaying device for displaying the multivalued image, wherein the shape information input accepting device accepts the input of the shape information that specifies shapes of the imaging object before image processing and after image processing on the multivalued image before image processing displayed in the image displaying device.
Further, according to another embodiment of the present invention, in the image processing apparatus, the image displaying device is provided with a display switching device for switching and displaying the multivalued image before image processing, the multivalued image after image processing, and the shape information the input of which has been accepted by the shape information input accepting device.
Further, according to another embodiment of the present invention, the image processing apparatus includes a transformation target area setting device for setting as a transformation target area a predetermined area including specific pixels in the area where the pixels of the multivalued image before image processing exist, wherein the projected area calculating device calculates the projected area, obtained by projectively transforming the transformation target area set by the transformation target area setting device, based upon the projective transformation parameter calculated by the projective transformation parameter calculating device.
Further, according to another embodiment of the present invention, in the image processing apparatus, the transformation target area setting device sets the predetermined area including specific pixels as the transformation target area out of areas divided by a predetermined straight line defined based upon the projective transformation parameter.
Further, according to another embodiment of the present invention, in the image processing apparatus, the transformation target area setting device sets, as the transformation target area, a predetermined area including pixels that exist inside the shape of the imaging object, the shape being specified by the shape information the input of which has been accepted by the shape information input accepting device.
Further, according to another embodiment of the present invention, in the image processing apparatus, the transformation target area setting device has: a first intersection calculating device for calculating a first intersection at which a predetermined straight line defined based upon the projective transformation parameter is intersected with an outer circumferential line of the area where the pixels of the multivalued image before image processing exist; a second intersection calculating device for calculating a second intersection at which outer circumferential lines of the area including pixels that exist inside the shape of the imaging object are intersected with each other out of the areas divided by the predetermined straight line, the shape being specified by the shape information the input of which has been accepted by the shape information input accepting device; and a polygonal area calculating device for calculating a polygonal area formed by connecting the first intersections calculated by the first intersection calculating device and the second intersections calculated by the second intersection calculating device, and the polygonal area calculated by the polygonal area calculating device is set as the transformation target area.
Moreover, according to another embodiment of the present invention, there is provided an image processing method capable of executing image processing on a multivalued image obtained by picking up an image of an imaging object with an imaging device, the method including the steps of calculating a predetermined projective transformation parameter for projectively transforming a multivalued image before image processing to correct perspective distortion; calculating a projected area, obtained by projectively transforming an area where pixels of the multivalued image before image processing exist to an output area where pixels of a multivalued image after image processing exist, based upon the calculated projective transformation parameter; specifying as an effective area an area in which the calculated projected area is overlapped with the output area where the pixels of the multivalued image after image processing exist; and performing coordinate transformation based upon the specified effective area and the calculated projective transformation parameter, to generate the multivalued image after image processing from the multivalued image before image processing.
Further, according to another embodiment of the present invention, the image processing method includes a step of accepting an input of shape information that specifies shapes of the imaging object before image processing and after image processing on the multivalued image, and calculates the projective transformation parameter based upon the shape information the input of which has been accepted.
Further, according to another embodiment of the present invention, the image processing method accepts the input of the shape information that specifies shapes of the imaging object before image processing and after image processing on the displayed multivalued image before image processing.
Further, according to another embodiment of the present invention, the image processing method switches and displays the multivalued image before image processing, the multivalued image after image processing, and the shape information the input of which has been accepted by the shape information input accepting device.
Further, according to another embodiment of the present invention, the image processing method includes a step of setting as a transformation target area a predetermined area including specific pixels in the area where the pixels of the multivalued image before image processing exist, and calculates the projected area, obtained by projectively transforming the set transformation target area, based upon the calculated projective transformation parameter
Further, according to another embodiment of the present invention, the image processing method sets the predetermined area including specific pixels as the transformation target area out of areas divided by a predetermined straight line defined based upon the projective transformation parameter.
Further, according to another embodiment of the present invention, the image processing method sets, as the transformation target area, a predetermined area including pixels that exist inside the shape of the imaging object, the shape being specified by the shape information the input of which has been accepted.
Further, according to another embodiment of the present invention, the image processing method includes the steps of calculating a first intersection at which a predetermined straight line defined based upon the projective transformation parameter is intersected with an outer circumferential line of the area where the pixels of the multivalued image before image processing exist; calculating a second intersection at which outer circumferential lines of the area including pixels that exist inside the shape of the imaging object are intersected with each other out of the areas divided by the predetermined straight line, the shape being specified by the shape information the input of which has been accepted; calculating a polygonal area formed by connecting the calculated first intersections and the calculated second intersections; and setting the calculated polygonal area as the transformation target area.
Yet, according to another embodiment of the present invention, there is provided a computer program capable of causing an image processing apparatus to execute image processing on a multivalued image obtained by picking up an image of an imaging object with an imaging device, wherein the image processing apparatus is made to function as: a projective transformation parameter calculating device for calculating a predetermined projective transformation parameter for projectively transforming a multivalued image before image processing to correct perspective distortion; a projected area calculating device for calculating a projected area, obtained by projectively transforming an area where pixels of the multivalued image before image processing exist to an output area where pixels of a multivalued image after image processing exist, based upon the projective transformation parameter calculated by the projective transformation parameter calculating device; an effective area specifying device for specifying as an effective area an area in which the projected area calculated by the projected area calculating device is overlapped with the output area where the pixels of the multivalued image after image processing exist; and an image transforming device for performing coordinate transformation based upon the effective area specified by the effective area specifying device and the projective transformation parameter calculated by the projective transformation parameter calculating device, to generate the multivalued image after image processing from the multivalued image before image processing.
Further, according to another embodiment of the present invention, in the computer program, the image processing apparatus is made to function as a shape information input accepting device for accepting an input of shape information that specifies shapes of the imaging object before image processing and after image processing on the multivalued image, and the projective transformation parameter calculating device is made to function as a device for calculating the projective transformation parameter based upon the shape information the input of which has been accepted by the shape information input accepting device.
Further, according to another embodiment of the present invention, in the computer program, the shape information input accepting device is made to function as a device for accepting the input of the shape information that specifies shapes of the imaging object before image processing and after image processing on the displayed multivalued image before image processing.
Further, according to another embodiment of the present invention, in the computer program, the image processing apparatus is made to function as a display switching device for switching and displaying the multivalued image before image processing, the multivalued image after image processing, and the shape information the input of which has been accepted by the shape information input accepting device.
Further, according to another embodiment of the present invention, in the computer program, the image processing apparatus is made to function as a transformation target area setting device for setting as a transformation target area a predetermined area including specific pixels in the area where the pixels of the multivalued image before image processing exist, and the projected area calculating device is made to function as a device for calculating the projected area, obtained by projectively transforming the transformation target area set by the transformation target area setting device, based upon the projective transformation parameter calculated by the projective transformation parameter calculating device.
Further, according to another embodiment of the present invention, in the computer program, the transformation target area setting device is made to function as a device for setting the predetermined area including specific pixels as the transformation target area out of areas divided by a predetermined straight line defined based upon the projective transformation parameter.
Further, according to another embodiment of the present invention, in the computer program, the transformation target area setting device is made to function as a device for setting as the transformation target area a predetermined area including pixels that exist inside the shape of the imaging object, the shape being specified by the shape information the input of which has been accepted by the shape information input accepting device.
Further, according to another embodiment of the present invention, in the computer program, the transformation target area setting device is made to function as: a first intersection calculating device for calculating a first intersection at which a predetermined straight line defined based upon the projective transformation parameter is intersected with an outer circumferential line of the area where the pixels of the multivalued image before image processing exist; a second intersection calculating device for calculating a second intersection at which outer circumferential lines of the area including pixels that exist inside the shape of the imaging object are intersected with each other out of the areas divided by the predetermined straight line, the shape being specified by the shape information the input of which has been accepted by the shape information input accepting device; a polygonal area calculating device for calculating a polygonal area formed by connecting the first intersections calculated by the first intersection calculating device and the second intersections calculated by the second intersection calculating device; and a device for setting the polygonal area, calculated by the polygonal area calculating device, as the transformation target area.
In the embodiment of the present invention, based upon a calculated projective transformation parameter, a projected area is calculated which is obtained by projectively transforming an area where pixels of the multivalued image before image processing exist to an output area where pixels of a multivalued image after image processing exist, an area is specified as an effective area in which the calculated projected area is overlapped with the output area where the pixels of the multivalued image after image processing exist, and coordinate transformation is performed based upon the specified effective area and the calculated projective transformation parameter, to generate the multivalued image after image processing from the multivalued image before image processing. Thereby, it is not necessary to perform projective transformation on a multivalued image out of the effective area, and processing time can be reduced, so as to complete image processing at high speed.
Further, an input of shape information that specifies shapes of the imaging object before image processing and after image processing on the multivalued image is accepted, and the projective transformation parameter is calculated based upon the shape information the input of which has been accepted. It is thereby possible to projectively transform the multivalued image before image processing, so as to calculate with high accuracy a projective transformation parameter for correcting perspective distortion.
Further, the input of the shape information is accepted, the information specifying shapes of the imaging object before image processing and after image processing on the displayed multivalued image before image processing. It is thereby possible to specify with ease shapes of the imaging object before image processing and after image processing, while visually recognizing the multivalued image before image processing which is projectively transformed.
Further, the multivalued image before image processing, the multivalued image after image processing, and the shape information the input of which has been accepted are switched and displayed. It is thereby possible to visually recognize the shape information of the imaging object before image processing and after image processing.
Further, a predetermined area including specific pixels is set as a transformation target area in the area where the pixels of the multivalued image before image processing exist, and based upon the calculated projective transformation parameter, the projected area is calculated which is obtained by projectively transforming the multivalued image before image processing corresponding to the set transformation target area. It is thereby possible to suppress appearance of an unnecessary multivalued image (virtual image), whose polarity based upon a predetermined condition for projective transformation is a reverse polarity, in the multivalued image after image processing. Meanwhile, it is not necessary to perform projective transformation on the multivalued image before image processing which corresponds to the unnecessary multivalued image (virtual image), and processing time can be reduced, so as to complete image processing at high speed.
Further, the predetermined area including specific pixels are set as the transformation target area out of areas divided by a predetermined straight line defined based upon the projective transformation parameter. It is thereby possible to suppress appearance of an unnecessary multivalued image (virtual image), whose polarity based upon a predetermined condition for projective transformation is a reverse polarity, in the multivalued image after image processing.
Further, a predetermined area including pixels that exist inside the shape of the imaging object is set as the transformation target area, the shape being specified by the shape information the input of which has been accepted. It is thereby possible to set with ease a transformation target area including the imaging object.
Further, a first intersection is calculated at which a predetermined straight line defined based upon the projective transformation parameter is intersected with an outer circumferential line of the area where the pixels of the multivalued image before image processing exist, and a second intersection is calculated at which outer circumferential lines of the area including pixels that exist inside the shape of the imaging object are intersected with each other out of the areas divided by the predetermined straight line, the shape being specified by the shape information the input of which has been accepted. A polygonal area formed by connecting the calculated first intersections and the calculated second intersections is calculated, and the calculated polygonal area is set as the transformation target area. It is thereby possible to reliably suppress appearance of an unnecessary multivalued image (virtual image) in the multivalued image after image processing.
According to the above configuration, a projected area is calculated, which is obtained by projectively transforming an area where pixels of a multivalued image before image processing exist to an output area where pixels of a multivalued image after image processing exist, an area is specified as a effective area in which the calculated projected area is overlapped with the output area where the pixels of the multivalued image after image processing exist, and coordinate transformation is performed based upon the specified effective area and a projective transformation parameter calculated by a projective transformation parameter calculating device, to generate the multivalued image after image processing from the multivalued image before image processing. Thereby, it is not necessary to perform projective transformation on a multivalued image in an area other than an effective area, and processing time can be reduced, so as to complete image processing at high speed. Further, by setting a predetermined area including specific pixels as the transformation target area, it is possible to suppress appearance of an unnecessary multivalued image (virtual image), whose polarity based upon a predetermined condition for projective transformation is a reverse polarity, in the multivalued image after image processing.
Image processing apparatuses according to embodiments of the present invention will be described below with reference to the drawings. It is to be noted that elements having the same or like structures or functions are denoted by the same or like reference numerals throughout the drawings to be referenced, and detailed examples thereof will not be given.
The image processing apparatus 2 is provided with a main control section 21 configured by at least a CPU (central processing unit), an LSI, or the like, a memory 22, a storage device 23, an input device 24, an output device 25, a communication device 26, an auxiliary storage device 27, and an internal bus 28 to which the above hardware components are connected. The main control section 21 is connected to the hardware components of the image processing apparatus 2 as described above via the internal bus 28, and controls operations of the hardware components and executes various software functions according to a computer program 5 stored in the storage device 23. The memory 22 is configured by a volatile memory such as an SRAM or an SDRAM, in which a load module is extracted when executing the computer program 5 and temporary data or the like created when executing the computer program 5 is stored.
The storage device 23 is configured by a built-in fixed storage device (hard disk or flash memory), an ROM, or the like. The computer program 5 stored in the storage device 23 is downloaded from a portable recording medium 4 such as a DVD, a CD-ROM, or a flash memory in which pieces of information such as the program and the data are stored to the auxiliary storage device 27. In execution, the computer program 5 is extracted from the storage device 23 to the memory 22 and executed. It should be appreciated that the computer program 5 can be a computer program downloaded via the communication device 26 from an external computer.
The communication device 26 is connected to the internal bus 28, and is able to transmit and receive data with an external computer and the like by being connected to an external network such as the Internet, a LAN, or a WAN. Specifically, the configuration of the storage device 23 is not limited to a built-in type in the image processing apparatus 2, and can be an external recording medium such as a hard disk provided for an external server computer or the like connected via the communication device 26.
The input device 24 represents a wide concept generally including a variety of devices that acquire inputted information of a touch panel or the like integrated with a liquid crystal panel or the like, in addition to data input media such as a keyboard and a mouse. The output device 25 refers to a printing device such as a laser printer or a dot printer.
The camera (imaging device) 1 is a CCD camera or the like provided with a CCD imaging device. The display device 3 is a display device provided with a CRT, a liquid crystal panel, or the like. The components such as the camera 1 and the display device 3 can be integrated with the image processing apparatus 2 or can be provided separately. External control equipment 6 is a control device connected via the communication device 26, and corresponds to a PLC (programmable logic controller), for example. As used herein, the external control equipment 6 represents a wide concept generally including a variety of devices that execute post-processing in response to a result of image processing by the image processing apparatus 2.
The camera 1 functions, for example, as a digital camera and picks up an image of, for example, an electronic component to be inspected, as an imaging object. The camera 1 acquires a multivalued image, and outputs the image to the image processing section 7.
The image processing section 7 includes a shape information setting device 70, a projective transformation matrix calculating device 71, a projected area calculating device 72, an effective area specifying device 73, and an image transforming device 74. Further, the image processing section 7 is configured to include a main control section 21, a memory 22, a variety of interfaces, and the like, as shown in
The storage device 23 functions as an image memory, and stores as needed an original multivalued image picked up by the camera 1, and a multivalued image after a variety of processing having been performed in the image processing section 7.
The input accepting/image displaying section 8 includes a display device 3 such as a monitor for a computer, and an input device 24 such as a mouse and keyboard. The input accepting section is, for example, provided on a display screen of the display device 3 as a dialog box, and includes a shape information input accepting device 80. An image displaying device 81 is provided adjacently to the input accepting section, and displays the multivalued image before image processing, the multivalued image after image processing, and shape information, the input of which has been accepted by the shape information input accepting device 80. A display switching device 82 can switch the multivalued image before image processing, the multivalued image after image processing, and the shape information the input of which has been accepted by the shape information input accepting device 80, based upon a switching instruction accepted in the input device 24, and display a switched screen in the image displaying device 81.
Next, each of configurations of the image processing section 7 will be described.
The shape information setting device 70 sets shape information as information to be used in the projective transformation matrix calculating device 71, the shape information specifying shapes of the imaging object before image processing and after image processing, an input of which has been made by a user and accepted by the shape information input accepting device 80 of the input accepting/image displaying section 8. The shape information that specifies the shapes of the imaging object can be coordinate values at four corners of an upper surface of an electronic component in the shape of a substantially rectangular parallelepiped in the case of using the electronic component as the imaging object.
It is to be noted that the shape information that specifies shapes of the imaging object before image processing and after image processing is information for calculating a projective transformation matrix as a projective transformation parameter of the entire multivalued image by making four points specified in the multivalued image before image processing correspond to four points specified in the multivalued image after image processing. The coordinate values at the four corners of imaging object are not necessarily specified. By using shape information that specifies a different shape that exists on the same imaging plane as the imaging object, the entire multivalued image including the imaging object can be projectively transformed. Further, in the image processing apparatus 2, accepting an input of shape information that specifies shapes of the imaging object before image processing and after image processing and once calculating a projective transformation parameter based upon the shape information can eliminate the need for new calculation of a projective transformation parameter at the time of image processing unless readjustment is needed. With no need for calculation of a projective transformation parameter at the time of image processing, the image processing apparatus 2 can perform image processing on the multivalued image picked up by the imaging device in real time by use of the once calculated projective transformation parameter.
When the input of shape information is accepted by the shape information input accepting device 80, based upon the shape information the input of which has been accepted, the projective transformation matrix calculating device (projective transformation parameter calculating device) 71 calculates a projective transformation matrix as a predetermined projective transformation parameter for projectively transforming the multivalued image before image processing to correct perspective distortion.
It is to be noted that the projective transformation matrix calculated by the projective transformation matrix calculating device 71 represents a projective transformation matrix that transforms a coordinate of the multivalued image after image processing to a coordinate of the multivalued image before image processing. Image processing associated with coordinate transformation normally requires previous calculation of a transformation parameter for transforming a coordinate of the multivalued image after image processing (coordinate of a transformation destination) to a coordinate of the multivalued image before image processing (coordinate of a transformation source). By previously calculating this transformation parameter, it is possible to thoroughly specify a coordinate of the multivalued image before image processing which correspond to a coordinate of each pixel in the multivalued image after image processing, so as to perform image processing with high accuracy without occurrence of problems such as pixels (defective pixels) with coordinates not transformed in the multivalued image after image processing.
The projected area calculating device 72 calculates a projected area, obtained by projectively transforming an area where pixels of the multivalued image before image processing exist to an output area where pixels of the multivalued image after image processing exist, based upon inverse transformation of the projective transformation matrix calculated by the projective transformation parameter calculating device 71. The effective area specifying device 73 specifies, as an effective area, an area in which the projected area calculated by the projected area calculating device 72 is overlapped with the output area where the pixels of the multivalued image after image processing exist. It is to be noted that the area where the pixels of the multivalued image before image processing exist may be an area including all the pixels of the multivalued image picked up by the camera 1, or the area can be restricted to a predetermined area including pixels that exist inside the shape of the imaging object specified by the shape information. Herein, the multivalued image before image processing is an original multivalued image picked up by the camera 1, and the multivalued image after image processing is a multivalued image outputted after projective transformation by the image processing section 7.
The image transforming device 74 performs coordinate transformation from coordinates of the multivalued image after image processing (coordinates of the transformation destination) to coordinates of the multivalued image before image processing (coordinates of the transformation source) based upon the projective transformation matrix exclusively, in the effective area specified by the effective area specifying device 73, thereby to generate the multivalued image after image processing with corrected perspective distortion. The multivalued image after projectively transformed by the image transforming device 74 may be outputted to external control equipment 6, while being displayed in the image displaying device 81.
In
For example, the user selects the pull-down box 52a with the input device 24 such as the mouse, to get the pull-down list 52c open, and specify “RECTANGLE” from the pull-down list 52c. The user specifies “RECTANGLE”, selects the edit button 53a with the input device 24 such as the mouse, and specifies the four corners of the electronic component 40 displayed in the image displaying device 81 with the input device 24 such as the mouse. In addition, when specifying “RECTANGLE” from the pull-down list 52c, the user specifies four points as shown in
Returning to
It is to be noted that, although a matrix having matrix elements a to i may be used for the projective transformation matrix H as in Equation 3, the matrix shown in Equation 1 is used in the first embodiment, where values of the matrix elements other than the matrix element i are divided by the matrix elements i, to reduce the matrix elements by one.
In step S303, for calculation of the projective transformation matrix H, four coordinates (x1′, y1′) to (x4′, y4′) as shape information that specifies a shape of the imaging object before image processing and four coordinates (x1, y1) to (x4, y4) as shape information that specifies a shape of the imaging object after image processing, the input of which has been accepted in step S302, are substituted into Equation 1 or Equation 2, and eight simultaneous equations shown in
The projective transformation matrix H shown in Equation 1 is a matrix for projectively transforming the coordinate (x, y) of the multivalued image after image processing to the coordinate (x′, y′) of the multivalued image before image processing. In order to projectively transform the coordinate (x′, y′) of the multivalued image before image processing to the coordinate (x, y) of the multivalued image after image processing, it is required to obtain an inverse matrix (Equation 5) of the projection transformation matrix H, as an inverse matrix of the projective transformation matrix H shown in Equation 1. It should be noted that in order to obtain the projective transformation matrix for transforming the coordinate (x′, y′) of the multivalued image before image processing to the coordinate (x, y) of the multivalued image after image processing, it is not necessarily required to calculate the matrix by obtaining an inverse matrix of the projective transformation matrix H, but the matrix may be directly calculated from a corresponding relation between the coordinate (x, y) of the multivalued image after image processing and the coordinate (x′, y′) of the multivalued image before image processing. Also in the inverse matrix of the projective transformation matrix H shown in Equation 5, values of the matrix elements other than the matrix element i′ are divided by the matrix elements i′, to reduce the matrix elements by one.
Returning to
Moreover,
Returning to
It is to be noted that the nearest pixel of the coordinate 66 obtained by projectively transforming the pixel 65 of the effective area 64 based upon the projective transformation matrix H can be taken, as it is, as the pixel 65 of the effective area 64. In order to generate a multivalued image after image processing with higher accuracy, a pixel in the vicinity of the obtained coordinate 66 is interpolated and taken as the pixel 65 of the effective area 64. As for an interpolation method, for example, bilinear interpolation may be employed where the four nearest pixels are linearly interpolated, but the present invention is not restricted to this interpolation method.
Herein, the effective area 64 shows an area where the multivalued image before image processing can be displayed in the output area 63 where the pixels of the multivalued image after image processing exist in the case of generating the multivalued image after image processing with corrected perspective distortion from the multivalued image before image processing based upon the projective transformation matrix H. Therefore, an area other than the effective area 64 out of the entire output area 63 is an area that does not exist in the multivalued image before image processing, and is not required to be projectively transformed based upon the projective transformation matrix H. It is to be noted that in the first embodiment, the projective transformation matrix H, used in image processing to generate the multivalued image after image processing with corrected perspective distortion from the multivalued image before image processing, is not necessarily calculated based upon shape information that specifies shapes of the imaging object before image processing and after image processing. For example, calibration processing may be performed using a multivalued image obtained by picking up a calibration pattern printed with a pattern of dots at fixed intervals, or the like, thereby to calculate a projective transformation matrix.
Although the description was given in the first embodiment to the image processing steps for generating the multivalued image after image processing with corrected perspective distortion after previous obtainment of the effective area, image processing may be performed while specifying the effective area in the process of generating the multivalued image after image processing. In other words, the multivalued image may be specified by determining whether the pixels of the multivalued image after image processing exist inside or outside the projected area 62 in the process of generating the multivalued image after image processing. When the pixels of the multivalued image after image processing exist inside the projected area 62, the nearest pixel of the coordinate 66 obtained by performing projective transformation based upon the projective transformation matrix H can be taken, as it is, as the pixel 65 of the effective area 64. When the pixels exist outside the projected area 62, image processing, such as processing of setting pixel values to 0 (zero) while not performing projective transformation, may be performed.
As described above, in the image processing apparatus 2 according to the first embodiment of the present invention, based upon inverse transformation of the projective transformation matrix H, the projected area calculating device 72 calculates the projected area 62, obtained by projectively transforming the area 61 where the pixels of the multivalued image before image processing exist to the output area 63 where the pixels of the multivalued image after image processing exist. The effective area specifying device 73 specifies, as the effective area 64, an area in which the projected area 62 is overlapped with the output area 63 where the pixels of the multivalued image after image processing exist. Based upon the projective transformation matrix, the image transforming device 74 performs coordinate transformation from coordinates of the multivalued image after image processing (coordinates of the transformation destination) to coordinates of the multivalued image before image processing (coordinates of the transformation source) exclusively in the effective area 64, to generate a multivalued image after image processing with corrected perspective distortion. Thereby, it is not necessary to perform projective transformation on a multivalued image out of the effective area 64, and processing time can be reduced, so as to complete image processing at high speed.
In an area where pixels of the multivalued image before image processing exist, the transformation target area setting device 721 sets, as a transformation target area, a predetermined area including pixels that exist inside the shape of the imaging object specified by shape information, the input of which has been accepted by the shape information input accepting device 80. In the area 61 shown in
Based upon inverse transformation of the projective transformation matrix H calculated by the projective transformation matrix calculating device 71, in the area 61 where the pixels of the multivalued image before image processing exist, the projected area calculating device 72 calculates a projected area obtained by projectively transforming the transformation target area set by the transformation target area setting device 721 to the output area 63 where pixels of a multivalued image after image processing exist.
Further, the effective area specifying device 73 specifies, as the effective area 64, an area in which the projected area obtained by projectively transforming the transformation target area of the multivalued image before image processing to the output area 63 where the pixels of the multivalued image after image processing exist, is overlapped with the output area 63 where the pixels of the multivalued image after image processing exist.
As shown in
In step S1101 which corresponds to step S301, the main control section 21 of the image processing section 7 acquires a multivalued image picked up by the camera 1. In step S1102 which corresponds to step S302, the main control section 21 accepts an input of shape information, made by the user, the information specifying shapes of the imaging object before image processing and after image processing. In step S1103 which corresponds to step S303, the main control section 21 projectively transforms the multivalued image before image processing based upon the shape information the input of which has been accepted, to calculate the projective transformation matrix H for correcting perspective distortion. It should be noted that the projection transformation matrix H represents a projective transformation matrix for performing coordinate transformation from a coordinate of the multivalued image after image processing to a coordinate of the multivalued image before image processing. In step S1104, the main control section 21 of the image processing section 7 sets, as a transformation target area, a predetermined area including pixels that exist inside the shape of the imaging object 91 specified by the shape information, the input of which has been made by the user and accepted.
The processing for obtaining a predetermined area to be set as a transformation target area will be further described.
[Mathematical Formula 5]
g′x+h′x+1=0 (Equation 6)
Hereinafter, processing for obtaining a predetermined area to be set as the transformation target area will be described by use of the boundary 68 representable by Equation 6.
The first intersection calculating device 721a calculates a first intersection at which the boundary 68 that is defined based upon matrix elements (g′, h′, 1) of the inverse matrix of the projective transformation matrix H is intersected with an outer circumferential line of the area 61 where the pixels of the multivalued image before image processing exist.
The second intersection calculating device 721b calculates a second intersection at which outer circumferential lines of an area are intersected with each other, the area including pixels that exist inside the shape of the imaging object specified by the shape information the input of which has been accepted by the shape information input accepting device 80, out of the areas 61 which were divided by the boundary 68 and in which the pixels of the multivalued image before image processing exist.
The polygonal area calculating device 721c calculates a polygonal area formed by connecting the first intersections calculated by the first intersection calculating device 721a and the second intersections calculated by the second intersection calculating device 721b. The polygonal area calculated by the polygonal area calculating device 721c is set as the transformation target area of the transformation target area setting device 721.
When the main control section 21 determines that the boundary 68 is intersected with the outer circumferential line of the area 61 where the pixels of the multivalued image before image processing exist (step S1401: YES), the main control section 21 calculates first intersections at which the boundary 68 is intersected with the outer circumferential line of the area 61 where the pixels of the multivalued image before image processing exist (step S1403).
Next, the main control section 21 calculates second intersections at which outer circumferential lines of an area are intersected with each other, the area including pixels that exist inside the shape of the imaging object 91 specified by the shape information, the input of which has been accepted in step S1102, out of the areas 61 which were divided by the boundary 68 and in which the pixels of the multivalued image before image processing exist (step S1404).
Next, the main control section 21 calculates a polygonal area formed by connecting the first intersections 69, 69 calculated in step S1403 and the second intersections 170, 170 calculated in step S1404 (step S1405).
Returning to
Returning to
Returning to
Although the description was given in the second embodiment to the image processing steps for generating the multivalued image after image processing with corrected perspective distortion after obtainment of the effective area, image processing may be performed while specifying the effective area in the process of generating the multivalued image after image processing. In other words, the effective area may be specified by determining whether the pixels of the multivalued image after image processing exist inside or outside the projected area 62 in the process of generating the multivalued image after image processing. When the pixels of the multivalued image after image processing exist inside the projected area 62, the nearest pixel of the coordinate 66 obtained by performing projective transformation based upon the projective transformation matrix H can be taken, as it is, as the pixel 65 of the effective area 64. When the pixels exist outside the projected area 62, image processing, such as processing of setting pixel values to 0 (zero) while not performing projective transformation, may be performed.
As described above, in the image processing apparatus 2 according to the second embodiment of the present invention, in the area 61 where the pixels of the multivalued image before image processing exist, a predetermined area including pixels that exist inside the shape of the imaging object 91 specified by shape information, the input of which has been accepted by the shape information input accepting device 80, is set as a transformation target area 190. The projected area 62 is calculated which is obtained by projectively transforming the transformation target area 190 of the multivalued image before image processing to the output area 63 where the pixels of the multivalued image after image processing exist. An area is specified as the effective area 64 in which the projected area 62 is overlapped with the output area 63 where the pixels of the multivalued image after image processing exist. It is thereby possible to suppress appearance of the multivalued image 93, as an unnecessary multivalued image (virtual image) shown in
Further, in the image processing apparatus 2 according to the second embodiment of the present invention, the first intersections 69, 69 are calculated at which the boundary 68 defined based upon the matrix element of inverse transformation of the projective transformation matrix H is intersected with the outer circumferential line of the area 61 where the pixels of the multivalued image before image processing exist. The second intersections 170, 170 are calculated at which the outer circumferential lines of an area are intersected with each other, the area including pixels that exist inside the shape of the imaging object 91 specified by the shape information out of the areas 61 which are divided by the boundary 68 and in which the pixels of the multivalued image before image processing exist. The polygonal area 180, formed by connecting the first intersections 69, 69 and the second intersections 170, 170, is set as the transformation target area 190. It is thereby possible to reliably suppress appearance of the multivalued image 93 as the unnecessary multivalued image (virtual image) in the multivalued image after image processing.
Number | Date | Country | Kind |
---|---|---|---|
2009-273983 | Dec 2009 | JP | national |