The present invention relates to an image processing device, an image processing method and an image processing program.
There has been proposed a device that generates a synthetic image by combining a plurality of captured images captured by a plurality of cameras (see Patent Reference 1, for example). This device corrects deviation in boundary parts of the plurality of captured images by calibrating a camera parameter of each of the plurality of cameras by using feature points in a captured image captured before a change in the posture of a vehicle and the feature points in a captured image captured after the change in the posture of the vehicle.
Patent Reference 1: WO 2017/069191 (see paragraph 0041 and FIG. 5, for example)
However, the aforementioned conventional device estimates a position posture change of an image capturing device occurring in a short time by performing the matching between the feature points in a captured image before the position posture change and the feature points in a captured image after the position posture change. Therefore, when estimating the position posture change of a camera in a long period (some days to some years), there is a possibility that the matching between the feature points fails due to great changes between the features in the captured image before the position posture change and the features in the captured image after the position posture change. Further, after the correction of the deviation, no evaluation is made of whether the deviation in the boundary parts of the plurality of captured images has been corrected accurately or not. Accordingly, there is a problem in that the deviation remains in the boundary parts in the synthetic image.
An object of the present invention, which has been made to resolve the above-described problems with the conventional technology, is to provide an image processing device, an image processing method and an image processing program capable of accurately correcting the deviation occurring in overlap regions of a plurality of captured images constituting the synthetic image due to the position posture change of a plurality of image capturing devices.
An image processing device according to an aspect of the present invention is a device for executing a process of combining a plurality of captured images captured by a plurality of image capturing devices, including: processing circuitry to record each of the plurality of captured images in a storage device while associating the captured image with identification information on the image capturing device that captured the captured image and time information indicating an image capture time; to calculate an estimated movement amount of each of the plurality of image capturing devices based on the plurality of captured images recorded in the storage device; and repeatedly to execute a deviation correction process including a process of obtaining an evaluation value of a deviation amount in each overlap region of the plurality of captured images constituting a synthetic image generated by combining the plurality of captured images whose image capture times are the same, a process of updating an external parameter of each of the plurality of image capturing devices based on the estimated movement amount and the evaluation value of the deviation amount, and a process of combining the plurality of captured images whose image capture times are the same by using the updated external parameters, wherein in regard to each of the plurality of image capturing devices, the processing circuitry acquires the captured images in a designated period from the storage device, obtains movement amounts in adjacent image periods based on a plurality of captured images arranged in chronological order, and obtains the estimated movement amount by calculation using the movement amounts in the adjacent image periods.
An image processing method according to another aspect of the present invention is a method of executing a process of combining a plurality of captured images captured by a plurality of image capturing devices, including: recording each of the plurality of captured images in a storage device while associating the captured image with identification information on the image capturing device that captured the captured image and time information indicating an image capture time; when calculating an estimated movement amount of each of the plurality of image capturing devices based on the plurality of captured images recorded in the storage device, in regard to each of the plurality of image capturing devices, acquiring the captured images in a designated period from the storage device, obtaining movement amounts in adjacent image periods based on a plurality of captured images arranged in chronological order, and obtaining the estimated movement amount by calculation using the movement amounts in the adjacent image periods; and repeatedly executing a deviation correction process including a process of obtaining an evaluation value of a deviation amount in each overlap region of the plurality of captured images constituting a synthetic image generated by combining the plurality of captured images whose image capture times are the same, a process of updating an external parameter of each of the plurality of image capturing devices based on the estimated movement amount and the evaluation value of the deviation amount, and a process of combining the plurality of captured images whose image capture times are the same by using the updated external parameters.
An image processing device according to another aspect of the present invention is a device for executing a process of generating a synthetic image by combining a plurality of camera images captured by a plurality of cameras, including: processing circuitry to provide a plurality of external parameters as camera parameters of the plurality of cameras; to generate synthesis tables, as mapping tables used at a time of combining projection images, based on the plurality of external parameters and to generate a plurality of projection images corresponding to the plurality of camera images by projecting the plurality of camera images onto the same projection surface by using the synthesis tables; to generate the synthetic image from the plurality of projection images; to calculate a plurality of external parameters after correction as camera parameters of the plurality of cameras by estimating movement amounts of the plurality of cameras based on reference data, including a plurality of reference images as camera images used as reference corresponding to the plurality of cameras and a plurality of external parameters corresponding to the plurality of reference images, and the plurality of camera images captured by the plurality of cameras; and to update the plurality of external parameters to the plurality of external parameters after the correction by using a result of the calculating of the plurality of external parameters after the correction.
According to the present invention, the deviation occurring in the overlap regions of the plurality of captured images constituting the synthetic image due to the position posture change of a plurality of image capturing devices can be corrected with high accuracy.
The present invention will become more fully understood from the detailed description given hereinbelow and the accompanying drawings which are given by way of illustration only, and thus are not limitative of the present invention, and wherein:
Image processing devices, image processing methods and image processing programs according to embodiments of the present invention will be described below with reference to the drawings. The following embodiments are just examples and a variety of modifications are possible within the scope of the present invention.
The cameras 1a to 1d have a function of capturing images. Each of the cameras 1a to 1d includes an image pickup device such as a CCD (Charged-Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor and a lens unit including one or more lenses. The cameras 1a to 1d do not need to be devices of the same type having the same configuration as each other. Each camera 1a-1d can be, for example, a fixed camera including a fixed lens unit and having no zoom function, a zoom camera including a movable lens unit and having the zoom function, a pan tilt zoom (PTL) camera, or the like. In the first embodiment, a case where the cameras 1a to 1d are fixed cameras will be described.
The cameras 1a to 1d are connected to the image input interface 14 of the image processing device 10. This connection may be either wired connection or wireless connection. The connection between the cameras 1a to 1d and the image input interface 14 is, for example, connection by an IP (Internet Protocol) network. The connection between the cameras 1a to 1d and the image input interface 14 may also be a different type of connection.
The image input interface 14 receives captured images (i.e., image data) from the cameras 1a to 1d. The received captured images are stored in the memory 12 or the storage device 13. The processor 11 generates a synthetic image (i.e., synthetic image data) by performing a synthesis process on a plurality of captured images received from the cameras 1a to 1d by executing a program stored in the memory 12 or the storage device 13. The synthetic image is sent to the display device 18 as the display via the display device interface 15. The display device 18 displays an image based on the received synthetic image.
The image processing device 10 receives the captured images 101a to 101d from the cameras 1a to 1d and generates one synthetic image by combining the captured images 101a to 101d together. The image recording unit 102 records the captured images 101a to 101d captured by the cameras 1a to 1d in the storage unit 114, the external storage unit 115 or both of the storage unit 114 and the external storage unit 115.
The timing determination unit 103 commands timing for the image recording unit 102 to record the captured images 101a to 101d.
The movement amount estimation unit 104 calculates an estimated movement amount (i.e., position posture deviation amount) of each of the cameras 1a to 1d. The movement amount is represented by, for example, translational movement components and rotational movement components of each camera 1a-1d. The translational movement components include three components in X-axis, Y-axis and Z-axis directions in an XYZ orthogonal coordinate system. The rotational movement components include three components of roll, pitch and yaw. Incidentally, the format of the parameters is not limited here as long as the movement amount of each camera can be uniquely determined. Further, the movement amount may also be formed with part of the plurality of components. The movement (i.e., position posture deviation) of each camera 1a-1d can be represented by, for example, a movement vector having three translational movement components and three rotational movement components as elements. An example of the movement vector will be shown as a movement vector Pt in
The outlier exclusion unit 113 judges whether each of movement amounts #1 to #N−1 in periods between adjacent images (hereinafter referred to also as “movement amounts in adjacent image periods”) corresponds to an outlier or not in a process of determining the movement amount of each camera 1a-1d in a designated period estimated by the movement amount estimation unit 104 (hereinafter referred to also as an “estimated movement amount”), and determines not to use the movement amounts in adjacent image periods corresponding to outliers for the calculation for determining the estimated movement amount generated by the movement amount estimation unit 104. Here, N is a positive integer. The judgment on whether one of the movement amounts in the adjacent image periods corresponds to an outlier or not can be made based on whether or not the movement amount in the adjacent image period is a value that cannot occur. For example, the outlier exclusion unit 113 judges that the movement amount in the adjacent image period is an outlier when the movement amount in the adjacent image period exceeds a predetermined threshold value. A concrete example of the judgment on whether the movement amount in the adjacent image period is an outlier or not will be described later with reference to
The feature point extraction unit 105 extracts feature points, to be used for calculating the estimated movement amounts of the cameras 1a to 1d, from the captured images 101a to 101d.
The parameter optimization unit 106 obtains optimum external parameters, for correcting the deviation in the overlap regions between the captured images constituting the synthetic image, based on the estimated movement amounts calculated by the movement amount estimation unit 104 and evaluation values of deviation amounts provided from the deviation amount evaluation unit 110 which will be described later, and updates the external parameters by using the obtained external parameters. The deviation in the overlap region between captured images will be referred to also as “deviation in the synthetic image”. This amount is shown in
The correction timing determination unit 107 determines timing for correcting the deviation in the synthetic image.
The synthesis table generation unit 108 generates a synthesis table as a mapping table of each of the captured images corresponding to the external parameter provided from the parameter optimization unit 106. The synthesis processing unit 109 generates the synthetic image by combining the captured images 101a to 101d into one image by using the synthesis tables provided from the synthesis table generation unit 108.
The deviation amount evaluation unit 110 calculates the amount of the deviation in the synthetic image, that is, the deviation amount, and outputs the calculated value of the deviation amount as the evaluation value of the deviation amount. The deviation amount evaluation value is provided to the parameter optimization unit 106. The overlap region extraction unit 111 extracts the overlap regions between the captured images 101a to 101d constituting the synthetic image when the synthesis processing unit 109 combines the captured images 101a to 101d together. The display image output unit 112 outputs the synthetic image in which the deviation has been corrected, that is, the synthetic image after a deviation correction process.
The image recording unit 102 records the captured images 101a to 101d in the storage unit 114, the external storage unit 115 or both of the storage unit 114 and the external storage unit 115 with timing designated by the timing determination unit 103. When recording each captured image 101a-101d, the image recording unit 102 also records a device ID as identification information for identifying the camera that generated the captured image 101a-101d and an image capture time while associating the device ID and the image capture time with each captured image 101a-101d. The device ID and the image capture time are referred to also as “accompanying information”. Namely, the image recording unit 102 stores the captured images 101a to 101d associated with the accompanying information in the storage unit 114, the external storage unit 115 or both of the storage unit 114 and the external storage unit 115.
As the method of recording each captured image 101a-101d and the accompanying information while associating them with each other, there are, for example, a method of including the accompanying information in the data of each captured image 101a-101d, a method of making the association by using a relational database such as RDBMS (Relational DataBase Management System), and so forth. The method of recording each captured image 101a-101d and the accompanying information while associating them with each other can be a method other than the aforementioned methods.
The timing determination unit 103 determines the timing for recording the captured images provided from the cameras 1a to 1d based on a condition designated by the user, for example, and notifies the image recording unit 102 of the determined timing. The designated condition can be at predetermined constant time intervals, at each time point when a predetermined situation occurs, or the like. The predetermined time interval is a constant time interval designated by using a unit such as second, minute, hour, day, month or the like. The time point when a predetermined situation occurs can be, for example, a time point when feature points are detected in the captured image from the camera 1a-1d (e.g., a certain time point in the daytime), a time point when no moving object is detected in the captured image from the camera 1a-1d, or the like. Further, the timing for recording the captured image may also be determined individually for each camera 1a-1d based on characteristics and installation position condition of the camera 1a-1d.
The feature point extraction unit 105 extracts feature points in each captured image 101a-101d and detects the coordinates of the feature points in order to calculate the estimated movement amount of each camera 1a-1d based on the captured images 101a to 101d. There is AKAZE as a typical example of feature point detection algorithm. However, the feature point detection algorithm is not limited to this example.
The movement amount estimation unit 104 calculates the estimated movement amount of each camera 1a-1d, i.e., calculates the estimated movement amount, based on the feature points of the captured images 101a to 101d recorded by the image recording unit 102. The estimated movement amount of each camera 1a-1d is, for example, a movement amount from a position at a reference time defined as the time point when the camera 1a-1d was installed. The estimated movement amount of each camera 1a-1d is, for example, a movement amount in a period between designated starting day and ending day. The estimated movement amount of each camera 1a-1d can also be the estimated movement amount of each camera 1a-1d in a period between a starting time and an ending time defined by designating the starting time and the ending time. The movement amount estimation unit 104 calculates the estimated movement amount of each camera 1a-1d based on the coordinates of the feature points of each captured image 101a-101d at two time points.
Further, the movement amount estimation unit 104 receives feedback information from the parameter optimization unit 106 when a parameter optimization process (i.e., the deviation correction process) has been executed by the deviation correction unit 100. Specifically, the movement amount estimation unit 104 sets (i.e., resets) the estimated movement amount calculated for each camera 1a-1d to zero at a time when the parameter optimization unit 106 has optimized and updated the external parameter of the camera 1a-1d. Alternatively, the movement amount estimation unit 104 may perform the calculation of the estimated movement amount on the basis of machine learning based on the feedback information received from the parameter optimization unit 106. Thereafter, the movement amount estimation unit 104 performs the calculation of the estimated movement amount by defining the reference time as the time point when the feedback information was received.
The estimated movement amount provided by the movement amount estimation unit. 104 is represented by the translational movement components and the rotational movement components of the camera 1a-1d. The translational movement components include the three components in the X-axis, Y-axis and Z-axis directions, and the rotational movement components include the three components of roll, pitch and yaw. Incidentally, the format of the parameters is not limited here as long as the movement amount of each camera can be uniquely determined. The translational movement components and the rotational movement components may be outputted in the format of a vector or a matrix. Incidentally, the process for calculating the estimated movement amount of each camera 1a-1d is not limited to the above-described process. For example, there is a method using a homography matrix as an example of the method of representing the movement amount between camera images. When an internal parameter of the camera is known, the external parameter can be calculated from the homography matrix. The rotational movement components of the estimated movement amount of each camera 1a-1d may also be acquired based on output from a rotary encoder or the like of a camera to which a sensor is attached or a camera including a built-in sensor (e.g., PTZ camera).
In regard to each camera judged by the correction timing determination unit 107 to be a target of the parameter optimization process (i.e., the deviation correction process), the parameter optimization unit 106 obtains the external parameter, to be used for correcting the deviation in the synthetic image, based on the estimated movement amount of each camera 1a-1d provided from the movement amount estimation unit 104 and the deviation amount evaluation value (referred to also as a “deviation amount calculation value”) in the synthetic image calculated by the deviation amount evaluation unit 110. The external parameter is made up of, for example, three components in the X-axis, Y-axis and Z-axis directions as translational movement components and three components of roll, pitch and yaw as rotational movement components. Incidentally, the format of the external parameter is not limited as long as the position posture of the camera can be uniquely determined.
The parameter optimization unit 106 calculates the external parameter, to be used for correcting the deviation in the synthetic image, so as to reduce the deviation amount in the synthetic image based on the estimated movement amount of each camera 1a-1d obtained by the movement amount estimation unit 104 and the deviation amount evaluation value in the synthetic image obtained by the deviation amount evaluation unit 110. The optimization process of the external parameter of each camera is executed by, for example, repeating the following processes (H2) to (H5) in this order after executing the following processes (H1) to (H5):
(H1) process in which the parameter optimization unit 106 updates the external parameter of each camera 1a-1d
(H2) process in which the synthesis table generation unit 108 generates the synthesis table corresponding to parameters (i.e., the internal parameter, a distortion correction parameter and the external parameter) of each camera 1a-1d
(H3) process in which the synthesis processing unit 109 generates the synthetic image by combining the captured images 101a to 101d by using the synthesis table of each camera 1a-1d
(H4) process in which the deviation amount evaluation unit 110 obtains the deviation amount evaluation value in the synthetic image and feeds back the deviation amount evaluation value
(H5) process in which the parameter optimization unit 106 updates the external parameter by using the deviation amount evaluation value as feedback information.
Further, when the position posture deviation has occurred to two or more cameras among the cameras 1a to 1d, the parameter optimization unit 106 executes a process of determining a captured image as the reference among the captured images 101a to 101d and a process of determining the order of the cameras as the targets of the deviation correction process. Furthermore, at a time when the deviation correction process has been executed, the parameter optimization unit 106 provides the movement amount estimation unit 104 with the feedback information for resetting the estimated movement amount of each camera. This feedback information includes the device ID indicating the camera as the target of the resetting of the movement amount and the external parameter after the correction.
The correction timing determination unit 107 provides the parameter optimization unit 106 with the timing satisfying the designated condition as the timing for executing the deviation correction process for correcting the deviation in the synthetic image. Here, the designated condition is a condition that the estimated movement amount of each camera 1a-1d acquired from the movement amount estimation unit 104 via the parameter optimization unit 106 has exceeded a threshold value, a condition that the deviation amount evaluation value in the synthetic image acquired from the deviation amount evaluation unit 110 has exceeded a predetermined threshold value, or the like. The condition that the estimated movement amount of each camera 1a-1d has exceeded a threshold value is, for example, a condition that the “estimated movement amount in a designated period” has exceeded a threshold value, or the like. The correction timing determination unit 107 outputs a command for making the parameter optimization unit 106 execute the deviation correction process for correcting the deviation in the synthetic image. Incidentally, the timing of the deviation correction process may also be designated by the user by using an input interface such as a mouse or a keyboard.
The synthesis table generation unit 108 generates the synthesis tables for generating the synthetic image based on the internal parameter and the distortion correction parameter of each camera 1a-1d and the external parameter of each camera 1a-1d provided from the parameter optimization unit 106.
The synthesis table generation unit 108 provides the synthesis processing unit 109 with the synthesis tables 204a to 204d based on the internal parameter and the distortion correction parameter of each camera 1a-1d and the external parameter of each camera 1a-1d provided from the parameter optimization unit 106. The synthesis processing unit 109 generates the synthetic image 205 based on the captured images 202a to 202d.
Incidentally, a bird's eye synthetic image, a panoramic synthetic image, an around view image or the like can be generated as the synthetic image by changing the positional relationship and image capture ranges of the cameras 1a to 1d. The synthesis table generation unit 108 outputs data indicating the correspondence between pixels of the captured images 202a to 202d and pixels of the synthetic image 205 as the synthesis tables. The synthesis table generation unit 108 arranges the captured images 202a to 202d in two rows and two columns in a case where the synthesis tables 204a to 204d are tables used for combining captured images of two rows and two columns, for example.
The synthesis table generation unit 108 provides the synthesis processing unit 109 with the synthesis tables 207a to 207d based on the internal parameter and the distortion correction parameter of each camera 1a-1d and the external parameter of each camera 1a-1d provided from the parameter optimization unit 106. The synthesis processing unit 109 generates the synthetic image 208 based on the captured images 206a to 206d.
Incidentally, a bird's eye synthetic image, a panoramic synthetic image, an around view image or the like can be generated as the synthetic image by changing the positional relationship and the image capture ranges of the cameras 1a to 1d. The synthesis table generation unit 108 outputs data indicating the correspondence between pixels of the captured images 206a to 206d and pixels of the synthetic image 208 as the synthesis tables. The synthesis table generation unit 108 arranges the captured images 206a to 206d in one row and four columns in a case where the synthesis tables 207a to 207d are tables used for combining captured images of one row and four columns, for example.
The synthesis processing unit 109 receives the synthesis table of each camera 1a-1d generated by the synthesis table generation unit 108 and the captured images captured by the cameras 1a to 1d and generates one synthetic image by combining the captured images together. The synthesis processing unit 109 performs a blending process on parts where captured images overlap with each other.
The deviation amount evaluation unit 110 calculates the deviation amount evaluation value indicating the magnitude of the deviation in the synthetic image based on the synthetic image generated by the synthesis processing unit 109 and the synthesis tables used at the time of the synthesis, and provides the deviation amount evaluation value to the parameter optimization unit 106 and thereby feeds hack the result of the deviation correction process for correcting the deviation in the synthetic image to the parameter optimization unit 106. The deviation in the synthetic image occurs in boundary parts where captured images transformed by using the synthesis tables (i.e., images after the transformation) are joined together. The boundary parts are referred to also as overlap regions or overlap parts. For the calculation of the deviation amount evaluation value in the synthetic image, numerical values such as luminance value difference, distance between corresponding feature points, an image similarity level or the like in the overlap regions of the captured images after the transformation to be joined together are used. The deviation amount evaluation value is calculated for each combination of captured images after the transformation. For example, when there exist the cameras 1a to 1d, the deviation amount evaluation value of the camera 1a is calculated in regard to the cameras 1a and 1b, the cameras 1a and 1c, and the cameras 1a and 1d. While a range used for the calculation of the deviation amount evaluation value is detected automatically, the range may also be designated by an operation by the user.
The overlap region extraction unit 111 extracts the overlap regions of the captured images after the transformation in the synthetic image generated by the synthesis processing unit 109. Information indicating the extracted overlap regions is provided to the deviation amount evaluation unit 110.
The display image output unit 112 outputs the synthetic image provided from the synthesis processing unit 109 to the display device (shown in
In the image recording process set S10, upon receiving a trigger from the timing determination unit 103 (step S11), the image recording unit 102 acquires the captured images 101a to 101d (step S12) and records the captured images 101a to 101d in the storage unit 114, the external storage unit 115 or both of the storage unit 114 and the external storage unit 115 (step S13).
In the movement amount estimation process set S20, the movement amount estimation unit 104 receives the captured images 101a to 101d from the image recording unit 102 and selects captured images not excluded by the outlier exclusion unit 113, i.e., captured images satisfying a predetermined condition (step S21). Subsequently, the movement amount estimation unit 104 receives the feature points in the selected captured images from the feature point extraction unit 105 (step S22). Subsequently, the movement amount estimation unit 104 calculates the estimated movement amount of each camera 1a-1d (step S23). The movement amount estimation unit 104 provides the parameter optimization unit 106 with the estimated movement amount when the estimated movement amount exceeds a threshold value (step S24).
In the parameter optimization process set S30, upon receiving a correction command from the correction timing determination unit 107 (step S31), the parameter optimization unit 106 acquires the estimated movement amount of each camera 1a-1d from the movement amount estimation unit 104 (step S32). The parameter optimization unit 106 sets initial values of the external parameters of the cameras 1a to 1d (step S33) and updates the external parameters (step S34). Subsequently, the synthesis table generation unit 108 generates the synthesis tables as the mapping tables (step S35) and the synthesis processing unit 109 synthesizes an image by using the synthesis tables (step S36). Subsequently, the deviation amount evaluation unit 110 calculates the deviation amount evaluation values in the synthetic image (step S37). The processing of the steps S34 to S37 is executed repeatedly until an optimum solution is obtained.
In the synthesis-display process set S40, the synthesis processing unit 109 acquires the captured images after the transformation (step S41) and combines together the captured images after the transformation by using the synthesis tables (step S42). The display image output unit 112 outputs the synthetic image to the display device. The display device displays a picture based on the synthetic image (step S43).
Upon receiving the trigger, the image recording unit 102 acquires the device ID of the camera (step S111). Subsequently, the image recording unit 102 acquires time information indicating the time of the occurrence of the trigger (step S112). For example, the image recording unit 102 acquires the time of the occurrence of the trigger from a clock installed in a computer forming the image processing device 10. Incidentally, the time information may also be information like an ordinal number clarifying ordinal relationship of captured images to be recorded.
Subsequently, the image recording unit 102 acquires a present captured image from the camera (step S113). Finally, the image recording unit 102 records the captured image in the storage unit 114, the external storage unit 115 or both of the storage unit 114 and the external storage unit 115 while associating the device ID of the camera and time information indicating the image capture time with the captured image (step S114). Incidentally, at the time of receiving the trigger, the image recording unit 102 may also record captured images from a plurality of cameras installed. Alternatively, at the time of receiving the trigger, the image recording unit 102 may exclusively record captured images from cameras satisfying a predetermined condition. Further, when a request for a recorded captured image is received from the movement amount estimation unit 104, the image recording unit 102 provides the movement amount estimation unit 104 with the requested captured image. In the request for a captured image, the movement amount estimation unit 104 designates the requested captured image by designating the device ID of the camera and an image capture time or an image capture period.
In the movement amount estimation process set S20, the feature points are extracted from the captured image from each camera 1a-1d recorded in the image recording process set S10 and the estimated movement amount of each camera 1a-1d is calculated. The estimated movement amount includes, for example, the three components in the X-axis, Y-axis and Z-axis directions as the translational movement components and the three components of roll, pitch and yaw as the rotational movement components. The calculation of the estimated movement amount is performed in parallel with a correction timing determination process performed by the correction timing determination unit 107. The timing for calculating the estimated movement amount can be each time a constant time interval elapses, or when the captured image is updated in the image recording process set S10.
First, the movement amount estimation unit 104 receives captured images 300a recorded in a designated period for performing the calculation of the estimated movement amount among the captured images from each camera recorded by the image recording unit 102 (step S120).
Subsequently, the movement amount estimation unit 104 sorts and arranges the received captured images 300a in the order of the recording by the image recording unit 102 (step S121). The captured images 300a are arranged in the order of captured images #1 to #N. Here, N is a positive integer indicating the order of the image capture time of a captured image.
Subsequently, the movement amount estimation unit 104 obtains the movement amounts 302 in the adjacent image periods by means of image analysis (step S122). As shown in
In the image analysis in this case, the coordinates of feature points detected by the feature point extraction unit 105 by image matching between captured images are used. When no feature point is detected by the image matching by the feature point extraction unit 105, the movement amount estimation unit 104 does not calculate the movement amount in the adjacent image period.
Finally, the movement amount estimation unit 104 totalizes the movement amounts 302 satisfying a predetermined condition among the movement amounts 302 in the adjacent image periods and outputs the sum total as the movement amount of each camera in the designated period, that is, the estimated movement amount 301. Here, the predetermined condition is the fact that the movement amount 302 does not correspond to a movement amount judged as an outlier among the movement amounts #1 to #N in the adjacent image periods. Namely, the sum total of the movement amounts obtained by excluding movement amounts judged as outliers among the movement amounts #1 to #N in the adjacent image periods obtained by the image analysis is calculated as the estimated movement amount 301. The process of previously excluding the movement amounts not satisfying the condition is executed by the outlier exclusion unit 113.
The outlier exclusion unit 113 has the function of preventing movement amounts judged as outliers among the movement amounts 302 in the adjacent image periods from being used by the movement amount estimation unit 104 for the calculation of the estimated movement amount 301 in the designated period. Specifically, the outlier exclusion unit 113 prevents a movement amount from being used for the calculation of the estimated movement amount 301 in the designated period when the movement amount is a value that cannot normally occur, such as when a translational movement component of each camera 1a-1d is a great value exceeding a threshold value or when a rotational movement component is a great value exceeding a threshold value.
As shown in
A plurality of captured images 310 shown in
Subsequently, the outlier exclusion unit 113 obtains G3=“movement amount 316” from the captured image (#M−1) 311 and the captured image (#M+1) 313 recorded immediately before and after the captured image (#M) 312 recorded the M-th (step S132). In this case, G1+G2=G3 holds if the movement amounts have been obtained ideally.
By using this property, the outlier exclusion unit 113 judges that G1=“movement amount 314” or G2=“movement amount 315” includes an outlier if G1+G2 differs greatly from G3 (step S133). In other words, the outlier exclusion unit 113 judges that G1 or G2 as a movement amount is an outlier when|G1+G2−G3| is greater than or equal to a predetermined threshold value.
In the case where |G1+G2−G3| is greater than or equal to the predetermined threshold value, for the exclusion of outliers, the outlier exclusion unit 113 excludes G1=“movement amount 314” and G2=“movement amount 315” and includes G3=“movement amount 316” in the calculation of the estimated movement amount. As above, the outlier exclusion unit 113 handles the movement amounts regarding the M-th captured image (#M) 312 as outliers and excludes G1=“movement amount 314” and G2=“movement amount 315”, as the movement amounts obtained by using the M-th captured image (#M) 312, from the calculation of the estimated movement amount (step S134).
In the parameter optimization process set S30, the correction timing determination unit 107 determines the device ID of the camera as the target of the parameter optimization process, i.e., the deviation correction process, based on the estimated movement amount of each camera 1a-1d provided from the movement amount estimation unit 104 and the deviation amount evaluation value of each camera 1a-1d in the synthetic image provided from the deviation amount evaluation unit 110. Thereafter, the parameter optimization unit 106 obtains the external parameter of the camera as the target of the parameter optimization process. The external parameter includes, for example, the three components in the X-axis, Y-axis and Z-axis directions as the translational movement components and the three components of roll, pitch and yaw as the rotational movement components.
After receiving the device ID of the camera as the target of the parameter optimization process from the correction timing determination unit 107, the parameter optimization unit 106 sets the values of the external parameter of the camera as the target of the parameter optimization process as the values of the external parameter of a moved camera.
Subsequently, the parameter optimization unit 106 changes the external parameter of the camera as the target of the parameter optimization process. The way of changing the external parameter varies depending on the method of the parameter optimization process. Then, the parameter optimization unit 106 provides the synthesis table generation unit 108 with the present external parameters of the plurality of cameras.
The synthesis table generation unit 108 generates the synthesis table for generating the synthetic image in regard to each camera based on the external parameter of each camera 1a-1d provided from the parameter optimization unit 106 and the internal, parameter and the distortion correction parameter of each camera 1a-1d.
The synthesis processing unit 109 generates one synthetic image by combining the captured images after the transformation corresponding to the captured images from the cameras 1a to 1d by using the synthesis tables generated by the synthesis table generation unit 108.
The deviation amount evaluation unit 110 obtains the deviation amount evaluation value in the generated synthetic image based on the generated synthetic image and the synthesis tables used at the time of generating the synthetic image, and feeds back the deviation amount evaluation value to the parameter optimization unit 106. The parameter optimization unit 106 executes the parameter optimization process so as to reduce the deviation amount evaluation value by changing the external parameter of the camera as the target of the parameter optimization process based on the deviation amount evaluation value obtained as the feedback.
A method of automatically determining the timing for the correction will be described below. First, the correction timing determination unit 107 acquires the estimated movement amount of each camera, the deviation amount evaluation value in the synthetic image, or both of them from the movement amount estimation unit 104 or the deviation amount evaluation unit 110 as an index for judging whether the parameter optimization process is necessary or not (steps S140 and S141).
Subsequently, the correction timing determination unit 107 compares the acquired estimated movement amount of each camera with a threshold value, or compares the acquired deviation amount evaluation value in the synthetic image with a threshold value (step S142). For example, when the estimated movement amount exceeds its threshold value or the deviation amount evaluation value exceeds its threshold value, the correction timing determination unit 107 notifies the parameter optimization unit 106 that the parameter optimization process should be executed (step S143). The condition using a threshold value for executing the deviation correction process can be set as various conditions, such as when the estimated movement amount of each camera exceeds its threshold value, when the deviation amount evaluation value in the synthetic image exceeds its threshold value, when both of these conditions are satisfied, and so forth.
Further, the correction timing determination unit 107 may include a configuration for detecting the occurrence of a situation where the deviation correction process cannot be executed based on the result of comparison between the deviation amount evaluation value in the synthetic image and a predetermined threshold value and notifying the user of the occurrence of the situation. The case where the deviation correction process cannot be executed is, for example, a case where position posture deviation that is so great as to disable the formation of an overlap region between captured images has occurred to a camera. The method of notifying the user can be, for example, displaying the notification in superimposition on the displayed synthetic image.
The parameter optimization unit 106 receives the estimated movement amount of each camera from the movement amount estimation unit 104, receives the deviation amount evaluation value in the synthetic image from the deviation amount evaluation unit 110, and outputs the external parameter for the deviation correction process. Incidentally, the parameter optimization process for the process of correcting the deviation, in the synthetic image is executed by the movement amount estimation unit 104 and the deviation correction unit 100.
Subsequently, the parameter optimization unit 106 receives the estimated movement amount of each camera as the target of the parameter optimization process from the movement amount estimation unit 104 (step S151). The estimated movement amount includes, for example, the three components in the X-axis, Y-axis and Z-axis directions as the translational movement components and the three components of roll, pitch and yaw as the rotational movement components.
Subsequently, the parameter optimization unit 106 changes the external parameter of the camera as the target of the parameter optimization process based on the estimated movement amount of each camera 1a-1d acquired from the movement amount estimation unit 104 (step S152). Incidentally, the external parameter at the time of the installation of the camera or at the initial startup of the camera is acquired by camera calibration work performed by using a calibration board having a camera calibration pattern.
P1=(X, Y, Z, roll, pitch, yaw)
Here, X, Y and Z represent external parameters in the X-axis, Y-axis and Z-axis directions, and roll, pitch and yaw represent external parameters in roll, pitch and yaw directions.
Further, the external parameter (namely, external parameter vector) P0 before the update (namely, at a time 0) is represented as follows:
P0=(X_0, Y_0, Z_0, roll_0, pitch_0, yaw_0)
Here, X_0, Y_0 and Z_0 represent external parameters in the X-axis, Y-axis and Z-axis directions, and roll_0, pitch_0 and yaw_0 represent external parameters in the roll, pitch and yaw directions.
Furthermore, a movement vector Pt indicating the movement, i.e., the position posture deviation, from the time 0 to the time t is represented as follows:
Pt=(X_t, Y_t, Z_t, roll_t, pitch_t, yaw_t)
Here, X_t, Y_t and Z_t represent movement amounts (i.e., distances) in the X-axis, Y-axis and Z-axis directions, and roll_t, pitch_t and yaw_t represent movement amounts (i.e., angles) in the roll, pitch and yaw directions.
In this case, the following expression (1) holds:
P1−P0+Pt (1)
Incidentally, the external parameter P0 before the update, at the time of the first update, is the external parameter obtained by the camera calibration. Namely, as shown in the expression (1), the external parameter after the update is a parameter obtained by adding the elements of the movement vector Pt obtained by the movement amount estimation unit 104 to the external parameter at the time of the installation.
Subsequently, the parameter optimization unit 106 judges the number of cameras as the targets of the parameter optimization process based on the number the device IDs of cameras received from the correction timing determination unit 107 (step S153). When there is no camera as the target of the parameter optimization process, the parameter optimization process by the parameter optimization unit 106 is ended.
When there is a camera as the target of the parameter optimization process (i.e., when the judgment is YES in the step S153), the parameter optimization process is executed in order to correct the deviation in the synthetic image (step S154). In this case, when the number of cameras as the targets of the parameter optimization process is two or more, the external parameter optimization process of a camera whose estimated movement amount acquired from the movement amount estimation unit 104 is small is executed first. This is because the camera whose estimated movement amount is small can be regarded as a camera with less errors and high reliability.
Subsequently, the parameter optimization unit 106 repeats a process of obtaining an external parameter for the deviation correction process and updating the external parameter of the camera by using the obtained external parameter (step S154), and excludes the camera whose deviation correction process is completed from the targets of the parameter optimization process and regards the camera as a deviation-corrected camera (step S155). Further, when the external parameter has been updated, the parameter optimization unit 106 feeds back the device ID of the deviation-corrected camera and the external parameter after the correction to the movement amount estimation unit 104 (step S156).
In the parameter optimization process (step S154), the parameter optimization unit 106 repeats the process so as to change the external parameter of the camera, receive the deviation amount evaluation value in the synthetic image at that time, and reduce the deviation amount evaluation value. Various methods such as a genetic algorithm are usable as the algorithm of the parameter optimization process used in this step.
First, the parameter optimization unit 106 acquires the deviation amount evaluation value of the camera as the target of the optimization from the deviation amount evaluation unit 110 (step S1541). The deviation amount evaluation value is acquired in regard to each captured image from a camera with which the targeted captured image overlaps at the time of the synthesis. The parameter optimization unit 106 receives the deviation amount evaluation value from the deviation amount evaluation unit. 110 in regard to each combination of captured images after the transformation. For example, when there exist the cameras 1a to 1d, the parameter optimization unit 106 receives the deviation amount evaluation value in an overlap region between captured images after the transformation corresponding to the captured images from the cameras 1a and 1b, the deviation amount evaluation value in an overlap region between captured images after the transformation corresponding to the captured images from the cameras 1a and 1c, and the deviation amount evaluation value in an overlap region between captured images after the transformation corresponding to the captured images from the cameras 1a and 1d as the deviation amount evaluation value of the camera 1a.
Thereafter, the parameter optimization unit 106 updates the external parameter of each camera based on the acquired deviation amount evaluation value (step S1542). The external parameter update process varies depending on an optimization algorithm that is used. As a typical optimization algorithm, there are methods such as the Newton's method and the genetic algorithm. However, the method of the external parameter update process of each camera is not limited to these methods.
Subsequently, the parameter optimization unit 106 sends the updated external parameter of the camera to the synthesis table generation unit 108 together with the external parameters of the other cameras (step S1543). The synthesis table generation unit 108 generates the synthesis table, to be used at the time of the synthesis, in regard to each camera based on the external parameter of each camera (step S1544).
The synthesis processing unit 109 generates one synthetic image by combining the captured images acquired from the cameras by using the synthesis tables of the cameras generated by the synthesis table generation unit 108 (step S1545).
The deviation amount evaluation unit 110 obtains the deviation amount evaluation value of each camera based on the synthesis tables of the cameras and the captured images used by the synthesis processing unit 109 at the time of the image synthesis, and outputs the obtained deviation amount evaluation value to the parameter optimization unit 106 (step S1546). The external parameter for correcting the deviation in the synthetic image is calculated by repeating the above process until the deviation amount evaluation value becomes less than or equal to a constant threshold value. Alternatively, it is also possible to calculate the external parameter for the correction by repeating the above process for a previously designated number of times.
As shown as step S11 in
Subsequently, as shown as step S12 in
Subsequently, as shown as step S13 in
Subsequently, as shown as step S14 in
Subsequently, as shown as step S15 in
The synthesis table generation unit 108 generates the synthesis tables, to be used at the time of the image synthesis, based on the parameters of each camera 1a-1d received from the parameter optimization unit 106. The parameters include the external parameter, the internal parameter and the distortion correction parameter.
Subsequently, the synthesis table generation unit 108 acquires the internal parameter and the distortion correction parameter of the camera. Incidentally, the internal parameter and the distortion correction parameter of the camera may also be previously stored in a memory of the synthesis table generation unit 108, for example.
Finally, the synthesis table generation unit 108 generates the synthesis table based on the received external parameter of each camera and the internal parameter and the distortion correction parameter in regard to the camera. The generated synthesis table is provided to the synthesis processing unit 109.
The above-described process is executed for each camera. Incidentally, the method of generating the synthesis table is changed depending on the camera used. For example, a projection method (e.g., central projection method, equidistant projection method, etc.) is used for generating the synthesis table. Further, a distortion model (e.g., radial direction distortion model, circumferential direction distortion model, etc.) is used for correcting lens distortion. However, the method of generating the synthesis table is not limited to the above examples.
As shown in
The deviation amount evaluation unit 110 may calculate the deviation amount evaluation value by accumulating luminance differences between pixels in the overlap region. Alternatively, the deviation amount evaluation unit 110 may calculate the deviation amount evaluation value by performing the matching of feature points in the overlap region and accumulating the distances between the matched feature points. Further, the deviation amount evaluation unit 110 may calculate the deviation amount evaluation value by obtaining the image similarity level by using an ECC (Elliptic Curve Cryptography) algorithm. Furthermore, the deviation amount evaluation unit 110 may calculate the deviation amount evaluation value between images by obtaining phase-limited correlation. It is also possible to use not an evaluation value that is optimized so as to minimize the deviation amount evaluation value but an evaluation value that is optimized by maximizing the evaluation value. Further, it is also possible to use an evaluation value that becomes optimum when the evaluation value reaches 0. The deviation amount evaluation value of each camera can be obtained by executing the above-described process for each camera.
In the synthesis-display process set S40 shown in
As described above, with the image processing device 10, the image processing method or the image processing program according to the first embodiment, the deviation amount evaluation values in the synthetic image are fed back to the parameter optimization process (i.e., the deviation correction process), and thus the deviation that has occurred to the overlap regions of the plurality of captured images after the transformation constituting the synthetic image due to the position posture change of the cameras 1a to 1d can be corrected with high accuracy.
Further, with the image processing device 10, the image processing method or the image processing program according to the first embodiment, the estimated movement amounts of the cameras 1a to 1d are calculated at time intervals facilitating the matching between feature points in the plurality of captured images after the transformation constituting the synthetic image, and thus the deviation that has occurred to the overlap regions of the plurality of captured images after the transformation constituting the synthetic image due to the position posture change of the cameras 1a to 1d can be corrected with high accuracy.
Furthermore, with the image processing device 10, the image processing method or the image processing program according to the first embodiment, the external parameter of each camera 1a-1d is optimized in order to correct the deviation that has occurred to the overlap regions of the plurality of captured images after the transformation constituting the synthetic image. Accordingly, the deviation occurring to the overlap regions in the synthetic image can be corrected without the need of performing the manual calibration work.
Moreover, with the image processing device 10, the image processing method or the image processing program according to the first embodiment, the maintenance cost in a monitoring system using a plurality of cameras for the purpose of monitoring can be reduced since the deviation can be corrected with high accuracy and without manual operations.
An image processing device according to a second embodiment differs from the image processing device 10 according to the first embodiment in processing performed by the parameter optimization unit 106. In regard to the other features, the second embodiment is the same as the first embodiment. Therefore,
In the second embodiment, the parameter optimization unit 106 obtains the external parameter, to be used for correcting the deviation in the synthetic image, for each camera 1a-1d based on the estimated movement amount of each camera 1a-1d acquired from the movement amount estimation unit 104 and the deviation amount evaluation value in the synthetic image acquired from the deviation amount evaluation unit 110. The external parameter is made up of the three components in the X-axis, Y-axis and Z-axis directions as the translational movement components and the three components of roll, pitch and yaw as the rotational movement components.
The parameter optimization unit 106 changes the external parameter so as to reduce the deviation amount evaluation value in the synthetic image based on the estimated movement amount of each camera 1a-1d obtained by the movement amount estimation unit 104 and the deviation amount evaluation value in the synthetic image obtained by the deviation amount evaluation unit 110. The optimization process of the external parameter of each camera is executed by, for example, repeating the aforementioned processes (H2) to (H5) in this order after executing the aforementioned processes (H1) to (H5).
Further, when the position posture deviation has occurred to two or more cameras among the cameras 1a to 1d, the parameter optimization unit 106 executes the process of determining a captured image as the reference among the captured images 101a to 101d and the process of determining the order of performing the deviation correction process. Furthermore, at the time when the deviation correction process has been executed, the parameter optimization unit 106 provides the movement amount estimation unit 104 with the feedback information for resetting the estimated movement amount of the camera. This feedback information includes the device ID indicating the camera as the target of the resetting of the estimated movement amount and the external parameter after the correction.
In the second embodiment, when the position posture deviation has occurred to two or more cameras among the cameras 1a to 1d, the parameter optimization unit 106 simultaneously corrects the deviation of all the cameras to which the position posture deviation has occurred. Further, at the time when the deviation correction process has been executed, the parameter optimization unit 106 provides the movement amount estimation unit 104 with the feedback information for resetting the estimated movement amounts of the cameras. This feedback information includes the device IDs indicating the cameras as the targets of the resetting of the estimated movement amounts and the external parameters after the correction.
Thereafter, the parameter optimization unit 106 receives the estimated movement amounts of the cameras from the movement amount estimation unit 104, receives the deviation amount evaluation values in the synthetic image from the deviation amount evaluation unit 110, and outputs the external parameters for the deviation correction process. Incidentally, the deviation correction process for correcting the deviation in the synthetic image is executed by a feedback loop formed of the movement amount estimation unit 104, the parameter optimization unit 106, the synthesis table generation unit 108, the synthesis processing unit 109 and the deviation amount evaluation unit 110.
Thereafter, the parameter optimization unit 106 receives the estimated movement amounts of the cameras as the targets of the parameter optimization process from the movement amount estimation unit 104 (step S211). The estimated movement amount includes, for example, the three components in the X-axis, Y-axis and Z-axis directions as the translational movement components and the three components of roll, pitch and yaw as the rotational movement components.
Subsequently, the parameter optimization unit 106 changes the external parameters of the cameras as the targets of the parameter optimization process based on the estimated movement amount of each camera 1a-1d acquired from the movement amount estimation unit 104 (step S212). Incidentally, the external parameter at the time of the installation of the camera or at the initial startup of the camera is acquired by camera calibration work performed by using a calibration board having a camera calibration pattern. The calculation formulas used for the update of the external parameter performed by the parameter optimization unit 106 are shown in
When there is a camera as the target of the parameter optimization process, the external parameter optimization process is executed (step S213). In this case, when the number of cameras as the targets of the parameter optimization process is two or more, the external parameters of the two or more cameras are optimized at the same time.
Subsequently, when the external parameters to be used for correcting the deviation have been obtained, the parameter optimization unit 106 performs the update by using the obtained external parameters as the external parameters of the cameras and ends the parameter optimization process. Further, when the external parameters are updated, the parameter optimization unit 106 feeds back the device IDs of the corrected cameras and the external parameters after the correction to the movement amount estimation unit 104 (step S214).
In the parameter optimization process (step S213), the parameter optimization unit 106 repeats the process so as to change the external parameters of the cameras, receive the deviation amount evaluation values in the synthetic image at that time, and reduce the deviation amount evaluation values. As the algorithm of the parameter optimization process, the genetic algorithm is usable, for example. However, the algorithm of the parameter optimization process can also be a different algorithm.
First, the parameter optimization unit 106 acquires the deviation amount evaluation value(s) of one or more cameras as the optimization target(s) from the deviation amount evaluation unit 110 (step S2131). This deviation amount evaluation value is acquired in regard to each captured image from a camera with which the targeted captured image overlaps at the time of the synthesis. The parameter optimization unit 106 receives the deviation amount evaluation value from the deviation amount evaluation unit 110 in regard to each combination of captured images. For example, when there exist the cameras 1a to 1d, the parameter optimization unit 106 acquires the evaluation values of the deviation amounts D3 and D1 in regard to the camera 1b as an optimization target #1 and the evaluation values of the deviation amounts D2 and D1 in regard to the camera 1c as an optimization target #2 as shown in
Thereafter, the parameter optimization unit 106 updates the external parameters of the plurality of cameras as the targets by using the sum total of all the acquired deviation amount evaluation values as the deviation amount evaluation value (step S2132). The external parameter update process varies depending on the optimization algorithm that is used. As a typical optimization algorithm, there are methods such as the Newton's method and the genetic algorithm. However, the method of the external parameter update process is not limited to these methods.
Subsequently, the parameter optimization unit 106 sends the updated external parameters of the cameras to the synthesis table generation unit 108 together with the external parameters of the other cameras (step S2133). The synthesis table generation unit 108 generates the synthesis table, to be used at the time of the synthesis, for each camera based on the external parameters of the plurality of cameras (step S2134).
The synthesis processing unit 109 generates one synthetic image by combining the captured images acquired from the cameras by using the synthesis tables of the cameras generated by the synthesis table generation unit 108 (step S2135).
The deviation amount evaluation unit 110 obtains the deviation amount evaluation value of each camera based on the synthesis tables of the cameras and the captured images after the transformation used by the synthesis processing unit 109 at the time of the image synthesis, and outputs the obtained deviation amount evaluation values to the parameter optimization unit 106 (step S2136). The external parameters used for correcting the deviation in the synthetic image are calculated by repeating the above process until the deviation amount evaluation values become less than or equal to a constant threshold value. Alternatively, it is also possible to calculate the external parameters for the correction by repeating the above process for a previously designated number of times.
As shown as step S21 in
Subsequently, as shown as step S22 in
Subsequently, as shown as step S23 in
As described above, with the image processing device, the image processing method or the image processing program according to the second embodiment, the deviation amount evaluation values in the synthetic image are fed back to the parameter optimization process (i.e., the deviation correction process), and thus the deviation that has occurred to the overlap regions of the plurality of captured images after the transformation constituting the synthetic image due to the position posture change of the cameras 1a to 1d can be corrected with high accuracy.
Further, with the image processing device, the image processing method or the image processing program according to the second embodiment, the number of calculations can be reduced since the parameter optimization process is executed based on the sum total of a plurality of deviation amount evaluation values.
An image processing device 610 according to a third embodiment executes the deviation correction process by using superimposition regions of a plurality of captured images (a plurality of camera images) and reference data. The reference data includes a reference image and a camera parameter at the time when the reference image was captured by a camera as an image capturing device. The reference image is a captured image, i.e., a camera image, captured by a camera in a calibrated state. The reference image is referred to also as a “corrected camera image”. The reference image is, for example, a camera image captured by a camera calibrated by using a calibration board when the camera was installed.
The auxiliary memory 613 stores, for example, a plurality of camera images captured by cameras 600_1 to 600_n. The reference character n represents a positive integer. The cameras 600_1 to 600_n are the same as the cameras 1a to 1d described in the first embodiment. Further, the auxiliary memory 613 stores information on the relationship among the installation positions of the cameras 600_1 to 600_n and the blending process at the time of the image synthesis, camera parameters calculated by previous camera calibration, and a lens distortion correction map. Furthermore, the auxiliary memory 613 may store a plurality of mask images to be used for a mask process performed on each of the plurality of camera images. The mask process and the mask images will be described in the fifth embodiment later.
The main processor 611 performs a process of loading information stored in the auxiliary memory 613 into the main memory 612. The main processor 611 stores a still image file in the auxiliary memory 613 when a process using a still image is executed. Further, the main processor 611 performs various calculation processes and various control processes by executing programs stored in the main memory 612. The programs stored in the main memory 612 may include an image processing program according to the third embodiment. The programs are stored in the main memory 612 from a record medium (i.e., a non-transitory computer-readable storage medium) storing information via a medium reading device (not shown), or via a communication interface (not shown) connectable to the Internet or the like.
The input interface 617 receives input information provided by a device input such as a mouse input, a keyboard input or a touch panel input. The main memory 612 stores input information inputted through the input interface 617.
The image processing memory 615 stores input images transferred from the main memory 612 and the synthetic image (i.e., synthetic image data) and projection images (i.e., projection image data) generated by the image processing processor 614.
The display device interface 15 outputs the synthetic image generated by the image processing device 610. The display device interface 15 is connected to the display device 18 by an HDMI (High-Definition Multimedia Interface) cable or the like. The display device 18 displays a picture based on the synthetic image provided from the display device interface 15.
The image input interface 14 receives image signals provided from the cameras 600_1 to 600_n connected to the image processing device 610. The cameras 600_1 to 600_n are network cameras, analog cameras, USB (Universal Serial Bus) cameras, HD-SDI (High Definition-Serial Digital Interface) cameras or the like, for example. The method of the connection between the image processing device 610 and the cameras 600_1 to 600_n is determined depending on the type of the cameras 600_1 to 600_n. Image information inputted through the image input interface 14 is stored in the main memory 612, for example.
The external storage device 17 and the display device 18 are the same as those described in the first embodiment. The external storage device 17 is a storage device connected to the image processing device 610. The external storage device 17 is a hard disk drive (HDD), an SSD or the like. The external storage device 17 is provided so as to supplement the capacity of the auxiliary memory 613, for example, and operates equivalently to the auxiliary memory 613. However, the image processing device without the external storage device 17 is also possible.
In the image processing device 610, the projection processing unit 603 generates synthesis tables, as mapping tables used at the time of combining projection images, based on a plurality of external parameters provided from the camera parameter input unit 601 and generates a plurality of projection images corresponding to the plurality of camera images by projecting the plurality of camera images onto the same projection surface by using the synthesis tables. The synthesis processing unit 602 generates the synthetic image from the plurality of projection images. The reference data readout unit 605 outputs reference data including a plurality of reference images as camera images used as the reference corresponding to the plurality of cameras and a plurality of external parameters corresponding to the plurality of reference images. The movement amount estimation-parameter calculation unit 607 calculates a plurality of external parameters after the correction corresponding to the plurality of cameras by estimating the movement amounts of the plurality of cameras based on the plurality of camera images and the reference data. The deviation detection unit 606 judges whether or not deviation has occurred to any one of the plurality of cameras. When the deviation detection unit 606 judges that deviation has occurred, the deviation correction unit 608 updates the plurality of external parameters provided by the camera parameter input unit 601 by using the plurality of external parameters after the correction calculated by the movement amount estimation-parameter calculation unit 607.
The camera image reception unit 609 shown in
The camera parameter input unit 601 shown in
Further, the camera parameter input unit 601 executes a process of updating the external parameters in the camera parameters stored in the storage device to external parameters corrected by the deviation correction unit 608 (referred to also as “external parameters after the correction”). The camera parameters including the external parameters after the correction are referred to also as “camera parameters after the correction”. Referring to
First, the synthesis processing unit 602 reads in blend information and the camera parameters to be used for the blending process from the camera parameter input unit 601 (steps S321 and S322).
Subsequently, the synthesis processing unit 602 acquires synthesis tables generated by the projection processing unit 603 by using the acquired camera parameters (step S323).
Subsequently, the synthesis processing unit 602 receives a plurality of camera images after undergoing the input process (step S324) and generates the synthetic image as one image by combining projection images made up of a plurality of camera images by making the projection processing unit 603 generate images projected on the same projection surface (i.e., projection images) by using the synthesis tables (step S325). Namely, the synthesis processing unit 602 provides the projection processing unit 603 with the camera parameters acquired from the camera parameter input unit 601 and the camera images read in by the camera image reception unit 609, receives the projection images regarding the cameras provided from the projection processing unit 603, and thereafter combines the received projection images regarding the cameras in the synthetic image generation unit 6021 (
Further, in the step S325, the synthetic image generation unit 6021 of the synthesis processing unit 602 may perform the blending process on joint parts between projection images by using the blend information inputted from the blend information read-in unit 6022. Referring to
Subsequently, the synthesis processing unit 602 outputs the synthetic image to the display processing unit 604 (step S346).
The synthesis processing unit 602 reads in the camera parameters from the camera parameter input unit 601 (step S327) and judges whether or not the camera parameters have changed. When the camera parameters have changed, the process advances to the step S323 and the synthesis processing unit 602 makes the projection processing unit 603 generate the synthesis tables to be used for the synthesis process by using the latest camera parameters acquired in the step S327 and further executes the processing of the steps S324 to S328. When the camera parameters have not changed, the process advances to the step S324 and the synthesis processing unit 602 newly receives a plurality of camera images (step S324) and further executes the processing of the steps S325 to S328.
Subsequently, the projection processing unit 603 reads in the camera parameters (step S303), reads in the camera images (step S304), and generates the projection images from the inputted camera images by using the generated synthesis tables (step S305). Namely, the synthesis table generation unit 6031 (
Subsequently, the projection processing unit 603 judges whether or not the inputted camera parameters have changed (step S306). When the camera parameters have changed, the process advances to step S307 and the projection processing unit 603 regenerates the synthesis tables by using the latest camera parameters acquired in the step S303 and thereafter executes the processing of the steps S303 to S306. When the camera parameters have not changed, the projection processing unit 603 newly receives a plurality of camera images (step S304) and thereafter executes the processing of the steps S305 to S306.
Further, the projection processing unit 603 may output the synthesis tables generated by the synthesis table generation unit 6031. When the inputted camera parameters have not changed, the projection processing unit 603 does not need to regenerate the synthesis tables. Therefore, when the inputted camera parameters have not changed, the synthesis table generation unit 6031 executes a process of leaving the synthesis tables as they are without regenerating the synthesis tables.
The display processing unit 604 executes a process of converting the synthetic image generated by the synthesis processing unit 602 to picture data that can be displayed by the display device and provides the picture data to the display device. The display device is the display device 18 shown in
The reference data readout unit 605 outputs the reference data in the image processing device 610. The reference data is, for example, data including the external parameters as the camera parameters of the cameras in the calibrated state and the reference images as the camera images at that time. The calibrated state is, for example, the state of the cameras 600_1 to 600_n when the calibration by using the calibration board is over at the time of installation of the image processing device 610 and the plurality of cameras 600_1 to 600_n. Referring to
Before the processes (R1) to (R4) by the deviation detection unit 606, processing shown as steps S321 to S326 in
(R1) The deviation detection unit 606 compares the reference images as the camera images in the reference data with present camera images acquired from the camera image reception unit 609 and judges positional deviation (displacement) of each camera 600_1-600_n based on the similarity level between the reference image and the present camera image. This process is shown in steps S334 and S335 in
(R2) The deviation detection unit 606 judges the positional deviation of each camera based on a deviation amount in a projection region. Namely, the deviation detection unit 606 evaluates the deviation amount based on a deviation amount calculated by the projection region deviation amount evaluation unit 6065 which will be described later. This process is shown in steps S327 and S328 in
(R3) The deviation detection unit 606 judges the positional deviation based on a deviation amount in a superimposition region in the synthetic image. Namely, the deviation detection unit 606 evaluates the deviation amount based on a deviation amount calculated by the superimposition region deviation amount evaluation unit 6064 which will be described later. This process is shown in steps S330 to S332 in
(R4) The deviation detection unit 606 compares the reference image with the present camera image acquired from the camera image reception unit 609 and judges the presence/absence of deviation based on the relative movement amount between these two images. This process is shown in step S333 in
The similarity level evaluation unit 6061 shown in
The relative movement amount estimation unit 6062 shown in
The relative movement amount estimation unit 6062 shown in
The superimposition region extraction unit 6063 shown in
The superimposition region deviation amount evaluation unit 6064 shown in
The projection region deviation amount evaluation unit 6065 shown in
The deviation judgment unit 6066 shown in
The movement amount estimation-parameter calculation unit 607 can use a publicly known method such as the five-point algorithm as the method of calculating the relative camera movement amount between two images. In the five-point algorithm, the movement amount estimation-parameter calculation unit 607 detects feature points in the two images, performs the matching between the feature points in the two images (step S342), and inputs the result of the matching to the five-point algorithm. Therefore, the movement amount estimation-parameter calculation unit 607 is capable of estimating the relative movement amount of each camera with respect to the reference data (relative movement amount at the time point of the input from the camera image reception unit 609) by inputting the camera image provided from the camera image reception unit 609 and the reference image to the aforementioned method (step S343).
The movement amount estimation-parameter calculation unit 607 can also output an external parameter indicating the relative movement amount by adding the external parameter of each camera at the time point of the input from the camera image reception unit 609 to the relative movement amount of each camera estimated above (step S344).
When the judgment result provided from the deviation detection unit 606 is “deviation has occurred”, the deviation correction unit 608 shown in
As the external parameter of each camera to which deviation has occurred, the deviation correction unit 608 uses the external parameter provided from the movement amount estimation-parameter calculation unit 607 or the camera parameter input unit 601. As the external parameter of each camera to which no deviation has occurred, the deviation correction unit 608 uses the external parameter provided from the camera parameter input unit 601.
The parameter optimization unit 6082 shown in
Based on the external parameter currently applied to the correction target camera, the parameter optimization unit 6082 calculates an evaluation value from the deviation amount in the superimposition region of the correction target camera and an adjacent camera acquired from the superimposition region deviation amount evaluation unit 6084 and the deviation amount with respect to the projection image of the reference data (reference data of the correction target camera acquired from the reference data readout unit 605) acquired from the projection region deviation amount evaluation unit 6085, and calculates an external parameter that maximizes or minimizes the evaluation value. The parameter optimization unit 6082 repeats the processing of steps S362 and S356 to S360 in
The parameter optimization unit 6082 carries out the optimization of the external parameter by newly obtaining an evaluation value based on a deviation amount E1 of superimposition region images as an evaluation value provided from the superimposition region deviation amount evaluation unit 6084 and a deviation amount E2 in a projection region as an evaluation value provided from the projection region deviation amount evaluation unit 6085. The evaluation value obtained in this case is, for example, the sum total of the deviation amount E1 and the deviation amount E2 or a weighted sum of the deviation amount E1 and the deviation amount E2. The weighted sum is calculated as w1χ E1+w2×E2, for example. Here, w1 and w2 are weight parameters of the deviation amount E1 and the deviation amount E2. Incidentally, the weight parameters w1 and w2 are obtained based on the areas of the superimposition region images and the projection images, for example. Further, by changing the weight parameters w1 and w2, it is also possible to execute the deviation correction process by using the evaluation exclusively with the deviation amount E1 as the evaluation value provided from the superimposition region deviation amount evaluation unit 6084 (w2=0) or the evaluation exclusively with the deviation amount E2 as the evaluation value provided from the projection region deviation amount evaluation unit 6085 (w1=0).
In the repetitive processing, the parameter optimization unit 6082 needs to recalculate an evaluation value corresponding to the updated external parameter, for which it is necessary to reacquire the deviation amount E1, as the evaluation value provided from the superimposition region deviation amount evaluation unit 6084, and the deviation amount E2, as the evaluation value provided from the projection region deviation amount evaluation unit 6085, corresponding to the updated external parameter. Thus, when the external parameter has been updated, the parameter optimization unit 6082 outputs the updated external parameter to the projection processing unit 603 and thereby reacquires the projection image regarding each camera corresponding to the external parameter. Here, the projection image is the projection image of each camera image acquired from the camera image reception unit 609. The parameter optimization unit 6082 inputs the reacquired projection image regarding each camera to the superimposition region extraction unit 6083, inputs outputted superimposition region images to the superimposition region deviation amount evaluation unit 6084, and reacquires the deviation amount E1 as an evaluation value. Further, the parameter optimization unit 6082 inputs the reacquired projection image regarding each camera to the projection region deviation amount evaluation unit 6085 and reacquires the deviation amount E2 as an evaluation value.
The superimposition region extraction unit 6083 shown in
The superimposition region deviation amount evaluation unit 6084 shown in
The projection region deviation amount evaluation unit 6085 shown in
As described above, with the image processing device 610, the image processing method or the image processing program according to the third embodiment, the deviation of camera images in the synthetic image can be corrected while maintaining the positional relationship among the camera images constituting the synthetic image.
Incidentally, it is also possible to employ methods described in the first embodiment as various processing methods in the third embodiment. Further, the deviation detection and deviation correction processes described in the third embodiment can be applied also to other embodiments.
As shown in
The image processing device 710 executes a process of generating a synthetic image by combining a plurality of camera images captured by a plurality of cameras. The camera image recording unit 701 records a plurality of camera images and a plurality of external parameters corresponding to the plurality of camera images in a storage device (e.g., the external storage device 17 in
Further, when recording an image, the camera image recording unit 701 also records the already set external parameter of the camera 600_k (k=1, . . . , n) in the camera parameter input unit 601 (steps S403 and S405). Furthermore, the camera image recording unit 701 also records the state of the deviation of the camera 600_k provided from the deviation detection unit 606 (e.g., whether deviation exists or not, the deviation amount, the direction of the deviation, and so forth) at the same time (steps S404 and S405). The camera image recording unit 701 may record a mask image. The mask image will be explained in a fifth embodiment which will be described later. Moreover, the camera image recording unit 701 provides the input data selection unit 702 with the camera image, the external parameter, the information indicating the order of the camera image, and so forth as a set of data. The processing of steps S402 to S406 is executed for all the cameras 600_1 to 600_n.
In regard to a camera in which deviation has been detected, from all camera images stored in the camera image recording unit 701 since the time point of the detection of the deviation (e.g., #7 and #8 in
Thereafter, the input data selection unit 702 outputs the camera image selected from all camera images stored in the camera image recording unit 701 since the time point of the detection of the deviation and the image selected from all camera images in the deviation corrected state recorded in the camera image recording unit 701 to the movement amount estimation-parameter calculation unit 607 and the deviation correction unit 608 (step S418 in
When there exists no image in a close condition in all the present camera images acquired from the camera image reception unit 609 or camera images recorded in the camera image recording unit 701 (in the past within some frames from the present time point), the input data selection unit 702 stays on standby until a camera image after the occurrence of the deviation is newly recorded in the camera image recording unit 701 and executes the aforementioned comparison process again while including the aforementioned newly recorded camera image (steps SS415 to S417 in
In regard to the camera to which no deviation has occurred, the input data selection unit 702 selects camera images in situations where no deviation has occurred (e.g., 001, 002, 004, 007 and 008 in
In regard to each camera judged by the deviation detection unit 606 to have the position posture deviation, the movement amount estimation-parameter calculation unit 607 receives the camera image and the reference data (i.e., the reference image and the external parameter) provided from the input data selection unit 702 as the input and calculates the external parameter based on these input data. Except for this feature, the movement amount estimation-parameter calculation unit 607 is the same as that in the third embodiment.
In regard to each camera judged by the deviation detection unit 606 to have the position posture deviation, the deviation correction unit 608 receives the camera image (i.e., the image captured by the camera in the deviated state), the reference image and the external parameter provided from the input data selection unit 702. In regard to each camera not judged by the deviation detection unit 606 to have the position posture deviation, the deviation correction unit 608 receives the camera image and the corresponding external parameter provided from the input data selection unit 702. In the third embodiment, values provided from the camera parameter input unit 601 are used as the external parameter of the camera having no position posture deviation. In contrast, in the fourth embodiment, the external parameter corresponding to the image selected by the input data selection unit 702 is used as the external parameter of the camera having no position posture deviation. However, in the fourth embodiment, the external parameter of the camera having no position posture deviation is not updated in the optimization process similarly to the third embodiment. Except for these features, the deviation correction unit 608 in the fourth embodiment is the same as that in the third embodiment.
As described above, with the image processing device 710, the image processing method or the image processing program according to the fourth embodiment, the deviation correction process and the movement amount estimation process are executed based on images in close conditions, and thus estimation accuracy of the movement amount or calculation accuracy of the deviation amount evaluation value can be increased. Further, robustness of the correction process can be increased and the condition in which the correction can be executed can be widened.
Except for the above-described features, the fourth embodiment is the same as the third embodiment. The deviation correction process and the movement amount estimation process described in the fourth embodiment can be applied also to other embodiments.
As shown in
The hardware configuration of the image processing device 810 is the same as that shown in
When the inputted camera image includes a masked region, the projection processing unit 603 shown in
When recording an image, the camera image recording unit 701 also records (i.e., stores) the already set external parameter of the camera in the camera parameter input unit 601 (steps S402, S403 and S405). Further, when recording an image, the camera image recording unit 701 also records the state of the deviation of the camera provided from the deviation detection unit 606 (steps S402, S404 and S405).
Furthermore, the camera image recording unit 701 inputs the image from each camera and the external parameters set to the camera parameter input unit 601 to the mask image generation unit 703 and acquires the mask image of each camera (step S501). When recording a camera image, the camera image recording unit 701 records the mask image provided from the mask image generation unit 703 while associating the mask image with the camera image (step S405).
Moreover, the camera image recording unit 701 outputs the data to be recorded (the camera image, the external parameter, the mask image, an ordinal number or time stamp, and so forth) to the input data selection unit 702 as a set of data. The data to be recorded are, for example, the camera image, the external parameter, the mask image, the ordinal number or time stamp, and so forth. The camera image recording unit 701 repeats the processing of the steps S402 to S404, S501 and S405 for all the cameras (step S406).
The initial mask image output unit 7033 shown in
The superimposition region mask image output unit 7035 shown in
The differential mask image output unit 7032 shown in
The differential mask image output unit 7032 shown in
While the differential mask image output unit 7032 may calculate the difference between received camera images, it is also possible to first transform the received camera images into projection images and calculate the difference between the projection images. In this case, the differential mask image output unit 7032 makes the projection processing unit 603 transform the input images into projection images based on the inputted camera images and camera parameters, calculates the difference between the projection images (
An integrated mask generated by the mask image integration processing unit 7036 shown in
The input data selection unit 702 shown in
(U1) When outputting the selected image (in the deviated state), the reference image and the external parameter in regard to a camera having the position posture deviation to the movement amount estimation-parameter calculation unit 607 and the deviation correction unit 608a, the input data selection unit 702 also outputs a mask image associated with the reference image and the external parameter.
(U2) When selecting images in close conditions, the input data selection unit 702 finds the images in close conditions by applying the mask image associated with the reference image and the external parameter. Namely, this process is a process of limiting an image range to be considered when obtaining the images in close conditions.
Except for these features, the input data selection unit 702 shown in
In regard to each camera judged by the deviation detection unit 606 to have the position posture deviation, the movement amount estimation-parameter calculation unit 607 receives the camera image, the reference image, the external parameter and the mask image provided from the input data selection unit 702 (step S521 in
In regard to each camera judged by the deviation detection unit 606 to have the position posture deviation, the deviation correction unit 608a shown in
When a mask region exists in an input image (i.e., a projection image, and a superimposition region image), the projection region deviation amount evaluation unit 6085 and the superimposition region deviation amount evaluation unit 6084 exclude the part from the target of the comparison process. When a mask region exists in the projection image provided from the projection processing unit 603, the superimposition region extraction unit 6083 extracts the superimposition region while maintaining the mask region and outputs the extracted superimposition region to the superimposition region deviation amount evaluation unit 6084.
A mask application unit 6086 executes the following processes (V1) and (V2):
(V1) The mask application unit 6086 receives selected reference data (i.e., the reference image and the external parameter) and the mask image corresponding to the reference data as the input, performs the mask process on the reference image, and outputs the masked reference image and the corresponding external parameter to the projection processing unit 603.
(V2) When an object exists in the mask region in the selected reference image, the mask application unit 6086 detects the object. Thereafter, if the detected object exists in the inputted camera image (camera image in the deviated state), the mask application unit 6086 outputs the image in which the object has been masked.
Except for the above-described features, the deviation correction unit 608a is the same as the deviation correction unit 608 in the fourth embodiment.
As described above, with the image processing device 810, the image processing method or the image processing program according to the fifth embodiment, image parts adversely affecting the estimation of the movement amount or the calculation of the deviation amount evaluation value are excluded from the images used for the deviation correction process, and thus the estimation accuracy of the movement amount or the calculation accuracy of the deviation amount evaluation value can be increased.
Except for the above-described features, the fifth embodiment is the same as the third or fourth embodiment. The processes for generating and using the mask image described in the fifth embodiment can be applied also to other embodiments.
As shown in
The input image transformation unit 911 classifies each of a plurality of camera images into one of a plurality of domains based on the states in which the plurality of camera images were captured, classifies each of a plurality of reference images into one of the plurality of domains based on the states in which the plurality of reference images were captured, and performs a transformation process, for causing a state in which the domain of a comparison target camera image among the plurality of camera images and the domain of a comparison target reference image among the plurality of reference images are close, on at least one of the comparison target camera image and the comparison target reference image. Further, also among the plurality of camera image, the input image transformation unit 911 performs a transformation process of causing a state in which the domains of camera images are close. The movement amount estimation-parameter calculation unit 607 estimates the movement amounts of the plurality of cameras based on the comparison target camera images and the comparison target reference images outputted from the input image transformation unit 911 and calculates the plurality of external parameters after the correction corresponding to the plurality of cameras. The transformation process is a process of making the domain of the comparison target camera image and the domain of the comparison target reference image coincide with each other, or a process of reducing the distance between the domains.
The relearning unit 913 generates and updates a learning model, indicating into which of the plurality of domains each of the plurality of camera images should be classified and into which of the plurality of domains the reference image should be classified, based on the plurality of camera images. Based on the learning model, the input image transformation unit 911 executes the classification of each of the plurality of camera images, the classification of each of the plurality of reference images, and the aforementioned transformation process. The relearning unit 913 generates and updates the learning model based on the plurality of camera images recorded by the camera image recording unit 914.
The reference data readout unit 605 shown in
The deviation detection unit 606 shown in
The movement amount estimation-parameter calculation unit 607 shown in
The deviation correction unit 608 shown in
Further, the deviation correction unit 608 performs the transformation of camera images by using the input image transformation unit 911 and calculates the deviation amount by using transformed images obtained as the result of the transformation. Similarly to the third embodiment, the deviation correction unit 608 executes the camera parameter optimization process by using the values evaluated by the projection region deviation amount evaluation unit and the superimposition region deviation amount evaluation unit (i.e., the evaluation values). The former evaluation value is represented as E1 and the latter evaluation value is represented as E2.
When calculating E1, the comparison between the reference image and the present camera image in regard to one camera is made, and thus the input image transformation unit 911 transforms the reference image into the domain to which the camera image provided from the camera image reception unit 609 belongs, or transforms the camera image provided from the camera image reception unit 609 into the domain to which the reference image belongs. The projection region deviation amount evaluation unit executes the calculation of the deviation amount by using the aforementioned images (i.e., performs bird's eye transformation on images and evaluates the deviation amount similarly to the third embodiment).
When calculating E2, the input image transformation unit 911 transforms the image from the correction target camera, the image from an adjacent corrected camera (i.e., camera in the non-deviated state), or both of the images into an appropriate domain. The superimposition region deviation amount evaluation unit executes the calculation of the deviation amount by using the aforementioned images after the transformation (i.e., performs bird's eye transformation on images, extracts the superimposition region, and calculates the deviation amount from the extracted superimposition region images similarly to the third embodiment).
Methods of determining the destination of the domain transformation between different cameras (i.e., transformation for the aforementioned evaluation value E2) are as described in the following (Y1) to (Y3):
(Y1) Previously obtain every domain-to-domain distance in regard to all the domains of the different cameras.
(Y2) Classify each of the image from the correction target camera and the image from the adjacent camera into the domain in regard to each camera and obtain the domain-to-domain distance between the domains of the different cameras.
(Y3) When there exists a domain that decreases the distance between the images based on the distances obtained in the above (Y1) and (Y2), transform the domains of the images from the correction target camera and the adjacent camera into the pertinent domain.
When there exist a plurality of adjacent cameras, domain transformation optimum for each image may be selected. Namely, different domain transformation is performed for each adjacent camera. For example, in the comparison of the domain-to-domain distance of the correction target camera and the adjacent camera (namely, the aforementioned (Y1)), the image similarly in the superimposition region is calculated by transforming the images into a domain “summer and daytime”. In the comparison of the domain-to-domain distance of the correction target camera and the adjacent camera (namely, the aforementioned (Y2)), the image similarly in the superimposition region is calculated by transforming the images into a domain “autumn and daytime”. Except for these features, the deviation correction unit 608 shown in
The camera image recording unit 914 shown in
The input image transformation unit 911 executes a transformation process for transforming at least one of the reference image provided from the reference data readout unit 605 and the camera image provided from the camera image reception unit 609 so as to make these images be in a condition in which the images are close to each other and provides the movement amount estimation-parameter calculation unit 607 with the reference image after the transformation process and the camera image after the transformation process. The “condition in which the reference image and the camera image are close to each other” include, for example, one or more of a condition in which the sunshine situations are close to each other, a condition in which the seasons are close to each other, a condition in which situations regarding the presence/absence of a person are close to each other, etc. For example, when the reference image provided from the reference data readout unit 605 is an image of the daytime and the camera image provided from the camera image reception unit 609 is an image of the nighttime, the input image transformation unit 911 transforms the camera image provided from the camera image reception unit 609 into a camera image in a daytime condition. When the present camera image captured by the camera A is a camera image captured in summer (e.g., camera image in a summer domain in a lower left part of
The image transformation destination determination unit 9111 shown in
For determining the transformation destination, it is necessary to judge to which domain each of the reference image in the reference data provided from the reference data readout unit 605 and the camera image provided from the camera image reception unit 609 belongs, and thus the image transformation destination determination unit 9111 also makes the judgment on to which domain each image belongs. The image transformation destination determination unit 9111 prepares a previously labeled image, that is, a standard image belonging to each domain, and judges the domain based on the similarity level to the standard image (i.e., distance to the image belonging to each domain). For the domain judgment, a machine learning algorithm such as t-SNE (T-distributed Stochastic Neighbor. Embedding) can be used. For example, in cases of classifying images into four domains of early morning, daytime, nightfall and nighttime, the image transformation destination determination unit 9111 previously prepares standard images respectively captured in the early morning, in the daytime, in the nightfall and in the nighttime, and judges the domain to which the reference image or the camera image belongs by obtaining the similarity level between the standard image belonging to each domain and the reference image provided from the reference data readout unit 605 or the camera image provided from the camera image reception unit 609. Incidentally, while the description has been given of an example in which the image transformation destination determination unit 9111 directly obtains the similarity level between the standard image and the reference image or the camera image as above, it is also possible to judge the domain based on the similarity level between an image obtained by convolution of each image (i.e., intermediate data) and an image obtained by convolution of the standard image (i.e., intermediate standard data).
As methods of determining the transformation destination, there are the following methods (Z1) to (Z3), for example:
(Z1) The first determination method is a method in which the reference image provided from the reference data readout unit 605 is transformed into the domain to which the camera image provided from the camera image reception unit 609 belongs. For example, when the reference image is an image of the nighttime and the camera image provided from the camera image reception unit 609 is an image of the daytime, the image transformation destination determination unit 9111 performs the transformation process on the reference image so that the domain to which the reference image belongs changes from the nighttime domain to the daytime domain.
(Z2) The second determination method is a method in which the camera image provided from the camera image reception unit 609 is transformed into the domain of the reference image provided from the reference data readout unit 605. For example, when the camera image provided from the camera image reception unit 609 is an image of the nighttime and the reference image is an image of the daytime, the image transformation destination determination unit 9111 performs the transformation process on the camera image so that the domain to which the camera image provided from the camera image reception unit 609 belongs changes from the nighttime domain to the daytime domain.
(Z3) The third determination method is a method in which the reference image provided from the reference data readout unit 605 and the camera image provided from the camera image reception unit 609 are transformed into a new domain. For example, when the camera image provided from the camera image reception unit 609 is an image of early morning and the reference image is an image of nightfall, the image transformation destination determination unit 9111 transforms the camera image provided from the camera image reception unit 609 from the image of early morning to an image of the daytime (i.e., transforms the domain from the early morning domain to the daytime domain) and transforms the reference image from the image of nightfall to an image of the daytime (i.e., transforms the domain from the nightfall domain to the daytime domain).
As a method of determining the method of the domain transformation, the method of the domain transformation is determined based on the similarity level (e.g., distance) between the reference image provided from the reference data readout unit 605 and the camera image provided from the camera image reception unit 609 and the distances to images respectively belonging to the domains.
In
Further, the input image transformation unit 911 may additionally employ reliability as data used for the correction of each domain in addition to the similarity level (distance) between images and determine the transformation destination based on both of the similarity level and the reliability. For example, since the accuracy of the correction increases in images of the daytime compared to images of the nighttime, the transformation destination is determined dynamically so as to increase the correction accuracy by setting the reliability of the daytime domain higher than the reliability of the nighttime domain.
Furthermore, the input image transformation unit 911 may also be configured to judge the similarity level between the reference image and the camera image based on the direct distance between the images instead of the distance between the domains to which the images belong.
A domain classification learning model-parameter input unit 9115 shown in
Based on the method of the image transformation process provided from the image transformation destination determination unit 9111, the image transformation learning model-parameter input unit 9112 shown in
The reference image transformation processing unit 9113 shown in
The input camera image transformation processing unit 9114 shown in
The learning model-parameter read-in unit 912 shown in
The relearning unit 913 shown in
The image transformation destination determination unit 9111 can make the judgment on whether the selected transformation destination is a suitable transformation destination or not based on the movement amount between the transformed camera image and the transformed reference image, the similarity level between the transformed camera image and the transformed reference image, or both of the movement amount and the similarity level. The estimation of the movement amount is performed by the same process as the process executed by the movement amount estimation-parameter calculation unit 607. For example, the image transformation destination determination unit 9111 can judge that the transformation destination is not suitable when the movement amount between the transformed camera image and the transformed reference image is an outlier. Alternatively, the image transformation destination determination unit 9111 can judge that the transformation destination is not suitable when the similarity level between the transformed camera image and the transformed reference image is lower than a predetermined threshold value.
As described above, with the image processing device 910, the image processing method or the image processing program according to the sixth embodiment, the movement amount estimation-parameter calculation unit 607 estimates the movement amount or calculates the deviation amount evaluation value by using images in conditions close to each other, and thus the estimation accuracy of the movement amount or the calculation accuracy of the deviation amount evaluation value can be increased and optimization accuracy of the camera parameter can be increased.
Further, with the image processing device 910, the image processing method or the image processing program according to the sixth embodiment, even in a period in which images in conditions close to each other have not been recorded (e.g., period within one year from the installation of the cameras in which images of all the seasons of the year have not been acquired), the images in conditions close to each other can be newly generated. Accordingly, the estimation accuracy of the movement amount or the calculation accuracy of the deviation amount evaluation value can be increased.
Except for the above-described features, the sixth embodiment is the same as one of the third to fifth embodiments. The function of transforming the domain to which the camera image belongs described in the sixth embodiment can be applied also to other embodiments.
It is possible to appropriately combine the configurations of the image processing devices according to the first to sixth embodiments described above. For example, the configuration of the image processing device according to the first or second embodiment can be combined with the configuration of the image processing device according to one of the third to sixth embodiments.
1
a-1d: camera, 10: image processing device, 11: processor, 12: memory, 13: storage device, 14: image input interface, 15: display device interface, 17: external storage device, 18: display device, 100: deviation correction unit, 101a-101d: captured image, 102: image recording unit, 103: timing determination unit, 104: movement amount estimation unit, 105: feature point extraction unit, 106: parameter optimization unit, 107: correction timing determination unit, 108: synthesis table generation unit, 109: synthesis processing unit, 110: deviation amount evaluation unit, 111: overlap region extraction unit, 112: display image output unit, 113: outlier exclusion unit, 114: storage unit, 115: external storage unit, 202a-202d, 206a-206d: captured image, 204a-204d, 207a-207d, 500a-500d: synthesis table, 205, 208: synthetic image, 600_1-600_n: camera, 601: camera parameter input unit, 602: synthesis processing unit, 603: projection processing unit, 604: display processing unit, 605: reference data readout unit, 606: deviation detection unit, 607: movement amount estimation-parameter calculation unit, 608, 608a: deviation correction unit, 609: camera image reception unit, 610, 710, 810, 910: image processing device, 611: main processor, 612: main memory, 613: auxiliary memory, 614: image processing processor, 615: image processing memory, 616: file interface, 617: input interface, 6061: similarity level evaluation unit, 6062: relative movement amount estimation unit, 6063: superimposition region extraction unit, 6064: superimposition region deviation amount evaluation unit, 6065: projection region deviation amount evaluation unit, 6066: deviation judgment unit, 6082: parameter optimization unit, 6083: superimposition region extraction unit, 6084: superimposition region deviation amount evaluation unit, 6085: projection region deviation amount evaluation unit, 701: camera image recording unit, 702: input data selection unit, 703: mask image generation unit, 7031: difference purposed camera image recording unit, 7032: differential mask image output unit, 7033: initial mask image output unit, 7034: superimposition region extraction unit, 7035: superimposition region mask image output unit, 7036: mask image integration processing unit, 911: input image transformation unit, 912: learning model-parameter read-in unit, 913: relearning unit, 914: camera image recording unit, 9111: image transformation destination determination unit, 9112: image transformation learning model-parameter input unit, 9113: reference image transformation processing unit, 9114: input camera image transformation processing unit, 9115: domain classification learning model-parameter input unit.
Number | Date | Country | Kind |
---|---|---|---|
PCT/JP2019/005751 | Feb 2019 | JP | national |
This application is a continuation application of International Application No. PCT/JP2019/036030 having an international filing date of Sep. 13, 2019, which claims priority to International Application No. PCT/JP2019/005751 having an international filing date of Feb. 18, 2019.
Number | Date | Country | |
---|---|---|---|
Parent | PCT/JP2019/036030 | Sep 2019 | US |
Child | 17393633 | US |