Patent Application
20120188118
The present application claims priority of European patent application 11150617.6 filed on Jan. 11, 2011.
The present invention relates to a passive radiometric imaging device and a corresponding method for scanning a scene and reconstructing an image of said scene. Further, the present invention relates to a processing apparatus and a corresponding processing method for use in a passive radiometric imaging device. Still further, the present invention relates to a computer readable medium.
Electromagnetic waves are classified into several types depending on the wave frequency. These waves have been applied to a lot of applications like in-vehicle radar devices for measuring the distance between moving vehicles in order to prevent collisions, in concealed weapon detection, or in detecting malignant cells. Further, improved generation and detection techniques as well as latest improvements in the integration and miniaturization of devices operating at various frequency ranges have created a lot of interest to exploit the properties of this electromagnetic radiation. Examples are milli-meter and sub-millimeter waves (30 GHz to a few THz) which have the ability to penetrate non-metal materials, including plastics, walls, clothes, smoke and fog.
Electromagnetic waves can be used in an active or a passive mode. A passive radiometric imaging system creates images by capturing the electromagnetic radiation emitted by the objects by using a radiometer. Such a passive radiometric imaging system is, for instance, known from US 2007/0221847 A1.
Passive electromagnetic waves are emitted naturally by any object. The strengths of these waves depend on the object itself. These signals are, however, weak and are degraded fast due to both the internal noise factors of the radiometer and the external noise factors of the environment. Therefore, the radiation samples of the passive electromagnetic waves captured by the radiometer are generally degraded by high noise variations.
Conventionally, in passive radiometric imaging devices it is tried to employ image de-noising and enhancement algorithms to reduce the effect of the noisy images obtained from the sensor (i.e. the radiometer). Such de-noising or enhancement algorithms can be based on image wavelets, on the Total Variation principle, on manifold learning algorithms, on filtering schemes like wiener or bilateral filtering, on diffusion algorithms or on image pyramids in combination with extrapolation in the frequency space. Many other techniques exist as a state of the art. However, all of these algorithms are mainly designed as a post-processing step of the noisy radiometer image. In other words, such algorithms do not directly process the radiometric samples when recovering the radiometer image, but they enhance, improve or de-noise an integrated image from all the samples of the radiometer by using an algorithm like one of the just mentioned algorithms.
Further, these techniques can suppress the noise to some extent but the resulting image will suffer from a lot of fluctuations due to the high variation of noise and interference at the sensor, especially in passive radiometers, in which the signal to noise ratio is low.
As a result, the obtained image after reconstruction will be degraded. This will make the applications of the passive radiometric imaging device for object detection in security screening applications a difficult task. This is mainly due to the degraded nature of the image that can result either in false alarms or in some suspicious objects being missed from the screening.
It is an object of the present invention to provide a passive radiometric imaging device and a corresponding method for imaging a scene providing a better image quality compared to known passive radiometric imaging devices and methods.
It is a further object of the present invention to provide a processing apparatus and a corresponding processing method for use in a passive radiometric imaging device, as well as to provide a corresponding computer readable medium.
According to an aspect of the present invention there is provided a passive radiometric imaging device for scanning a scene and reconstructing an image of said scene, said device comprising:
a cost calculation unit configured to calculate costs for the radiation samples of said data sub-set according to a predetermined cost function, said costs indicating the level of noise in the respective radiation sample, and
an optimization unit configured to determine a pixel value as a label value from several label values, a label indicating a radiation sample or a group of radiation samples of the respective data sub-set, by determining an extremum of energy values determined for different radiation samples or different groups of radiation samples of the respective data sub-set by use of an energy function, said energy function forming a sum adding a first summand including the costs of the radiation sample or group of radiation samples, for which the energy value is determined, and a second summand taking account of the costs of at least one radiation sample or group of radiation samples of at least one neighboring data sub-set.
According to further aspects of the present invention there are provided a processing apparatus for use in a passive radiometric imaging device, a corresponding passive radiometric imaging method and a corresponding processing method.
According to still a further aspect of the present invention there is provided a computer readable non-transitory medium having instructions stored thereon which, when carried out on a computer, cause the computer to perform the steps of the processing method proposed according to the present invention.
Preferred embodiments of the invention are defined in the dependent claims. It shall be understood that the claimed processing apparatus, processing method, passive radiometric imaging method and the claimed computer program have similar and/or identical preferred embodiments as the claimed passive radiometric imaging device and as defined in the dependent claims.
The present invention is based on the idea to take account of the favorable property of a radiometric surface which is generally a piece-wise smooth surface. The variation of the surface within an object in the image is smooth. The jumps and the high variations happen at the edges of the scene captured by the radiometer. Hence, according to the present invention all the radiation samples detected by the radiometer are taken into account depending on the sampling rate and the objective image resolution. Further, according to the present invention the properties of the radiometric surface are taken into account when calculating the pixel values of the pixels of the image. Said pixel values may then directly form the image or may be further processed for reconstructing the image, which, preferably, is simultaneously de-noised or enhanced.
Further, according to the present invention an optimization technique designated for radiometric image reconstruction is applied. Starting from the above explained general idea an optimization algorithm is provided, which can, for instance, be considered as a labeling problem, choosing a label value for each pixel of the image out of several label values. The invention includes deriving the necessary cost functions needed to reconstruct the radiometric images from the raw radiation samples of the radiometer. Further, an energy function is established to be used by the optimization for determining energy values associated with all the label values, from which an extremum (in particular a minimum or maximum) is searched. Said extremum corresponds to the sought label for each pixel or the reconstructed radiometric image. In the sum formed by said energy function two terms are added, in particular a first summand including the costs of the radiation sample or group of radiation samples (being assigned with a label), for which the energy value is determined, and a second summand taking account of the costs of at least one radiation sample or group of radiation samples (being assigned with a label) of at least one neighboring data sub-set.
In preferred embodiments the optimization can be done locally, per column and/or row (i.e. per scanline) or globally, i.e. it can be controlled if and of how many neighboring data sub-sets the costs of radiation samples or groups of radiation samples are taken into account when calculating the energy value for each label by use of the energy function. Further, various embodiments exist for determining the energy values for all labels and the extremum of all energy values corresponding to the reconstructed image. Preferred embodiments use belief propagation or dynamic programming.
These and other aspects of the present invention will be apparent from and explained in more detail below with reference to the embodiments described hereinafter. In the following drawings
In the following, details and different embodiments of the invention are explained.
Generally, the radiometer 20 comprises a single radiometer unit by which the scene is scanned spot by spot. Thus, depending on the integration time of the radiometer unit, from each spot one or more radiation samples are obtained. For scanning over the scene 1 the radiometer 20, or at least its antenna, is mechanically moved along a predetermined trajectory, e.g. in a meandering scheme, over the scene 1. In other embodiments, the sensitivity profile of the radiometer can be electronically moved over the scene 1. In further embodiments, sensing the scene 1 is done by placing a static or a moving reflector next to the radiometer 20. The radiometer 20 may also comprise two or more radiometer units, which can simultaneously detect radiation from two or more radiation spots so that scanning time can be decreased. These and other embodiments of a passive radiometer 20 are generally known and will thus not be explained in more detail in this application.
Generally, pixels of the image to be reconstructed should coincide with the spots 2, at which the radiation samples 4 have been acquired, so that a data sub-set comprises the radiation samples detected at the spot, which coincides with the pixel, and the radiation samples of neighboring spots. In practical implementations this condition is not always completely satisfied, thus creating a noisy image if the samples are simply integrated. This also proofs that there is a need for a solution as proposed by the invention to compensate for this disadvantage.
Therefore, the data set of radiation samples 4, which may be stored in a storage unit (not shown), is processed by a processing unit 30 for subsequently determining their contribution to the reconstruction of each of the pixel values 6a of pixels of the image to be reconstructed. It shall be noted that the determination of the pixel values 6a can be done serially, in parallel, or in a mixed structure.
The determined pixel values 6a either directly represent the pixels of the image, which can thus be, for instance, displayed on a monitor. Alternatively or in addition, a post-processing of the determined pixel values 6a may optionally be performed, e.g. using image processing tools such as interpolation or de-blurring.
Thus, the constructed image 6 may simply be a depiction of the determined pixel values of all the pixels. However, other embodiments for post-processing the determined pixel values and/or the image are possible, and it is, for instance, also possible to generate an image from more or less pixel, wherein more pixels may be generated by known interpolation of available pixel values 6a and wherein less pixels may be generated by combination and/or averaging of pixel values 6a generated by the processing unit 30.
Since, as mentioned above, according to the present invention the properties of the radiometric surfaces are taken into account when reconstructing the image and since directly the (raw) radiation samples are processed and not the integrated image, the obtained images have a better quality than images obtained with conventional passive radiometric imaging devices and methods. In particular, the images have more contrast, less noise and the shapes of the objects are much more preserved and less varying. This allows a better post-analysis of the radiometric image for various applications, e.g. security applications for weapon detection at an airport.
According to the present invention the processing unit 30 is configured to subsequently determine pixel values of pixels of the image to be reconstructed, wherein a pixel value of one of said pixels is determined from a data sub-set of radiation samples from said data set, said data sub-set including the radiation samples detected at the spot corresponding to or being closest to the pixel, whose pixel value is to be determined, and/or radiation samples detected at neighboring spots. For this purpose the processing unit 30 comprises a cost calculation unit 40 configured to calculate costs 5 for the radiation samples of said data sub-set according to a predetermined cost function, said costs indicating the level of noise in (also referred to as the degree of noisiness of) the respective radiation sample. Further, the processing unit 30 comprises an optimization unit 50 that receives the determined costs 5 and is configured to determine a pixel value 6a by finding a label value out of several label values. Here, a label value is indicating a radiation sample or a group of radiation samples, e.g. indicating the number of the radiation sample or the number of the group of radiation samples within a data sub-set. The label value and hence the reconstructed value of the pixel is obtained by determining an extremum of energy or cost values corresponding to the label values of the different radiation samples or different groups of radiation samples of the respective data sub-set using an energy function, said energy function forming a sum adding a first summand including the costs of the radiation sample or group of radiation samples, for which the energy value is determined, and a second summand taking account of the costs of at least one radiation sample or group of radiation samples of at least one neighboring data sub-set.
A first embodiment of a passive radiometric imaging device 10a according to the present invention is shown in
Then, depending on the image resolution to be reconstructed and the sampling rate of the radiometer 20, the radiation samples 4 are mapped into a sample space image (SSI) using a SSI former 31. An example SSI 7 is shown in
Once the SSI 7 is formed, the aim of the next processing steps becomes finding the piece-wise smooth curve (which can also be regarded as an image) within the radiation samples of the SSI 7 that best describes the scene under consideration from all the radiation samples 4 in the SSI 7. Thereby, it is kept in mind, however, that the radio-metric scene 1 is a piece-wise smooth surface. This is justified since the radiation emitted from any real-life object has smooth variations within the geometric edges. The abrupt jumps or discontinuities can generally only be found at the edges of the object; therefore, each piece (surface) is smooth and this is why it is designated by a piecewise-smooth surface. Therefore, in order to construct or reconstruct (both terms are interchangeably used throughout this application, and using one these terms shall not be understood such that the meaning of the other term is excluded) the image according to the present invention, this constraint will be taken into account in the reconstruction process, which is one of the elements, by which the invention distinguishes from the known methods. Preferred embodiments of the invention additionally take the corresponding samples of the pixel under consideration along with the samples of other pixels inside the SSI into account as will be explained below.
To reconstruct a pixel of the radiometric image, an optimization algorithm is used to choose a label for each pixel from a set of label values. For that, costs arc computed for each label using the corresponding samples of the pixel in the SSI 7 under consideration, i.e. all the samples obtained at the spot, whose position corresponds to the position of the pixel or is closest to the position of the pixel as one preferred embodiment. Depending on the type of the optimization algorithm used, i.e. local, scanline or global (which will be explained below in more detail), it might be necessary to also use the samples of the neighboring spots (pixels) when computing the costs. Therefore, it is preferred according to embodiments of the present invention to take the samples of the pixel to be reconstructed and the neighboring samples into account when reconstructing the pixel value with local or scanline based optimization algorithms. Thus, a patch selector 32 is provided for assigning a patch 8 for each pixel, when the costs of the pixel assuming different label values is to be computed. In one preferred embodiment, the patch can be the samples of the pixels itself when using a global optimization algorithm to reconstruct the image. In another preferred embodiment, the patch includes the samples of the pixel where the costs are computed along with all the neighboring samples in the SSI 7. Such a patch 8 can be regarded as a data sub-set of the (complete) data set represented by the SSI 7.
For simplicity of the explanation two cases will be referred to in the following, in particular a first case when the samples of the pixel itself arc used for cost computations and the optimization is done with a global algorithm, and a second case, when the optimization algorithm used is scanline or local-based and the samples of the pixel and those of its neighbors are used to compute the costs. The first case is simple since the patch consists of the samples of the pixel itself. In the second case, the patch should contain the neighboring samples and will be explained next. It should be noted, however, that an expert in the field can apply other combinations according to the present invention depending on the optimization algorithm applied.
In the simplest embodiment, the patch around each pixel can be uniform in size by using a pre-defined fix-window size. An example of such a patch 8 is shown in
Further, in the embodiment shown in
In the previous case, the patch selector 32 then computes the neighbors around each pixel. These neighbors can be referred to also as the optimal neighbors that mostly corresponds to the pixel itself or the segment to which it belongs. In other words, optimal neighbors are considered to be neighbors that the samples of the pixel values are as close together as possible. Example algorithms that can be used for this purpose are variable windows, segment support or similar techniques. In the first one, windows of several sizes and shapes are defined around each pixel. The window that best describes the region around the pixel is then chosen as the patch. In the second one, the SSI is segmented or clustered into several regions. Then, the segment to which the pixel belongs is chosen as the patch. More details about these two algorithms and a summary of schemes that can be applied to determine an optimal patch 8 in the SSI 7 can be found, for example, in F. Tombari, S. Mattoccia, and L. Di Stefano, “Classification and evaluation of cost aggregation methods for stereo correspondence”, IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8, June 2008. It should, however, be noted that these algorithms are not the only ones that can be used to determine the adaptive or optimal patch of each pixel.
The aim of the present invention is to reconstruct, using an optimization algorithm, an image from the raw samples of the radiometer by assuming it as a piece-wise smooth surface. This is done by finding a label from a set of labels for each pixel. Hence, the costs are calculated by the cost calculation unit 40 needed for the optimization by the optimization unit 50 should reflect this property. In addition, the optimization should be able to find its solution by locating the optimal point in the costs corresponding to the optimal label for each pixel. Depending on the costs used for the optimization, the optimal point is defined to be the location where the cost values associated with the labels have an extremum, i.e. a minimum or a maximum. Therefore, any type of costs that satisfies these properties can be used for the optimization according to the present invention.
To calculate the costs, according to an embodiment two criteria should preferably be fulfilled. First, at least one reference or representative point should be calculated for each pixel that should be reconstructed. Second, the distances of the samples to the one or more reference points should be estimated.
Using patches, it is possible to reduce the noise effect (especially when using local or scanline based optimization algorithms) since the cost value of a label that a pixel can have in the optimization will be computed in correspondence with its neighboring pixels. As shown in
Let Xi be the value of a sample in the patch 8, let N be the number of the overall samples in the patch 8 and let R be the value of the representative point. In this preferred embodiment of the invention, the value of one representative point R is the magnitude of the centroid. It is given by
Taking the magnitude of the centroid of the patch 8 is the preferred embodiment of the invention since it is simple and fast. In other embodiments of the invention, as will be explained below, the data can be transformed and will have complex values. In case no such transformations are done to the data, the value of the centroid can be used as an alternative embodiment. It is also possible to use as different embodiments the median or its magnitude; however, the magnitude of the centroid gives an improved performance over them and is the preferred embodiment. It is also possible to apply other methods as alternative embodiments to calculate a representative point of the patch. One alternative embodiment is to take cluster the data in the SSI as generally described in O. Schall, A. Belyaev, and H. P. Seidel, “Sparse meshing of uncertain and noisy surface scattered data,” Technical Report, Max—Planck—Institut Fur Informatik, February 2005. Other alternative embodiment uses the center of the sphere that contains the most of the points of the patch in the SSI as generally described in Y. Ohtake, A. Belyaev, and H. P. Seidel, “An integrating approach to meshing scattered point data,” in ACM Symposium on Solid and Physical Modelling, pp. 61-69, June 2005. However, these embodiments are not the only ones that can be used to calculate the representative point of each patch, but any algorithm with the goal to find a reference point of the patch can generally be applied.
Once at least one representative point from the samples used to compute the costs for each pixel is computed, the cost calculation unit 40a will use it to compute its distances to each of the samples. These distances can be actually the costs that can be used in the subsequent optimization. Therefore, in a preferred embodiment of the invention, the calculation of costs block is actually composed of a reference or a representative point estimator 33 and a distance calculator 34 as shown in
In an embodiment each representative point will be used by an distance calculator 34 to determine the distances to the samples. The distance di of any point Xi in the SSI 7 to the representative point R is given by
d
i
=|R−X
i|. (2)
The distance can also be defined by the squared difference between R and Xi, as in (R−Xi)2. Other equivalent similarity measures can be also be applied. Since the samples are noisy and varying, it is better to regularize the distances so as to prevent extreme values that can bias the computations. An example preferred embodiment is to use the truncated version of the distance {circumflex over (d)}i to enhance the quality of the reconstructed image. By using this preferred embodiment, extreme values of the distances can be cut off. The truncated distance is given by
{circumflex over (d)}
i=min(di,c), (3)
where c is a real constant representing the truncation value. Typical values of c are in the range between 0 and 20 if the data of the radiometer is standardized. Otherwise, the value of c can be adapted depending on the samples output of the radiometer. The truncation value can be preset across the whole patches 8 of the SSI 7 or computed adaptively depending on the statistics of the SSI 7 or the patch 8 itself; one example is the standard deviation. Other variations on how to compute c can also be used according to the invention.
The truncated distance presents the preferred embodiment of the invention, but other embodiments are also possible as long as they try to regularize the distances like the truncated distances do. One alternative embodiment is the Potts model which assigns two values for the distances depending if they are below or above a certain threshold similar to c. Another alternative embodiment is to assign three values of the distances depending on two threshold values of the distances K1 and K2, where K1≦K2. In other words, the distances will have a value V1 if a distance is less than K1, V2 if it is between K1 and K2 and V3 if the distance is larger than K2. More details on these functions can be found in in C. Leung, “Efficient methods for 3D reconstruction from multiple images”, Chapter 4, pp. 109-112, PhD thesis, University of Queensland, February 2006. It is to be noted again, however, that these alternative embodiments are not the only possible ones, but others can be used as long as they tend to regularize the distances.
As mentioned above the distances determined in distance determination unit 34 can be used as costs 5 by the optimization unit 40. However, it also possible to weight the calculated distance before using them in the optimization to reconstruct the image. Depending on the level of noise in the samples of the radiometer and the optimization algorithm used, weighting the distances might help in reaching a solution with less noise. Consequently, another preferred embodiment of the cost calculation unit 40b is shown in
In an embodiment the adaptive distances are now used to determine the weights that will be used to reconstruct the pixel value of the radiometric image by the weighing factor calculation unit 35. The weights that should be used according to this embodiment should preserve the smoothness and the structure of the radiometric image. In other words, it should ensure that the radiometric surface properties are taken into account in the reconstructed pixel values. In general, the weight for a radiation sample is determined based on its similarity to the representative point, in particular by assigning a higher weight with increasing similarity or a lower distance. Therefore, the weights in general should be defined as a function inversely proportional to the distances. The higher the distance is, the more the noise content is in the sample. Hence, the weight of sample in the reconstruction process should be smaller. In general, the weight of the sample should be defined as
where Ki is the weight of the sample in the reconstruction process, Q can be a function of {circumflex over (d)} or a scalar and f({circumflex over (d)}) is a function of {circumflex over (d)}. It should be noted that this equation can be applied directly to the distances d as well if no regularization is done to the distances.
As one preferred embodiment, the general multiquadric spline smoothing kernel, as for instance described in R. K. Beatson and N. Dyn, “Multiquadric b-splines”, Journal of Approximation Theory, pp. 1-24, October 1996, to reconstruct these weights. The multiquadric spline kernel will define the weights K1 in equation (4). It is given at each sample Xi of a patch as a function of the distance di (or preferably its truncated version {circumflex over (d)}i) by
where l is a pre-defined real constant. Lower values of l will result in a smoother image while larger values of l will result in a noisier image. A typical value for l is between 0 and 2 if the data of the radiometer is standardized. However, other values can be used depending on the data output of the radiometer. Further, 2n is the order of the polynomial of the multiquadric spline, n being a positive integer. A typical value of n is in the range between 3 and 7. However, the value of n can be used depending on the data output of the radiometer. Higher values of n will lead to smoother surfaces and vice versa. A is the function defined by
Other smoothing kernels can also be used according to the present invention as long as they take the properties of the radiometric surface into account, when reconstructing the image from the radiation samples. In addition, the embodiment should satisfy equation (3a). One alternative embodiment to the multiquadric splines is the kernel of the bilateral filter which is defined by applying two Gaussian filters to all of the samples of the patch in the SSI. Embodiments with other kernels can be used as long as they take the properties of the radiometric surface into account when reconstructing the image and satisfy equation (3a). Once the weights are computed, they are preferably normalized to obtain the costs 5 as an alternative embodiment to the distances. It is also possible according to embodiment of the present invention to convolve them with the computed distances in equation (2) or equation (3) to obtain the costs as another alternative embodiment to the distances and weights.
An essential aspect of the present invention is the optimization process to construct the radiometric image. The optimization can operate directly on the distances as shown in the embodiment of
The radiometric image can be thought of as a surface where each pixel has single associated value from all the possible label values. The aim of the optimization is to estimate the best sample label L at each pixel in the image that best describes the scene captured by the radiometer, where x is the row number of the pixel and y is its column number. The label L is the sample number in the SSI in one embodiment of the invention or, in another embodiment, refers to other labeling schemes which will be described below. Irrespective of the labeling scheme used, the label is defined to be the best label, hence the reconstructed pixel value of the radiometric image, if it is associated with an extremum in the computed costs and if it satisfies the properties of the radiometric image at the same time. This can be expressed mathematically as follows
where Δ is one possible solution radiometric image of equation (6) from all the possible solutions, E(Δ) is the energy function which corresponds to the possible image solution Δ, C(x, y, L) is the cost of a label using the methods explained above, ψ(L, Li) is a function that describes the relation between the label L at one pixel and the label of one of its neighbors Li and λ is a weighting factor that regularizes the terms in E(Δ).
Hence, as shown in equation (6), according to the present invention an energy function is generally used forming a sum adding a first summand (the first term in equation (6)) including the costs of a pixel assuming the radiation sample or group of radiation samples, for which the energy value is determined, and a second summand (the second term in equation (6)) taking account of the costs of at least one radiation sample or group of radiation samples of at least one neighboring data sub-set.
The term to the left in equation (6) describes the cost of a pixel assuming a label value while the term to the right in equation (6) describes the interactions within the neighboring pixels. Therefore, the right term is the term that should enforce the properties of the radiometric surface according to the present invention when constructing the image and λ is the term that weighs the costs of the samples with respect to the interaction with its neighbors.
The above equation is only a preferred embodiment of the present invention and several variations of this equation exist and can be easily derived by the skilled person. For example, it is possible to place λ in front of the left term in equation (6) or it is possible to optimize equation (6) while considering only one of the above two terms or also by adding an extra term to the equation. It is, however, important to consider the radiometric surface properties in the optimization. λ is typically chosen according to the present invention as a positive real number.
Now, the remaining three issues will be discussed to make the description of the invention complete: The labeling scheme that defines the number of labels L where E(Δ) should be optimized, the function ψ(L, Li) that describes the interaction between each pixel in the image with its neighbors and the optimization algorithm that can be used to reconstruct the radiometric image.
The labeling scheme describes where equation (6) should be optimized in order to construct the radiometric image 6 as shown in
In a preferred embodiment the samples of the radiometer are grouped into several groups. Let M, where 0<M<N and N is the total number of samples for each pixel, be the number of these groups. The optimization according to equation (6) will correspond in this case to finding the optimal label L for each pixel such that 0≦L≦M.
The preferred embodiment for grouping the samples according to the present invention is to perform a uniform grouping the computed distances. The uniform grouping is a simple and efficient solution for the labeling problem. It consists of sub-dividing the computed distances into different bins according to the order of appearance and the number of groups required. An example is shown in
It should be noted, however, that any algorithm that groups the samples of the pixel is possible to use and can present an alternative embodiment to group the samples in the present invention. Grouping of the samples can be done for example using clustering algorithms like K-means. The latter will cluster the samples of each pixel into M different groups where each in this case belongs to one of the K-means clusters.
Whatever grouping methodology is used, the result of this step is a grouped SSI (GSSI) 7′ in which the optimization of equation (6) should be performed to find the optimal labels. These labels correspond to the possible solutions that each pixel can have in the reconstructed radiometric image 6. If no grouping is performed, the GSSI 7′ is equal to the SSI 7 and the number of groups is equal to the number of the samples for each pixel, i.e. M=N. Otherwise, the number of groups will be M such that 0<M<N.
The right term of equation (6) expresses the interaction between the neighboring pixels. This means that the role of this term is mainly to enforce the properties of the radiometric surface on the reconstructed image. Therefore, it is necessary that the term ψ(L, Li) be chosen according to the properties that are to be enforced on the image to be reconstructed. In general, this term should allow the reconstructed radiometric image to vary smoothly except when there are edges where there are some jumps. Preferably, the term ψ(L, Li) should be proportional to the difference between the label of each pixel and that of its neighbors. One way to express this mathematically is by writing ψ in the form
ψ(L,Li)=φ(|L−Li|l) (7)
The form of the term ψ(L, Li) presented in equation (7) is the general way to express the interaction among the neighboring pixels according to this invention. These follow the same concept that was applied to equation (3) in the distance calculator. In other words, it is possible to take the Potts energy model which assigns two values for the differences between the labels depending if |L−Li|l is below or above a certain threshold Q. Another alternative embodiment is to assign three values |L−Li|l depending on two threshold values Q1 and Q2, where Q1≦Q2. |L−Li|l will lead in this case to a cost value J1 if its value is less than Q1, |L−Li|l will lead to a cost value J2 if its value is between Q1 and Q2. |L−Li|l will lead to a cost value J3 if its value is larger than Q2. More details on these variations can be found in C. Leung, “Efficient methods for 3D reconstruction from multiple image”, Chapter 4, pp. 80-134, PhD thesis, University of Queensland, February 2006.
As one preferred embodiment of the invention, the term ψ(L, Li) can be defined while respecting equation (7) as follows
ψ(L,Li)=min(|L−Li|,τ) (8)
In this way, the label difference follows a linear model. It increases up to a threshold τ after which it becomes constant. Therefore, the preferred embodiment of equation (6) that shall be optimized according to this invention is
The values of λ and τ that should be used depend on the optimization algorithm used. But in general, these two terms are positive real numbers. The solution or configuration Δ associated with the extremum of the energy, i.e. where E(Δ) is a global extremum, will be the reconstructed radiometric image according to the present invention. But it should be noted that the global extremum depends on the optimization algorithm.
The aim of the optimization algorithm in the present invention is to find the global optimum of equation (6) that corresponds to the radiometric image. Therefore, the aim of the optimization according to the invention is to try to determine the optimal label L for each pixel from the GSSI such that 0≦L≦M while satisfying equation (6). The optimization can be performed according to this invention in three ways: local, scanline and global. The choice among the three optimization schemes depends on the computational power available and the minimum required quality of the radiometric images. The different embodiments will be explained in the following.
The local optimization is the simplest way to calculate the radiometric image according to equation (6). Substantially, it comprises the step of taking the extremum value, i.e. minimum or maximum functions, of the cost defined by equation (6) at each pixel. This can be expressed mathematically in this case as
In other words, the energy function E(Δ) is calculated for each pixel separately and the optimal label L for each pixel will be the one that corresponds to E(Δ) being an extremum. This can be calculated by computing the energy function for each label using equation (10) and finding the one that corresponds to the extremum. In case the costs in equations (2) and (3) are used along with the interaction in equation (8), the optimal label will correspond to the label that results in the minimum value of E(Δ) in equation (10).
Many variations of the local optimization exist. Generally, according to local optimization an energy function is used including a second summand taking account of the costs of at least one radiation sample or group of radiation samples of all directly neighboring data sub-sets.
A preferred embodiment of an optimization unit 50a for performing local optimization is schematically shown in
The advantage of the local optimization is that it can be done usually very fast. To improve the quality of the reconstructed images in local optimization, it is preferred according to a further embodiment that the patch selection unit 32 chooses some patches in the SSI around each pixel to be reconstructed before calculating the costs. In still another embodiment the patch selection unit 32 is incorporated with an algorithm that calculates the optimal patch or segment around each pixel before calculating the costs.
Scanline optimization proposed according to further embodiments means optimizing equation (6) in each scanline of the GSSI. Looking at
SL denotes in equation (11) the scanline. Scanline optimization splits the optimization of equation (6) over all the GSSI into an optimization over line-slices of the GSSI. This means that the optimization is done for each scanline and consequently, the optimal label for each pixel in the reconstructed radiometric image will be dependent on the other pixels of the scanline where the pixel belongs. This illustrates the difference to the local optimization where the result only depends on the pixel itself.
Many variations of the scanline optimization exist. Generally, according to scanline optimization an energy function is used including a second summand taking account of the costs of at least one radiation sample or group of radiation samples of all neighboring data sub-sets of the same row or column.
A preferred embodiment of a scanline optimization unit 50b for performing scanline optimization is schematically shown in
In a preferred embodiment of the optimization according to the present invention the dynamic programming (DP) concept is used. DP in terms of this invention tries to find the optimal path where the cost defined by equation (11) is an extremum. In case the costs in equations (2) and (3) are used along with the interaction in equation (8), the optimal path according to DP corresponds to the minimum-cost path or to the value of E(Δ) in equation (11) when it is at a minimum. From the minimum path, the label of each pixel in the scanline that are optimal according to DP are extracted. This process is repeated until all the scanline slices in the GSSI are processed. More information about DP can be found in Y. Ohta and T. Kanade, “Stereo by intra- and inter scanline search using dynamic programming”, IEEE Transactions on pattern analysis and machine intelligence, vol. 7, 1985 for example. In terms of this invention, DP optimization is composed of two steps. The first one is a forward accumulation stage which builds up a cost volume for each scanline using the costs derived previously for the labels. For that, a matrix M is inititated to 0. Then for each scanline, it is filled with the costs defined in equation (11) as such
M(x,y,L)=C(x,y,L)+min(C(x−1,y,L−1)+λ,C(x−1,y, L),C(x,y,L+1)) (11a)
After filling M with the costs as defined in equation (11a), a backward tracing step follows to extract the optimal label path for each scanline corresponding to where the energy E(Δ) in equation (11) is an extremum. This step is repeated for every scanline and the resulting labels computed optimally for a scanline will be the reconstructed radiometric image once they are stacked together.
It shall be noted that DP is a preferred embodiment of scanline optimization for reconstructing the radiometric image, but other embodiments with scanline optimization exist, which can be applied as an alternative to DP. For example, it is possible to use any variant of a convex optimization like Newton as described in S. Boyd and L. Vandenberghe, “Convex Optimization”, Cambridge University Press, 2009 or any energy optimization method like graph cuts or belief propagation shown in J. Yedidia, W. T. Freeman and Y. Weiss, “Understanding Belief Propagation and its Generalizations”, Technical Report-2001-22, Mitsubishi Electric Research Laboratories, January 2002 defined on a single scanline of the image as alternative embodiments to DP.
Scanline optimization is slower than local optimization, but leads to better quality radiometric images. To obtain radiometric images with even better qualities in scanline optimization, the same improvements that were explained above for local optimization can be applied.
Global optimization proposed according to further embodiments means optimizing equation (6) over the whole GSSI at the same time. Different from local or scanline optimization, global optimization does not split the optimization problem into several smaller ones. The optimal label for each pixel in the reconstructed image will therefore be dependent on all the other pixels. This can be expressed mathematically as follows
To reconstruct the radiometric image using a global optimization concept, it is necessary to optimize equation (12) over the whole GSSI. Good candidates for doing that are similar to what can applied for the scanline optimization, i.e. any convex or energy optimization method but defined over the whole GSSI and not a scanline slice. Many example algorithms are available in the art and some are shown in C. Leung, “Efficient methods for 3D reconstruction from multiple image”, Chapter 4, pp. 80-134, PhD thesis, University of Queensland, February 2006 and S. Boyd and L. Vandenberghe, “Convex Optimization”, Cambridge University Press, 2009.
Many variations of the global optimization exist. Generally, according to global optimization an energy function is used including a second summand taking account of the costs of at least one radiation sample or group of radiation samples of data sub-sets from different rows and/or column, in particular from of all data sub-sets.
A preferred embodiment of a global optimization unit 50c for performing global optimization is schematically shown in
Irrespective of which embodiment used, global optimization results in improved reconstructed radiometric images compared to local or scanlinc optimization, but has a larger computational complexity. When performing such an optimization, it is also possible that the patch selection unit 32 chooses some patches in the SSI before calculating the costs. This is, however, not necessary here opposed to local or scanline optimization (where this is preferred) since the global optimization will consider the effect of all the other pixels when reconstructing the image. Therefore, it is usually sufficient that the costs are calculated using only the samples of the pixel themselves when using a global optimization strategy to reconstruct the radiometric image.
Another preferred embodiment of a global optimization unit 50d for performing global optimization with belief propagation (BP) is schematically shown in
Irrespective of the connectivity assumed on the grid, let p and q be two neighboring nodes in the graph. In order to be consistent with equation (12), let p have the coordinates (xp, yp) in the GSSI and a possible label LP and let q have the coordinates (xq, yq) in the GSSI and a possible label Lq. The message that has to be passed according to this embodiment from node p to node q at iteration t and using equation (12) has the form
where G(p) denotes the neighbors of p in the connected graph and the term gεG(p)g≠q denotes all the neighbors of node p in the graph to the exception of q. The messages are initialized with 0 at the first iteration. After T iterations, a belief vector will be computed at each node or pixel in the graph which has the form
The dimension of this vector is the number of all the possible labels M in the GSSI. Consequently, the label Lq that optimizes individually V(Lq) at each node (or pixel), i.e. extremum of V(Lq), is the optimal label that has to be chosen for each pixel in the reconstructed radiometric image. In addition, this label is the globally optimal label for the pixel that satisfies equation (12) in terms of BP used according to this embodiment for which an optimization unit 50d is depicted in
This optimization unit 50d comprises a GSSI construction unit 51 in which a GSSI is constructed from the SSI. Thereafter, in a message initialization unit the belief propagation messages are initialized to zero. For iteration 1=T, in a message passing unit 61 for every node and for every possible label every message is passed from each node to its neighbors according to equation (13). In an iteration unit 62 for every node the following steps are performed. In all extremum finding unit 63 the extremum of the belief vector at each node is found according to equation (14). Then, in an image update unit 64 the constructed image is updated with the found labels of the extremum.
It shall be noted that the present embodiment of BP is not the only variation that can be used for optimization. There many variations of the BP concept in the state of the art, for instance as described in J. Yedidia, W. T. Freeman and Y. Weiss, “Understanding Belief Propagation and its Generalizations”, Technical Report-2001-22, Mitsubishi Electric Research Laboratories, January 2002. Further, the presented embodiment can be easily exported to other optimization algorithms that solve the inference problem. One example is closely related to BP is Graph Cuts as described e.g. in Y. Boykov, O. Veksler and R. Zabih, “Fast approximate energy minimization via graph cuts”, IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1222-1239, November 2001.
As a result, the optimization algorithms that solve the inference problem, e.g. BP and graph cuts, and all the other convex or energy optimization algorithms, are alternative embodiments to global optimization. Any of these can generally be used to reconstruct the radiometric image according to the present invention. These can also be formulated as a scanline optimization or as a global optimization as described above to reconstruct a radiometric image while not deviating from the general principle of the invention.
In the above, embodiments of the optimization unit have been described which are based on finding for each pixel the optimum label. For this purpose the samples of the SSI are grouped into a GSSI as explained above. In other embodiments, however, no grouping of samples is made, but the number of each sample in the SSI is directly used as a label and the optimum label (i.e. sample number) is searched for each pixel. An embodiment of such an optimization unit 50e is depicted in
The radiometric samples are in general noisy samples due to many internal and external factors and due to the weakness of the passive radiometric waves themselves. In addition, there might be other deteriorating factors like the blurring of the antenna of the radiometer for example. The presented invention can be easily coped with these distortions by taking further additional measures. If the noise under consideration has a multiplicative form, the radiation samples can be transformed with a logarithmic transform before the computations starts. An embodiment of such an imaging device 10 is depicted in
The patch selector unit was set for one pixel only and hence the costs were computed using only the samples of each pixel in the SSI with equation (3). The SSI was grouped via uniform grouping to construct a GSSI which has 5 groups. The preferred embodiment of the invention with BP shown in
It shall be noted that other combinations of the elements described in the above embodiments are also possible according to the present invention. For instance, in the embodiment disclosed in
The invention has been illustrated and described in detail in the drawings and foregoing description, but such illustration and description are to be considered illustrative or exemplary and not restrictive. The invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.
In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single element or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.
A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.
Any reference signs in the claims should not be construed as limiting the scope.
| Number | Date | Country | Kind |
|---|---|---|---|
| 11150617.6 | Jan 2011 | EP | regional |
