Spatial-temporal regulation method for robust model estimation

Abstract

A computerized spatial-temporal regulation method for accurate spatial-temporal model estimation receives a spatial temporal sequence containing object confidence mask. A spatial-temporal weight regulation is performed to generate weight sequence output. A weighted model estimation is performed using the spatial temporal sequence and the weight sequence to generate at least one model parameter output. An iterative weight update is performed to generate weight sequence output. A weighted model estimation is performed to generate estimation result output. A stopping criteria is checked and the next iteration iterative weight update and weighted model estimation is performed until the stopping criteria is met. A model estimation is performed to generate model parameter output. An outlier data identification is performed to generate outlier data output. A spatial-temporal data integrity check is performed and the outlier data is disqualified.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

The preferred embodiment and other aspects of the invention will become apparent from the following detailed description of the invention when read in conjunction with the accompanying drawings, which are provided for the purpose of describing embodiments of the invention and not for limiting same, in which:

FIG. 1 shows the processing flow for the basic spatial-temporal model estimation method;

FIG. 2 shows the processing flow for the spatial-temporal regulation method;

FIG. 3A shows the processing flow chart for finding the optimal a value;

FIG. 3B shows the processing flow chart for the tau_estimation( ) function;

FIG. 4 shows the processing flow for the iterative improvement method for model estimation;

FIG. 5 shows the processing flow for the iterative spatial-temporal regulation for model estimation;

FIG. 6 shows the processing flow for the data integrity assurance method.

DETAILED DESCRIPTION OF THE INVENTION

I. Basic Spatial-Temporal Model Estimation

The basic spatial-temporal model estimation processing flow is shown in FIG. 1. As shown in the figure, a spatial temporal sequence 100 is subjected to a computerized model estimation stage 104 to estimate at least one model parameter 102. In one preferred embodiment of the invention, the model estimation is performed using a nonlinear regression method.

II. Spatial-Temporal Regulation for Model Estimation

The Spatial-temporal Model estimation could be improved when using different confidence weights for different time points. In a subcellular assay, it is known that the pixel intensities do not fluctuate rapidly. If the intensity value of the pixel at a time point is significantly different from the pixel intensities of the previous and the next time points, the pixel may be given low confidence for model estimation. Spatial-Temporal Regulation (STR) makes use of the distribution within each object of the adjacent frame temporal intensity difference, ΔF_t(x,y), to identify those low-confident pixels. The unreliable pixels are discounted from feature measurements by assigning very low weights, w_t, for that time frame.

The processing flow for the spatial-temporal regulation method is shown in FIG. 2. As shown in FIG. 2, a spatial temporal sequence 100 containing a plurality of image frames enclosing at least one object in each frame is inputted. The object is represented by its confidence mask. The confidence mask represents an object by its associated pixels and the confidence values of the pixels. This is different from the binary mask representation that represents an object only by its associated pixels having “ON” or “OFF” states. The confidence mask can be time dependent. That is, the mask and values are different at different time points (frames), we call the time dependent confidence mask as confidence mask sequence.

The spatial temporal sequence input is processed by a computerized spatial-temporal weight regulation step 204 to generate weight sequence 200 for weighted model estimation 202. The weighted model estimation 202 outputs at least one model parameter 102.

Spatial-Temporal Weight Regulation

In one embodiment of the invention, the spatial-temporal weight regulation method determines weight sequence using spatial-temporal confidence based regulation. The spatial-temporal confidence based regulation method generates a spatial-temporal confidence sequence C_t(x, y) integrating the object confidence mask S_t(x, y) and temporal confidence sequence T_t(x, y).

In one embodiment of the invention wherein the image pixel is 8 bit deep and the temporal confidence sequence, T_t(x,y), is calculated by

T _t(x, y)=255−(255×ΔF_t(x, y))

where the variation of the adjacent frame temporal intensity, ΔF_t(x,y), is defined by

$\Delta F_t\left(x,y\right)=\frac{\sum _{i=-2}^1\frac{I_{t+i}\left(x,y\right)-I_{t+i+1}\left(x,y\right)}{I_{t,\mathrm{avg}}\left(x,y\right)}}{4}$ $I_{t,\mathrm{avg}}\left(x,y\right)=\frac{I_{t-1}\left(x,y\right)+I_t\left(x,y\right)+I_{t+1}\left(x,y\right)}{3}$

Since the temporal confidence, T_t(x,y), has to be greater than 0, we set T_t(x,y)=max(0, T_t(x,y)).For the second frame and the second to last frame, only two intensity variations are involved to get temporal confidence T_t(x,y). Those ordinary skill in the art should recognize that other methods of temporal confidence sequence generation could be applied within the scope of this invention.

In one embodiment of the invention, the integration can be done by multiplication assuming the confidence are independently derived:

C _t(x, y)=S_t(x, y)×T_t(x, y)

Other integration method such as linear combination, division (normalization) can also be used for integration.

The confidence sequence is used to create the weight sequence. In one embodiment of the invention, the weight sequence, w_t, is derived by accumulating the confidence values for the object as follows:

$w_t=\frac{\sum _{\left(x,y\in O\right)}C_t\left(x,y\right)}{\sum _{\left(x,y\in O\right)}C_{\max }\left(x,y\right)}$

where C_max(x, y): the maximum value of C_t.

In this embodiment, w_t is the summation of the confidence for all pixels of the object. The summation is normalized by dividing it by the summation of the maximum spatial-temporal confidence value in the object.

Weighted Model Estimation

In one embodiment of the invention, the modified nonlinear regression method is used for model parameter estimation. The modification allows weighted fitting. This method is general purpose and is applicable to other model estimation not just for the destaining model.

In the destining model example, to find the optimized I(t_i)=Ae^−αti+B, which fits the input signal y_i with the weight w_i, the cost function c is defined as

$c=\frac{1}{N}\sum _i{w_i\left[A{}^{-\alpha t_i}+B-y_i\right]}^2,\mathrm{where}N=\sum _iw_i$

To minimize this cost function, the derivatives to the three parameters are set to zero. That is,

$\frac{\partial c}{\partial A}=0,\frac{\partial c}{\partial B}=0,\frac{\partial c}{\partial \alpha }=0$

The solution are done in three steps:

• • (1) To solve

$\frac{\partial c}{\partial \alpha }=0,$

the α derivative of the cost function c is calculated as follows:

$\frac{\partial c}{\partial \alpha }=\frac{2}{N}\sum _it_i{}^{-\alpha t_i}\left[{\mathrm{Aw}}_i{}^{-\alpha t_i}+{\mathrm{Bw}}_i-y_iw_i\right]=0$

By arranging the formula, we can define a function ƒ(α)

ƒ(α)=A<te^−2αt>_w+B<te^−αt>_w−<yte^−αt>_w

α is determined by solving ƒ(α)=0. This is performed by a nonlinear regression procedure.

• • (2) With given α, we can find B by solving

$\frac{\partial c}{\partial B}=0$

$c=\frac{1}{T}\sum _i{w_i\left[A{}^{-\alpha t_i}+B-y_i\right]}^2$ $\begin{array}{c}\frac{\partial c}{\partial B}=\frac{2}{T}\sum _i\left[{\mathrm{Aw}}_i{}^{-\alpha t_i}+{\mathrm{Bw}}_i-y_iw_i\right]\\ =0\\ =A<{}^{-\alpha t}{>}_w+B+<y{>}_w\\ =0\end{array}\therefore B=<y{>}_w-A<{}^{-\alpha t}{>}_w$

• • (3) With given B=<y>_w−A<e^−αt>_w, we can find A by solving

$\frac{\partial c}{\partial A}=0$

$c=\frac{1}{N}\sum _i{\left[A\left({}^{-\alpha t_i}-<{}^{-\alpha t}{>}_w\right)-\left(y_i-<y{>}_w\right)\right]}^2$ $\begin{array}{c}\frac{\partial c}{\partial A}=\frac{1}{N}\sum _i\left({}^{-\alpha t_i}-<{}^{-\alpha t}{>}_w\right)[A\left({}^{-\alpha t_i}-<{}^{-\alpha t}{>}_w\right)-\\ \left(y_i-<y{>}_w\right)]\\ =0\end{array}$ $A<{\left({}^{-\alpha t_i}-<{}^{-\alpha t}{>}_w\right)}^2{>}_w-<\left({}^{-\alpha t_i}-<{}^{-\alpha t}{>}_w\right)\left(y_i-<y{>}_w\right){>}_w=0\therefore A=\frac{<{}^{-\alpha t}y{>}_w-<{}^{-\alpha t}{>}_w<y{>}_w}{<{}^{-2\alpha t}{>}_w-<{}^{-\alpha t}{>}_w^2}$

The flow chart for finding the optimal α is shown in FIG. 3A and the processing flow chart for the tau_estimation( ) function is shown in FIG. 3B. As shown in the Figures, an initial low and high α values containing f(α)=0 within its range are defined. One of the low or high values is reduced to its half distance value in the next iteration. This allows quick convergence to find the a value that is close to f(α)=0.

III. Iterative Improvement Method for Model Estimation

The model estimation is improved with appropriate temporal confidence values based on their reliabilities. The confidence values are determined iteratively from the previous iteration fitting result. The processing flow for the iterative improvement method for model estimation is shown in FIG. 4. As shown in FIG. 4, the spatial temporal sequence input 100 is processed by a weighted model estimation step 202. The weighted model estimation 202 outputs estimation result 402 using the spatial temporal sequence 100 and the weight sequence 200. The weight sequence 200 is generated and updated iteratively from the estimation result 402 by an iterative weight update step 404. The weight sequence 200 generated from the current iteration is applied to the next iteration weighted model estimation 202. The iterative process continues until a stopping criteria 400 is met. In one embodiment of the invention, the stopping criteria includes the maximum number of iterations. In another embodiment of the invention, the stopping criteria checks the difference of the weight sequence between consecutive iterations and stops when the difference is small. In yet another embodiment of the invention, the stopping criteria checks the difference of the estimation result between consecutive iterations and stops when the difference is small. When the stopping criteria is met (“Yes” 406), the estimation result 402 at the current iteration becomes the model parameter 102 result of the process. Otherwise (“No 408), the iteration continues.

The τ fitting result at one iteration including the input data and the estimated result. The iterative weight update method updated the weights for the data points wherein the data with large errors are given smaller weights. The updated weights will be used for the model estimation in the next iteration.

In one embodiment of the invention, the steps for the iterative improvement method for model estimation are listed as follows:

• • Unweighted fitting is performed at the first iteration.
  • After each fitting iteration, errors between input signal (spatial temporal sequence) and expected signal (sequence) based on the current τ estimation result are used to derive confidence weight sequence for the next iteration.
  • The algorithm for iterative method is described as

τ_curvet,fitting = curvefitting(It(x, y))
for (iteration = 2; iteration<=n; iteration++)
{
w                                                              t                                                                                                                                                                                                                                                    exp                                                                                      {                                                                                                                                    -                                    c                                                                    ·                                                                      ɛ                                    t                                    2                                                                                                                                    (                                                                                                            max                                                                              err                                        t                                                                                                                                                                                        -                                                                              min                                                                                  err                                          i                                                                                                                                                                                      )                                                                                            }                                                                                ,                                                      c                                                                                                                                                                          is                                                                                                                                                                          a                                                                                                                                                                                                        const                              .                                                                                                                                                                                          Where                                                                                                                                                                  ɛ                            t                                                    =                                                                                                                                                                                                                      ∑                                                                          (                                                                              x                                        ,                                                                                  y                                          ∈                                          0                                                                                                                    )                                                                                                                                                                                                                                                                                                                                                              I                                      t                                                                                                                                                  (                                                                              x                                        ,                                        y                                                                            )                                                                                                                                      N                                                            -                                                              τ_curve                                                                  t                                  ,                                  fitting                                                                                                                      
}
τ_curvet,fitting = weighted_curvefitting(It(x, y), wt)

To assign appropriated weight values to each time point, we used exponential function so that it can discriminate higher confidence from lower one effectively. If the error, which is derived from the difference between the actual intensity and the estimated intensity by τ estimation, is smaller, much lower weight value is mapped by the exponential function than the linear function.

IV. Iterative Spatial-Temporal Regulation for Model Estimation

The spatial-temporal regulation method and the iterative improvement method can be combined for improved model estimation as shown in FIG. 5. As shown in FIG. 5, the spatial-temporal weight regulation 500 is applied to generate initial weight sequence 502 for the initial estimation and the iterative weight update 404 is then applied to generate updated weight sequence 200 to improve the initial model estimation through weighted model estimation 202. The weight sequence 202 generated from the current iteration is applied to the next iteration weighted model estimation 202. The iteration continues until a stopping criteria 400 is met. When the stopping criteria is met (“Yes” 406), the estimation result 402 at the current iteration becomes the model parameter 102 result of the process. Otherwise (“No 408), the iteration continues.

V. Data Integrity Assurance

The iterative improvement method has ability to identify and reduce the temporal image frame with large errors by adjusting their weight values. This is a passive approach. It changes the way the data is used but it does not change the data even if they are in error.

The data assurance method of this invention is an active approach. In this approach, if a point with large error is detected and the error is determined as an outlier, the data is disqualified for model estimation. Alternatively, the error of the data is corrected and the corrected data is used for updated model estimation. One large source of measurement error is caused by the spatial positional shift of the object between time frames. This could be due to the movement of the objects and/or the camera and/or the positioning system. Therefore, the data integrity assurance method searches the misplaced objects in the neighborhood and restores them for model estimation update.

The processing flow for the data integrity assurance is shown in FIG. 6. The spatial temporal sequence 100 is used for model estimation 104 and the model estimation result, model parameter 102, is used to identify outlier data 610 by an outlier data identification step 600. The outlier data 610 is subjected to the spatial-temporal data integrity check 602. The outlier status 612 result determines whether the data is a true outlier 604 or it is the result of a measurement error. If it is determined as a legitimate outlier (“Yes” 616), the data is disqualified 614 for model fitting. Otherwise (“No” 618), the measurement error of the data is corrected by a spatial-temporal data collection step 606 and the corrected data 608 is used for updated model estimation 104. Note that the model estimation could also use the spatial-temporal regulation or iterative improvement methods to enhance the results.

Outlier Data Identification

The outlier data identification step identifies the temporal image frames with large error. The error between the estimation result derived from the estimated model parameter and the real value from the spatial temporal sequence is calculated. If the error is larger than a threshold, the error is considered as an outlier, and the data point is subjected to the spatial-temporal data integrity check.

Spatial-Temporal Data Integrity Check

One large source of measurement error is caused by the spatial position shifting of the object between different time-lapse frames. This could be due to the movement of the objects and/or the camera and/or the positioning system. Therefore, in one embodiment of the invention, the spatial-temporal data integrity check first identifies the image frame that introduces the large error. It then checks the temporal variation values of the object around the frame. If the temporal variation values are significantly greater than the average value of the object, the data is potentially not a true outlier. In this case, we can shift and re-measure the object feature for multiple shift positions around this temporal frame. The new feature values after the position shifts are used to further determine the outlier data status.

If at least one of the new values makes the data become a non-outlier, the data is considered not a true outlier and the best (closest to the model) new (shifted) value replaces the old one for model estimation. This is performed by the spatial-temporal data correction step. If all new values are also outliers, the data is considered a true outlier and it is disqualified from the model fitting.

VI. Model Adequacy Regulation

This invention allows a model adequacy regulation step to identify multiple populations when data points that appear to be outliers are in fact members of distinctive populations. When there are many true outliers detected by the iterative method, the model adequacy regulation process performs a new model fitting for the true outliers. If a significant number of previous outliers can be fitted into a new model, the new model is considered a different population.

Our iterative model estimation method performs the model fitting to the dominant model parameter, and leaves the data points with non-dominant model parameter as outliers. By repetition, the result of the model estimation comes closer to the dominant model parameter. Since the initial model parameter estimation is based on the combination of multiple model parameter population it is difficult to detect a new model in the first estimation. However, the iterative method drives estimated model parameter closer to the dominant model parameter value out of the multiple populations after repetition.

The invention has been described herein in considerable detail in order to comply with the Patent Statutes and to provide those skilled in the art with the information needed to apply the novel principles and to construct and use such specialized components as are required. However, it is to be understood that the inventions can be carried out by specifically different equipment and devices, and that various modifications, both as to the equipment details and operating procedures, can be accomplished without departing from the scope of the invention itself.

Claims

1. A computerized spatial-temporal regulation method for more accurate spatial-temporal model estimation comprising the steps of: a) Input a spatial temporal sequence containing object confidence mask;b) Perform spatial-temporal weight regulation using the spatial temporal sequence having weight sequence output;c) Perform weighted model estimation using the spatial temporal sequence and the weight sequence having at least one model parameter output.

2. The spatial-temporal regulation method of claim 1 wherein the spatial-temporal weight regulation method uses spatial-temporal confidence based regulation integrating the object confidence mask and temporal confidence sequence.

3. The spatial-temporal regulation method of claim 2 wherein the temporal confidence sequence is derived from variation of adjacent frame temporal intensity.

4. The spatial-temporal regulation method of claim 2 wherein the weight sequence is derived by accumulating the spatial-temporal confidence values for object.

5. The spatial-temporal regulation method of claim 1 wherein the weighted model estimation uses modified nonlinear regression method.

6. The spatial-temporal regulation method of claim 1 wherein the at least one model parameter is the destaining constant τ.

7. A computerized iterative improvement method for more accurate spatial-temporal model estimation comprising the steps of: a) Input a spatial temporal sequence containing object confidence mask;b) Perform iterative weight update to generate weight sequence output;c) Perform weighted model estimation using the spatial temporal sequence and the weight sequence having estimation result output;d) Perform stopping criteria check and continue the next iteration of iterative weight update and weighted model estimation until the stopping criteria is met;e) Output the estimation result as the model parameter output when the stopping criteria is met.

8. The iterative improvement method of claim 7 wherein the iterative weight update method uses exponential function to discriminate higher confidence from lower one.

9. The iterative improvement method of claim 7 wherein the iterative weight update method uses errors between the input spatial temporal sequence and expected sequence based on the estimation result to generate weight sequence.

10. The iterative improvement method of claim 7 further comprises a spatial-temporal weight regulation step to generate initial weight sequence for weighted model estimation.

11. The spatial-temporal regulation method of claim 10 wherein the spatial-temporal weight regulation method uses spatial-temporal confidence based regulation integrating the object confidence mask and temporal confidence sequence.

12 The spatial-temporal regulation method of claim 11 wherein the temporal confidence sequence is derived from variation of adjacent frame temporal intensity.

13. The spatial-temporal regulation method of claim 11 wherein the weight sequence is derived by accumulating the spatial-temporal confidence values for object.

14. The iterative improvement method of claim 7 wherein the model parameter is the destaining constant τ.

15. A computerized data integrity assurance for more accurate spatial-temporal model estimation comprising the steps of: a) Input a spatial temporal sequence containing object confidence mask;b) Perform model estimation using the spatial temporal sequence having model parameter output;c) Perform outlier data identification using the model parameter and the spatial temporal sequence having outlier data output;d) Perform spatial-temporal data integrity check using the outlier data having outlier status output;e) Disqualify the outlier data if its outlier status is true.

16. The data integrity assurance method of claim 15 further comprises a spatial-temporal data correction step to generate corrected data output.

17. The data integrity assurance method of claim 15 further comprises a model adequacy regulation step to identify multiple populations when data points that appear to be outliers are in fact members of distinctive populations.

18. The data integrity assurance method of claim 15 wherein the outlier data identification step calculates error between estimation result derived from estimated model parameter and real value from the spatial temporal sequence.

19. The data integrity assurance method of claim 15 wherein the spatial-temporal data integrity check step checks temporal variation values of object around the frame with large errors.

20. The data integrity assurance method of claim 15 wherein the model parameter is the destaining constant τ.