Claims
- 1. A method of efficiently encoding a portion of a digital image frame comprised of image segments having arbitrary shapes, wherein each image segment has a boundary and each image segment is comprised of pixels having values, the method comprising:
obtaining a first image frame and a second image frame; determining newly uncovered image segments in the second frame. determining which segments are adjacent to the newly uncovered image segment in the first image frame and designating them as boundary segments; carrying out a reference filling routine; determining which boundary segments are to be used for filling the exposed area and designating them as fill segments; carrying out a predictive filling routine; recalculating if necessary and determining the final set of fill segments.
- 2. The method of carrying out a reference filling routine, the steps comprising:
considering a region of adaptive dimensions around each unfilled pixel at the boundary of the exposed area; determining the statistical distribution of colors for all pixels within the region belonging to each boundary segment;
- 3. The method of claim 2 further comprising determining to which segment each pixel is adjacent.
- 4. The method of claim 2 further comprising calculating a statistical parameter of the color values for each adjacent segment determined from the said statistical distribution of colors.
- 5. The method of claim 2 further comprising calculating the difference between the actual color values of the pixel and the value of the calculated statistical parameter for each boundary segment and identifying the smallest difference value and the segment which provides the smallest difference value.
- 6. The method of claim 2 further comprising filling the pixel with the value of the calculated statistical parameter of the segmentadjacent to the selected pixel that has the smallest difference if this difference is less than a threshold value.
- 7. The method of claim 2 further comprising assigning to the pixel, the segment identifier of the segmentadjacent to the selected pixel that has the smallest difference between the actual color value of that pixel and the calculated statistical parameter if this difference is less than a threshold value.
- 8. The method of claim 2 further comprising leaving the pixel unfilled if the difference between the actual color value of that pixel and the calculated statistical parameter is greater than a threshold value.
- 9. The method of claim 2 further comprising increasing the threshold value if a certain percentage or less of pixels can be filled with the current threshold.
- 10. The method of claim 8 where the percentage of pixels is zero.
- 11. The method of claim 2 further comprising repeating the steps in claims 2 through 7 for each unfilled pixel at the boundary of the exposed area until the entire exposed area is filled.
- 12. The method of claim 2 where statistical distributions of three color components Y, U, V are determined.
- 13. The method of claim 2 where statistical distributions of each component of a multi-spectral image is determined.
- 14. The method of claim 4 where the statistical parameter is the median color value of the statistical distribution.
- 15. The method of claim 4 where the statistical parameter is any statistical moment of the distribution of color values.
- 16. The method of determining the segments to be used for filling, the steps comprising:
calculating the percentage of pixels within the exposed area that are filled by each of the boundary segments; calculating a parameter that represents the geometric shape of the area filled by each of the boundary segments.
- 17. The method of claim 16 where the parameter representing the geometric shape is a function of the perimeter length, or a function of the area.
- 18. The method of claim 16 where the parameter representing the geometric shape is a function of both the perimeter length and the area.
- 19. The method of claim 16 further comprising selecting a set of tentative fill segments, the steps comprising:
selecting a segment as a tentative fill segment, if their contribution to the filled region is greater than a predetermined contribution of all of the boundary segments; selecting a segment as a tentative fill segment, if their contribution is less than the predetermined contribution but greater than a certain threshold value, and the geometric parameter of the region filled by this segment is within a threshold range; rejecting a segment as a tentative fill segment if one of the above two criteria are not met.
- 20. The method of claim 19 where the predetermined contribution is the average contribution of all of the segments.
- 21. The method of claim 19 where the predetermined contribution is any statistical parameter of all of the segments.
- 22. The method of claim 15 further comprising calculating a function of the perimeter lengths that each of the tentative fill segments contribute to the perimeter of the exposed area.
- 23. The method of claim 22 where the function of the perimeter lengths is the normalized ratio of the squares of the perimeter lengths.
- 24. The method of claim 15 further comprising calculating a function of the areas of the regions contributed by the each of the tentative fill segments.
- 25. The method of claim 24 where the function of the areas is the normalized ratio of the areas.
- 26. The method of claim 15 further comprising calculating a function of the difference between the ratios.
- 27. The method of claim 26 where the function of the difference between the ratios is the sum of the absolute values of the difference between the ratios.
- 28. The method of claim 15 further comprising:
repeating the steps in claims 15 through 26 after excluding the segment where the value obtained by subtracting the normalized area from the normalized length squared is the greatest; determining if the sum of the absolute values of the differences recalculated in this step is smaller than that obtained in the previous calculation of the same parameter.
- 29. The method of claim 15 further comprising repeating the steps in claim 15 until the sum of the absolute values of the difference is greater than that calculated during the previous recalculation.
- 30. The method of claim 15 further comprising selecting the segments used to calculate the lowest sum of absolute values of the differences as the fill segments.
- 31. The method of carrying out a predictive fill routine, the steps comprising:
considering a region of adaptive dimensions around each unfilled pixel at the boundary of the exposed area; determining the statistical distribution of colors for all pixels within the region belonging to each boundary segment;
- 32. The method of claim 20 further comprising determining to which boundary segment each pixel is adjacent.
- 33. The method of claim 20 further comprising calculating a statistical parameter of the color values for each adjacent segment determined from the said statistical distribution of colors.
- 34. The method of claim 20 further comprising assigning to the pixel, the segment identifier of the segment that contributes the greatest number of pixels to the said region of adaptive dimensions around the pixel.
- 35. The method of claim 20 further comprising filling the pixel with the value of the statistical parameter of the segment that contributes the greatest number of pixels to the said region of adaptive dimensions around the pixel.
- 36. The method of claim 20 further comprising repeating the steps in claims 18 through 22 for each unfilled pixel at the boundary of the exposed area until the entire exposed area is filled.
- 37. The method of claim 20 where statistical distributions of three color components Y, U, V are determined.
- 38. The method of claim 20 where statistical distributions each component of a multi-spectral image is determined.
- 39. The method of claim 22 where the statistical parameter is the median color value of the statistical distribution.
- 40. The method of claim 22 where the statistical parameter is any statistical moment of the distribution of color values.
- 41. The method of determining whether the tentative fill segments chosen above sufficiently approximate the actual value of the exposed area, the steps comprising:
calculating the value of a function of the actual values and the reference fill values for regions filled by each of the fill segments; calculating the value of a function of the actual values and the predictive fill values for regions filled by each of the fill segments; determining whether for any segment, the value obtained by subtracting the function of the actual values and the reference fill values, from the function of the actual values and the predictive fill values is above a threshold.
- 42. The method of claim 30 where the function of the actual values and the reference fill values is the average of the absolute values of the color differences between the actual values and the reference fill values.
- 43. The method of claim 30 where the function of the actual values and the predictive fill values is the average of the absolute values of the color differences between the actual values and the predictive fill values.
- 44. The method of determining to undertake a recalculation, if for any segment or segments, the value obtained by subtracting the function of the actual values and the reference fill values, from the function of the actual values and the predictive fill values is above a threshold.
- 45. The method of claim 33 where the function of the actual values and the reference fill values is the average of the absolute values of the color differences between the actual values and the reference fill values.
- 46. The method of claim 33 where the function of the actual values and the predictive fill values is the average of the absolute values of the color differences between the actual values and the predictive fill values.
- 47. The method of recalculating the fill segments and refilling, the steps comprising:
rejecting the segment or segments if, for any segment, the value obtained by subtracting the function of the actual values and the reference fill values, from the function of the actual values and the predictive fill values is above a threshold. Repeating the steps described in claims 2 through 29 after excluding the rejected segment or segments.
- 48. The method of claim 36 where the function of the actual values and the reference fill values is the average of the absolute values of the color differences between the actual values and the reference fill values.
- 49. The method of claim 36 where the function of the actual values and the predictive fill values is the average of the absolute values of the color differences between the actual values and the predictive fill values.
- 50. The method of claim 29 where recalculation is carried out multiple times.
- 51. The method of determining the final set of fill segments, if no recalculation is done, the steps comprising designating the set of fill segments used for the predictive fill as the final fill segments.
- 52. The method of determining the final set of fill segments, if a recalculation is performed, the steps comprising:
calculating the value of a function of the first predictive fill values and the actual values of all of the pixels in the exposed area; calculating the value of a function of the recalculated predictive fill values and the actual values of all of the pixels in the exposed area; designating the set of fill segments used for the first predictive fill as the final fill segments, if the function of the recalculated predictive fill values and the actual values is greater than the function of the first predictive fill values and the actual values; designating the set of fill segments used for the recalculated predictive fill as the final fill segments, if the function of the recalculated predictive fill values and the actual values is less than the function of the first predictive fill values and the actual values.
- 53. The method of claim 41 where the function of the first predictive fill values and the actual values is the average of the absolute values of the color differences between the first predictive fill values and the actual values.
- 54. The method of claim 41 where the function of the recalculated predictive fill values and the actual values is the average of the absolute values of the color differences between the recalculated predictive fill values and the actual values.
- 55. The method of efficiently encoding exposed areas with multiple subregions, the steps comprising:
Determining final fill segments for each of the exterior subregions. Calculating the value of a function that represents the error in the predictive filling for each external subregion. If the function representing the error is less than a threshold value, then filling the exterior subregion. Determining again the identities of the exterior subregions which include the subregions that were not filled in the first pass as well as regions that were previously in the interior of the exposed area and previously had no boundary segments. Filling the exterior subregions using fill segments which may include the subregions that were filled in the previous step.
- 56. The method of claim 44 further comprising repeating the steps in claim 44 until all subregions have been filled.
- 57. The method of claim 44 where the function that represents the error in predictive filling is the average of the absolute value of the difference between the predictive fill values and the actual pixel color values.
- 58. Transmission of the identities of the fill segments to a decoder
- 59. Transmitting the order of filling and the fill segment identities to the decoder if the exposed area is divided into multiple subregions.
- 60. Coupling the information of the fill segments with any set of residue encoder to improve local quality.
CROSS-REFERENCES TO RELATED APPLICATIONS
[0001] This application claims priority from U.S. Provisional Application No. 60/257,844, entitled “Filling by coding”, filed on Dec. 20, 2000, the disclosure of which is hereby incorporated by reference.
[0002] This invention is related to co-pending U.S. patent application Ser. No. 09/591,438, entitled “Method and Apparatus for Digital Image Segmentation,” filed Jun. 9, 2000 (hereinafter ‘Prakash I’), the disclosure of which is hereby incorporated by reference.
Provisional Applications (1)
|
Number |
Date |
Country |
|
60257844 |
Dec 2000 |
US |