Patent Application
20220012308
The present invention relates to a data processing technique to process data of an image and the like.
A tone conversion function to correct or convert the tone of an image, or a geometrical conversion function to correct or transform the shape of an image can be approximated by a global function or a local function that take the coordinate position of an image as a variable and many of those are expressed by a polynomial and a rational polynomial such as projection conversion. However, even in a case where it is possible to approximate them by a local function, it is difficult to appropriately determine the application area of the local function and switch the local function to another function at the area boundary.
Consequently, in data processing in which the tone correction, the shape transformation and the like are performed, generally, a value obtained by evaluating a local function value at a grid point for each certain pixel interval is stored in an LUT (lookup table) and the function value at the grid point read from the LUT is interpolated and used as a correction value or a transformation value.
It is not necessary to calculate the data stored in the LUT real time, and therefore, the data is calculated and generated with an accuracy as high as possible by using high-order polynomials and various functions that require a complicated computation. In contrast to this, it is necessary to perform the interpolation processing and the like of the data read from the LUT real time, and therefore, as described in Japanese Patent Laid-Open No. 2011-081477, generally, linear interpolation or cubic interpolation is performed. Further, Japanese Patent Laid-Open No. H09-511079 has also disclosed a technique relating to quadratic interpolation whose performance and computation scale are intermediate between the linear interpolation and the cubic interpolation described previously.
However, in a case where an attempt is made to perform interpolation with a high-order polynomial, it is necessary to perform multiplication and addition many times and in addition to this, it is necessary to calculate the coefficients of the polynomial real time, and therefore, the circuit configuration becomes a large scale. For example, in a video device, the pixel rate and the operating frequency of the computation circuit are substantially the same and in order to perform the computation for each pixel, the multipliers corresponding to the number of times of multiplication that appear in the computation formula are necessary. Particularly, in recent years, a high-resolution video, such as an 8K×4K video, appears and in the situation in which the pixel rate is higher than the operating frequency of the computation circuit, it is necessary to drive the computation circuits in parallel, and therefore, the circuit scale increases further.
Here, in order to implement data conversion by a function whose nonlinearity is strong, in other words, by a high-order function with a high accuracy by table data and interpolation processing, it becomes necessary to narrow the grid interval of the table data or to perform high-order interpolation processing. However, the narrower the grid interval, the more the memory amount increases and the higher the order of the interpolation processing, the more the computation scale increases. Consequently, there is such a problem that the circuit scale to improve the interpolation accuracy increases and the cost is raised.
Consequently, the applicant of the present application has disclosed in Japanese Patent Laid-Open No. 2020-135283 a technique relating to quadratic interpolation of an uneven grid interval, which enables implementation of data processing by a high-order function with a high accuracy and with a small amount of computation, and which enables miniaturization of the circuit scale and a reduction in cost.
Further, in recent years, an HDR (High Dynamic Range) video has also prevailed and in this case, the bit width of the data that is handled increases, and therefore, this also results in an increase in the circuit scale. Because of this, as a method of reducing the bit width of data while maintaining the High Dynamic Range, it is effective to convert data into a floating point.
However, with the technique described in Japanese Patent Laid-Open No. 2020-135283, in order to convert input data from fixed-point data into floating-point data, additional computation is necessary, and therefore, there is such a problem that the circuit scale increases.
Consequently, in view of the above-described problem, an object of one embodiment of the present invention is to implement a data conversion method by a high-order function with a small-scale circuit.
One embodiment of the present invention is an apparatus including: a determination unit configured to determine, based on input data, a first reference address indicating a plurality of interpolation ranges for calculating quadratic function coefficients and an interpolation distance including the plurality of interpolation ranges; a coefficient computation unit configured to calculate the quadratic function coefficients based on the first reference address; an adjustment unit configured to perform cut-out processing for the interpolation distance represented by bits and output cut-out data as an adjustment interpolation distance; and a computation unit configured to calculate output data by performing a quadratic interpolation computation based on the quadratic function coefficients and the adjustment interpolation distance.
Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.
Hereinafter, with reference to the attached drawings, the present invention is explained in detail in accordance with preferred embodiments. Configurations shown in the following embodiments are merely exemplary and the present invention is not limited to the configurations shown schematically.
The data processing apparatus 110 is a personal computer or the like and as shown in
In the ROM 113, various programs and the like including data processing programs according to the present embodiment are stored. The program stored in the ROM 113 is loaded onto the RAM 112 and executed by the CPU 111. A program is loaded onto the RAM 112 and in addition to this, the RAM 112 is used as a work area of the CPU 111 and input image data and image data and the like being processed are stored temporarily.
The communication I/F 114 performs communication, for example, between the imaging device 130 and the network 131. For example, in a case where the communication I/F 114 receives image data and the like sent from the imaging device 130 or the network 131, the image data is stored temporarily in the RAM 112 under the control of the CPU 111. Then, various kinds of image processing and the like are performed for the image data by the CPU 111 as needed. After that, based on the image data, an image is displayed on the screen of the display device 121 via the display I/F 116, the image data is stored in the storage medium 120 via the storage I/F 115, and so on.
The CPU 111 performs various kinds of processing and control in the data processing apparatus 110 and performs various kinds of image processing including data conversion processing, as will be described later, by executing the data processing programs and the like according to the present embodiment. In the present embodiment, explanation is given on the assumption that the processing of each function unit of the data conversion apparatus 100 shown in
In the following, with reference to
First, by using the flowchart in
At step S1001, the CPU 111 determines whether input data 10 is received. In a case where determination results at this step are affirmative, the processing advance to step S1002. On the other hand, in a case where the determination results at this step are negative, the series of processing is terminated.
At step S1002, a range determination unit 102 calculates and outputs a reference address 12 and an interpolation distance 11 based on the input data 10.
At step S1003, a coefficient computation unit 103 calculates and outputs quadratic function coefficients 14 based on the reference address 12.
At step S1004, a bit width adjustment unit 105 adjusts the bit width of the interpolation distance 11 and outputs as an adjustment interpolation distance 16.
At step S1005, an interpolation computation unit 104 performs quadratic interpolation processing based on the quadratic function coefficients 14 and the adjustment interpolation distance 16 and outputs output data 15.
Next, the operation of each component is explained in detail. The data conversion apparatus 100 determines interpolation range information that is necessary to calculate the quadratic function coefficients 14 in accordance with the input data 10 in the range determination unit 102 and outputs as the reference address 12 in order to perform data interpolation processing. And even as, the range determination unit 102 outputs the interpolation distance 11 that is used for the computation in the interpolation computation unit 104, to be described later. Then, the coefficient computation unit 103 calculates the quadratic function coefficients 14.
In a case of calculating the quadratic function coefficients 14, the coefficient computation unit 103 calculates the quadratic function coefficients 14 based on an LUT, that is, data (referred to as stored reference data 13) stored within a reference value table stored in the reference data storage unit 101. The coefficient computation unit 103 first generates new reference data in a reference data updating unit 201 (see
The input data 10 is image data, data of luminance of an image, coordinates and the like. The quadratic function coefficients 14 are data determined by the tone conversion function that corrects (specifically, converts) the tone of an image and the geometrical conversion function that corrects (specifically, transforms) the shape of an image. Among the reference data that includes the generated new reference data and which is output by the reference data updating unit 201, the data that is used for calculation of the quadratic function coefficients 14 is called updating reference data 21.
Here, only part of the updating reference data 21 is stored in the reference data storage unit 101 as LUT data and the LUT data is supplied to the reference data updating unit 201 as the stored reference data 13. Then, by performing computation based on the stored reference data 13, the reference data updating unit 201 generates reference data that is lacking by a method, to be described later.
Hereinafter, the interval between each discrete grid point of the updating reference data 21 is called a grid interval DIS. Details of the grid interval DIS will be described later and for example, the narrower the grid interval DIS, the higher the data conversion accuracy in interpolation processing is, but the larger the amount of table data that needs to be stored is. On the other hand, the wider the grid interval DIS, the lower the data conversion accuracy is, but the smaller the amount of table data that needs to be stored is.
Here, for example, in a case where the input data 10 is X1, the closest updating reference data 21 corresponding to the data X1 is Y2 and the two pieces (before and after Y2) of the updating reference data 21 in the vicinity of the Y2 as a center are Y1 and Y3. The interpolation range is found by the range determination unit 102 and in the example in
Further, the interpolation computation unit 104 calculates a segment, to be described later, and finds a quadratic approximation curve based on the segment and the middle points M1, M2, and generates interpolation output data Y0 as the output data 15 by performing interpolation processing using the quadratic approximation curve.
The theoretical formula of the interpolation processing performed by the interpolation computation unit 104 is expressed by formula (1) to formula (4). In formula (1), x is obtained by normalizing the relative position of the input data 10 within the interpolation range with the grid interval DIS and takes a value between 0 and 1. Further, Y in formula (1) to formula (4) is the output data 15.
Y=(Ca·x+Cb)·x+Cc formula (1)
Ca=(Y(−1)−2·Y(0)+Y(1))/2 formula (2)
Cb=Y(0)−Y(1) formula (3)
Cc=(Y(1)+Y(0))/2 formula (4)
The interpolation computation unit 104 performs the quadratic interpolation computation based on the quadratic approximation curve expressed by formula (1). Here, the quadratic function coefficients 14 described previously are Ca, Cb, and Cc in formula (1) to formula (4) and the interpolation distance 11 corresponds to x in formula (1).
The quadratic approximation curve is, in the interpolation range A in
That is, at the boundary portion between the interpolation range A and the interpolation range B adjacent to each other, the middle point M2 is the same point and the segment (Y2-Y3) is the same segment. In other words, at the boundary portion between the interpolation range A and the interpolation range B adjacent to each other, the middle point M2 and the segment (Y2-Y3) are used in common, and therefore, the quadratic approximation curves at the boundary portion between both the interpolation ranges are connected smoothly without exception. The conversion characteristic from input data into output data of this quadratic approximation curve is determined by the series of reference data stored as table data without performing the processing corresponding to the processing by the reference data generation unit described in Japanese Patent Laid-Open No. 2020-135283. Consequently, in the data conversion apparatus 100, it is possible to implement data interpolation suitable to gamma conversion that implements a conversion characteristic determined in advance, and the like.
<Outline of Quadratic Interpolation Processing in a Case where Grid Interval DIS is not Constant>
One of the characteristics of the present embodiment is that a case where each value of the grid interval in table data is not constant, that is, a case where the grid interval is uneven is taken into consideration. In the following, a case is explained where the grid interval to be applied varies depending on input data.
The uneven grid interval such as this is suitable to the tone conversion function and so on whose nonlinearity is strong. For example, by applying a narrow grid interval to a low-tone area whose nonlinearity is strong and applying a wide grid interval to a high-tone area whose linearity is strong, it is made possible to reduce the amount of table data that needs be stored while keeping high the tone conversion accuracy of the entire area from the low-tone area to the high-tone area. However, in this case, there is a possibility that a state where the grid interval and the interpolation interval are different at the boundary portion between the interpolation ranges adjacent to each other is brought about, and therefore, it becomes necessary to take steps to smoothly connect the quadratic approximation curves in both the interpolation ranges. The quadratic interpolation in a case where the grid interval is uneven in the present embodiment makes it possible to implement the smooth connection between the quadratic approximation curves in both the interpolation ranges in a case where the grid interval and the interpolation interval are different at the boundary portion between the interpolation ranges adjacent to each other.
In the reference data storage unit 101, only part of the necessary updating reference data 21 is stored as table data, and therefore, reference data that is lacking is generated in the reference data updating unit 201. Specifically, from the characteristic of the quadratic interpolation that is adopted in the present embodiment, two pieces of reference data Y1′ and Y2′ that satisfy formula (5) to formula (9) below are generated in order to smoothly connect the quadratic approximation curves at the boundary portion between different interpolation ranges, that is, at the middle point M2 in
X
2
+X
3
=X
1′
+X
2′ formula (5)
Y
2
+Y
3
=Y
1′
+Y
2′ formula (6)
Formula (5) means that the middle point in each piece of input data is the same and formula (6) means that the middle point in each piece of output data is the same. Consequently, by satisfying formula (5) and formula (6), the coordinates of the middle point M2 that is the boundary portion between the interpolation range A and the interpolation range B are the same, and therefore, even across both the interpolation ranges, the quadratic approximation curves are connected continuously and smoothly.
(Y3−Y2)/(X3−X2)=(Y2−Y1′)/(X2−X1) formula (7)
X
3
−X
2
=DIS_A formula (8)
X
2
−X
1
=DIS_B formula (9)
Formula (7) means that the gradient of the segment connecting reference data adjacent to each other is the same. Consequently, by satisfying formula (7), it is possible to make the same the gradients of the quadratic approximation curves at the coordinates of the middle point M2 that is the boundary portion between the interpolation range A and the interpolation range B.
Y
1′=(½)·[(Y2+Y3)+(DIS_B/DIS_A)·(Y3−Y2)] formula (10)
Y
2=(½)·[(Y2+Y3)−(DIS_B/DIS_A)·(Y3−Y2)] formula (11)
In the present embodiment, by satisfying formula (10) and formula (11), it is possible to make the same the gradients of the quadratic approximation curves at the coordinates of the middle point M2 that is the boundary portion between the interpolation range A and the interpolation range B.
As described above, in the present embodiment, it is possible to reduce the table data. It is desirable for the value of each grid interval in the explanation described previously to be a power of 2 at the time of selecting the table data and at the time of calculating the relative position x, and in order to reduce the computation load of the relational formulas of formula (10) and formula (11), but the value is not limited to this. Further, the grid interval DIS_B shown in
A reference data reading unit 302 acquires, in accordance with the first reference address 31, all the data of the relevant grid interval of the data stored in the reference data storage unit 101 and the data of the different grid interval adjacent to the data as the stored reference data 13. The reference data reading unit 302 outputs all the data of the relevant grid interval described previously as second acquired reference data 34. Further, the reference data reading unit 302 outputs the data of the different grid interval adjacent to the second acquired reference data 34 as first acquired reference data 33. The number of pieces of the data that is acquired by the reference data reading unit 302 and which has the same grid interval is a predetermined number larger than or equal to four.
By using
An extrapolation processing unit 303 performs processing based on formula (10) and formula (11), that is, extrapolation processing and outputs two pieces of extrapolation processing data 35 of Y1′ and Y2′ by using two pieces of data (Y5 and Y6 in
A data selection unit 304 selects three consecutive pieces of data from the extrapolation processing data 35 and the second acquired reference data 34 (that is, the six pieces of data Y1, Y2, Y3′, Y4′, Y5, and Y6′ in
The coefficient data computation unit 204 calculates and outputs the quadratic function coefficients 14 (corresponding to Ca, Cb, and Cc) from the updating reference data 21 (corresponding to Y(−1), Y(0), and Y(1)) based on formula (1) to formula (4). The reference data updating unit 201 may generate the updating reference data 21 of all the interpolation ranges at the time of a computation start signal given by an upper control unit is input.
<Conversion of Input Data into Floating Point>
The present embodiment reduces the computation bit width by converting input data into a floating point and is characterized in that interpolation processing is implemented with a small-scale circuit by reducing the bit width of valid data.
The way the reference data storage unit 101 in the present embodiment stores the table data is shown specifically, and even as, processing relating to conversion of the input data 10 into a floating point, which is performed by the range determination unit 102 and the bit width adjustment unit 105, is shown. The table data in the present embodiment is the same as the contents in
ADD_VAL=DIS_INI*NUM+ADD_INI formula (12)
DIS_INI and NUM are each limited to powers of 2 and in the present embodiment, DIS_INI=1, NUM=4. Further, ADD_INI=0, and therefore, ADD_VAL=4.
For the correction input data, address information is extracted by the concept shown in
DIS=2{circumflex over ( )}(L−2) formula (13)
The first reference address 31 in the present embodiment is obtained by formula (14).
The first reference address 31=4×(L−2) formula (14)
Information on two high-order bits next to the most significant bit information L (20th bit and 19th bit in
As above, the calculation of the first reference address 31, the second reference address 32, and the interpolation distance 11 in the range determination unit 102 is implemented mainly by addition processing, bit shift, and processing to cut out data from a specific bit.
Reducing the bit width of the valid data and reducing the computation bit width of the interpolation computation unit 104 by converting the input data 10 into a floating point are equivalent to reducing the bit width of the interpolation distance 11. That is, in the present embodiment, it is sufficient to perform the conversion of data into a floating point for the interpolation distance 11. In general, in the process of the conversion of data into a floating point, the exponent data is acquired by the bit shift process to search for the most significant bit and the mantissa data is obtained by the process to cut out a predetermined bit width necessary for the interpolation processing from the bit-shifted data. Here, as described in the explanation of the processing of the range determination unit 102, actually, the results of completing the bit shift necessary for the conversion of data into a floating point are obtained as the interpolation distance 11. Because of that, by only the process to cut out a predetermined bit width necessary for the interpolation processing, the conversion of data into a floating point of the interpolation distance 11 is implemented.
In the present embodiment, the bit width of the interpolation distance 11 is adjusted in the bit width adjustment unit 105. Specifically, in the present embodiment, the bit width adjustment unit 105 acquires a predetermined bit width necessary for the interpolation processing from the high-order bits and outputs it as the adjustment interpolation distance 16.
The present invention reduces the computation bit width by the conversion of input data into a floating point and is characterized in that interpolation processing is implemented with a small-scale circuit by reducing the bit width of valid data.
Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.
While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
For example, part of the coefficient computation unit 103 operates by a computation start signal given by an upper control unit and it is possible to reduce power by storing computation results in advance and only by referring to the data stored here while an image is being input.
According to the present invention, it is possible to implement data processing by a high-order function with a high accuracy and with a small amount of computation, and it is made possible to reduce the circuit scale.
This application claims the benefit of Japanese Patent Application No. 2020-118787, filed Jul. 9, 2020 which is hereby incorporated by reference wherein in its entirety.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2020-118787 | Jul 2020 | JP | national |
