Patent Application
20230009580
The present technology relates to an image processing device and an image processing method, and particularly relates to, for example, an image processing device and an image processing method that enable simplification of processing.
In the Joint Video Experts Team (JVET), which is a joint standardization organization of ITU-T and ISO/IEC, standardization work of Versatile Video Coding (VVC), which is a next-generation image encoding system, is in progress for the purpose of further improving encoding efficiency over H.265/HEVC.
In the standardization work of VVC, it has been proposed to perform matrix-based intra prediction (MIP), which is intra prediction using matrix operation, on a prediction block (see, for example, Non Patent Document 1).
In the MIP, parameters of a matrix (weight matrix) obtained by parameter learning are defined, and an operation using (the parameters of) the matrix is performed.
An MIP operation (operation performed by the MIP) is performed using an offset factor fO. The offset factor fO is changed according to MipSizeId representing the matrix size of the matrix and modeId representing the mode number of the MIP in order to increase bit accuracy.
As described above, in the MIP operation, since the offset factor fO is changed according to the MipSizeId and the modeId, it is necessary to set the offset factor fO for each combination of the MipSizeId and the modeId, and the process becomes complicated.
The present technology has been made in view of such a situation, and enables simplification of processing.
A first image processing device of the present technology is an image processing device including an intra prediction unit that, when performing matrix-based intra prediction that is intra prediction using a matrix operation on a current prediction block to be encoded, performs the matrix-based intra prediction using a coefficient related to a sum of change amounts of pixel values and set to a fixed value, and generates a predicted image of the current prediction block, and an encoding unit that encodes the current prediction block using the predicted image generated by the intra prediction unit.
A first image processing method of the present technology is an image processing method including an intra prediction step of, when performing matrix-based intra prediction that is intra prediction using a matrix operation on a current prediction block to be encoded, performing the matrix-based intra prediction using a coefficient related to a sum of change amounts of pixel values and set to a fixed value, and generating a predicted image of the current prediction block, and an encoding step of encoding the current prediction block using the predicted image generated in the intra prediction step.
In the first image processing device and image processing method of the present technology, when performing matrix-based intra prediction that is intra prediction using a matrix operation on a current prediction block to be encoded, the matrix-based intra prediction is performed using a coefficient related to a sum of change amounts of pixel values and set to a fixed value, and a predicted image of the current prediction block is generated. Then, the current prediction block is encoded using the predicted image.
A second image processing device of the present technology is an image processing device including an intra prediction unit that, when performing matrix-based intra prediction that is intra prediction using a matrix operation on a current prediction block to be decoded, performs the matrix-based intra prediction using a coefficient related to a sum of change amounts of pixel values and set to a fixed value, and generates a predicted image of the current prediction block, and a decoding unit that decodes the current prediction block using the predicted image generated by the intra prediction unit.
A second image processing method of the present technology is an image processing method including an intra prediction step of, when performing matrix-based intra prediction that is intra prediction using a matrix operation on a current prediction block to be decoded, performing the matrix-based intra prediction using a coefficient related to a sum of change amounts of pixel values and set to a fixed value, and generating a predicted image of the current prediction block, and a decoding step of decoding the current prediction block using the predicted image generated in the intra prediction step.
In the second image processing device and image processing method of the present technology, when performing matrix-based intra prediction that is intra prediction using a matrix operation on a current prediction block to be decoded, the matrix-based intra prediction is performed using a coefficient related to a sum of change amounts of pixel values and set to a fixed value, and a predicted image of the current prediction block is generated. Then, the current prediction block is decoded using the predicted image.
Note that the image processing device may be an independent device or an internal block constituting one device.
Furthermore, the image processing device can be achieved by causing a computer to execute a program. The program can be provided by transmitting via a transmission medium or by recording on a recording medium.
The scope disclosed in the present description is not limited to the contents of the embodiment, and the contents of the following References REF 1 to REF 11 known at the time of filing are also incorporated herein by reference. That is, the contents described in the following References REF 1 to REF 11 are also the basis for determining the support requirement. Moreover, the documents referred to in References REF 1 to REF 11 are also grounds for determining the support requirement.
For example, a quad-tree block structure, a quad tree plus binary tree (QTBT) block structure, and a multi-type tree (MTT) block structure are within the scope of the present disclosure and satisfy the support requirements of the claims even in a case where they are not directly defined in the detailed description of the invention. Furthermore, for example, technical terms such as parsing, syntax, and semantics are similarly within the scope of the present disclosure and satisfy the support requirements of the claims even in a case where not directly defined in the detailed description of the invention.
REF 1: Recommendation ITU-T H.264 (April 2017) “Advanced video coding for generic audiovisual services”, April 2017
REF 2: Recommendation ITU-T H.265 (February 2018) “High efficiency video coding”, February 2018
REF 3: Benjamin Bross, Jianle Chen, Shan Liu, Versatile Video Coding (Draft 7), JVET-P2001-v14 (version 14—date Nov. 14, 2019)
REF 4: Jianle Chen, Yan Ye, Seung Hwan Kim, Algorithm description for Versatile Video Coding and Test Model 7 (VTM 7), JVET-P2002-v1 (version 1—date Nov. 10, 2019)
REF 5: JVET-N0217-v3: CE3: Affine linear weighted intra prediction (CE3-4.1, CE3-4.2) (version 7—date Jan. 17, 2019)
REF 6: JVET-M0043-v2: CE3: Affine linear weighted intra prediction (test 1.2.1, test 1.2.2) (version 2—date Jan. 9, 2019)
REF 7: JVET-00408-v3: Non-CE3: On rounding shift of MIP (version 3 date Jul. 4, 2019)
REF 8: JVET-00925-v3: Non-CE3: Simplifications of MIP (version 3 date Jul. 4, 2019)
REF 9: Kenji Kondo, Masaru Ikeda, Teruhiko Suzuki, Junyan Huo, Yanzhuo Ma, Fuzheng Yan, Shuai Wan, Yuanfang Yu, CE3-2: On rounding shift of MIP, JVET-P0056-v3 (version 3—date Oct. 3, 2019)
REF 10: Junyan Huo, Haixin Wang, Yu Sun, Yanzhuo Ma, Shuai Wan, Fuzheng Yang. Yuanfang Yu, Yang Liu, Non-CE3: MIP simplification. JVET-P0136-v2 (version 3—date Oct. 4, 2019)
REF 11: Thibaud Biatek, Adarsh K. Ramasubramonian, Geert Van der Auwera, Marta Karczewicz, Non-CE3: simplified MIP with power-of-two offset. JVET-P0625-v2 (version 2—date Oct. 2, 2019)
<Definitions>
To be adjacent includes, for a pixel, not only a case where the pixel is adjacent to the current pixel of interest by one pixel (one line) but also a case where the pixel is adjacent to the current pixel of interest by a plurality of pixels (a plurality of lines). Therefore, an adjacent pixel includes a pixel at a position corresponding to a plurality of pixels continuously adjacent to the current pixel in addition to a pixel at a position corresponding to one pixel directly adjacent to the current pixel. Furthermore, an adjacent block includes a block in a range corresponding to a plurality of blocks continuously adjacent to the current block in addition to a block in a range corresponding to one block directly adjacent to the current block of interest. Moreover, the adjacent block can include a block located in the vicinity of the current block as necessary.
A prediction block means a block (prediction unit (PU)) serving as a processing unit when intra prediction or inter prediction is performed, and also includes a sub-block in the prediction block. In a case where the prediction block, an orthogonal transform block (transform unit (TU)), and an encoding block (coding unit (CU)) are unified into the same block, the prediction block, the orthogonal transform block, and the coding block mean the same block. The orthogonal transform block is a block serving as a processing unit when orthogonal transform is performed, and the encoding block is a block serving as a processing unit when encoding is performed.
The intra prediction mode collectively means variables (parameters) referring to when deriving the intra prediction mode, such as a mode number, a mode number index, a block size of the prediction block, and a size of a sub-block serving as a processing unit in the prediction block when intra prediction is performed.
The matrix-based intra prediction mode (MIP mode) means variables (parameters) referring to when deriving the matrix-based intra prediction mode, such as an MIP mode number, a mode number index, a type of a matrix used when the MIP operation is performed, and a type of a matrix size of a matrix used when the MIP operation is performed.
The parameter is a generic term for data required when encoding or decoding, and is typically a syntax of a bit stream, a parameter set, or the like. Moreover, the parameters include variables and the like used in a derivation process. For the MIP, various data used when the MIP operation is performed corresponds to the parameter. For example, the offset factor fO, a shift amount sW, and (the component of) the weight matrix mWeight[i][j], and the like described in REF 3 correspond to the parameters.
Changing means changing determined contents, for example, changing contents described in the publicly known document based on date before the present application date. Therefore, for example, being different from the content described in Reference REF 3 (values, arithmetic expressions, variables, and the like) corresponds to the change.
In the present technology, identification data for identifying a plurality of patterns can be set as syntax of a bit stream obtained by encoding an image. The bit stream can include identification data identifying various patterns.
In a case where the identification data is included in the bit stream, a decoder that decodes the bit stream can perform processing more efficiently by parsing and referring to the identification data.
<First MIP Method>
The first MIP method is a method of generating a predicted image of the MIP proposed in Reference REF 3 (JVET-P2001-v14).
In the first MIP method, (part of) pixels (pixel values of) predMip[x][y] of a predicted image of a current prediction block that is a prediction block to be encoded or decoded is generated according to the following expression described as Expression (258) in Reference REF 3.
predMip[x][y]=(((ΣmWeight[i][y*predSize+x]*p[i])+oW)>>sW)+pTemp[] (258)
In Expression (258), a variable oW is calculated according to the following expression described as Expression (257) in Reference REF 3.
oW=(1<<(sW−1))−fO*(Σp[i]) (257)
A<<B and A>>B represent that A is shifted leftward and rightward by B bits, respectively.
Each of the variables in Expressions (258) and (257) is described in Reference REF 3, and thus the description thereof is appropriately omitted.
The predMip[x][y] represents (the pixel value of) a pixel whose horizontal position is x and whose vertical position is y in the predicted image. The pixel of the predicted image will be also referred to as a predicted pixel.
According to Expressions (258) and (257), in the first MIP method, MIP is performed using the weight matrix mWeight[i][j], the shift amount sW, and the offset factor fO set according to the MipSizeId and the modeId, and the pixel predMip[x][y] of the predicted image is generated.
Here, in Expressions (258) and (257), p[i] represents a change amount of (the pixel value of) the pixel pTemp[i] in the current prediction block. For example, p[i] is a change amount of the pixel pTemp[i] with reference to an upper left pixel pTemp[0] of the current prediction block, and is expressed by, for example, an expression p[i]=pTemp[i+1]−pTemp[0] or an expression p[i]=pTemp[i]−pTemp[0].
In Expression (257), the offset factor fO is a coefficient relating to the sum Σp[i] of the change amounts p[i] of pixel values.
The shift amount sW is set according to MipSizeId and modeId in accordance with Table 23 described in Reference REF 3. The offset factor fO is set according to MipSizeId and modeId in accordance with Table 24 described in Reference REF 3.
Note that although Reference REF 3 describes that “the variable sW is derived using mipSizeId and modeId as specified in Table 8-5)”, the “Table 8-5” is a clerical error of “Table 23”. Furthermore, although Reference REF 3 describes that “the variable fO is derived using mipSizeId and modeId as specified in Table 8-6”, the “Table 8-6” is a clerical error of “Table 24”.
Meanwhile, Reference REF 10 (JVET-P0136-v2) proposes deleting Table 24 of Reference REF 3, that is, not using the offset factor fO.
Furthermore, Reference REF 11 (JVET-P0625-v2) proposes to use a value represented by a power of two as the offset factor fO defined in Table 24 of Reference REF 3.
As proposed in Reference REF 10, by not using the offset factor fO, it is not necessary to set the offset factor fO according to the MipSizeId and the modeId, so that the MIP processing can be simplified.
However, in a case where the offset factor fO is not used, the range of the weight matrix mWeight[i][j] increases due to the influence of not using the offset factor fO. Consequently, the storage capacity for storing the weight matrix mWeight[i][j] increases.
As proposed in Reference REF 11, by using a value represented by a power of two as the offset factor fO defined in the Table 24, multiplication of the offset factor fO can be performed by shift operation, so that the MIP processing can be simplified.
However, even in a case where a value represented by a power of two is used as the offset factor fO defined in the Table 24, it is necessary to reset the offset factor fO according to the MipSizeId and the modeId similarly to a case where the current Table 24 is used, and the processing becomes complicated.
For example, in a case where the first MIP method is implemented by hardware, a selector for switching the offset factor fO is required, and the circuit scale increases. Furthermore, in a case where the first MIP method is implemented by software, it is necessary to prepare a table in which the offset factor fO represented by a power of two is defined and refer to the table, and the processing speed decreases.
Therefore, in the present technology, the MIP is performed using the offset factor fO set to a fixed value. That is, for example, operations of Expressions (258) and (257) are changed according to the offset factor fO set to the fixed value, and the predicted image of the MIP is generated according to the changed operation.
Hereinafter, the offset factor fO set to a fixed value as appropriate will be also referred to as a fixed offset coefficient. Since the fixed offset coefficient is the offset factor fO set to the fixed value, the fixed offset coefficient is a coefficient relating to the sum Σp[i] of the change amounts p[i] of the pixel value, similarly to the offset factor fO.
<Second MIP Method>
In the second MIP method, the offset factor fO of Expression (257) for generating the predicted image of the MIP proposed in Reference REF 3 is set to a fixed offset coefficient that is a fixed value.
By setting the offset factor fO to the fixed offset coefficient that is a fixed value, Table 24 of Reference REF 3 becomes unnecessary, and the MIP processing can be simplified.
As the fixed value used as the fixed offset coefficient, for example, a value that causes the range of the weight matrix mWeight[i][j] to be in a predetermined range can be employed.
That is, the weight matrix mWeight[i][j] changes from the value described in Reference REF 3 due to the influence of using the fixed offset coefficient. As the fixed offset coefficient, a value that causes the range of the weight matrix mWeight[i][j] after the change to fall within the predetermined range can be employed.
Moreover, as the fixed value used as the fixed offset coefficient, a value represented by a power of two or a value represented by a sum of powers of two can be employed.
In a case where the value represented by a power of two is employed as the fixed offset coefficient, multiplication of the sum Σp[i] of the change amounts p[i] of the pixel value at the time of calculating the variable oW in Expression (257) and the fixed offset coefficient can be performed only by shift operation. Therefore, the MIP processing can be simplified, that is, the calculation amount of the MIP processing can be reduced.
In a case where a value represented by the sum of powers of two is employed as the fixed offset coefficient, multiplication of the sum Σp[i] of the change amounts p[i] of the pixel value at the time of calculating the variable oW in Expression (257) and the fixed offset coefficient can be performed by shift operation and addition. Therefore, the MIP processing can be simplified, that is, the calculation amount of the MIP processing can be reduced.
Note that in a case where a value represented by a power of two is employed as the fixed offset coefficient, the degree of reduction in the calculation amount of the MIP processing is larger than that in a case where the value represented by the sum of powers of two is employed.
However, calculation accuracy can be enhanced, that is, a prediction error of the predicted image of the MIP can be reduced more in a case of employing the value represented by the sum of powers of two as the fixed offset coefficient than in a case of employing the value represented by a power of two.
As the value represented by a power of two, which is the fixed offset coefficient, for example, 32=25 or the like can be employed. As the value represented by the sum of powers of two as the fixed offset coefficient, for example, 48=23+24, 96=26+25, or the like can be employed.
In the second MIP method, the shift amount sW of Expressions (258) and (257) for generating the predicted image of the MIP proposed in Reference REF 3 can be set to a fixed shift amount that is a fixed value.
By setting the shift amount sW to the fixed shift amount that is a fixed value, Table 23 of Reference REF 3 becomes unnecessary, and the MIP processing can be further simplified.
As the fixed shift amount, for example, one of 5, 6, and 7, which are three (types) shift amounts sW defined in Table 23 of Reference REF 3, can be employed.
In the second MIP method, the operation of Expression (258) is changed according to the fixed offset coefficient and the three shift amounts sW defined in Table 23 of Reference REF 3, or according to the fixed offset coefficient and the fixed shift amount, and the predicted image of the MIP is generated according to the changed operation.
That is, in the second MIP method, the weight matrix mWeight[i][j] is changed from the value described in Reference REF 3 according to the fixed offset coefficient and the three shift amounts sW defined in Table 23 of Reference REF 3, or according to the fixed offset coefficient and the fixed shift amount.
Hereinafter, in order to simplify the description, attention will be paid to a case where the fixed offset coefficient and the fixed shift amount are employed.
A predicted pixel predMip[x][y] obtained in a case where the offset factor fO and the shift amount sW are employed, that is, a predicted pixel predMip[x][y] (of the predicted image of the MIP) obtained according to Reference REF 3 will be also referred to as a standard predicted pixel predMip[x][y].
In the second MIP method, for example, the weight matrix mWeight[i][j] described in Reference REF 3 is changed according to the fixed offset coefficient and the fixed shift amount so that a predicted pixel (hereinafter also referred to as a fixed predicted pixel) obtained in a case where the fixed offset coefficient and the fixed shift amount are employed, that is, the predicted pixel predMip[x][y] obtained according to an expression obtained by replacing the offset factor fO and the shift amount sW in Expressions (258) and (257) with the fixed offset coefficient and the fixed shift amount, respectively, has a value approximate to the standard predicted pixel predMip[x][y]. Alternatively, the weight matrix mWeight[i][j] described in Reference REF 3 is changed according to the fixed offset coefficient and the fixed shift amount so that a prediction error of the fixed predicted pixel predMip[x][y] becomes close to a prediction error of the standard predicted pixel predMip[x][y].
For example, 32 which is a value represented by a power of two can be employed as the fixed offset coefficient, and six can be employed as the fixed shift amount. In this case, the changed weight matrix mWeight[i][j] can fall within a range that can be represented by seven bits from zero to 127.
In the second MIP method, a predicted image of MIP is generated according to the calculation including the weight matrix mWeight[i][j] after the change.
Therefore, in the second MIP method, the offset factor fO of Expressions (258) and (257) and further the shift amount sW are fixed regardless of the combination of MipSizeId and modeId, and thus the MIP processing can be simplified. Consequently, it is not necessary to define Table 24 and further Table 23 of Reference REF 3 by the standard, and the standard can be simplified.
Furthermore, for example, in a case where the second MIP method is implemented by hardware, a selector for switching the offset factor fO and the shift amount sW becomes unnecessary, and increase in the circuit scale can be suppressed. Moreover, in a case where the second MIP method is implemented by software, it is not necessary to refer to Table 24 or Table 23, and decrease in processing speed can be suppressed as compared with a case where Table 24 or Table 23 is referred to.
Note that, as the fixed offset coefficient, a value represented by a power of two other than 32 or a value represented by the sum of powers of two other than 48 and 96 can be employed. Furthermore, as the fixed shift amount, a value other than five, six, and seven can be employed.
<MIP in Case Where 48 is Employed as Fixed Offset Coefficient and Five is Employed as Fixed Shift Amount>
In this case, in the MIP, as illustrated in
A of
B of
In the case where the multiplication 48*(Σp[i]) of the sum Σp[i] of the change amounts p[i] of the pixel values and 48 that is the fixed offset coefficient is performed as it is as the multiplication in the operation in which the offset factor fO and the shift amount sW in Expression (257) are replaced with 48 that is the fixed offset coefficient and five that is the fixed shift amount, respectively, the operation illustrated in A of
That is, an operation of an expression in which the offset factor fO and the shift amount sW in Expressions (258) and (257) are replaced with 48 that is the fixed offset coefficient and five that is the fixed shift amount, respectively, is performed.
In the case where the multiplication 48*(Σp[i]) of the sum Σp[i] of the change amounts p[i] of the pixel values and 48 that is the fixed offset coefficient is performed by the shift operation and the addition in the operation in which the offset factor fO and the shift amount sW in Expression (257) are replaced with 48 that is the fixed offset coefficient and five that is the fixed shift amount, respectively, the operation illustrated in B of
That is, the sum=Σp[i] of the change amounts p[i] of the pixel values is calculated. Then, the multiplication 48*(Σp[i]) of the sum Σp[i] of the change amounts p[i] of the pixel values and 48 that is the fixed offset coefficient is performed by the shift operation (sum<<5) and (sum<<4) of the sum and addition of results of the shift operation (sum<<5) and (sum<<4). Thus, the variable oW of Expression (257) is calculated.
Thereafter, in both cases of A and B of
Here, a combination of the MipSizeId and the modeId is represented as (M, m). M represents the MipSizeId, and m represents the modeId. Furthermore, the weight matrix mWeight[i][j] in a case where the fixed offset coefficient and the fixed shift amount are employed will be also referred to as a fixed weight matrix mWeight[i][j].
In
Note that the fixed weight matrix mWeight[i][j] of (M, m)=(0, 0) in the case where 48 is employed as the fixed offset coefficient and five is employed as the fixed shift amount is not limited to the values illustrated in
The fixed weight matrix mWeight[i][j] of (M, m)=(0, 0) can be appropriately changed as long as within a range in which the operation is performed using the fixed offset coefficient and the fixed shift amount and technical effects are exhibited correspondingly. Furthermore, since the range in which the technical effects are exhibited varies depending on the approximation level to be set, the fixed weight matrix mWeight[i][j] of (M, m)=(0, 0) can be changed as appropriate within the range. For example, the change can be made within the range of ±1, and the change can be made within the range of ±3. Moreover, not only all the values can be changed uniformly, but also only some of the values can be changed. Furthermore, it is also possible to individually set a range of values to be changed with respect to existing values.
That is, for example, the fixed weight matrix mWeight[i][j] of (M, m)=(0, 0) can be appropriately changed within a range in which a technical effect such as ensuring predetermined prediction accuracy or more for the predicted image is appropriately exhibited when the operations of Expressions (258) and (257) are performed using the fixed offset coefficient and the fixed shift amount.
Furthermore, the range (degree) in which the technical effect is exhibited changes depending on how much the approximation level is set.
The approximation level means the degree to which the fixed predicted pixel predMip[x][y] is approximated to the true value of the fixed predicted pixel predMip[x][y] or the standard predicted pixel predMip[x][y]. The fixed predicted pixel predMip[x][y] means (a pixel value of) a predicted pixel obtained by MIP using the fixed offset coefficient and the fixed shift amount. The standard predicted pixel predMip[x][y] means a predicted pixel obtained by MIP using the offset factor fO and the shift amount sW described in Reference REF 3.
The fixed weight matrix mWeight[i][j] of (M, m)=(0, 0) can be appropriately changed within a range in which the set approximation level can be maintained. For example, the fixed weight matrix mWeight[i][j] of (M, m)=(0, 0) can be changed within a range of ±1 or within a range of ±3 with reference to the value illustrated in
The change of the fixed weight matrix mWeight[i][j] of (M, m)=(0, 0) can be performed for all of the fixed weight matrices mWeight[i][j] of (M, m)=(0, 0), and can be performed for only a part of the fixed weight matrix mWeight[i][j] of (M, m)=(0, 0).
Moreover, as the range of values for changing the fixed weight matrix mWeight[i][j] of (M, m)=(0, 0), a uniform range can be employed for all of the fixed weight matrices mWeight[i][j] of (M, m)=(0, 0), or an individual range can be employed for each fixed weight matrix mWeight[i][j] of (M, m)=(0, 0).
For example, the range of values for changing the fixed weight matrix mWeight[i][j] of (M, m)=(0, 0) can be individually set for each value of the corresponding weight matrix mWeight[i][j].
The above point similarly applies to the fixed weight matrix mWeight[i][j] other than (M, m)=(0, 0).
<MIP in Case Where 96 is Employed as Fixed Offset Coefficient and Six is Employed as Fixed Shift Amount>
In this case, in the MIP, as illustrated in
A of
B of
In a case where the multiplication 96*(Σp[i]) of the sum Σp[i] of the change amounts p[i] of the pixel values and 96 that is the fixed offset coefficient is performed as it is as the multiplication in the operation in which the offset factor fO and the shift amount sW in Expression (257) are replaced with 96 that is the fixed offset coefficient and six that is the fixed shift amount, respectively, the operation illustrated in A of
That is, the calculation of the expressions in which the offset factor fO and the shift amount sW in Expressions (258) and (257) are replaced with 96 that is the fixed offset coefficient and six that is the fixed shift amount, respectively, is performed.
In a case where the multiplication 96*(Σp[i]) of the sum Σp[i] of the change amounts p[i] of the pixel values and 96 that is the fixed offset coefficient is performed by the shift operation and the addition in the operation in which the offset factor fO and the shift amount sW in Expression (257) are replaced with 96 that is the fixed offset coefficient and six that is the fixed shift amount, respectively, the operation illustrated in B of
That is, the sum=Σp[i] of the change amounts p[i] of the pixel values is calculated. Then, the multiplication 96*(Σp[i]) of the sum Σp[i] of the change amounts p[i] of the pixel values and 96 that is the fixed offset coefficient is performed by shift operation (sum<<6) and (sum<<5) of the sum and addition of results of the shift operation (sum<<6) and (sum<<5). Thus, the variable oW of Expression (257) is calculated.
Thereafter, in both cases of A and B of
<Image Processing System to Which Present Technology is Applied>
The image processing system 10 includes an image processing device as an encoder 11 and an image processing device as a decoder 51.
The encoder 11 encodes an original image that is an encoding target supplied thereto, and outputs a coded bit stream obtained by the encoding. The coded bit stream is supplied to the decoder 51 via a recording medium or a transmission medium that is not illustrated.
The decoder 51 decodes the coded bit stream supplied thereto, and outputs a decoded image obtained by the decoding.
<Configuration Example of Encoder 11>
Note that in the block diagram described below, a description of a line that supplies information (data) required for processing of each block is omitted as appropriate in order to avoid complication of the diagram.
In
The A/D conversion unit 21 A/D-converts an original image (encoding target) of an analog signal into an original image of a digital signal, and supplies the original image to the rearrangement buffer 22 for storage. Note that in a case where the original image of the digital signal is supplied to the encoder 11, the encoder 11 can be configured without the A/D conversion unit 21.
The rearrangement buffer 22 rearranges the frames of the original image from a display order to an encoding (decoding) order according to a group of picture (GOP), and supplies the original image to the operation unit 23, the intra prediction unit 34, and the motion prediction compensation unit 35.
The operation unit 23 subtracts the predicted image supplied from the intra prediction unit 34 or the motion prediction compensation unit 35 via the predicted image selection unit 36 from the original image from the rearrangement buffer 22, and supplies a residual (prediction residual) obtained by the subtraction to the orthogonal transform unit 24.
The orthogonal transform unit 24 performs orthogonal transform such as discrete cosine transform or Karhunen-Loeve transform on the residual supplied from the operation unit 23, and supplies an orthogonal transform coefficient obtained by the orthogonal transform to the quantization unit 25.
The quantization unit 25 quantizes the orthogonal transform coefficient supplied from the orthogonal transform unit 24. The quantization unit 25 sets a quantization parameter on the basis of the target value (code amount target value) of the code amount supplied from the rate control unit 37 and quantizes the orthogonal transform coefficient. The quantization unit 25 supplies the coded data, which is the quantized orthogonal transform coefficient, to the reversible encoding unit 26.
The reversible encoding unit 26 encodes the quantized orthogonal transform coefficient as the coded data from the quantization unit 25 by a predetermined reversible encoding method.
Furthermore, the reversible encoding unit 26 acquires, from each block, encoding information necessary for decoding by the decoding device 170 out of the encoding information related to predictive encoding by the encoder 11.
Here, examples of the encoding information include prediction modes of intra prediction and inter prediction, motion information such as a motion vector, a code amount target value, a quantization parameter, a picture type (I, P, B), filter parameters of the deblocking filter 31a and the adaptive offset filter 41, and the like.
The prediction mode can be acquired from the intra prediction unit 34 or the motion prediction compensation unit 35. The motion information can be acquired from the motion prediction compensation unit 35. The filter parameters of the deblocking filter 31a and the adaptive offset filter 41 can be acquired from the deblocking filter 31a and the adaptive offset filter 41, respectively.
The reversible encoding unit 26 encodes the encoding information by, for example, variable-length encoding or arithmetic encoding such as context-adaptive variable length coding (CAVLC) or context-adaptive binary arithmetic coding (CABAC), or any other reversible encoding method, generates a coded bit stream (multiplexed) including the encoding information after encoding and the coded data from the quantization unit 25, and supplies the coded bit stream to the accumulation buffer 27.
Here, the operation unit 23 to the reversible encoding unit 26 described above constitute an encoding unit that encodes an image, and processing (step) performed by the encoding unit is an encoding step.
The accumulation buffer 27 temporarily accumulates the coded bit stream supplied from the reversible encoding unit 26. The coded bit stream accumulated in the accumulation buffer 27 is read and transmitted at a predetermined timing.
The coded data that is the orthogonal transform coefficient quantized in the quantization unit 25 is supplied to the reversible encoding unit 26 and also supplied to the inverse quantization unit 28. The inverse quantization unit 28 inversely quantizes the quantized orthogonal transform coefficient by a method corresponding to the quantization by the quantization unit 25, and supplies the orthogonal transform coefficient obtained by the inverse quantization to the inverse orthogonal transform unit 29.
The inverse orthogonal transform unit 29 inversely orthogonally transforms the orthogonal transform coefficient supplied from the inverse quantization unit 28 by a method corresponding to orthogonal transform processing by the orthogonal transform unit 24, and supplies the residual obtained as a result of the inverse orthogonal transform to the operation unit 30.
The operation unit 30 adds the predicted image supplied from the intra prediction unit 34 or the motion prediction compensation unit 35 via the predicted image selection unit 36 to the residual supplied from the inverse orthogonal transform unit 29, thereby obtaining and outputting (a part of) the decoded image obtained by decoding the original image.
The decoded image output from the operation unit 30 is supplied to the deblocking filter 31a or the frame memory 32.
The frame memory 32 temporarily stores the decoded image supplied from the operation unit 30 and the decoded image (filtered image), to which the deblocking filter 31a, the adaptive offset filter 41, and the ALF 42 are applied, that is supplied from the ALF 42. The decoded image stored in the frame memory 32 is supplied to the selection unit 33 as a reference image to be used for generating a predicted image at a necessary timing.
The selection unit 33 selects a supply destination of the reference image supplied from the frame memory 32. In a case where intra prediction is performed in the intra prediction unit 34, the selection unit 33 supplies the reference image supplied from the frame memory 32 to the intra prediction unit 34. In a case where inter prediction is performed in the motion prediction compensation unit 35, the selection unit 33 supplies the reference image supplied from the frame memory 32 to the motion prediction compensation unit 35.
The intra prediction unit 34 performs intra prediction (intra screen prediction) using the original image supplied from the rearrangement buffer 22 and the reference image supplied from the frame memory 32 via the selection unit 33. The intra prediction unit 34 selects an optimum prediction mode of intra prediction on the basis of a predetermined cost function, and supplies a predicted image generated from the reference image in the optimum prediction mode of intra prediction to the predicted image selection unit 36. Furthermore, the intra prediction unit 34 appropriately supplies the prediction mode of intra prediction selected on the basis of the cost function to the reversible encoding unit 26 and the like.
The motion prediction compensation unit 35 performs motion prediction using the original image supplied from the rearrangement buffer 22 and the reference image supplied from the frame memory 32 via the selection unit 33. Moreover, the motion prediction compensation unit 35 performs motion compensation according to a motion vector detected by motion prediction, and generates a predicted image. The motion prediction compensation unit 35 performs inter prediction in a plurality of prediction modes of inter prediction prepared in advance, and generates a predicted image from the reference image.
The motion prediction compensation unit 35 selects an optimum prediction mode of inter prediction from among a plurality of prediction modes of inter prediction on the basis of a predetermined cost function. Moreover, the motion prediction compensation unit 35 supplies the predicted image generated in the optimum prediction mode of inter prediction to the predicted image selection unit 36.
Furthermore, the motion prediction compensation unit 35 supplies the optimum prediction mode of inter prediction selected on the basis of the cost function, the motion information such as the motion vector necessary for decoding the coded data that is coded in the prediction mode of inter prediction, and the like to the reversible encoding unit 26.
The predicted image selection unit 36 selects a supply source of the predicted image to be supplied to the operation unit 23 and the operation unit 30 from among the intra prediction unit 34 and the motion prediction compensation unit 35, and supplies the predicted image supplied from the selected supply source to the operation unit 23 and the operation unit 30.
The rate control unit 37 controls the rate of the quantization operation of the quantization unit 25 so that overflow or underflow does not occur on the basis of the code amount of the coded bit stream accumulated in the accumulation buffer 27. That is, the rate control unit 37 sets a target code amount of the coded bit stream so that overflow and underflow of the accumulation buffer 27 do not occur, and supplies the target code amount to the quantization unit 25.
The deblocking filter 31a applies a deblocking filter to the decoded image from the operation unit 30 as necessary, and supplies a decoded image (filtered image) to which the deblocking filter is applied or a decoded image to which the deblocking filter is not applied to the adaptive offset filter 41.
The adaptive offset filter 41 applies an adaptive offset filter to the decoded image from the deblocking filter 31a as necessary, and supplies the decoded image (filtered image) to which the adaptive offset filter is applied or the decoded image to which the adaptive offset filter is not applied to the ALF 42.
The ALF 42 applies the ALF to the decoded image from the adaptive offset filter 41 as necessary, and supplies the decoded image to which the ALF is applied or the decoded image to which the ALF is not applied to the frame memory 32.
<Encoding Processing>
Note that the order of respective steps of the encoding processing illustrated in
In the encoder 11, in step S11, the A/D conversion unit 21 A/D-converts the original image and supplies the original image to the rearrangement buffer 22, and the processing proceeds to step S12.
In step S12, the rearrangement buffer 22 stores the original image from the A/D conversion unit 21, rearranges the original image in the encoding order, and outputs the original image, and the processing proceeds to step S13.
In step S13, the intra prediction unit 34 performs intra prediction (intra prediction step), and the processing proceeds to step S14. In step S14, the motion prediction compensation unit 35 performs inter prediction for performing motion prediction or motion compensation, and the processing proceeds to step S15.
In the intra prediction of the intra prediction unit 34 and the inter prediction of the motion prediction compensation unit 35, cost functions of various prediction modes are calculated, and a predicted image is generated.
In step S15, the predicted image selection unit 36 determines the optimum prediction mode on the basis of each cost function obtained by the intra prediction unit 34 and the motion prediction compensation unit 35. Then, the predicted image selection unit 36 selects and outputs a predicted image in the optimum prediction mode from the predicted image generated by the intra prediction unit 34 and the predicted image generated by the motion prediction compensation unit 35, and the processing proceeds from step S15 to step S16.
In step S16, the operation unit 23 calculates the residual between the encoding target image that is the original image output from the rearrangement buffer 22 and the predicted image output from the predicted image selection unit 36, and supplies the residual to the orthogonal transform unit 24, and the processing proceeds to step S17.
In step S17, the orthogonal transform unit 24 orthogonally transforms the residual from the operation unit 23 and supplies an orthogonal transform coefficient obtained as a result to the quantization unit 25, and the processing proceeds to step S18.
In step S18, the quantization unit 25 quantizes the orthogonal transform coefficient from the orthogonal transform unit 24 and supplies a quantization coefficient obtained by the quantization to the reversible encoding unit 26 and the inverse quantization unit 28, and the processing proceeds to step S19.
In step S19, the inverse quantization unit 28 inversely quantizes the quantization coefficient from the quantization unit 25 and supplies the orthogonal transform coefficient obtained as a result to the inverse orthogonal transform unit 29, and the processing proceeds to step S20. In step S20, the inverse orthogonal transform unit 29 inversely orthogonally transforms the orthogonal transform coefficient from the inverse quantization unit 28 and supplies the residual obtained as a result to the operation unit 30, and the processing proceeds to step S21.
In step S21, the operation unit 30 adds the residual from the inverse orthogonal transform unit 29 and the predicted image output by the predicted image selection unit 36, and generates a decoded image corresponding to the original image that is the target of operation of the residual in the operation unit 23. The operation unit 30 supplies the decoded image to the deblocking filter 31a, and the processing proceeds from step S21 to step S22.
In step S22, the deblocking filter 31a applies the deblocking filter to the decoded image from the operation unit 30 and supplies the filtered image obtained as a result to the adaptive offset filter 41, and the processing proceeds to step S23.
In step S23, the adaptive offset filter 41 applies the adaptive offset filter to the filtered image from the deblocking filter 31a and supplies the filtered image obtained as a result to the ALF 42, and the processing proceeds to step S24.
In step S24, the ALF 42 applies the ALF to the filtered image from the adaptive offset filter 41 and supplies the filtered image obtained as a result to the frame memory 32, and the processing proceeds to step S25.
In step S25, the frame memory 32 stores the filtered image supplied from the ALF 42, and the processing proceeds to step S26. The filtered image stored in the frame memory 32 is used as a reference image from which the predicted image is generated in steps S13 and S14.
In step S26, the reversible encoding unit 26 encodes the coded data that is the quantization coefficient from the quantization unit 25, and generates a coded bit stream including the coded data. Moreover, the reversible encoding unit 26 encodes, as necessary, encoding information such as the quantization parameter used for quantization by the quantization unit 25, the prediction mode obtained by intra prediction by the intra prediction unit 34, the prediction mode and motion information obtained by inter prediction by the motion prediction compensation unit 35, and filter parameters of the deblocking filter 31a and the adaptive offset filter 41, and includes the encoding information in the coded bit stream.
Then, the reversible encoding unit 26 supplies the coded bit stream to the accumulation buffer 27, and the processing proceeds from step S26 to step S27.
In step S27, the accumulation buffer 27 accumulates the coded bit stream from the reversible encoding unit 26, and the processing proceeds to step S28. The coded bit stream accumulated in the accumulation buffer 27 is appropriately read and transmitted.
In step S28, the rate control unit 37 controls the rate of the quantization operation of the quantization unit 25 so that overflow or underflow does not occur on the basis of the code amount (generated code amount) of the coded bit stream accumulated in the accumulation buffer 27, and the encoding processing ends.
<Configuration Example of Decoder 51>
In
The accumulation buffer 61 temporarily accumulates the coded bit stream transmitted from the encoder 11, and supplies the coded bit stream to the reversible decoding unit 62 at a predetermined timing.
The reversible decoding unit 62 receives the coded bit stream from the accumulation buffer 61, and decodes the coded bit stream by a method corresponding to the encoding method of the reversible encoding unit 26 in
Then, the reversible decoding unit 62 supplies the quantization coefficient as the coded data included in the decoding result of the coded bit stream to the inverse quantization unit 63.
Furthermore, the reversible decoding unit 62 has a function of performing parsing. The reversible decoding unit 62 parses necessary encoding information included in the decoding result of the coded bit stream, and supplies the encoding information to the intra prediction unit 71, the motion prediction compensation unit 72, the deblocking filter 31b, the adaptive offset filter 81, and other necessary blocks.
The inverse quantization unit 63 inversely quantizes the quantization coefficient as the coded data from the reversible decoding unit 62 by a method corresponding to the quantization method of the quantization unit 25 in
The inverse orthogonal transform unit 64 inversely orthogonally transforms the orthogonal transform coefficient supplied from the inverse quantization unit 63 by a method corresponding to the orthogonal transform method of the orthogonal transform unit 24 in
The operation unit 65 is supplied with the residual from the inverse orthogonal transform unit 64, and is supplied with the predicted image from the intra prediction unit 71 or the motion prediction compensation unit 72 via the selection unit 73.
The operation unit 65 adds the residual from the inverse orthogonal transform unit 64 and the predicted image from the selection unit 73, generates a decoded image, and supplies the decoded image to the deblocking filter 31b.
Here, the reversible decoding unit 62 to the operation unit 65 described above constitute a decoding unit that decodes an image, and processing (step) performed by the decoding unit is a decoding step.
The rearrangement buffer 67 temporarily stores the decoded image supplied from the ALF 82, rearranges the arrangement of frames (pictures) of the decoded image from the encoding (decoding) order to the display order, and supplies the decoded image to the D/A conversion unit 68.
The D/A conversion unit 68 D/A converts the decoded image supplied from the rearrangement buffer 67, outputs the decoded image to a display (not illustrated), and causes the display to display the decoded image. Note that in a case where a device connected to the decoder 51 receives an image of a digital signal, the decoder 51 can be configured without the D/A conversion unit 68.
The frame memory 69 temporarily stores the decoded image supplied from the ALF 82. Moreover, the frame memory 69 supplies the decoded image to the selection unit 70 as the reference image to be used for generation of the predicted image at a predetermined timing or on the basis of an external request of the intra prediction unit 71, the motion prediction compensation unit 72, or the like.
The selection unit 70 selects a supply destination of the reference image supplied from the frame memory 69. In a case of decoding an image encoded by intra prediction, the selection unit 70 supplies the reference image supplied from the frame memory 69 to the intra prediction unit 71. Furthermore, in a case of decoding an image encoded by inter prediction, the selection unit 70 supplies the reference image supplied from the frame memory 69 to the motion prediction compensation unit 72.
The intra prediction unit 71 performs intra prediction similar to that of the intra prediction unit 34 in
The motion prediction compensation unit 72 performs inter prediction using the reference image supplied from the frame memory 69 via the selection unit 70, similarly to the motion prediction compensation unit 35 in
The motion prediction compensation unit 72 supplies the predicted image obtained by the inter prediction to the selection unit 73.
The selection unit 73 selects the predicted image supplied from the intra prediction unit 71 or the predicted image supplied from the motion prediction compensation unit 72, and supplies the selected predicted image to the operation unit 65.
The deblocking filter 31b applies the deblocking filter to the decoded image from the operation unit 65 according to the filter parameter included in the encoding information supplied from the reversible decoding unit 62. The deblocking filter 31b supplies the decoded image (filtered image) to which the deblocking filter is applied or the decoded image to which the deblocking filter is not applied to the adaptive offset filter 81.
The adaptive offset filter 81 applies the adaptive offset filter to the decoded image from the deblocking filter 31b as necessary according to the filter parameter included in the encoding information supplied from the reversible decoding unit 62. The adaptive offset filter 81 supplies the ALF 82 with the decoded image (filtered image) to which the adaptive offset filter is applied or the decoded image to which the adaptive offset filter is not applied.
The ALF 82 applies the ALF to the decoded image from the adaptive offset filter 81 as necessary, and supplies the decoded image to which the ALF is applied or the decoded image to which the ALF is not applied to the rearrangement buffer 67 and the frame memory 69.
<Decoding Processing>
In the decoding processing, in step S51, the accumulation buffer 61 temporarily accumulates the coded bit stream transmitted from the encoder 11 and supplies the coded bit stream to the reversible decoding unit 62 as appropriate, and the processing proceeds to step S52.
In step S52, the reversible decoding unit 62 receives and decodes the coded bit stream supplied from the accumulation buffer 61, and supplies the quantization coefficient as the coded data included in the decoding result of the coded bit stream to the inverse quantization unit 63.
Furthermore, the reversible decoding unit 62 parses the encoding information included in the decoding result of the coded bit stream. Then, the reversible decoding unit 62 supplies necessary encoding information to the intra prediction unit 71, the motion prediction compensation unit 72, the deblocking filter 31b, the adaptive offset filter 81, and other necessary blocks.
Then, the processing proceeds from step S52 to step S53, and the intra prediction unit 71 or the motion prediction compensation unit 72 performs intra prediction or inter prediction for generating a predicted image according to the reference image supplied from the frame memory 69 via the selection unit 70 and the encoding information supplied from the reversible decoding unit 62 (intra prediction step or inter prediction step). Then, the intra prediction unit 71 or the motion prediction compensation unit 72 supplies the predicted image obtained by the intra prediction or the inter prediction to the selection unit 73, and the processing proceeds from step S53 to step S54.
In step S54, the selection unit 73 selects the predicted image supplied from the intra prediction unit 71 or the motion prediction compensation unit 72 and supplies the predicted image to the operation unit 65, and the processing proceeds to step S55.
In step S55, the inverse quantization unit 63 inversely quantizes the quantization coefficient from the reversible decoding unit 62 and supplies the orthogonal transform coefficient obtained as a result to the inverse orthogonal transform unit 64, and the processing proceeds to step S56.
In step S56, the inverse orthogonal transform unit 64 inversely orthogonally transforms the orthogonal transform coefficient from the inverse quantization unit 63 and supplies the residual obtained as a result to the operation unit 65, and the processing proceeds to step S57.
In step S57, the operation unit 65 generates the decoded image by adding the residual from the inverse orthogonal transform unit 64 and the predicted image from the selection unit 73. Then, the operation unit 65 supplies the decoded image to the deblocking filter 31b, and the processing proceeds from step S57 to step S58.
In step S58, the deblocking filter 31b applies the deblocking filter to the decoded image from the operation unit 65 according to the filter parameter included in the encoding information supplied from the reversible decoding unit 62. The deblocking filter 31b supplies the filtered image obtained as a result of applying the deblocking filter to the adaptive offset filter 81, and the processing proceeds from step S58 to step S59.
In step S59, the adaptive offset filter 81 applies the adaptive offset filter to the filtered image from the deblocking filter 31b according to the filter parameter included in the encoding information supplied from the reversible decoding unit 62. The adaptive offset filter 81 supplies the filtered image obtained as a result of the application of the adaptive offset filter to the ALF 82, and the processing proceeds from step S59 to step S60.
The ALF 82 applies the ALF to the filtered image from the adaptive offset filter 81 and supplies the filtered image obtained as a result to the rearrangement buffer 67 and the frame memory 69, and the processing proceeds to step S61.
In step S61, the frame memory 69 temporarily stores the filtered image supplied from the ALF 82, and the processing proceeds to step S62. The filtered image (decoded image) stored in the frame memory 69 is used as the reference image that is the source for generating the predicted image in the intra prediction or the inter prediction in step S53.
In step S62, the rearrangement buffer 67 rearranges the filtered images supplied from the ALF 82 in the display order and supplies the rearranged filtered images to the D/A conversion unit 68, and the processing proceeds to step S63.
In step S63, the D/A conversion unit 68 D/A-converts the filtered image from the rearrangement buffer 67, and the decoding processing ends. The filtered image (decoded image) after the D/A conversion is output to and displayed on a display (not illustrated).
The intra prediction performed by the intra prediction unit 34 of
<Others>
The present technology can be applied to any image encoding and decoding method. That is, as long as it does not contradict the present technology described above, the specifications of various processes related to the image encoding and decoding, such as conversion (inverse conversion), quantization (inverse quantization), encoding (decoding), and prediction are arbitrary, and are not limited to the above-described examples. Furthermore, some of these processes may be omitted as long as they do not contradict the present technology described above.
Furthermore, in the present description, a “block” (not a block indicating a processing unit) used in the description as a partial area of an image (picture) or a processing unit indicates an arbitrary partial area in the picture unless otherwise specified, and does not limit its size, shape, characteristics, and the like. For example, the “block” includes any partial area (processing unit) such as transform block (TB), transform unit (TU), prediction block (PB), prediction unit (PU), smallest coding unit (SCU), coding unit (CU), largest coding unit (LCU), coding tree block (CTB), coding tree unit (CTU), conversion block, sub-block, macroblock, tile, or slice, and the like described in References REF 1 to REF 3 above.
The data units in which the various information described above is set and the data units targeted by the various processes are arbitrary and are not limited to the above-described examples. For example, these pieces of information and processes may be set in every transform unit (TU), transform block (TB), prediction unit (PU), prediction block (PB), coding unit (CU), largest coding unit (LCU), sub-block, block, tile, slice, picture, sequence, or component, or data in those data units may be targeted. Of course, this data unit can be set for every piece of information or process, and it is not necessary that the data units of all the pieces of information or processes are unified. Note that the storage location of these pieces of information is arbitrary, and may be stored in a header, parameter set, or the like of the above-described data units. Furthermore, it may be stored in a plurality of places.
The control information related to the present technology described above may be transmitted from the encoding side to the decoding side. For example, control information (for example, enabled flag) for controlling whether or not to permit (or prohibit) the application of the present technology described above may be transmitted. Furthermore, for example, control information indicating a target to which the above-described present technology is applied (or a target to which the present technology is not applied) may be transmitted. For example, control information specifying a block size (upper limit or lower limit or both), a frame, a component, a layer, or the like to which the present technology is applied (or permit or prohibit the application) may be transmitted.
Upon specifying the size of the block by which the present technology is applied, not only the block size may be specified directly, but also the block size may be indirectly specified. For example, the block size may be designated using identification data for identifying the size. Furthermore, for example, the block size may be designated by a ratio or a difference from the size of a reference block (for example, an LCU, an SCU, or the like). For example, in a case where information for designating a block size is transmitted as a syntax element or the like, information for indirectly designating a size as described above may be used as the information. In this manner, the amount of information of the information can be reduced, and encoding efficiency may be improved. Furthermore, the specification of the block size also includes a specification of the range of the block size (for example, the specification of the range of an allowable block size, or the like).
Note that in the present description, “identification data” is information for identifying a plurality of states, and includes “flag” and other names. Furthermore, the “identification data” includes not only information used to identify two states of true (1) and false (0) but also information capable of identifying three or more states. Therefore, the value that this “identification data” can take may be, for example, a binary of 1 or 0 or a ternary or more. That is, the number of bits constituting the “identification data” is arbitrary, and may be one bit or a plurality of bits. Furthermore, since the identification data is assumed to include not only the identification data in the bit stream but also include difference information of the identification data with respect to certain reference information in the bit stream, in the present description, the “identification data” includes not only the information but also difference information with respect to the reference information.
Furthermore, various types of information (metadata and the like) related to the coded data (bit stream) may be transmitted or recorded in any form as long as the information is associated with the coded data. Here, the term “associate” means, for example, that one data can be used (linked) when the other data is processed. That is, the data associated with each other may be combined as one piece of data or may be individual pieces of data. For example, information associated with coded data (image) may be transmitted on a transmission path different from that of the coded data (image). Furthermore, for example, the information associated with the coded data (image) may be recorded in a recording medium (or another recording area of the same recording medium) different from the coded data (image). Note that this “association” may be for a part of data instead of the entire data. For example, an image and information corresponding to the image may be associated with each other in an arbitrary unit such as a plurality of frames, one frame, or a part in a frame.
Note that in the present description, terms such as “combine”, “multiplex”, “add”, “integrate”, “include”, “store”, “put in”, “plug in”, and “insert” mean to combine a plurality of items into one, for example, such as combining coded data and metadata into one piece of data, and mean one method of the above-described “association”.
The present technology can also be implemented as any configuration constituting a device or a system, for example, a processor as a system large scale integration (LSI) or the like, a module using a plurality of processors or the like, a unit using a plurality of modules or the like, a set obtained by further adding other functions to a unit, or the like (that is, a configuration of a part of the device).
<Description of Computer to Which Present Technology is Applied>
Next, all or part of the above-described series of processes can be performed by hardware or software. In a case where all or a part of the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.
The program can be pre-recorded on a hard disk 905 or ROM 903 as a recording medium incorporated in the computer.
Alternatively, the program can be stored (recorded) in a removable recording medium 911 driven by a drive 909. Such a removable recording medium 911 can be provided as what is called package software. Here, examples of the removable recording medium 911 include, for example, a flexible disk, a compact disc read only memory (CD-ROM), a magneto optical (MO) disk, a digital versatile disc (DVD), a magnetic disk, a semiconductor memory, and the like.
Note that in addition to installing the program on the computer from the removable recording medium 911 as described above, the program can be downloaded to the computer via a communication network or a broadcasting network and installed on the incorporated hard disk 905. That is, for example, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer by wire via a network such as a local area network (LAN) or the Internet.
The computer has an incorporated central processing unit (CPU) 902, and an input-output interface 910 is connected to the CPU 902 via a bus 901.
If a command is input by a user through the input-output interface 910 by operating an input unit 907 or the like, the CPU 902 executes the program stored in the read only memory (ROM) 903 accordingly. Alternatively, the CPU 902 loads the program stored in the hard disk 905 into a random access memory (RAM) 904 and executes the program.
Thus, the CPU 902 performs the processing according to the above-described flowchart or the processing performed according to the above-described configuration of the block diagram. Then, the CPU 902 outputs a processing result thereof from an output unit 906 or sends the processing result from a communication unit 908 if necessary via the input-output interface 910 for example, and further causes recording of the processing result on the hard disk 905, or the like.
Note that the input unit 907 includes a keyboard, a mouse, a microphone, and the like. Furthermore, the output unit 906 includes a liquid crystal display (LCD), a speaker, and the like.
Here, in the present description, the processing performed by the computer according to the program does not necessarily have to be performed in time series in the order described as the flowchart. That is, the processing performed by the computer according to the program also includes processing that is executed in parallel or individually (for example, parallel processing or object processing).
Furthermore, the program may be processed by one computer (processor) or may be processed in a distributed manner by a plurality of computers. Moreover, the program may be transferred to a distant computer and executed.
Moreover, in the present description, a system means a set of a plurality of components (devices, modules (parts), and the like), and it does not matter whether or not all components are in the same housing. Therefore, both of a plurality of devices housed in separate housings and connected via a network and a single device in which a plurality of modules is housed in one housing are systems.
Note that the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology. For example, the present technology can employ a configuration of cloud computing in which one function is shared by a plurality of devices via a network and processed jointly.
Furthermore, each step described in the above-described flowcharts can be executed by one device, or can be executed in a shared manner by a plurality of devices.
Moreover, in a case where a plurality of processes is included in one step, the plurality of processes included in the one step can be executed in a shared manner by a plurality of devices in addition to being executed by one device.
Furthermore, the effects described in the present description are merely examples and are not limited, and other effects may be provided.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2020/047393 | 12/18/2020 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 62951177 | Dec 2019 | US |
