Patent Application
20170366813
This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2016-119176, filed on Jun. 15, 2016, the entire contents of which are incorporated herein by reference.
The embodiment discussed herein is related to an image encoding apparatus, an image encoding method, and an image encoding program stored in a non-transitory computer-readable storage medium.
In image data encoding apparatuses, techniques for reducing idle time in encoding processing have been proposed.
As an example, a data encoding apparatus has been proposed in which a data holding unit (buffer) is provided between a coefficient bit modeling unit and a multiplier quotient (MQ) coder, and in which coding information generated in the coefficient bit modeling unit is temporarily held in the data holding unit and then is read and processed by the MQ coder. In this data encoding apparatus, the timing when coding information is generated in the coefficient bit modeling unit does not have to coincide with the timing when coding information is processed in the MQ coder, thereby reducing wasted idle time (see Japanese Laid-open Patent Publication No. 2004-320157).
Furthermore, a method for a video encoding process has been proposed in which mode decision processing and bit stream packing are performed in parallel for various frames in a sequence. This method reduces the amount of idle time for both mode decision processing logic and bit stream packing logic (see Japanese National Publication of International Patent Application No. 2014-529256).
As an example, there is an image encoding apparatus that has a high processing capability of encoding high-resolution image data, such as 4K (3,840×2,160 pixels, a picture rate of 60 p) or 8K (7,680×4,320 pixels, a picture rate of 120 p) image data, with real-time processing. Such an image encoding apparatus is also capable of encoding image data that is able to be encoded with a lower throughput than a maximum throughput with which real time processing is enabled.
In this case, however, encoding processing of one-frame image data of image data that is able to be encoded with a low throughput is completed in a shorter time period than a time period that is basically usable for encoding one-frame image data, and thus idle time occurs in the encoding processing performed by the image encoding apparatus. That is, no good use is made of the processing capability of the image encoding apparatus.
In an aspect of the embodiment, image quality is enhanced.
According to an aspect of the invention, an image encoding method causing a computer to execute a process, the process includes: calculating, from an encoding throughput for image data to be encoded that is obtained based on a parameter regarding an encoding throughput, and a maximum throughput of an image encoding apparatus, an available throughput that occurs in a process of encoding the image data to be encoded; allocating, in accordance with the calculated available throughput, a function of enhancing precision of encoding to the image data to be encoded; and encoding, by using the allocated function, the image data to be encoded.
The object and advantages of the invention will be realized and attained by means of the elements and combinations particularly pointed out in the claims.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the invention, as claimed.
An example of the embodiment will be described in detail below with reference to the drawings.
As illustrated in
As illustrated in
The acquisition unit 11 acquires parameters of original image data of each channel that have been set externally. Parameters here are parameters regarding a throughput in encoding of original image data. Examples of parameters include an image horizontal size, an image vertical size, a picture rate, a pixel bit depth, and a YUV format.
An image horizontal size and an image vertical size refer to an image size of one picture specified in units of pixels. In some cases, image sizes are represented below by 4K (3,840×2,160 pixels), high definition (HD) (1,920×1,080 pixels), and standard definition (SD) (720×480 pixels). A picture rate is the number of pictures per second. A pixel bit depth is the number of bits by which one pixel is represented, and 8 bits or 10 bits are typically used. A YUV format is a typical image format used in inputs in an image encoding apparatus, and examples of the type of YUV format include YUV422 and YUV420 based on densities of color difference components.
The acquisition unit 11 transfers the acquired parameters of each channel to the calculation unit 12.
The calculation unit 12 calculates, based on the parameters of each channel transferred from the acquisition unit 11, an encoding throughput for the original image data of each channel. The calculation unit 12 also calculates a maximum value of throughput (hereinafter referred to as “maximum throughput”) usable for encoding the original image data in the image encoding apparatus 10. Then, the calculation unit 12 calculates, based on the encoding throughput for the original image data and the maximum throughput, an available throughput that is an unused portion of the maximum throughput. Note that “maximum throughput” is determined in accordance with hardware specifications of an apparatus and implementation program specifications, and thus “maximum throughput” may be stored in a certain nonvolatile memory in advance. In calculation of “maximum throughput”, “maximum throughput” stored in the nonvolatile memory may be read. Alternatively, “maximum throughput” may be calculated based on the hardware specifications and the implementation program specifications every time the apparatus is powered on, thereby enabling changes in program specifications to be accommodated.
Specifically, the calculation unit 12 calculates, as an index indicating an encoding throughput for the original image data of each channel, “throughput per second for channel k” represented by, for example, the following equation (1) based on the parameters acquired by the acquisition unit 11.
throughput per second for channel k=image horizontal size×image vertical size×picture rate×α (1)
In equation (1), α denotes a weight variable used in the case where an encoding throughput for original image data of channel k is changed depending on a pixel bit depth and a YUV format. Specifically, in the case where the image encoding apparatus 10 is capable of encoding image data having a pixel bit depth of 10 bits and being in a YUV422 format, assume that α for the case of the pixel bit depth of 10 bits and the YUV422 format is 1. Then, α=1 serves as a standard, and α is adjusted according to a pixel bit depth and the type of YUV format acquired for channel k.
For example, assume that, in the case of a pixel bit depth of 8 bits, a throughput is 0.9 times as much as in the case of a pixel bit depth of 10 bits, and that, in the case of a YUV420 format, a throughput is 0.8 times as much as in the case of the YUV422 format. In this case, in calculating a throughput of equation (1) for original image data having a pixel bit depth of 8 bits and being in the YUV420 format, α is 0.9×0.8=0.72.
For example, assume that original image data of a channel 0 has an image size of HD (1,920×1,080 pixels), a picture rate of 60 p, and a pixel bit depth of 10 bits, and is in the YUV422 format. In this case, a throughput per second for the channel 0 is as follows.
throughput per second for channel 0=1,920×1,080×60×1=124,416,000
For example, assume that original image data of a channel 1 has an image size of SD (720×480 pixels), a picture rate of 60 p, and a pixel bit depth of 10 bits, and is in the YUV422 format. In this case, a throughput per second for the channel 1 is as follows.
throughput per second for channel 1=720×480×60×1=20,736,000
Furthermore, the calculation unit 12 calculates an available throughput per second of the image encoding apparatus 10 using the following equation (2).
available throughput=maximum throughput per second of image encoding apparatus 10−Σ[k=0, n−1] throughput per second for channel k (2)
In equation (2), “maximum throughput per second of image encoding apparatus 10” is a fixed value dependent on the processing capability of the image encoding apparatus 10. For example, assume that the image encoding apparatus 10 is capable of processing image data having an image size of 4K (3,840×2,160 pixels), a picture rate of 60 p, and a pixel bit depth of 10 bits, and being in the YUV422 format. In this case, a maximum throughput per second of the image encoding apparatus 10 is as follows. From the pixel bit depth and the YUV format, assume that α is 1.
maximum throughput per second of image encoding apparatus 10=3,840×2,160×60×1=497,664,000
In the case where the above-described image encoding apparatus 10 processes two channels of the channel 0 (HD, 60 p, 10 bits, YUV422) and the channel 1 (SD, 60 p, 10 bits, YUV422), an available throughput is as follows.
available throughput=497,664,000−(124,416,000+20,736,000)=352,512,000
The calculation unit 12 transfers the calculated available throughput to the allocation unit 13.
The allocation unit 13 distributes the available throughput transferred from the calculation unit 12 to each channel and allocates, to each channel, an image quality enhancement function based on an available throughput distributed to the channel.
Specifically, the allocation unit 13 distributes the available throughput in accordance with a ratio between throughputs per second for the respective channels, and priorities of the channels that have been assigned in advance. For example, in the case where multi-channel encoding of two channels of the channel 0 (HD) and the channel 1 (SD) is performed, the throughput for SD is about one-sixth of the throughput for HD, and thus the available throughput is distributed in a ratio of channel 0 (HD):channel 1 (SD)=6×β:1×γ, where β and γ are each a weight variable equal to or greater than 0 based on a priority of each channel. A priority of each channel and a weight variable may be appropriately set in accordance with a specification, such as giving priority to HD over SD in terms of image quality enhancement. For the sake of simplicity, assume below that β=1, and γ=1. That is, the above calculated available throughput (352,512,000) is distributed to the channel 0 (HD) and the channel 1 (SD) as follows.
available throughput distributed to channel 0 (HD)=352,512,000×6/7=302,153,142
available throughput distributed to channel 1 (SD)=352,512,000×1/7=50,358,857
Examples of an image quality enhancement function typically include vector search functions using reduced images with different sizes, a vector search function using a prediction image, and a prediction mode determination function. An image quality enhancement function is not limited to these examples and may be a function capable of enhancing the precision of encoding when used in encoding, that is, a function capable of reducing degradation of image quality due to encoding. In each image quality enhancement function, in the case where the number of images or the number of modes that are referred to in the function is increased, processing is repeated the number of times corresponding to the increased number of images or modes, thereby increasing a throughput in proportion to the number of repetitions. A throughput and a maximum number of repetitions are fixed values dependent on the processing capability of the image encoding apparatus 10, such as the number of parallelizable processes in the image encoding apparatus 10 or the bandwidth of a memory.
In the embodiment, throughputs and maximum numbers of repetitions for these image quality enhancement functions are set in the image quality enhancement function table 21. Then, the allocation unit 13 allocates, to each channel, an image quality enhancement function based on an available throughput distributed to the channel with reference to the image quality enhancement function table 21.
Allocation priorities indicate which of the image quality enhancement functions set in the image quality enhancement function table 21 is allocated on a priority basis. For example, allocation priorities may be assigned so that priority is given in descending order of image quality enhancement effectiveness obtained by performing encoding using each image quality enhancement function.
An example will be described where image quality enhancement functions are allocated to two channels of the channel 0 (HD, 60 p, 10 bits, YUV422) and the channel 1 (SD, 60 p, 10 bits, YUV422) with reference to the image quality enhancement function table 21 illustrated in
The allocation unit 13 calculates a throughput ratio between each throughput estimated in the image quality enhancement function table 21 and a throughput for each channel. Here, with respect to the channel 0, a throughput ratio is 1/4 because there is a difference between image sizes 4K and HD. With respect to the channel 1, a throughput ratio is 1/24 because there is a difference between image sizes 4K and SD.
Then, the allocation unit 13 allocates, in accordance with the available throughput of 302,153,142 distributed to the channel 0 (HD), “vector search using 1/16 (length and width 1/4)-reduced image” ranked first in allocation priority to the channel 0 (HD) from the image quality enhancement function table 21. Because of 302,153,142≧(6,000,000 (throughput per second)×1/4 (throughput ratio)×16 (maximum number of repetitions)), the allocation unit 13 allocates “vector search using 1/16 (length and width 1/4)-reduced image” for 16 times. A remaining available throughput is 302,153,142−6,000,000×1/4×16=278,153,142.
The allocation unit 13 similarly calculates, in accordance with the remaining available throughput, in descending order of allocation priority, how many times each image quality enhancement function is able to be repeated and allocates the image quality enhancement function to the channel 0 (HD). The allocation unit 13 similarly performs calculations and allocates each image quality enhancement function to the channel 1 (SD).
In this example, the following image quality enhancement functions are finally allocated to each channel.
The image encoding unit 14 encodes the original image data of each channel by using the image quality enhancement functions allocated by the allocation unit 13 and outputs it as encoded image data. That is, the image encoding unit 14 performs, in addition to typical encoding processing based on a predetermined standard, encoding by allocating the image quality enhancement functions. Examples of a predetermined standard may include moving picture experts group (MPEG)-2 video, H.264/MPEG4 advanced video coding (AVC), and H.265/high efficiency video coding (HEVC).
The audio encoding unit 15 encodes original audio data of each channel in accordance with a predetermined standard and outputs it as encoded audio data. Examples of a predetermined standard may include MPEG-1 audio, MPEG2/4 advanced audio coding (MC), and MPEG4 high-efficiency (HE)-AAC.
The multiplexing unit 16 generates, in accordance with a predetermined standard, multiplex data involved in multiplexing and audio-video (AV) synchronization of the encoded image data output from the image encoding unit 14 and the encoded audio data output from the audio encoding unit 15. Examples of a predetermined standard may include an MPEG-2 transport stream (TS) standard and an MPEG-H MPEG media transport (MMT) standard. Data involved in multiplexing is, according to the MPEG-2 TS standard, a TS header, a packetized elementary stream (PES) header, a program association table (PAT), a program map table (PMT), and a program clock reference (PCR), for example. Data involved in AV synchronization is time of presentation (presentation time stamp (PTS)) for image data and audio data to be written in a PES header, for example.
Then, the multiplexing unit 16 multiplexes the encoded image data, the encoded audio data, and the multiplex data into one stream in accordance with the predetermined standard.
The image encoding apparatus 10 may be implemented by a computer 40 illustrated in
The storage unit 43 may be implemented by a hard disk drive (HDD), a solid state drive (SSD), a flash memory, or the like. The storage unit 43 serving as a storage medium stores an image encoding program 50 for causing the computer 40 to function as the image encoding apparatus 10. The image encoding program 50 includes an acquisition process 51, a calculation process 52, an allocation process 53, an image encoding process 54, an audio encoding process 55, and a multiplexing process 56. The storage unit 43 further includes a function information storage area 60 in which function information constituting the image quality enhancement function table 21 is stored.
The CPU 41 reads the image encoding program 50 from the storage unit 43 to load it into the memory 42 and executes the processes included in the image encoding program 50 sequentially. The CPU 41 operates as the acquisition unit 11 illustrated in
A function implemented by the image encoding program 50 may be implemented by a semiconductor integrated circuit, or more specifically, an application specific integrated circuit (ASIC), for example.
Next, the action of the image encoding apparatus 10 according to the embodiment will be described. When original image data and original audio data are input to the image encoding apparatus 10, image encoding processing illustrated in
In step S11, the calculation unit 12 calculates a maximum throughput per second of the image encoding apparatus 10.
Then, in step S12, the acquisition unit 11 acquires, for each processing unit in real time processing in encoding, parameters of original image data of each channel that have been set externally.
Subsequently, in step S13, the acquisition unit 11 determines, based on the acquired parameters, whether a channel structure of the input original image data has been changed. When at least one of the number of channels of original image data and the parameters of the original image data of each channel differs from that of original image data in a previous processing unit, a determination of YES is made, and processing proceeds to step S14. On the other hand, when the channel structure has not been changed, processing proceeds to step S17.
In step S14, the calculation unit 12 calculates, for each channel, “throughput per second for channel k” represented by, for example, equation (1) based on the parameters of each channel acquired by the acquisition unit 11.
Then, in step S15, the calculation unit 12 calculates, according to, for example, equation (2), an available throughput per second of the image encoding apparatus 10 by using the maximum throughput per second of the image encoding apparatus 10 calculated in step S11 and the throughput per second for each channel calculated in step S14.
Then, in step S16, the allocation unit 13 distributes the available throughput calculated by the calculation unit 12 to each channel and allocates, to each channel, an image quality enhancement function based on an available throughput distributed to the channel with reference to the image quality enhancement function table 21.
Subsequently, in step S17, the image encoding unit 14 encodes the original image data of each channel by using the image quality enhancement function allocated by the allocation unit 13 and outputs it as encoded image data.
In parallel with the process of step S17, the audio encoding unit 15 encodes original audio data of each channel and outputs it as encoded audio data in step S18.
Then, in step S19, the multiplexing unit 16 generates multiplex data involved in multiplexing and AV synchronization of the encoded image data output from the image encoding unit 14 and the encoded audio data output from the audio encoding unit 15. Then, the multiplexing unit 16 multiplexes the encoded image data, the encoded audio data, and the multiplex data into one stream.
Subsequently, in step S20, the acquisition unit 11 determines whether original image data and original audio data in a subsequent processing unit have been input to determine whether to end encoding. If encoding is to be performed continuously, processing returns to step S12, and if encoding is to be ended, the image encoding processing ends.
As described above, an image encoding apparatus according to the embodiment calculates an available throughput of the image encoding apparatus, allocates an image quality enhancement function based on the available throughput to original image data, and encodes the original image data by using the image quality enhancement function. This enables maximum use of the processing capability of the image encoding apparatus and enhancement of image quality.
As a comparative example,
As illustrated in
On the other hand, in the embodiment, an available throughput is allocated to an image quality enhancement function, therefore enabling maximum use of the processing capability of the image encoding apparatus as illustrated in
In the embodiment, the case where processes of encoding and multiplexing audio data are included is described, whereas the embodiment is also applicable to the case where only image data is encoded.
In the embodiment, the case where multi-channel original image data is encoded is described, whereas the embodiment is also applicable to the case where one-channel original image data is encoded. In this case, all of available throughputs of the image encoding apparatus may be used for an image quality enhancement function allocated to the one-channel original image data.
As described above, the embodiment where the image encoding program 50 is stored (installed) in the storage unit 43 in advance is described, whereas the image encoding program 50 recorded in a storage medium, such as a compact disc read-only memory (CD-ROM) or a digital versatile disc ROM (DVD-ROM), may be provided.
All examples and conditional language recited herein are intended for pedagogical purposes to aid the reader in understanding the invention and the concepts contributed by the inventor to furthering the art, and are to be construed as being without limitation to such specifically recited examples and conditions, nor does the organization of such examples in the specification relate to a showing of the superiority and inferiority of the invention. Although the embodiment of the present invention has been described in detail, it should be understood that the various changes, substitutions, and alterations could be made hereto without departing from the spirit and scope of the invention.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2016-119176 | Jun 2016 | JP | national |
