Patent Application
20240040135
The present disclosure is generally related to video coding and, in particular, to supplemental enhancement information (SEI) messages used in image/video coding.
Digital video accounts for the largest bandwidth use on the internet and other digital communication networks. As the number of connected user devices capable of receiving and displaying video increases, it is expected that the bandwidth demand for digital video usage will continue to grow.
The disclosed aspects/embodiments provide techniques that use a scalability dimension information (SDI) view identifier length minus L syntax element to prevent a length of an SDI view ID value syntax element, which specifies a view identifier of an i-th layer in a bitstream, from being zero. The disclosed aspects/embodiments further provide techniques that prevent a bitstream from having a multiview acquisition information supplemental enhancement information (SEI) message or an auxiliary information SEI message when an SDI message is not present in the bitstream. The disclosed aspects/embodiments also provide techniques that prevent the multiview acquisition information SEI message from being scalable-nested.
A first aspect relates to a method of processing video data. The method includes using a scalability dimension information (SDI) supplemental enhancement information (SEI) message to indicate an SDI view identifier length minus L syntax element; and performing a conversion between a video media file and the bitstream based on the SDI SEI message.
Optionally, in any of the preceding aspects, another implementation of the aspect provides that the SDI view identifier length minus L syntax element is configured to prevent a length of an SDI view identifier value syntax element, which specifies a view identifier of an i-th layer in a bitstream, from being zero.
Optionally, in any of the preceding aspects, another implementation of the aspect provides that L is equal to 1.
Optionally, in any of the preceding aspects, another implementation of the aspect provides that the SDI view identifier length minus L syntax element is designated sdi_view_id_len_minus1.
Optionally, in any of the preceding aspects, another implementation of the aspect provides that the SDI view identifier value syntax element is designated sdi_view_id_val[i].
Optionally, in any of the preceding aspects, another implementation of the aspect provides that the SDI view identifier length minus L syntax element, plus one, specifies the length of the SDI view identifier value syntax element.
Optionally, in any of the preceding aspects, another implementation of the aspect provides that the SDI view identifier length minus L syntax element is coded as an unsigned integer using N bits.
Optionally, in any of the preceding aspects, another implementation of the aspect provides that N is equal to 4.
Optionally, in any of the preceding aspects, another implementation of the aspect provides that the SDI view identifier length minus L syntax element is coded as a fixed-pattern bitstring using N bits, a signed integer using N bits, a truncated binary, a signed integer K-th order Exp-Golomb-coded syntax element where K is equal to 0, or an unsigned integer M-th order Exp-Golomb-coded syntax element where M is equal to 0.
Optionally, in any of the preceding aspects, another implementation of the aspect provides that the bitstream is a bitstream in scope.
Optionally, in any of the preceding aspects, another implementation of the aspect provides that a multiview information SEI message and an auxiliary information SEI message are not present in a coded video sequence (CVS) unless the SDI SEI message is present in the CVS.
Optionally, in any of the preceding aspects, another implementation of the aspect provides that the multiview information SEI message comprises a multiview acquisition information SEI message.
Optionally, in any of the preceding aspects, another implementation of the aspect provides that the auxiliary information SEI message comprises a depth representation information SEI message.
Optionally, in any of the preceding aspects, another implementation of the aspect provides that the auxiliary information SEI message comprises an alpha channel information SEI message.
Optionally, in any of the preceding aspects, another implementation of the aspect provides that one or more of an SDI multiview information flag and an SDI auxiliary information flag are equal to 1 when the multiview information SEI message or the auxiliary information SEI message are present in the bitstream.
Optionally, in any of the preceding aspects, another implementation of the aspect provides that the multiview information SEI message comprises a multiview acquisition information SEI message, and wherein the multiview acquisition information SEI message is not scalable-nested.
Optionally, in any of the preceding aspects, another implementation of the aspect provides that an SEI message in the bitstream and having a payload type equal to 179 is constrained from being included in a scalable nesting SEI message.
Optionally, in any of the preceding aspects, another implementation of the aspect provides that an SEI message in the bitstream and having a payload type equal to 3, 133, 179, 180, or 205 is constrained from being included in a scalable nesting SEI message.
A second aspect relates to an apparatus for processing video data comprising a processor and a non-transitory memory with instructions thereon, wherein the instructions upon execution by the processor cause the processor to: use a scalability dimension information (SDI) supplemental enhancement information (SEI) message to indicate an SDI view identifier length minus L syntax element; and convert between a video media file and the bitstream based on the SDI SEI message.
A third aspect relates to a non-transitory computer readable medium comprising a computer program product for use by a coding apparatus, the computer program product comprising computer executable instructions stored on the non-transitory computer readable medium that, when executed by one or more processors, cause the coding apparatus to: use a scalability dimension information (SDI) supplemental enhancement information (SEI) message to indicate an SDI view identifier length minus L syntax element; and convert between a video media file and the bitstream based on the SDI SEI message.
A fourth aspect relates to a non-transitory computer-readable storage medium storing instructions that cause a processor to: use a scalability dimension information (SDI) supplemental enhancement information (SEI) message to indicate an SDI view identifier length minus L syntax element; and convert between a video media file and the bitstream based on the SDI SEI message.
A fifth aspect relates to a non-transitory computer-readable recording medium storing a bitstream of a video which is generated by a method performed by a video processing apparatus, wherein the method comprises: use a scalability dimension information (SDI) supplemental enhancement information (SEI) message to indicate an SDI view identifier length minus L syntax element; and convert between a video media file and the bitstream based on the SDI SEI message.
A sixth aspect relates to a method for storing bitstream of a video, comprising: using a scalability dimension information (SDI) supplemental enhancement information (SEI) message to indicate an SDI view identifier length minus L syntax element; generating the bitstream based on the SDI SEI message; and storing the bitstream in a non-transitory computer-readable recording medium.
For the purpose of clarity, any one of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create a new embodiment within the scope of the present disclosure.
These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.
For a more complete understanding of this disclosure, reference is now made to the following brief description, taken in connection with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.
It should be understood at the outset that although an illustrative implementation of one or more embodiments are provided below, the disclosed systems and/or methods may be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.
Video coding standards have evolved primarily through the development of the well-known International Telecommunication Union-Telecommunication (ITU-T) and International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) standards. The ITU-T produced H.261 and H.263, ISO/IEC produced Moving Picture Experts Group (MPEG)-1 and MPEG-4 Visual, and the two organizations jointly produced the H.262/MPEG-2 Video and H.264/MPEG-4 Advanced Video Coding (AVC) and H.265/High Efficiency Video Coding (HEVC) standards. See ITU-T and ISO/IEC, “High efficiency video coding”, Rec. ITU-T H.265|ISO/IEC 23008-2 (in force edition). Since H.262, the video coding standards are based on the hybrid video coding structure wherein temporal prediction plus transform coding are utilized. To explore the future video coding technologies beyond HEVC, the Joint Video Exploration Team (JVET) was founded by Video Coding Experts Group (VCEG) and MPEG jointly in 2015. Since then, many new methods have been adopted by JVET and put into the reference software named Joint Exploration Model (JEM). See J. Chen, E. Alshina, G. J. Sullivan, J.-R. Ohm, J. Boyce, “Algorithm description of Joint Exploration Test Model 7 (JEM7),” JVET-G1001, August 2017. The JVET was later renamed to be the Joint Video Experts Team (JVET) when the Versatile Video Coding (VVC) project officially started. VVC is the new coding standard, targeting at 50% bitrate reduction as compared to HEVC, that has been finalized by the JVET at its 19th meeting ended at Jul. 1, 2020. See Rec. ITU-T H.266|ISO/IEC 23090-3, “Versatile Video Coding”, 2020.
The VVC standard (ITU-T H.266|ISO/IEC 23090-3) and the associated Versatile Supplemental Enhancement Information (VSEI) standard (ITU-T H.274|ISO/IEC 23002-7) have been designed for use in a maximally broad range of applications, including both the traditional uses such as television broadcast, video conferencing, or playback from storage media, and also newer and more advanced use cases such as adaptive bit rate streaming, video region extraction, composition and merging of content from multiple coded video bitstreams, multiview video, scalable layered coding, and viewport-adaptive 360° immersive media. See B. Bross, J. Chen, S. Liu, Y.-K. Wang (editors), “Versatile Video Coding (Draft 10),” JVET-S2001, Rec. ITU-T Rec. H.274|ISO/IEC 23002-7, “Versatile Supplemental Enhancement Information Messages for Coded Video Bitstreams”, 2020, and J. Boyce, V. Drugeon, G. Sullivan, Y.-K. Wang (editors), “Versatile supplemental enhancement information messages for coded video bitstreams (Draft 5),” JVET-S2007.
The Essential Video Coding (EVC) standard (ISO/IEC 23094-1) is another video coding standard that has recently been developed by MPEG.
Layer based prediction 100 is applied between pictures 111, 112, 113, and 114 and pictures 115, 116, 117, and 118 in different layers. In the example shown, pictures 111, 112, 113, and 114 are part of layer N+1 132 and pictures 115, 116, 117, and 118 are part of layer N 131. A layer, such as layer N 131 and/or layer N+1 132, is a group of pictures that are all associated with a similar value of a characteristic, such as a similar size, quality, resolution, signal to noise ratio, capability, etc. In the example shown, layer N+1 132 is associated with a larger image size than layer N 131. Accordingly, pictures 111, 112, 113, and 114 in layer N+1 132 have a larger picture size (e.g., larger height and width and hence more samples) than pictures 115, 116, 117, and 118 in layer N 131 in this example. However, such pictures can be separated between layer N+1 132 and layer N 131 by other characteristics. While only two layers, layer N+1 132 and layer N 131, are shown, a set of pictures can be separated into any number of layers based on associated characteristics. Layer N+1 132 and layer N 131 may also be denoted by a layer ID. A layer ID is an item of data that is associated with a picture and denotes the picture is part of an indicated layer. Accordingly, each picture 111-118 may be associated with a corresponding layer ID to indicate which layer N+1 132 or layer N 131 includes the corresponding picture.
Pictures 111-118 in different layers 131-132 are configured to be displayed in the alternative. As such, pictures 111-118 in different layers 131-132 can share the same temporal identifier (ID) and can be included in the same access unit (AU) 106. As used herein, an AU is a set of one or more coded pictures associated with the same display time for output from a decoded picture buffer (DPB). For example, a decoder may decode and display picture 115 at a current display time if a smaller picture is desired or the decoder may decode and display picture 111 at the current display time if a larger picture is desired. As such, pictures 111-114 at higher layer N+1 132 contain substantially the same image data as corresponding pictures 115-118 at lower layer N 131 (notwithstanding the difference in picture size). Specifically, picture 111 contains substantially the same image data as picture 115, picture 112 contains substantially the same image data as picture 116, etc.
Pictures 111-118 can be coded by reference to other pictures 111-118 in the same layer N 131 or N+1 132. Coding a picture in reference to another picture in the same layer results in inter-prediction 123, which is compatible unidirectional inter-prediction and/or bidirectional inter-prediction. Inter-prediction 123 is depicted by solid line arrows. For example, picture 113 may be coded by employing inter-prediction 123 using one or two of pictures 111, 112, and/or 114 in layer N+1 132 as a reference, where one picture is referenced for unidirectional inter-prediction and/or two pictures are referenced for bidirectional inter-prediction. Further, picture 117 may be coded by employing inter-prediction 123 using one or two of pictures 115, 116, and/or 118 in layer N 131 as a reference, where one picture is referenced for unidirectional inter-prediction and/or two pictures are referenced for bidirectional inter-prediction. When a picture is used as a reference for another picture in the same layer when performing inter-prediction 123, the picture may be referred to as a reference picture. For example, picture 112 may be a reference picture used to code picture 113 according to inter-prediction 123. Inter-prediction 123 can also be referred to as intra-layer prediction in a multi-layer context. As such, inter-prediction 123 is a mechanism of coding samples of a current picture by reference to indicated samples in a reference picture that are different from the current picture where the reference picture and the current picture are in the same layer.
Pictures 111-118 can also be coded by reference to other pictures 111-118 in different layers. This process is known as inter-layer prediction 121, and is depicted by dashed arrows. Inter-layer prediction 121 is a mechanism of coding samples of a current picture by reference to indicated samples in a reference picture where the current picture and the reference picture are in different layers and hence have different layer IDs. For example, a picture in a lower layer N 131 can be used as a reference picture to code a corresponding picture at a higher layer N+1 132. As a specific example, picture 111 can be coded by reference to picture 115 according to inter-layer prediction 121. In such a case, the picture 115 is used as an inter-layer reference picture. An inter-layer reference picture is a reference picture used for inter-layer prediction 121. In most cases, inter-layer prediction 121 is constrained such that a current picture, such as picture 111, can only use inter-layer reference picture(s) that are included in the same AU 106 and that are at a lower layer, such as picture 115. When multiple layers (e.g., more than two) are available, inter-layer prediction 121 can encode/decode a current picture based on multiple inter-layer reference picture(s) at lower levels than the current picture.
A video encoder can employ layer based prediction 100 to encode pictures 111-118 via many different combinations and/or permutations of inter-prediction 123 and inter-layer prediction 121. For example, picture 115 may be coded according to intra-prediction. Pictures 116-118 can then be coded according to inter-prediction 123 by using picture 115 as a reference picture. Further, picture 111 may be coded according to inter-layer prediction 121 by using picture 115 as an inter-layer reference picture. Pictures 112-114 can then be coded according to inter-prediction 123 by using picture 111 as a reference picture. As such, a reference picture can serve as both a single layer reference picture and an inter-layer reference picture for different coding mechanisms. By coding higher layer N+1 132 pictures based on lower layer N 131 pictures, the higher layer N+1 132 can avoid employing intra-prediction, which has much lower coding efficiency than inter-prediction 123 and inter-layer prediction 121. As such, the poor coding efficiency of intra-prediction can be limited to the smallest/lowest quality pictures, and hence limited to coding the smallest amount of video data. The pictures used as reference pictures and/or inter-layer reference pictures can be indicated in entries of reference picture list(s) contained in a reference picture list structure.
Each AU 106 in
Previous H.26x video coding families have provided support for scalability in separate profile(s) from the profile(s) for single-layer coding. Scalable video coding (SVC) is the scalable extension of the AVC/H.264 that provides support for spatial, temporal, and quality scalabilities. For SVC, a flag is signaled in each macroblock (MB) in enhancement layer (EL) pictures to indicate whether the EL MB is predicted using the collocated block from a lower layer. The prediction from the collocated block may include texture, motion vectors, and/or coding modes. Implementations of SVC cannot directly reuse unmodified H.264/AVC implementations in their design. The SVC EL macroblock syntax and decoding process differs from H.264/AVC syntax and decoding process.
Scalable HEVC (SHVC) is the extension of the HEVC/H.265 standard that provides support for spatial and quality scalabilities, multiview HEVC (MV-HEVC) is the extension of the HEVC/H.265 that provides support for multi-view scalability, and 3D HEVC (3D-HEVC) is the extension of the HEVC/H.264 that provides support for three dimensional (3D) video coding that is more advanced and more efficient than MV-HEVC. Note that the temporal scalability is included as an integral part of the single-layer HEVC codec. The design of the multi-layer extension of HEVC employs the idea where the decoded pictures used for inter-layer prediction come only from the same AU and are treated as long-term reference pictures (LTRPs), and are assigned reference indices in the reference picture list(s) along with other temporal reference pictures in the current layer. Inter-layer prediction (ILP) is achieved at the prediction unit (PU) level by setting the value of the reference index to refer to the inter-layer reference picture(s) in the reference picture list(s).
Notably, both reference picture resampling and spatial scalability features call for resampling of a reference picture or part thereof. Reference picture resampling (RPR) can be realized at either the picture level or coding block level. However, when RPR is referred to as a coding feature, it is a feature for single-layer coding. Even so, it is possible or even preferable from a codec design point of view to use the same resampling filter for both the RPR feature of single-layer coding and the spatial scalability feature for multi-layer coding.
Some of the layers in the coded video sequence (CVS) 290 of
Each of the different OLSs may contain any number of layers. The different OLSs are generated in an effort to accommodate the coding capabilities of a variety of different devices having varying coding capabilities. For example, OLS 1, which contains only two layers, may be generated to accommodate a mobile phone with relatively limited coding capabilities. On the other hand, OLS 2, which contains four layers, may be generated to accommodate a big screen television, which is able to decode higher layers than the mobile phone. OLS 3, which contains three layers, may be generated to accommodate a personal computer, laptop computer, or a tablet computer, which may be able to decode higher layers than the mobile phone but cannot decode the highest layers like the big screen television.
The layers in
In an embodiment, when the layers are simulcast layers, only one layer is selected for decoding and output. In an embodiment, when some layers use ILP, all of the layers (e.g., the entire bitstream) are specified to be decoded, and certain layers among the layers are specified to be output layers. The output layer or layers may be, for example, 1) only the highest layer, 2) all the layers, or 3) the highest layer plus a set of indicated lower layers. For example, when the highest layer plus a set of indicated lower layers are designated for output by a flag in the VPS, Layer N+3 234 (which is the highest layer) and Layers N 231 and N+1 232 (which are lower layers) from OLS 2 are output.
Some layers in
Unfortunately, existing standards have drawbacks.
Disclosed herein are techniques that solve one or more of the foregoing problems. For example, the present disclosure provides techniques that use a scalability dimension information (SDI) view identifier length minus L syntax element to prevent a length of an SDI view ID value syntax element, which specifies a view identifier of an i-th layer in a bitstream, from being zero. The disclosed aspects/embodiments further provide techniques that prevent a bitstream from having a multiview acquisition information supplemental enhancement information (SEI) message or an auxiliary information SEI message when an SDI message is not present in the bitstream. The disclosed aspects/embodiments also provide techniques that prevent the multiview acquisition information SEI message from being scalable-nested.
The DCI 302, which may also be referred to a decoding parameter set (DPS) or decoder parameter set, is a syntax structure containing syntax elements that apply to the entire bitstream. The DCI 302 includes parameters that stay constant for the lifetime of the video bitstream (e.g., bitstream 300), which can translate to the lifetime of a session. The DCI 302 can include profile, level, and sub-profile information to determine a maximum complexity interop point that is guaranteed to be never exceeded, even if splicing of video sequences occurs within a session. It further optionally includes constraint flags, which indicate that the video bitstream will be constraint of the use of certain features as indicated by the values of those flags. With this, a bitstream can be labelled as not using certain tools, which allows among other things for resource allocation in a decoder implementation Like all parameter sets, the DCI 302 is present when first referenced, and referenced by the very first picture in a video sequence, implying that it has to be sent among the first network abstraction layer (NAL) units in the bitstream. While multiple DCIs 302 can be in the bitstream, the value of the syntax elements therein cannot be inconsistent when being referenced.
The VPS 304 includes decoding dependency or information for reference picture set construction of enhancement layers. The VPS 304 provides an overall perspective or view of a scalable sequence, including what types of operation points are provided, the profile, tier, and level of the operation points, and some other high-level properties of the bitstream that can be used as the basis for session negotiation and content selection, etc.
In an embodiment, when it is indicated that some of the layers use ILP, the VPS 304 indicates that a total number of OLSs specified by the VPS is equal to the number of layers, indicates that the i-th OLS includes the layers with layer indices from 0 to i, inclusive, and indicates that for each OLS only the highest layer in the OLS is output.
The SPS 306 contains data that is common to all the pictures in a sequence of pictures (SOP). The SPS 306 is a syntax structure containing syntax elements that apply to zero or more entire CLVSs as determined by the content of a syntax element found in the PPS referred to by a syntax element found in each picture header. In contrast, the PPS 308 contains data that is common to the entire picture. The PPS 308 is a syntax structure containing syntax elements that apply to zero or more entire coded pictures as determined by a syntax element found in each picture header (e.g., PH 312).
The DCI 302, the VPS 304, the SPS 306, and the PPS 308 are contained in different types of Network Abstraction Layer (NAL) units. A NAL unit is a syntax structure containing an indication of the type of data to follow (e.g., coded video data). NAL units are classified into video coding layer (VCL) and non-VCL NAL units. The VCL NAL units contain the data that represents the values of the samples in the video pictures, and the non-VCL NAL units contain any associated additional information such as parameter sets (important data that can apply to a number of VCL NAL units) and supplemental enhancement information (timing information and other supplemental data that may enhance usability of the decoded video signal but are not necessary for decoding the values of the samples in the video pictures).
In an embodiment, the DCI 302 is contained in a non-VCL NAL unit designated as a DCI NAL unit or a DPS NAL unit. That is, the DCI NAL unit has a DCI NAL unit type (NUT) and the DPS NAL unit has a DPS NUT. In an embodiment, the VPS 304 is contained in a non-VCL NAL unit designated as a VPS NAL unit. Therefore, the VPS NAL unit has a VPS NUT. In an embodiment, the SPS 306 is a non-VCL NAL unit designated as a SPS NAL unit. Therefore, the SPS NAL unit has an SPS NUT. In an embodiment, the PPS 308 is contained in a non-VCL NAL unit designated as a PPS NAL unit. Therefore, the PPS NAL unit has a PPS NUT.
The PH 312 is a syntax structure containing syntax elements that apply to all slices (e.g., slices 318) of a coded picture (e.g., picture 314). In an embodiment, the PH 312 is in a type of non-VCL NAL unit designated a PH NAL unit. Therefore, the PH NAL unit has a PH NUT (e.g., PH_NUT).
In an embodiment, the PH NAL unit associated with the PH 312 has a temporal ID and a layer ID. The temporal ID identifier indicates the position of the PH NAL unit, in time, relative to the other PH NAL units in the bitstream (e.g., bitstream 300). The layer ID indicates the layer (e.g., layer 131 or layer 132) that contains the PH NAL unit. In an embodiment, the temporal ID is similar to, but different from, the picture order count (POC). The POC uniquely identifies each picture in order. In a single layer bitstream, temporal ID and POC would be the same. In a multi-layer bitstream (e.g., see
In an embodiment, the PH NAL unit precedes the VCL NAL unit containing the first slice 318 of the associated picture 314. This establishes the association between the PH 312 and the slices 318 of the picture 314 associated with the PH 312 without the need to have a picture header ID signaled in the PH 312 and referred to from the slice header 320. Consequently, it can be inferred that all VCL NAL units between two PHs 312 belong to the same picture 314 and that the picture 314 is associated with the first PH 312 between the two PHs 312. In an embodiment, the first VCL NAL unit that follows a PH 312 contains the first slice 318 of the picture 314 associated with the PH 312.
In an embodiment, the PH NAL unit follows picture level parameter sets (e.g., the PPS) or higher level parameter sets such as the DCI (a.k.a., the DPS), the VPS, the SPS, the PPS, etc., having both a temporal ID and a layer ID less than the temporal ID and layer ID of the PH NAL unit, respectively. Consequently, those parameter sets are not repeated within a picture or an access unit. Because of this ordering, the PH 312 can be resolved immediately. That is, parameter sets that contain parameters relevant to an entire picture are positioned in the bitstream before the PH NAL unit. Anything that contains parameters for part of a picture is positioned after the PH NAL unit.
In one alternative, the PH NAL unit follows picture level parameter sets and prefix supplemental enhancement information (SEI) messages, or higher level parameter sets such as the DCI (a.k.a., the DPS), the VPS, the SPS, the PPS, the APS, the SEI message, etc.
The picture 314 is an array of luma samples in monochrome format or an array of luma samples and two corresponding arrays of chroma samples in 4:2:0, 4:2:2, and 4:4:4 color format.
The picture 314 may be either a frame or a field. However, in one CVS 316, either all pictures 314 are frames or all pictures 314 are fields. The CVS 316 is a coded video sequence for every coded layer video sequence (CLVS) in the video bitstream 300. Notably, the CVS 316 and the CLVS are the same when the video bitstream 300 includes a single layer. The CVS 316 and the CLVS are only different when the video bitstream 300 includes multiple layers (e.g., as shown in
Each picture 314 contains one or more slices 318. A slice 318 is an integer number of complete tiles or an integer number of consecutive complete coding tree unit (CTU) rows within a tile of a picture (e.g., picture 314). Each slice 318 is exclusively contained in a single NAL unit (e.g., a VCL NAL unit). A tile (not shown) is a rectangular region of CTUs within a particular tile column and a particular tile row in a picture (e.g., picture 314). A CTU (not shown) is a coding tree block (CTB) of luma samples, two corresponding CTBs of chroma samples of a picture that has three sample arrays, or a CTB of samples of a monochrome picture or a picture that is coded using three separate color planes and syntax structures used to code the samples. A CTB (not shown) is an N×N block of samples for some value of N such that the division of a component into CTBs is a partitioning. A block (not shown) is an M×N (M-column by N-row) array of samples (e.g., pixels), or an M×N array of transform coefficients.
In an embodiment, each slice 318 contains a slice header 320. A slice header 320 is the part of the coded slice 318 containing the data elements pertaining to all tiles or CTU rows within a tile represented in the slice 318. That is, the slice header 320 contains information about the slice 318 such as, for example, the slice type, which of the reference pictures will be used, and so on.
The pictures 314 and their slices 318 comprise data associated with the images or video being encoded or decoded. Thus, the pictures 314 and their slices 318 may be simply referred to as the payload or data being carried in the bitstream 300.
The bitstream 300 also contains one or more SEI messages, such as SEI message 322, SEI message 326, and SEI message 328. The SEI messages contain supplemental enhancement information. SEI messages can contain various types of data that indicate the timing of the video pictures or describe various properties of the coded video or how the coded video can be used or enhanced. SEI messages are also defined that can contain arbitrary user-defined data. SEI messages do not affect the core decoding process, but can indicate how the video is recommended to be post-processed or displayed. Some other high-level properties of the video content are conveyed in video usability information (VUI), such as the indication of the color space for interpretation of the video content. As new color spaces have been developed, such as for high dynamic range and wide color gamut video, additional VUI identifiers have been added to indicate them.
In an embodiment, the SEI message 322 may be an SDI SEI message. The SDI SEI message may be used to indicate which primary layers are associated with an auxiliary layer when auxiliary information is present in a bitstream. For example, the SDI SEI message may include one or more syntax elements 324 to indicate which primary layers are associated with the auxiliary layer when the auxiliary information is present in the bitstream. A discussion of various SEI messages and the syntax elements included in those SEI messages is provided below.
In an embodiment, the SEI message 326 is a multiview information SEI message, such as a multiview acquisition information SEI message. When present in the bitstream 300, the multiview information SEI message includes one or more syntax elements 324 that specify various parameters of the acquisition environment, e.g., intrinsic and extrinsic camera parameters. These parameters are useful for view warping and interpolation.
In an embodiment, the SEI message 328 may be an auxiliary information SEI message, such as a depth representation information SEI message or an alpha channel information SEI message. When present in the bitstream 300, the depth representation information SEI message includes one or more syntax elements 324 that specify various depth representation for depth views for the purpose of processing decoded texture and depth view components prior to rendering on a three dimensional (3D) display, such as view synthesis. The SEI message may be associated with an instantaneous decoder refresh (IDR) access unit for the purpose of random access. When present in the bitstream 300, the alpha channel information SEI message includes one or more syntax elements 324 that provide information about alpha channel sample values and post-processing applied to the decoded alpha plane auxiliary pictures, and one or more associated primary pictures. Blending is the process of combining two images into a single image. An image to be blended is associated with an auxiliary image identified as an alpha plane. The alpha channel information SEI message may be used to specify how the pixel values of the image to be blended are converted to another image comprising interpretation values.
Those skilled in the art will appreciate that the bitstream 300 may contain other parameters and information in practical applications.
To solve the above problems, methods as summarized below are disclosed. The techniques should be considered as examples to explain the general concepts and should not be interpreted in a narrow way. Furthermore, these techniques can be applied individually or combined in any manner.
Below are some example embodiments for some of the examples summarized above. Each embodiment can be applied to VVC. Most relevant parts that have been added or modified are depicted in a bold italic font, and some of the deleted parts are depicted in an italic font. There may be some other changes that are editorial in nature and thus not highlighted.
Each scalability dimension SEI message syntax described below includes one or more syntax elements. A syntax element may be, for example, one or more values, flags, variables, phrases, indications, indices, mappings, data elements, or a combination thereof included in the scalability dimension SEI message syntax disclosed herein. In an embodiment, the syntax elements may be organized into a group of values, flags, variables, phrases, indications, indices, mappings, and/or data elements.
Scalability Dimension SEI Message Semantics
The scalability dimension SEI message provides the scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) when bitstreamInScope may be a multiview bitstream, the view ID of each layer; and 2) when there may be auxiliary information (such as depth or alpha) carried by one or more layers in bitstreamInScope, the auxiliary ID of each layer.
The bitstreamInScope is the sequence of AUs that consists, in decoding order, of the AU containing the current scalability dimension SEI message, followed by zero or more AUs, including all subsequent AUs up to but not including any subsequent AU that contains a scalability dimension SEI message.
NOTE 1—The interpretation of auxiliary pictures associated with sdi_aux_id in the range of 128 to 159, inclusive, is specified through means other than the sdi_aux_id value.
Scalability Dimension SEI Message Semantics
The scalability dimension SEI message provides the scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) when bitstreamInScope may be a multiview bitstream, the view ID of each layer; and 2) when there may be auxiliary information (such as depth or alpha) carried by one or more layers in bitstreamInScope, the auxiliary ID of each layer.
The bitstreamInScope is the sequence of AUs that consists, in decoding order, of the AU containing the current scalability dimension SEI message, followed by zero or more AUs, including all subsequent AUs up to but not including any subsequent AU that contains a scalability dimension SEI message.
NOTE 1—The interpretation of auxiliary pictures associated with sdi_aux_id in the range of 128 to 159, inclusive, is specified through means other than the sdi_aux_id value.
Scalability Dimension SEI Message Semantics
The scalability dimension SEI message provides the scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) when bitstreamInScope may be a multiview bitstream, the view ID of each layer; and 2) when there may be auxiliary information (such as depth or alpha) carried by one or more layers in bitstreamInScope, the auxiliary ID of each layer.
The bitstreamInScope is the sequence of AUs that consists, in decoding order, of the AU containing the current scalability dimension SEI message, followed by zero or more AUs, including all subsequent AUs up to but not including any subsequent AU that contains a scalability dimension SEI message.
NOTE 1—The interpretation of auxiliary pictures associated with sdi_aux_id in the range of 128 to 159, inclusive, is specified through means other than the sdi_aux_id value.
Scalability Dimension SEI Message Semantics
The scalability dimension SEI message provides the scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) when bitstreamInScope may be a multiview bitstream, the view ID of each layer; and 2) when there may be auxiliary information (such as depth or alpha) carried by one or more layers in bitstreamInScope, the auxiliary ID of each layer.
The bitstreamInScope is the sequence of AUs that consists, in decoding order, of the AU containing the current scalability dimension SEI message, followed by zero or more AUs, including all subsequent AUs up to but not including any subsequent AU that contains a scalability dimension SEI message.
Alternatively, the following applies:
NOTE 1—The interpretation of auxiliary pictures associated with sdi_aux_id in the range of 128 to 159, inclusive, is specified through means other than the sdi_aux_id value.
.
Scalability Dimension SEI Message Semantics
The scalability dimension SEI message provides the scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) when bitstreamInScope may be a multiview bitstream, the view ID of each layer; and 2) when there may be auxiliary information (such as depth or alpha) carried by one or more layers in bitstreamInScope, the auxiliary ID of each layer.
The bitstreamInScope is the sequence of AUs that consists, in decoding order, of the AU containing the current scalability dimension SEI message, followed by zero or more AUs, including all subsequent AUs up to but not including any subsequent AU that contains a scalability dimension SEI message.
NOTE 1—The interpretation of auxiliary pictures associated with sdi_aux_id in the range of 128 to 159, inclusive, is specified through means other than the sdi_aux_id value.
,
.
.
.
Scalability Dimension SEI Message Semantics
The scalability dimension SEI message provides the scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) when bitstreamInScope may be a multiview bitstream, the view ID of each layer; and 2) when there may be auxiliary information (such as depth or alpha) carried by one or more layers in bitstreamInScope, the auxiliary ID of each layer.
The bitstreamInScope is the sequence of AUs that consists, in decoding order, of the AU containing the current scalability dimension SEI message, followed by zero or more AUs, including all subsequent AUs up to but not including any subsequent AU that contains a scalability dimension SEI message.
NOTE 1—The interpretation of auxiliary pictures associated with sdi_aux_id in the range of 128 to 159, inclusive, is specified through means other than the sdi_aux_id value.
,
.
,
.
,
.
,
.
,
.
.
,
.
Scalability Dimension SEI Message Semantics
The scalability dimension SEI message provides the scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) when bitstreamInScope may be a multiview bitstream, the view ID of each layer; and 2) when there may be auxiliary information (such as depth or alpha) carried by one or more layers in bitstreamInScope, the auxiliary ID of each layer.
The bitstreamInScope is the sequence of AUs that consists, in decoding order, of the AU containing the current scalability dimension SEI message, followed by zero or more AUs, including all subsequent AUs up to but not including any subsequent AU that contains a scalability dimension SEI message.
NOTE 1—The interpretation of auxiliary pictures associated with sdi_aux_id in the range of 128 to 159, inclusive, is specified through means other than the sdi_aux_id value.
.
,
.
,
.
,
.
,
.
.
,
.
.
.
Scalability Dimension SEI Message Semantics
The scalability dimension SEI message provides the scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) when bitstreamInScope may be a multiview bitstream, the view ID of each layer; and 2) when there may be auxiliary information (such as depth or alpha) carried by one or more layers in bitstreamInScope, the auxiliary ID of each layer.
The bitstreamInScope is the sequence of AUs that consists, in decoding order, of the AU containing the current scalability dimension SEI message, followed by zero or more AUs, including all subsequent AUs up to but not including any subsequent AU that contains a scalability dimension SEI message.
NOTE 1—The interpretation of auxiliary pictures associated with sdi_aux_id in the range of 128 to 159, inclusive, is specified through means other than the sdi_aux_id value.
.
,
.
,
,
.
.
.
Scalability Dimension SEI Message Semantics
The scalability dimension SEI message provides the scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) when bitstreamInScope may be a multiview bitstream, the view ID of each layer; and 2) when there may be auxiliary information (such as depth or alpha) carried by one or more layers in bitstreamInScope, the auxiliary ID of each layer.
The bitstreamInScope is the sequence of AUs that consists, in decoding order, of the AU containing the current scalability dimension SEI message, followed by zero or more AUs, including all subsequent AUs up to but not including any subsequent AU that contains a scalability dimension SEI message.
NOTE 1—The interpretation of auxiliary pictures associated with sdi_aux_id in the range of 128 to 159, inclusive, is specified through means other than the sdi_aux_id value.
.
,
.
,
.
,
.
.
.
Scalability Dimension SEI Message Semantics
The scalability dimension SEI message provides the scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) when bitstreamInScope may be a multiview bitstream, the view ID of each layer; and 2) when there may be auxiliary information (such as depth or alpha) carried by one or more layers in bitstreamInScope, the auxiliary ID of each layer.
The bitstreamInScope is the sequence of AUs that consists, in decoding order, of the AU containing the current scalability dimension SEI message, followed by zero or more AUs, including all subsequent AUs up to but not including any subsequent AU that contains a scalability dimension SEI message.
NOTE 1—The interpretation of auxiliary pictures associated with sdi_aux_id in the range of 128 to 159, inclusive, is specified through means other than the sdi_aux_id value.
.
,
.
,
.
,
.
,
,
,
.
.
,
.
.
,
.
—
.
Scalability Dimension SEI Message Semantics
The scalability dimension SEI message provides the scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) when bitstreamInScope may be a multiview bitstream, the view ID of each layer; and 2) when there may be auxiliary information (such as depth or alpha) carried by one or more layers in bitstreamInScope, the auxiliary ID of each layer.
The bitstreamInScope is the sequence of AUs that consists, in decoding order, of the AU containing the current scalability dimension SEI message, followed by zero or more AUs, including all subsequent AUs up to but not including any subsequent AU that contains a scalability dimension SEI message.
NOTE 1—The interpretation of auxiliary pictures associated with sdi_aux_id in the range of 128 to 159, inclusive, is specified through means other than the sdi_aux_id value.
Scalability Dimension SEI Message Semantics
The scalability dimension SEI message provides the scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) when bitstreamInScope may be a multiview bitstream, the view ID of each layer; and 2) when there may be auxiliary information (such as depth or alpha) carried by one or more layers in bitstreamInScope, the auxiliary ID of each layer.
The bitstreamInScope is the sequence of AUs that consists, in decoding order, of the AU containing the current scalability dimension SEI message, followed by zero or more AUs, including all subsequent AUs up to but not including any subsequent AU that contains a scalability dimension SEI message.
,
:
NOTE 1—The interpretation of auxiliary pictures associated with sdi_aux_id in the range of 128 to 159, inclusive, is specified through means other than the sdi_aux_id value.
Depth Representation Information SEI Message Syntax
Depth Representation Information SEI Message Semantics
The syntax elements in the depth representation information SEI message specify various parameters for auxiliary pictures of type AUX_DEPTH for the purpose of processing decoded primary and auxiliary pictures prior to rendering on a 3D display, such as view synthesis. Specifically, depth or disparity ranges for depth pictures are specified.
When present, the depth representation information SEI message shall be associated with one or more layers with sdi_aux_id value equal to AUX_DEPTH. The following semantics apply separately to each nuh_layer_id targetLayerId among the nuh_layer_id values to which the depth representation information SEI message applies.
When present, the depth representation information SEI message may be included in any access unit. It is recommended that, when present, the SEI message is included for the purpose of random access in an access unit in which the coded picture with nuh_layer_id equal to targetLayerId is an Intra Random Access Picture (IRAP) picture.
For an auxiliary picture with sdi_aux_id[targetLayerId] equal to AUX_DEPTH, an associated primary picture, if any, is a picture in the same access unit having sdi_aux_id[nuhLayerIdB] equal to 0 such that ScalabilityId[LayerIdxInVps[targetLayerId]][j] is equal to ScalabilityId[LayerIdxInVps[nuhLayerIdB]][j] for all values of j in the range of 0 to 2, inclusive, and 4 to 15, inclusive.
The information indicated in the SEI message applies to all the pictures with nuh_layer_id equal to targetLayerId from the access unit containing the SEI message up to but excluding the next picture, in decoding order, associated with a depth representation information SEI message applicable to targetLayerId or to the end of the CLVS of the nuh_layer_id equal to targetLayerId, whichever is earlier in decoding order.
.
The variable maxVal is set equal to (1<<(8+sps_bitdepth_minus8))−1, where sps_bitdepth_minus8 is the value included in or inferred for the active SPS of the layer with nuh_layer_id equal to targetLayerId.
NOTE 1—disparity_ref_view_id is present only if d_min_flag is equal to 1 or d_max_flag is equal to 1 and is useful for depth_representation_type values equal to 1 and 3.
The variables in the x column of Table Y2 are derived from the respective variables in the s, e, n and v columns of Table Y2 as follows:
NOTE 1—The above specification is similar to that found in IEC 60559:1989.
The DMin and DMax values, when present, are specified in units of a luma sample width of the coded picture with ViewId equal to ViewId of the auxiliary picture.
The units for the ZNear and ZFar values, when present, are identical but unspecified.
NOTE 2—When depth_representation_type is equal to 3, an auxiliary picture contains nonlinearly transformed depth samples. The variable DepthLUT[i], as specified below, is used to transform decoded depth sample values from the nonlinear representation to the linear representation, i.e., uniformly quantized disparity values. The shape of this transform is defined by means of line-segment approximation in two-dimensional linear-disparity-to-nonlinear-disparity space. The first (0, 0) and the last (maxVal, maxVal) nodes of the curve are predefined. Positions of additional nodes are transmitted in form of deviations (depth_nonlinear_representation_model[i]) from the straight-line curve. These deviations are uniformly distributed along the whole range of 0 to maxVal, inclusive, with spacing depending on the value of nonlinear_depth_representation_num_minus1.
The variable DepthLUT[i] for i in the range of 0 to maxVal, inclusive, is specified as follows:
When depth_representation_type is equal to 3, DepthLUT[dS] for all decoded luma sample values dS of an auxiliary picture in the range of 0 to maxVal, inclusive, represents disparity that is uniformly quantized into the range of 0 to maxVal, inclusive.
The syntax structure specifies the value of an element in the depth representation information SEI message.
The syntax structure sets the values of the OutSign, OutExp, OutMantissa and OutManLen variables that represent a floating-point value. When the syntax structure is included in another syntax structure, the variable names OutSign, OutExp, OutMantissa and OutManLen are to be interpreted as being replaced by the variable names used when the syntax structure is included.
Depth Representation Information SEI Message Syntax
Depth Representation Information SEI Message Semantics
The syntax elements in the depth representation information SEI message specify various parameters for auxiliary pictures of type AUX_DEPTH for the purpose of processing decoded primary and auxiliary pictures prior to rendering on a 3D display, such as view synthesis. Specifically, depth or disparity ranges for depth pictures are specified.
When present, the depth representation information SEI message shall be associated with one or more layers with sdi_aux_id value equal to AUX_DEPTH. The following semantics apply separately to each nuh_layer_id targetLayerId among the nuh_layer_id values to which the depth representation information SEI message applies.
When present, the depth representation information SEI message may be included in any access unit. It is recommended that, when present, the SEI message is included for the purpose of random access in an access unit in which the coded picture with nuh_layer_id equal to targetLayerId is an TRAP picture.
For an auxiliary picture with sdi_aux_id[targetLayerId] equal to AUX_DEPTH, an associated primary picture, if any, is a picture in the same access unit having sdi_aux_id[nuhLayerIdB] equal to 0 such that ScalabilityId[LayerIdxInVps[targetLayerId]][j] is equal to ScalabilityId[LayerIdxInVps[nuhLayerIdB]][j] for all values of j in the range of 0 to 2, inclusive, and 4 to 15, inclusive.
The information indicated in the SEI message applies to all the pictures with nuh_layer_id equal to targetLayerId from the access unit containing the SEI message up to but excluding the next picture, in decoding order, associated with a depth representation information SEI message applicable to targetLayerId or to the end of the CLVS of the nuh_layer_id equal to targetLayerId, whichever is earlier in decoding order.
.
.
The variable maxVal is set equal to (1<<<(8+sps_bitdepth_minus8))−1, where sps_bitdepth_minus8 is the value included in or inferred for the active SPS of the layer with nuh_layer_id equal to targetLayerId.
NOTE 1—disparity_ref_view_id is present only if d_min_flag is equal to 1 or d_max_flag is equal to 1 and is useful for depth_representation_type values equal to 1 and 3.
The variables in the x column of Table Y2 are derived from the respective variables in the s, e, n and v columns of Table Y2 as follows:
NOTE 1—The above specification is similar to that found in IEC 60559:1989.
The DMin and DMax values, when present, are specified in units of a luma sample width of the coded picture with ViewId equal to ViewId of the auxiliary picture.
The units for the ZNear and ZFar values, when present, are identical but unspecified.
NOTE 2—When depth_representation_type is equal to 3, an auxiliary picture contains nonlinearly transformed depth samples. The variable DepthLUT[i], as specified below, is used to transform decoded depth sample values from the nonlinear representation to the linear representation, i.e., uniformly quantized disparity values. The shape of this transform is defined by means of line-segment approximation in two-dimensional linear-disparity-to-nonlinear-disparity space. The first (0, 0) and the last (maxVal, maxVal) nodes of the curve are predefined. Positions of additional nodes are transmitted in form of deviations (depth_nonlinear_representation_model[i]) from the straight-line curve. These deviations are uniformly distributed along the whole range of 0 to maxVal, inclusive, with spacing depending on the value of nonlinear_depth_representation_num_minus1.
The variable DepthLUT[i] for i in the range of 0 to maxVal, inclusive, is specified as follows:
When depth_representation_type is equal to 3, DepthLUT[dS] for all decoded luma sample values dS of an auxiliary picture in the range of 0 to maxVal, inclusive, represents disparity that is uniformly quantized into the range of 0 to maxVal, inclusive.
The syntax structure specifies the value of an element in the depth representation information SEI message.
The syntax structure sets the values of the OutSign, OutExp, OutMantissa and OutManLen variables that represent a floating-point value. When the syntax structure is included in another syntax structure, the variable names OutSign, OutExp, OutMantissa and OutManLen are to be interpreted as being replaced by the variable names used when the syntax structure is included.
Alpha Channel Information SEI Message Syntax
Alpha Channel Information SEI Message Semantics
The alpha channel information SEI message provides information about alpha channel sample values and post-processing applied to the decoded alpha planes coded in auxiliary pictures of type AUX_ALPHA and one or more associated primary pictures.
For an auxiliary picture with nuh_layer_id equal to nuhLayerIdA and sdi_aux_id[nuhLayerIdA] equal to AUX_ALPHA, an associated primary picture, if any, is a picture in the same access unit having sdi_aux_id[nuhLayerIdB] equal to 0 such that ScalabilityId[LayerIdxInVps[nuhLayerIdA]][j] is equal to ScalabilityId[LayerIdxInVps[nuhLayerIdB]][j] for all values of j in the range of 0 to 2, inclusive, and 4 to 15, inclusive.
When an access unit contains an auxiliary picture picA with nuh_layer_id equal to nuhLayerIdA and sdi_aux_id[nuhLayerIdA] equal to AUX_ALPHA, the alpha channel sample values of picA persist in output order until one or more of the following conditions are true:
The following semantics apply separately to each nuh_layer_id targetLayerId among the nuh_layer_id values to which the alpha channel information SEI message applies.
Let currPic be the picture that the alpha channel information SEI message is associated with. The semantics of alpha channel information SEI message persist for the current layer in output order until one or more of the following conditions are true:
NOTE—When both alpha_channel_incr_flag and alpha_channel_clip_flag are equal to one, the clipping operation specified by alpha_channel_clip_type_flag should be applied first followed by the alteration specified by alpha_channel_incr_flag to obtain the interpretation sample value for the auxiliary picture luma sample.
Alpha Channel Information SEI Message Syntax
Alpha Channel Information SEI Message Semantics
The alpha channel information SEI message provides information about alpha channel sample values and post-processing applied to the decoded alpha planes coded in auxiliary pictures of type AUX_ALPHA and one or more associated primary pictures.
For an auxiliary picture with nuh_layer_id equal to nuhLayerIdA and sdi_aux_id[nuhLayerIdA] equal to AUX_ALPHA, an associated primary picture, if any, is a picture in the same access unit having sdi_aux_id[nuhLayerIdB] equal to 0 such that ScalabilityId[LayerIdxInVps[nuhLayerIdA]][j] is equal to ScalabilityId[LayerIdxInVps[nuhLayerIdB]][j] for all values of j in the range of 0 to 2, inclusive, and 4 to 15, inclusive.
When an access unit contains an auxiliary picture picA with nuh_layer_id equal to nuhLayerIdA and sdi_aux_id[nuhLayerIdA] equal to AUX_ALPHA, the alpha channel sample values of picA persist in output order until one or more of the following conditions are true:
The following semantics apply separately to each nuh_layer_id targetLayerId among the nuh_layer_id values to which the alpha channel information SEI message applies.
Let currPic be the picture that the alpha channel information SEI message is associated with. The semantics of alpha channel information SEI message persist for the current layer in output order until one or more of the following conditions are true:
NOTE—When both alpha_channel_incr_flag and alpha_channel_clip_flag are equal to one, the clipping operation specified by alpha_channel_clip_type_flag should be applied first followed by the alteration specified by alpha_channel_incr_flag to obtain the interpretation sample value for the auxiliary picture luma sample.
Multiview Acquisition Information SEI Message Syntax
Multiview Acquisition Information SEI Message Semantics
The multiview acquisition information (MAI) SEI message specifies various parameters of the acquisition environment. Specifically, intrinsic and extrinsic camera parameters are specified. These parameters could be used for processing the decoded views prior to rendering on a 3D display.
The following semantics apply separately to each nuh_layer_id targetLayerId among the nuh_layer_id values to which the multiview acquisition information SEI message applies.
When present, the multiview acquisition information SEI message that applies to the current layer shall be included in an access unit that contains an TRAP picture that is the first picture of a CLVS of the current layer. The information signalled in the SEI message applies to the CLVS.
An MAI SEI message that has payloadType equal to 179 (multiview acquisition) shall not be contained in a scalable nesting SEI message.
Let the current AU be the AU containing the current MAI SEI message, and the current CVS be the CVS containing the current AU.
When a CVS does not contain an SDI SEI message, the CVS shall not contain an MAI SEI message.
When an AU contains both an SDI SEI message and an MAI SEI message, the SDI SEI message shall precede the MAI SEI message in decoding order.
When the multiview acquisition information SEI message is contained in a scalable nesting SEI message, the syntax elements sn_ols_flag and sn_all_layers_flag in the scalable nesting SEI message shall be equal to 0.
The variable numViewsMinus1 is derived as follows:
Some of the views for which the multiview acquisition information is included in a multiview acquisition information SEI message may not be present.
In the semantics below, index i refers to the syntax elements and variables that apply to the layer with nuh_layer_id equal to NestingLayerId[i].
The extrinsic camera parameters are specified according to a right-handed coordinate system, where the upper left corner of the image is the origin, i.e., the (0, 0) coordinate, with the other corners of the image having non-negative coordinates. With these specifications, a 3-dimensional world point, wP=[x y z] is mapped to a 2-dimensional camera point, cP[i]=[u v 1], for the i-th camera according to:
s*cP[i]=A[i]*R
−1
[i]*(wP−T[i]) (X)
where A[i] denotes the intrinsic camera parameter matrix, R−1[i] denotes the inverse of the rotation matrix R[i], T[i] denotes the translation vector and s (a scalar value) is an arbitrary scale factor chosen to make the third coordinate of cP[i] equal to 1. The elements of A[i], R[i] and T[i] are determined according to the syntax elements signalled in this SEI message and as specified below.
The length of the mantissa_focal_length_y[i] syntax element is variable and determined as follows:
The intrinsic matrix A[i] for i-th camera is represented by
prec_rotation_param specifies the exponent of the maximum allowable truncation error for r[i][j][k] as given by 2−prec_rotation_param. The value of prec_rotation_param shall be in the range of 0 to 31, inclusive.
The rotation matrix R[i] for i-th camera is represented as follows:
The translation vector T[i] for the i-th camera is represented by:
The association between the camera parameter variables and corresponding syntax elements is specified by Table ZZ. Each component of the intrinsic and rotation matrices and the translation vector is obtained from the variables specified in Table ZZ as the variable x computed as follows:
NOTE—The above specification is similar to that found in IEC 60559:1989.
Depth Representation Information SEI Message Syntax
Depth Representation Information SEI Message Semantics
The syntax elements in the depth representation information SEI message specify various parameters for auxiliary pictures of type AUX_DEPTH for the purpose of processing decoded primary and auxiliary pictures prior to rendering on a 3D display, such as view synthesis. Specifically, depth or disparity ranges for depth pictures are specified.
When present, the depth representation information SEI message shall be associated with one or more layers with sdi_aux_id value equal to AUX_DEPTH. The following semantics apply separately to each nuh_layer_id targetLayerId among the nuh_layer_id values to which the depth representation information SEI message applies.
When present, the depth representation information SEI message may be included in any access unit. It is recommended that, when present, the SEI message is included for the purpose of random access in an access unit in which the coded picture with nuh_layer_id equal to targetLayerId is an TRAP picture.
It is a requirement of bitstream conformance that the depth representation information SEI message shall not be present in the bitstream in which the scalability dimension information SEI message is not present.
For an auxiliary picture with sdi_aux_id[targetLayerId] equal to AUX_DEPTH, an associated primary picture, if any, is a picture in the same access unit having sdi_aux_id[nuhLayerIdB] equal to 0 such that ScalabilityId[LayerIdxInVps[targetLayerId]][j] is equal to ScalabilityId[LayerIdxInVps[nuhLayerIdB]][j] for all values of j in the range of 0 to 2, inclusive, and 4 to 15, inclusive.
The information indicated in the SEI message applies to all the pictures with nuh_layer_id equal to targetLayerId from the access unit containing the SEI message up to but excluding the next picture, in decoding order, associated with a depth representation information SEI message applicable to targetLayerId or to the end of the CLVS of the nuh_layer_id equal to targetLayerId, whichever is earlier in decoding order.
The variable maxVal is set equal to (1<<<(8+sps_bitdepth_minus8))−1, where sps_bitdepth_minus8 is the value included in or inferred for the active SPS of the layer with nuh_layer_id equal to targetLayerId.
NOTE 1—disparity_ref_view_id is present only if d_min_flag is equal to 1 or d_max_flag is equal to 1 and is useful for depth_representation_type values equal to 1 and 3.
The variables in the x column of Table Y2 are derived from the respective variables in the s, e, n and v columns of Table Y2 as follows:
NOTE 1—The above specification is similar to that found in IEC 60559:1989.
The DMin and DMax values, when present, are specified in units of a luma sample width of the coded picture with ViewId equal to ViewId of the auxiliary picture.
The units for the ZNear and ZFar values, when present, are identical but unspecified.
NOTE 2—When depth_representation_type is equal to 3, an auxiliary picture contains nonlinearly transformed depth samples. The variable DepthLUT[i], as specified below, is used to transform decoded depth sample values from the nonlinear representation to the linear representation, i.e., uniformly quantized disparity values. The shape of this transform is defined by means of line-segment approximation in two-dimensional linear-disparity-to-nonlinear-disparity space. The first (0, 0) and the last (maxVal, maxVal) nodes of the curve are predefined. Positions of additional nodes are transmitted in form of deviations (depth_nonlinear_representation_model[i]) from the straight-line curve. These deviations are uniformly distributed along the whole range of 0 to maxVal, inclusive, with spacing depending on the value of nonlinear_depth_representation_num_minus1.
The variable DepthLUT[i] for i in the range of 0 to maxVal, inclusive, is specified as follows:
When depth_representation_type is equal to 3, DepthLUT[dS] for all decoded luma sample values dS of an auxiliary picture in the range of 0 to maxVal, inclusive, represents disparity that is uniformly quantized into the range of 0 to maxVal, inclusive.
The syntax structure specifies the value of an element in the depth representation information SEI message.
The syntax structure sets the values of the OutSign, OutExp, OutMantissa and OutManLen variables that represent a floating-point value. When the syntax structure is included in another syntax structure, the variable names OutSign, OutExp, OutMantissa and OutManLen are to be interpreted as being replaced by the variable names used when the syntax structure is included.
Depth Representation Information SEI Message Syntax
Depth Representation Information SEI Message Semantics
The syntax elements in the depth representation information (DRI) SEI message specify various parameters for auxiliary pictures of type AUX_DEPTH for the purpose of processing decoded primary and auxiliary pictures prior to rendering on a 3D display, such as view synthesis. Specifically, depth or disparity ranges for depth pictures are specified.
When a CVS does not contain an SDI SEI message with sdi_aux_id[i] equal to 2 for at least one value of i, no picture in the CVS shall be associated with a DRI SEI message.
When an AU contains both an SDI SEI message with sdi_aux_id[i] equal to 2 for at least one value of i and a DRI SEI message, the SDI SEI message shall precede the DRI SEI message in decoding order.
When present, the depth representation information SEI message shall be associated with one or more layers that are indicated as depth auxiliary layers by an SDI SEI message with sdi_aux_id value equal to AUX_DEPTH. The following semantics apply separately to each nuh_layer_id targetLayerId among the nuh_layer_id values to which the depth representation information SEI message applies.
When present, the depth representation information SEI message may be included in any access unit. It is recommended that, when present, the SEI message is included for the purpose of random access in an access unit in which the coded picture with nuh_layer_id equal to targetLayerId is an TRAP picture.
For an auxiliary picture with sdi_aux_id[targetLayerId] equal to AUX_DEPTH, an associated primary picture, if any, is a picture in the same access unit having sdi_aux_id[nuhLayerIdB] equal to 0 such that ScalabilityId[LayerIdxInVps[targetLayerId]][j] is equal to ScalabilityId[LayerIdxInVps[nuhLayerIdB]][j] for all values of j in the range of 0 to 2, inclusive, and 4 to 15, inclusive.
The information indicated in the SEI message applies to all the pictures with nuh_layer_id equal to targetLayerId from the access unit containing the SEI message up to but excluding the next picture, in decoding order, associated with a depth representation information SEI message applicable to targetLayerId or to the end of the CLVS of the nuh_layer_id equal to targetLayerId, whichever is earlier in decoding order.
The variable maxVal is set equal to (1<<<(8+sps_bitdepth_minus8))−1, where sps_bitdepth_minus8 is the value included in or inferred for the active SPS of the layer with nuh_layer_id equal to targetLayerId.
NOTE 1—disparity_ref_view_id is present only if d_min_flag is equal to 1 or d_max_flag is equal to 1 and is useful for depth_representation_type values equal to 1 and 3.
The variables in the x column of Table Y2 are derived from the respective variables in the s, e, n and v columns of Table Y2 as follows:
NOTE 1—The above specification is similar to that found in IEC 60559:1989.
The DMin and DMax values, when present, are specified in units of a luma sample width of the coded picture with ViewId equal to ViewId of the auxiliary picture.
The units for the ZNear and ZFar values, when present, are identical but unspecified.
NOTE 2—When depth_representation_type is equal to 3, an auxiliary picture contains nonlinearly transformed depth samples. The variable DepthLUT[i], as specified below, is used to transform decoded depth sample values from the nonlinear representation to the linear representation, i.e., uniformly quantized disparity values. The shape of this transform is defined by means of line-segment approximation in two-dimensional linear-disparity-to-nonlinear-disparity space. The first (0, 0) and the last (maxVal, maxVal) nodes of the curve are predefined. Positions of additional nodes are transmitted in form of deviations (depth_nonlinear_representation_model[i]) from the straight-line curve. These deviations are uniformly distributed along the whole range of 0 to maxVal, inclusive, with spacing depending on the value of nonlinear_depth_representation_num_minus1.
The variable DepthLUT[i] for i in the range of 0 to maxVal, inclusive, is specified as follows:
When depth_representation_type is equal to 3, DepthLUT[dS] for all decoded luma sample values dS of an auxiliary picture in the range of 0 to maxVal, inclusive, represents disparity that is uniformly quantized into the range of 0 to maxVal, inclusive.
The syntax structure specifies the value of an element in the depth representation information SEI message.
The syntax structure sets the values of the OutSign, OutExp, OutMantissa and OutManLen variables that represent a floating-point value. When the syntax structure is included in another syntax structure, the variable names OutSign, OutExp, OutMantissa and OutManLen are to be interpreted as being replaced by the variable names used when the syntax structure is included.
Alpha Channel Information SEI Message Syntax
Alpha Channel Information SEI Message Semantics
The alpha channel information SEI message provides information about alpha channel sample values and post-processing applied to the decoded alpha planes coded in auxiliary pictures of type AUX_ALPHA and one or more associated primary pictures.
For an auxiliary picture with nuh_layer_id equal to nuhLayerIdA and sdi_aux_id[nuhLayerIdA] equal to AUX_ALPHA, an associated primary picture, if any, is a picture in the same access unit having sdi_aux_id[nuhLayerIdB] equal to 0 such that ScalabilityId[LayerIdxInVps[nuhLayerIdA]][j] is equal to ScalabilityId[LayerIdxInVps[nuhLayerIdB]][j] for all values of j in the range of 0 to 2, inclusive, and 4 to 15, inclusive.
When an access unit contains an auxiliary picture picA with nuh_layer_id equal to nuhLayerIdA and sdi_aux_id[nuhLayerIdA] equal to AUX_ALPHA, the alpha channel sample values of picA persist in output order until one or more of the following conditions are true:
.
The following semantics apply separately to each nuh_layer_id targetLayerId among the nuh_layer_id values to which the alpha channel information SEI message applies.
Let currPic be the picture that the alpha channel information SEI message is associated with. The semantics of alpha channel information SEI message persist for the current layer in output order until one or more of the following conditions are true:
NOTE—When both alpha_channel_incr_flag and alpha_channel_clip_flag are equal to one, the clipping operation specified by alpha_channel_clip_type_flag should be applied first followed by the alteration specified by alpha_channel_incr_flag to obtain the interpretation sample value for the auxiliary picture luma sample.
Alpha Channel Information SEI Message Syntax
Alpha Channel Information SEI Message Semantics
The alpha channel information (ACI) SEI message provides information about alpha channel sample values and post-processing applied to the decoded alpha planes coded in auxiliary pictures of type AUX_ALPHA and one or more associated primary pictures.
For an auxiliary picture with nuh_layer_id equal to nuhLayerIdA and sdi_aux_id[nuhLayerIdA] equal to AUX_ALPHA, an associated primary picture, if any, is a picture in the same access unit having sdi_aux_id[nuhLayerIdB] equal to 0 such that ScalabilityId[LayerIdxInVps[nuhLayerIdA]][j] is equal to ScalabilityId[LayerIdxInVps[nuhLayerIdB]][j] for all values of j in the range of 0 to 2, inclusive, and 4 to 15, inclusive.
When a CVS does not contain an SDI SEI message with sdi_aux_id[i] equal to 1 for at least one value of i, no picture in the CVS shall be associated with an ACI SEI message.
When an AU contains both an SDI SEI message with sdi_aux_id[i] equal to 1 for at least one value of i and an ACI SEI message, the SDI SEI message shall precede the ACI SEI message in decoding order.
When an access unit contains an auxiliary picture picA in a layer that is indicated as an alpha auxiliary layer by an SDI SEI message with nuh_layer_id equal to nuhLayerIdA and sdi_aux_id[nuhLayerIdA] equal to AUX_ALPHA, the alpha channel sample values of picA persist in output order until one or more of the following conditions are true:
The next picture, in output order, with nuh_layer_id equal to nuhLayerIdA is output.
The following semantics apply separately to each nuh_layer_id targetLayerId among the nuh_layer_id values to which the alpha channel information SEI message applies.
Let currPic be the picture that the alpha channel information SEI message is associated with. The semantics of alpha channel information SEI message persist for the current layer in output order until one or more of the following conditions are true:
A new CLVS of the current layer begins.
The bitstream ends.
A picture picB with nuh_layer_id equal to targetLayerId in an access unit containing an alpha channel information SEI message is
. with nuh_layer_id equal to targetLayerId is output having PicOrderCnt(picB) greater than PicOrderCnt(currPic), where PicOrderCnt(picB) and PicOrderCnt(currPic) are the PicOrderCntVal values of picB and currPic, respectively, immediately after the invocation of the decoding process for picture order count for picB.
NOTE—When both alpha_channel_incr_flag and alpha_channel_clip_flag are equal to one, the clipping operation specified by alpha_channel_clip_type_flag should be applied first followed by the alteration specified by alpha_channel_incr_flag to obtain the interpretation sample value for the auxiliary picture luma sample.
Scalability Dimension SEI Message Syntax
Scalability Dimension SEI Message Semantics
The scalability dimension SEI message provides the scalability dimension information for each layer in bitstreamInScope (defined below), such as 1) when bitstreamInScope may be a multiview bitstream, the view ID of each layer; and 2) when there may be auxiliary information (such as depth or alpha) carried by one or more layers in bitstreamInScope, the auxiliary ID of each layer.
The bitstreamInScope is the sequence of AUs that consists, in decoding order, of the AU containing the current scalability dimension SEI message, followed by zero or more AUs, including all subsequent AUs up to but not including any subsequent AU that contains a scalability dimension SEI message.
The variable NumViews is derived as follows:
NOTE 3— The interpretation of auxiliary pictures associated with sdi_aux_id in the range of 128 to 159, inclusive, is specified through means other than the sdi_aux_id value.
Multiview Acquisition Information SEI Message
Multiview Acquisition Information SEI Message Syntax
Multiview Acquisition Information SEI Message Semantics
The multiview acquisition information SEI message specifies various parameters of the acquisition environment. Specifically, intrinsic and extrinsic camera parameters are specified. These parameters could be used for processing the decoded views prior to rendering on a 3D display.
.
,
.
,
.
,
.
The following semantics apply separately to each nuh_layer_id targetLayerId among the nuh_layer_id values to which the multiview acquisition information SEI message applies.
When present, the multiview acquisition information SEI message that applies to the current layer shall be included in an access unit that contains an TRAP picture that is the first picture of a CLVS of the current layer. The information signalled in the SEI message applies to the CLVS.
When the multiview acquisition information SEI message is contained in a scalable nesting SEI message, the syntax elements sn_ols_flag and sn_all_layers_flag in the scalable nesting SEI message shall be equal to 0.
The variable numViewsMinus1 is derived as follows:
Some of the views for which the multiview acquisition information is included in a multiview acquisition information SEI message may not be present.
In the semantics below, index i refers to the syntax elements and variables that apply to the layer with nuh_layer_id equal to NestingLayerId[i].
The extrinsic camera parameters are specified according to a right-handed coordinate system, where the upper left corner of the image is the origin, i.e., the (0, 0) coordinate, with the other corners of the image having non-negative coordinates. With these specifications, a 3-dimensional world point, wP=[x y z] is mapped to a 2-dimensional camera point, cP[i]=[u v 1], for the i-th camera according to:
s*cP[i]=A[i]*R
−1
[i]*(wP−T[i]) (X)
where A[i] denotes the intrinsic camera parameter matrix, R−1[i] denotes the inverse of the rotation matrix R[i], T[i] denotes the translation vector and s (a scalar value) is an arbitrary scale factor chosen to make the third coordinate of cP[i] equal to 1. The elements of A[i], R[i] and T[i] are determined according to the syntax elements signalled in this SEI message and as specified below.
The length of the mantissa_focal_length_y[i] syntax element is variable and determined as follows:
If exponent_focal_length_y[i] is equal to 0, the length is Max(0, prec_focal_length−30).
The intrinsic matrix A[i] for i-th camera is represented by:
The rotation matrix R[i] for i-th camera is represented as follows:
The translation vector T[i] for the i-th camera is represented by:
The association between the camera parameter variables and corresponding syntax elements is specified by Table ZZ. Each component of the intrinsic and rotation matrices and the translation vector is obtained from the variables specified in Table ZZ as the variable x computed as follows:
NOTE—The above specification is similar to that found in IEC 60559:1989.
The video processing system 400 may include a coding component 404 that may implement the various coding or encoding methods described in the present document. The coding component 404 may reduce the average bitrate of video from the input 402 to the output of the coding component 404 to produce a coded representation of the video. The coding techniques are therefore sometimes called video compression or video transcoding techniques. The output of the coding component 404 may be either stored, or transmitted via a communication connected, as represented by the component 406. The stored or communicated bitstream (or coded) representation of the video received at the input 402 may be used by the component 408 for generating pixel values or displayable video that is sent to a display interface 410. The process of generating user-viewable video from the bitstream representation is sometimes called video decompression. Furthermore, while certain video processing operations are referred to as “coding” operations or tools, it will be appreciated that the coding tools or operations are used at an encoder and corresponding decoding tools or operations that reverse the results of the coding will be performed by a decoder.
Examples of a peripheral bus interface or a display interface may include universal serial bus (USB) or high definition multimedia interface (HDMI) or Displayport, and so on. Examples of storage interfaces include SATA (serial advanced technology attachment), Peripheral Component Interconnect (PCI), Integrated Drive Electronics (IDE) interface, and the like. The techniques described in the present document may be embodied in various electronic devices such as mobile phones, laptops, smartphones or other devices that are capable of performing digital data processing and/or video display.
Source device 610 may include a video source 612, a video encoder 614, and an input/output (I/O) interface 616.
Video source 612 may include a source such as a video capture device, an interface to receive video data from a video content provider, and/or a computer graphics system for generating video data, or a combination of such sources. The video data may comprise one or more pictures. Video encoder 614 encodes the video data from video source 612 to generate a bitstream. The bitstream may include a sequence of bits that form a coded representation of the video data. The bitstream may include coded pictures and associated data. The coded picture is a coded representation of a picture. The associated data may include sequence parameter sets, picture parameter sets, and other syntax structures. I/O interface 616 may include a modulator/demodulator (modem) and/or a transmitter. The encoded video data may be transmitted directly to destination device 620 via I/O interface 616 through network 630. The encoded video data may also be stored onto a storage medium/server 640 for access by destination device 620.
Destination device 620 may include an I/O interface 626, a video decoder 624, and a display device 622.
I/O interface 626 may include a receiver and/or a modem. I/O interface 626 may acquire encoded video data from the source device 610 or the storage medium/server 640. Video decoder 624 may decode the encoded video data. Display device 622 may display the decoded video data to a user. Display device 622 may be integrated with the destination device 620, or may be external to destination device 620 which may be configured to interface with an external display device.
Video encoder 614 and video decoder 624 may operate according to a video compression standard, such as the High Efficiency Video Coding (HEVC) standard, Versatile Video Coding (VVC) standard, and other current and/or further standards.
Video encoder 700 may be configured to perform any or all of the techniques of this disclosure. In the example of
The functional components of video encoder 700 may include a partition unit 701, a prediction unit 702 which may include a mode selection unit 703, a motion estimation unit 704, a motion compensation unit 705 and an intra prediction unit 706, a residual generation unit 707, a transform unit 708, a quantization unit 709, an inverse quantization unit 710, an inverse transform unit 711, a reconstruction unit 712, a buffer 713, and an entropy encoding unit 714.
In other examples, video encoder 700 may include more, fewer, or different functional components. In an example, prediction unit 702 may include an intra block copy (IBC) unit. The IBC unit may perform prediction in an IBC mode in which at least one reference picture is a picture where the current video block is located.
Furthermore, some components, such as motion estimation unit 704 and motion compensation unit 705 may be highly integrated, but are represented in the example of
Partition unit 701 may partition a picture into one or more video blocks. Video encoder 614 and video decoder 624 of
Mode selection unit 703 may select one of the coding modes, intra or inter, e.g., based on error results, and provide the resulting intra- or inter-coded block to a residual generation unit 707 to generate residual block data and to a reconstruction unit 712 to reconstruct the encoded block for use as a reference picture. In some examples, mode selection unit 703 may select a combination of intra and inter prediction (CLIP) mode in which the prediction is based on an inter prediction signal and an intra prediction signal. Mode selection unit 703 may also select a resolution for a motion vector (e.g., a sub-pixel or integer pixel precision) for the block in the case of inter-prediction.
To perform inter prediction on a current video block, motion estimation unit 704 may generate motion information for the current video block by comparing one or more reference frames from buffer 713 to the current video block. Motion compensation unit 705 may determine a predicted video block for the current video block based on the motion information and decoded samples of pictures from buffer 713 other than the picture associated with the current video block.
Motion estimation unit 704 and motion compensation unit 705 may perform different operations for a current video block, for example, depending on whether the current video block is in an I slice, a P slice, or a B slice. I-slices (or I-frames) are the least compressible but don't require other video frames to decode. S-slices (or P-frames) can use data from previous frames to decompress and are more compressible than I-frames. B-slices (or B-frames) can use both previous and forward frames for data reference to get the highest amount of data compression.
In some examples, motion estimation unit 704 may perform uni-directional prediction for the current video block, and motion estimation unit 704 may search reference pictures of list 0 or list 1 for a reference video block for the current video block. Motion estimation unit 704 may then generate a reference index that indicates the reference picture in list 0 or list 1 that contains the reference video block and a motion vector that indicates a spatial displacement between the current video block and the reference video block. Motion estimation unit 704 may output the reference index, a prediction direction indicator, and the motion vector as the motion information of the current video block. Motion compensation unit 705 may generate the predicted video block of the current block based on the reference video block indicated by the motion information of the current video block.
In other examples, motion estimation unit 704 may perform bi-directional prediction for the current video block, motion estimation unit 704 may search the reference pictures in list 0 for a reference video block for the current video block and may also search the reference pictures in list 1 for another reference video block for the current video block. Motion estimation unit 704 may then generate reference indexes that indicate the reference pictures in list 0 and list 1 containing the reference video blocks and motion vectors that indicate spatial displacements between the reference video blocks and the current video block. Motion estimation unit 704 may output the reference indexes and the motion vectors of the current video block as the motion information of the current video block. Motion compensation unit 705 may generate the predicted video block of the current video block based on the reference video blocks indicated by the motion information of the current video block.
In some examples, motion estimation unit 704 may output a full set of motion information for decoding processing of a decoder.
In some examples, motion estimation unit 704 may not output a full set of motion information for the current video. Rather, motion estimation unit 704 may signal the motion information of the current video block with reference to the motion information of another video block. For example, motion estimation unit 704 may determine that the motion information of the current video block is sufficiently similar to the motion information of a neighboring video block.
In one example, motion estimation unit 704 may indicate, in a syntax structure associated with the current video block, a value that indicates to the video decoder 624 that the current video block has the same motion information as another video block.
In another example, motion estimation unit 704 may identify, in a syntax structure associated with the current video block, another video block and a motion vector difference (MVD). The motion vector difference indicates a difference between the motion vector of the current video block and the motion vector of the indicated video block. The video decoder 624 may use the motion vector of the indicated video block and the motion vector difference to determine the motion vector of the current video block.
As discussed above, video encoder 614 may predictively signal the motion vector. Two examples of predictive signaling techniques that may be implemented by video encoder 614 include advanced motion vector prediction (AMVP) and merge mode signaling.
Intra prediction unit 706 may perform intra prediction on the current video block. When intra prediction unit 706 performs intra prediction on the current video block, intra prediction unit 706 may generate prediction data for the current video block based on decoded samples of other video blocks in the same picture. The prediction data for the current video block may include a predicted video block and various syntax elements.
Residual generation unit 707 may generate residual data for the current video block by subtracting (e.g., indicated by the minus sign) the predicted video block(s) of the current video block from the current video block. The residual data of the current video block may include residual video blocks that correspond to different sample components of the samples in the current video block.
In other examples, there may be no residual data for the current video block, for example in a skip mode, and residual generation unit 707 may not perform the subtracting operation.
Transform unit 708 may generate one or more transform coefficient video blocks for the current video block by applying one or more transforms to a residual video block associated with the current video block.
After transform unit 708 generates a transform coefficient video block associated with the current video block, quantization unit 709 may quantize the transform coefficient video block associated with the current video block based on one or more quantization parameter (QP) values associated with the current video block.
Inverse quantization unit 710 and inverse transform unit 711 may apply inverse quantization and inverse transforms to the transform coefficient video block, respectively, to reconstruct a residual video block from the transform coefficient video block. Reconstruction unit 712 may add the reconstructed residual video block to corresponding samples from one or more predicted video blocks generated by the prediction unit 702 to produce a reconstructed video block associated with the current block for storage in the buffer 713.
After reconstruction unit 712 reconstructs the video block, loop filtering operation may be performed to reduce video blocking artifacts in the video block.
Entropy encoding unit 714 may receive data from other functional components of the video encoder 700. When entropy encoding unit 714 receives the data, entropy encoding unit 714 may perform one or more entropy encoding operations to generate entropy encoded data and output a bitstream that includes the entropy encoded data.
The video decoder 800 may be configured to perform any or all of the techniques of this disclosure. In the example of
In the example of
Entropy decoding unit 801 may retrieve an encoded bitstream. The encoded bitstream may include entropy coded video data (e.g., encoded blocks of video data). Entropy decoding unit 801 may decode the entropy coded video data, and from the entropy decoded video data, motion compensation unit 802 may determine motion information including motion vectors, motion vector precision, reference picture list indexes, and other motion information. Motion compensation unit 802 may, for example, determine such information by performing the AMVP and merge mode signaling.
Motion compensation unit 802 may produce motion compensated blocks, possibly performing interpolation based on interpolation filters. Identifiers for interpolation filters to be used with sub-pixel precision may be included in the syntax elements.
Motion compensation unit 802 may use interpolation filters as used by video encoder 614 during encoding of the video block to calculate interpolated values for sub-integer pixels of a reference block. Motion compensation unit 802 may determine the interpolation filters used by video encoder 614 according to received syntax information and use the interpolation filters to produce predictive blocks.
Motion compensation unit 802 may use some of the syntax information to determine sizes of blocks used to encode frame(s) and/or slice(s) of the encoded video sequence, partition information that describes how each macroblock of a picture of the encoded video sequence is partitioned, modes indicating how each partition is encoded, one or more reference frames (and reference frame lists) for each inter-encoded block, and other information to decode the encoded video sequence.
Intra prediction unit 803 may use intra prediction modes for example received in the bitstream to form a prediction block from spatially adjacent blocks. Inverse quantization unit 804 inverse quantizes, i.e., de-quantizes, the quantized video block coefficients provided in the bitstream and decoded by entropy decoding unit 801. Inverse transform unit 805 applies an inverse transform.
Reconstruction unit 806 may sum the residual blocks with the corresponding prediction blocks generated by motion compensation unit 802 or intra prediction unit 803 to form decoded blocks. If desired, a deblocking filter may also be applied to filter the decoded blocks in order to remove blockiness artifacts. The decoded video blocks are then stored in buffer 807, which provides reference blocks for subsequent motion compensation/intra prediction and also produces decoded video for presentation on a display device.
In block 902, the coding apparatus uses a scalability dimension information (SDI) supplemental enhancement information (SEI) message to indicate an SDI view identifier length minus L syntax element. The SDI SEI message is a type of SEI message like, for example, the SEI message in the bitstream 300 of
In block 904, the coding apparatus converts between a video media file and the bitstream based on the SDI SEI message.
When implemented in an encoder, converting includes receiving a media file (e.g., a video unit) and encoding an SEI message into a bitstream. When implemented in a decoder, converting includes receiving the bitstream including the SEI message, and decoding the SEI message in the bitstream to generate the video media file.
In an embodiment, the SDI view identifier length minus L syntax element is configured to prevent a length of an SDI view identifier value syntax element, which specifies a view identifier of an i-th layer in a bitstream, from being zero. In an embodiment, L is equal to 1. In an embodiment, the SDI view identifier length minus L syntax element is designated sdi_view_id_len_minus1. In an embodiment, the SDI view identifier value syntax element is designated sdi_view_id_val[i]. In an embodiment, the SDI view identifier length minus L syntax element, plus one, specifies the length of the SDI view identifier value syntax element.
In an embodiment, the SDI view identifier length minus L syntax element is coded as an unsigned integer using N bits. By way of example, an unsigned integer is an integer (e.g., a whole number) that does not have a sign (e.g., positive or negative) associated therewith. In an embodiment, N is equal to 4.
In an embodiment, the SDI view identifier length minus L syntax element is coded as a fixed-pattern bitstring using N bits, a signed integer using N bits, a truncated binary, a signed integer K-th order Exp-Golomb-coded syntax element where K is equal to 0, or an unsigned integer M-th order Exp-Golomb-coded syntax element where M is equal to 0. A bitstring is an array data structure that compactly stores bits. A fixed-pattern bitstring is an array data structure having a fixed pattern. A signed integer is an integer (e.g., a whole number) that has a sign (e.g., positive or negative) associated therewith. Truncated binary, or truncated binary encoding, is an entropy encoding typically used for uniform probability distributions with a finite alphabet. An exponential-Golomb code (Exp-Golomb code) is a type of universal code.
In an embodiment, the bitstream is a bitstream in scope. In an embodiment, the bitstream in scope is a sequence of access units (AUs) that consists, in decoding order, of an initial AU containing the SDI SEI message followed by zero or more subsequent AUs up to, but not including, any subsequent AU that contains another SDI SEI message.
In an embodiment, a multiview information SEI message and an auxiliary information SEI message are not present in a coded video sequence (CVS) unless the SDI SEI message is present in the CVS.
In an embodiment, the multiview information SEI message comprises a multiview acquisition information SEI message. In an embodiment, the auxiliary information SEI message comprises a depth representation information SEI message. In an embodiment, the auxiliary information SEI message comprises an alpha channel information SEI message.
In an embodiment, one or more of an SDI multiview information flag (e.g., sdi_multiview_info flag) and an SDI auxiliary information flag (e.g., sdi_auxiliary_info_flag) are equal to 1 when the multiview information SEI message or the auxiliary information SEI message are present in the bitstream. A flag is a variable or single-bit syntax element that can take one of the two possible values: 0 and 1.
In an embodiment, the multiview information SEI message comprises a multiview acquisition information SEI message, and the multiview acquisition information SEI message is not scalable-nested. A scalable-nested SEI message is an SEI message within a scalable nesting SEI message. A scalable nesting SEI message is a message that contains a plurality of scalable-nested SEI messages that correspond to one or more output layer sets or one or more layers in a multi-layer bitstream.
In an embodiment, an SEI message in the bitstream and having a payload type equal to 179 is constrained from being included in a scalable nesting SEI message. In an embodiment, an SEI message in the bitstream and having a payload type equal to 3, 133, 179, 180, or 205 is constrained from being included in a scalable nesting SEI message.
In an embodiment, the method 900 may utilize or incorporate one or more of the features or processes of the other methods disclosed herein.
A listing of solutions preferred by some embodiments is provided next.
The following solutions show example embodiments of techniques discussed in the present disclosure (e.g., Example 1).
The following documents may include additional details related to the techniques disclosed herein:
The disclosed and other solutions, examples, embodiments, modules and the functional operations described in this document can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this document and their structural equivalents, or in combinations of one or more of them. The disclosed and other embodiments can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more them. The term “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus.
A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
The processes and logic flows described in this document can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random-access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and compact disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
While this patent document contains many specifics, these should not be construed as limitations on the scope of any subject matter or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular techniques. Certain features that are described in this patent document in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Moreover, the separation of various system components in the embodiments described in this patent document should not be understood as requiring such separation in all embodiments.
Only a few implementations and examples are described and other implementations, enhancements and variations can be made based on what is described and illustrated in this patent document.
| Number | Date | Country | Kind |
|---|---|---|---|
| PCT/CN2021/085292 | Apr 2021 | WO | international |
This application is a continuation of International Patent Application No. PCT/CN2022/084992, filed on Apr. 2, 2022, which claims the benefit of International Application No. PCT/CN2021/085292, filed on Apr. 2, 2021. All the aforementioned patent applications are hereby incorporated by reference in their entireties.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/CN2022/084992 | Apr 2022 | US |
| Child | 18476650 | US |
