Patent Application
20140140417
This disclosure relates in general to the field of communications and, more particularly, to providing alignment of multiple transcoders for adaptive bitrate streaming in a network environment.
Adaptive streaming, sometimes referred to as dynamic streaming, involves the creation of multiple copies of the same multimedia (audio, video, text, etc.) content at different quality levels. Different levels of quality are generally achieved by using different compression ratios, typically specified by nominal bitrates. Various adaptive streaming methods such as Microsoft's HTTP Smooth Streaming “HSS”, Apple's HTTP Live Streaming “HLS”, and Adobe's HTTP Dynamic Streaming “HDS”, MPEG Dynamic Streaming over HTTP “DASH”, involve seamlessly switching between the various quality levels during playback, for example, in response to changes in available network bandwidth. To achieve this seamless switching, the video and audio tracks have special boundaries where the switching can occur. These boundaries are designated in various ways, but should include a timestamp at fragment boundaries. These fragment boundary timestamps should be the same in all of the video tracks and all of the audio tracks of the multimedia content. Accordingly, they should have the same integer numerical value and refer to the same sample from the source content.
To provide a more complete understanding of the present disclosure and features and advantages thereof, reference is made to the following description, taken in conjunction with the accompanying figures, wherein like reference numerals represent like parts, in which:
A method is provided in one example and includes receiving source video including associated video timestamps and determining a theoretical fragment boundary timestamp based upon one or more characteristics of the source video and the received video timestamps. The theoretical fragment boundary timestamp identifies a fragment including one or more video frames of the source video. The method further includes determining an actual fragment boundary timestamp based upon the theoretical fragment boundary timestamp and one or more of the received video timestamps, transcoding the source video according to the actual fragment boundary timestamp, and outputting the transcoded source video including the actual fragment boundary timestamp.
In more particular embodiments, the one or more characteristics of the source video include a fragment duration associated with the source video and a frame rate associated with the source video. In still other particular embodiments, determining the theoretical fragment boundary timestamp includes determining the theoretical fragment boundary timestamp from a lookup table. In still other particular embodiments, determining the actual fragment boundary timestamp includes determining the first received video timestamp that is greater than or equal to the theoretical fragment boundary timestamp.
In other more particular embodiments, the method further includes determining a theoretical segment boundary timestamp based upon one or more characteristics of the source video and the received video timestamps. The theoretical segment boundary timestamp identifies a segment including one or more fragments of the source video. The method further includes determining an actual segment boundary timestamp based upon the theoretical segment boundary timestamp and one or more of the received video timestamps.
In other more particular embodiments, the method further includes receiving source audio including associated audio timestamps, determining a theoretical re-framing boundary timestamp based upon one or more characteristics of the source audio, and determining an actual re-framing boundary timestamp based upon the theoretical audio re-framing boundary timestamp and one or more of the received audio timestamps. The method further includes transcoding the source audio according to the actual re-framing boundary timestamp, and outputting the transcoded source audio including the actual re-framing boundary timestamp. In more particular embodiments, determining the actual re-framing boundary timestamp includes determining the first received audio timestamp that is greater than or equal to the theoretical re-framing boundary timestamp.
Referring now to
First transcoder device 104a, second transcoder device 104b, and third transcoder device 104c are each configured to receive the source video and/or audio and transcode the source video and/or audio to a different quality level such as a different bitrate, framerate, and/or format from the source video and/or audio. In particular, first transcoder 104a is configured to produce first transcoded video/audio, second transcoder 104b is configured to produce second transcoded video/audio, and third transcoder 104b is configured to produce third transcoded video/audio. In various embodiments, first transcoded video/audio, second transcoded video/audio, and third transcoded video/audio are each transcoded at a different quality level from each other. First transcoder device 104a, second transcoder device 104b and third transcoder device 104c are further configured to produce timestamps for the video and/or audio such that the timestamps produced by each of first transcoder device 104a, second transcoder device 104b and third transcoder device 104c are in alignment with one another as will be further described herein. First transcoder device 104a, second transcoder device 104b and third transcoder device 104c then each provide their respective timestamp aligned transcoded video and/or audio to encapsulator device 105. Encapsulator device 105 performs packet encapsulation on the respective transcoded video/audio and sends the encapsulated video and/or audio to media server 106.
Media server 106 stores the respective encapsulated video and/or audio and included timestamps within storage device 108. Although the embodiment illustrated in
Media server 106 is further configured to stream one or more of the stored transcoded video and/or audio files to one or more of first destination device 110a and second destination device 110b. First destination device 110a and second destination device 110b are configured to receive and decode the video and/or audio stream and present the decoded video and/or audio to a user. In various embodiments, the video and/or audio stream provided to either first destination device 110a or second destination device 110b may switch between one of the transcoded video and/or audio streams to another of the transcoded video and/or audio streams, for example, due to changes in available bandwidth, via adaptive streaming. Due to the alignment of the timestamps between each of the transcoded video and/or audio streams, first destination device 110a and second destination device 110b may seamlessly switch between presentation of the video and/or audio.
Adaptive streaming, sometimes referred to as dynamic streaming, involves the creation of multiple copies of the same multimedia (audio, video, text, etc.) content at different quality levels. Different levels of quality are generally achieved by using different compression ratios, typically specified by nominal bitrates. Various adaptive streaming methods such as Microsoft's HTTP Smooth Streaming “HSS”, Apple's HTTP Live Streaming “HLS”, Adobe's HTTP Dynamic Streaming “HDS”, and MPEG Dynamic Streaming over HTTP involve seamlessly switching between the various quality levels during playback, for example, in response to changes in available network bandwidth. To achieve this seamless switching, the video and audio tracks have special boundaries where the switching can occur. These boundaries are designated in various ways, but should include a timestamp at fragment boundaries. These fragment boundary timestamps should be the same for all of the video tracks and all of the audio tracks of the multimedia content. Accordingly, they should have the same integer numerical value and refer to the same sample from the source content.
Several transcoders exist that can accomplish an alignment of timestamps internally within a single transcoder. In contrast, various embodiments described herein provide for alignment of timestamps for multiple transcoder configurations such as those used for teaming, failover, or redundancy scenarios in which there are multiple transcoders encoding the same source in parallel (“teaming” or “redundancy”) or serially (“failover”). A problem that arises when multiple transcoders are used is that although the multiple transcoders are operating on the same source video and/or audio, the transcoders may not receive the same exact sequence of input timestamps. This may be a result of, for example, a transcoder A starting later than a transcoder B. Alternately, this could occur as result of corruption/loss of signal between source and transcoder A and/or transcoder B. Each of the transcoders should still compute the same output timestamps for the fragment boundaries.
Various embodiments described herein provide for aligning of video and audio timestamps for multiple transcoders without requiring communication of state information between transcoders. Instead, in various embodiments described herein first transcoder device 104a, second transcoder device 104b, and third transcoder device 104c “pass through” incoming timestamps to an output and rely on a set of rules to produce identical fragment boundary timestamps and audio frame timestamps from each of first transcoder device 104a, second transcoder device 104b, and third transcoder device 104c. Discontinuities in the input source, if they occur, are passed through to the output. If the input to the transcoder(s) is continuous and all frames have an explicit Presentation Time Stamp (PTS) value, then the output of the transcoder(s) can be used directly by an encapsulator. In practice, it is likely that there will be at least occasional loss of the input signal, and some input sources group multiple video frames into one packetized elementary stream (PES) packet. In order to be tolerant of all possible input source characteristics, it is possible that there will still be some differences in the output timestamps of two transcoders that are processing the same input source. However, the procedures as described in various embodiments result in “aligned” outputs that can be “finalized” by downstream components to meet their specific requirements without having to re-encode any of the video or audio. Specifically, in a particular embodiment, the video closed Group of Pictures (GOP) boundaries (i.e. Instantaneous Decoder Refresh (IDR) frames) and the audio frame boundaries will be placed consistently. The timestamps of the transcoder input source may either be used directly as the timestamps of the aligned transcoder output, or they may be embedded elsewhere in the stream, or both. This allows downstream equipment to make any adjustments that may be necessary for decoding and presentation of the video and/or audio content.
Various embodiments are described with respect to a ISO standard 13818-1 MPEG2 transport stream input/output to a transcoder, however the principles described herein are similarly applicable to other types of video streams such as any system in which an encoder ingests baseband (i.e. SDI or analog) video or an encoder/transcoder that outputs to a format other than, for example, an ISO 13818-1 MPEG2 transport stream.
An MPEG2 transport stream transcoder receives timestamps in Presentation Time Stamp (PTS) “ticks” which represent 1/90000 of 1 second. The maximum value of the PTS tick is 2̂33 or 8589934592, approximately 26.5 hours. When it reaches this value it “wraps” back to a zero value. In addition to the discontinuity introduced by the wrap, there can be jumps forward or backward at any time. An ideal source does not have such jumps, but in reality such jumps often do occur. Additionally, it cannot be assumed that all video and audio frames will have an explicit PTS associated with them.
First, assume a situation in which the frame rate of the source video is constant and there are no discontinuities in the source video. In such a situation, video timestamps may then simply be passed through the transcoder. However there is an additional step of determining which video timestamps are placed as fragment boundaries. To ensure that all transcoders place fragment boundaries consistently, the transcoders compute nominal frame boundary PTS values based on the nominal frame rate of the source and a user-specified nominal fragment duration. For example, for a typical frame rate of 29.97 fps (30/1.001), the frame duration is 3003 ticks. In a particular embodiment, the nominal fragment duration can be specified in terms of frames. In a specific embodiment, the nominal fragment duration may be set to a typical value of sixty (60) frames. In this case, the nominal fragment boundaries may be set at 0, 180180, 360360, etc. The first PTS value received that is equal to or greater than a nominal boundary and less than the next nominal boundary may be used as an actual fragment boundary.
For an ideal source having a constant frame rate and no discontinuities, the above-described procedure produces the same exact fragment boundary timestamps on each of multiple transcoders. In practice, the transcoder input may have at least occasional discontinuities. In the presence of discontinuities, if first transcoder device 104a receives a PTS at 180180 and second transcoder device 104bB does not, then each of first transcoder device 104a and second transcoder device 104b may produce one fragment with mismatched timestamps (180180 vs. 183183 for example). Downstream equipment, such as an encapsulator associated with media server 106, may detect this difference and compensate as required. The downstream equipment may, for example, use knowledge of the nominal boundary locations and the original input PTS values to the transcoders. To allow for reduced video frame rate in some of the output streams, care has to be taken to ensure that the lower frame rate streams do not discard the video frame that the higher frame rate stream(s) would select as their fragment boundary frame. Various embodiments of video boundary PTS alignment are further described herein.
With audio, designating fragment boundaries can be performed in a similar manner as to video if needed. However, there is an additional complication with audio streams, because while it is not always necessary to designate fragment boundaries, it is necessary to group audio samples into frames. In addition, it is often impossible to pass through audio timestamps because input audio frame duration is often different from output audio frame duration. The duration of an audio frame depends on the audio compression format and audio sample rate. Typical input audio compression formats are AC-3 developed by Dolby Laboratories, Advanced Audio Coding (AAC), and MPEG. A typical input audio sample rate is 48 kHz. Most of the adaptive streaming specs support AAC with a sample rates from the 48 kHz “family” (48 kHz, 32 kHz, 24 kHz, 16 kHz . . . ) and the 44.1 kHz family (44.1 kHz, 22.05 kHz, 11.025 kHz . . . ).
Various embodiments described herein exploit the fact that while audio PTS values cannot be passed through directly, there can still be a deterministic relationship between the input timestamp and output timestamp. Regarding an example in which the input is 48 kHz AC-3 and the output is 48 kHz AAC. In this case, every 2 AC-3 frames form 3 AAC frames. Of each pair of input AC-3 frame PTS values, the first or “even” AC3 PTS is passed through as the first AAC PTS, and the remaining two AAC PTS values (if needed) are extrapolated from the first by adding 1920 and 3840. For each AC3 PTS a determination is made whether the given AC3 PTS is “even” or “odd.” In various embodiments, the determination of whether a particular PTS is even or odd can be determined either via a computation or equivalent lookup table. Various embodiments of audio frame PTS alignment are further described herein.
In one particular instance, communication system 100 can be associated with a service provider digital subscriber line (DSL) deployment. In other examples, communication system 100 would be equally applicable to other communication environments, such as an enterprise wide area network (WAN) deployment, cable scenarios, broadband generally, fixed wireless instances, fiber to the x (FTTx), which is a generic term for any broadband network architecture that uses optical fiber in last-mile architectures. Communication system 100 may include a configuration capable of transmission control protocol/internet protocol (TCP/IP) communications for the transmission and/or reception of packets in a network. Communication system 100 may also operate in conjunction with a user datagram protocol/IP (UDP/IP) or any other suitable protocol, where appropriate and based on particular needs.
Referring now to
In one implementation, transcoder device 200 is a network element that includes software to achieve (or to foster) the transcoding and/or timestamp alignment operations as outlined herein in this Specification. Note that in one example, each of these elements can have an internal structure (e.g., a processor, a memory element, etc.) to facilitate some of the operations described herein. In other embodiments, these transcoding and/or timestamp alignment operations may be executed externally to this element, or included in some other network element to achieve this intended functionality. Alternatively, transcoder device 200 may include software (or reciprocating software) that can coordinate with other network elements in order to achieve the operations, as outlined herein. In still other embodiments, one or several devices may include any suitable algorithms, hardware, software, components, modules, interfaces, or objects that facilitate the operations thereof.
In order to support video and audio services for Adaptive Bit Rate (ABR) applications, there is a need to synchronize both the video and audio components of these services. When watching video services delivered over, for example, the internet, the bandwidth of the connection can change over time. Adaptive bitrate streaming attempts to maximize the quality of the delivered video service by adapting its bitrate to the available bandwidth. In order to achieve this, a video service is encoded as a set of several different video output profiles, each having a certain bitrate, resolution and framerate. Referring again to
Since combining files from different video profiles should result in a seamless viewing experience, video chunks associated with the different profiles should be synchronized in a frame-accurate way, i.e. the corresponding chunk of each profile should start with exactly the same frame to avoid discontinuities in the presentation of the video/audio content Therefore, when generating the different profiles for a video source, the encoders that generate the different profiles should be synchronized in a frame-accurate way. Moreover, each chunk should be individually decodable. In a H264 data stream, for example, each chunk should start with an instantaneous decoder refresh (IDR) frame.
A video service normally also contains one or more audio elementary streams. Typically, audio content is stored together with the corresponding video content in the same file or as a separate file on the file server. When switching from one profile to another, the audio content may be switched together with the video. In order to provide a seamless listening experience, chunks should start with a new audio frame and corresponding chunks of the different profiles should start with exactly the same audio sample.
Referring now to
At a Time 0, first destination device 110a begins receiving video/audio stream 302a from media server 106 according the bandwidth available to first destination device 110a. At Time A, the bandwidth available to first destination device 110a remains sufficient to provide first video/audio stream 302a to first destination device 110a. At Time B, the bandwidth available to first destination device 110a is greatly reduced, for example due to network congestion. According to an adaptive bitrate streaming procedure, first destination device 110a begins receiving third audio/video stream 302c. At Time C, the bandwidth available to first destination device 110a remains reduced and first destination device 110a continues to receive third video/audio stream 302c. At Time D, greater bandwidth is available to first destination device 110a and first destination device 110a begins receiving second video/audio stream 302b from media server 106. At Time E, the bandwidth available to first destination device 110a is again reduced and first destination device 110a begins receiving third video/audio stream 302c once again. As a result of adaptive bitrate streaming, first destination device 110a continues to seamlessly receive a representation of the original video/audio source despite variations in the network bandwidth available to first destination device 110a.
As discussed, there is a need to synchronize the video over the different video profiles in the sense that corresponding chunks, also called fragments or segments (segments being typically larger than fragments), should start with the same video frame. In some cases, a segment may be comprised of an integer number of fragments although this is not required. For example, when two chunk sizes are being produced simultaneously in which the smaller chunks are called fragments and the larger chunks are called segments, the segments are typically sized to be an integer number of fragments. In various embodiments, the different output profiles can be generated either in a single codec chip, in different chips on the same board, in different chips on different boards in the same chassis, or in different chips on boards, for example. Regardless of where these profiles are generated, the video associated with each profile should be synchronized.
One procedure that could be used for synchronization is to use a master/slave architecture in which one codec is the synchronization master that generates one of the profiles and decides where the fragment/segment boundaries are. The master communicates these boundaries in real-time to each of the slaves and the slaves perform based upon what the master indicates should be done. Although this is conceptually a relatively simple solution, it is difficult to implement properly because it is not easily amendable to the use of backup schemes and configuration is complicated and time consuming.
In accordance with various embodiments described herein, each of first transcoder device 104a, second transcoder device 104b, and third transcoder device 104c use timestamps in the incoming service, i.e. a video and/or audio source, as a reference for synchronization. In a particular embodiment, a PTS within the video and/or audio source is used as a timestamp reference. In a particular embodiment, each transcoder device 104a-104c receives the same (bit-by-bit identical) input service with the same PTS's. In various embodiments, each transcoder uses a pre-defined set of deterministic rules to perform a synchronization process given the incoming PTS's. In various embodiments, rules define theoretical fragmentation/segmentation boundaries, expressed as timestamp values such as PTS values. In at least one embodiment, these boundaries are solely determined by the fragment/segment duration and the frame rate of the video.
First Video Synchronization Procedure
Theoretical Fragment and Segment Boundaries
In one embodiment of a video synchronization procedure theoretical fragment and segment boundaries are determined. In a particular embodiment, theoretical fragment boundaries are determined by following rules:
A first theoretical fragment boundary, PTS_Ftheo[1], starts at:
PTS—Ftheo[1]=0
Theoretical fragment boundary n starts at:
PTS—Ftheo[n]=(n−1)*Fragment Length
With: Fragment Length=fragment length in 90 kHz ticks
The fragment length expressed in 90 kHz ticks is calculated as follows:
FragmentLength=90000/FrameRate*ceiling(FragmentDuration*FrameRate)
An issue that arises with using a PTS value as a time reference for video synchronization is that the PTS value wraps around back to zero after approximately 26.5 hours. In general one PTS cycle will not contain an integer number of equally-sized fragments. In order to address this issue in at least one embodiment, the last fragment in the PTS cycle will be extended to the end of the PTS cycle. This means that the last fragment before the wrap of the PTS counter will be longer than the other fragments and the last fragment ends at the PTS wrap.
The last theoretical normal fragment boundary in the PTS cycle starts at following PTS value:
PTS—Ftheo[Last-1]=[floor(2̂33/FragmentLength)−2]*FragmentLength
PTS—Ftheo[Last]=PTS—Ftheo[Last-1]+FragmentLength
As explained above a segment is a collection of an integer number of fragments. Next to the rules to define the theoretical fragment boundaries, there is also a need to define the theoretical segment boundaries.
PTS—Stheo[1]=0
PTS—Stheo[n]=(n−1)*Fragment Length*N
Just like for fragments, the PTS cycle will not contain an integer amount of equally-sized segments and hence the last segment will contain less fragments than the other segments.
The last normal segment in the PTS cycle starts at following PTS value:
PTS—Stheo[Last-1]=[floor(2̂33/(FragmentLength*N))−2]*(FragmentLength*N)
PTS—Stheo[Last]=PTSLast-1+FragmentLength*N
Actual Fragment and Segment Boundaries
Referring now to
As discussed above the theoretical fragment boundaries depend upon the input frame rate. The above description is applicable for situations in which the output frame rate from the transcoder device is identical to the input frame rate received by the transcoder device. However, for ABR applications the transcoder device may generate video corresponding to different output profiles that may each have a different frame rate from the source video. Typical reduced output frame rates used in ABR are output frame rates that are equal to the input framerate divided by 2, 3 or 4. Exemplary resulting output frame rates in frames per second (fps) are shown in the following table (Table 1) in which frame rates below approximately 10 fps are not used:
When limiting the output frame rates to an integer division of the input framerate an additional constraint is added to ensure that all output profiles stay in synchronization. According to various embodiments, when reducing the input frame rate by a factor x, one input frame out of the x input frames is transcoded and the other x−1 input frames are dropped. The first frame that is transcoded in a fragment should be the frame that corresponds with the actual fragment boundary. All subsequent x−1 frames are dropped. Then the next frame is transcoded again, the following x−1 frames are dropped and so on.
Referring now to
The following table (Table 2) gives an example of the minimum fragment duration for the different output frame rates as discussed above. All fragment durations that are a multiple of this value are valid durations.
Table 2 shows input frame rates of 50.00 fps, 59.94 fps, 25.00 fps, and 29.97 fps along with corresponding least common multiples, and minimum fragment durations. The minimum fragment durations are shown in both 90 kHz ticks and seconds (s).
Frame Alignment at PTS Wrap
Referring now to
Second Video Synchronization Procedure
In order to accommodate the PTS discontinuity issue at the PTS wrap point for frame rate reduced profiles, a modified video synchronization procedure is described. Instead of considering just one PTS cycle for which the first theoretical fragment/segment boundary starts at PTS=0, in accordance with another embodiment of a video synchronization procedure multiple successive PTS cycles are considered. Depending upon the current cycle as determined by the source PTS values, the position of the theoretical fragment/segment boundaries will change.
In at least one embodiment, the first cycle starts arbitrarily with a theoretical fragment/segment boundary at PTS=0. The next fragment boundary starts at PTS=Fragment Length, and so on just as described for the previous procedure. At the wrap of the first PTS cycle, the next fragment boundary timestamp doesn't start at PTS=0 but rather at the last fragment boundary of the first PTS cycle+Fragment Length (modulo 2̂33). In this way, the fragments and segments have the same length at the PTS wrap and no PTS discontinuities occur for the frame rate reduced profiles. Given the video frame rate, the number of frames per fragment and the number of fragments per segment, in a particular embodiment a lookup table 212 (
In one or more embodiments, the total number of theoretical PTS cycles that needs to be considered is not infinite. After a certain number of cycles the first cycle will be arrived at again. The total number of PTS cycles that need to be considered can be calculated as follows:
#PTSCycles=lcm(2̂33, 90000/Frame Rate)/2̂33
The following table (Table 3) provides two examples for the number of PTS cycles that need to be considered for different frame rates.
When all the PTS cycles of the source video have been passed through, the first cycle will be arrived at again. When arriving again at the first cycle, the first theoretical fragment/segment boundary timestamp will be at PTS=0 and in general there will be a PTS discontinuity in the frame rate reduced profiles at this transition to the first cycle. Since this occurs very infrequently, it may be considered a minor issue.
When building a lookup table this manner, in general it is not necessary to include all possible PTS values in lookup table 212. Rather, a limited set of evenly spread PTS values may be included in lookup table 212. In a particular embodiment, the interval between the PTS values (Table Interval) is given by:
Table Interval=Frame Length/#PTS Cycles
Table 4 below provides an example table interval for different frame rates.
One can see that for 29.97 Hz video all possible PTS values are used. For 25 Hz video, the table interval is 16. This means that when the first video frame starts at PTS value 0 it will never get a value between 0 and 16, or between 16 and 32, etc. Accordingly, all PTS values in the range 0 to 15 can be treated identically as if they were 0, all PTS values in the range 16 to 31 may be treated identically as if they were 64, and so on.
Instead of building lookup tables that contain all possible fragment and segment boundaries for all PTS cycles, a reduced lookup table 212 may be built that only contains the first PTS value of each PTS cycle. Given a source PTS value, the first PTS value in the PTS cycle (PTS First Frame) can be calculated as follows:
PTS First Frame=[(PTSa MOD Frame Length)DIV Table Interval]*Table Interval
The PTS First Frame value is then used to find the corresponding PTS cycle in lookup table 212 and the corresponding First Frame Fragment Sequence and First Frame Segment Sequence number of the first frame in the cycle. The First Frame Fragment Sequence is the location of the first video frame of the PTS cycle in the fragment. When the First Frame Fragment Sequence value is equal to 1, the video frame starts a fragment. The First Frame Segment Sequence is the location of the first video frame PTS cycle in the segment. When the First Frame Segment Sequence is equal to 1, the video frame starts a segment.
The transcoder then calculates the offset between PTS First Frame and PTSa in number of frames:
Frame OffsetPTSa=(PTSa−PTSFirstFrame)DIV FrameLength
Fragment SequencePTSa=[(First Frame Fragment Sequence−1+Frame Offset PTSa)MOD(Number Of Frames Per Fragment)]+1
The SegmentSequenceNumber of PTSa is then calculated as:
Segment SequencePTSa=[(First Frame Segment Sequence−1+Frame Offset PTSa)MOD(Number Of Frames Per Fragment*N)]+1
The following table (Table 5) provides several examples of video synchronization lookup tables generated in accordance with the above-described procedures.
Complications with 59.54 Hz Progressive Video
When the input video source is 59.54 Hz video (e.g. 720p59.94) an issue that may arise with this procedure is that the PTS increment for 59.94 Hz video is either 1501 or 1502 (1501.5 on average). Building a lookup table 212 for this non-constant PTS increment brings a further complication. To perform the table lookup for 59.94 Hz video, in one embodiment only the PTS values that differ by either 1501 or 1502 compared to the previous value (in transcoding order—i.e. at the output of the transcoder) are considered. By doing so only every other PTS value will be used for table lookup, which makes it possible to perform a lookup in a half-rate table.
Complications with Sources Containing Field Pictures
Another complication that may occur is with sources that are coded as field pictures. The PTS increment for the pictures in these sources is only half the PTS increment of frame coded pictures. When transcoding these sources to progressive video, the PTS of the output frames will increase by the frame increment. This means that only half of the input PTS values are actually present in the transcoded output. In one particular embodiment, a solution to this issue includes first determining whether the source is coded as Top-Field-First (TFF) or Bottom-Field-First (BFF). For field coded pictures, this can be done by checking the first I-picture at the start of a GOP. If the first picture is a top field then the field order is TFF, otherwise it is BFF. In the case of TFF field order, only the top fields are considered when performing table lookups. In the case of BFF field order, only the bottom fields are considered when performing table lookups. In an alternative embodiment, the reconstructed frames at the output of the transcoder are considered and use the PTS values after the transcoder to perform the table lookup.
Complications with 3/2 Pull-Down 29.97 Hz Sources
For 29.97 Hz interlaced sources that originate from film content and that are intended to be 3/2 pulled down in the transcoder (i.e. converted from 24 fps to 30 fps), the PTS increment of the source frames is not constant because of the fact that some frames last 2 field periods while others last 3 field periods. When transcoding these sources to progressive video, the sequence is first converted to 29.97 Hz video in the transcoder (3/2 pull-down) and afterwards the frame rate of the 29.97 Hz video sequence is reduced. Because of the 3/2 pull-down manner of decoding the source, not all output PTS values are present in the source. For these sources the standard 29.97 Hz table is used. The PTS values that are used for table lookup however are the PTS values at the output of the transcoder, i.e. after the transcoder has converted the source to 29.97 Hz.
Robustness Against Source PTS Errors
Although the second video synchronization procedure described above gives better performance on PTS cycle wraps, it may be less robust against errors in the source video since it assumes a constant PTS increment in the source video. Consider, for example, a 29.97 Hz source where the PTS increment is not constant but varies by +/−1 tick. Depending upon the actual nature of the errors, the result for the first procedure may be that every now and then the fragment/segment duration is one frame more or less, which may not be a significant issue although there will be a PTS discontinuity in the frame rate reduced profiles. However, for the second procedure there may be a jump to a different PTS cycle each time the input PTS differs 1 tick from the expected value, which may result each time in a new fragment/segment. In such situations, it may be more desirable to use the first procedure for video synchronization as described above.
Audio Synchronization Procedure
As previously discussed audio synchronization may be slightly more complex than video synchronization since the synchronization should be done on two levels: the audio encoding framing level and the audio sample level. Fragments should start with a new audio frame and corresponding fragments of the different profiles should start with exactly the same audio sample. When transcoding audio from one compression standard to another the number of samples per frame is in general not the same. The following table (Table 6) gives an overview of frame size for some commonly used audio standards (AAC, MP1Lll, AC3, HE-ACC):
Accordingly, when transcoding from one audio standard to another, the audio frame boundaries often cannot be maintained, i.e. an audio sample that starts an audio frame at the input will in general not start an audio frame at the output. When two different transcoders transcode the audio, the resulting frames will in general not be identical which will make it difficult to generate the different ABR profiles on different transcoders. In order to solve this issue, in at least one embodiment, a number of audio transcoding rules are used to instruct the transcoder how to map input audio samples to output audio frames.
In one or more embodiments, the audio transcoding rules may have the following limitations: limited support for audio sample rate conversion, i.e. the sample rate at the output is equal to the sample rate at the input, although some sample rate conversions can be supported (e.g. 48 kHz to 24 kHz), and no support for audio that is not locked to a System Time Clock (STC). Although it should be understood that in other embodiments, such limitations may not be present.
First Audio Re-Framing Procedure
As explained above the number of audio samples per frame is different for each audio standard. However, according to an embodiment of a procedure for audio re-framing it is always possible to map m frames of standard x into n frames of standard y.
This may be calculated as follows:
m=lcm(#samples/framex,#samples/framey)/#samples/framex
n=lcm(#samples/framex,#samples/framey)/#samples/framey
The following table (Table 7) gives the m and n results when transcoding from AAC, AC3, MP1Lll or HE-AAC (=standard x) to AAC (=standard y):
For example, when transcoding from AC3 to AAC, two AC3 frames will generate exactly 3 AAC frames.
Accordingly, a first audio transcoding rule generates an integer amount of frames at the output from an integer amount of frames of the input. The first sample of the first frame of the input standard will also start the first frame of the output standard. The remaining issue is how to determine if a frame at the input is the first frame or not since only the first sample of the first frame at the input should start a new frame at the output. In at least one embodiment, determining if an input frame is the first frame or not is performed based on the PTS value of the input frame.
Theoretical Audio Re-Framing Boundaries
In accordance with various embodiments, audio re-framing boundaries in the first audio re-framing procedure are determined in a similar manner as for the first video fragmentation/segmentation procedure. First, the theoretical audio re-framing boundaries based on source PTS values are defined:
Some examples of audio frame durations are depicted in the following table (Table 8).
Actual Audio Re-Framing Boundaries
In the previous section, calculation of theoretical re-framing boundaries were described. The theoretical boundaries are used to determine the actual re-framing boundaries which is performed as follows: the first incoming actual PTS value that is greater than or equal to PTS_RFtheo[n] determines an actual re-framing boundary timestamp.
PTS Wrap Point
Referring now to
Second Audio Re-Framing Procedure
An issue with the first audio re-framing procedure discussed above is that there may be an audio glitch at the PTS wrap point (See
#PTS_Cycles=lcm(2̂33,m*AudioFrameLength)/2̂33
An example for AC3 to AAC @ 48 kHz is as follows: #PTS_Cycles=lcm(2̂33, 2*2880)/233=45. This means that 45 PTS cycles fit an integer amount of 2 AC3 input audio frames.
Next, an audio re-framing rule is defined that runs over multiple PTS cycles. The rule includes a lookup in a lookup table that runs over multiple PTS cycles (# cycles=#PTS_Cycles). In one embodiment, the table may be calculated in real-time by the transcoder or in other embodiments, the table may be calculated off-line and used as a look-up table such as lookup table 212.
In order to calculate the lookup table, the procedure starts from the first PTS cycle (cycle 0) and it is arbitrarily assumed that the first audio frame starts at PTS value 0. It is also arbitrarily assumed that the first audio sample of this first frame starts a new audio frame at the output. For each consecutive PTS cycle the current location in the audio frame numbering is calculated. In a particular embodiment, audio frame numbering increments from 1 to m in which the first sample of frame number 1 starts a frame at the output.
An example of a resulting table (Table 9) for AC3 formatted input audio at 48 kHz is as follows:
As can be seen in Table 9, the table repeats after 45 PTS cycles.
In various embodiments, when building a table in this manner, in general it is not necessary to use all possible PTS values but rather a limited set of evenly spread PTS values. In a particular embodiment, the interval between the PTS values is given by: Table Interval=AudioFrameLength/#PTS_Cycles
For AC3 @48 kHz, the Table Interval=2880/45=64. This means that when the first audio frame starts at PTS value 0 it will never get a value between 0 and 64, or between 64 and 128, etc. This means that all PTS values in the range 0-63 can be treated identically as if they were 0, all PTS values in the range 64-127 are treated identically as if they were 64, and so on.
This is depicted in the following simplified table (Table 10).
When a transcoder starts up and begins transcoding audio it receives an audio frame with a certain PTS value designated as PTSa. The first calculation that is performed is to find out where this PTS value (PTSa) fits in the lookup table and what the sequence number of this frame is in order to know whether this frame starts an output frame or not.
In order to do so, the corresponding first frame is calculated as follows:
PTSFirst Frame=[(PTSa MOD Audio Frame Length)DIV Table Interval]*Table Interval
The PTS First Frame value is then used to find the corresponding PTS cycle in the table and the corresponding First Frame Sequence Number.
The transcoder then calculates the offset between PTS First Frame and PTSa in number of frames as follows:
Frame OffsetPTSa=(PTSa−PTSFirst Frame)DIV Audio Frame Length
The sequence number of PTSa is then calculated as:
SequencePTSa=[(First Frame Sequence Number−1+FrameOffsetPTsa)MOD m]+1
If SequencePTSa is equal to 1 then the first audio sample of this input frame starts a new output frame. For example, assume a transcoder transcodes from AC3 to AAC at a 48 kHz sample rate. The first received audio frame has a PTS value equal to 4000. The PTS First Frame is determined as follows: PTS First Frame=(4000 MOD 2880) DIV (2880/45)*(2880/45)=1088
First Frame Sequence Number=2
Frame Offset PTSa=(4000−1088)DIV 2880=1
Sequence PTSa=[(2−1+1)MOD 2]+1=1
Transcoded Audio Fragment Synchronization
In the previous sections a procedure was described to deterministically build new audio frames after transcoding of an audio source. The re-framing procedure makes sure that different transcoders generate audio frames that start with the same audio sample. For some ABR standards, there is a requirement that transcoded audio streams are fragmented (i.e. fragment boundaries are signaled in the audio stream) and different transcoders should insert the fragment boundaries at exactly the same audio frame boundary.
A procedure to synchronize audio fragmentation in at least one embodiment is to align the audio fragment boundaries with the re-framing boundaries. As discussed herein above, in at least one embodiment for every m input frames the re-framing is started based on the theoretical boundaries in a look-up table. The look-up table may be expanded to also include the fragment synchronization boundaries. Assuming the minimum distance between two fragments is m, the fragment boundaries can be made longer by only inserting a fragment every x re-framing boundaries, which means only 1 out of x re-framing boundaries is used as a fragment boundary, resulting in fragment lengths of m*x audio frames. Determining whether a re-framing boundary is also a fragmentation boundary is performed by extending the re-framing look-up table with the fragmentation boundaries. It should be noted that in general if x is different from 1, the fragmentation boundaries will not perfectly fit into the multi-PTS re-framing cycles and will result in a shorter than normal fragment at the multi-PTS cycle wrap.
Referring now to
In 904, first transcoder device 104a determines theoretical fragment boundary timestamps based upon one or more characteristics of the source video using one or more of the procedures as previously described herein. In a particular embodiment, the one or more characteristics include one or more of a fragment duration and a frame rate associated with the source video. In still other embodiments, the theoretical fragment boundary timestamps may be further based upon frame periods associated with a number of output profiles associated with one or more of first transcoder device 104a, second transcoder device 104b, and third transcoder device 104c. In a particular embodiment, the theoretical fragment boundary timestamps are a function of a least common multiple of a plurality of frame periods associated with respective output profiles. In some embodiments, the theoretical fragment boundary timestamps may be obtained from a lookup table 212. In 906, first transcoder device 104a determines theoretical segment boundary timestamps based upon one or more characteristics of the source video using one or more of the procedures as previously discussed herein. In a particular embodiment, the one or more characteristics include one or more of a segment duration and frame rate of associated with the source video.
In 908, first transcoder device 104a determines the actual fragment boundary timestamps based upon the theoretical fragment boundary timestamps and received timestamps from the source video using one or more of the procedures as previously described herein. In a particular embodiment, the first incoming actual timestamp value that is greater than or equal to the particular theoretical fragment boundary timestamp determines the actual fragment boundary timestamp. In 910, first transcoder device 104a determines the actual segment boundary timestamps based upon the theoretical segment boundary timestamps and the received timestamps from the source video using one or more of the procedures as previously described herein.
In 912, first transcoder device 104a transcodes the source video according to the output profile and the actual fragment boundary timestamps using one or more procedures as discussed herein. In 914, first transcoder device 104a outputs the transcoded source video including the actual fragment boundary timestamps and actual segment boundary timestamps. In at least one embodiment, the transcoded source video is sent by first transcoder device 104a to encapsulator device 105. Encapsulator device 105 encapsulated the transcoded source video and sends the encapsulated transcoded source video to media server 106. Media server 106 stores the encapsulated transcoded source video in storage device 108. In one or more embodiments, first transcoder device 104a signals the chunk (fragment/segment) boundaries in a bitstream sent to encapsulator device 105 and encapsulator device 105 for use by the encapsulator device 105 during the encapsulation.
It should be understood that the video synchronization operations may also be performed on the source video by one or more of second transcoder device 104b and third transcoder device 104b in accordance with one or more output profiles such that the transcoded output video associated with each output profile may have different video formats, resolutions, bitrates, and/or framerates associated therewith. At a later time, a selected one of the transcoded output video may be streamed to one or more of first destination device 110a and second destination device 110b according to available bandwidth. The operations end at 916.
In 1004, first transcoder device 104a determines theoretical fragment boundary timestamps using one or more of the procedures as previously described herein. In 1006, first transcoder device 104a determines theoretical segment boundary timestamps using one or more of the procedures as previously discussed herein. In 1008, first transcoder device 104a determines the actual fragment boundary timestamps using one or more of the procedures as previously described herein. In a particular embodiment, the first incoming actual timestamp value that is greater than or equal to the particular theoretical fragment boundary timestamp determines the actual fragment boundary timestamp. In 1010, first transcoder device 104a determines the actual segment boundary timestamps based upon the theoretical segment boundary timestamps and the received timestamps from the source video using one or more of the procedures as previously described herein.
In 1012, first transcoder device 104a determines theoretical audio re-framing boundary timestamps based upon one or more characteristics of the source audio using one or more of the procedures as previously described herein. In a particular embodiment, the one or more characteristics include one or more of an audio frame length and a number of grouped source audio frames needed for re-framing associated with the source audio. In some embodiments, the theoretical audio re-framing boundary timestamps may be obtained from lookup table 212.
In 1014, first transcoder device 104a determines the actual audio re-framing boundary timestamps based upon the theoretical audio re-framing boundary timestamps and received audio timestamps from the source audio using one or more of the procedures as previously described herein. In a particular embodiment, the first incoming actual timestamp value that is greater than or equal to the particular theoretical audio re-framing boundary timestamp determines the actual audio re-framing boundary timestamp.
In 1016, first transcoder device 104a transcodes the source audio according to the output profile, the actual audio-reframing boundary timestamps, and the actual fragment boundary timestamps using one or more procedures as discussed herein. In 1018, first transcoder device 104a outputs the transcoded source audio including the actual audio re-framing boundary timestamps, actual fragment boundary timestamps, and the actual segment boundary timestamps. In at least one embodiment, the transcoded source audio is sent by first transcoder device 104a to encapsulator device 105. Encapsulator device 105 sends the encapsulated transcoded source audio to media server 106, and media server 106 stores the encapsulated transcoded source audio in storage device 108. In one or more embodiments, the transcoded source audio may be stored in association with related transcoded source video. It should be understood that the audio synchronization operations may also be performed on the source audio by one or more of second transcoder device 104b and third transcoder device 104b in accordance with one or more output profiles such that the transcoded output audio associated with each output profile may have different audio formats, bitrates, and/or framerates associated therewith. At a later time, a selected one of the transcoded output audio may be streamed to one or more of first destination device 110a and second destination device 110b according to available bandwidth. The operations end at 1012.
Note that in certain example implementations, the video/audio synchronization functions outlined herein may be implemented by logic encoded in one or more non-transitory, tangible media (e.g., embedded logic provided in an application specific integrated circuit [ASIC], digital signal processor [DSP] instructions, software [potentially inclusive of object code and source code] to be executed by a processor, or other similar machine, etc.). In some of these instances, a memory element [as shown in
In one example implementation, transcoder devices 104a-104c may include software in order to achieve the video/audio synchronization functions outlined herein. These activities can be facilitated by transcoder module(s) 208, video/audio timestamp alignment module 210, and/or lookup tables 212 where these modules can be suitably combined in any appropriate manner, which may be based on particular configuration and/or provisioning needs). Transcoder devices 104a-104c can include memory elements for storing information to be used in achieving the intelligent forwarding determination activities, as discussed herein. Additionally, transcoder devices 104a-104c may include a processor that can execute software or an algorithm to perform the video/audio synchronization operations, as disclosed in this Specification. These devices may further keep information in any suitable memory element [random access memory (RAM), ROM, EPROM, EEPROM, ASIC, etc.], software, hardware, or in any other suitable component, device, element, or object where appropriate and based on particular needs. Any of the memory items discussed herein (e.g., database, tables, trees, cache, etc.) should be construed as being encompassed within the broad term ‘memory element.’ Similarly, any of the potential processing elements, modules, and machines described in this Specification should be construed as being encompassed within the broad term ‘processor.’ Each of the network elements can also include suitable interfaces for receiving, transmitting, and/or otherwise communicating data or information in a network environment.
Note that with the example provided above, as well as numerous other examples provided herein, interaction may be described in terms of two, three, or more network elements. However, this has been done for purposes of clarity and example only. In certain cases, it may be easier to describe one or more of the functionalities of a given set of flows by only referencing a limited number of network elements. It should be appreciated that communication system 100 (and its teachings) are readily scalable and can accommodate a large number of components, as well as more complicated/sophisticated arrangements and configurations. Accordingly, the examples provided should not limit the scope or inhibit the broad teachings of communication system 100 as potentially applied to a myriad of other architectures.
It is also important to note that the steps in the preceding flow diagrams illustrate only some of the possible signaling scenarios and patterns that may be executed by, or within, communication system 100. Some of these steps may be deleted or removed where appropriate, or these steps may be modified or changed considerably without departing from the scope of the present disclosure. In addition, a number of these operations have been described as being executed concurrently with, or in parallel to, one or more additional operations. However, the timing of these operations may be altered considerably. The preceding operational flows have been offered for purposes of example and discussion. Substantial flexibility is provided by communication system 100 in that any suitable arrangements, chronologies, configurations, and timing mechanisms may be provided without departing from the teachings of the present disclosure.
Although the present disclosure has been described in detail with reference to particular arrangements and configurations, these example configurations and arrangements may be changed significantly without departing from the scope of the present disclosure. Additionally, although communication system 100 has been illustrated with reference to particular elements and operations that facilitate the communication process, these elements and operations may be replaced by any suitable architecture or process that achieves the intended functionality of communication system 100.
