Patent Application
20090154347
1. Technical Field of the Invention
This invention relates generally to multimedia content transport, and more particularly to the receipt and processing of such multimedia content.
2. Related Art
The broadcast of digitized audio/video information (multimedia content) is well known. Limited access communication networks such as cable television systems, satellite television systems, and direct broadcast television systems support delivery of digitized multimedia content via controlled transport medium. In the case of a cable modem system, a dedicated network that includes cable modem plant is carefully controlled by the cable system provider to ensure that the multimedia content is robustly delivered to subscribers' receivers. Likewise, with satellite television systems, dedicated wireless spectrum robustly carries the multi-media content to subscribers' receivers. Further, in direct broadcast television systems such as High Definition (HD) broadcast systems, dedicated wireless spectrum robustly delivers the multi-media content from a transmitting tower to receiving devices. Robust delivery, resulting in timely receipt of the multimedia content by a receiving device is critical for the quality of delivered video and audio.
Some of these limited access communication networks now support on-demand programming in which multimedia content is directed to one, or a relatively few number of receiving devices. The number of on-demand programs that can be serviced by each of these types of systems depends upon, among other things, the availability of data throughput between a multimedia source device and the one or more receiving devices. Generally, this on-demand programming is initiated by one or more subscribers and serviced only upon initiation.
Publicly accessible communication networks, e.g., Local Area Networks (LANs), Wireless Local Area Networks (WLANs), Wide Area Networks (WANs), Wireless Wide Area Networks (WWANs), and cellular telephone networks, have evolved to the point where they now are capable of providing data rates sufficient to service streamed multimedia content. The format of the streamed multimedia content is similar/same as that that is serviced by the limited access networks, e.g., cable networks, satellite networks. However, each of these communication networks is shared by many users that compete for available data throughput. Resultantly, data packets carrying the streamed multimedia content are typically not given preferential treatment by these networks.
Generally, streamed multimedia content is formed/created by a first electronic device, e.g., web server, transmitted across one or more commutation networks, and received and processed by a second electronic device, e.g., personal computer, laptop computer, cellular telephone, WLAN device, or WWAN device. In creating the multimedia content, the first electronic device obtains/retrieves multimedia content from a video camera or from a storage device, for example, and encodes the multimedia content to create encoded audio and video frames according to a standard format, e.g., MPEG-2 format. The audio and video frames are placed into data packets that are sequentially transmitted from the first electronic device onto a servicing communication network, the data packets addressed to one or more second electronic device(s). One or more communication networks carry the data packets to the second electronic device. The second electronic device receives the data packets, reorders the data packets, if required, and extracts the audio and video frames from the data packets. A decoder of the second electronic device receives the data packets and reconstructs a program clock based upon Program Clock References (PCRs) contained in the transport packets. The decoder then uses the reconstructed clock to decode the audio and video frames to produce audio and video data. The second electronic device then stores the video/audio data and/or presents the video/audio data to a user via a user interface.
To be compliant to the MPEG-2 Systems specification, PCRs are required to be inserted at least every 100 milliseconds in a MPEG-2 transport stream. In order for the second electronic device to accurately reconstruct the program clock, e.g., at 27 MHz, incoming transport packets must arrive at the decoder of the second electronic device with jitter that is less than approximately 1-2 milliseconds/30 parts-per-million (PPM). Data packets that are transported by communication networks such as the Internet, WANs, LANs, WWANs, WLANs, and/or cellular networks, for example, using Internet Protocol (IP) addressing, for example, may travel via differing routes across one or more communication networks and arrive with various transmission latencies. In many operations, the data packets carrying timestamps arrive with significant jitter, sometimes approaching 200-400 parts-per-million jitter. With this large jitter, the receiving decoder may be unable to recreate the program clock from the received data packets. Further, even if the decoder is able to recreate the program clock, the recreated program clock may have a significantly different frequency than the program clock of the encoding first device. Such differences in the recreated program clock of the decoder of the second electronic device as compared to the program clock of the encoder of the first electronic device results in buffer overflow or underflow at the second electronic device. Buffer overflow causes some of the incoming data to be purged and not buffered resulting in lost data and poor audio or video quality. Buffer underflow causes starvation of the decoder also resulting in poor audio or video quality.
Thus, a need exists for a streamed transport system that operates satisfactorily with a high jitter transport stream, e.g. the Internet, and that produces video and audio output of high quality. Further limitations and disadvantages of conventional and traditional approaches will become apparent to one of skill in the art, through comparison of such systems with some aspects of the present invention as set forth in the remainder of the present application with reference to the drawings.
The present invention is directed to apparatus and methods of operation that are further described in the following Brief Description of the Drawings, the Detailed Description of the Invention, and the claims. Other features and advantages of the present invention will become apparent from the following detailed description of the invention made with reference to the accompanying drawings.
Generally, the Internet/WWW 102 (and in many cases the WANs/LANs) cannot guarantee delivery of data packets (and audio/video frames carried therein) with jitter of 1-2 milliseconds. It is possible for the jitter of data packets carried by the Internet/WWW to exceed hundreds of milliseconds. Thus, any transport stream (transport packet stream) traversing the Internet/WWW 102 could have jitter of 100s of milliseconds. WANs/LANs 104 and 106 support data transport of a transport stream with less jitter than that provided by the Internet/WWW 102 in some operations. However, in other operations, the WAN/LANs 104 and 106 cannot guarantee delivery of the transport packets of the transport stream with a lower jitter figure.
The WLAN/WWAN/Cellular networks 108 and 110 operate according to one or more wireless interface standards, e.g., IEEE 802.11x, WiMAX, GSM, EDGE, GPRS, WCDMA, CDMA, 1xEV-DO, 1xEV-DV, etc. The WLAN/WWAN/Cellular networks 108 and 110 include a back-haul network that couples to the Internet/WWW 102 and service wireless links for wirelessly enabled electronic devices 122, 124, 126, 128, 130, 132, and 134. In providing this wireless service, the WLAN/WWAN/Cellular networks 108 and 110 include infrastructure devices, e.g., Access Points and base stations to wirelessly service the electronic devices 122, 124, 126, 128, 130, 132, and 134. The wireless links serviced by the WLAN/WWAN/Cellular networks 108 and 110 are shared amongst the wirelessly enabled electronic devices 124-134 and are generally data throughput limited. Such data throughput limitations result because the wireless links are shared, the wireless links are degraded by operating conditions, and/or simply because the wireless links have basic data throughput limitations. Thus, the WLAN/WWAN/Cellular networks 108 and 110 typically introduce additional jitter into a serviced transport stream that is in addition to the jitter introduced by the Internet/WWW 102.
Generally, the servicing networks 104, 106, 102, 108, and 110 illustrated in
With an example of operation according to the present invention, a web server 112 establishes an audio/video session with one or more of electronic devices 114, 120, 122, and/or 130. The web server 112 initially sets up the audio/video session with these devices 114, 120, 122, and/or 130 and then creates a transport stream that carries encoded audio/video frames/data. Some of the transport packets include Program Clock References (PCRs). The web server 112 may package the transport packets into larger/other data packets, address the data packets to devices 114, 120, 122, and/or 130 and transmit the data packets onto WAN/LAN 104. Networks 104, 106, 102, 108, and/or 110 deliver the data packets to the receiving electronic devices 114, 120, 122, and/or 130. However, the servicing networks 102, 104, 106, 108, and/or 110 introduce jitter into the transported data packets. Differences in transport latency from data packet to data packet are exhibited as jitter of the transport stream. The jitter of the transport of the data packets from the web server 112 to each of the electronic device 114, 120, 122, and/or 130 may differ. Further, the jitter of each transport stream (between web server 112 and a corresponding receiving device) may vary over time, depending upon network loading, wireless link limitations, equipment outages, the weather, etc. The reader should note that transport packets and data packets, although referred to separately herein, may be the same data structures in some embodiments and need not be separate or differing structures. In some embodiments, the transport packets that are made up of audio/video frames are carried by IP packets, which are one example of data packets that differ from transport packets. In still other embodiments, audio frames and/or video frames may be packaged into differing data packets. Further, “transport packet stream” and “transport stream” may be referred to interchangeably herein for clarity purposes and, in some embodiments, are the same thing.
Each electronic device 114, 120, 122, and/or 130 eventually receives most or all of the transport packets of the transport stream. However, the transport packets are effectively received on an irregular basis as compared to the PCRs that they carry. According to one example of the operations of system 100 of
Thus, according to one or more embodiments of the present invention, electronic devices 114, 120, 122, and 130 includes transport stream jitter removal circuitry that removes the jitter from the received transport packets of the transport packet stream prior to reconstructing the program clock from the PCRs. The structure and operation of the jitter removal circuitry will be described further with reference to
According to other operations of the system 100 of
The audio/video source 202 transmits the data packets (carrying the transport packets of the transport packets stream) upon a communication path 204 that includes at least one jittery network. The communication path 204 carries the data packets carrying the transport packets to audio/video player 206. However, the transport packets of the transport packet stream are received by audio/video player 206 with significant jitter, e.g. hundreds of milliseconds of jitter. The audio/video 206 includes transport stream jitter removal circuitry 210 and a decoder 212. The decoder 212 cannot operate upon the jittery transport stream to recreate a program clock accurately. Because the received transport stream is too jittery, the decoder 212 would create an inaccurate program clock that would cause buffer underflow or overflow resulting in poor quality of output audio/video. Thus, according to the present invention, the transport stream jitter removal circuitry 210 of the audio/video 206 operates upon the transport stream (after extraction from the data packet stream) to remove jitter prior to providing the transport stream to decoder 212. In such case, the transport stream jitter removal circuitry 210 substantially/fully removes the jitter introduced by the communication path 204 prior to operation upon the transport stream by the decoder 212.
Thus, according to the embodiment of
The transport stream jitter removal circuitry 402 includes a data buffer 404, pacing control circuitry 406, pacing counter clock circuitry 408, and PCR packet pacing circuitry 410. The data buffer 404 is operable to receive the transport packets of the transport stream jittery input transport stream) and to store the transport packets. The pacing counter clock circuitry 408 is operable to produce a pacing counter clock and to adjust a frequency of the pacing counter clock based upon a pacing counter clock adjust signal. The pacing control circuitry 406 is operable to produce the pacing counter clock adjust signal based upon receipt of the transport packets. The PCR packet pacing circuitry 410 is operable to receive the pacing counter clock from pacing counter clock circuitry 408 and, based upon the pacing counter clock, to receive transport packets from the data buffer 404. The PCR pacing circuitry 410 is further operable to transmit the retrieved transport packets as an output transport stream. The output transport stream is also referred to in
According to one operation of the embodiment of
In one embodiment of operation according to the present invention, pacing control circuitry 406 is operable to process the PCRs of the transport packets to estimate a frequency of a program clock of an electronic device producing the transport stream. For example, referring again to
Then, referring again to
According to another operation of the pacing control circuitry 406, in producing the pacing counter clock adjust signal based upon the transport packets, the pacing control circuitry 406 characterizes the fullness of the data buffer 404. The pacing control circuitry 406 then compares the fullness of the data buffer 404 to a fullness threshold. Then, based upon this comparison the pacing control circuitry 406 produces the pacing counter clock adjust signal. For example, in one particular embodiment, it may be desirable for the data buffer 404 to be approximately 50% full at all times. When the data buffer 404 becomes more than 50% full, the pacing control circuitry 406 directs an increase in frequency of the pacing counter clock using the pacing counter clock adjust signal. Alternatively, when the data buffer 404 is less than 50% full, the pacing control circuitry 406 determines that the frequency of the pacing counter clock should be decreased and produces an appropriate pacing counter clock adjust signal.
In still another embodiment, the fullness thresholds may be set at more than one level. For example, if the data buffer 404 is more than 70% full, the pacing control circuitry 406 may direct the pacing counter clock circuitry 408 to increase the frequency of the pacing counter clock via the pacing counter clock adjust signal. Further, when the data buffer is less than 30% full, the pacing control circuitry 406 may direct the pacing counter clock circuitry 408 to decrease the frequency of the pacing counter clock via the pacing counter clock adjust signal. In this fashion, the pacing control circuitry 406 will effectively control the pacing counter clock frequency so that the data buffer 404 remains between 30% and 70% full. With these data fullness thresholds met, the likelihood that the decoder that is decoding the transport stream will not underflow or overflow increases.
According to another aspect of the transport stream jitter removal circuitry 402 of
The transport demultiplexer 502 demultiplexes the transport stream into its audio and video components. The audio components of the transport stream are operated upon by audio decoder 510 while the video components of the transport stream are operated upon by video decoder 506. The audio output block 512 receives the reconstructed program clock in the output of audio decoder 510 and produces audio output based thereupon. The video output block 508 receives the output of video decoder 506 and the recreated program clock from PCR recovery loop 504. The video output block 508 outputs video for subsequent display and processing.
Generally, the electronic device 602 includes processing circuitry 604, memory 606, first network interface 608, second network interface 610, user input interfaces 612, and user output interfaces 614. The user input interfaces 612 couple to headset 622, mouse 620, and keyboard 618. The user output interfaces 614 couple to audio/video display device 616. The user output interface 614 may also couple to headphone 622. The display device 616 may include a monitor, projector, speakers, and other components that are used to present the audio and video output to a user.
The electronic device 602 embodies the structure and performs operations of the present invention with respect to jitter removal from a received transport stream. In one particular construct of the electronic device 602, dedicated hardware is employed for jitter removal purposes. In such case, the electronic device 602 includes jitter removal circuitry 632. The electronic device may also include decoding circuitry 634. Further, the electronic device 602 may include encoding circuitry 636 that is employed to encode video information into a transport stream that carries transport packets.
Alternatively, the electronic device 602 may include non-dedicated jitter removal circuitry and/or encoding and decoding operations. In such case, the jitter removal operations of electronic device 602 are serviced by processing circuitry 604. The processing circuitry 604 performs, in addition to its PC operations, jitter removal operations 638 and may perform encoding/decoding operations 640. In such case, particular hardware may be included in the processing circuitry 604 to perform the operations 638 and 640. Alternatively, the jitter removal operations 638 and encoding/decoding operations 640 may be accomplished by the execution of software instructions. In this case, the processing circuitry 604 retrieves video processing instruction 624, jitter removal instructions 626, decoding instructions 628, and/or encoding instructions 630 from memory 608. The processing circuitry 604 executes these various instructions 624, 626, 628, and/or 630 to perform the indicated functions. Processing circuitry 604 may include one or more processing devices such as microprocessors, digital signal processors, application specific processors, or other processing type devices.
The audio/video processing device 602 includes the first network interface 608 and the second network interface 610. Generally, the electronic device 602 receives a transport stream (within data packets) via one of the first and second network interfaces 608 and 610. In its other operations, the electronic device 602 may output a transport stream (within data packets) from one of network interfaces 608 or 610. In another operation, the electronic device 602 simply removes jitter from a transport stream carrying video data and outputs the de-jittered transport stream via one of its interfaces 608 and 610.
Generally, the device 702 receives a jittery transport stream via one of the network interfaces 708, 710 and removes the jitter from the jittery transport stream. The structure and operations of the jitter removal circuitry was previously described with reference to
According to one operation, the device 702 receives a jittery transport stream via first network interface 708. Then, after de-jittering the transport stream, the electronic device 702 outputs the de-jittered transport stream onto another network via second network interface 710. Alternately, the device 702 may receive the jittery transport stream and transmit the de-jittered transport stream via a single one of the network interfaces 708 and 710.
The pacing control circuitry receives the PCR either by extraction or by receipt from the data buffer (Step 810). Based upon the receipt of the PCR, or based upon another event, e.g., counter event, the pacing control circuitry initiates operations to produce the pacing counter clock adjust signal, if necessary, based upon the PCR. In a first embodiment of this operation, the pacing control circuitry examines the buffer fullness of the data buffer (Step 812). Such examination may be performed absent receipt of the PCR during its normal operations. Based upon the examination of the buffer fullness at Step 812, the pacing control circuitry adjusts the pacing counter frequency if necessary (Step 814). An alternate embodiment described herein, the pacing control circuitry attempts to reconstruct a program clock of the device that created the transport stream and adjusts the pacing counter frequency if necessary at Step 814 based upon this estimation.
The jittery removal circuitry produces the de-jittered transport stream as its output. Upon investigating the pacing counter clock, the PCR pacing circuitry determines when a pacing counter event has been met (Step 816). Upon meeting the pacing counter event, the PCR packet pacing circuitry retrieves a transport packet from the data buffer (Step 818). The pacing control circuitry then outputs the retrieved transport packet (Step 822). Steps 818 and or 822 may include retrieving audio data and transmitting the audio data as well. From Steps 808, 814, and 822 operation returns to Step 800.
The terms “circuit” and “circuitry” as used herein may refer to an independent circuit or to a portion of a multifunctional circuit that performs multiple underlying functions. For example, depending on the embodiment, processing circuitry may be implemented as a single chip processor or as a plurality of processing chips. Likewise, a first circuit and a second circuit may be combined in one embodiment into a single circuit or, in another embodiment, operate independently perhaps in separate chips. The term “chip”, as used herein, refers to an integrated circuit. Circuits and circuitry may comprise general or specific purpose hardware, or may comprise such hardware and associated software such as firmware or object code.
The present invention has also been described above with the aid of method steps illustrating the performance of specified functions and relationships thereof. The boundaries and sequence of these functional building blocks and method steps have been arbitrarily defined herein for convenience of description. Alternate boundaries and sequences can be defined so long as the specified functions and relationships are appropriately performed. Any such alternate boundaries or sequences are thus within the scope and spirit of the claimed invention.
The present invention has been described above with the aid of functional building blocks illustrating the performance of certain significant functions. The boundaries of these functional building blocks have been arbitrarily defined for convenience of description. Alternate boundaries could be defined as long as the certain significant functions are appropriately performed. Similarly, flow diagram blocks may also have been arbitrarily defined herein to illustrate certain significant functionality. To the extent used, the flow diagram block boundaries and sequence could have been defined otherwise and still perform the certain significant functionality. Such alternate definitions of both functional building blocks and flow diagram blocks and sequences are thus within the scope and spirit of the claimed invention. One of average skill in the art will also recognize that the functional building blocks, and other illustrative blocks, modules and components herein, can be implemented as illustrated or by discrete components, application specific integrated circuits, processors executing appropriate software and the like or any combination thereof.
As may be used herein, the terms “substantially” and “approximately” provides an industry-accepted tolerance for its corresponding term and/or relativity between items. Such an industry-accepted tolerance ranges from less than one percent to fifty percent and corresponds to, but is not limited to, component values, integrated circuit process variations, temperature variations, rise and fall times, and/or thermal noise. Such relativity between items ranges from a difference of a few percent to magnitude differences. As may also be used herein, the term(s) “coupled to” and/or “coupling” and/or includes direct coupling between items and/or indirect coupling between items via an intervening item (e.g., an item includes, but is not limited to, a component, an element, a circuit, and/or a module) where, for indirect coupling, the intervening item does not modify the information of a signal but may adjust its current level, voltage level, and/or power level. As may further be used herein, inferred coupling (i.e., where one element is coupled to another element by inference) includes direct and indirect coupling between two items in the same manner as “coupled to”. As may even further be used herein, the term “operable to” indicates that an item includes one or more of power connections, input(s), output(s), etc., to perform one or more its corresponding functions and may further include inferred coupling to one or more other items. As may still further be used herein, the term “associated with”, includes direct and/or indirect coupling of separate items and/or one item being embedded within another item. As may be used herein, the term “compares favorably”, indicates that a comparison between two or more items, signals, etc., provides a desired relationship. For example, when the desired relationship is that signal 1 has a greater magnitude than signal 2, a favorable comparison may be achieved when the magnitude of signal 1 is greater than that of signal 2 or when the magnitude of signal 2 is less than that of signal 1.
The present invention has also been described above with the aid of method steps illustrating the performance of specified functions and relationships thereof. The boundaries and sequence of these functional building blocks and method steps have been arbitrarily defined herein for convenience of description. Alternate boundaries and sequences can be defined so long as the specified functions and relationships are appropriately performed. Any such alternate boundaries or sequences are thus within the scope and spirit of the claimed invention.
Moreover, although described in detail for purposes of clarity and understanding by way of the aforementioned embodiments, the present invention is not limited to such embodiments. It will be obvious to one of average skill in the art that various changes and modifications may be practiced within the spirit and scope of the invention, as limited only by the scope of the appended claims.
