The present invention relates to wireless communication and in particular to transmission of uncompressed video over wireless communication channels.
With the proliferation of high quality video, an increasing number of electronic devices (e.g., consumer electronic devices) utilize high-definition (HD) video. Conventionally, most devices compress the HD video, which can be around 1 Gbps (giga bits per second) in bandwidth, to a fraction of its size to allow for transmission between devices. However, with each compression and subsequent decompression of the video, some video information can be lost and the picture quality is degraded.
The High-Definition Multimedia Interface (HDMI) specification defines an interface for uncompressed HD transmission between devices through HDMI cables (wired links). Three separate channels are used to transmit three component streams (R, G, B or Y, Cb, Cr). For each channel, pixels are transmitted in pixel-by-pixel order for each video line, and line-by-line for each video frame or field. The HDMI provides pixel-repetition functionality which repeats each pixel one or multiple times. The copies of each pixel directly follow the original pixel during the transmission at each component channel.
Existing Wireless Local Area Networks (WLANs) and similar technologies do not have the bandwidth needed to carry uncompressed HD video, such as providing an air interface to transmit uncompressed video over a 60 GHz bandwidth. Further, existing WLANs can suffer from interference issues when several devices are connected, leading to video signal degradation.
The present invention provides a method and system for spatial video pixel partitioning and encoding for transmission of uncompressed video over wireless communication channels. In one implementation, this allows transmission of uncompressed HD video from a transmitter to a receiver over a wireless channel, and further allows error recovery at the receiver.
An example involves inputting a frame of pixel information, and partitioning spatially correlated pixels. Then, the partitioned pixels are placed into different packets and error detection information is generated for each packet and appended thereto. The packets are then transmitted by a transmitter to a receiver over a wireless channel. Based on the appended error detection data in each packet, the receiver determines if a received packet is corrupt. If a packet is corrupt, then the receiver recovers the corrupt pixels using pixel information in other received packets containing neighboring pixels. As a result, retransmission of corrupt pixels is not required. This improves transmission robustness and reduces channel bandwidth requirements.
Preferably, the partitioned pixels are placed into packets such that pixels with minimal spatial distance (i.e., neighboring pixels) are placed into different packets for transmission over the wireless channel. This can further include partitioning spatially correlated pixels into K different partitions, and selecting the value of the nth pixel at each K pixel block as base information. Then, the base information is placed in a packet as BASE pixels, and information of other pixels in the pixel block is encoded within the same block and placed in another packet as DIFF pixels.
In accordance with further features of a preferred embodiment of the present invention, recovering the corrupt pixels further includes determining a difference between each corrupt pixel in a corrupt packet with a corresponding pixel of an adjacent non-corrupt packet, and if the difference is greater than a threshold, then correcting each corrupt pixel using pixel information in that adjacent non-corrupt packet. Such correcting steps can be performed by replacing each corrupt pixel with a corresponding pixel in the adjacent non-corrupt packet, or by using an average of neighboring pixels of a corresponding pixel in the non-corrupt packet.
These and other features, aspects and advantages of the present invention will become understood with reference to the following description, appended claims and accompanying figures.
In the drawings, like references refer to similar elements.
The present invention provides a method and system for spatial video pixel partitioning and encoding for transmission of uncompressed video, such as over wireless communication channels. In one implementation, this allows transmission of uncompressed HD video from a transmitter to a receiver over a wireless channel.
There are two categories of HD video formats according to a display scan scheme: interlaced and progressive. In the progressive scheme the pixels are scanned line-by-line. However, in the interlaced scheme the pixels are scanned every other line and one video frame is divided into two sub-frames called odd line field and even line field. In each video frame, usually the neighboring pixels have very similar or even the same values. This type of spatial redundancy can be used in wireless transmission to improve video quality.
According to said embodiment of the present invention, neighboring pixels in a video frame are partitioned into different packets and transmitted separately over wireless channels from a transmitter to a receiver. If one packet is received corrupted (i.e., pixels lost or received with errors), then packets carrying the neighboring pixels are used to recover the pixels in the corrupt packet. As such, retransmission of lost information is not required, which saves transmission bandwidth.
Wireless transmission of uncompressed HD video (WiHD) requires higher Medium Access Control (MAC) packet transmission efficiency. Due to such high MAC efficiency requirements (and a relatively static channel), a WiHD packet can be very long (i.e., typically 300K-600K bits long). The present invention further provides the option of encoding pixels based on said spatial pixel partitioning method, which conserves transmission bandwidth.
The pixels 102 are split horizontally into two groups: (1) the first group of pixels (marked as “X”) have indices i=0, 2, 4, . . . , etc., per line and indices j=0, 1, 2, . . . , etc.; and (2) the second group of pixels (marked as “0”) have indices i=1, 3, 5, . . . , etc., per line and indices j=0, 1, 2, . . . , etc. Then, as shown in
Packet size is selected depending on transmitter and receiver buffer requirements. One or more lines worth of pixels can be placed in each packet. A CRC for each packet is computed and appended at the end of the packet before transmission to a receiver over a wireless channel.
In the uncompressed video frame 100, geographically neighboring (spatially correlated) pixels usually have very similar, or even the same values. Regardless of how pixel partitioning is performed, so long as spatially neighboring pixels are partitioned and placed into different packets for transmission, then if pixel information in a received packet is corrupted (i.e., lost or damaged), one or more other packets which contain pixels that are spatially related to the corrupt pixel(s) can be used to recover (compensate for) the corrupt pixel information.
There are many approaches for recovering a lost or erroneous pixel P. One approach involves simply copying a pixel Q from a neighboring packet, wherein preferably pixel Q is spatially correlated to pixel P. Another approach involves using a combination (e.g., the average value) of pixels R in neighboring packets, wherein preferably pixels R are spatially correlated to pixel P. Other approaches for recovering a lost or erroneous pixel based on the neighboring pixels can be utilized, as those skilled in the art will recognize.
Preferably, partitioning is performed such that pixels with minimal spatial distance are placed into different packets for transmission over a wireless channel. Further, partitioning can be performed by distributing y number of spatially correlated pixels into z number of different packets, wherein y≠z. In one example, y can be greater than z, whereby at least one of the packets includes two or more spatially correlated (neighboring) pixels from a partition. It is also possible to split pixels vertically. However, for interlaced format, since two neighboring lines are already split into two separate fields, it is preferable to partition horizontally for each field if only two partitions are required.
If more than two partitions are needed, then a combination of horizontal and vertical partitioning can be considered, as well as horizontal splitting or vertical splitting. Additional examples of partitioning according to the present invention wherein pixels are partitioned into more than two groups are provided below.
Specifically, for the type 0 pixels, the indices i and j are even numbers (i.e., i=0, 2, 4, . . . , etc., and j=0, 2, 4, . . . , etc.), and the type 0 pixels are placed in the Packet 0. For the type 1 pixels, the index i is odd (i.e., i=1, 3, 5, . . . , etc.), the index j is even (i.e., j=0, 2, 4, . . . , etc.), and the type 1 pixels are placed in the Packet 1. For the type 2 pixels, the index i is even (i.e., i=0, 2, 4, . . . , etc.), the index j is odd (i.e., j=1, 3, 5, . . . , etc.), and the type 2 pixels are placed in the Packet 2. For the type 3 pixels, the indices i and j are odd numbers (i.e., i=1, 3, 5, . . . , etc., and j=1, 3, 5, . . . , etc.), and the type 3 pixels are placed in the Packet 3. A CRC for each packet is appended at the end of the packet before transmission to a receiver of a wireless channel.
If during transmission, a pixel in one packet (e.g., Packet 0) is corrupted, then spatially related pixels in the other three packets (e.g., Packets 1, 2, or 3) can be used at the receiver to compensate for the corrupted pixel. As such, if pixel information in position P in a packet (e.g., Packet 0 in
Different packets can be transmitted at a single channel or at different channels/paths. In addition to robustness improvement, in the case when one channel/path cannot meet the bandwidth requirement for a HD stream, spatial pixel partitioning can take advantage of multi-channel/path to transmit all data of a HD video stream.
In general, square/rectangular blocks 104 (each block including multiple pixels therein), can be used for partitioning the multiple pixels in each block into corresponding multiple packets, wherein for each block, preferably each pixel in that block is placed in a different packet for transmission.
In the example show in
At the receiver, the received packets are processed for errors. When based on a CRC check a packet is determined to be corrupt, in order to determine the corrupted pixels, all the pixels in the corrupt packet are compared with corresponding pixels in an adjacent non-corrupt packet, on a pixel-by-pixel basis. If there is a sharp change (i.e., greater than a pre-defined threshold) between two corresponding pixels which belong to different partitions/packets, then the pixel in the corrupt packet is likely wrong, and it is corrected based on adjacent packets (described below). Otherwise, the pixel is used as is.
Each received packet is processed according to the above steps for error detection and recovery.
The transmitter 202 further includes a partitioning module 210 that receives video frames and implements the above partitioning steps on uncompressed video from higher layers, and a packetization and encoding module 212 that generates packets of data. The MAC layer 208 converts each data packet into a MAC packet by adding a MAC header to each data packet, and further calculates and adds CRC information to the data packet. The MAC packets are then provided to the PHY layer 206. The PHY layer 206 adds a PHY header to each MAC packet for transmission to the WiHD receiver 204 via transmit antennas 203.
In the receiver 204, the PHY layer 214 receives transmitted packets. The MAC layer 216 processes each received packet and performs error detection and error recovery according to the steps in the flowchart of
The receiver 204 further includes an error detection module 219 and a repair module 220. The error detection module 219 detects lost or damaged pixels in packets (e.g., using CRC information). The repair module 220 utilizes information from neighboring pixels, as described, to compensate for that lost or damaged pixel when the de-partitioning module 218 performs said inverse partitioning steps. In one example, the detection module 219 and a repair module 220 performs the process steps in the flowchart of
In another example implementation, the MAC layer 208, the partitioning module 210, and the packetization and encoding 212 in the WiHD transmitter 202 are logical modules. As such, though in the example of
In accordance with further features of the present invention, an encoding method is provided for conserving transmission bandwidth. A typical HD video frame has M rows and N lines (columns), and a total of M*N pixels (* means multiplication). Each pixel has D bits, wherein at a frame updating frequency of f frames per second (frames/sec), the data rate required for transmission of an HD video frame is M*N*D*f bits per second. For example, in the 1080 p video format, each frame has 1920 rows and 1080 lines, and each pixel has 24 bits, such that if the frame updating frequency is 60 frame/sec, then the transmission data rate is 1920*1080*24*60=2,985,984,000 bps. In some cases, it may be difficult for wireless hardware and PHY layer to meet the bandwidth requirements of uncompressed HD video transmission.
To solve this problem, according to another aspect of the present invention, first a pixel partitioning process as described above is applied to partition the neighboring video pixels into K partitions wherein every K pixels form a block. Then, instead of placing all original video pixel information into packets, the value of the nth fixed position pixel is selected at every K pixel block as base information (n<K), and information of other pixels in the block is encoded within the same block (e.g., by using differential pulse code modulation (DPCM) or binary XOR (bXOR) encoding). The encoded pixels are termed DIFF pixels and transmitted in DIFF packets, and the pixels carrying the original video data are termed BASE pixels and transmitted in BASE packets.
Because spatially correlated pixels usually have very similar or even the same values, after DPCM or bXOR encoding, the Most Significant Bits (MSBs) of the DIFF pixels are mostly zero. In order to conserve transmission bandwidth, the zero bits need not be transmitted. Two example implementations of such an approach are hard truncation and Run Length Coding (RLC) as described below.
An example of hard truncation involves truncating the high order zero bits of encoded DIFF pixels, such that less number of bits is needed for transmission. Assuming one pixel has D bits, for a BASE pixel all of the D bits are used to carry the original data information. However, for a DIFF pixel, D1 bits (D1<D) are used for DPCM or bXOR encoding. Preferably, the exact value of D1 is selected in advance, according to video content types. If D1 is selected to be less than the bits required to carry encoded information for a DIFF pixel, then the D1 bits are set to the value closest to the real encoded value.
A first packet 107 (i.e., Packet 0) is constructed which includes D bits 107A of the original data per pixel for said first group of pixels (BASE pixels). The first packet 107 is an example of a BASE packet. A second packet 109 (i.e., Packet 1) is constructed which includes D1 bits 109A for said second group of pixels that are DPCM or bXOR encoded per pixel (DIFF pixels). The second packet 109 is an example of a DIFF packet. In this example where K=2, then D=24 and D1=12, wherein for each DIFF pixel, 12 bits are used for transmission after DPCM or bXOR encoding.
The above hard truncation example is a simple solution to reduce the transmission bandwidth requirement. To avoid introduction of error for DIFF pixels, if D1 bits are insufficient for all of the bits of the DPCM or bXOR encoded value, then RLC is used for the DIFF pixels and the bit order is re-organized in each DIFF packet to carry DIFF pixels.
A first packet 110 (i.e., Packet 0) is constructed which includes D bits 110A of the original data per pixel from said first group of pixels (BASE pixels). The first packet 110 is another example of a BASE packet. A second packet 112 (i.e., Packet 1) is constructed which includes data information after DPCM or bXOR encoding, re-organization and RLC, per pixel from said second group of pixels. The second packet 112 is another example of a DIFF packet.
The bits of the DIFF pixels in a DIFF packet 112 are grouped and re-ordered according to their information significance in the pixel. For example, the first MSBs of all pixels are grouped together, which are followed by the second MSBs of all pixels, and so on, up to the LSBs (Least Significant Bits) of all pixels. Then, RLC is applied either to the reorganized DIFF bits stream or to just the MSBs part of the reorganized bit stream. Since after DPCM or bXOR encoding most MSBs are zero, RLC can achieve high compression ratio without loss of any information. Note that it is also possible to aggregate multiple DIFF packets together to reduce various MAC layer overhead in wireless transmission.
The above encoding methods can be implemented in the transmitter 202 of system 200 in
As is known to those skilled in the art, the aforementioned example architectures described above, according to the present invention, can be implemented in many ways, such as program instructions for execution by a processor, as logic circuits, as an ASIC, as firmware, etc.
The present invention has been described in considerable detail with reference to certain preferred versions thereof; however, other versions are possible. Therefore, the spirit and scope of the appended claims should not be limited to the description of the preferred versions contained herein.
This application claims priority from U.S. Provisional Patent Application Ser. No. 60/773,826, filed on Feb. 15, 2006, incorporated herein by reference.
Number | Name | Date | Kind |
---|---|---|---|
4661862 | Thompson | Apr 1987 | A |
5189510 | Henaff et al. | Feb 1993 | A |
5289190 | Shimoda et al. | Feb 1994 | A |
5453840 | Parker et al. | Sep 1995 | A |
5936669 | Niesen | Aug 1999 | A |
5969764 | Sun et al. | Oct 1999 | A |
6052159 | Ishii et al. | Apr 2000 | A |
6088045 | Lumelsky et al. | Jul 2000 | A |
6094453 | Gosselin et al. | Jul 2000 | A |
6115420 | Wang | Sep 2000 | A |
6201834 | Zhu | Mar 2001 | B1 |
6239886 | Klassen et al. | May 2001 | B1 |
6298085 | Kondo et al. | Oct 2001 | B1 |
6418240 | Yu | Jul 2002 | B1 |
6512218 | Canini et al. | Jan 2003 | B1 |
6571016 | Mehrotra et al. | May 2003 | B1 |
6757435 | Kondo | Jun 2004 | B2 |
6868186 | Sadeh | Mar 2005 | B1 |
6973221 | Xue | Dec 2005 | B1 |
7027515 | Lin | Apr 2006 | B2 |
7075993 | O'Brien, Jr. | Jul 2006 | B2 |
7099678 | Vaidyanathan | Aug 2006 | B2 |
7113556 | Heegard et al. | Sep 2006 | B1 |
7227900 | Porter et al. | Jun 2007 | B2 |
7283165 | Alderson et al. | Oct 2007 | B2 |
7339993 | Brooks et al. | Mar 2008 | B1 |
7627348 | Lysejko et al. | Dec 2009 | B2 |
7643558 | Garudadri et al. | Jan 2010 | B2 |
7734106 | Zhang et al. | Jun 2010 | B1 |
7991055 | Cancemi et al. | Aug 2011 | B2 |
8098741 | Suh et al. | Jan 2012 | B2 |
20020116715 | Apostolopoulos | Aug 2002 | A1 |
20030063676 | Prakash et al. | Apr 2003 | A1 |
20040032516 | Kakarala | Feb 2004 | A1 |
20060013299 | Sato et al. | Jan 2006 | A1 |
20060013320 | Oguz et al. | Jan 2006 | A1 |
20060013321 | Sekiguchi et al. | Jan 2006 | A1 |
20060146940 | Gomila et al. | Jul 2006 | A1 |
20060239360 | Kadono et al. | Oct 2006 | A1 |
20060268760 | Fang et al. | Nov 2006 | A1 |
20070014360 | Botzko et al. | Jan 2007 | A1 |
20070091999 | Nissan-Cohen et al. | Apr 2007 | A1 |
20070098063 | Reznic et al. | May 2007 | A1 |
20070189383 | Shao et al. | Aug 2007 | A1 |
20070296822 | Lan et al. | Dec 2007 | A1 |
20080101467 | MacMullan et al. | May 2008 | A1 |
20080107330 | Cotman et al. | May 2008 | A1 |
20080144553 | Shao et al. | Jun 2008 | A1 |
20080232478 | Teng et al. | Sep 2008 | A1 |
20080267299 | Hannuksela et al. | Oct 2008 | A1 |
20080285651 | Au et al. | Nov 2008 | A1 |
20090021646 | Shao et al. | Jan 2009 | A1 |
20090063935 | Singh et al. | Mar 2009 | A1 |
20100014584 | Feder et al. | Jan 2010 | A1 |
20100150463 | Yeung et al. | Jun 2010 | A1 |
20100166057 | Huchet et al. | Jul 2010 | A1 |
20100265392 | Shao et al. | Oct 2010 | A1 |
20110109792 | Montag | May 2011 | A1 |
Number | Date | Country |
---|---|---|
10-2009-0100219 | Sep 2009 | KR |
2004073201 | Aug 2004 | WO |
WO 2004073201 | Aug 2004 | WO |
2008060025 | May 2008 | WO |
Entry |
---|
International Search Report and Written Opinion mailed Dec. 6, 2010 in PCT/KR2010/002192, 6 pp., Korean Intellectual Property Office, Republic of Korea. |
Schwarz, H. et al., “Overview of the Scalable Video Coding Extension of the H.264/AVC Standard”, IEEE Transactions on Circuits and Systems for Video Technology, vol. 17, No. 9, Sep. 2007, pp. 1-18, United States. |
PCT International Search Report, International Application No. PCT/KR2007/003251, dated Oct. 12, 2007. |
Hitachi Ltd. et al. High-Definition Multimedia Interface (HDMI) Specifications version 1.2, Aug. 22, 2005. |
International Search Report and Written Opinion dated Oct. 12, 2007 for International Application No. PCT/KR2007/003251 from Korean International Property Office, filed Jul. 4, 2007, 10 pages, Seo-gu, Daejeon, Republic of Korea. |
Korean Office Action dated Nov. 11, 2009 for Korean Patent Application No. 10-2008-7006604, pp. 1-3, Korean Intellectual Property Office, Seo-gu, Daejeon, Republic of Korea. |
Korean Final Office Action dated Feb. 18, 2010 for Korean Patent Application No. 10-2008-7006604, pp. 1-2, Korean Intellectual Property Office, Seo-gu, Daejeon, Republic of Korea. |
Korean Office Action dated Nov. 11, 2009 for Korean Patent Application No. 10-2008-7006604, pp. 1-3, Korean Intellectual Property Office, Seo-gu, Daejeon, Republic of Korea (Machine-generated English translation attached, pp. 1-2). |
Korean Final Office Action dated Feb. 18, 2010 for Korean Patent Application No. 10-2008-7006604, pp. 1-2, Korean Intellectual Property Office, Seo-gu, Daejeon, Republic of Korea (Machine-generated English translation attached, p. 1). |
U.S. Non-Final Office Action for U.S. Appl. No. 12/048,126 mailed Dec. 28, 2011. |
U.S. Notice of Allowance for U.S. Appl. No. 11/897,087 mailed Apr. 25, 2012. |
U.S. Non-Final Office Action for U.S. Appl. No. 12/048,126 mailed Oct. 22, 2012. |
U.S. Notice of Allowance for U.S. Appl. No. 11/897,087 mailed Jan. 20, 2012. |
U.S. Final Office Action for U.S. Appl. No. 12/048,126 mailed Mar. 20, 2013. |
U.S. Non-Final Office Action for U.S. Appl. No. 12/754,522 mailed Jan. 18, 2013. |
U.S. Final Office Action for U.S. Appl. No. 12/754,522 mailed May 16, 2013. |
U.S. Non-Final Office Action for U.S. Appl. No. 12/048,126 mailed Aug. 21, 2013. |
Number | Date | Country | |
---|---|---|---|
20070202842 A1 | Aug 2007 | US |
Number | Date | Country | |
---|---|---|---|
60773826 | Feb 2006 | US |