The present invention relates to the field of video processing. More specifically, the present invention is related to automatic detection and conversion of video from one of a number of formats to a format of choice.
Advances in microprocessor, networking and related technologies have led to wide spread deployment and adoption of server-client based applications. In particular, with the advance of high speed public networks, such as the Internet, increasing amounts of rich content, such as video, are being served and available for consumption by networked clients.
For a variety of reasons, most client devices are equipped with progressive scan monitors. Resultantly, video contents are typically encoded and streamed to client devices for consumption in a progressive format. While a substantial volume of video content is available in the progressive format, there exist also large volumes of video contents in other formats, such as the interlaced format or the telecine format (which may be progressive or interlaced).
Interlaced format video typically consists of fields and frames. See
Telecine generally refers to the process of converting 24 frame-per second (fps) progressive format (e.g. from film) to 30 fps interlaced (e.g. NTSC, National Television Standard Committee). The process typically involves repeating certain fields of the video, in e.g. a 3:2 pattern. See
A number of deinterlacers and telecine inverters to deinterlace interlaced or invert telecine formatted videos to a progressive format are known in the art, and available for use by video providers to accomplish the conversion. However, under the prior art, the responsibility of selecting and invoking the appropriate reformatter, i.e. a deinterlacer or a telecine inverter, rests on the user operators of a video provider.
Unfortunately, user operators of video providers often do not know the content format well enough to invoke the appropriate “reformatter”. The task is further complicated by the fact that many videos are mixed formats, e.g. a telecine formatted video may contain one or more portions of interlaced video.
Thus, availability of methods and/or apparatuses to automatically detect and reformat video from one of a number of formats to a format of choice is desired.
The present invention will be described by way of exemplary embodiments, but not limitations, illustrated in the accompanying drawings in which like references denote similar elements, and in which:
a illustrates interlaced formatting of video;
b illustrates telecine formatting of video;
c illustrates a “sub-sampling” arrangement for determining whether a frame is interlaced formatted by examining only a subset of the pixel blocks of a frame, in accordance with one embodiment;
The present invention includes components of a video device, methods of operations practiced thereon, video devices and systems so formed, and other related subject matters.
In the following description, various aspects of the present invention will be described. However, it will be apparent to those skilled in the art that the present invention may be practiced with only some or all aspects of the present invention. For purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without the specific details. In other instances, well-known features are omitted or simplified in order not to obscure the present invention.
Parts of the description will be presented in video processing terminology consistent with the manner commonly employed by those skilled in the art to convey the substance of their work to others skilled in the same art. These terms, for the purpose of this application, are to be accorded their commonly understood meaning. In particular, in a video processing device, a video and its associated data may take the form of electrical, magnetic, or optical signals capable of being stored, transferred, combined, and otherwise manipulated through electrical and/or optical components of a processor, and its subsystems.
Section headings are merely employed to improve readability, and they are not to be construed to restrict or narrow the present invention.
Various operations will be described as multiple discrete steps in turn, in a manner that is most helpful in understanding the present invention, however, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations need not be performed in the order of presentation.
The phrase “in one embodiment” is used repeatedly. The phrase generally does not refer to the same embodiment, however, it may. The terms “comprising”, “having” and “including” are synonymous, unless the context dictates otherwise.
Together, elements 102-110 cooperate to automatically determine whether an input video 122 is one of telecine, non-telecine progressive or non-telecine interlaced formatted. Further, if input video 122 is determined to be either telecine or non-telecine interlaced formatted, elements 102-110 cooperate to automatically convert the video from either telecine or non-telecine interlaced format to non-telecine progressive formatted video 124.
More specifically, telecine and non-telecine progressive detector 102 is used to automatically determine whether input video 122 is telecine or non-telecine formatted, and to automatically determine whether a non-telecine formatted video 122 is progressive formatted. Telecine inverter 104 is used to automatically invert telecine formatted video 122 into non-telecine progressive formatted video 122.
Interlaced detector 106 is used to automatically determine whether non-telecine formatted video 122 is interlaced, and deinterlacer 108 is used to deinterlace interlaced video 122.
Resultantly, a user of a video device (e.g. video device 700 of
For the illustrated embodiment, telecine inverter 104 also informs telecine and non-telecine progressive detector 102 of frames dropped during the inversion process, to facilitate detector 102 in detecting progressive patterns by analyzing the number of frames of video 122 consistent with the interlaced or progressive pattern, to be described more fully below. Detector 102 also notifies deinterlaced detector 106 of whether detector 102 was able to detect progressive patterns (in the event input video 122 is not considered to be telecine formatted). Further, both detectors 102 and 106 also notify selector 110 of the results of their respective determinations, allowing selector 110 to select the proper source, i.e. video 122 itself, output of telecine inverter 104 or output of deinterlacer 108 to output as non-telecine progressive formatted video 124.
In an alternate embodiment, e.g. a software implementation, where inverter 104, interlaced detector 106, and deinterlacer 108 are selectively enabled/invoked to operate on input video 122 on an as needed basis, the present invention may be practiced without employing selector 110.
Telecine and non-telecine progressive detector 102 (except for the concurrent progressive detection aspect) is the subject matter of U.S. patent application Ser. No. 09/732,217, entitled Automated Inverse Telecine Conversion, filed on Dec. 6, 2000. Telecine inverter 104 and deinterlacer 108 are known in the art. Accordingly, these elements or the known aspects of these elements will not be further described.
The progressive detection aspect of telecine and non-telecine progressive detector 102, and interlaced detector 106, along with other aspects of the present invention, will be further described in turn below.
Before doing so, it should be noted that, while for ease of understanding, the present invention is being described with an embodiment equipped to automatically detect telecine, non-telecine progressive and non-telecine interlaced video, and automatically convert telecine and non-telecine interlaced video to non-telecine progressive video, the present invention anticipates the described embodiment may also be practiced with automatic detection and conversion of other known formats.
If the input video is automatically determined to be telecine formatted, block 204, the input video is automatically inverted to be non-telecine progressive formatted, block 206, before being outputted, block 212, e.g. to an encoder.
On the other hand, if the input video is automatically determined to be non-telecine, but progressive formatted, block 208, the input video is outputted directly, block 212, e.g. to an encoder.
However, if the input video is determined to be neither telecine nor non-telecine progressive formatted, the input video is further analyzed to determine if the input video is interlaced formatted, block 210. If the input video is determined to be non-telecine interlaced formatted, block 214, the input video is automatically deinterlaced, block 216, before being outputted, block 212.
Identification of frames to be dropped is a by-product of the telecine inversion process, as described earlier, telecine typically involves converting 24 fps progressive format (e.g. from film) to 30 fps (e.g. NTSC). The process involves repeating certain frames of the video in e.g. a well known 3:2 pattern. Thus, inverse telecine will necessarily involve converting 30 fps (e.g. NTSC) back to 24 fps, resulting in one in every 5 input frames being dropped.
As illustrated, the progressive detection portion of telecine and non-telecine progressive detector 102 monitors for frames being dropped, block 302. On detection, detector 102 determines an amount of time elapsed since the last time a frame was dropped, block 304.
On determination, detector 102 determines whether the amount of time elapsed since the last time a frame was dropped is within an expected range (encompassing the expected elapsed time), block 306. In one embodiment, the expected range may be from 145 ms to 175 ms (encompassing the expected elapsed time of 167 ms).
If the amount of time elapsed since the last time a frame was dropped is within an expected range, detector 102 increments either a progressive counter or an interlaced counter by an amount, block 308. Whether the progressive counter or the interlaced counter gets incremented depends on whether detector 102 has detected progressive or interlaced telecine patterns, as part of its mainline telecine analysis. If detector 102 has detected progressive telecine patterns, the progressive counter is incremented. On the other hand, if detector 102 has detected interlaced telecine patterns, the interlaced counter is incremented.
For the embodiment, as detector 102 increments either the progressive or interlaced counter as described above, it also decrements the other counter by an amount. In one embodiment, the amounts are the same. More specifically, in one embodiment, the counters are initialized to zero on start up, and the increment/decrement amount may be set to 5.
On incrementing/decrementing the progressive/interlaced counter as described, the process continues back at block 302.
Back at block 306, if the amount of elapsed time is determined not to be within the expected range, for the embodiment, detector 102 further determines whether the amount of elapsed time is below an upper threshold, block 310. If the amount of elapsed time is below the upper threshold, detector 102 similarly decrements one of the progressive or interlaced counter by an amount, block 320. Again, whether the progressive or interlaced counter is decremented depends on whether detector 102 has detected progressive or interlaced telecine patterns in its mainline analysis. If detector 102 has detected progressive telecine patterns, the progressive counter is decremented. On the other hand, if detector 102 has detected interlaced telecine patterns, the interlaced counter is decremented. In one embodiment, the amounts are the same, and may be set to 5 also.
Similarly, on incrementing/decrementing the progressive/interlaced counter as described, the process continues back at block 302.
Referring now to
If the frame is to be considered interlaced formatted, detector 106 increments an interlaced detection measure (detection_measure), block 410. On the other hand, if the frame is to be considered progressive formatted, detector 106 decrements the interlaced detection measure, block 412. Upon incrementing/decrementing the interlaced detection measure, the process continues back at block 402.
If the video cannot be considered as either progressive or interlaced formatted, the interlaced detection measure is left unchanged, and the process immediately continues back at block 402.
In one embodiment, on start up, the interlaced detection measure is initialized to zero, and on conclusion that a frame is to be considered as interlaced formatted, the interlaced detection measure may be incremented as follows:
detection_measure=(31*detection_measure+256)/32 (1)
On conclusion that a frame is to be considered as progressive formatted, the interlaced detection measure may be decremented as follows:
detection_measure=(31*detection_measure+0)/32 (2)
In one embodiment, the video is to be considered interlaced if the interlaced detection measure reaches a threshold. In one embodiment, the threshold may be 16.
In one embodiment, where each frame of video 122 has more than 242 lines, the interlaced detection measure may be initialized at 16 instead.
Referring now to
On selecting a pixel block, detector 106 computes difference measurements for even and odd lines of the pixel block to determine if there are sufficient movements in the pixel block to facilitate interlaced detection, block 504. The operational is performed as adjacent fields of an interlaced frame often contain data of slightly different times, e.g. by 1/60 or 1/50 of a second, and sufficient presence of movements enable discernment of such time differential.
In one embodiment, the sum of absolute differences (SAD) is used for the difference measurement for even and odd lines (SAD_even and SAD_odd) and may be computed as follows:
where p(x,y) is the luma-plane pixel value located x pixels to the right and y pixels down from the upper-left edge of the current 8×8 block;
q(x,y) is the pixel located at a corresponding position in the previous frame.
In alternate embodiments, the reference point may be a pixel at another location other than the upper-left edge of the pixel block.
In one embodiment, the SADs may be sub-sampled to reduce the amount of calculations. Specifically, the SADs may be calculated as follows:
On computing SAD_even and SAD_odd, detector 106 compares the computed SAD_even and SAD_odd to a movement threshold to determine whether the pixel block is to be considered as having sufficient movement to facilitate interlaced detection, block 506. In one embodiment, the movement threshold may be 400.
If the pixel block is determined to have sufficient movement to facilitate interlaced detection, detector 106 computes a number of one or more step vertical and/or horizontal differences for the pixel block, and accumulates the computed vertical/horizontal differences, block 508.
In one embodiment, a 1-step vertical difference (step1v), a 2-step vertical difference (step2v), a 1-step horizontal difference (step1h), and a 2-step horizontal difference (step2h) are computed. The 1 and 2-step vertical and horizontal differences may be computed as follows:
On computing the step differences, detector 106 determines whether the pixel block is to be considered interlaced or progressive, block 509. If so, for the embodiment, interlaced detector 106 increments an interlaced block counter interlaced_blocks or a progressive block counter progressive_blocks by 1 accordingly.
In one embodiment, detector 106 increments the interlaced and progressive block counters, interlaced_blocks and progressive_blocks as follows:
If step1v>step2v and step1h<T1 and step2v<T2, then the block is classified as interlaced, and the interlaced counter interlaced_blocks is incremented by 1.
If step1v<2×step2v and step1h<T3 and step1v<T4, then the block is classified as progressive, and the progressive counter progressive_blocks is incremented by 1.
In one embodiment, T1=T2=T3=T4=T, and T may be 25600.
In one embodiment, the operation at block 508 also includes accumulating the computed values of step1v, step2v, step1h, and step2h of each analyzed block into four corresponding running variables step1v_total, step2v_total, step1h_total, and step2h_total for the frame. For the embodiment, these variables, in addition to the counters interlaced_blocks and progressive_blocks are also later used in the determination whether the frame is to be considered interlaced.
On computing step1v, step2v, step1h, and step2h, accumulating step1v_total, step2v_total, step1h_total, and step2h_total, and determining whether the analyzed block is to be considered interlaced or progressive, blocks 508-509, detector 106 determines if there are more blocks to analyze, block 510. If so, the process continues back at block 502. If all pixel blocks have been analyzed, then detector 106 proceeds to determine whether the frame is to be considered interlaced formatted.
As described earlier, in one embodiment, the pixel block analysis is performed using a subset of the pixel blocks. In one embodiment, the pixel blocks of a frame analyzed are the “shaded” pixel blocks of a frame as illustrated in
In one embodiment, detector 106 determines whether the frame is to be considered interlaced formatted as follows:
A frame is considered to be interlaced
A frame is to be considered progressive
where width and height are in units of pixels.
As described earlier, pre-filter arrangement 100 is employed to automatically determine at least whether an input video is telecine, non-telecine progressive or non-telecine interlaced formatted, and if the input video is determined to be telecine or non-telecine interlaced formatted, automatically convert the input video to the non-telecine progressive format, before providing the input video to encoder 702 for encoding.
As described earlier, resultantly, a user of video device 700 need not be responsible for knowing the format of the input video nor for invoking the appropriate inverter or deinterlacer at the right time to properly process the input video.
For the embodiment, video device 800 includes storage medium 802 to store at least a portion of a working copying of the programming instructions implementing the software embodiment of pre-filter arrangement 100, and at least one processor 804 coupled to storage medium 802 to execute the programming instructions.
Video device 800 may be any one of a number of video devices known in the art, including but not limited to video servers. Article 810 may e.g. be a diskette, a compact disk (CD), a DVD or other computer readable medium of the like. In other embodiments, article 810 may be a distribution server distributing the programming instructions implementing the present invention to video device 800 on line, via private and/or public networks, such as the Internet. In one embodiment, article 810 is a web server.
The format of video 904 is automatically detected, and converted if necessary, by pre-filter arrangement 100, as earlier described, before being provided to encoder 906 to be encoded and sent to video receiver device 902.
Video receiver device 912 employs decoder 914 to decode the provided video for player 916 to render for consumption by a user of video receiver device 912.
Video provider and receiver devices 902 and 912 may be any one of a number of video providing and receiving devices known in the art. Video provider 902 may e.g. be a video server. Video receiving device 912 may e.g. be a daughter card, a motherboard, a palm sized computing device, a wireless mobile phone, a digital personal assistant, a laptop computing device, a desktop computing device, a set-top box, a server, a compact disk player, a digital versatile disk player, a television, or a display monitor.
Video provider and receiver devices 902 and 912 may be coupled to each other via a wired or a wireless connection. The connection may e.g. be a bus connection, such as the Universal Serial Bus (USB), the Peripheral Component Interconnect (PCI) bus, or other buses. The connection may e.g. also be a local or wide area networking connection, such as an Ethernet, a Token Ring, Transmission Control Protocol/Internet Protocol (TCP/IP) or other networking protocols/transports of like kind.
Thus, it can be seen from the above descriptions, an improved video processing device, including methods practiced thereon, and other related aspects, have been described.
While the present invention has been described in terms of the foregoing embodiments and example applications, those skilled in the art will recognize that the invention is not limited to the embodiments and example application described. The present invention can be practiced with modification and alteration within the spirit and scope of the appended claims.
Thus, the description is to be regarded as illustrative instead of restrictive on the present invention.
This application is a continuation of and claims benefit of priority under 35 U.S.C. § 120 of U.S. patent application Ser. No. 10/340,376, filed Jan. 10, 2003 now U.S. Pat. No. 7,154,555, entitled “AUTOMATIC DEINTERLACING AND INVERSE TELECINE,” which is hereby incorporated herein by reference in its entirety.
Number | Name | Date | Kind |
---|---|---|---|
4982280 | Lyon et al. | Jan 1991 | A |
5742351 | Guede | Apr 1998 | A |
5754248 | Faroudja | May 1998 | A |
5757435 | Wells | May 1998 | A |
5812202 | Ng et al. | Sep 1998 | A |
5821991 | Kwork | Oct 1998 | A |
5828786 | Rao et al. | Oct 1998 | A |
5847772 | Wells | Dec 1998 | A |
5864368 | Kato et al. | Jan 1999 | A |
5929902 | Kwork | Jul 1999 | A |
6014182 | Swartz | Jan 2000 | A |
6055018 | Swan | Apr 2000 | A |
6157412 | Westerman et al. | Dec 2000 | A |
6160589 | Sadowski | Dec 2000 | A |
6222589 | Faroudja et al. | Apr 2001 | B1 |
6380978 | Adams et al. | Apr 2002 | B1 |
6525774 | Sugihara | Feb 2003 | B1 |
6538688 | Giles | Mar 2003 | B1 |
6545727 | Pau et al. | Apr 2003 | B1 |
6549668 | Pezzoni et al. | Apr 2003 | B1 |
6563550 | Kahn et al. | May 2003 | B1 |
6661464 | Kokkosoulis et al. | Dec 2003 | B1 |
6670996 | Jiang | Dec 2003 | B2 |
6700622 | Adams et al. | Mar 2004 | B2 |
6704055 | He et al. | Mar 2004 | B1 |
6724433 | Lippman | Apr 2004 | B1 |
6839094 | Tang et al. | Jan 2005 | B2 |
6842194 | Sugihara | Jan 2005 | B1 |
6867814 | Adams et al. | Mar 2005 | B2 |
6870568 | Hui | Mar 2005 | B1 |
6873368 | Yu et al | Mar 2005 | B1 |
6891571 | Shin et al. | May 2005 | B2 |
6897903 | Hu | May 2005 | B1 |
6904173 | Bagni et al. | Jun 2005 | B2 |
6952212 | Nister et al. | Oct 2005 | B2 |
7075581 | Ozgen et al. | Jul 2006 | B1 |
7116828 | Wells | Oct 2006 | B2 |
20010002853 | Lim | Jun 2001 | A1 |
20010002921 | Bagni et al. | Jun 2001 | A1 |
20010026328 | Del Corson | Oct 2001 | A1 |
20020057368 | Fakhruddin | May 2002 | A1 |
20020130970 | Jiang | Sep 2002 | A1 |
20020149703 | Adams et al. | Oct 2002 | A1 |
20030189667 | Chow | Oct 2003 | A1 |
20030206242 | Choi | Nov 2003 | A1 |
20040001159 | Shen et al. | Jan 2004 | A1 |
Number | Date | Country | |
---|---|---|---|
20070024703 A1 | Feb 2007 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 10340376 | Jan 2003 | US |
Child | 11541767 | US |