Patent Application
20110265134
This invention relates to a system and method for encoding and decoding video signals to and from a video transport stream over IP. Video signals are typically MPEG-2 and H.264 compliant but may further include raw video ingress and egress via DVB-ASI and HD-SDI.
The challenges created by the ever evolving video encoding and transport standards force new generations of video equipment that video producers and distributors are required to manage, control and invest in. This is particularly true of distributers who provide high definition video to a large number of customers using a variety of encodings such as the MPEG-2 and H.264 standards, which are not compatible in bit-rate or in encoding technology. To manage in this environment, advanced but economical video compression techniques are required to transmit video. Furthermore, a dynamic platform is required to accommodate the ever evolving standards in order to reduce equipment churn.
Conventional approaches require complex ASICS or arrays of DSPs to manage the intensive signal processing. These approaches reduce flexibility, compromise quality and add non-recurring engineering costs inherent in ASIC production. What is needed is a high performance, high speed, low cost hardware platform in combination with software programmability so that future video signal processing standards may be incorporated into the platform as those standards evolve.
Traditionally, satellite broadcast systems have sent compressed video signals to cable television stations for redistribution using cable TV head-ends using related point to multipoint hardware architectures. But with the increasing performance of the internet network and the internet protocol (IP) for transport of constant bit-rate signals such as video, customers are being offered new options including high definition television HDTV on-demand. In many cases, the video sources may be standard definition SDTV product which need to be converted. In other cases, the video sources may be in a streaming format such as MPEG-2, supporting compression for HD-TV broadcast transport at about 20 Mb/s. However, bandwidth limited IP networks that need to carry multiple HDTV signals suffer congestion. It would be advantageous to convert the MPEG-2 video streams to H.264 standard video streams, that offer better compression efficiencies with bit-rates approximately 50% of MPEG-2. It would be also be advantageous to provide additional statistical multiplexing for IP transport of the lower bit-rate H.264 video streams.
U.S. Pat. no. 7,818,442 to Hershey et al. discloses a streaming media encoder for encoding media content. The media encoder has a confidence monitor for the input and output video data streams and a network interface.
U.S. Patent Patent Application Pub. No. 2008/0240093 to Morad et al. disloses a stream multiplexer/de-multiplexer for operating on packetized digital data streams including DVB receivers and DVB transmitters. The host interface for control can be a PCI, PCIe bus or Ethernet. The apparatus includes a central switch for switching data streams between various video and audio processors. The disclosure is primarily concerned with an architecture for a micro controller unit for video packet processing.
U.S. Patent Application Pub. No. 2009/0201988 to Gazier et al. discloses systems and methods for video processing in network edge devices. Gazier et al further disclose a method including decoding, transcoding and encoding a video stream into a specific format and disclose a system including multiple transcoder line cards connected to a switch fabric.
U.S. Patent Application Pub. No. 2008/0091845 to Mills et al. discloses a system and method for processing video content. The system of Mills et al. is concerned with moving content files and transcoding content files while in the process of moving them, although the modification of content files into a display of low bit-rate video streams is considered.
U.S. Patent Application Pub. No. 2006/0168637 to Vyotsky et al. discloses a multiple-channel codec environment that utilizes a plurality of reconfigurable media signal processors to provide on-demand network functions such as A/V transcoding.
The present invention addresses the need for a programmable video signal processor through a combination of a hardware and dynamic software platform for video compression and decompression processing suitable for broadcast level high definition (HD) video encoding, decoding and multiplexing. None of the above mentioned references provide a suitably modular hardware design that is realized at low cost and yet capable of software reconfiguration without considerable design effort. As there are varied needs within the media industry for video processing, in relation to video signal formats and applications, the present invention address the need for a modular video server component which may be resused in many different designs to meet a particular video processing requirement. The dynamic software platform is implemented on a low cost multicore DSP architecture that allows for easy reconfiguration of software to meet the video processing requirement.
The present invention is a programmable codec system with sufficient flexibility to provide encoding, decoding and multiplexing functions in a plurality of application environments.
A video TS/IP server for transcoding and transrating is disclosed for transforming a set of input video streams from video sources into a set of output video streams for video destinations. The video TS/IP server is managed from a management console which accesses the video TS/IP server via a web enabled interface or a computer host via a PCI bus. The set of input video streams can include transport streams over IP with ingress from an IP network via Ethernet. The set of output video streams can include transport streams over IP with egress to an IP network via Ethernet. In an alternate embodiment, the video TS/IP server can encode uncompressed video streams directly from video applications into the set of output video streams, the uncompressed video streams being in the form of HD-SDI or DVB-ASI video streams. In another alternate embodiment, the video TS/IP server can decode the set of input video streams into uncompressed video streams and egress the uncompressed video streams directly to video applications, the uncompressed video streams being in the form of HD-SDI or DVB-ASI video streams. In the preferred embodiments, the transport streams include digital video formats MPEG-2, H.264, DV25 and DVC-pro/25/50/100.
The video TS/IP server comprises a set of video server blocks which may be configured to perform a variety of video processing functions. A video server block comprises at least two codecs, a primary FPGA, at least two VioIP processors, four Ethernet PHY interfaces and a set of supporting components including DDR memory, Flash memory and random-access memory. A first codec connects to the primary FPGA via a first set of data and control buses. The second codec connects to the primary FPGA via a second set of data and control buses. The first and second sets of data and control buses include a bidirectional bus for transporting uncompressed video streams, a bidirectional bus for transporting compressed video data streams, a bidirectional bus for transporting I2S audio data, and a bidirectional bus for peripheral data and control. Each VioIP processor connects to the primary FPGA via a bidirectional bus for transporting compressed audio and video (A/V) data streams. Flash memory containing program instructions for the codecs is attached to the primary FPGA. DDR memory is attached to each codec on which a set of video data buffers are implemented. A first Ethernet PHY interface is connected to a first VioIP processor. A second Ethernet PHY interface is connected to a second VioIP processor. A third Ethernet PHY interface is connected to the first codec. A fourth Ethernet PHY is connected to the second codec. The Ethernet PHY interfaces are terminated in RJ-45 connectors. A JTAG test CPLD is included for video server block debug activities.
The primary FPGA comprises a set of multiplexers and a pair of stream loaders. A first multiplexer selectably interconnects two input data and control buses suitable for compressed A/V data to two output data and control buses also suitable for compressed A/V data. A second multiplexer selectably interconnects two input data and control buses suitable for compressed A/V data to two output data and control buses also suitable for compressed A/V data. A third multiplexer selectably interconnects a flash memory interface to a first and second stream loader. The first stream loader is further connected to a first input data and control bus for uncompressed video data. The second stream loader is further connected to a second input data and control bus for uncompressed video data. The first and second stream loaders can be selectably configured to pass data from their respective input data and control bus or from the third multiplexer.
In a PCI-hosted video TS/IP server embodiment comprising a PCI host computer and a video server block, the video server block is augmented with a PCI bus driven by each of the two codecs. The PCI bus can be converted to a PCIe bus with a PCI to PCIe converter suitable to attach the video TS/IP server to the PC host computer. The primary FPGA can be configured by the PCI host computer via communication links between the codecs and the primary FPGA.
In another aspect of the invention, additional embodiments are realized by augmenting a video server block with a set of video and audio interfaces to transmit and receive uncompressed video streams.
A first video interface is a receiver for receiving uncompressed video data transmitted on a coaxiable cable connected to an adaptive cable equalizer to which an SDI receiver is connected. The output of the SDI receiver is connected to one of the codecs through a data and control bus.
The first video interface can be augmented with a multiplexer so that the SDI receiver is selectably connected to multiple codecs through multiple data and control buses.
A second video interface is a transmitter for transmitting uncompressed video data over a coaxiable cable connected to an SDI transmitter. The input of the SDI transmitter is connected to one of the codecs through a data and control bus.
The second video interface can be augmented with a multiplexer so that the SDI transmitter is selectably connected to multiple codecs through multiple data and control buses.
A first audio interface is a receiver for receiving uncompressed audio data transmitted on a suitable cable connected to an AES/EBU to I2S converter. The output of the AES/EBU to I2S converter is connected to at least one of the group of the first codec, the second codec and the primary FPGA.
A second audio interface is a transmitter for transmitting uncompressed audio data over a suitable cable connected to an I2S to AES/EBU converter. The input of the AES/EBU to I2S converter is connected to at least one of the group of the first codec, the second codec and the primary FPGA.
A stand-alone video TS/IP server embodiment comprises a set of N video server blocks, an Ethernet switch, an uncompressed video signal multiplexer, and a host processor. The host processor is further connected to a set of components including flash memory, random access memory, a set of status indicators, and a set of Ethernet ports. The uncompressed video signal multiplexer is an 2N port by 2N port digital bus multiplexer where any one of N/2 external inputs can be selectable connected to any one of N internal outputs and any one of N external outputs can be selectably connected to any one of N internal inputs. The N internal inputs and N internal outputs are connected to the set of video server blocks. A first set of 2N Ethernet ports are connected to the set of video server blocks for transmitting and receiving transport streams over internet protocol. A second set of 2N Ethernet ports are connected to the Ethernet switch block which is further connected to the host processor. A dual redundant power supply supplies power to the video TS/IP server and a set of cooling fans and includes a power button and power input connector.
The video TS/IP server includes a set of operable programs stored in flash memory and operated by a host processor and by the codecs comprising the video server block to selectably encode and decode audio and video data streams. The set of operable programs include a user interface, preferably a web application, usable to supply codec configurations for encoding and decoding tasks on the video TS/IP server. The set of operable programs include hardware drivers for the codecs to move data between the codec and other devices, including DDR memory and the set of data and control buses. The set of operable programs include communication, management and process functions to configure and control the encoding and decoding tasks. The set of operable programs include DSP related functions including intensive video processing that when executed operate on an input video/audio stream to transform it into an output video/audio stream according to the codec configurations.
The set of programs implement a set of methods for encoding and decoding video streams. A first method includes the steps of initiating the video server block, waiting for changes to the codec configurations to define encoding and decoding tasks, configuring the input and output video streams for processing according to the codec configurations, enabling the data flow of video streams on prescribed ports to and from the codecs, receiving a compressed audio/video data stream from an IP network as a transport stream and storing into a buffer, decoding the compressed audio/video data stream according to the codec configurations into a second buffer as an uncompressed audio/video data stream, encoding the uncompressed audio/video data stream into a second compressed audio/video data stream according to the codec configurations, and transmitting the second compressed audio/video data stream to an IP network as a transport stream.
These and other inventive aspects will be described in the detailed description below.
The disclosed embodiments will be described with reference to the accompanying drawings.
A first embodiment includes video distribution across an internet protocol (IP) network. In
Transport stream (TS) is a standard format for transmission and storage of audio, video, and data, and is used in broadcast systems. Transport Stream is specified in MPEG-2 Part 1, Systems (formally known as ISO/IEC standard 13818-1 or ITU-T Rec. H.222.0). The transport stream standard specifies a container format encapsulating packetized elementary streams (ES), with error correction and stream synchronization features for maintaining transmission integrity when the signal is degraded.
Management console 4 is provided to configure video TS/IP server 1. In a preferred embodiment, a management console operating as a web application on the video TS/IP server is served over IP network 3. A client computer operating a web browser may interact with the management console over IP network 3 to perform management functions.
In an alternate embodiment, video TS/IP server is a PCI-based hardware card hosted in a computer. The management console in the alternate embodiment is deployed as a standalone computer software application operating within an operating system running on the computer, such as Microsoft® Windows™ or Linux. The management console interacts directly with the video TS/IP server over PCI bus 10 of the computer.
Management functions include determining the nature of the transformation of each video data stream from ingress to egress; routing each of the video data streams to a destination transport stream and ultimately to a video destination; setting the bit-rate of each video data stream; and all other functions related to the operation of the video TS/IP server. Examples of the nature of the transformation include transforming an MPEG-2 video data stream on ingress to an H.264 video data stream on egress and vice versa; transforming a raw video data stream on ingress to an MPEG-2 video data stream on egress and vice versa; and transforming a raw video data stream on ingress to an H.264 video data stream on egress and vice versa. These examples of transformations are not exhaustive but illustrative of the function of the video TS/IP server.
A second embodiment is in video production across an interne protocol (IP) network. In
HD-SDI is the high-definition version of the Serial Digital Interface (SDI) specified in SMPTE-292M. This standard transmits audio and video over a single coaxial cable with a data rate of 1.485 Gbit/second and is typically used in the video production environment. DVB-ASI is the Digital Video Broadcasting Asynchronous Serial Interface which is a standard for the broadcast of digital television signals frequently used in satellite transmission, interfacility links, and telephony communications. This standard transmits audio and video data over a single coaxial cable with a data rate of 270 Mbit/second.
In operation, video data streams are encapsulated into a set of ingress transport streams by the set of video sources 5, HD-SDI video signals by video application 7, or DVB-ASI video signals by video application 8 and sent to video TS/IP server 1. The set of ingress transport streams are ingested by the video TS/IP server where they are transformed to a set of egress transport streams. An egress transport stream can be sent to one of the set of video destinations 6 over IP network 2 as a TS over IP packet stream. In one embodiment, the egress transport stream can be conditioned to be sent over a HD-SDI connection to video application 7. In another embodiment, the egress transport system can be conditioned to be sent over a DVB-ASI connection to video application 8. An egress transport stream may also be addressed to one or more video destinations in the set of video destinations. Each transport stream of the set of egress transport streams is ingested by the video destination or application to which it is addressed.
The supporting components connected to master codec 53 are DDR2 memory 55 and Ethernet PHY 70 (Ethernet physical layer interface device).
The supporting components connected to slave codec 53 are DDR2 memory 56 and Ethernet PHY 71 (Ethernet physical layer interface device).
The supporting components connected to the primary FPGA are clock generator 59 to provide timing for entire video server block 50, and primary flash memory 58 for holding program instructions for the primary FPGA functions and for the master and slave codecs.
The supporting components connected to VioIP processor 60 are a random access memory (RAM) 64, flash memory 66, DDR2 memory 65, and Ethernet PHY 80. VioIP processor 60 further comprises a FPGA incorporating a processor core which is configured for packetization of video transport streams over internet protocol and requires the flash memory for booting, RAM 64 and DDR2 memory 65 for operations.
The supporting components connected to second VioIP processor 61 are a random access memory (RAM) 67, flash memory 69, DDR2 memory 68, and Ethernet PHY 81. VioIP processor 61 further comprises a FPGA incorporating a processor core which is configured for packetization of video transport streams over internet protocol and requires the flash memory for booting, RAM 67 and DDR2 memory 68 for operations.
The supporting components connected to video server block 50 and appropriately interconnected to all components described so far, are power supply 75 for powering all components and JTAG test CPLD 76 for operating a test console.
The Ethernet PHY devices, Ethernet PHY 80 and Ethernet PHY 81 are further terminated with RJ45 connectors to allow for connection to an external Ethernet network. Ethernet PHY 70 and Ethernet PHY 71 can be terminated with RJ45 connectors as required by the application depending upon whether they are tied to an internal or external Ethernet device or network, respectively.
The master and slave codecs are each preferably implemented as a single chip solution on a high speed data parallel DSP type processing capability that is optimized for video signal processing. In the preferred embodiment, a multi-core processor having at least a system CPU core for handling system level processing and I/O operations and a DSP co-processor core optimized for parallel video processing. A suitable codec processor is the STORM-1 processor part SP16HP-G220 from Stream Processors Inc capable of over 400 GOPS. A suitable component for the primary FPGA is a Virtex series FPGA from Xilinks Corporation. A suitable component for VioIP processors 60 and 61 is a Cyclone series FPGA from Altera which further includes the NIOS II processor core operating a video processing application. For example, see the reference design found at http://www.altera.com/support/refdesigns/sys-sol/broadcast/ref-video.html.
In the preferred embodiment, the primary FPGA is configured as shown in
AMPEGE data bus and BMPEGE data bus are connected as inputs to MUX 100. AMPEG_E data bus and BMPEG_E data bus are connected as outputs of MUX 100. MUX 100 can selectively shift data from one of either the AMPEGE or BMPEGE data buses to one of either the AMPEG_E or BMPEG_E data buses. MUX 100 is further capable to simultaneously shift data from the both inputs to both outputs as selected. In the preferred embodiment, AMPEGE, AMPEG_E, BMPEGE and BMPEG_E are 8 bit data buses and MUX 100 is configured as a 16 bit×16 bit cross-connect switch.
AMPEG_I data bus and BMPEG_I data bus are connected as inputs to MUX 110. AMPEGI data bus and BMPEGI data bus are connected as outputs of MUX 110. MUX 110 can selectively shift data from one of either the AMPEG_I or BMPEG_I data buses to one of either the AMPEGI or BMPEGI data buses. MUX 110 is further capable to simultaneously shift data from the both inputs to both outputs as selected. In the preferred embodiment, AMPEGI, AMPEG_I, BMPEGI and BMPEG_I are 8 bit data buses and MUX 110 is configured as a 16 bit×16 bit cross-connect switch.
Stream loader 130 can selectively load data from FLASH1, or pass data through from the AYUVE data bus, to the BYUVI data bus. Stream loader 131 can selectively load data from FLASH1, or pass data through from the BYUVE data bus, to the AYUVI data bus. MUX 132 can selectively receive an address from either of the first or second stream loaders and send the address to the FLASH1 address bus. Correspondingly, MUX 132 can selectively transmit data appearing on the FLASH1 data bus to either the first or second stream loaders. AYUVE, BYUVE, AYUVI and BYUVI are preferably 20 bit data buses to carry raw YUV video data; FLASH1 preferably includes a 32 bit address bus and 16 bit data bus connected to the primary flash memory; the stream loaders are preferably 32 bit address and 20 bit data devices; MUX 132 preferably comprises a set of 32 (1×2) data switches for address bits and 16 (1×2) data switches for data bits.
In the preferred embodiment, the primary FPGA is configured to pass through audio data from AI2S to BI2S and vice versa.
The master and slave codecs are configured with programmed instructions stored in the FPGA flash memory and accessed via AYUVI, BYUVI and FLASH1, the programmed instructions suitable to operate video TS/IP server codec functions. Codec software will be described in more detail below.
The video server block is a flexible hardware design that can be stacked and augmented with additional hardware components to arrive at exemplary embodiments of the video TS/IP server. In PC hosted embodiment, a video TS/IP server is conceived where video server block 50 is implemented on a PC card and hosted in a PC computer utilizing a native peripheral interface. A common peripheral interface found in PC computers is the Peripheral Component Interface (PCI) and the Peripheral Component Interconnect Express (PCIe). Recent PC computer architectures support the PCIe. The processor chosen to embody the codec components of the video server block preferably includes a built-in PCI or a built-in PCIe interface bus for interfacing to a host computer system.
Within a set of alternate embodiments, video TS/IP server is configured to ingress raw YUV video data streams from native video interfaces to perform encoding functions and placement into transport streams for IP transport. Within the set of alternate embodiments, the video TS/IP server is also configured to egress YUV video data streams decoded from TS over IP video streams. Within the set of alternate embodiments, the video TS/IP server is further configured as a stand-alone device, incorporating a set of video server blocks and a host processor in communication with the set of video server blocks for management and file I/O functions over Ethernet.
In a first alternate embodiment, video server block 50 of
In a second alternate embodiment, video server block 50 of
SDI receiver 92 is preferably a multirate SDI receiver which operates in one of at least two modes selected from an SMPTE mode and a DVB-ASI mode and is capable to de-embed audio-signals. In SMPTE mode, the SDI receiver can perform full SMPTE processing of an HD-SDI serial digital data stream. In DVB-ASI mode, 8b/10b decoding is applied to a received serial digital data stream.
Adaptive cable equalizer 91 is preferably a high-speed integrated circuit designed to equalize and restore signals received over 75Ω coaxial cable optimized for performance at 1.485 Gb/s and 2.97 Gb/s, and further designed to compensate for the DC content of SMPTE pathological test patterns.
A suitable device for SDI receiver 92 is part number GS2970 from Gennum Corporation. A suitable device for adaptive cable equalizer 91 is part number GS2974 also from Gennum Corporation.
In a third alternate embodiment, video server block 50 of
In a fourth alternate embodiment, video server block 50 of
In a fifth alternate embodiment, video server block 50 of
SDI transmitter 93 is capable to generate at least a SMPTE 292M or DVB-ASI compliant serial digital output signal. In SMPTE mode, the SDI transmitter can perform all SMPTE processing features. In DVB-ASI mode, the SDI transmitter will perform 8b/10b encoding prior to transmission on the 75-ohm coaxial cable. A suitable device for SDI transmitter 93 is part number GS2972 from Gennum Corporation.
In a sixth alternate embodiment, video server block 50 of
In a seventh alternate embodiment, video server block 50 of
In an eighth alternate embodiment, video server block 50 of
In the seventh and eighth embodiments, the audio signal converter can be a device suitable to convert any of the standard digital audio signal formats to and from I2S. Examples of standard digital audio formats for carrying raw audio data streams is the balanced AES/EBU, unbalanced AES/EBU and S/PDIF.
The alternate embodiments of
Host processer 156 is connected to flash memory 167, random access memory RAM 166 and a set of status indicators 33. Host processor block 156 is further connected to Ethernet switch block 154 and to FPGA video multiplexer 160. A pair of host Ethernet management ports 165a and 165b operated by the host processor is connected to a pair of physical RJ-45 ports, one port for file I/O, the other port for operational management of the video TS/IP server. The host processor is configured with programmed instructions stored in the flash memory and suitable to operate the video TS/IP server. Host processor software will be described in more detail below.
The dual redundant hot swappable power supply incorporates a power input connector 38 normally connected to AC power such as 110 V and a power button 34 for connecting and disconnecting power to the other components of the video TS/IP server.
QikStak block 145a has a pair of Ethernet interfaces 157a connected to Ethernet switch block 154 for management and file I/O, Ethernet interfaces 155a and 155b for sending and receiving video TS over IP, HD-SDI transmit port 151a and HD-SDI receive port 152a. QikStak block 145b has a pair of Ethernet interfaces 157b connected to Ethernet switch block 154 for management and file I/O, Ethernet interfaces 155c and 155d for sending and receiving video TS over IP, HD-SDI transmit port 151b and HD-SDI receive port 152b. QikStak block 145c has a pair of Ethernet interfaces 157c connected to Ethernet switch block 154 for management and file I/O, Ethernet interfaces 155e and 155f for sending and receiving video TS over IP, HD-SDI transmit port 151c and HD-SDI receive port 152c. QikStak block 145c has a pair of Ethernet interfaces 157d also connected to Ethernet switch block 154 for management and file I/O, Ethernet interfaces 155g and 155h for sending and receiving video TS over IP, HD-SDI transmit port 151d and HD-SDI receive port 152d. Ethernet interfaces 155a, 155b, . . . 155h are connected to a set of physical RJ-45 ports for connection to an external IP network. HD-SDI receive ports 151a-151d and HD-SDI transmit ports 152a-152d are connected to FPGA video multiplexer 160.
FPGA video multiplexer 160 has a set of input HD-SDI ports 162a-162d and a set of output HD-SDI ports 161a-161b connected to a set of coaxial cable connectors. The FPGA video multiplexer can be configured to switch any one port of the set of HD-SDI input ports to any one of HD-SDI receive ports 152a-152d. The FPGA video multiplexer can be further configured to dynamically switch any one of HD-SDI transmit ports 151a-151d to any one port of the set of HD-SDI output ports. In another alternate embodiment, raw YUV video data may be input or output over DVB-ASI I/O ports, instead of or in combination with HD-SDI input and output ports, the DVB-ASI I/O ports connected to the FPGA video multiplexer.
The software framework and programmable aspects of the present invention are explained with reference to
Software framework 200 executes under three operating systems, host OS 210, codec system OS 206 and codec DSP OS 216. The host OS operates on a host processor interacting with one or more video server blocks. The codec system OS and codec DSP OS operate on the master and slave codecs. In the preferred embodiment, the codec system OS is an embedded Linux operating system and the DSP OS is RTOS. If the host system is a PC as in the PC hosted embodiment, the host OS can be Microsoft® Windows™. If the host system is embedded as in the stand-alone embodiment a Linux operating system is preferred. Under these operating systems, software framework 200 comprises a set of modules that permit rapid adaptation to changing standards as well as customization to users' specific needs and requirements with a short development cycle.
Codec system OS 206 interfaces to a set of hardware drivers 203 and a set of hardware control APIs 205. A particularly important set of hardware drivers and hardware API for moving video and audio data into and out of the codec is direct memory access (DMA) functions 204. Codec system OS 206 utilizes system interfaces library 207 along with the communications and peripheral functions module 209 to handle the system work load. System interfaces library 207 contains library interfaces for functions such as video device drivers and audio device drivers while communications and peripheral functions module 209 contains functions such as device drivers for PCI bus and Ethernet interfaces, and panel control functions if required. Codec system OS 206 also handles the system function of servicing the host processor over the PCI bus or Ethernet interfaces in a hosted environment.
Codec DSP OS 216 handles the execution of DSP centric tasks and comprises DSP codec library 217, DSP intensive computation and data flow 218, and system scheduler 219. Examples of DSP centric tasks include codec algorithmic functions including encoding and decoding and video data streaming functions. System scheduler 219 manages thread and process execution between the codec system OS and the codec DSP OS.
The operating systems, libraries, drivers, functions and APIs of a codec are stored in persistent storage media attached to the codec. Persistent storage media is preferably flash memory, but can include hard drives, CD drives, USB flash devices and other forms of static memory. The operating systems, libraries, drivers, functions and APIs of the codec access RAM memory such as DDR memory attached to the codec and utilize the CPU and DSP cores of the codec to perform operations essential to the video TS/IP server.
Host OS 210 includes a set of hardware drivers 212 for communicating with the various components of a video server block including at least the primary FPGA, the master codec and the slave codec. Codec interface library 213 provides support for higher level system management functions 214 related to more complex codec specific configurations, for example, choosing the encoder and decoder compression algorithms. User interface API 215 is provided. Preferably, user interface API 215 includes a web hosting application for configuration, suitable for access over an IP network using a web browser. User interface API 215 may also include standard TCP/IP protocol functions such as a file transfer protocol (FTP) application suitable for moving files into and out of the host, and for updating the software in the host and on the codecs.
The host OS, libraries, drivers, functions and APIs of the host are stored in persistent storage media attached to the host. Persistent storage media include flash memory, hard drives, CD drives, USB flash devices. The host OS, libraries, drivers, functions and APIs of the host access RAM memory within the host and utilize the CPU of the host to perform operations essential to the video TS/IP server.
User interface API 215 and system management functions 214 are further described with the help of
The codec portion of software framework 200 is organized into a set of modular components operating as a codec system.
Within system OS components 225 are process manager 251, codec manager 261, PCI manager 271 and Ethernet manager 281. Process manager 251 controls all the states and processes of the codec including the codec manager, the PCI manager, and the Ethernet manager via communications links 260 shown in dashed lines. Ethernet manager 281 controls all communications between the codec and Ethernet network 221. PCI manager 271 controls all communications including PCI interrupt generation and handling between the codec and host PCI bus 222.
Within hardware driven components 226 are a video device driver, V4L2 252, audio device driver, I2S 262, VioIP driver 272 and DMA controller 282. V4L2 252 is implemented as the standard Linux V4L2 video driver and I2S 262 is implemented as the standard Linux I2S audio driver. The video device driver manages video data input and output requirements of the codec as required by configuration, operating on video output buffers to create egress video elementary streams and operating on ingress video streams to store video elementary streams in video input buffers. Similarly, the audio device driver manages audio data input and output to and from audio data buffers. The video and audio device drivers utilize DMA controller 282 to move data to and from external data buses 292 and DDR memory 291. Process manager 251 utilizes VioIP driver 272 to control external VioIP engine 290 attached to the codec.
101051 Shared memory components 227 manage data held within DDR memory 291. Shared memory components 227 include the methods: video_capture 253, video_output 263, audio_capture 273, audio_output 283, ENC 254 and DEC 264. The video_capture and video_output methods operate on video stream data including metadata contained in suitable data buffers existing within the DDR memory. The audio_capture and audio_output methods operate on audio stream data including metadata contained in suitable data buffers existing within the DDR memory. Video_capture, video_output, audio_capture and audio_output methods can be accessed by the system OS components and the DSP OS components, and these methods can access the hardware driven components. For example, the video_output object method can utilize the V4L2 video device driver along with the DMA controller to stream video data to a data bus, such as the AMPEG_E, from a video buffer in the DDR memory. Codec manager 261 controls video-capture 253, video_output 263, audio_capture 273, audio_output 283 and ENC 254. Data flows between V4L2 252 and video_capture 253 and between V4L2 252 and video_output 263. Data also flows between I2S 262 and audio_capture 273 and between I2S 262 and audio_output 283.
DSP OS components comprise the ENC 254 and DEC 264 and further comprise DSP codec library 217 of encoder and decoder algorithms optimized for DSP processing unit, DPU 293.
ENC 254 encapsulates a set of video and audio processing methods for encoding a first data buffer containing raw YUV video and audio data into a second data buffer containing MPEG2 and H.264 video data. ENC 254 can utilize the configurable compression algorithms in the DSP codec library. DEC 264 encapsulates a set of video and audio processing methods for decoding a third data buffer containing MPEG2 and H.264 video data streams video data streams into a fourth data buffer containing raw YUV video and audio data streams. DEC 264 can utilize the configurable compression algorithms in the DSP codec library. First, second, third and fourth data buffers can be accessed by the video_capture, video_output, audio_capture and audio_output methods.
In operation, the codec system follows a sequence of operational states according to state diagram 350 of
The codec system starts from initialization state 355 while booting and does not require a host system interaction. However, the codec system may be put into this state by sending a message from the host system to the process manager. During the initialization state the codec system boots, loading program instructions and operational data from flash memory. Once initialization is complete, the codec system transitions automatically to idle state 360, wherein the codec system is operational and ready for host communication. The process manager keeps the codec system in idle state 360 until a “start encode” or “start decode” command is received from the PCI or Ethernet manager. From idle state 360, the codec system may transition to either encode standby state 365 or decode standby state 380 depending upon the operational mode of the codec system being configured to encode or decode, respectively, according to the current configuration.
Upon entering encode standby state 365, the codec system loads an encoder algorithm and is ready to begin encoding immediately upon receiving a “start encode” command from the host system via the PCI or Ethernet manager. When the “start encode” command is received by the codec manager, the codec system transitions from encode standby state 365 to encode running state 370. Encode standby state 365 may also transition to configuration update state 375 or to shutdown state 390 upon a configuration change request or a shutdown request from the host system, respectively. One other possible transition from encode standby state 365 is to maintenance state 395.
Encode running state 370 is a state in which the codec system, specifically the DSP OS and DPU, is actively encoding video and audio data. The only allowed transition from encode running state 370 is to encode standby state 365.
When entering decode standby state 380, the codec system loads a decoder algorithm and is ready to begin decoding immediately upon receiving a “start decode” command from the host system via the PCI or Ethernet manager. When the “start decode” command is received by the codec manager, the codec system transitions from decode standby state 380 to decode running state 385. Decode standby state 380 may also transition to configuration update state 375 or to shutdown state 390 upon a configuration change request or a shutdown request, respectively, from the host system. One other possible transition from decode standby state 380 is to maintenance state 395.
Decode running state 385 is a state in which the codec system, specifically the DSP OS and DPU, is actively decoding video and audio data. The only allowed transition from decode running state 385 is to decode standby state 380.
In configuration update state 375, a new configuration set is selected to be the current configuration or the current configuration is altered via the PCI or Ethernet manager. The only allowed transitions from the configuration update is to encode standby state 365 or decode standby state 380, depending upon the configuration setting.
Transitions to maintenance state 395 only arrive from encode standby state 365 or decode standby state 380 when a major codec system issue fix or a software update is required. The software update process is managed by the PCI or Ethernet manager. The only possible transition from maintenance state 395 is to initialization state 355.
Transitions to shutdown arrive from encode state 365 or decode standby state 380 upon a power down request from PCI or Ethernet manager, wherein the codec system promptly powers down.
Referring to flow charts in
In a preferred embodiment, the master and slave codecs become independent devices after the boot process completes, independently receiving and transmitting data to/from management interfaces and processing video to/from video I/O ports. In TS/IP only configurations, video decoded by one codec is encoded by the other codec, the video stream being transmitted from one codec to the other via the AYUVE, AYUVI, BYUVE and BYUVI data buses.
At step 530, the decoder function is configured. Possible decoder configurations in the preferred embodiment include transport stream formats: MPEG2, H.264, DV25 and DVC-pro/25/50/100. Video resolution, compression level and bit rate are inherent to the ingress transport stream and therefore autodetected.
At step 532, the output port is defined for the egress video data stream as either a TS/IP assigned to an Ethernet port or as a raw YUV video output. At step 534, if raw YUV video output is defined then step 535 is executed, otherwise step 537 is executed. At step 534, if a video multiplexer is available to select from multiple output HD-SDI output ports, then a selection defining the HD-SDI output port is made.
At step 537, the encoder function is configured. Possible encoder configurations in the preferred embodiment include transport stream formats: MPEG2, H.264, DV25 and DVC-pro/25/50/100. Video resolution, compression level and bit rate are also selected as a part of the encoder configuration. Selecting a transport stream format for the encoder function that is different from the decoder function results in an overall transcoding process. Selecting a compression level or a bit-rate for the encoder function that is different from the compression level or bit-rate for the decoder function results in an overall transrating process.
In step 538, the hardware components are set on the selected video server block to match the configuration. Each codec configures itself appropriately as an encoder or decoder and the host processor configures the primary FPGA to accomplish the correct data flow, for example, enabling a multiplexer to connect the AMPEG_I data bus to the AMPEGI data bus.
Note that the steps 530 and 537 illustrate a transrating capability for the video TS/IP server in addition to a transcoding capability. In transrating operations, the output video stream bit-rate can be configured to be different than the input video stream bit-rate. Transrating typically involves selecting a different compression algorithm for the output video stream than what is used for the input video stream. In transcoding operations, the output video stream resolution or format is different from the input video stream resolution or format. For example, capturing an MPEG2 video stream and sending out an H.264 video stream is a transrating operation.
The method continues at step 542, where if raw YUV video was selected for input, then step 559 is performed, otherwise a decoder function is required and step 553 is performed wherein the DMA controller of a second codec is enabled to stream ingress and egress of video and audio data to/from ingress and egress data buffers, respectively. At step 554, the codec manager of the second codec executes a command to the video_capture method to read video data from the ingress data buffer. At step 555, the codec manager of the second codec executes a command to the DEC method to load and run a decoder process on the ingress data buffer. At step 556, the decoder process operates on data in the ingress data buffer and stores the result in the egress data buffer. At step 558, the codec manager configures a second VioIP engine to process transport streams received over Ethernet from the IP network to audio/video (A/V) data for decoding by the second codec.
In step 559, video streams are captured from input ports (Ethernet or HD-SDI) and output via output ports (Ethernet or HD-SDI) according to the configurations. The sequence diagrams of
Beginning with
A video server block can be optionally configured so that the uncompressed video and audio data is encoded and transmitted out as TS/IP to an IP network and simultaneously transmitted out over HD-SDI. This option allows for monitoring of an A/V transport streams as it passes through the video TS/IP server.
In
If the video server block is configured to receive uncompressed YUV video from an external source, the uncompressed video and audio data is transferred from an SDI receiver to AYUVE data bus 567 at point 600 and AI2S data link at point 608, respectively. The SDI receiver can be configured to receive either an HD-SDI or DVB-ASI signal.
Continuing with
In
If the video server block is configured to receive uncompressed YUV video from an external source, the uncompressed video and audio data is transferred from an SDI receiver to BYUVE data bus 767 at point 800 and BI2S data link at point 808, respectively. The SDI receiver can be configured to receive either an HD-SDI or DVB-ASI signal.
The foregoing descriptions of configuring and processing a video stream for a video server block, including encoding and decoding functions, is repeated for each video server block contained within a video TS/IP server. For the stand-alone embodiment of the video TS/IP server, for example, four configurations and four video streams are processed simultaneously having an input and output defined for each video stream. The foregoing descriptions are also not intended to limit the number of video streams processed simultaneously by a video server block. It is expected that the DSP processing and DMA capability of future codec devices will support simultaneous processing of multiple HD-SDI, MPEG2, H.264, DV25 and DVC-pro/25/50/100 based transport streams. It is within the capability of the existing hardware to process multiple MPEG2 transport streams. It is therefore within the scope of the present invention, with the video server hardware embodiments as described, to simultaneously process multiple video streams through a single video server block.
Other embodiments may be conceived, for example, for current and future studio quality video formats which may include 3-D image and video content of current and future consumer formats. Still other embodiments may be conceived, that combine larger numbers of video server blocks together to provide a higher capacity video TS/IP server. Another embodiment may be conceived whereby a stand-alone video TS/IP server incorporates a single video server block with video I/O over Ethernet ports only. Another embodiment may be conceived whereby multiple PCI hosted video TS/IP servers are installed into a host computer server and wherein the host computer software provides a management function for all installed video TS/IP servers. The specifications and description described herein are not intended to limit the invention, but to simply show a set of embodiments in which the invention may be realized.
This application claims priority to U.S. Provisional Application No. 61/280,429 filed Nov. 4, 2009.
| Number | Date | Country | |
|---|---|---|---|
| 61280429 | Nov 2009 | US |
