Patent Application
20090254683
The present invention relates to the field of video processing systems, and more particularly, to video multiviewers and related methods.
As broadcasters continue the transition from analog television to digital video television, the television production process is increasingly conducted in an all-digital domain, that is, from the initial camera shot to the display in the consumer's living room. This move to digital technology permits broadcasters to simultaneously broadcast multiple video streams using a single connection. Indeed, for popular live events, broadcasters typically deploy mobile broadcast units to route and manipulate, i.e., producing the numerous video streams that come from respective cameras throughout the event before being transmitted.
An approach to manipulating and monitoring the video streams is a multiviewer, which typically includes a monitor and associated processor receiving the high resolution, digital video streams. The processor scales the video stream to fit all the video streams onto a single display. Some multiviewers use a plurality of monitors, thereby permitting the viewing of even more video streams. A potential drawback to the typical multiviewer is the difficulty in rearranging the video streams on the monitor in real time. For example, a user viewing the multiviewer monitor displaying four video streams split equally over quarters of the monitor may desire to expand a first video stream and correspondingly reduce the other video streams. This operation may cause the processor to adjust scaling operations in real time based upon requests from the user. More specifically, to provide advanced features to the user, the typical multiviewer may have to include significant hardware to provide adequate processing power, thereby possibly increasing the form factor and housing size to undesirable levels.
Some exemplary multiviewer systems include the DX series and Centrio™ Multiviewers from present assignee Harris Corporation. The DX series Multiviewers delivers modular configurations for 4, 8, 12 or 16 SDI or composite inputs typically using different input modules, with auto detect of NTSC, PAL or SDI formats. Composite, component, SDI and line doubled VGA outputs provide customers with a wide range of flexible display options. Front panel controls and remote control via RS-232/422 provide the user with the ability to view real-time multiple images in various pre-configured displays. Optional software allows the user configuration of displays, and multi-system control, while options also exist for in-picture audio monitoring of SDI embedded audio. Standard alarm features include loss of video sync, black picture and frozen picture detection.
Another exemplary multiviewer system is the 7767VIP4 signal monitoring module from the Evertz corporation of Burlington, Ontario, Canada. The module simultaneously accepts, auto-detects, analyzes and displays four synchronous or asynchronous HD/SD/Analog video signals. An additional fifth computer graphic input may be used for display of a dynamic background image. A similar multiviewer system is disclosed in U.S. Pat. No. 7,023,488 to Szybiak et al.
Despite the advantages of such multiviewer systems, further enhancements may be desirable in some applications. For example, system memory constraints often impose restrictions on the number of video inputs that may be simultaneously displayed. Additionally, the ability of such systems to display added graphic content (e.g., backgrounds, indicators, etc.) along with the video inputs on the display may be encumbered by the requirement for separate inputs for such content, or for performing offline modifications to video inputs or display templates, for example.
One popular multiviewer system is the Centrio™ multiviewer as manufactured by Harris Corporation. This multiviewer includes a video input module that in an early version supported eight DMA channels. The configuration/state registers were implemented fully in Field Programmable Gate Array (FPGA) flip flops and associated circuit logic. Different performance requirements, however, necessitated over 20 DMA channels. Future performance requirements necessitated scaling this larger number of DMA channels to over 1,000 DMA channels supported in a single Field Programmable Gate Array. Field Programmable Gate Arrays, however, typically do not incorporate and support sufficient logic and flip flops to support such a vast number of DMA channel configuration/state registers, where such large logic would run slow for practical implementations. It is therefore desirable to enhance this type of system design and achieve an uninterrupted DMA flow through a larger number of DMA channels, such as over a thousand or more DMA channels.
A video multiviewer system includes a Graphics Processing Unit (GPU) that includes a GPU memory. A video input module is operative with the GPU for receiving video data and transferring the video data to the GPU memory via a Direct Memory Access (DMA). A programmable circuit such as a Field Programmable Gate Array (FPGA) includes a multi-ported and in one aspect a dual ported block Random Access Memory (RAM) configured for a plurality of DMA channels for receiving video data and allowing uninterrupted operation of consecutive DMA transfers of video data to the GPU memory. A display displays the multiple video windows based upon video data received within the GPU memory.
The system includes a DMA controller. The dual ported Block RAM includes first and second ports. The first port is operative for accessing video data based on independent read/write access and the second port is operative for accessing video data based on simultaneous read/write access by the DMA controller. The system also includes a read/write controller and plurality of registers. The DMA controller operates on DMA channel “n” and is operative for pre-reading registers for DMA channel “n+1.” The DMA controller can change state variables in a register set when data is transferred within DMA channel “n.” The DMA controller is also operative for modifying data within registers and passing the data back to the Read/write controller to be stored back within the Block RAM.
This Block RAM is scalable as to the number of DMA channels required for passing data to the GPU memory. A video data controller is operative with the video input module for assigning addresses to be used for video data within the GPU memory.
A method aspect is also set forth.
Other objects, features and advantages of the present invention will become apparent from the detailed description of the invention which follows, when considered in light of the accompanying drawings in which:
Different embodiments will now be described more fully hereinafter with reference to the accompanying drawings, in which preferred embodiments are shown. Many different forms can be set forth and described embodiments should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope to those skilled in the art. Like numbers refer to like elements throughout.
In this prior art implementation shown in
In accordance with a non-limiting example of the present invention, most of this excess logic using FPGA flip flops is replaced using a multi-ported and in one aspect a dual ported Block RAM. As is known, RAM implementations typically permit access to a limited number of bits at a time, making Block RAM typically not applicable to general register files. In case of an application where only a subset of registers needs to be used at a time, such as with multi-channel DMA applications, it becomes possible to build a highly scalable solution using an FPGA Block RAM. It should be understood that different programmable circuits besides FPGA's can be used, including different types of ASIC's and other processors using Block RAM.
A high-level diagram of a DMA configured Block RAM system 50 as part of a video input module of a video multiviewer system having DMA registers in Block RAM is shown in
An uninterrupted DMA video or other data for a video multiviewer is achieved through a large number of DMA channels while using Block RAM, which effectively has only a single read/write port.
The DMA register access logic is generally shown in this block diagram of
This FPGA Block RAM solution as described allows for scalability that is only limited by the size of available Block RAM memory in the FPGA. It can easily be scaled up to thousands of DMA channels without noticeable impact on speed and requires typically only additional storage space in Block RAM. A sample implementation can support 2,000 DMA channels, while utilizing a relatively small portion of the FPGA.
The flexible DMA configuration using FPGA Block RAM as described is operable with the Graphic Processing Units (GPU) used in video multiviewer systems in which DMA signals are sent to texture memory or other Graphics Processing Units. These frames of video data are typically split into many blocks and sent as separate blocks over separate DMA channels into different memory locations for a video multiviewer system that has extensive scaling in one non-limiting embodiment. Different Field Programmable Gate Arrays could be used for this system, including FPGA's manufactured by Xylinx and Altera that use dual port Block RAMs. Thus, the system dual ports a Read/Write Memory Interface 54 and interacts with the DMA controller 66 through a Read/Write Memory Interface 62 as shown in
Although a number of DMA channels could be implemented using prior art configuration registers and flip flops such as described relative to
From a design point of view, when using a minimum number of DMA channels, it is easier to implement a circuit for such minimum number of DMA channels in regular flip flops for speed and efficiency. When greater than 20 and 50 DMA channels are required, however, and expansion is necessary, the use of regular flip flops is not advantageous. When a large amount of video data for a multiviewer must be brought into the system, the greater number of DMA channels are required as described in accordance with a non-limiting example of the present invention. This system also allows greater ability to scale.
For example, a DMA burst could be a block of 512 bytes for one DMA channel. The Read/write Controller 74 reads a corresponding configuration in the registers and processes data for the Pre-Read Registers 76 and passes data onto the DMA controller 66, which operates on the registers such as on regular flip flops in function. Thus, the system increments the last written address and how much data has been sent across. When it returns, the system knows it has written the 512 bytes and is done sending those 512 bytes. It passes the resulting and updated information to any registers as status registers and back to the Write Back Registers 80. For example, if there is a list of a number of different DMA data bursts coming through during the first DMA, the Read/Write Controller 74 can pre-fetch the configuration information from the next DMA and the DMA controller 66 does not have to wait for the configuration for the second DMA burst to be read out since it is read out already and now ready to be used. Once the DMA controller 66 has processed the second DMA burst, the configuration in the status registers for the first DMA burst is in the Write Back registers 80 and written back into the memory by the Read/Write Controller 74. Thus, the system can achieve an uninterrupted DMA data flow through a large number of DMA channels while using effectively only a single read/write port as described.
It should be understood that it is possible to use the system as described further with different video multiviewer systems as described below.
As shown in
One technology that is currently used for video rendering is Microsoft DirectShow, part of the DirectX family of API applications. DirectShow allows the transfer of video streams to video memory, but it requires CPU resources to manage the transfers. Typical prior art techniques use an input/output (I/O) controller to copy video stream data to system memory from the hardware device. The CPU copies the video stream data to the display device. This requires two separate DMA transfer operations. For relatively high bandwidth video streams, this may pose a significant limiting factor in a video system's capabilities, since DMA bandwidth has a fixed limit.
As mentioned above, the system 130 illustratively includes a video I/O controller 131, a system memory 132, and a Graphics Processing Unit (GPU) 133 including a GPU memory 134 in which video data is input using a video input module that includes the flexible DMA configuration and FPGA Block RAM as described before. By way of example, the GPU memory 134 may include one or more texture buffers, which may be associated with respective video input streams or feeds, as will be appreciated by those skilled in the art. The system 130 also illustratively includes a central processing unit (CPU) 135. In the illustrated example, access between the video I/O controller 131, GPU 133, CPU 135, and system memory 132 is provided via a system bus 136, as will also be appreciated by those skilled in the art. The GPU is operable with a display 138. The video input module 140 receives data along the bus 136 and can be individual input modules that are used to DMA data into the GPU. PCI Express can be used as one non-limiting implementation.
The video I/O controller 131 assigns addresses to be used for video data in the GPU memory 133 (i.e., texture buffers). By way of comparison, using the DirectShow application discussed above, in a typical prior art arrangement the texture buffers would temporarily be locked while textures are loaded from the video I/O controller to the texture buffers via the system memory 132. Drawing or rendering operations would not be permitted during such locked periods. Yet, the constant locking and unlocking of the texture buffers requires additional overhead, and therefore delays memory transfers. Moreover, this can also result in a video I/O controller changing destination addresses in the GPU memory 134, which may result in a still further overhead penalty if the video I/O controller 131 has to be reset or re-programmed to desired texture buffer addresses.
The CPU 135 advantageously operates the video I/O controller 131 to transfer video data to the GPU memory 134 via direct memory access (DMA) without using the system memory 132. Accordingly, the system may advantageously increase the amount of video data transferred to the GPU 133 within a given amount of time, and thereby allow more video inputs to be displayed on the display than would otherwise be possible if the same video had to be routed through the system memory.
The CPU may cooperate with the I/O controller 131 to lock the assigned addresses. By way of example, the GPU memory 133 may include at least one texture buffer 134 in which the CPU locks the assigned addresses. Furthermore, the video I/O controller may receive a plurality of video input streams, and the at least one texture buffer may include a respective texture buffer for each video input stream. Moreover, the CPU may operate the video I/O controller 131 and the GPU 133 to transfer video data in real time. Also, the video multiviewer system may further include at least one data communications bus coupled to the video I/O controller 131, the system memory 132, the GPU 133 and the CPU 135.
By way of example, the video data may include video frame data. Furthermore, the CPU may use at least one Microsoft DirectX application programming interface (API) for operating the video I/O controller 131 and the GPU 133, for example. By locking the texture buffer of the Direct3D texture, the physical memory can address to allow the hardware I/O controller 131 to DMA transfer video stream data from the hardware device directly to the texture buffer. This technique does not require an intermediate host memory buffer for the video stream data thus saving this memory for other uses.
This technique can rely on the fact that the GPU device driver does not change the physical memory address of the texture buffer and it remains available for the duration of the transfer. The system exploits an undocumented and unsupported behavior of the GPU device driver and the Microsoft DirectX run time. This allows a Centrio™ multiviewer as described above to make more efficient use of the hardware platform, thus providing more functionality with less hardware than competitive products. Since video frames are passed via DMA transfer once instead of twice, it effectively doubles the number of video streams the system could transfer otherwise.
Further examples of a video multiviewer system that is adapted for enhanced scaling and can be modified for use with the flexible DMA configuration using the FPGA Block RAM described relative to
Referring initially to
The multiviewer system 220 illustratively includes a processing unit 227 coupled downstream from the video scalers 22la-221d for generating additionally scaled video streams (video output streams) by performing additional video scaling on the initially scaled video streams 225a-225d, and a display 230 cooperating with the processing unit for displaying multiple video windows based upon the additionally scaled video streams. Although illustrated as a single screen display, the display 230 may comprise a plurality of screens, for example, 94 displays. Advantageously, the video multiviewer system 220 may scale the video input streams 224a-224d more efficiently by distributing the computationally intensive process of scaling the video input streams.
More particularly, each of the video scalers 221a-221d illustratively performs video scaling in only one dimension, for example, in a horizontal dimension of video frames. The processing unit 227 illustratively performs video scaling in a vertical dimension of video frames to complete the scaling of the video input streams 224a-224d. Advantageously, the computationally intensive vertical scaling, which may use large amounts of storage and logic resources, may be performed by the processing unit 227. As will be appreciated by those skilled in the art, other distributions of the scaling processes may be implemented. In general, less intensive scaling processes should be allocated to the video scalers 221a-221d while the more complex scaling processes should be allocated to the processing unit 227.
Additionally, each of the video scalers 221a-221d may comprise a hardware implemented video scaler. In other words, the processing power of the video scalers 221a-221d may be limited and static. Advantageously, the video multiviewer system 220 allocates the horizontal scaling processes, which use fewer resources than the vertical scaling processes, to the video scalers 221a-221d. Moreover, each of the video scalers 221a-221d may perform video scaling as a selectable power of 2, for example, 1, 2, 4, 8, and so forth, further reducing computational intensity demands on the video scalers. Advantageously, since the computational demand on the hardware implemented video scalers 221a-221d is limited, the size of the packaging and housing used for the video scalers is reduced. For example, a single field-programmable gate array (FPGA) may be used to implement the video scalers 221a-221d to prescale 8 3G-SDI video input streams.
The video scalers 221a-221d and the processing unit 227 illustratively cooperate to process data other than video stream data using distributed processing, for example, metadata extraction and audio ballistics metering. More specifically, the video scalers 221a-221d may also perform bit data extraction, thereby advantageously reducing bandwidth passed on to the processing unit 227. The processing unit 227 may perform data decoding and interpretation based upon the bit data extraction. Furthermore, to reduce the computational payload of a Central Processing Unit (CPU) 223 for audio processing, the video scalers 221a-221d may calculate raw ballistics values while the processing unit 227 cooperates to interpret the data and render appropriate audio amplitudes and phase meters.
The processing unit 227 is preferably formed as a Graphics Processing Unit (GPU) including a GPU processor and GPU memory coupled thereto as noted relative to
The video multiviewer system 220 illustratively includes a Direct Memory Access (DMA) engine 222 coupled between the video scalers 221a-221d and the processing unit (GPU) 227. Also, a second FPGA, in addition to the FPGA that may implement the video scalers 221a-221d, may be used to implement the DMA engine 222.
The video multiviewer system 220 illustratively includes a data communications bus 226 coupled between the DMA engine 222 and the processing unit (GPU) 227, and the CPU 223 coupled to the data communications bus. Advantageously, after the video input streams 224a-224d are prescaled, the DMA engine 222 “DMAs” the initially scaled video streams 225a-225d into the processing unit 227 for final scaling.
As will be appreciated by those skilled in the art, the data communications bus 226 has an associated bandwidth and corresponding data throughput that may limit processing for scaling in the processing unit 227. Advantageously, in the video multiviewer system 220, since the scaling is distributed between the video scalers 221a-221d and the processing unit 227, the bandwidth bottleneck of the data communication bus 226 is less likely to be exceeded. Indeed, the distributed scaling of the video multiviewer system 220 may allow for sufficiently offloading the processing unit 227 so that it handles the remaining scaling work, and the distributed scaling sufficiently reduces the DMA engine 222 bandwidth to “DMA” up to 64 video streams over modern local bus architectures, such as, PCI Express (1st generation) and Hyper Transport.
Advantageously, since the video multiviewer system 220 consumes limited physical space, the system may be installed into a Platinum multiviewer, as available from the Harris Corporation of Melbourne, Florida (Harris Corp.), the assignee of the present application, or a router. Additionally, the video multiviewer system 220 may control routing of the video input streams 224a-224d. More specifically, the video multiviewer system 220 may access any of the router inputs, for example, that is all 512 inputs in a 28RU Platinum router, as available from the Harris Corp.
Although illustrated with a single plurality of video scalers 221a-221d and a corresponding processing unit 227, data communications bus 226, CPU 223, and DMA engine 222 (all together hereinafter referenced as a “set”), in other embodiments, the video multiviewer system 220 may additionally include multiple sets, all of which may be installed in the 28RU Platinum router, for example.
The video multiviewer system 220 may selectively choose which set a video input stream routes to. Thereby, the video multiviewer system 220 may actively balance internal resource utilization between sets, and thus makes it possible to get better performance out of the same hardware.
Moreover, if a user of the video multiviewer system 220 chooses to display the same video input streams 224a-224d in two picture-in-pictures (PIPs) of different size, the video multiviewer system can route it to two of its inputs, and apply different pre-scaling ratios to yield the best quality picture for both PIPs.
Advantageously, the number of video input streams 224a-224d received by the video multiviewer system 220 is scalable. In particular, the number of inputs may be scalable based on the scalability of the 28RU Platinum Router. Additionally, the number of outputs can be scaled up by inserting additional “sets” into the Platinum router frame, and each set may work independently of each other while having no limiting effect on other sets in the frame.
Since all sets in a frame have access to the same inputs, a user may readily build a video multiviewer system 220 that spans a single video frame across two or more displays 230 driven by two or more sets, each set routing the same video stream to its input and displaying the corresponding portion of the video frame. This may allow for spanning a single video frame across a wall of displays.
Referring to
Each Centrio module 273a-272b illustratively includes a plurality of video input modules 274a-274d feeding into a local data bus 275 cooperating with a CPU 279, and a GPU 276. The local data bus 275 may comprise, for example, a PCI Express (1st generation) data bus or a Hyper Transport data bus. The GPU 276 illustratively includes a data bus 277, a Random Access Memory module 281 cooperating with the data bus, and a scaler/rendering engine 280 cooperating with the data bus. The GPU 276 outputs to the input-output module 282 including a Digital Visual Interface (DVI) to SDI converter 283.
The video multiviewer system 290 shown in
Each PPIM 291a-291n illustratively includes respective equalizers 296a-296h for each video input stream, and a prescaler module 294 including a deembedder 298 cooperating with a prescaler multiplexer 293. The deembedder 298 also includes an optional Dolby decoder 295. The outputs of the PPIMs 291a-291n are illustratively received by the Platinum cross-point matrix 297 and a Platinum TDM cross-point module 298, both as will be available from the Harris Corp. The outputs of the Platinum cross-point matrix 297 and the Platinum TDM cross-point module 298 are received by a plurality of multiviewer modules 300a-300n. Each multiviewer module 300a-300n illustratively includes a converter 301, a scaler module 302 receiving the output of the converter, and a DVI card (GPU) 303 receiving the output of the scaler module.
The communication link between the PPIMs 291a-291n and the multiviewer modules 300a-300n may have a payload bandwidth divided into 8 equal parts, each allocated to one of the 8 video input streams (channels) on a given PPIM. After reserving space for a given channel's pre-processed ancillary space and audio ballistics information, the remaining bandwidth may be allocated for the video stream. The video pre-scaling ratio may be hardcoded on a per-video standard basis, and stored in a table for hardware automatic lookup. The pre-scaling ratio may be selected to result in the highest bandwidth that fits in the allocated payload bandwidth.
The pre-processing parameters may be hardcoded at design or system integration phase and may not be dynamically changed. The packet size in the communications link may match the packet size used by multiviewer modules 300a-300n local bus and DMA engine, thus minimizing logic required to handle the pre-processed data.
Advantageously, the communications links in the video multiviewer system 290 follow the same profile. Any communication link may be used by the multiviewer modules 300a-300n without these modules interfering with each others operation. A single communication link may carry all 8 channels from a given input module, i.e., a multiviewer module 300a-300n may have full simultaneous access to up to 32 PPIMs 291a-291n, permitting greater access.
Referring now to
The outputs 314 are received by a plurality of second 28RU Platinum router frames 316a-136d, each also including a plurality of inputs 317, a cross-point switcher 318 coupled thereto, and a plurality of outputs 319 upstream of the cross-point switcher. The video multiviewer system 310 illustratively receives 2048 SD/GD/3G-SDI video input streams and outputs 128 DVI outputs, or alternatively 256 HD-SDI outputs. As will be appreciated by those skilled in the art, the video multiviewer system 310 may be scaled to have more or less inputs and outputs.
Many modifications and other embodiments of the invention will come to the mind of one skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is understood that the invention is not to be limited to the specific embodiments disclosed, and that modifications and embodiments are intended to be included within the scope of the appended claims.
