Patent Application
20080117965
This invention relates to video communications and signal processing, and more specifically to the compression, decompression, transcoding, and/or combining of audio and/or video signals among various digital and/or analog formats.
The invention comprises an environment for integrating a collection of video and audio compression and decompression engines into a system ideally suited for a common electronic circuit board or yet more compact subsystem. These compression and decompression engines, which will be called “media processors,” may be autonomous, operate under external control, be managed by a separate common chaperoning processor, or combinations of each of these.
The chaperoning processor may divide session management, resource allocation, and housekeeping tasks among itself, the media processors, and any external processing elements in various ways, or may be configured to operate in a completely autonomous and self-contained manner.
The resulting configuration may be used as an analog/digital codec bank, codec pool, fixed or variable format transcoder or transcoder pool, continuous presence multimedia control unit (MCU), network video broadcast source, video storage transcoding, as well as other functions in single or multiple simultaneous signal formats.
One aspect of the invention provides for flexible environments where a plurality of reconfigurable media signal processors cooperatively coexist so as to support a variety of concurrent tasks.
In a related aspect of the invention, several independent codec sessions can be supported simultaneously.
In another aspect of the invention, the reconfigurable media signal processors include abilities to cooperatively interwork with each other.
In another related aspect of the invention, flexibly reconfigurable transcoding is provided for signals conforming to one compression standard to be converted to and from that of another compression standard.
In another aspect of the invention, encoder/decoder pair software is unbundled into separately executable parts which can be allocated and operate independently.
In another aspect of the invention, resource availability is increased for cases when signal flow is unidirectional by not executing unneeded portions of bidirectional compression algorithms.
In another aspect of the invention, a common incoming signal can be converted into a plurality of outgoing signals conforming to differing compression standards.
In another aspect of the invention, the system can provide needed functions involved in implementing a video conferencing MCU supporting a variety of analog and digital signal formats.
In another aspect of the invention, the system can provide functions involved in implementing a streaming transcoding video storage playback system, supporting a variety of analog and digital signal formats.
In a related aspect of the invention, the system can implement a streaming transcoding video storage system broadcasting video conforming to a variety of analog and digital signal formats.
In another related aspect of the invention, the system can implement a streaming transcoding video storage system simultaneously broadcasting a plurality of video signals, each conforming to selected plurality of differing video signal formats.
In another aspect of the invention, the system can provide functions involved in implementing a streaming transcoding video storage system in record modes, and in this receiving video and audio in any of a variety of analog and digital signal formats.
In another related aspect of the invention, the system can implement a video call the recording of a video call.
In another related aspect of the invention, the system can implement the recording of a video conference.
In another related aspect of the invention, the system can implement a recording function of a video answering system.
In another related aspect of the invention, the system can implement a playback function of a video answering system.
In another aspect of the invention, the system can be reconfigured in response to on-demand service requests.
In another aspect of the invention, the system software includes modularization of lower level tasks in such a way that facilitates efficient reconfiguration on demand.
In another aspect of the invention, the system software is structured so that some tasks may be flexibly allocated between local controlling processor and a media processor.
In another aspect of the invention, the system grows gracefully in supporting a larger number of co-executing tasks as software algorithms become more efficient.
In another aspect of the invention, the system provides important architectural continuity as future reconfigurable processors become more powerful.
In another related aspect of the invention, the system can implemented with standard signal connectors rather than bus-based I/O connections so as to provide stand-alone implementation without physical installation in a host system chassis.
The above and other aspects, features and advantages of the present invention will become more apparent upon consideration of the following description of exemplary and preferred embodiments taken in conjunction with the accompanying drawing figures.
a illustrates a basic configuration involving a number of analog-to-digital and digital-to-analog elements and a number of encoder/decoder elements.
b illustrates the addition of a locally controlling processor.
a and 2b illustrate the incorporation of reconfiguration capabilities within the invention.
a-3c illustrate the incorporation of analog and digital I/O switching capabilities within the invention.
a-5c illustrates reconfiguration capabilities that may be added to the arrangement of
a-6d illustrates various configurations for transcoding operations as provided for by the invention.
a-7b illustrates the computational load implications of encoding or decoding four video images of quarter size versus one video image of full size. This is useful in flexible task allocation as well as for exemplary video MCU function implementations as provided for by the invention.
a-8d illustrate resource allocation abstractions useful in session management as provided for by the invention.
a-10c illustrate increasing degrees of flexible resource allocation as associations between encode tasks, decode tasks, and real-time media processors are unbundled.
a illustrates an exemplary high-level architecture for implementing analog and digital I/O aspects of the invention applicable to contemporary commercially available components.
a illustrates an exemplary signal flow for a bidirectional codec operation that could readily be executed in the parallelized multi-task environment of the exemplary embodiment depicted in
a illustrates an exemplary tasks associated with implementing an instance of the signal flow procedure of
In the following detailed description, reference will be made to the accompanying drawing(s), in which identical functional elements are designated with like numerals. The aforementioned accompanying drawings show by way of illustration, and not by way of limitation, specific embodiments and implementations consistent with principles of the present invention. These implementations are described in sufficient detail to enable those skilled in the art to practice the invention and it is to be understood that other implementations may be utilized and that structural changes and/or substitutions of various elements may be made without departing from the scope and spirit of present invention. The following detailed description is, therefore, not to be construed in a limited sense. Additionally, the various embodiments of the invention as described may be implemented in the form of a software running on a general purpose computer, in the form of a specialized hardware, or combination of software and hardware.
High-performance video and audio compression/encoding and decompression/decoding systems are commonly in use today and have been available in increasingly miniature forms for many years. In production environments, encoders are used in isolation to record DVDs and to create MPEG video clips, movies, and streaming video. These encoders are typically hardware engines, but can be implemented as batch software programs. In delivery environments, decoders are used in isolation to render and view DVDs, MPEG video clips, movies, and streaming video on computers, set-top boxes, and other end-user hardware. Recently, such decoders are typically implemented in software, but higher-performance hardware systems are also common. In video editing systems, both encoders and decoders often exist in a common system, and there may be more than one decoder available in order to support multiple decoding sessions as part of commonplace video editing tasks. The multiple decoders may be software only. In some cases, several high-performance decoders may coexist in a single board-level system. Single board-level systems comprising an encoder/decoder pair also exist. These, too, are used in video editing but are more commonplace in video conferencing systems where they regularly comprise any of a wide variety of video codecs.
In these single board-level systems comprising an encoder/decoder pair, typically only one compression standard (such as MPEG1/2/4, H.261/263/264, etc.) is supported. These typically provide parameter adjustments such as bit rate, quantization granularity, inter-frame prediction parameters, etc., as provided for in the standard Software decoders initially were similar, although there is increasing support for more than one compression standard. Recently, new powerful media signal processors have appeared which can support pre-execution downloads of a full high-performance video and audio encoder/decoder pair of essentially arbitrary nature, specifically targeting existing video and audio compression standards. This, in principle, makes it possible to create a video and audio encoder/decoder pair within the scope of a physically small single board-level system.
The present invention develops such emergent capability further by creating environments where a plurality of reconfigurable media signal processors cooperatively coexist so as to support a variety of concurrent tasks. In the most straightforward implementation, several independent codec sessions can be supported simultaneously, wherein “session” will be taken to mean not only a granted request for the allocation of resources for a contiguous interval of time but, in a further aspect of the invention, a configuration of those resources maintained for a contiguous interval of time. Considerable additional value is obtained by further providing the reconfigurable media signal processors with abilities to cooperatively interwork. One example of this is providing for transcoding signals conforming to one compression standard to and from that of another compression standard. Yet more value can be obtained by unbundling encoder/decoder pair software into separately executable parts that can be allocated and operate independently. One example of this is the conversion of a common incoming signal into one or more outgoing signals conforming to differing compression standards. Another is increased resource availability when signal flow is unidirectional or bidirectional (“two-way”) compression sessions are not needed. Further, such a system can provide the needed functions involved in implementing a video conferencing MCU or streaming transcoding video storage system, each supporting a variety of analog and digital signal formats sequentially or simultaneously. Additionally, such a system grows gracefully in supporting a larger number of co-executing tasks as software algorithms become more efficient. No less importantly, such a system also provides important architectural continuity as future reconfigurable processors become more powerful and agile.
The overview of the functionalities, capabilities, utility, and value of the invention thus provided, the invention is now described in further detail.
1. Basic Structure and Functionality
a depicts a simply-structured exemplary system 100 provided for by the invention. This exemplary system 100 comprises a plurality of encoder/decoder pairs 110a-110n, each uniquely associated with bidirectional analog/digital conversion elements 120a-120n. Other arrangements provided for by the invention also include those without the bidirectional analog/digital conversation elements 120a-120n and those with additional elements such as digital switches, analog switches, one or more locally controlling processors, bus interfaces, networking and telecommunications interfaces, etc. These will be described later in turn.
Referring to
The analog composite video signals 121a-121n, 122a-122n similarly are typically in compliance with a published industry-wide standard (for example NTSC, PAL, SECAM, etc.). The compressed signals 111a-111n, 112a-112n themselves and the operations performed by encoder/decoder pairs 110a-110n are typically in compliance with a published industry-wide standard (for example H.261, H.263, H.264, MPEG-1, MPEG-2, MPEG-4, etc.) or may be a proprietary standard (such as the wavelet compression provided by Analog Devices ADV601™ chip, etc.). Although not explicitly included nor excluded in this view, the encoder/decoder pairs 110a-110n may or may not further internally provide support for various existing and emerging venues and protocols of digital transport (for example, IP protocol, DS0/DS1 formats for T carrier, ISDN, etc.).
The encoder/decoder pairs 110a-110n may each be implemented as a dedicated hardware engine, as software (or firmware) running on a DSP or generalized processor, or a combination of these. When implemented as software, the encoding and decoding algorithms may be implemented as a common routine, as separate routines timesharing a common processor, or a combination of these. When encoders and decoders are implemented as separate routines permitting timeshared concurrent execution on a common processor, a wide range of new functionality is made cost-effectively possible. Several aspects of the invention leverage this capability in a number of ways as will be subsequently discussed.
Each encoder/decoder of the encoder/decoder pairs 110a-110n may operate independently, or may have various aspects and degrees of its operation governed by common shared coordinating processing. The common shared coordinating processing can be performed by one or more processors, each of which may be local to the system, external to the system, or a combination of these.
or any combination of these, as well as other possible functions. Examples of other possible functions include IP connection implementation, Q.931 operation, H.323 functions, etc. The locally controlling processor 150 may also control some of the additional elements to be described later such as digital switches, analog switches, one or more locally controlling processors, bus interfaces, networking and telecommunications interfaces, etc.
The arrangements described thus far and forward now through
An important note going forward: in order to simplify
As stated earlier, the encoder/decoder pairs 110a-110n may each be implemented as a dedicated hardware engine, as software (or firmware) running on a DSP or generalized processor, or a combination of these. In any of these situations it is often advantageous or necessary to at least set the value of parameters of operation. In the case where encoder/decoder pairs 110a-110n are implemented in part or in full as software running on a DSP or generalized processor, it may be desirable to download parts or all of the software into the DSP or generalized processor on a session-by-session, or perhaps even intra-session, basis. For ease of discussion, the entire range of reconfiguring anything between parameter settings to entire algorithms will be referred to as “reconfiguration.”
In a similar way, it may be advantageous or necessary to set the value of parameters of operation pertaining to the analog/digital conversation elements 120a-120n. For example, each analog/digital conversation element may support a variety of analog protocols (such as NTSC, PAL, SECAM). The conversion may also support a range of parameters such as sampling rate/frame rate, sampling resolution, color models (YUV, RGB, etc.) and encoding (4:2:2, 4:1:1, etc.). The digital stream may have additional adjustable protocol parameters as well.
The invention provides for expanding upon the arrangement illustrated in
The invention further provides for expanding upon the arrangement illustrated in
c combines the switches 170 and 180 of
This arrangement also facilitates a wide range of additional capabilities when additional features are included and leveraged as will become clear in the discussion that follows.
As stated earlier, the invention provides for further expansions upon the arrangement illustrated in
The resulting aggregated arrangement provides reconfigurable access to unbundled lower-level capabilities and as such gives rise to a rich set of higher-level capabilities as will be discussed.
a illustrates the literal combination of
The invention also provides for the incorporation or merging the Digital Bus or Matrix Switch 190 and the Internal Digital Stream Bus 180 into a common digital stream interconnection entity 580 as shown in
Such an arrangement clearly supports a wide range of time-varying demands for codec, transcoding, single-protocol broadcast, and multi-protocol broadcast services. The same arrangement can also implement additional services as will be discussed in Section 1.7. In such an arrangement where common digital stream interconnection entity 580 is used in this fashion (i.e., as in
Standard PCI bus implementations have been 32 bit wide and operate at 33-66 MHz in contemporary practice, so PCI bandwidth is roughly 1-2 GB/sec, supporting 5 to 11 unidirectional full-CIF flows or 2 to 5 bidirectional CIF sessions. Recent higher-bit rate 64-bit PCI/PCI-X extensions operate up to 32 Gbps, supporting up to sixteen times these upper limits (i.e., up to roughly 175 unidirectional full-CIF flows or 80 bidirectional CIF sessions). These relaxed limitations can be even further expanded by utilizing a plurality of PCI busses, each supporting a number of buffered analog/digital conversation elements 520a-520n and encoder/decoder pairs 110a-110m implemented via real-time media processors. Such segregating PCI busses may be linked by means of bus bridges. An example of such an arrangement is shown in
In the context of this invention, transcoding refers to a real-time transformation from one (video) coding (and compression) scheme to another. For example, a live video conferencing stream encoded via H.263 may be converted into MPEG 2 streaming video, or a proprietary video encoding method using run-length encoding may be converted to H.264, etc. These would be accomplished by the invention by in one manner or another connecting a decoder (configured to decode and decompress according to one encoding and compression scheme) to an encoder (configured to encode and compress according to another scheme), where each uses a different compression protocol. The invention can provide for such a capability in a number of ways. Illustrating a first approach,
It is also noted that the transcoding paths 601, 602, 603 described above are also useful as loopback paths for diagnostics purposes.
Additionally, a decoded signal from one of a plurality of decoders is fed to encoders through the internal digital bus or switch matrix 190 as shown in
With exemplary hardware environments provided for by the invention established, attention is now directed towards obtaining even further reconfigurable flexibility, giving rise to yet more new systems level functions, by unbundling the encoder/decoder pairs 110a-110n into encoder algorithms, decoder algorithms, and processors which may freely execute one, or concurrently more than one, instances of these algorithms simultaneously.
Modern high-performance “media” signal processing chips, such as the Equator BSP-15 or Texas Instruments C6000, are capable of concurrently executing an encoding algorithm and a decoding algorithm simultaneously, each at the level of complexity of a bidirectional 768 Kbps H.263 or 2 Mbps MPEG stream. Although some overhead is involved, for a fixed resolution, quantization level, motion-compensation quality-level, and frame-rate the computational load increases roughly linearly with image area. By way of illustration,
A contemporary media processor, such as the Equator BSP-15™ or Texas Instruments C6000™, can concurrently perform a CIF encode and decode, corresponding to 20 of the load units cited above. The same media processor then can alternatively perform, for example, any of the following simultaneous combinations:
16FE+4FD+4QE+QD≦20
As DSP media processors become faster, the right-hand-side increases in magnitude, increasing the flexibility and capabilities of the overall system. Similarly, as algorithms become more efficient, the numbers on the left-hand-side of the constraint equations become smaller, also increasing the flexibility and capabilities of the overall system.
This kind of flexible real-time concurrent task computation arrangement subject to this sort of overall proportion-of-demand resource constraint can readily be extended to other combinations of tasks, types of tasks, task resource requirements, etc.
For example, in an exemplary embodiment of the inventive concept, at least two types of sessions are supported, each drawing from a common collection or pool of shared resources with different requirements. Each type of session may utilize a differing formally defined service, or may involve differing ad-hoc type (or even collection) of tasks. To understand and design such a system with good performance and relatively high utilization of expensive resources, the common collection or pool of shared resources may be thought of at any moment as being divided into those resources allocated to a first type of session/service/task, those resources allocated to a second type of session/service/task, and those resources not currently allocated. One useful way of doing this so as to facilitate practical calculation is to represent the current number of active sessions in a geometric arrangement, each type on an individual mutually-orthogonal axis, and represent resource limitations by boundaries defining the most extreme permissible numbers of each type of session/service/task that are simultaneously possible with the resource limitations.
a illustrates a such geometric representation for the sharing of computation resources between two types of sessions, services, tasks, or collections of tasks whose resource requirements are roughly in a 2:1 ratio. This two-axis plot, as depicted, comprises a vertical axis 801 measuring the number of simultaneously active service sessions requiring the higher number of shared resources and a horizontal axis 802 measuring the number of simultaneously active service sessions requiring the lower number of shared resources. In this example the “higher resource service” associated with the vertical axis 801 requires approximately twice as many instances of real-time resource as the “lower resource service” associated with the horizontal axis 802. As, in this representation, the sessions require integer-valued numbers of the shared computational resource the resulting possible states are shown as the lattice of dots 851 inclusively bounded by the axes 801, 802 (where one or the other services has zero active sessions) and the constraint boundary 804 on the total number of simultaneously available units of resource (here, units of simultaneous real-time computation power). As the “higher resource service” associated with the vertical axis 801 requires approximately twice as many instances of real-time resource as the “lower resource service” associated with the horizontal axis 802, the constraint boundary 804 would be of the form:
2Y+X≦C
wherein the constraint boundary 804 intersects the horizontal axis 802 at the value X=C (i.e., the system is serving C sessions of the “lower resource service”) and also intersects the vertical axis 801 at the value Y=C/2 (i.e., the system is serving C/2 sessions of the “higher resource service”). If, instead, an instance of the “higher resource service” required four times as much real-time computational resource as the “lower resource service,” the constraint boundary 804 would be of the form:
4Y+X≦C;
If it used eight times as much, the constraint boundary 804 would be of the form:
8Y+X≦C,
etc., i.e. the slope of the constraint boundary 804 gets increasingly less steep. One of the results of this ‘open’ policy is that services requiring higher numbers of shared resource experience statistically higher blocking (resource unavailability) than services requiring lower numbers of shared resource. This is because, using the last example, two higher resource sessions require 16 units of resource and if there are more than four lower resource sessions active, less than 16 units of resource would be available. The general phenomenon is suggested by
The general mathematics for specific computations for cases with ‘time-reversible’ (i.e., self-adjoint) stochastic dynamics (which include standard Erlang and Engset blocking models, typically directly relevant here) is given by J. S. Kaufman “Blocking in a Shared Resource Environment, IEEE Transactions on Communications, Vol COM-29 (10), 1474-1481, among many others. Although there are notable curve variations as well as pathologies and exceptions,
2Y≦Ymax (for Y boundary 825a at intercept 821);
8X≦Xmax (for X boundary 825b at intercept 822).
These reservation constraints can be calculated from algebraic equations resulting from various fairness policies. This results in a non-triangular region of permissible states 852. The reservation constraints for the exemplary two-service case of
These general resource allocation structures provide a basis for informed design of embodiments of the invention whose potential flexibility adds predictable value;
a-10d illustrate increasing degrees of unbundling of functionality components and making flexible allocations of the resulting unbundled processes and hardware resources.
b shows an unbundled approach where multiple encoder sessions 1022a-1022n, etc. run on a more specialized class of processor 1012a-1012p optimized for encoding while multiple decoder sessions 1032a-1032m, etc. run on a more general class of processor 1042a-1042q as decoding is typically a less-demanding task than encoding. Allocations are made by session allocation mechanism 1002.
d shows the processing environment of
Reflecting the opportunities and concerns cited above, the invention also provides for further expanding the scope of hardware elements that are profitably manageable in flexible configurations;
In each of these cases, the multi-service allocation mechanisms described earlier, or extensions of them, may be used to manage resources according to various allocation policies. Typically allocation policies determine the bounding convex hull (edges and surfaces 804, 824, 824a, 824b, 834, 844, 844a-844c as shown in
In addition to analog-to-digital/encoding sessions, decoding/digital-to-analog sessions, and transcoding sessions, the invention provides a valuable substrate for the support of other types of functions and operations.
A first example of additional capabilities provided for by the invention is an MCU function, useful in multi-party conferencing and the recording of even two-party video calls. As another example, a video storage and playback encode/decode/transcode engine is illustrated, malting use of the invention's encoder, decoder, and transcode capabilities in conjunction with a high-throughput storage server.
1.7.1 Continuous Presence MCU Applications
The invention provides for using the system to be configured so as to implement an MCU function, useful in multi-party conferencing and the recording of even two-party video calls. This configuration may be a preprogrammed configuration or configured “on-demand” in response to a service request from unallocated encoders and decoders.
It is noted that the topology of the multipoint connection and the associated functions the encoders and decoders are performing determine the source of the streams directed to the MCU functionality. For example:
As to the range of MCU functionalities that can be realized, it is noted that contemporary MCUs implement one or more of a number of types of output streams:
In these, a local controlling processor is typically somewhat to heavily involved in coordinating the operations among the various encoders, decoders, and any other allocated entities.
1.7.2 Video Storage Applications
The invention provides for the system to be configured to implement a video storage and playback encode/decode/transcode engine. This makes use of encoder, decoder, and transcode capabilities in conjunction with a high I/O-throughput storage server. This configuration may be a preprogrammed configuration or configured on-demand in response to a service request involving unallocated encoders and decoders.
In one implementation, a high I/O-throughput storage server connects with the system through a network connection such as high-speed Ethernet. In another implementation, the system further comprises one or more disk interfaces such as IDE/ATA, ST-506, ESDI, SCSI, etc. Such a disk interface would connect with, for example, the internal digital stream bus. Other configurations are also possible.
There are several reasons for adding video storage capabilities and applications to certain implementations of the invention. These include:
2. Example Implementations of the Invention
The discussion now turns to and some exemplary embodiments. Four general exemplary types are considered, distinguished by the type of bus interface technology provided by the hosting system:
Analog A/V bus (
High performance digital A/V bus for D1, D2, ATSC/8-VS B, etc. (
Optical A/V video bus (
The initial discussion is directed to the analog A/V bus case, and the others are then considered as variations. This is followed by a unified description of data flows and task management.
a illustrates a high-level architecture for a single-card implementation 1100a suitable for interfacing with the backplane of a high-performance analog audio/video switch. Such a switch may be part of a networked video collaboration system, such as the Avistar AS2000, or part of a networked video production system, networked video broadcast system, networked video surveillance system, etc.
Referring to
In this exemplary embodiment, the media processors are each assumed to be the Equator BSP-15 ™ or Texas Instruments C6000™ which natively include PCI bus support 1110a-1110n. Each of these communicate with the locally controlling processor 1118 by means of a fully implemented PCI bus 1111 linked via a 60x/PCI bus protocol bridge 1120, such as the Tundra Powerspan™ chip, to an abbreviated implementation of a “PowerPC” 60x bus 1119. It is noted that most contemporary signal processing chips capable of implementing real-time media processors 1109a-1109n natively support the PCI bus rather than directly usable with 60x bus 1119, so the use of a transparent bus protocol bridge 1120 as shown in
The locally controlling processor 1118 provides higher-level packetization and IP protocol services for the input and output streams of each of the real-time media processors 1109a-1109n and directs these streams to and from an Ethernet port 1131 supported by an Ethernet interface subsystem 1130, such as the Kendin KS8737/PHY™ interface chip or equivalent discrete circuitry. Alternatively, other protocols, such as Firewire™, DS-X, Scramnet™, USB, SCSI-II, etc., may be used in place of Ethernet.
The locally controlling processor 1118 also most likely will communicate with the host system control bus 1150; in this exemplary embodiment a bus interface connection 1115 connects the host system control bus 1150 with a communications register 1116 which connects 1117 with the locally controlling processor 1118 and acts as an asynchronous buffer.
For diagnostics purposes, locally controlling processor 1118 may also provide a serial port 1135 interface. Alternatively, a wide range of other protocols, including USB, IEEE instrumentation bus or Centronix™ parallel port, may be employed.
Again referring to
The bidirectional digital video signals 1106a-1106n exchanged between the analog-to-digital (A/D) and digital-to-analog (D/A) converters 1105a-1105n and real-time media processors 1109a-1109n are carried in digital stream format, for example via the CCIR-656™ protocol although other signal formats may be employed. The bidirectional digital audio signals 1107a-1107n exchanged between the analog-to-digital (A/D) and digital-to-analog (D/A) converters 1105a-1105n and real-time media processors 1109a-1109n are also carried in digital stream format, for example via the IIS protocol although other signal formats may be employed.
Bidirectional control signals 1108a-1108n exchanged between the analog-to-digital (A/D) and digital-to-analog (D/A) converters 1105a-1105n and real-time media processors 1109a-1109n may be carried according to a control signal protocol and format, for example via the I2C protocol although others may be employed. In this exemplary embodiment, the real-time media processors 1109a-1109n serve in the “Master” role in the “master/slave” I2C protocol. In this way the media processors can control the sampling rate, resolution, color space, synchronization reconstruction, and other factors involved in the video and analog conversion.
Each of the analog-to-digital (A/D) and digital-to-analog (D/A) converters 1105a-1105n handles incoming and outgoing analog video signals 1103a-1103n and analog audio signals 1104a-1104n. These signals are exchanged with associated analog A/V multiplexers/demultiplexers 1102a-1102n. The incoming and outgoing analog video signals 1103a-1103n may be in or near a standardized analog format such as NTSC, PAL, or SECAM.
In this exemplary embodiment, the analog A/V multiplexers/demultiplexers 1102a-1102n exchange bidirectional multiplexed analog video signals 1101a-1101n with an analog crossbar switch 1112a that connects directly with an analog bus 1140a via an analog bus interface 1113a. In this exemplary embodiment, the analog crossbar switch 1112a is directly controlled by the host control processor 1160 via signals carried over the host system control bus 1150 and accessed by host system control bus interfaces 1151 and 1114. Alternatively, the analog crossbar switch 1112a, if one is included, may be controlled by the local controlling processor 1118 or may be under some form of shared control by both the host control processor 1160 and the local controlling processor 1118.
Internally, each of the analog A/V multiplexers/demultiplexers 1102a-1102n, should they be used in an implementation, may further comprise an A/V multiplexer (for converting an outgoing video signal and associated outgoing audio signal into an outgoing A/V signal) and an A/V demultiplexer (for converting an incoming A/V signal into incoming an video signal and associated incoming audio signal). Typically, the bidirectional paths 1101a-1101n comprise a separate analog interchange circuit in each direction. This directional separation provides for maximum flexibility in signal routing and minimal waste of resources in serving applications involving unidirectional signals. Alternatively, the two directions can be multiplexed together using analog bidirectional multiplexing techniques such as frequency division multiplexing, phase-division multiplexing, or analog time-division multiplexing. The host system, particularly the analog A/V bus 1140a, will typically need to match the chosen scheme used for handling signal direction separation or multiplexing. The invention also provides for other advantageous approaches to be used as is clear to one skilled in the art.
Returning to the transcoding configurations of
The latter configuration can be exploited further by routing a decoded signal into a plurality of decoders as shown in
It is further noted that many or in fact all of the transcoding streams may be routed through the networking port 1131. If more bandwidth is required the network protocol processing path (here involving the bus bridge 1120, the local controlling microprocessor 1118) can be re-architected to provide dedicated high-performance protocol processing hardware.
Although an interface for an analog A/V bus is described above, the core architecture is essentially identical for a raw high-performance digital stream such as the D1 and D2 formats used in digital video production, ATSC/8-VS B, etc.
For a high-performance digital stream host bus implementation, the analog-to-digital (A/D) and digital-to-analog (D/A) converters 1105a-1105n are omitted and the analog bus 1140a and analog bus interface 1113a are replaced by their high-throughput digital counterparts 1140b and 1113b. The analog crossbar switch 1112a and analog A/V multiplexers/demultiplexers 1102a-1102n could be omitted altogether, or replaced by their high-throughput digital counterparts 1112b and 1162a-1162n as shown in the figure. Here, the bidirectional video 1106a-1106n, audio 1107a-1107n, and control 1108a-1108n paths connect directly to these optional high-throughput digital A/V multiplexers/demultiplexers 1162a-1162n. Alternatively, the media processors 1109a-1109n could do the optional A/V stream multiplexing/demultiplexing internally. The high-throughput multiplexed digital A/V signals 1162a-1162n can either be directed to an optional high-throughput digital crossbar switch 1112b as shown or else connect to the high-throughput digital A/V bus 1140b. Such busses are typically time-division multiplexed, but in the case they are not either time-division-multiplexed or provide space-divided channels, additional bus arbitration hardware would be required. If the optional high-throughput digital crossbar switch 1112b is used, it connects to the high-throughput digital A/V bus 1140b. Otherwise the operation is similar or identical to that of the analog I/O bus implementation described in Section 2.1.
The exemplary high-level architecture of
c shows an exemplary embodiment adapting the basic design of
Here two exemplary signal flows for codec and transcoding functions are provided. In these, configurations and routing involved in moving the analog signals to and from the host system bus 1140a through the analog crossbar switch 1112a and the digital signals to and from the network port 1131 through the PCI bus 1111 and other subsystems 1120, 1118, 1130 are not depicted.
2.4.1 Bidirectional Codec Example
a illustrates an exemplary signal flow for a bidirectional codec (two-way analog compression/decompression) operation using the system depicted in
2.4.2 Transcoding Example
b illustrates an exemplary signal flow for a unidirectional transcoding operation. an incoming digital stream 1211 is queued in a queuing operation 1210 for dejittering and then provided in a statistically-smoothed steady stream 1211a to a decompression operation 1212 to create a wideband digital signal 1223. This wideband digital signal 1223 is then encoded into a different signal format in a compression step 1204 to create an outgoing digital stream 1205.
Although not required in many embodiments, it can be advantageous for the exemplary lower-level tasks and operations depicted above to be aggregated to form higher-level steps and operations. In various implementations this allows for useful modularity, better software structure, and better matching to a generalized operational framework.
In particular, in situations where multiple types of compression or decompression algorithms co-execute on the same media processor this would provide ready and rapid reconfigurable support for multiple types of protocols in a common execution environment. This includes self contained means, or other standardized handling, for initiation, resource operation, resource release, and clean-up.
Such modularization allows for rapid reconfiguration as needed for larger network applications settings. In cases with explicit control of network elements, such as the AvistarVOS™, the system can natively reconfigure ‘on demand.’ In more primitive or autonomous network configurations, the invention provides for the system to rapidly reconfigure ‘behind the scenes’ so as to flexibly respond to a wide range of requests on-demand.
The real-time job manager manages the execution of all other real-time jobs or active objects. It can itself be a co-executed real-time job or active object, as will be described below. The real-time job manager accepts, and in more sophisticated implementations also selectively rejects, job initiation requests. Should job request compliance not be handled externally, it may include capabilities that evaluate the request with respect to remaining available resources and pertinent allocation policies as discussed in Section 1.5. The jobs themselves are best handled if modularized into a somewhat standardized form as described in Section 2.5.
The left portion of
The right portion of
In addition to the exemplary Additional Processing Job 1355,
An exemplary aggregation of low-level tasks associated with implementing an instance of the signal flow is now considered. Such aggregation results in a smaller collection of real-time jobs or active objects with a more uniform structure to ease reconfiguration actions, all in keeping with the points of Section 2.5. The resulting jobs would be those of the type to be handled in the exemplary real-time process management environment depicted in
The example chosen and depicted in
a shows the individual steps involved in the two directional paths of data flow for this example. The first path in this flow is the analog capture step 1401 involving an analog-to-digital converter. The captured sample value is reformatted at 1402 and then presented for encoding at 1403. The media processor transforms a video frame's worth of video samples into a data sequence for RTP-protocol packetization, which occurs in a packetization step 1404. The packet is then transmitted by 1405 out to the local controlling processor I/O 1406a for transmission onto the IP network by subsequent actions of the local controlling processor. The second task in this flow begins with a local controlling I/O exchange 1406b into a packet receive task 1407 which loads a packet queue 1408a. When this packet queue is polled and found to be non-empty, the packet is removed at 1408b and depacketized at the RTP level 1409. The resulting payload data is then directed to a decoding operation 1410. The result is reformatted 1411 and directed to a digital-to-analog converter for analog rendering 1412.
Although the individual steps may be handled in somewhat different ways from one implementation to another, this exemplary implementation is representative in identifying fourteen individual steps. Modularizing groups of these steps into a smaller number of real-time jobs in a structurally and functionally cognoscente manner as described in Section 2.5 makes the initiation, periodic servicing, management, and deactivation far easier to handle. One example aggregation, represented in
Aggregate steps 1406b, 1407, and 1408b into a second job. In some implementations this job may be viewed as just an instance of other similar tasks that match the function of the Real-Time Job Manager job 1325 which checks the local controlling processor message queue. In other implementations, the received and transmitted packets may be routed through (a) separate ‘non-message’ local controlling processor packet I/O path(s);
In this exemplary implementation, all three of these jobs would execute on the media processor. Other arrangements are also possible and provided for in the invention.
In reference to the discussion above, the invention provides for alternative implementations which split the tasks of
With regards to protocol processing,
In the first example (“Partition 1”), the selected media processor from the collection 1109a-1109n would be responsible for RTP protocol processing 1504, codec-specific protocol processing 1505, and finally the operations on the actual data payload 1506. The rest of the protocol stack implementation would be handled by the local controlling processor 1118. In the second example (“Partition 2”), the selected media processor is only responsible for operations on the actual data payload 1506, leaving two additional protocol stack implementation tasks 1504, 1505 to instead also be handled by the local controlling processor.
In comparison, Partition 1 spares the local controlling processor from a number of processing tasks and thus scales to larger implementations more readily than Partition 2. However, Partition 2 limits the loading on the media processors, giving more computational capacity for protocol handling.
In the preceding description, reference was made to the accompanying drawing figures which form a part hereof, and which show by way of illustration specific embodiments of the invention. It is to be understood by those of ordinary skill in this technological field that other embodiments may be utilized, and structural, electrical, as well as procedural changes may be made without departing from the scope of the present invention. The various principles, components and features of this invention may be employed singly or in any combination in varied and numerous embodiments without departing from the spirit and scope of the invention as defined by the appended claims. For example, the system need not be hosted in a bus-based system but rather those I/O connections may be brought out as standard signal connectors, allowing the system essentially as described in a freely stand-alone implementation without physical installation in a host system chassis.
Finally, it should be understood that processes and techniques described herein are not inherently related to any particular apparatus and may be implemented by any suitable combination of components. Further, various types of general purpose devices may be used in accordance with the teachings described herein. It may also prove advantageous to construct specialized apparatus to perform the method steps described herein. The present invention has been described in relation to particular examples, which are intended in all respects to be illustrative rather than restrictive. Those skilled in the art will appreciate that many different combinations of hardware, software, and firmware will be suitable for practicing the present invention. For example, the described software may be implemented in a wide variety of programming or scripting languages, such as Assembler, C/C++, per, shell, PHP, Java, etc.
Moreover, other implementations of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.
This application claims the benefit of priority to U.S. Provisional Patent Application No. 60/647,168 filed on Jan. 25, 2005, under the same title, which is incorporated by reference in its entirety for all purposes as if fully set forth herein.
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/US06/01358 | 1/12/2006 | WO | 00 | 1/24/2008 |
| Number | Date | Country | |
|---|---|---|---|
| 60647168 | Jan 2005 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 11246867 | Oct 2005 | US |
| Child | 11814671 | US |
