1. Technical Field
Embodiments of the present invention are related to the field of communications, and in particular, to communication devices.
2. Description of Related Art
In the current art, negotiation of media processing operations (e.g., agreed upon encode/decode operations for codecs) in a communications session may be limited by the capabilities of the endpoints that are common to all parties. In other words, for a communications session to be established between the two parties, they may negotiate and agree on a common media processing operation, but that operation must exist at both endpoints prior to the negotiation. In some cases, this may result in sub-optimal quality of service for one or more of the parties or may result in the communications not being possible.
In the following description, for purposes of explanation, numerous details are set forth in order to provide a thorough understanding of the disclosed embodiments of the present invention. However, it will be apparent to one skilled in the art that these specific details are not required in order to practice the disclosed embodiments of the present invention. In other instances, well-known electrical structures and circuits are shown in block diagram form in order not to obscure the disclosed embodiments of the present invention.
With reference to
Media Processing Component (MPC)—an executable, computer code implementation of a media-processing operation(s). The media-processing operation may be described as one that takes a stream of media data, such as digitized voice or video, and performs certain computations and/or translations on the data and outputs the resultant data. Examples of such media-processing operations may include operations used for audio/video codecs, audio gain control, or audio/video mixers. In some embodiments, the MPC may be self-identifying.
Media Processing Framework Module (Framework Module)—software that enables an application, such as an Application Agent defined below, to create, modify and/or control a collection of Media Processing Components.
Compute Device—hardware (e.g., CPU, memory etc.) that execute the Framework Module and Media Processing Components.
Application Agent (M)—one or more software application(s) executing on a Compute Device and controlling MPC(s) through the Framework Module. An AA may represent a participant in a multimedia session. One example of many possible examples of an AA may be a personal voice mail that physically runs on a cell phone. In some embodiments, the AA may consist of multiple application programs (e.g., may be a distributed program)
Network Interface Controller (NIC)—hardware and software that enable communication between AAs.
Media Processing Protocol—set of rules defined herein for the negotiation, transport, and instantiation of Media Processing Components in a mono or multi-media communications session.
Referring to
For the purposes of simplification, in
AAs 16A and 16B may be coupled to the input/output devices 14A and 14B, respectively, with the AA 16A being configured to be a sending AA and the AA 16B being configured to be a receiving (target) AA in
As previously described, the AAs 16A and 16B may each represent a party in an N-way communications session over the network 12. On the sender's side in the input device 14A, the media data stream may be digitized and typically encoded/compressed before transmission over the network 12. The data then may be decoded/uncompressed on the receiver's side in the output device 14B where it then may be appropriately rendered, such as out of a speaker or a video screen. There are many techniques or operations for encoding and decoding audio and video data. The different encoding/decoding techniques may exhibit different characteristics with regard to compression, fidelity loss, and processing load. In order for a communications session to be established between the two parties, they may negotiate and agree on a common encode/decode techniques to use. Further, there may be additional processing performed on the media data stream in the input/output devices 14A and 14B to realize the quality desired by the user. One or more of these media processing techniques are incorporated into the self-identifying MPCs defined above.
In some embodiments, the components of the Compute Devices 20A and 20B may be the same; hence, the same reference numbers will be used for the same components and not all components will be shown for both of the Compute Devices 20A and 20B. Likewise, in some embodiments, the AAs 16A and 16B may have the same functionality, e.g. having both the sending and receiving functionalities described herein. With the configuration shown in
Referring to
The Framework Module 32 is shown in
Upon the MPCs 30 being installed in the Framework Module 32, the MPCs 30 may be executed by the Compute Device 20B to process a media stream 34 from the sending AA 16A. In some embodiments, the media steam 34 may be transmitted over the communication links 24 and 26 to and from the network 12.
In
In some embodiments, the MPCs used by the N Application Agents 16 (N=2 in the example of
Referring to a flow chart of
In an operation 44, there is a negotiation phase where the 2 parties (two AAs) attempt to negotiate and agree on media transmission properties (e.g., encode/decode MPCs, bandwidth parameters, etc). In some embodiments, this negotiation may be undertaken during the session establishment phase. In other embodiments, this negotiation may be undertaken after a session has been established, perhaps because of network congestion conditions.
In the operation 44, the AAs advertise their conformance with the previously described MPC sharing capability by specifying a “special” known tag in their list of acceptable media. As an example, using the SIP protocol, an AA advertises the media types and acceptable formats in a structure known as the Session Description Protocol (SDP). An AA that implements the MPC sharing capability may specify that they support pushable MPCs by specifying the special known tag in its list of supported media types. In the following example, the SDP may be used to indicate the AA will accept audio in International Telecommunication Union (ITU)-T Recommendation G.711 plaw (operations for uncompressed codecs available for packet-based networks, at http://www.itu.int) as indicated by the media type 0 in the m=line. It further may specify that it conforms to the MPC sharing capability by providing a dynamic payload type 200 and indicating that 200 maps to PUSHABLE. Here the tag PUSHABLE is being used as the special known identifier for the MPC sharing capability.
v=0
o=jgrecco 2890844526 2890842807 IN IP4 192.168.1.1
s=SDP PushableCodec
m=audio 49170 RTP/AVP 0 200
a:rtpmap:200 PUSHABLE/INTC/PXA27x
As previously described, the N-way IP communications session 40 may enable one AA to push (i.e., upload) the implementation of MPCs to another AA in the course of the IP communications session. In the operation 44, to ensure that the MPC is executable on the Compute Device having the receiving AA, it is desirable that the receiving AA identify its execution environment. In the above SDP example, this information is passed in the rtpmap line (INTC/PXA27x). It may be advantageous to provide additional information in this way. For example, the version of the Framework Module may also be provided.
The next stage of the N-way IP communications session 40 may be generally referred to as the MPC push. Once it has been established that both AAs implement the MPC sharing capacity, the AAs may now exchange the MPC's that realize the desired media processing operations. Unless otherwise indicated during the negotiation operation 44, it is assumed that either AA in the communications session is capable of sending and receiving a MPC.
In an operation 46, one of the AAs may be selected for pushing the MPC(s) and becomes the sending AA. In some embodiment, the protocol defined by this N-way IP communications session 40 may specify that the contacting AA takes precedence over the contacted AA for sending MPCs, and thereby automatically selects the contacting AA to be the sending AA for pushing the MPC(s) and the contacted AA becomes the receiving AA. Alternatively, in the operation 46 the contacting AA may defer the MPC exchange to the contacted AA by sending it a specifically defined message.
If the contacting AA decides to send MPC(s) as the sending AA, it undertakes the following operations. In an operation 48, the sending AA may select the appropriate MPC(s) implementation based on the receiving AA execution environment. In an operation 50, in some embodiments, the sending AA may encode/compress/encrypt the MPC(s) for reliable, efficient and secure transport. In other embodiments, the operation 50 may be eliminated. In an operation 52, the sending AA may send a specifically defined message or datagram containing the encapsulated MCP(s) using an appropriate mechanism/protocol. For example, the MPC may be sent as the MIME encoded body of a SIP INFO message. If the contacting AA defers the MPC exchange to the contacted AA by sending it a specifically defined message in the operation 46u, then upon acknowledging that message, the contacted AA may perform the operations 48, 50 and 52 of the sending AA described above.
The next stage of the N-way IP communications session 40 may be referred to as a validation and acknowledgement stage. In an operation 54, upon receipt of the MPC, the receiving AA may unpack it and may verify that it has arrived intact. In an operation 56, the receiving AA may acknowledge the successful receipt by sending a specifically defined message to the sending AA.
The next stage of the N-way IP communications session 40 may be referred to as a MPC Installation and activation stage. Once all of the MPCs have been received, in an operation 58, the receiving AA may install them into its Framework Module using the framework and interfaces defined by the Framework Module and any knowledge it has for the purpose (function) of the MPC. In some embodiments, the MPCs may be self-describing, as will be discussed hereinafter. In some embodiments, it may be assumed that the receiving AAs have intimate knowledge of the function of a MPC and may use that knowledge when configuring the Framework Module and controlling the media streams therein. In other embodiments, there may be a base set of interfaces in the MPC and Framework Module to allow the receiving AAs to interact with them while having a minimal knowledge of their function. In some embodiments, as an example of how this may be implemented, the receiving AA may unpack the executable binary MPC, store it in a specifically defined folder in local storage. The AA then may load the executable image (e.g., “LoadLibrary”) of the MPC into mass memory and use Framework Module interfaces to instantiate instances of the MPC and “wire” them into a media stream graph.
With respect to the MPC 30 being self-identifying, in some embodiments, two mechanisms may be used. The MPC 30 may be encapsulated in a digital envelope for transmission from the sending AA 16A to the receiving AA 16B. Headers in the envelope may contain information describing the MPC 30, perhaps including digital signatures, unpacking instructions, interface information and like information. Once the MPC 30 is unpacked and loaded at the receiving AA 16B, it may have interfaces that would allow it to be identified at runtime as well.
The next stage of the N-way IP communications session 40 may be referred to as a MPCs usage stage. Once the MPCs have been successfully installed in the Framework Module, in an operation 60, the receiving AA then notifies the sending AA that it is ready to communicate. In some embodiments, this may be accomplished via existing mechanisms such as via realtime transfer protocol (RTP) or may be done via a message specifically defined for the purpose.
In some embodiments, the next stage of the N-way IP communications session 40 may be referred to as a clean-up. In other embodiments, this stage may be eliminated. It may be desirable for the MPCs to be actively deleted from the receiving AA after use or otherwise rendered unusable after use. In this event, in an operation 62, upon termination of the communication's session, the receiving AA may delete the MPCs from local storage and memory or disable the MPCs. In these embodiments, the receiving AA in effect may make certain assurances that the MPCs that have been uploaded will be deleted or disabled after the authorized use. In some embodiments, if the sending AA wants to loan the receiving AA a MPC for use during a session, the sending AA may specify that the MPC cannot be used by the receiving AA after the session has been completed.
In some embodiments, the MPC may be implemented as a dynamically linked library (DLL) or shared library (SL) using established operating system (OS) specific methods. The MPC may be up loaded as a compressed zip archive file. The Framework Module may receive the upload as a sequence of packets. Once the Framework Module has received the entire MPC archive, it may unpack it into a DLL or SL. Using OS specific established methods, it may “load” the DLL (or SL) into memory and enumerate its entry points (interface). At this point, the Framework Module is able to interoperate with the MPC (call its interface). Using the MPC's interfaces, the Framework module may exchange its own entry points to establish a two-way communications between Framework Module and the MPC. This may allow the MPC to use services provided by the Framework Module and may allow the Framework Module to control the MPC.
Examples of what might be some of the interfaces on a MPC include an interface for pushing in a media packet of the media stream for processing specific to that MPC. Another might be an interface allowing the MPC to present a media packet (perhaps after processing). In some embodiments, these interfaces may be considered “generic” and no understanding of the processing done by the MPC is needed by the Framework Module, on behalf of the AA, to use the MPC. Hence, the Framework Module may simply feed packets through the MPC with no need for knowing what is being done inside. In some embodiments, the MPCs may publish other interfaces that are specific to the operations the MPC perform. These interfaces may need a priori (or otherwise discovered) knowledge to use. For example, a gain control might have an interface specifically for setting the value of the gain up or down. A digit detector might have an interface for reporting digit events. Described above is one of many ways to accomplish these operations. Java, activeX, .NET have other ways of accomplishing these operations.
The N-way IP communications session 40, as described above, is an example of how the session 40 may be implemented using SIP/SDP. However, the session 40 may be implemented using other protocols. The network 12 of
Referring to
Referring to
As an illustrative example, assume that the media stream is being transmitted from the endpoint Compute Device 20 which is coupled to the gateway 72 by way of communication link 24. If this endpoint Compute Device 20 transmitting the media stream does not have the power to run the codec operation to encode the media stream, then the codec operation may run on the intermediate Compute Device 20 in the gateway 72 to encode at least a portion of the media stream. Likewise, if the endpoint Compute Device 20 coupled to the gateway 74 by the communication link 26 does not have the power to run the codec MPC in decoding the media steam, then the MPC may run on the intermediate Compute Device 20 in the gateway 74 to decode at least a portion of the media stream.
In summary, the arrangement of Compute Devices in the communication network 70 may take many different configurations. In some embodiments, uploading the MPC may only be to one or both of the intermediate Compute Devices 20 to help in sharing the computational tasks. In other embodiments, the MPC may only be uploaded to at least one endpoint Compute Devices 20 to allow both endpoints to share the same MPC. In some embodiments, the MPC may be distributed to one or more intermediate Compute Devices 20 and to one or more endpoint Compute Devices. In some embodiments, the computational tasks may be divided up between an endpoint Compute Device 20 and an intermediate Compute Device 20, e.g., encryption/decryption undertaken at the gateway 72 or 74 and encoding/decoding or codec undertaken at the sending or receiving AA. In general, with respect to the intermediate Compute Devices 20, the network 70, according to one embodiment of the present invention, may allow for the distribution of the processing to devices anywhere between the endpoints.
Examples of the main memory 76 include, but are not limited to, static random access memory (SRAM) and dynamic random access memory (DRAM). Examples of the local storage 28 include, but are not limited to, a hard disk drive, a compact disk drive (CD), a digital versatile disk driver (DVD), a floppy diskette, a tape system and so forth.
Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that any arrangement which is calculated to achieve the same purpose may be substituted for the specific embodiment shown. This application is intended to cover any adaptations or variations of the present invention. Therefore, it is manifestly intended that this invention be limited only by the claims and the equivalents thereof.