Patent Grant
11882165
Interest in avatar-based virtual reality communications systems has grown with the increased availability of computing systems that have high-processing-power and high-bandwidth network connections. A primary goal of such a virtual reality system is to create a virtual space in which users can interact and communicate using realtime data streams, such as audio, video and text chat streams. The virtual space typically is defined by a computer graphics specification that describes the visual geometry of the space, the colors and textures that are mapped onto the visual geometry, the collision properties that control how users maneuver within the space, and auditory properties, such as, reverberation and sound absorption properties, of the space.
In a typical virtual reality system, the users communicate with each other from respective computers through an interface that is a source, a sink, or both a source and a sink of one or more of the realtime data streams that are supported by the system. A virtual reality software application running on each of the user's computers configures its own audio and graphic rendering based on position information describing the positions of avatars in the virtual space. The position information typically is received either directly from the other users' computers or indirectly from a central presence server. By default, the virtual reality software application typically connects each source represented in the virtual space to every sink represented in the virtual space, subject to conditions specified in global switching rules, local user preferences, and the properties of objects within the virtual space. These conditions typically are specified in terms of relative distances between objects. For example, some virtual reality software applications are configured so that realtime data stream connections are not established if the separation distance between avatars exceeds a maximum threshold distance.
A successful virtual reality communications system typically should have relatively low computational resource requirements so that realtime communications performance can be achieved using currently available computing devices and network bandwidth constraints. In addition, such a system typically should be implemented in a way that encourages area designers to develop virtual areas that increase the adoption of the system by users.
In one aspect, the invention features a method in accordance with which one or more stream handling instructions are received at a local network node from a remote network node. The one or more stream handling instructions include a specification of a stream handler for processing at least one realtime data stream. At the local network node, a stream handler is created in accordance with the specification. A resultant data stream is produced at the local network node. In this process, a realtime data stream is processed through the created stream handler.
In another aspect, the invention features a method in accordance with which a specification of a realtime stream handler is parsed from one or more stream handling instructions. In this process, an input source identifier, an output sink identifier, and a respective identifier of each of one or more data processing objects are parsed from the one or more stream handing instructions. Realtime stream handling objects corresponding to respective ones of the identifiers are instantiated, A directed graph that includes ones of the instantiated realtime stream handling objects are created in accordance with the specification. A realtime data stream is received from an input source corresponding to the input source identifier. A resultant data stream is produced at an output sink corresponding to the output sink identifier. In this process, the realtime data stream is processed through the directed graph.
In another aspect, the invention features a method in accordance with which at least one realtime data stream connection is established between a local network node and at least one remote network node. At the local network node, at least one realtime data stream that is sourced by the remote network node is processed. In this process, the at least one realtime data stream is processed through one or more realtime data processing operations to produce a resultant data stream. The processing is monitored. In response to a determination based on the monitoring that the processing deviates from a performance target, the processing is modified in accordance with a realtime performance targeting routine.
In another aspect, the invention features a method in accordance with which, on a local network node, a first session is established with a remote network node on a transport stream in accordance with a connectionless transport protocol. On behalf of one or more software entities on the local network node, one or more channels over which data is transmitted between the local network node and the remote network node in the first session automatically are opened. In the first session, a table is maintained. The table identifies open ones of the channels and associates respective attribute values with the identified channels. In response to a determination that the first session has failed, a second session with the remote network node is automatically attempted to be established on a second transport stream in accordance with the connectionless transport protocol. In response to successful establishment of the second session, each of the channels identified in the table automatically is opened.
In another aspect, the invention features a method in accordance with which a list of kernel components that include one or more kernel service components is parsed. All the kernel components in the parsed list that are missing from a local repository are determined. Each of the kernel components determined to be missing is retrieved. Kernel services are instantiated from ones of the kernel service kernel components. The instantiated kernel services are executed to communicate with one or more remote network nodes in a communication environment defined with respect to a virtual area.
In another aspect, the invention features a method that is performed on a local network node. In accordance with this method the local network node is configured to support realtime communications with at least one remote network node in a context defined by a virtual area. The configuration process includes: returning a list comprising identifiers of all plugins associated with the specified API in a plugin database in response to a call to enumerate all plugins that support a specified application programming interface (API); delivering a list comprising identifiers of all variants of the given API that are supported by the identified plugin in response to a call to enumerate variants of a given API supported by an identified one of the plugins; and loading the identified plugin and providing a pointer to an instance of the identified variant in response to a call to instantiate an identified one of the variants of an identified API supported by an identified one of the plugins. At least one realtime data stream connection is established between the configured local network node and the at least one remote network node.
The invention also features apparatus operable to implement the inventive methods described above and computer-readable media storing computer-readable instructions causing a computer to implement the inventive methods described above.
Other features and advantages of the invention will become apparent from the following description, including the drawings and the claims.
Referring to
In the following description, like reference numbers are used to identify like elements. Furthermore, the drawings are intended to illustrate major features of exemplary embodiments in a diagrammatic manner. The drawings are not intended to depict every feature of actual embodiments nor relative dimensions of the depicted elements, and are not drawn to scale.
A “computer” is any machine, device, or apparatus that processes data according to computer-readable instructions that are stored on a computer-readable medium either temporarily or permanently.
A “computer operating system” is a software component of a computer that manages and coordinates the performance of tasks and the sharing of software and hardware resources. A “kernel” is a collection of software components that can be invoked by software applications to provide specific functions for accessing computer resources (e.g., CPU, memory, network links, and peripheral resources). A “software application” (also referred to as software, an application, computer software, a computer application, a program, and a computer program) is a set of instructions that a computer can interpret and execute to perform one or more tasks.
An “application programming interface” (or API) is a set of declarations of the functions (or procedures) that an operating system, library, or service provides to support requests made by a software application. An API specifies an interface and the behavior of the identifiers specified in that interface. An implementation of an API refers to the software application code that provides the functionality described by the API. A “computer data file” is a bock of information that durably stores data for use by a software application.
A “service” is a process that performs a task on its own initiative (independently of other processes).
A “manager” is a gateway for a service to perform tasks. A manager does not perform a task on its own initiative.
A “database” is an organized collection of records that are presented in a standardized format that can be searched by computers. A database may be stored on a single computer-readable data storage medium on a single computer or it may be distributed across multiple computer-readable data storage media on one or more computers.
A “data sink” (referred to herein simply as a “sink”) is any of a device, part of a device (e.g., a computer), or software that receives data.
A “data source” (referred to herein simply as a “source”) is any of a device, part of a device (e.g., a computer), or software that originates data.
A “network node” (also referred to simply as a “node”) is a junction or connection point in a communications network. Exemplary network nodes include, but not limited to, a terminal, a computer, and a network switch. A “server network node” is a host computer on a network that responds to requests for information or service. A “client network node” is a computer on a network that requests information or service from a server. A “network connection” is a link between two communicating network nodes. The term “local network node” refers to a network node that currently is the primary subject of discussion. The term “remote network node” refers to a network node that is connected to a local network node by a network communications link.
“Presence” refers to the ability and willingness of a networked entity (e.g., a communicant, service, or device) to communicate, where such willingness affects the ability to detect and obtain information about the state of the entity on a network and the ability to connect to the entity.
A “realtime data stream” is data that is structured and processed in a continuous flow and is designed to be received with no delay or only imperceptible delay. Realtime data streams include digital representations of voice, video, user movements, facial expressions and other physical phenomena, as well as data within the computing environment that may benefit from rapid transmission, rapid execution, or both rapid transmission and rapid execution, including for example, avatar movement instructions, text chat, realtime data feeds (e.g., sensor data, machine control instructions, transaction streams and stock quote information feeds), and file transfers.
A “stream mix” is a combination of two or more realtime data streams of the same or semantically consistent type (e.g., audio, video, chat, and motion data). For example, a set of voice streams might be mixed into a single voice stream or a voice stream might be mixed into the audio portion of a video stream.
A “switching rule” is an instruction that specifies a connection or disconnection of one or more realtime data sources and one or more realtime data sinks subject to one or more conditions precedent.
A “virtual area” (also referred to as an “area” or a “place”) is a representation of a computer-managed space or scene. Virtual areas typically are one-dimensional, two-dimensional, or three-dimensional representations; although in some embodiments a virtual area may correspond to a single point. Oftentimes, a virtual area is designed to simulate a physical, real-world space. For example, using a traditional computer monitor, a virtual area may be visualized as a two-dimensional graphic of a three-dimensional computer-generated space. However, virtual areas do not require an associated visualization to implement switching rules. A virtual area typically refers to an instance of a virtual area schema, where the schema defines the structure and contents of a virtual area in terms of variables and the instance defines the structure and contents of a virtual area in terms of values that have been resolved from a particular context.
A “virtual area application” (also referred to as a “virtual area specification”) is a description of a virtual area that is used in creating a virtual area communication environment. The virtual area application typically includes definitions of geometry, physics, and realtime switching rules that are associated with one or more zones of the virtual area.
A “virtual communication environment” is a representation of a computer-managed space that includes at least one virtual area and supports realtime communications between communicants.
A “zone” is a region of a virtual area that is associated with at least one switching rule or governance rule. A switching rule controls switching (e.g., routing, connecting, and disconnecting) realtime data streams between network nodes communicating in the context of a virtual area. A governance rule controls a communicant's access to a resource (e.g., an area, a region of an area, or the contents of that area or region), the scope of that access, and follow-on consequences of that access (e.g., a requirement that audit records relating to that access must be recorded).
A “position” in a virtual area refers to a location of a point or an area or a volume in the virtual area. A point typically is represented by a single set of one-dimensional, two-dimensional, or three-dimensional coordinates (e.g., x, y, z) that define a spot in the virtual area. An area typically is represented by the three-dimensional coordinates of three or more coplanar vertices that define a boundary of a closed two-dimensional shape in the virtual area. A volume typically is represented by the three-dimensional coordinates of four or more non-coplanar vertices that define a closed boundary of a three-dimensional shape in the virtual area.
In the context of a virtual area, an “object” is any type of discrete element in a virtual area that may be usefully treated separately from the geometry of the virtual area. Exemplary objects include doors, portals, windows, view screens, and speakerphone. An object typically has attributes or properties that are separate and distinct from the attributes and properties of the virtual area. An “avatar” is an object that represents a communicant in a virtual area.
A “communicant” is a person who communicates or otherwise interacts with other persons over a network connection, where the communication or interaction may or may not occur in the context of a virtual area. A “user” is a communicant who is operating a particular network node that defines a particular perspective for descriptive purposes.
An “area server” is a network node that includes an area network infrastructure service, which administers a virtual area that hosts a virtual area application by managing sessions of client nodes associated with objects in the virtual area.
As used herein, the term “includes” means includes but not limited to, and the term “including” means including but not limited to.
The embodiments that are described herein provide a realtime kernel that supports realtime communications between communicants operating on respective network nodes. The realtime kernel handles the complex tasks of connecting to communicants, virtual areas, and other network resources, switching those connections in response to user inputs, and mixing realtime data streams. The realtime kernel enables developers to focus on developing high-level communications functionality instead of low-level plumbing code. The realtime kernel imposes relatively low computational resource requirements so that realtime communications performance can be achieved using a wide range of computing devices and network connections that currently are available.
In some embodiments, the realtime kernel supports remote configuration and execution of audio and graphic rendering engines, as well as switching of realtime data streams in response to instructions (also referred to as definitions) that are received from a remotely hosted virtual area application. In this way, the realtime kernel enables virtual area designers to maintain control over the presentation of immersive virtual communication environments on remote client network nodes, thereby encouraging the development of a wide variety of different types of virtual areas and increasing the number of users who will want to adopt the communications system.
In some embodiments, the realtime kernel monitors the processing of a realtime data stream and adapts the processing based on deviations of the processing from a performance target. In this way, the realtime kernel increases the likelihood that realtime performance can be achieved regardless of the computing environment in which the realtime data stream processing is being performed.
In some embodiments, the realtime kernel implements a stream transport protocol that is efficient in connection and disconnection, as well as in transport. In some of these embodiments, the stream transport protocol provides a connection-oriented, encrypted connection over a connectionless transport protocol (e.g., UDP). The realtime kernel additionally provides between a client application and the transport layer a reconnection mechanism that automatically attempts to reestablish failed connections without intervention by the client application, thereby adding reliability on top of an inherently unreliable communication protocol.
In some embodiments, the realtime kernel has a plugin architecture that allows the functionalities of kernel components to be provided by one or more plugins that can be loaded dynamically on client network nodes. In this way, kernel components can be independently developed and remotely managed and updated. The plugin architecture additionally allows the installation footprint of the realtime kernel to be reduced substantially, thereby allowing the kernel to be installed on a wide range of client devices including those with significant computing and memory resource constraints.
A. Introduction
In the process of creating the shared virtual area communication environment, the area service 26 remotely configures the realtime kernels in the first and second client network nodes 12, 14 in accordance with the remote virtual area application 30 subject to a set of constraints 32 on the virtual area application 30. The constraints 32 typically include controls on access to the virtual area. The access controls typically are based on one or more of capabilities (where access is granted to communicants or client nodes having proper capabilities or permissions) and an access control list (where access is granted to communicants or client nodes having identities that are on the list). In some embodiments, the constraints 32 are administered by a security network infrastructure service (described below). Client software applications operating on the first and second client network nodes 12, 14 allow communicants to access the shared virtual area communication environment by presenting respective views of the virtual area in accordance with data received from the area service 26 via the realtime kernel 20 and by providing an interface for receiving commands from the communicants. The communicants typically are represented in the virtual area by respective avatars, which move about the virtual area in response to commands that are input by the communicants at their respective network nodes. Each communicant's view of the virtual area typically is presented from the perspective of the communicant's avatar, which increases the level of immersion that is experienced by the communicant. Each communicant typically is able to view any part of the virtual area around his or her avatar. The realtime kernels operating on the first and second client network nodes 12, 14 establish realtime data stream connections with other network nodes sharing the virtual area communication environment based at least in part on the positions of the communicants' avatars in the virtual area.
In some embodiments, the process of configuring the I/O hardware 24 involves dynamically configuring at least one of the stream handlers 22 in accordance with the instructions that are received from the remote network node 14 and the position data. For example, the virtual area application 30 may specify one or more audio effects that should be applied to audio streams that are associated with the objects currently in the virtual area, in which case the area service 26 sends to the realtime kernels executing on the first and second client network nodes 12, 14 instructions that configure their respective audio stream handlers to implement the specified effects in accordance with the positions of the respective objects in the virtual area.
The realtime kernel 20 processes realtime data streams that are associated with the communicant objects through each of the configured stream handlers 22 to produce a respective output 33. Depending on its content, the output 33 may be stored on a computer-readable medium or converted into a human-perceptible output by the I/O hardware operating on the first and second network nodes 12, 14. For example, audio output signals are converted into audible sounds by audio hardware (e.g., a sound card and speakers) and graphic output signals are converted into visible images by graphic hardware (e.g., a video card and a display). In some embodiments, the output 33 that is produced by at least one of the stream handlers 22 is processed by one or more downstream software components that in turn produce an output that either may be stored on a computer-readable medium or converted into a human-perceptible output.
B. An Exemplary Operating Environment
The realtime kernel 20 operates in the context of the virtual area communication environment 10, which includes the network 18 and a network infrastructure service environment that includes a number of network infrastructure services including the area service 26. The realtime kernel 20 and the network infrastructure service environment constitute a platform for creating virtual area communication environments for communicants.
1. Network Environment
The network 18 may include any of a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN) (e.g., the internet). The network 18 typically includes a number of different computing platforms and transport facilities that support the transmission of a wide variety of different media types (e.g., text, voice, audio, and video) between network nodes.
The realtime kernel 20 typically operates on a network node that includes software and hardware resources which—together with administrative policies, user preferences (including preferences regarding the exportation of the user's presence and the connection of the user to areas and connection targets), and other settings—define a local configuration that influences the administration of realtime connections with other network nodes. The network connections between network nodes may be arranged in a variety of different stream handling topologies, including a peer-to-peer architecture, a server-mediated architecture, and hybrid architectures that combine aspects of peer-to-peer and server-mediated architectures.
2. Network Infrastructure Services
One or more network infrastructure services typically cooperate with the realtime kernel 20 in the process of establishing and administering the network connections with other network nodes. The network infrastructure services may run on a single network node or may be distributed across multiple network nodes. The network infrastructure services typically run on one or more dedicated network nodes (e.g., a server computer or a network device that performs edge services, such as routing and switching). In some embodiments, however, one or more of the network infrastructure services run on at least one of the communicants' network nodes. Among the network infrastructure services that are included in the exemplary embodiment of the virtual area communication environment 10 are an account service, a security service, the area service 26, a rendezvous service, and an interaction service.
The account service manages communicant accounts in the network infrastructure service environment. The account service also manages the creation and issuance of authentication tokens that can be used by client network nodes to authenticate themselves to any of the network infrastructure services.
The security service controls communicants' access to the assets and other resources of the virtual area communication environment 10. The access control method implemented by the security service typically is based on one or more of capabilities (where access is granted to entities having proper capabilities or permissions) and an access control list (where access is granted to entities having identities that are on the list). After a particular communicant has been granted access to a resource, that communicant typically uses the functionality provided by the other network infrastructure services to interact in the virtual area communication environment 10.
The area service 26 administers a virtual area. In this process, the area service 26 manages connections that are associated with the virtual area subject to the capabilities of the requesting entities, maintains global state information for the virtual area, and serves as a data server for the client network nodes participating in a shared communication session in a context defined by the virtual area. The global state information includes a list of all the objects that are in the virtual area and their respective locations in the virtual area. The area service 26 sends instructions that configure the client network nodes. The area service 26 also registers and transmits initialization information to other client network nodes that request to join the communication session. In this process, the area service 26 transmits to each joining client network node a list of components (e.g., plugins) that are needed to render the virtual area 28 on the client network node in accordance with the virtual area application 30. The area service 26 also ensures that the client network nodes can synchronize to a global state if a communications fault occurs.
The rendezvous service manages the collection, storage, and distribution of presence information and provides mechanisms for network nodes to communicate with one another (e.g., by managing the distribution of connection handles) subject to the capabilities of the requesting entities. The rendezvous service typically stores the presence information in a presence database.
The interaction service maintains an interaction database that records interactions between communicants and supports queries on the interaction database subject to the capabilities of the requesting entities. For every interaction between communicants, one or more services in the virtual area communication environment 10 (e.g., the area service 26) transmit interaction data to the interaction service. In response, the interaction service generates one or more respective interaction records in the relationship database. Each interaction record describes the context of an interaction. For example, in some embodiments, an interaction record contains an identifier for each of the communicants, an identifier for the place of interaction (e.g., a virtual area instance), a description of the hierarchy of the interaction place (e.g., a description of how the interaction room relates to a larger area), start and end times of the interaction, and a list of all files and other streams shared during the interaction. Thus, for each realtime interaction, the interaction service tracks when it occurred, where it occurred, and what happens during the interaction in terms of communicants involved (e.g., entering and exiting), objects that are activated/deactivated, and the files that were shared.
The interaction service is able to present the results of queries on the interaction database records in a sorted order (e.g., most frequent or most recent) based on place. The query results can be used to drive a frequency sort of contacts whom a communicant has met in which virtual areas, as well as sorts of who the communicant has met with regardless of virtual area and sorts of the virtual areas the communicant frequents most often. The query results also may be used by application developers as part of a heuristic system that automates certain tasks based on relationships. An example of a heuristic of this type is a heuristic that permits communicants who have visited a particular virtual area more than five times to enter without knocking by default, or a heuristic that allows communicants who were present in an area at a particular time to modify and delete files created by another communicant who was present in the same area at the same time. Queries on the interaction database can be combined with other searches. For example, queries on the interaction database may be combined with queries on contact history data generated for interactions with contacts using a communication system (e.g., Skype, Facebook, and Flickr) that is outside the domain of the network infrastructure service environment.
3. Virtual Areas
The realtime kernel 20 administers the realtime connections with network nodes in a communication context that is defined by an instance of a virtual area. The virtual area instance may correspond to an abstract (non-geometric) virtual space that is defined with respect to abstract coordinates. Alternatively, the virtual area instance may correspond to a visual virtual space that is defined with respect to one-, two- or three-dimensional geometric coordinates that are associated with a particular visualization. Abstract virtual areas may or may not be associated with respective visualizations, whereas visual virtual areas are associated with respective visualizations.
As explained above, communicants typically are represented by respective avatars in a virtual area that has an associated visualization. The avatars move about the virtual area in response to input commands that are input by the communicants at their respective network nodes. The communicant's view of a virtual area instance that has an associated visualization typically is presented from the perspective of the communicant's avatar, and each communicant typically is able to view any part of the visual virtual area around his or her avatar, increasing the level of immersion that is experienced by the communicant.
As explained in detail below, the virtual area 66 includes zones 74, 76, 78, 82 that are associated with respective rules that govern the switching of realtime data streams between the network nodes that are represented by the avatars 68-72 in the virtual area 66. (During a typical communication session, the dashed lines demarcating the zones 74-82 in
A virtual area is defined by a specification that includes a description of geometric elements of the virtual area and one or more rules, including switching rules and governance rules. The switching rules govern realtime stream connections between the network nodes. The governance rules control a communicant's access to resources, such as the virtual area itself, regions with the virtual area, and objects within the virtual area. In some embodiments, the geometric elements of the virtual area are described in accordance with the COLLADA—Digital Asset Schema Release 1.4.1 Apr. 2006 specification (available from http://www.khronos.org/collada/), and the switching rules are described using an extensible markup language (XML) text format (referred to herein as a virtual space description format (VSDL)) in accordance with the COLLADA Streams Reference specification described in U.S. application Ser. Nos. 11/923,629 and 11/923,634.
The geometric elements of the virtual area typically include physical geometry and collision geometry of the virtual area. The physical geometry describes the shape of the virtual area. The physical geometry typically is formed from surfaces of triangles, quadrilaterals, or polygons. Colors and textures are mapped onto the physical geometry to create a more realistic appearance for the virtual area. Lighting effects may be provided, for example, by painting lights onto the visual geometry and modifying the texture, color, or intensity near the lights. The collision geometry describes invisible surfaces that determine the ways in which objects can move in the virtual area. The collision geometry may coincide with the visual geometry, correspond to a simpler approximation of the visual geometry, or relate to application-specific requirements of a virtual area designer.
The switching rules typically include a description of conditions for connecting sources and sinks of realtime data streams in terms of positions in the virtual area. Each rule typically includes attributes that define the realtime data stream type to which the rule applies and the location or locations in the virtual area where the rule applies. In some embodiments, each of the rules optionally may include one or more attributes that specify a required role of the source, a required role of the sink, a priority level of the stream, and a requested stream handling topology. In some embodiments, if there are no explicit switching rules defined for a particular part of the virtual area, one or more implicit or default switching rules may apply to that part of the virtual area. One exemplary default switching rule is a rule that connects every source to every compatible sink within an area, subject to policy rules. Policy rules may apply globally to all connections between the area clients or only to respective connections with individual area clients. An example of a policy rule is a proximity policy rule that only allows connections of sources with compatible sinks that are associated with respective objects that are within a prescribed distance (or radius) of each other in the virtual area.
In some embodiments, governance rules are associated with a virtual area to control who has access to the virtual area, who has access to its contents, what is the scope of that access to the contents of the virtual area (e.g., what can a user do with the contents), and what are the follow-on consequences of accessing those contents (e.g., record keeping, such as audit logs, and payment requirements). In some embodiments, an entire virtual area or a zone of the virtual area is associated with a “governance mesh.” In some embodiments, a governance mesh is implemented in a way that is analogous to the implementation of the zone mesh described in U.S. application Ser. Nos. 11/923,629 and 11/923,634. A governance mesh enables a software application developer to associate governance rules with a virtual area or a zone of a virtual area. This avoids the need for the creation of individual permissions for every file in a virtual area and avoids the need to deal with the complexity that potentially could arise when there is a need to treat the same document differently depending on the context.
In some embodiments, a virtual area is associated with a governance mesh that associates one or more zones of the virtual area with a digital rights management (DRM) function. The DRM function controls access to one or more of the virtual area or one or more zones within the virtual area or objects within the virtual area. The DRM function is triggered every time a communicant crosses a governance mesh boundary within the virtual area. The DRM function determines whether the triggering action is permitted and, if so, what is the scope of the permitted action, whether payment is needed, and whether audit records need to be generated. In an exemplary implementation of a virtual area, the associated governance mesh is configured such that if a communicant is able to enter the virtual area he or she is able to perform actions on all the documents that are associated with the virtual area, including manipulating the documents, viewing the documents, downloading the documents, deleting the documents, modifying the documents and re-uploading the documents. In this way, the virtual area can become a repository for information that was shared and discussed in the context defined by the virtual area.
Additional details regarding the specification of a virtual area are described in U.S. Application No. 61/042,714 (which was filed on Apr. 4, 2008), Ser. No. 11/923,629 (which was filed on Oct. 24, 2007), and Ser. No. 11/923,634 (which was filed on Oct. 24, 2007).
4. Other Platform Components
The realtime kernel 20 is designed to work as a component of a local network node as part of a client software package that additionally includes:
a. Heads-Up Display (HUD)
The Heads-up Display (HUD) is an application interface to the realtime kernel 20, which operates on each client network node. The HUD is a small, lightweight interface that a user can keep up and running all the time on his or her desktop. It is the user's interface for launching virtual area applications, providing him or her with immediate access to realtime contacts and realtime collaborative places (or areas). A virtual area is integrated with the user's desktop through the HUD and the realtime kernel 20 such that the user can drag and drop files into the virtual area communications environment, use files stored in association with the virtual area using the native client software applications independently of the virtual area communications environment while still present in a virtual area, and more generally treat presence and position within a virtual area as an aspect of their operating environment analogous to other operating system functions rather than just one of several applications.
In this embodiment, the HUD 84 is implemented by a substantially transparent (semi-translucent) user interface overlay that provides a persistent interface and access to controls. In the embodiment shown in
The communicant is able to work in a normal desktop computing environment while the realtime kernel 260 and the HUD 84 are running and ready to initiate realtime communications sessions. For example, the communicant may work with other applications, such as Microsoft® Excel®, to create a document that can later be shared in a realtime communications session. The virtual area 86 is integrated with the communicant's desktop such that the communicant can drag and drop files into the virtual area, use files stored in association with the virtual area using native client software applications independently of the virtual area communications environment while still present in a virtual area, and more generally treat presence and position within a virtual area as an aspect of the operating environment analogous to other operating system functions rather than one of several applications.
While the communicant interacts in the virtual area 86, the HUD 84 provides the communicant with independent control over his or her desired visualization. For example, a communicant may display a minimized view of the virtual area (minimized to the lower right-hand corner of the desktop) and participate in an audio conversation with another communicant in the virtual area while working in different application such as Microsoft® Excel®. A communicant then can change his or her visualization schema and enter into a more immersive three-dimensional rendering of the virtual area 86. This is accomplished by changing the setting of the progressive immersion slider 88 in the HUD 84 from “Desktop” to “3D”. Once in the 3D visualization mode, the communicant's desktop displays a 3D rendition of the virtual are 86 (as shown in
Any data that is associated with the virtual area 86 can be displayed on view screens 102, 104, 106. A view screen is a generic data rendering component that can be used to render any arbitrary data. Examples of the types of data that can be rendered on a view screen include:
As shown in
The HUD 84 shows an ordered set of place tiles 108, 110, 112. Clicking on one of the place tiles brings the user to the virtual area represented by the selected place tile. For people, we have a basic metaphor of Go (to the communicant's area) and Get (bring, them to the user's area). This is refined in the HUD 84 by allowing communicants to queue requests to go or get, as well as communicate with people via text or voice without “moving.” The HUD 84 notifies the communicant when each communication request is received from another communicant. The communicant can accept the request, ignore it, or add it to a communications queue. In this way, communicants can respond to non-priority communications at a later time. For example, a communicant can queue communications received during a time when the communicant is busy (e.g., engaged in a current communications session) and, after the communicant is free, the communicant can respond to communication requests in the communications queue.
As described above, the interaction network infrastructure service maintains an interaction database that records who the communicant meets with and where. The interaction service responds to queries on the relationship database with query results that may be presented in a sorted order (e.g., most frequent or most recent) based on place. In this way, the relationship database information can be used to drive a frequency sort of who the communicant met in which areas, as well as sorts of who the communicant has met with regardless of area, and sorts of the areas the communicant has frequented most often. This data is used in the HUD 84. This data also may be used by virtual area application developers as part of a heuristic system (e.g. a rule that permits people who have visited a particular virtual area more than five times to enter without knocking by default, or people who were present in a virtual area at a particular time to modify and delete files created by another communicant there at the same time).
In
Queries on the relationship database can be combined with other searches. For example, queries on the relationship database may be combined with queries on contact history data generated for interactions with contacts using another communication system (e.g., Skype, Facebook, and Flickr). In one example, the Skype virtual area 112 may be associated with a query on a communicant's relationship data associated with the Skype virtual area 112 and the communicant's Skype history data to produce a sorted list of realtime contacts of the user that are associated with the Skype virtual area 112.
Should any communicant exit a virtual area or enter a virtual area, the presence indicators (i.e., the sprites shown by circles, which typically are associated with names or other identifiers) in that virtual area will automatically be updated in realtime. This feature demonstrates the ability of a virtual area designer to put application-specific realtime data into a place tile. The place tile may appear either associated with a communicant, or with the communicant's places. For example, a game developer may export a map of where a communicant is in their game environment such that other people connected to that communicant through the relationship database receive a realtime feed of that communicant's current activities. These people can use this virtual area tile to navigate to that communicant, communicate with him or her, or contact him or her (e.g., send an invitation to enter a virtual area). The HUD 84 manages this interface to contacts and virtual areas for many different virtual areas simultaneously.
The realtime data used in the HUD virtual area tiles 84, 108, 110, 112 is provided by an interface that is managed by the area service hosting the relevant area via the realtime kernel 20. Each area service may provide a different respective HUD virtual area tile data feed to communicants based on the communicants' permissions to view the hosted virtual area. For example, if a communicant enters a virtual area that the communicant does not have permission to view, the HUD virtual area tile may show limited or no detailed information. In addition, the HUD virtual area tile data feed that is provided by the hosting area service may be customized by the virtual area provider operating that area service to present an application-specific view of the virtual area to subscribing HUDs.
b. Local Human Interface Device (HID) and Audio Playback Devices
The local HID devices enable a communicant to input commands and other signals into the client network node while participating in a virtual area communications session. Exemplary HID devices include a computer keyboard, a computer mouse, a touch screen display, and a microphone.
The audio playback devices enable a communicant to playback audio signals received during a virtual area communications session. Exemplary audio playback devices include audio processing hardware (e.g., a sound card) for manipulating (e.g., mixing and applying special effects) audio signals, and speakers for outputting sounds.
c. So3D Graphical Display, Avatar, and Physics Engine
The So3D engine is a three-dimensional visualization engine that controls the presentation of a respective view of a virtual area and objects in the virtual area on a display monitor. The So3D engine typically interfaces with a graphical user interface driver and the HID devices to present the views of the virtual area and to allow the communicant to control the operation of the HUD application.
The So3D engine typically receives graphics rendering instructions from the area service 26 via the realtime kernel 20. In some embodiments, the So3D engine also reads a communicant avatar database that contains images needed for rendering the communicant's avatar in the virtual area. Based on this information, the So3D engine generates a visual representation (i.e., an image) of the virtual area and the objects in the virtual area from the point of view (position and orientation) of the communicant's avatar in the virtual area. The visual representation typically is passes to the graphics rendering components of the operating system, which drive the graphics rendering hardware to render the visual representation of the virtual area on the client network node.
The communicant can control the presented view of the virtual area by transmitting commands from a HID device (e.g., a computer mouse) to the realtime kernel 20, which transmits view control commands to the So3D engine. The So3D engine updates the view of the virtual area in accordance with the view control commands. The So3D engine also updates the graphic representation of the virtual area on the display monitor in accordance with updated object position information that is received from the area service 26 via the realtime kernel 20.
d. System Database and Storage Facility
The system database and storage facility stores various kinds of information that is used by the platform. Exemplary information that typically is stored by the storage facility includes the presence database, the interaction database, an avatar database, a real user id (RUID) database, an art cache database, and a virtual area specification database. This information may be stored on a single network node or it may be distributed across multiple network nodes.
C. Exemplary Communication Session
Referring back to
In some embodiments, the area service 26 maintains global state information that includes a current specification of the virtual area, a current register of the objects that are in the virtual area, and a list of any stream mixes that currently are being generated by the network node that is hosting the area service 26. The objects register typically includes for each object in the virtual area a respective object identifier (e.g., a label that uniquely identifies the object), a connection handle (e.g., a URI, such as an IP address) that enables a network connection to be established with a network node that is associated with the object, and interface data that identifies the realtime data sources and sinks that are associated with the object (e.g., the sources and sinks of the network node that is associated with the object). The objects register also typically includes for each object one or more optional role identifiers, which may be assigned explicitly to the objects by either the communicants or the area service 26, or may be inferred from other attributes of the objects. In some embodiments, the objects register also includes the current position of each of the objects in the virtual area as determined by the area service 26 from an analysis of the realtime motion data streams received from the network nodes associated with objects in the virtual area. In this regard, the area service 26 receives realtime motion data streams from the network nodes associated with objects in the virtual area, tracks the communicants' avatars and other objects that enter, leave, and move around in the virtual area based on the motion data. The area service 26 updates the objects register in accordance with the current locations of the tracked objects.
In the process of administering realtime data stream connections with other network nodes, the area service 26 maintains for each of the client network nodes a set of configuration data, including interface data, a zone list, and the positions of the objects that currently are in the virtual area. The interface data includes for each object associated with each of the client network nodes a respective list of all the sources and sinks of realtime data stream types that are associated with the object. The zone list is a register of all the zones in the virtual area that currently are occupied by the avatar associated with the corresponding client network node. When a communicant first enters a virtual area, the area service 26 typically initializes the current object positions database with position initialization information. Thereafter, the area service 26 updates the current object positions database with the current positions of the objects in the virtual area as determined from an analysis of the realtime motion data streams received from the other client network nodes sharing the virtual area.
The area service 26 determines a set of target realtime data stream types that are defined for the zones in the occupied zones list (
In some exemplary embodiments, after the area service 26 has determined the set of realtime data stream data that enables the user to participate in a collaborative communication session with other network nodes in the shared virtual area instance (
In some of these embodiments, the area service 26 determines a realtime data stream handling topology that delivers the set of realtime data streams to the computer system 120 based at least in part on bandwidth capabilities of the computer system 120. In this process, the area service 26 determines a respective form in which to receive each of the realtime data streams from an unmixed realtime data stream and a stream mix derived from a combination of realtime data streams. The area service 26 also determines a network route over which each of the realtime streams is received from a direct peer-to-peer network route and a network route mediated by one or more of the other network nodes. After the stream handling topology has been determined, the area service 26 sends instructions to the realtime kernel operating on the computer system 120. The instructions specify the required realtime data stream connections between the computer system 120 and other ones of the network nodes in accordance with the determined stream handling topology.
In accordance with this method, the realtime kernel 20 determines if the computer system 120 has sufficient bandwidth to receive the set of required realtime data stream data 186 directly from the other network nodes (
If the available bandwidth is at least equal to the overall required bandwidth, the realtime kernel 20 establishes direct connections with the other network nodes that provide the required realtime data stream data (
If the available bandwidth is less than the required bandwidth (
A. Introduction
A communicant typically connects to the network 18 from a client network node, which typically is implemented by a general-purpose computer system or a dedicated communications computer system (or “console”, such as a network-enabled video game console). The network node executes communications processes that establish realtime data stream connections with other network nodes and typically executes visualization rendering processes that present a view of each virtual area entered by the communicant.
A communicant may interact (e.g., input commands or data) with the computer system 120 using one or more input devices 130 (e.g. one or more keyboards, computer mice, microphones, cameras, joysticks, physical motion sensors such Wii input devices, and touch pads). Information may be presented through a graphical user interface (GUI) that is presented to the communicant on a display monitor 132, which is controlled by a display controller 134. The computer system 120 also may include other input/output hardware 136 (e.g., peripheral output devices, such as speakers and a printer). The computer system 120 connects to other network nodes 138, 140, and 142 through a network adapter 138 (also referred to as a “network interface card” or NIC).
A number of program modules may be stored in the system memory 124, including an operating system (OS) 144 (e.g., the Windows XP® operating system available from Microsoft Corporation of Redmond, Washington U.S.A.), the realtime kernel 20, drivers 146 (e.g., a GUI driver), network protocols 148, a local software application 150 (e.g., the HUD 84), and data (e.g., input data, output data, program data, a registry 156, and the configuration settings 152).
B. Operating System
The operating system 144 hosts software applications by providing the base operating system services for creating a run-time execution environment on the computer system 120. Among the exemplary types of services that typically are provided by the operating system are resource management, file management, security, authentication, verification, notification, and user interfaces (e.g., windowing, menus, dialogs, etc.).
The services relating to the management of the resources (e.g., memory, processors, and I/O devices) of the computer system 120 typically are implemented by an operating system kernel. File management may be implemented by the operating system kernel or it may be implemented by a separate file system manager (e.g., the installable file system, which is provided in some Microsoft® Windows® operating systems). In the process of opening a file (e.g., a computer data file or a software application file), the file system manager typically calls an appropriate file system driver that looks up the disk storage location of the file in a database (e.g., a file allocation table, such as FAT, FAT98, VFAT, MFT, and CDFS) that maps out the storages locations of the file on the disk. Other operating system functions, such as security, authentication, verification, notification, and user interfaces, may be provided by one or more other components of the operating system (e.g., the executive services layer in some Microsoft® Windows® operating systems).
Among the exemplary types of services that typically are provided by the operating system kernel are process management, memory management, device management, and system call handling. Process management includes running applications and providing an application programming interface (API) to hardware components of the computer system. In the process of running a software application, the operating system kernel typically sets up an address space in memory for the software application, loads a file that contains the software application code into the address space, and executes the loaded software application code. Memory management involves managing software application accesses to the system memory 124. Device management involves providing access to hardware devices through device drivers. System call handling involves providing an API that exposes the operating system kernel services to user mode software applications. By invoking the API (e.g., through inter-process communication mechanisms and system calls), a software application can request a service from the operating system kernel, pass parameters, and receive results that are generated by the service in response to the request.
The operating system 144 typically stores hardware and software configuration information, user preferences, and setup information in the registry 156. For example, the registry 156 typically contains the following information: parameter values that are needed to boot and configure the system; system-wide software settings that control the operation of the operating system 144; a security database; and per-user profile settings. In some embodiments, the connection rules 32 are stored in the registry 156 instead of a separate database.
C. Network Protocols
The network protocols 148 control or enable the connection, communication, and transfer of data between the computer system 120 and other network nodes. Exemplary types of network protocols include the Transmission Control Protocol/Internet Protocol (TCP/IP), the User Datagram Protocol/Internet Protocol (UDP/IP), the realtime Transport Protocol (RTP), and the Session Initiation Protocol (SIP).
The TCP/IP includes a TCP portion and an IP portion. The TCP portion of the protocol provides the transport function by breaking a message into smaller packets, reassembling the packets at the other end of the communication network, and re-sending any packets that get lost along the way. The IP portion of the protocol provides the routing function by assigning to the data packets addresses for the destination network and the target node at the destination network. Each data packet that is communicated using TCP/IP includes a header portion that contains the TCP and IP information. The IP provides no guarantee of packet delivery to the upper layers of the communications stack. The TCP, on the other hand, provides a connection-oriented, end-to-end transport service with guaranteed, in-sequence packet delivery. In this way, the TCP protocol provides a reliable, transport layer connection.
The UDP is a message-oriented transport layer protocol that provides an interface between the application layer and the internet layer. UDP does not guarantee message delivery to the application layer. UDP is a connectionless protocol in that there is no effort made to setup a dedicated end-to-end connection. A UDP sender retains no state information about UDP messages after they are sent. Communication is based on transmission of messages in one direction from source to destination without checking the state of the receiver.
The RTP defines a standardized packet format for delivering audio and video over network connections. A variety of network protocols may be used in transmitting and receiving RTP data between network nodes, including peer-to-peer networking frameworks, a centralized server using TCP sockets alone or in combination with UDP, and multicast protocols.
The SIP provides means for users to locate one another, establish communicative sessions, and terminate active sessions. With a SIP transaction, session negotiations processes are handled in accordance with a Session Description Protocol (SDP).
D. Device Drivers
The device drivers 146 typically are implemented by software applications that enable other software applications (e.g., user-mode software applications and the operating system) to interact with hardware devices that are connected to the computer system 120. A device driver typically provides an API for functions that can be invoked by calls made by software processes in order to translate commands and data that are transferred between the software processes and the hardware device.
E. Realtime Kernel
1. Introduction
The realtime kernel 20 includes services that control the processing and switching of realtime data streams between the computer system 120 and other network nodes sharing a virtual area communication environment, as well as the presenting of a respective view of a virtual area and objects in the virtual area on the display monitor 132. In these processes, the realtime kernel interfaces with the operating system functions that communicate with the drivers 148 to translate commands and data to and from the hardware components of the computer system 120 in order to exchange realtime data streams with other network nodes and to present an immersive virtual area communication experience to the communicant.
Implementations of the realtime kernel 20 include one or more discrete modules or libraries (e.g., dynamic linked libraries) that are not limited to any particular hardware, firmware, or software configuration. In general, these modules may be implemented in any computing or data processing environment, including in digital electronic circuitry (e.g., an application-specific integrated circuit, such as a digital signal processor (DSP)) or in computer hardware, firmware, device driver, or software. In some embodiments, the functionalities of the modules are combined into a single data processing component. In some embodiments, the respective functionalities of each of one or more of the modules are performed by a respective set of multiple data processing components. In some implementations, process instructions (e.g., computer-readable code, such as computer software) for implementing the methods that are executed by the embodiments of the realtime kernel 20, as well as the data they generate, are stored in one or more computer-readable media. Storage devices suitable for tangibly embodying these instructions and data include all forms of non-volatile computer-readable memory, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks and removable hard disks, magneto-optical disks, DVD-ROM/RAM, and CD-ROM/RAM.
2. Exemplary Realtime Kernel Functionality
The realtime kernel 20 cooperates with one or more of the network infrastructure services in establishing and administering the network connections between the computer system 120 and the other network nodes sharing a virtual area communication environment. Among the exemplary functionalities that are involved in the process of establishing and administering network connections are session administration, remote-controlled stream handling, and realtime task scheduling.
e. Session Administration
In accordance with the method of
The realtime kernel 20 establishes a session with a network infrastructure service that hosts the designated virtual area (
The realtime kernel 20 subscribes to state data describing the current state of the virtual area instance (
In some embodiments, So3D engine of the realtime kernel 20 invokes a user interface service of the operating system 144 to render a human-perceptible view of the state data. For example, the So3D engine may invoke the interface service to render a representation of each of the communicants associated with objects currently in the area on the display 132. In some embodiments, the communicants may be represented by an icon, thumbnail image, or other graphic that optionally is labeled with the communicant's name. In some embodiments, the state data is presented in a graphical interface of a software application that triggered the invocation of the realtime kernel 20. In some embodiments, the state data is presented in an embodiment of the heads-up display (HUD) interface 84 (see
After a connection has been established with a virtual area instance, the software application that triggered the invocation of the realtime kernel 20 can give the user an option to request entry into the virtual area instance or can automatically request entry into the virtual area instance on behalf of the user.
In accordance with the method of
The realtime kernel 20 initiates transfer of at least one realtime data stream over at least one network connection with at least one realtime data stream source respectively associated with at least one object in the virtual area (
In the illustrated embodiment, the realtime kernel 20 processes the initiated realtime data streams in accordance with at least one stream handling definition in the specification of the virtual area instance (
f. Remote-Controlled Stream Handling
In accordance with the method of
g. Realtime Task Scheduling
In accordance with the method of
The realtime kernel 20 processes at least one realtime data stream that is sourced by the remote network node (
The realtime kernel 20 monitors the processing of the at least one realtime data stream (
In response to a determination that the processing of the at least one realtime data stream varies from a performance target, the realtime kernel 20 modifies the processing in accordance with a realtime performance targeting routine (
If the processing of the at least one realtime data stream satisfies the performance target (
In some embodiments, the realtime kernel 20 instantiates processing objects that perform respective ones of the data processing operations on the at least one realtime data stream. The realtime kernel 20 builds a directed graph from ones of the instantiated processing objects and processes the at least one realtime data stream through the directed graph. Depending on the realtime performance targeting routine, the realtime kernel 20 may modify the processing of the realtime data stream by pruning one or more of the instantiated processing objects from the directed graph. In some embodiments, the processing objects are assigned respective priority values, and the realtime kernel 20 prunes processing objects by removing ones of the instantiated processing objects from the directed graph based on the assigned priority values. For example, in some of these embodiments, the pruning includes removing from the directed graph ones of the instantiated processing objects having assigned respective priority values that fail to satisfy a priority threshold.
In some embodiments, the realtime kernel 20 builds from ones of the instantiated processing objects a second directed graph that is used to process a second realtime data stream that is sourced by one of the local network node and the at least one remote network node. In some of these embodiments, the first and second directed graphs are assigned respective priority values, and the realtime kernel modifies the processing of the first and second realtime data streams by preferentially modifying one of the first and second directed graphs based on the assigned priority values. For example, the realtime kernel may tear down the one of the first and second directed graphs that is assigned a lowest priority value.
In some embodiments, the realtime kernel 20 processes a second realtime data stream through the directed graph, where the second realtime data stream is sourced by one of the local network node and the at least one remote network node. In some of these embodiments, the first and second realtime data streams are assigned respective priority values, and the realtime kernel 20 preferentially modifies the processing of one of the first and second realtime data streams based on the assigned priority values.
In some embodiments, the realtime kernel 20 establishes respective realtime data stream connections between the local network node and multiple remote network nodes. The realtime kernel 20 processes through the directed graph realtime data streams that are sourced by respective ones of the remote network nodes. In some of these embodiments, the realtime data streams are assigned respective priority values, and the realtime kernel 20 preferentially modifies the processing of one or more of the realtime data streams based on the assigned priority values. The directed graph typically includes multiple directed chains of respective ones of the instantiated processing objects. The realtime kernel 20 typically processes a respective one of the realtime data streams through each of the directed chains. In some of these embodiments, the realtime kernel 20 iteratively modifies the processing of the realtime data streams until the processing is within the specified performance target. During each of the iterations, the modifying typically includes performing one or more of (i) removing one or more of the chains from the directed graph and (ii) pruning one or more of the instantiated processing objects from the directed graph.
A. Introduction
The realtime kernel 260 includes a set of managers and services. Among the realtime kernel managers are a connection and service mix manager 262, an area/zone manager 264, and a plugin manager 266. Among the realtime kernel services are a STRAW service 268, a SODA handler service 270, a media service 271, an audio stream service 272, a So3D interface service 274, an asset cache service 275, one or more social processors 277, a recording, playback, and transport bus service 276, a realtime scheduler service 278, a time service 280, a SIP service 282, a local HID/RDS driver handler service 284, and interface services for local audio devices including local audio playback 286, a local speaker 288, a local microphone 290, and Skype® audio. In one exemplary embodiment, the realtime kernel 260 is implemented by the following runtime package components:
B. Realtime Kernel Design
As shown in
The following sub-sections describe the installation, design, and operation of embodiments of the realtime kernel 260 and its components.
1. Installation
a. Overview
In some embodiments, the virtual area based rendering platform is downloaded as a software package over the internet as an installation package. It is delivered by HTTP download from a download server. On client network nodes operating a Microsoft® Windows® operating system, the platform software package is a .msi package. The initial installation is a single package that is amended on the download server as updates become available. When a new client network node performs a current installation, no other updates are needed until such time as subsequent updates are created.
The realtime kernel 260 makes use of plugins to customize applications. Necessary plugins are included in the installation package. From time to time components may be updated independently (e.g., realtime kernel services may be point-released and plugins may be added). In this case a separate Windows®.msi installation package may be created for the point release and registered with an update server. The installed platform software will be informed of the update and will offer the communicant the option of upgrading. Some communicants may delay upgrading until more than one update is available. When the communicant finally agrees to upgrade, all available updates will be loaded and applied sequentially.
Multiple versions of a plugin may be present on a client network node at the same time. This is because the client network node typically negotiates features, and chooses the plugin that suits the API and version requirements. Each plugin advertises its API and variants. The plugins will have different file names to avoid name collision in the file system. Two plugins with the same API and different variants are different implementations, and the choice is made by the service requesting the plugin (perhaps by negotiation with a server for instance). When a plugin is loaded with the same API and variant as an existing plugin, this is a bug-fix. The new plugin replaces the old one. Services are always replaced by an upgrade. There are never two services with the same API. The Windows® installation uses a manifest and bundled Windows® dependent DLLs, to guarantee a functioning product regardless of the update state of the Windows® environment. The Windows® side-by-side feature is used to avoid conflicting with other product installation requirements.
b. Update Server
The update server contains installation packages for each supported host operating environment, and upgrade packages for each previous supported installation package.
The client network node and the update server communicate over a reliable STRAW channel. The update server publishes available upgrade definitions for each supported host operating environment. The virtual area based rendering platform software that is installed on the client network node subsequently may subscribe to an upgrade. The update server begins sending the desired software piecemeal.
In some embodiments, a client version and upgrade tool is installed on each client network node to allow the user to see the current client software version, list available upgrades and start and monitor the upgrade process. The client network node will keep a table of the GUIDs of the upgrade packages that have been applied. It will present this list to the update server, and in return get a list of pending upgrades by GUID, in order of application. They will have attached description, size and date attributes.
An upgrade is marked “applied” only when the download completes, and the automatic install reports success. The automatic install process includes stopping any running SORK services so the DLLs can be overwritten. Downloading an upgrade is done through a sequence of SODA records, so the process can be interrupted and resumed without repeating any data transfer. The records include the upgrade GUID and an offset.
Since there is no requirement for rolling back or uninstalling, there is no need for any Microsoft® Windows® “side-by-side” library manifests. Any required libraries can be loaded as part of the upgrade.
The upgrade tool will make a registry entry or keep a file, containing the GUIDs of applied upgrades as well as the GUID of any current loading upgrade, its offset and a reference to the file on disk containing the data “so far”. Upgrade packages are deleted once applied. If it is desirable to cache an upgrade package for multiple client network nodes, then point them each at the same client proxy which will do the caching.
c. Update Local Database
The client network node stores the virtual area based rendering platform services and plugins in an asset directory in local file system. The services and plugins are self-describing, through APIs and attached resources. No further information is kept on the client network node software state. When a client network node reinstalls the virtual area based rendering platform software, perhaps after an OS upgrade, existing plugins typically are revalidated. The fresh installation includes all basic services and plugins, but there may be present on the machine optional or application-specific plugins, which typically are deleted or re-validated. In some embodiments, the binary content of valid plugins is hashed and one-way encrypted and the resulting value is stored as an attached resource that is used to check whether or not the plugins are authentic. To validate a suspected plugin, the current plugin content is rehashed and encrypted, and the resulting value is compared with the existing resource. If the content does not match the resource, then the plugin is invalid.
d. Client Authentication
Network authentication typically is made once each time the realtime kernel 260 is launched. In some embodiments, an account server running the account network infrastructure service is used to authenticate the communicant and establish a real user identifier (RUID) for the communicant. The account server creates a token (subsequently included as part of the RUID) and gives it to the client network node to authenticate itself to other servers. In this process, the client network node is securely issued a credential at installation time. The credential typically is a CA-defined certificate that is signed by a certificate authority. The certificate contains a private and public key. The virtual area based rendering platform installation package creates a new credential containing just the public key. The private key is stored securely on the client network node. The virtual area based rendering platform installation package creates a signature using the private key to encrypt a digest of a communicant-supplied password, and transmits the signature securely to the account server. The account server recovers the digest and stores it as the client identifying secret.
When establishing connections, the realtime kernel 260 shares the credential with the account server. The account server responds with its credential (e.g., a server-side certificate). The client network node and the account server validate the credentials using a registration authority. Once verified, the server-side credential is valid for any server anywhere.
In some embodiments, the account server also provides a random 128-bit challenge phrase to the client network node. The client network node hashes the challenge phrase with a cryptographic digest of the communicant-provided password and returns this as a response. The account server also hashes the challenge phrase with the previously-obtained digest for that communicant and verifies that the response from the client network node matches. The network connection is now authenticated and the communicant is identified as the owner of the private key.
In some embodiments, the account server assigns to the communicant a random Client ID with attached signatures. The signature is a 128-bit hash of the Client ID encrypted using the account server private key. The signature can only be created by the account server. Anyone receiving the token can validate the communicant by decrypting the digest using the public key published by the account server and comparing it with the Client ID.
e. Account Server Authentication
2. Initialization Sequence
In some embodiments, the following services of the realtime kernel 260 are loaded at boot time as Windows® service DLLs:
In these embodiments, services are loaded by name, not by GUID. Only one copy of each service is present on a client network node at one time. After loading, the SODA channel service 270, the media service 271, the audio stream service 272, the area/zone manager, and the realtime scheduler service wait idle. The connection and server mix manager leaves audio un-configured and waits for a definition of a connection to an area server. The default plugins are registered by GUID as API class objects. They are loaded when referenced by GUID in a definition. The HUD 84 contacts the account server, authenticates and identifies the communicant. The HUD 84 creates a stream to the rendezvous network infrastructure service and the interaction network infrastructure service and populates its most recently used (MRU) friends and area lists and its frequency friends and area lists. The asset cache service 275 typically contacts the art database server and begins to cache digital resources according to a heuristic and update its GUID map.
3. Sessions
The realtime kernel 260 manages sessions between the client network node and other network nodes. During a session, data is shared between a server and the client network node as SODA definition records over STRAW sockets. Data is shared in a publish/subscribe model. The realtime kernel 260 subscribes only to the data the client network node needs. To subscribe, the realtime kernel 260 creates a STRAW channel to the desired server. A STRAW channel is negotiated by well-known GUID for a particular virtual area. In some embodiments, STRAW sockets are connected using an address that is provided through a configured DNS.
The area service will send publish messages indicating the data streams that are available to the communicant, tagging each with a GUID handle. The realtime kernel 260 then sends subscribe messages for the desired data streams. Any changes to area service data for the subscribed channels are sent as SODA definition records to all client network nodes that have subscribed to those channels.
There are two primary types of sessions: (a) a HUD session, which involves displaying current relationship and presence information in the HUD 84; and (b) an area session, which involves either lurking or entering a virtual area instance.
a. HUD Session
In a HUD session, the HUD 84 contacts the account server, the RUID server, and the rendezvous server, and through STRAW channels subscribes to the communicant's own account and relationship information. The HUD 84 then subscribes to presence information for closely-related contacts and virtual areas. At this point the HUD 84 can display dynamic presence information for closely-related contacts.
b. Area Session
In an area session, the HUD 84 subscribes to information about related virtual areas. In some embodiments, a directory server is consulted to determine the current area server hosting a virtual area specified by the HUD. A STRAW stream is created to the current area server.
The HUD subscribes to the presence data that is associated with the virtual area and updates its 2D head-up display with the names of the other communicants currently participating in the virtual area. At this point the communicant is “lurking” in the virtual area. The presence of a communicant can be displayed in a pop-up list, and an icon is displayed in the HUD area representation (e.g., in the Office place tile shown in
If the communicant directs the HUD 84 to enter a virtual area, then the realtime kernel informs the rendezvous service of the communicant's request to enter the virtual area. Other communicants that are subscribed to the presence information that is associated with the virtual area are informed of the new communicant that has entered the virtual area.
The realtime kernel 260 directs the So3D engine to launch an interactive environment. The So3D engine subscribes to the area server environment data (e.g., rendering and motion data). The area server begins to stream the requested area server environment data to the realtime kernel 260. The realtime kernel passes the requested data to the So3D engine, which renders the data according to the current visualization mode (e.g., 2D overhead view, low-resolution view, or fully immersive 3D view).
The area server defines raw microphone audio media streams between the client network nodes that are associated with objects in the virtual area. The area server also creates definitions of audio mix elements according to audio handling instructions (e.g., spatial effects definitions and zone definitions) in the virtual area specification. The connection and server mix manager 262 listens for audio definitions, which include GUID handles for each P2P audio stream, and creates media streams for each definition. Each of the media streams is registered with the local transport bus 276 and appropriate audio mixing components are created by the audio stream service 272. The area/zone manager 264 also subscribes to SODA definitions for audio and for avatar motion and orientation data. The area/zone manager 264 controls gain/mute of each audio stream as the communicant's avatar navigates the virtual area.
In some embodiments, the area/zone manager 264 additionally subscribes to relationship data, which the area/zone manager 264 uses to control avatar orientation/movement/pose within the virtual area via social processors 277 (see
4. Managing Sessions
In accordance with the method of
On behalf of one or more software entities on the local network node, the STRAW service 268 automatically opens one or more channels over which data is transmitted between the local network node and the remote network node in the first session (
In the first session, the STRAW service 268 maintains a table that identifies open ones of the channels and associates respective attribute values with the identified channels (
The STRAW service 268 transmits data between the local network node and the remote network node on the one or more open channels in the session. In some embodiments, the data is transmitted in the form of records each of which includes a set of fields. Each of the fields of a record is defined by a respective field type and an associated field value, and each of the field types is identified by a respective Gum. Some of the records are media records that contain media data, which includes packets of renderable data. Other records are configuration records that contain configuration data, which includes definitions of configuration settings. The media records and the configuration records typically are encapsulated in transport records over the transport stream. The media records typically are compressed using a first data compression service and the configuration records typically are compressed using a second data compression service. On transmission, the STRAW service 268 associates the transport records with identifiers of respective ones of the channels on which they are transmitted, encrypts the transport records, and sequences the encrypted transport records. On reception, the STRAW service 268 decrypts the transport records and dispatches the media records and the configuration records contained in the decrypted transport records to subscribing ones of the software entities.
In response to a determination that the first session has failed, the STRAW service 268 automatically attempts to establish a second session with the remote network node on a second transport stream in accordance with the connectionless transport protocol (
In response to successful establishment of the second session, the STRAW service 268 automatically opens each of the channels identified in the table (
5. Processing Data Streams
The realtime kernel 260 supports remote configuration of stream handlers for processing data streams that are received by a client network node from other network nodes. In response to instructions that are received from the area service 26, various services and other components of the realtime kernel 260 cooperatively construct and configure directed graphs of processing elements into stream handlers that are used to process data streams. The area service instructions configure the stream handlers in accordance with a virtual area application being hosted by a virtual area that is managed by the area service 26.
In accordance with the method of
The connection and server mix manager 262 instantiates realtime stream handling objects corresponding to respective ones of the identifiers (
The transport bus 276 creates a directed graph that includes ones of the instantiated realtime stream handling objects in accordance with the specification (
The STRAW service 268 receives a realtime data stream from an input source corresponding to the input source identifier (
The stream handler produces a resultant data stream at an output sink corresponding to the output sink identifier (
6. Services and Other Components of the Realtime Kernel
The components of the realtime kernel 260 include services, plugins, and libraries.
a. Compressor Library
The compressor library implements an optional compression layer for transport data. It is not intended to compress protocol headers or link negotiation exchanges.
The compressor library is used by the SODA channel service 270 and by the media service 271. When encryption is configured, these services 270, 271 create two compressor instances and pipe channel data through. One compressor instance is used for transmit and the other compressor instance is used for receive.
The compressor library uses a compression/decompression plugin that is configured by variant. It is up to the service to negotiate the compression variant, and provide it to compressor.
b. Audio Stream Service
In some embodiments, the audio stream service 272 is a Windows® service DLL.
The audio stream service 272 manages audio stream mixing. It defines APIs for creating and configuring audio processing graph elements (also referred to as AudioComponent objects), which are manipulated by the area/zone manager 264. The audio stream service 272 is a client of the transport bus 276. All audio processing graph elements are registered with the transport bus 276.
Audio processing graph elements are created through the plugin manager 266 (PlugInMgr) via the following API calls:
The invocation of the PlugInMgr::CreateInstance( ) API yields a guidPlugin, representing an instance of the desired variant of the API.
The plugin APIs used for audio processing graph elements are:
The caller supplies the variant, which is just passed along by the audio stream service 272. The variant represents an implementation of an audio processing graph element, as defined in the virtual area application. The audio stream service 272 then creates an AudioComponent object that encapsulates the plugin instance (guidPlugin). The AudioComponent uses the plugin manager 266 to access the methods of the plugin. The audio stream service 272 creates the correct type of derived AudioComponent for each audio API:
AudioComponent objects are registered with the transport bus 276. The transport bus API is used to link components into a graph. The AudioComponent API supports the operations of the transport bus 276 and the area/zone manager 264. The base class AudioComponent has the API of an AudioSource and an AudioSink, both of which are aspects of the same plugin.
c. STRAW Service
(i) Overview
In some embodiments, the STRAW service 268 is a Windows® service DLL.
The STRAW service 268 implements a STRAW transport protocol that enables connection-oriented, encrypted secure socket connections between network nodes over a connectionless transport protocol (e.g., UDP). The STRAW transport protocol uses fixed-length globally unique identifiers (GUIDs) to identify all records and all field types in the records. For example, in some embodiments, a network node (or station) is defined by an IP_Address and a Port. In these embodiments, a STRAW station identification record defines a particular network node with the following set of GUIDs: {GUID1, GUID2, GUID3, GUID4, GUID5, GUIDE}, where
Referring to
The STRAW service 268 manages two kinds of channels: media channels that contain streaming data (e.g., audio); and SODA channels that contain SODA records of definitions (or instructions). STRAW records encapsulate SODA records and media records over a stream. STRAW records are encrypted, sequenced, and include a message integrity field. The sequence is independent of the record source or purpose—it is a link-level feature used to detect out-of-order or missing records.
STRAW records are identified by channel. GUIDs are used as channel identifiers. SODA and media records may be compressed at the channel level, as a stream irrespective of STRAW record encapsulation. Each SODA record contains one or more SODA definitions 344. Examples of SODA definitions include processing graph elements (e.g., AudioMix and AudioEffect), 3D rendering assets (e.g., texture and mesh), and RDS (e.g., avatar motion checkpoints). Each media record contains one media packet 346. Examples of media packets include audio codec and text.
Applications publish channels on a session using a well-known GUID ID. Kernel managers subscribe to channels. The publish/subscribe model is connectionless. A kernel manager that subscribes to a channel registers to receive notification of channel state changes and channel records as they arrive.
(ii) Stream Vs. Session Vs. Channel Vs. Record
In the context of STRAW sessions, a stream is a bi-directional UDP socket between two network nodes defined by two IP address/port pairs, and a transport GUID. A stream supports sessions of channels. A session is a logical node-to-node connection. Sessions transport channels for the two nodes. Sessions may pass through one or more proxy stations and are transported over streams that may contain multiple sessions.
A channel is a logical construct that transfers SODA or media records between two network nodes in a session. A channel can be reliable or unreliable, compressed or non-compressed. The content of a channel is identified by a content GUID. Channel records are transported in a sequence of STRAW records sharing the same header CHANNEL_CLIENT ID and with sequential packet numbers and a MAC field. The MAC calculation depends upon the packet sequence on the given channel in one direction only. All records transmitted on a single channel share a single set of configuration parameters (e.g., {Client, reliable, compressed}). Records on a single channel are compressed as a serial stream. Only reliable channels normally can be compressed. In some embodiments unreliable channels can be compressed with a compression process in which compression restarts on each key frame. In the case of a lost packet on an unreliable channel, records on that channel are discarded until a key frame is reached (because they cannot be decompressed out of order).
Compression uses Compress.lib. To improve compression a channel definition can include preload data, which is run through the compressor but not transmitted. The purpose is to prime the compression state tables with common phrases. The compression state table is reset and rebuilt each time a key frame is received.
(iii) SODA Records
SODA records are nested structures with an initial GUID ID and one or more SODA definitions. A SODA definition has a definition type, a definition length and one or more fields. The definition type is a well-known GUID (e.g., guidAsset). The length indicates total size of fields. Fields are a combination of type-specific fixed fields and nested SODA definitions. That is,
SODA Record:
SODA Definition:
Field:
Or SODA Definition
For example,
SODA records are encapsulated within a STRAW record:
(iv) Channel Reliability and Link-Level Protocol
STRAW records are numbered and contain a channel ID. After receiving a packet and after a short time delay the transport sends an ACK record containing the number of the next expected packet for each channel so that the sender can confirm transmitted records were received and can release local resources. There is no reliability feature for this ACK beyond periodic transmission. This scheme uses the minimum network resources for reliability, assuming that almost all records are successfully received.
A MAC field is calculated for each STRAW record transmitted. It is checked on receive.
For Reliable Channels:
If records in a channel are received out-of-order, a NACK is transmitted for the missing record. A MAC failure also results in a NACK being transmitted for the expected record. Up to four NACKs for a single record are permitted, and then the transport queues a failure message to any subscribing kernel managers and erases the channel definition.
For Unreliable Channels:
If records in a channel are received out-of-order, the missed packet number is signaled to any managers subscribed to the channel and no NACK is sent. A MAC failure is also indicated as a missed packet to any kernel managers subscribed to the channel and no NACK is sent. There is no threshold for missed packets, and the channel is never closed by the transport.
There is no need to “close” a channel. If all kernel managers unsubscribe, then data transmission over the channel stops. Since a channel is a logical entity, no operating system resources are used.
(v) Publish/Subscribe
The STRAW service 268 maintains a list of local publish and subscribe entries. Each entry contains
The list is initialized with
The STRAW service 268 also maintains a table of all arrived publish definitions, for use in case a late subscribe is registered in the local list.
When the STRAW service 268 receives a session definition for a desired connection to another station, the STRAW service 268 establishes the stream, sends the session definition, and then sends all of the local table publish entries in a SODA record on the session channel. When a publish definition arrives at a STRAW service 268, the STRAW service 268 enters that definition into the publish definition table and then sends a subscribe definition on the session channel for each subscribe entry in the local list that had a matching Channel ID in the publish record. When a subscribe definition arrives, the STRAW service 268 begins sending definition updates (piped from the publishing Applications) on the given channel as STRAW records containing the SODA record for that definition. The records may be sent on more than one channel.
When a kernel manager desires to participate in a channel with a server, the kernel manager defines a subscribe request, whether or not any STRAW Streams exist to any servers. If a virtual area application publishes later (i.e., after stream is established) then the change in the local table triggers re-sending of the publish entries in the table, which automatically triggers any latent subscribe on the other end of the link. If a kernel manager subscribes later and there is an entry in the publish table, then the STRAW service 268 sends the subscribe request automatically. This process ensures that channel data is sent over a link only if it is desired by the receiver.
(vi) Channel Record Dispatching
STRAW records are decrypted as they arrive. If valid, their embedded records are uncompressed and then dispatched to all subscribing kernel managers. The list of local subscribe entries is examined, and all entries matching the Channel ID (in the subscribe transport info) receive a copy of the record on their message queue.
The subscribing kernel manager is responsible for freeing messages as they are processed. The bulk data portion of a message is not copied, but points to the original network buffer containing the STRAW Record. Each kernel manager frees messages such that when they are all freed the network buffer can be recycled.
(vii) Establishing a STRAW Stream
The client network node connects sessions over streams with servers and peer network nodes. In this process, each party authenticates itself to the other.
STRAW streams are authentic and secure. This means that:
Part of a session definition is a list of stream transport plugin GUIDs. If the client network node responding to the definition supports at least one of the GUIDs, it loads the plugin and uses it to establish the session. The server creating the definition may examine the support list of each client network node involved and decide which transport plugin GUID to include in the definition.
Part of a session definition is a list of stream encryption plugin GUIDs. If the client network node responding to the definition supports at least one of the GUIDs, it loads the plugin and uses it to encrypt the Session. The server creating the definition may examine the support list of each client network node involved and decide which stream encryption pluginGUID to include in the definition.
(viii) Server Stream
In some embodiments, a stream from a client network node 344 to a server 346 is established using an address that is obtained from a server, such as a directory server, a map server, or an area server. Exemplary purposes of the stream include obtaining presence information, rendering a public space using rendering definitions from a map server, and rendering a virtual area using rendering definitions from an area server.
(ix) Client Stream
Referring to
If the stream or session fails for one of the client network nodes 358, 360, that client network node informs the area server 356 of the failure using a SessionFailure SODA definition. Reasons for failure include, for example, no compatible transport, no available channel, and reliable channel failure. In some embodiments, the area server 356 responds to the SessionFailure SODA definition by attempting to re-route the stream (e.g., by reflecting an audio stream through a proxy or server).
In some embodiments, the client network nodes 358, 360 communicate P2P in accordance with the Simple Traversal of UDP through Network Address Translators (NATs) (abbreviated STUN) network protocol. In these embodiments, the clients 358, 360 operate through respective NATs. A server (e.g., the area server 356) acts as a STUN server, which listens at two IP addresses in the network on the public side of the NATs and reports the mapped IP addresses and ports on the outside of the NATs. From this information, the client network nodes 358, 360 are able to discover the presence and specific type of NAT, and obtain the mapped (external) IP address (NAT address) and port number that the NAT has allocated for the clients' UDP connections to remote hosts. The client network nodes 358, 360 then use the external IP addresses to communicate with one another P2P in accordance with the UDP protocol. Additional details regarding the STUN protocol can be obtained from Jonathan Rosenberg et al., “STUN—Simple Traversal of User Datagram Protocol (UDP) Through Network Address Translators (NATs),” Internet proposed standard RFC 3489 (March 2003).
(x) Keep-Alive
In some embodiments, once a stream is established, the transport on the client network node issues periodic idle StreamKeepAlive definitions. The partner network node returns a StreamKeepAlive definition with the timeout set to whatever it estimates is the maximum interval it can tolerate. The purpose of this message is to keep any NAT firewall pinhole active. The partner network node records the desired timeout and lengthens the interval each time. If the next message comes from a different IP and port, then the NAT timed out and a new pinhole was created by the keepalive. The interval should then be shortened.
If either the client network node or the partner network node notices a StreamKeepAlive is missing either because the idle timer expired and no message was received, or no answer was received to a keepalive message, then it issues an immediate Stream KeepAlive with a very small timeout. This is to distinguish between a dropped station and a dropped UDP packet. Several retries can be attempted. If no answer is received after the retries then a local StreamFailure event is generated with the failed Session ID(s), and the stream definition is deleted.
In some embodiments, the STRAW service 268 responds to a broken link by automatically reestablishing the link and re-linking all subscriptions and data flows based on the local publish and subscribe table entries without application (or client) intervention.
f. Media Channel Service
In some embodiments, the media channel service 271 is a Windows® service DLL.
The media channel service 271 is used to robustly communicate P2P audio and text chat. The media channel service 271 will compress the stream if configured to do so. The compressor library accomplishes compression as requested. The media channel service uses an audio codec plugin configured by variant. The variant GUID is taken from the stream definition.
g. SODA Channel Service
In some embodiments, the SODA channel service 270 is a Windows® service DLL.
The SODA channel service 270 is used to robustly communicate SODA definitions. The SODA channel service 270 will compress the stream if configured to do so. The compressor library accomplishes compression as requested. This is the place where data structures are converted to network byte order. In some embodiments, little-endian (Intel) network byte order is used.
h. Connection and server mix manager
In some embodiments, the connection and server mix manager 262 is a Windows® service DLL.
In some embodiments, the connection and server mix manager 262 exports a procedural API available for the local conference scheduling tool to initiate a session with an area server. This API is the initial entry point for creating a session. The connection and server mix manager 262 publishes an active session definition. The area server receives the session definition on a SODA channel.
The connection and server mix manager 262 also constructs audio graphs from audio graph processing elements. The audio graph processing elements are configured by the area service 26, either directly through SODA records or indirectly through VSDL scripts. In any case SODA definitions are the result. In some embodiments, the connection and server mix manager 262 processes the following SODA definitions sent by area service 26:
These SODA definitions are described in the following paragraphs.
AudioDevice is a definition of a local audio device to be registered with the audio transport bus (which is a component of the transport bus 276) as an AudioSource. The well-known ID of a standard local audio source (microphone, headset) or a registered local audio recording (file, streaming CD audio) is provided, along with an instance ID (which headset if more than one is available). The device is given a new ID for use on the audio transport bus. The connection and server mix manager 262 creates an instance of the appropriate variant of the AudioDevice plug-in using the well-known audio source ID, and hands it off to the audio transport bus.
AudioStream is a definition of an incoming audio stream to be registered with the audio transport bus as an AudioSource. An incoming audio stream is defined by the Channel ID it is transported over. For purposes of dynamic mixing (done by the area/zone manager 264) it is necessary to associate a stream with an Avatar ID. The device uses the Channel ID as its audio transport bus ID. The connection and server mix manager 262 creates an instance of the appropriate variant of the AudioStream plug-in based on the Channel Type ID, and hands it off to the audio transport bus.
AudioMix is a definition of a combination AudioSource and AudioSink plug-in. The definition fully specifies the plug-in API ID, variant ID, one or two Audio Transport Bus source IDs, and an associated Avatar ID (for the area/zone manager 264). The connection and server mix manager 262 creates the indicated variant of the AudioMix plug-in based on the IDs provided, and hands it off to the Audio Transport Bus.
AudioEffect is a definition of a combination AudioSource and AudioSink plug-in. The definition fully specifies the plug-in API ID, variant ID, one audio transport bus Source IDs, and an associated Avatar ID (for the area/zone manager 264). The connection and server mix manager 262 creates the indicated variant of the AudioEffect plug-in based on the IDs provided, and hands it off to the audio transport bus.
AudioCalculation is a definition of an AudioCalculation plug-in. The definition fully specifies the plug-in API ID, variant ID, associated audio transport bus AudioSource object ID, the component's own audio transport bus ID, and two situation-specific parameters. The AudioCalculation objects are not processing audio data directly in audio chains. Instead the AudioCalculation objects calculate settings for other audio graph components based on a “domain object model”, external information such as manual settings (mute, volume control in the HUD), avatar position and motion, reverb spaces and Windows® settings (speaker selection in the Control Panel for instance). AudioCalculation objects are executed on a different rendering timer event—much less often than normal audio rendering. This is because the data they use as inputs to calculations change slowly. The connection and server mix manager 262 creates the indicated variant of the AudioCalculation plug-in based on the IDs provided, and hands it off to the audio transport bus.
AudioRecord is a definition of an AudioS ink plug-in. The definition links a point in the audio graph with a storage component. At rendering time, an AudioRecord component doesn't trigger rendering itself. But if rendering is triggered by another AudioSink component, then the rendered audio data is provided to the AudioRecord object for transfer to the indicated storage component. The connection and server mix manager 262 creates the AudioSink plug-in and hands it off to the audio transport bus.
AudioPlayback is a definition of an AudioSource plug-in. The definition links a point in the audio graph with a storage component. If an audio chain references this component, then at frame-preparation time one time-slice worth of audio data is fetched from the storage component and provided as output of this component. The connection and server mix manager 262 creates the AudioSource plug-in and hands it off to the audio transport bus.
The connection and server mix manager 262 configures the transport bus 276 and the audio stream service 272 according to definitions received from the area server. Each definition results in the creation of an audio processing graph element, which is an audio stream plugin, an audio calculation plugin, or an audio source plugin. The local audio devices (e.g., microphone, speaker(s), and Skype audio) can be configured according to settings chosen through a HID configuration tool. The HID configuration tool allows the user to choose keyboard and mouse options for navigating the 3D collaborative space. For example, shortcut keys may be defined, and mouse gestures bound to avatar behaviors. In some embodiments, the audio output selection follows the Windows® Control Panel “Sounds and Audio Device” settings for Audio and Voice. This ensures that the same audio settings are used for virtual area communications as are used for ordinary VOIP conversations.
In the event of an area server session transport failure, the connection and server mix manager 262 attempts recovery. It tears down the session and re-launches it on a different area server. In the event of a media stream failure, the connection and server mix manager 262 attempts recovery. In this process, the connection and server mix manager 262 tries to reconnect to the client network node. If the reconnection attempt fails, the connection and server mix manager 262 defines the communicant state as inaudible to the area server.
i. Area/Zone Manager
In some embodiments, the area/zone manager 264 is a Windows® service DLL.
The area/zone manager 264 adjusts the audio stream service 272 mix parameters according to zone definitions and avatar position definitions. The area service 26 publishes to the area/zone manager 264 SODA definitions that relate each avatar to the audio processing graph element that responds to that avatar's motion. The avatar position data is used to mix the audio streams from each of the client network nodes participating in a virtual area in a way that allows each communicant to hear the other communicants at the right audio location with the right volume according to local zone definitions. The parameter values that are applied to the audio processing graph elements typically depend upon a calculation that includes relative position, orientation of communicants, zone definitions, audio properties of the virtual area, and manual settings (e.g., mute, volume) that are configured by the communicant.
In some embodiments, the area/zone manager 264 processes the following SODA definitions, which pertain to gross features of the simulated audio space in which the current audio graph is rendering.
These SODA definitions are described in the following paragraphs.
AudioReverb is a definition of a reverb space which is a “hollow space” that results in a certain reverb or echo effect. The definition identifies a simple geometry with a location. This definition is provided in a domain object model (DOM) to all AudioCalculation objects when they are invoked.
AudioAperture is a definition of a connection between two reverb spaces. It identifies two reverb spaces by ID, and specifies an audio connection between them. The connection is a circle at a certain location and orientation. This definition is provided in a domain object model (DOM) to all AudioCalculation objects when they are invoked.
AudioObstruction is a definition of a physical barrier to sound propagation. It is modeled as a sphere at a location. This definition is provided in a domain object model (DOM) to all AudioCalculation objects when they are invoked.
The SODA definitions described above are inputs for AudioCalculation objects, which are scriptable calculation plugins that take the following parameters as an argument:
Initial Audio Calculation plugins include:
Some calculations are appropriate for individual audio sources; some for whole-room final mix. The virtual area application can introduce new plugins at will by referring to them in audio definitions. The area/zone manager 264 will subscribe to plugins that it doesn't have, and receive their definition from the area server.
The arrows 382, 384, 386, 388, 390 represent AudioSources, which are all dry mono audio sources. Avatars 1, 2 and 3 are network streams from remote client network nodes. Whisper is an optional local audio feed from a designated source. Everything to the left of the audio Panners is mono with a series of effects added. These effects include adjusting volume according to Zone and speaker Orientation and applying a Doppler shift to account for relative velocity of speaker and listener. The audio Panners position each adjusted mono signal in the three hundred sixty degree audio space of the currently occupied zone of a virtual area. The Location of the speaker relative to the listener is used. Everything to the right of an audio Panner is 5.1 audio. The Room audio processing graph element calculates the effect of the room acoustics on the audio signal. It takes into account position of speaker and listener, room characteristics, and obstructions. The Final Mix audio processing graph element adds all of the processed audio signals together to produce a resultant stream that is piped to the designated audio output device (i.e., SPKR, which represents the local speaker(s) in the illustrated example).
Some audio processing graph elements (inserts) have fixed parameters and, therefore, are not associated with any runtime calculation plugin scripts. These elements include echo and noise cancellation, automatic gain control (AGC), silence detection, fixed-source Panner, and Final Mix.
j. Asset Cache Service
In some embodiments, the asset cache service 275 is a Windows® service DLL.
Assets are recorded in a local database or table service indexed by GUID. The asset data is kept in an amorphous storage pool. The ID on a particular asset never changes, thereby avoiding any cache consistency issues. Assets are cached by SODA record. This means that large assets may be stored in many pieces, since SODA records are limited in size to the UDP MTU (around 1000 bytes). The asset cache must index records by GUID and the data offset, which is a field in the SODA record.
(i) Asset Index
Assets are represented in a class table, and an optional attribute table.
The asset class table maps the asset GUID to a class GUID and a data store reference.
The asset attribute table attaches a tagged attribute scalar value to an asset.
(ii) Data Store
The asset storage interface of the virtual area based rendering platform allows heaping data, storing a separate index transactionally, and scavenging unused asset storage for reuse. A combination of a database and file will serve for asset storage. The database contains two tables: a storage allocation table, and a GUID/offset index. The file is created with a fixed size according to the cache size configured. Assets are stored in the file using a heap-bucket algorithm.
The asset cache data store will be indexed by GUID and offset. The heap is partitioned into buckets by data size, in powers of two. The smallest bucket is 32 bytes; the largest is 2 kilobytes, which makes a total of 7 buckets. Each bucket is budgeted the same amount of storage, which means there are half as many items in each successive bucket. Each bucket is a heap table large enough to hash enough items to fulfill the storage budget, which makes the chance of hash collision reasonably small.
(iii) SODA Asset Definition Records
Assets are encapsulated in SODA records for transmission. The definition includes the asset GUID, its content (unstructured data) and it's offset if it is larger than one record, and a table of attributes. A SODA record encapsulating an asset never contains any reference to the storage system.
The offset and data fields can be repeated as long as they fit in one record.
A query for an asset is a definition
An asset is forgotten using a definition
(iv) Attribute Hierarchy
Assets have attributes, the most important of which are Type and Default. Type specifies the purpose of the asset. Default specifies a base asset that can be used in place of the given asset. An exemplary attribute hierarchy is shown in the following table:
In this example, a virtual area scene that includes the SococoTable texture will be rendered using the first texture available, searching from SococoTable through ConfTableWood, then TableTop1, etc.
The Artbase will have a huge hierarchical tree of assets based on default assets, all finally based on a small number of fundamental assets. These fundamental assets are installed as part of the client software package. This feature is intended to allow designing a level with specific art assets called out, and to allow rendering it before all of the art assets are actually designed. Also it may be desirable to start rendering a virtual area before all of the art assets are loaded.
Optional attributes include Default, Author, Application and Collada ID which is a reference to the Collada source from which the asset is derived. Browsing tools running on an author station will index assets by any and all Attributes.
k. Audio Transport Bus
The audio transport bus is a component of the transport bus 276 that handles audio streams. In some embodiments, the audio transport bus is implemented by a library that manages an audio graph as a collection of component objects. All of the audio graph objects are registered with the audio transport bus by a unique ID. The audio transport bus is responsible for managing the audio graph objects when rendering audio. The audio transport bus traces the audio graph components by ID. In this process, the audio transport bus invokes each audio graph component in turn, providing the audio data from the input component named by ID.
The audio transport bus buffers one time-interval of each audio stream available on the client network node. The audio transport bus feeds these streams to zero or more subscribers as configured by the audio stream service 272. Streaming data uses a pull model, where the final output stage calls preceding stages for data as needed. Each stage calls the one before until the original audio stream source is reached. If the source needs to control rate (flow control) it typically does its own buffering and has its own particular signaling scheme. For instance a local file source can double-buffer and read ahead one time-interval while processing the preceding one. A network file source can signal flow rates and buffer limits over the network to the server. A local microphone source, on the other hand, has no ability to control flow rate at all.
The audio transport bus operates in two phases: upon a rendering timer event, it provides existing rendered data to AudioSink components; the audio transport bus then traverses the audio graph, causing the next time-slice worth of audio data to be rendered and buffered. This technique gives the audio graph a good chance of providing continuous playback, even in the presence of variable-latency audio source data.
In some embodiments, the audio transport bus measures the rendering latency of each audio graph component, and aggregates each rendering chain latency by adding up all dependent (source) audio component latencies. The audio transport bus collects and registers the rendering latency statistics. Based on these statistics, the realtime scheduler 278 determines when and how the audio graph should be modified in order to achieve an audio graph processing target. In some embodiments, the realtime scheduler 278 executes one or more of the methods described above in connection with
Another function of the audio transport bus is to invoke AudioCalculation objects periodically. The AudioCalculation objects are used to change settings of associated ones of the audio graph processing elements. The period of AudioCalculation execution typically is much longer (less often) than the audio graph rendering period.
The audio transport bus typically has the ability to record streams and replay recorded streams. The raw audio streams typically are recorded so that during playback the mix can be re-rendered according to the viewer's point of view. Some embodiments include a hub that receives all of the raw audio streams. In these embodiments, the hub typically handles the recording of sessions. When it is not desirable to re-render a session, the audio transport bus typically only records audio streams at the client network node.
The AudioSource object is the base for all audio sources. This object delivers data when polled, and defines its desired latency and channels (e.g., mono, stereo, 5.1). Derived objects include Microphone, MediaStream, Clip, WaveFile, DirectX audio, and the output side of the Mix plugins.
The AudioSink object is the base object for audio output devices. This object requests data from an AudioSource when polled. Derived objects include Speaker, MediaStream and the input side of the Mix plugins.
(i) Audio Plugin API
In some embodiments, the audio plugins incorporate VST audio effect C++ objects that are available from Steinberg Media Technologies GmbH. In particular, the audio plugins incorporate VST objects wrapped as plugins. A shim library is provided for wrapping a VST object as an audio plugin. This wrapper supplies the audio plugin API. The API of the VST objects will be used as the audio plugin class-specific API. This API includes:
In these embodiments, a VST plugin is wrapped as an AudioSource and AudioSink. For instance, the AudioSource::Frame(data&, size) call will be implemented as a call to the preceding AudioSource::Frame(data&, size&), followed by setupProcessing(process) and process(data&). The Configuration(latency&, channelLayout&) call is implemented in terms of getLatencySamples( ) and getBusArrangements(output, i, channelLayout&) for each supported channel. The existence of a wrapper means that VST source code is required to shim an existing VST plugin into the audio bus.
(ii) OpenAL
Most audio processing graph mixing and effect elements are executed using the OpenAL cross-platform audio API available from www.openal.org. The OpenAL library is capable of calculating all parameters listed above in the area/zone manager section using the best features of the available soundcard to implement the features. In particular, OpenAL Sources, Listeners, and program Buffers are created for each operation in the audio graph from mute through final mix. Before each update, the buffer parameters are modified according to the calculation plugins.
In some embodiments the GIPS componentized audio library available from Global IP Solutions, Inc. is used for implementing stream-processing components (inserts). The GIPS audio library directly supports the following audio plugins: Codecs, Error Concealment, Jitter control, Echo and Noise cancellation, AGC, and Silence detection.
l. Local Stream Sources
Local Stream sources are the microphone, local sound sources such as recorded way files and music, and sound art resources. A Windows® operating system API is used to attach each of these sources and present them to the audio transport bus for distribution. Each source is “wrapped” as an AudioSource derived class. Source object wrappers are created when a definition is received (see the AudioDevice SODA definition in section VI). DirectSound APIs are used on Windows® based computing systems for microphone, clip, Skype, and CD sound. RTP streaming is simply an AudioSource wrapper around the Real-Time Protocol (RTP) service, which is used to deliver streaming data (e.g., audio) over a UDP socket. Streaming audio over the internet supports web-based download-and-play and also Microsoft Media Server streams.
In some embodiments, the realtime kernel 260 supports sourcing and mixing of sessions of virtual area based communications with non-virtual area based communications (e.g., Skype and VOIP audio). In these embodiments, the realtime kernel intercepts non-virtual area based communications and presents them as local audio sources. The non-virtual area based communication session is initiated by one of the communicants on a client network node that is responsible for sharing the non-virtual area based communication session raw audio with other client network nodes. The hosting client network node also mixes audio associated with a virtual area communication environment for the communicant who is communicating via the non-virtual area based communication application.
In operation of the computer system 389, the I/O Multiplexer/Demultiplexer sends the audio signals 1 and 2 received from Systems 1 and 2 to both System 3 and System 4. The I/O Multiplexer/Demultiplexer also sends the audio signals 3 and 4 received from Systems 3 and 4 to the P Route component of the stream handler 391. The P Route component sends the audio signals 1, 3, and 4 to the playback component of System 2 and passes the audio signals 3 and 4 to the P mix component of System 2. The P Mix component of the stream handler 391 mixes the audio signals 2, 3, 4 and passes the mixed signal to the integration component of System 2. The integration component 393 passes the mixed signal to the audio capture component of an alternative communications application (e.g., Skype) that is running on System 2 and that corresponds to the communications application 395 (e.g., Skype) that is used by System 1. The alternative audio capture system (C A) passes the captured mixed signal 2+3+4 to the playback component of the alternative communications application 395 running on System 1.
In some embodiments of the communication infrastructure 389, P Mix subscribes to I/O Multiplexer/Demultiplexer directly so that the system is more symmetric. In these embodiments, P Route becomes P Mix 1, and receives 3, 4 from I/O and 1 from C Split 1. Since these are sent as independent channels, the output of C Split 1 could be sent directly to the Playback Component, but that isn't quite as flexible (since P Mix could perform an actual mix instead of a pass through of independent channels, see 3 below). In this case, P Mix becomes P Mix 2 and receives 3, 4 from I/O and 2 from C Split 2. The output of this mixer is a true mix, since the Alternate Audio System is a single channel communication system (even if the channel is stereo, we there is no multi-track mixer at the other end to combine signals from multiple sources).
In
m. Realtime Scheduler
In some embodiments, the realtime scheduler service 278 is a Windows® service DLL.
In some embodiments, rendering of audio and the 3D scene is done on a frame-by-frame basis. Initially, the streams are started, then after a delay the realtime scheduler service 278 begins processing the first frame. The delay is calibrated by the combined desired latency of each audio and video processing chain. The realtime scheduler service 278 initiates consumption of the previously prepared frame and then processes the next frame, on a time Tick that has a period of 50 milliseconds.
The final rendering objects in each chain are registered with the realtime scheduler service 278. The objects are derived from a SoFrameRenderer class, which has a method
This method prepares one frame for the time indicated, from data sources that are particular to the rendering chain (audio or video). The SoFrameRenderer class includes another method
A Windows® operating system has no “hard scheduling” ability. In some embodiments, the realtime scheduler service 278 is configured to call one or more SoFrameRenderer classes, which include audio processors, graphical processors, physical modeling and scripts. The SoFrameRenderer classes enable the realtime scheduler service 278 to readjust frame processing in response to a determination that the client network node cannot keep up with the target processing level. In some embodiments, the realtime scheduler 278 implements one or more of the methods described above in connection with
In some embodiments, the realtime scheduler service 278 additionally is able to drop stages within a chain. For this purpose, the realtime scheduler service 278 is able to call the method:
Where the processing chain is responsible for “dropping links” that are lower than the indicated priority. The realtime scheduler service 278 can start calling at the maximum priority and count backwards until all frames render within the desired total latency. A combined heuristic of iteratively dropping low-priority chains and pruning the chains themselves typically terminates at some priority level. If the priority level is below a threshold, a log entry can be made and, in some cases, the session is closed.
The realtime scheduler service 278 also is responsible for managing the interface to a SIS time service (a network resource), so as to synchronize the client network node clock with the clocks of the other client network nodes in the network. In some embodiments the time services are implemented by plugins, as described below.
7. Failure Recovery
The realtime kernel 260 makes a best-effort attempt to continue operating in the presence of network and server failures. In this process, the realtime kernel 260 implements a two-tier failure recovery algorithm. First the SODA channel service 270 and the Media service 271 independently attempt to reestablish connections upon failure. The media recovery will allow a conference to continue in the presence of individual audio channel failure, or to recover in the case of NAT timeout. Second, if the SODA channel service 270 and the media service 271 fail to reestablish connections, they will signal failure to their clients. In some embodiments, the actions of multiple communicants simultaneously attempting recovery are coordinated by registering the client node sessions with an area vault server that can synchronize the recovery attempts.
In some embodiments, the area/zone manager 264 also attempts a recovery in the event of a server failure. The area/zone manager 264 fields the stream failure and then tears down the session and re-launched it on a different area server.
8. Plugins
In some embodiments, the realtime kernel 260 has a componentized, open, and platform-independent architecture that allows developers to independently and develop and remotely add and update components of the realtime kernel 260.
a. Plugin Design
Plugins are platform-specific binary images that are downloaded from a plugin server. On Windows® based computer systems, plugins are implemented as DLLs (e.g., .NET or COM style plugins). Plugins differ from normal dynamic linked libraries, however, by the way in which they are loaded. In particular, during loading there is no need to link to the plugin library; nor is there any need for any compilation of software code. Instead, the plugin manager 266 simply loads the plugins that are contained in the plugin directory. In this way, plugins can be added or removed from a client network node without any further configuration of the station (e.g. registry entries), simply by downloading or deleting the executable. The plugins connect to each plugin host in a well defined way.
As shown in
Plugins are identified by GUID as well as human-readable name. A single plugin can have multiple APIs supported as well as multiple variants on the same API. In some embodiments, a plugin API is a C language data structure containing function pointers to the plugin functions implementing the services defined by the interface.
The following table shows an example of a fictitious Physics plugin that is identified as guidPhysicsSet1.
In this example there are four variants of a “Joint” API and two of the “Collision Detection” API. The file name is name under which the plugin is stored in the file system. The Internal Name is intended to describe the plugin as a whole. The Identifier identifies the entire plugin binary image, and is only used by the plugin manager 266. The API is a well-known GUID that specifies a procedural API that is supported by the plugin. The Variant is a well-known GUID that client applications can use to choose a plugin for a particular purpose. For instance the SODA channel service 270 may negotiate a compression plugin variant for use on a session.
Each plugin executable supports a core Plugin API that allows the plugin manager 266 to manage the plugin. The core Plugin API makes the plugin executables self-describing. In particular, each plugin exports a function that can be called by the plugin manager 266 to allow the plugin to register itself. In some embodiments, each plugin executable exports the following core Plugin API:
(i) Plugin Variant
An active instance of a plugin variant is derived from the class PlugInVariant. They are always created using a PlugIn object. All useful plugins extend this class to include API-specific methods. The only common methods are generic configuration methods.
(ii) Plugin Manager
In accordance with the method of
The plugin manager 266 queries the discovered plugins for all APIs that they support (
Based on the results of the querying, the plugin manager 266 stores associations between the plugins and the APIs that they respectively support in the plugin database 426 (
The Plugin Manager exports the following API:
(iii) Plugin Server
Plugins are present on a server in appropriate downloadable form for each supported platform. Servers always support a minimum set of encryption, compression and authentication plugins, so client network nodes always succeed when attempting a server connection.
When first registered on a server by a third party developer, the plugin is examined by an automated tool to ensure it conforms to the API specification, and is checked for unacceptable client station API references. For instance, no dynamic binding is allowed to any native interfaces not already in the Plugin environment. Then the plugin is scanned for viruses. All images on the server are then virus-safe.
Plugins are access-controlled by authenticating users. In this way a user who has paid for access can use the plugin from any location. In some embodiments, user is authenticated for plugin download via an electronic commerce engine.
b. Classes of Plugin
All plugins conform to the plugin API. Some plugins are specific to certain tools (e.g., OpenGL) and have their own standard. Each class of plugin has APIs that are class-specific. In some embodiments, plugins depend upon a native application runtime for all client station access. In each component where plugins are used, protocols allow negotiation of the feature using a GUID namespace. For example, in some instances, when connecting a network stream the offering client network node offers its encryption plugin GUID list in order of preference and the receiving network node chooses among those offered and responds with a choice or a refusal.
In some exemplary embodiments, the following classes of plugins are defined and used by developers to develop plugins for use in creating virtual area communicant environments.
Encryption Algorithm
Encryption is a stream transport feature, and is negotiated at stream creation time. The session definition will include the encryption Variant ID. Exemplary encryption variants include AES and RC4 block encryption.
Compression Algorithm
Compression is an optional Channel content feature. Channels choose compression when the transport negotiates a channel definition. The subscribing station offers available compression options, and the publishing station indicates its selection when the publication is offered. Audio codecs may choose to skip compression, since their content is already compressed. An exemplary audio compression plugin variant implements the ITU-T V.44 compression process with a channel-specific priming stream.
Authentication Algorithm
Network infrastructure servers require certain authentication protocols in order to connect. Third party servers may have specific authentication requirements. Authentication protocols may be amended from time to time. In addition to the core Plugin API, an authentication plugin has APIs for executing authentication protocols and gaining access to local credentials. Exemplary authentication plugin variants include plugin variants that support SSL type of authentication for an initial server and that support subsequent server login using a signed ID as a token.
Credential
Credentials may be created and must be encapsulated in a plugin for storage and use by the authentication algorithm. Exemplary credentials are certificates containing public keys.
Graphical Effect
OpenGL and Collada support scriptable shaders, samplers, profiles and annotations. Some embodiments support Collada cg_surface_type and glsl_surface_type which are shaders, and gl_samplerX and cg_samplerX which are samplers.
Physics Extension
In some embodiments, the Open Dynamics Engine (ODE) is amended to included specific hooks into the dynamic behavior loop. Virtual area applications typically specify physics by applying physical properties to entities in the scene graph. Physics extension plugins have an API for querying associated property GUID(s) that are processed by that plugin. In this way, the physics plugins are only invoked for pertinent objects and virtual areas. Some embodiments support Collada rigid_constraint which is a joint in a model, and OpenGL collision detection which is a global motion algorithm for avatars and artifacts.
Script Extension
Script extension plugins have an additional API to allow wrapping a scripting runtime (e.g., Java and JavaScript) and to provide to it the native application runtime. Scripts are defined through a SODA record that includes the GUID of the script extension plugin.
Input Device Hosting
Input device plugins generate SODA records for standard input events (e.g., events generated by computer keyboard, computer mouse, Xbox® controller, and Wii® controller) that are processed by application logic.
Audio Mix, Audio Source, and Audio Insert
Audio processing is desirable at the origin (e.g., microphone) and effects and mixing at the destination (e.g., speaker). Audio plugins typically are not able to change audio network routing, because that affects other user's experience. Audio plugins on the Windows® platform are DLLs that conform to the Audio Plugin API. They are registered with the realtime kernel 260 and are available for reference by SODA. Virtual area developers may request audio plugins as part of a virtual area application (e.g., using extended Collada fields or VSDL semantics).
Avatar scripts can request audio plugins as well.
Mix variants include echo, echo cancellation, reverb and compose.
Source variants include microphone, clip, media, Skype and streaming file.
Insert variants include panner, volume, iLBC, RCU, iPCM-wb, equalizer, LPF, HPF, AGC, Noise Cancellation, Error Concealment, Jitter Control, mute, delay, silence detection, comfort noise.
Stream Transport
STRAW uses a stream transport plugin to host sessions on a network. A Stream transports packets and provides reliability, authentication and encryption.
Time Service
The realtime scheduler 260 is interested in synchronizing time between client network nodes. For this purpose each client network node will synchronize with an internet time standard, such as SIS, NTP (Network Time Protocol), or ITS (Internet Time Service).
c. Plugin Class APIs
All Plugin class objects are based on Plug InVariant.
Encryption Algorithm
This plugin is based on PlugInVariant. It performs a block encryption algorithm including keying. This plugin is stateless except for the key i.e. each encryption is independent of any other.
Compression Algorithm
This plugin is based on PlugInVariant. It performs one or more lossless compression/decompression algorithms. It compresses is a continuous stream, and may keep an internal state table. The algorithm is restarted on each KeyFrame, which may include processing a “priming” block which is not emitted but contributes to the compression state.
Authentication Algorithm
This plugin is based on PlugInVariant. It accesses local credentials, and implements a client authentication state machine of one or more states. The authentication state machine does NOT control the communication link to the authentication server. It only processes messages and produces subsequent messages.
Credential
This plugin is based on PlugInVariant. It accesses platform-specific credential stores.
Graphical Effects—Shader
This plugin is based on PlugInVariant. It affects 3D rendering.
The Shader API supports shaders (scripted procedural surface generators).
Graphical effects—Sampler
The Sampler API is based on PlugInVariant. It supports mapping texture to surfaces.
Physics Extension—Joint
This plugin is based on PlugInVariant. It extends the dynamics of motion.
The Joint API supports joint constraints and dynamics for avatar and artifact subassemblies.
Joint::Torque(torque)
Joint::GetState(fps&, erp&, jLin1&, jAng1&, jLin2&, jAng2&, c&, cfm&, low&, high&)
Physics Extension—Collision
The Collision API is based on PlugInVariant. It supports rapidly calculating collision detection. In some embodiments, a collision plugin variant may implement the quad-tree class dxSpace algorithm in the Open Dynamics Engine (ODE).
Script Extension
This plugin is based on PlugInVariant. It supports execution of code supplied by the virtual area application. This code is in binary form and labeled by the language and the API the particular code definition supports. The plugin varies by language. It loads a particular code definition and calls it, supplying a native application runtime object appropriate for the API.
Input Device Hostinq
This plugin is based on PlugInVariant. It supports user event capture (other than audio). This includes everything from a mouse to a video camera with facial recognition. The plugin generates user events to the queue.
Audio Mix
This plugin is based on AudioComponent. It combines two audio streams.
Audio Source
This plugin is based on AudioComponent. It provides a pollable audio stream.
Audio Insert
This plugin is based on AudioComponent. It translates an audio stream. It may compress, decompress or simply modify the stream.
Audio Send
This plugin is based on AudioComponent. It divides an audio stream into two equal streams (copies the stream). Any AudioSource can be routed to any other audio element.
Stream Transport
This plugin is based on PlugInVariant. An exemplary stream transport plugin variant supports an IP UDP transport with configurable authentication and encryption.
Time Service
This plugin is based on PlugInVariant. It synchronizes a client network node time with an Internet standard.
Session
SessionFailure
Station
Publish
Subscribe
StreamKeepAlive
Stream Failure
ChannelFailure
CommunicantState
HID
AudioParameter
AudioDevice
AreaSession
PhysicsCheckpoint
Zone
AudioAperture
AudioObstruction
AudioReverb
AudioMix
AudioEffect
AudioStream
AudioCalculation
PlugIn
UpgradeDependencyList
Upgrade
AudioRecord
AudioPlayback
NetworkDNS
NetworkProxy
Address String
DebugInfo
DebugTrap
HID
The embodiments that are described herein provide a realtime kernel that supports the realtime communications between communicants operating on respective network nodes. The realtime kernel handles the complex tasks of connecting to communicants, virtual areas, and other network resources, switching those connections in response to user inputs, and mixing realtime data streams. The realtime kernel enables developers to focus on developing high-level communications functionality instead of low-level plumbing code. The realtime kernel imposes relatively low computational resource requirements so that realtime communications performance can be achieved using a wide range of currently available computing devices and network connections.
Other embodiments are within the scope of the claims.
Under 35 U.S.C. § 119(e), this application claims the benefit of U.S. Provisional Application No. 61/120,372, filed Dec. 5, 2008, the entirety of which is incorporated herein by reference. This application also relates to the following co-pending patent applications, the entirety of each of which is incorporated herein by reference: U.S. patent application Ser. No. 12/418,243, filed Apr. 3, 2009;U.S. patent application Ser. No. 11/923,629, filed Oct. 24, 2007; andU.S. patent application Ser. No. 11/923,634, filed Oct. 24, 2007.
| Number | Name | Date | Kind |
|---|---|---|---|
| 10972523 | Van Wie | Apr 2021 | B2 |
| 20090276771 | Nickolov | Nov 2009 | A1 |
| 20100005531 | Largman | Jan 2010 | A1 |
| Number | Date | Country |
|---|---|---|
| 2004-30571 | Jan 2004 | JP |
| Entry |
|---|
| Notice of Allowance dated Nov. 27, 2020, filed in U.S. Appl. No. 16/221,334, pp. 1-7. |
| Terminal Disclaimer Filed (Electronic) Oct. 28, 2020, filed in U.S. Appl. No. 16/221,334, pp. 1-2. |
| Terminal Disclaimer Approved (Electronic) Oct. 28, 2020, filed in U.S. Appl. No. 16/221,334, p. 1. |
| Response to Non-Final Office Action dated Oct. 28, 2020, filed in U.S. Appl. No. 16/221,334, pp. 1-8. |
| Non-Final Rejection dated May 28, 2020, filed in U.S. Appl. No. 16/221,334, pp. 1-8. |
| Response to Non-Final Office Action dated Feb. 24, 2020, filed in U.S. Appl. No. 16/221,334, pp. 1-13. |
| Non-Final Rejection dated Aug. 22, 2019, filed in U.S. Appl. No. 16/221,334, pp. 1-9. |
| Number | Date | Country | |
|---|---|---|---|
| 20210185105 A1 | Jun 2021 | US |
| Number | Date | Country | |
|---|---|---|---|
| 61120372 | Dec 2008 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 12630973 | Dec 2009 | US |
| Child | 14045562 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16221334 | Dec 2018 | US |
| Child | 17188831 | US | |
| Parent | 14722156 | May 2015 | US |
| Child | 16221334 | US | |
| Parent | 14045562 | Oct 2013 | US |
| Child | 14722156 | US |
