Patent Application
20160119626
The present invention relates to the field of data stream processing and is particularly directed toward video decoding and other image processing tasks using various processing resources. In addition, the present invention relates to human-machine interaction with applications involving said image processing tasks, particularly in the fields of the security, surveillance, and access control.
The demands placed on computers in recent years, both in terms of the variety of tasks to perform, the amount of data to process, and the performance expected from the computing systems themselves, have grown considerably. The advance of computing technology has coincided with and arguably fostered the pervasiveness of large sets of data. Indeed, the emergence of the term “Big data” as a blanket term to describe large and complex data sets in addition to the latter's numerous attendant challenges bears witness to the ongoing realities of the present climate.
Among the increased demands made of computing systems are those pertaining to the acquisition and processing of multimedia data. Content—and indeed much multimedia content—is typically packaged into streams. Such streams may represent and encapsulate, for instance, video, audio, and any other sorts of data, in addition to metadata of several types and intended for various purposes. Fields of application in which such data acquisition and processing are encountered include those involving security and surveillance-related activities. Of particular importance in such fields is the ability for humans to view streams of data (sometimes called “feeds”) with the best possible quality.
Too often, security desks or video walls are populated with a number of camera feeds whose image quality or framerates (and sometimes both) are low. An impaired ability to properly view the contents of a feed, whether through jittery feeds or poor image quality, may in some cases constitute mere visual inconvenience; however, in other cases, it may severely hamper the ability of security or law enforcement personnel to adequately perform their duties when responding to incidents. Particularly in the latter cases, the authorities' inability to respond to incidents in a timely manner often carries a cost, whether in terms of damage to property or compromised health and safety of individuals.
Very often, the aforementioned inability to adequately view feeds, especially when such feeds are captured at high resolutions and framerates, is the result of poor decoding capabilities of the system(s) involved in displaying them. Decoding and other processing of streams—particularly media streams—to ultimately present resulting streams to human operators in an acceptable form and quality is a comparatively processing-intensive activity.
Processing such data may be done by one or more general purpose processors present within a computer system. One paradigm proposed in various cases and implemented in different settings over the years is the offloading of specific types of processing. This paradigm namely involves routing certain tasks away from general purpose processors and onto specialized hardware, with the latter having varying degrees of autonomy. The prevalence of video data is associated with growing ubiquity of capture sources in addition to enhanced transmission means, such as broadband. Typically, such data streams are encoded in any one of several formats by their capture devices or in attendant source systems or capture modules. Likewise, a corresponding decoding action, in addition to any additional and contextually relevant image processing operations, must be completed for such streams to be viewed and their contents understood by a human operator with any subsequent action taken thereupon.
Methods for accelerating the viewing, decoding, and/or processing of (particularly video) streams have been proposed in the art. Some of these methods involve the use of graphics processing units, with various levels of human operator intervention or resource monitoring. Unfortunately, many of these methods are not optimally adapted to the tasks of adequately monitoring the content of the video frames and determining how best to perform the processing from among available processing means. Further still, these methods are inadequately adapted to contemporary user experience expectations; this is particularly evident in display wall interfaces incapable of displaying large numbers of high-definition streams with sufficiently rapid response, both in terms of rapid response to user input specified through increasingly common means (such as tactile displays) as well as in carrying out stream decoding tasks, displaying numerous high-resolution streams at full framerate, performing zooming on a region of interest while retaining said full framerate, or other processing tasks, in many cases as a function of human input. Furthermore, the possibility of specifying said processing-intensive operational parameters in bulk (i.e. applied to a large number of streams at once) with minimal (if any) corresponding jitter, is lacking in present systems. Likewise, the ability for a human user to trade off or relax one preference to correspondingly favor another (as between decoding or zooming quality versus retaining full framerate), is similarly lacking at the present time. Accordingly, the ability for a system to dynamically respond to such user-specified preferences—by switching or routing processing commands among multiple available processing resources while operation is underway—all while affording the aforementioned possibilities to users, is likewise not attested in systems known in the art.
Such is an object of the invention described herewith.
The present invention may typically be implemented on any of several platforms, including a security video monitoring workstation, one or several computers, and/or homologous processors (
According to a first aspect of the present invention, input media streams 101, 102, 103, typically carrying video, but possibly any other media type, are received by an input module 100 from a source 099, 099′. The streams' payload is detected 150 by a stream detection engine 200 which may inspect the streams' headers as a key part of the payload detection 150 action to positively identify the nature of a given stream 101, 102, 103 and generate a stream payload analysis result 250. In another embodiment, and particularly in cases where a stream's data format is unknown or not otherwise identified or detectable, the nature of a stream may be inferred through a deeper inspection of the characteristics of said stream.
The result 250, 250′ is provided to the attribution module 600 and forms a partial basis of the elements upon which said attribution 600 module relies to determine which of any processing resources 501, 502, 503 available within the resource pool 500 may be called upon to perform a specified processing task, such as decoding an encoded video input stream 101. Such processing resources may include GPUs for video processing, particularly for use in compute-intensive processing operations. Other operations may include decompression (whether as part of, or separate from, a generic understanding of decoding) as well as any other processing task. Other elements involved in this process include configuration information or user-provided instructions 350, additionally supplied or otherwise provided through an operator input module 300 which may include a user interface and the ability to transmit and receive information from an external system, and which may optionally include other human interface devices. The resource pool 500 communicates with the attribution module 600, in some cases by way of an intermediary resource monitoring engine 400. When it is present, the resource monitoring engine 400 monitors the load 450 of the resource pool 500 at configurable intervals and communicates information, and in turn communicates resource utilization data information 640 to the attribution module 600.
In accordance with another aspect of embodiments of the present invention, the operator input module 300 may be absent, with configuration information and instructions 350 being received externally.
The attribution module 600 thus receives up to three distinct varieties of information; this information consists, namely, of the exact nature of the streams to be processed 250, on the current processing resources and capabilities to perform said processing 640, and user-supplied configuration instructions 350. On the basis of these three types of information, the attribution module 600 formulates an explicit assignment, called a routing command 650, of media stream(s) 101, 102, 103 to available processing resource(s) 501, 502, 503.
The attribution module 600 typically generates routing commands 650 to respond to evolving conditions that it 600 gleans from aforementioned input elements 250, 350, 640. A single routing command 650 may be generated to assign a single processing resource 501 to a single input media stream 101; alternatively, permutations involving a plurality of the foregoing may be likewise envisioned.
In accordance with a further aspect of embodiments of the present invention, the resource monitoring engine 400 may be absent, with load monitoring 450 and utilization data query 640 tasks subsumed into a more simplified and direct link between resource pool 500 and attribution module 600. In such cases, operator input module 300 is present and plays a role in establishing more rigid load attribution policy among resources 501, 502, 503 present in the pool 500.
The dispatching module 700 receives routing commands 650 generated by the attribution module 600. It subsequently 700 puts these 650 into action by accessing the input media stream(s) 101, 102, 103 present in the input module 100 and by directing 750 said stream(s) to the processing resource(s) 501, 502, 503 so that requisite processing may be performed.
Alternatively, and in accordance with a still further aspect of the present invention, an update on a previously issued routing command 650 may be issued by the attribution module 600. Such an amended routing command 650′ may result from changing processing demand observed at the input module 100 or from changing processing resource capability in the resource pool 500 (such as through the engagement/allocation, freeing, addition or removal of processing resources). The amended routing command 650′ directs the dispatching module 700 to carry out a previously-issued routing command 650 under a resource use policy more consistent with, compatible to, or representative of previously supplied configuration information and/or instructions 350.
In accordance with a still further aspect of embodiments of the present invention, said amended routing command 650′ may be directed toward an already issued routing command 650 whose execution may already be in progress.
In accordance with a further aspect of embodiments of the present invention, an output module 800 may receive from the pool 500 the results of aforementioned processing. The contents of said output module 800 typically consist of output media stream(s) 801 corresponding to the result of processing initially requested via the aforementioned configuration information and/or instructions 350.
In accordance with a still further aspect of embodiments of the present invention, output streams 801, in addition to other externally available streams 099 may be received by a display management module 900 (
The result 801 of said processing may be additionally forwarded to a peripheral destination 999.
The invention will be better understood by way of the following detailed description of embodiments of the invention with reference to the appended drawings, in which:
Embodiments of the present system may be implemented and/or deployed on one or several computers, monitoring workstations, and/or homologous processors. With reference to
In other embodiments, the types and formats of data streams accepted may be far less restricted. Accordingly, sources 099 for the aforementioned media streams may unlimitedly include any one or more of an image or video capture device such as a camera, a local hard drive, media server, or remote network location. Additional sources for input media streams may include sensors capable of receiving, measuring, or converting information or data—whether simultaneously or non-concurrently—that is contextually relevant to the aforementioned media stream sources. Such additional input media sources may include, without limitation, sound, GPS coordinates, human, biometric, object identification, validation, or other telemetric data.
In other embodiments, metadata resulting from more complex event processing may form part or all of a media stream. In various embodiments, the acceptability of an input media stream 101, 102, 103 follows said stream's intelligibility to the system; such as may be enabled, in several embodiments, through the presence of one or more appropriate media codecs or other software or hardware capability, to decode or otherwise render intelligible one or more streams stored in one or more different formats.
In at least one embodiment of the present invention, input media streams 101, 102, 103, each whether of identical or differing types, and in accordance with the capacity of the system, are received 099 by the input module 100 for handling by said embodiment. In a further series of embodiments, said input module 100 may passively receive or actively fetch input media streams from any source 099 external to the system, or admit a mixture of these stream inputting means. In a still further series of embodiments, streams received by the input module 100 may be duplicates or dedicated local copies of those received from external sources 099 for the exclusive use of embodiments of the present invention. Such dedicated copies may be desired for the added efficiency and convenience they offer to the deployment or execution of an embodiment of the present invention, such as to minimize latency at the moment of processing, or to help ensure the integrity of one or more input streams 101, 102, 103. In a still further series of embodiments, streams received by the input module 100 may be implicit or explicit references to said streams 101, 102, 103. Said access of streams 101, 102, 103 by reference in lieu of an implicit duplication streams is ideally governed by an appropriate semaphore or equivalent mechanism to support mutual exclusion of said stream use by multiple systems for handling by one or more embodiments of the present invention.
In another series of embodiments of the present invention, the organization and layout of the input module 100 may vary. As with various modules further described herein, input module 100 need not be understood to be necessarily contiguous, either in a physical or conceptual sense; for example, portions of the module may be implemented in just one or alternatively across multiple locations and interconnected through a network. In at least one embodiment, for example, one or more additional input modules 100′ may coexist and correspondingly provide input media streams 101′, 102′, 103′ to the system. The purpose of such additional input modules 100′ may reflect any number of abstractions or criteria, including without limitation, additional physical locations of streams, distinct stream fetching and/or access policies, or specific types or classes of image or other data contained.
In addition to the one or more media sources 099 described previously, input module 100 and any one or more complements 101′ communicate information, when so inspected and detected 150, about the payload of every stream 101, 102, 103 entering the input module 100 for handling by an embodiment of the present invention. Extraction of such information is key to allowing embodiments of the present invention to identify each of the input media streams 101, 102, 103 received from external sources 099 to prepare embodiments of the present invention for the eventual processing tasks to be performed on them, as will be further described herein. Details surrounding the necessary identification and subsequent communication of such information 150 regarding said streams 101, 102, 103 will be discussed presently.
Knowledge about the specific nature of input media streams 101, 102, 103 received by the input module 100 is vital to the function of embodiments of the present invention. The behavior of modules further discussed herein and indeed progress and execution under different operational scenarios and circumstances depend significantly upon the types of data received by said embodiments. Accordingly, embodiments of the present invention may obtain such knowledge via any one or several means. A stream detection engine 200, dedicated to the positive identification of incoming streams 101, 102, 103, provides an expedient modular abstraction obtain such knowledge.
For example, a finite number of stream input types known to some embodiments of the invention may be recognized in said embodiments through a parsing of corresponding streams' packet headers. In further embodiments, a detection algorithm may be present to arrive at a positive identification of streams through payload analysis of said streams (
Of course, the broad input stream reckoning strategies presented above are not intended to be an exhaustive enumeration of all possible approaches that may be envisioned for all embodiments, but are merely a suggestion of certain strategies that may be envisioned for deployment. In still further embodiments, it is possible that a type or even a specific aspect of one or more input streams may not be positively or definitively identified at all; in such cases, said streams may be identified as such and an error handling approach suited to the deployment objectives of said embodiments may be executed. These may include, in some non-limiting examples, notifying or otherwise prompting a human operator for specific action, and attempting to proceed nonetheless.
In some embodiments, the stream detection engine 200 may be configured to have limited access to the input module 100 for purposes of detecting the type and other attributes of streams 101, 102, 103 received by said module 100. The stream detection engine 200 may likewise be configured, in certain embodiments, to inspect 150 the input module 100 with one or more reckoning strategies such as discussed herein at a specified interval. In other embodiments, the engine 200 may be configured to dynamically or asynchronously detect the addition of a stream 101, 102, 103 to the input module 100 and perform detection and query operations 150 on the new stream added. Likewise, further embodiments where such stream detection and payload query operations 150 are carried out dynamically or asynchronously may be further configured to provide such detection abilities on an ongoing basis upon said media streams 101, 102, 103 for the entire time that said streams are available within the input module 100, rather than merely once. Multiple inquiries 150 of the streams 101, 102, 103 over the interval during which they are present within the input module 100 allow embodiments of the present invention to detect changes within the nature of a specific input media stream (including, but not limited to a change in a video stream's frame rate, resolution, color space), and allow other modules further discussed herein, to respond to such changes by modifying the behavior of the system as appropriate. In an embodiment of the present invention, an inspection and detection operation 150 may require or otherwise include a granting, to the stream detection engine 200, of proper mutual exclusion or access permissions through a synchronization mechanism of any one or more input media streams 101, 102, 103 prior to actual detection or parsing of any part of said streams.
Following identification of a stream, the engine 200 generates a stream payload analysis result 250. The particular encoding standard and format of said analysis result 250 may vary in accordance with the deployment needs and scenarios of specific embodiments of the present invention. In some embodiments, a single result 250 at a time may be returned; in other embodiments, a series of such results collected over a given time interval may be compiled or otherwise merged and retained in accordance with a given condition or policy. In a related embodiment, said results retention (including, without limitation, number of individual records and the amount of time each is kept) may be configured as a parameter of the system, as discussed further herein. In a further embodiment, the analysis results 250 may likewise be requested or otherwise communicated in accordance with similar considerations and needs of said embodiment. As a non-exhaustive enumeration, the engine 200 may be configured in such a way as to communicate or share such results 250 once the detection operation for an individual stream 101 has completed, or once the nature and attributes pertinent to the respective embodiment of the present invention have been identified for a number of streams 101, 102, 103.
In another series of embodiments, particularly those in which rapid identification of some aspect(s) of input stream types is of particular or time-sensitive importance (such as in a mission-critical deployment scenario in which a large number of input streams is received by an embodiment of the invention in a short period of time), the specific record of a payload analysis result 250 may configured for various levels of granularity and to query and deliver certain aspects of the analysis result 250 deemed of greater importance before others. In such embodiments, for example, it might be necessary or desirable for an embodiment of the present invention to ascertain the video format, frame rate, and resolution of a particular stream input as a coarse means by which to arrive at a heuristic to assess and anticipate the processing load associated with said stream input 101. In said embodiments, returning such information prior to performing any deep packet inspection of said input's 101 bitstream to determine information deemed of secondary or lesser importance, is useful. A subsequent result 250 with greater or full granularity may be requested and provided subsequently and/or at a later time when less input module 100 traffic is received by an embodiment of the invention. In a further embodiment, a priority policy may be specified or configured such that additional or full information about one particular (or particular category) of stream may be provided in a result 250 before comparable information is compiled and/or ascertained by the engine 200 for another (category of) input stream.
In some embodiments, the stream detection engine 200 be configured with information about the stream types supported by the system. In such embodiments, results record 250 may accordingly contain information specifying whether an incoming stream is of a type that is partially recognized or even entirely unsupported by the system, in which case corrective action (including but not limited to execution of the aforementioned error handling procedures) may be undertaken.
It will be appreciated that for various embodiments of the present invention, the modules discussed herein may accept configuration settings of various kinds to allow said embodiments to alternatively alter, adjust, vary, or simply set their behavior in accordance with the deployment objectives or scenarios of said embodiments. Such configuration settings, which may affect any one or all modules described herein, may be specified, in various embodiments, by one or even several human operators. In further embodiments, access policies or privileges may be specified to variously allow or restrict the specifying of some or all of the configuration settings available to said embodiments by specific users or, in still further embodiments, by categories of users.
Categories of users may include, without limitation, human operators of an embodiment of the present invention for whom a limited interaction profile has been specified. In an embodiment, such limited interaction may include access to or restriction from using any one or more of the modules described herein, whether in whole or in part. In a further embodiment, categories of users may include, to provide a non-exhaustive enumeration, system integrators, deployment professionals, human operator super users, and system maintenance staff. Indeed, in a still further series of embodiments, the notion of “user” need not be limited to a human operator or other class of human individual with a specific level of access to any whole or part of the system; in such an embodiment, the notion of what constitutes a user may be extended to include an external non-human actor, such as a third-party computer program or related application programming interface providing partial or total access to one or more aspects of embodiments of the present invention. In a still further embodiment of the present invention, examples of human users might include law enforcement and security personnel.
In various scenarios, interactions between embodiments of the present invention and humans and/or external non-human actors may ultimately affect the operation of various modules of embodiments of the present invention. Nonetheless, it is convenient to visualize all such interactions as being received by or otherwise consolidated within a dedicated operator input module 300. Consistent with this conceptualization, it is from said operator input module 300 that configuration information and instructions 350 to embodiments of the present invention emanate. Such configuration and instructions 350 may contain key policy or function-specific parameters for the operation of embodiments of the present invention, and are further discussed herein. In particular, aspects of modules described herein as being configurable may for convenience be integrated or featured as (including but not limited to a graphical user interface, menu, submenu, or control panel) encapsulated within the operator input module 300 (
In certain embodiments of the present invention, the scope of system parameters that may be specified via the operator input module 300 may be severely restricted or, in further embodiments, be entirely absent. In the latter case, such parameters may be independently specified within other modules described herein. In still further embodiments, said modules may be configured to operate in accordance with a contained set or finite number of combinations. In still further embodiments, such parameter selection may be carried out autonomously by the system's various modules, obviating the need for an operator input module 300 entirely. In such embodiments, parameters such as the maximum allowable load to place on one or more processing resources, further discussed herein, may be specified. Likewise, it may be expedient in further embodiments to specify, within the operator input module 300, additional policy pertaining utilization patterns of resources present to adopt or, in related embodiments, to prefer.
For example, in certain embodiments of the present invention, utilization policies for various processing resources 501, 502, 503, may include, without limitation, instructions 350 on whether to enable specific hardware acceleration capabilities (when such capabilities are present) as further discussed herein; such instructions 350 may be specified within the operator input module 300. In further embodiments, such instructions 350 may likewise include a list detailing the sequence of specific processing operations to perform on input streams 101, 102, 103.
In still further embodiments, portions of such instructions 350 may be received from one or more modules external to an embodiment of the present invention and routed through the operator input module 300 before being issued and applied to the system. In still further embodiments of the present invention, an optional display management module 900, further discussed herein, may alternatively or complementarily receive said portions of said instructions 350 before routing the latter to an implementation of the operator input module 300.
It will be appreciated that in certain embodiments of the present invention, it is advantageous to dedicate a module to monitor the state of processing resources 501, 502, 503 available 500 to the system overall. The pool of processing resources available to embodiments of the system 500 are mentioned in the present section merely for contextual relevance; they 500 are discussed in further detail and within a more dedicated section herein.
It will be further appreciated that rigorous monitoring of such processing resources 500, 501, 502, 503, is appropriate and indeed useful in high performance or mission critical embodiments. In particular, combining such monitoring with a feedback reporting and a corresponding reaction mechanism—in cases where these are needed or desirable, renders the implementation all the more robust and valuable. The foregoing considerations provide a compelling impetus for at least certain embodiments of the present invention to include a resource monitoring engine 400. While the various functions of such a monitoring engine 400 may in certain embodiments be contained within an explicit and dedicated module, it will be further appreciated that such functions might, in other embodiments, be limited or otherwise encapsulated within other modules of the system or in still further embodiments be entirely absent.
In embodiments where it is present, the resource monitoring engine 400 monitors 450 the load of the processing resources 501, 502, 503 that make up the processing pool 500 of said embodiments of the present invention. The nature of such monitoring, including but not limited to the frequency with which it occurs and indeed the granularity of the usage analysis to perform, may vary in accordance with the specific objectives and deployment scenario of each specific embodiment of the invention (
In certain embodiments, monitoring 450 of the resource pool 500 may likewise follow a custom and non-uniform policy for the entirety of resources 501, 502, 503 contained within the resource pool 500. For example, one processing resource 501 may be subject to a load monitoring 450 policy or even specific load reporting requirements that may differ from those of 502. In various embodiments of the present invention, such differences may include, without limitation, the frequency with which each one or more resources is monitored, the type, category, or specific name of the processing operation done by each at the time at a given moment, in addition to any other statistical information (including but not limited to active time, disk usage, or other performance metrics) deemed valuable for the specific scenario and purpose in which said embodiments are deployed.
In a further series of embodiments, the load monitoring information 450 may be stored within the resource monitoring engine 400 until said engine 400 is queried for such information 450 in a manner analogous to that seen in the stream detection engine 200. In a still further series of embodiments, the resource monitoring engine 400 may be configured to retain such information in accordance with a given criteria set, such as a specific period of time. In such embodiments, the resource monitoring engine 400 may store such data while the engine 400 awaits a resource utilization data query 640 from the system's attribution module 600, the latter of which will be further described in a subsequent section herein. Nonetheless, said attribution module 600 receives the aforementioned stream payload analysis result 250, the system configurations and specific operational instructions 350, in addition to completing a resource utilization data query 640, all in an effort to generate an overall directive to dictate and manage processing to be carried out by the system.
In a related series of embodiments, in the course of monitoring 450, the resources 501, 502, 503 may be polled for their status or for details regarding their operation, with the result of such polling being communicated back to the resource monitoring engine 400. In a further series of embodiments, specific policies on said polling and any related monitoring activity 450 may be specified as part of the configuration information 350, the latter of which is indirectly shared with the resource monitoring engine 400 by the attribution module 600 in the course of a resource utilization data query 640 that the latter 600 issues to the former 400. In a further related series of embodiments, the result of such polling 450 may be compiled by the engine 400 into a matrix or matrix-like series of data which matches the resource 501, 502, 503 with any number of load and/or performance parameters and with tasks currently occupying the aforementioned resources, with said resulting matrix forming the basis of the resource utilization data query 640 provided to the attribution module 600 by the engine 400. It will be appreciated that the information populating the aforementioned matrix and delivered to the attribution module 600 may likewise deliberately constitute a limited or partial (rather than integral) set of data available. Such limitation might, in a manner analogous to that discussed previously for stream detection, be desirable in embodiments or even in specific operational situations in which a specifically defined minimal and/or rapidly collected quantity of information are sufficient for use by the attribution module 600.
In another series of embodiments, the resource monitoring engine 400 may be entirely absent by design. In such cases, no load monitoring of the resource pool 500 need be done 450 or accordingly queried 640. Such embodiments may typically be implemented in situations and/or on systems characterized by such a conservative usage policy of resources present in the processing pool 500 as to obviate the need for such monitoring capability.
It will be appreciated that embodiments of the present invention require the ability to process data commensurate both with the streams 101, 102, 103 received 099 as inputs 100 as well as the desired processing operations to perform. Accordingly, it is convenient to contemplate the processing ability of embodiments of the present invention as a resource pool 500 comprising any of several identical or distinct processing resources 501, 502, 503.
Said resources 501, 502, 503 may be processors of any kind, and accordingly need not be constrained to any one type or technology. The resource pool 500 may be made up of any number of processing resources 501, 502, 503, each having any microarchitecture and electrical layout operative to permit the processing tasks required for scenarios in which the various embodiments of the present invention will be deployed. Thus, in various embodiments, said processing resources may be one or more general purpose CPUs, including but not limited, in at least one embodiment, to the one or more CPUs on which the present system executes. The processing pool may likewise comprise, for any one or more embodiments of the present invention, one or more GPUs, ASICs, DSPs, physics processing units, image processors, network processors, audio processors, or any other processing means 501′, 502′, 503′. Indeed, such processing resources may include GPUs for video processing, particularly for use in compute-intensive processing operations and for purposes which include freeing the CPU for other tasks. Likewise, when the processing abilities of said processing means other than the central CPU of a workstation are exceeded, a portion of the workstation's own CPU's load may in some embodiments be procured to provide surplus processing capability to meet said demand. Other processing operations may include decompression (whether as part of or separate from a generic understanding of decoding), image scaling, as well as any other processing task.
It will also be appreciated that in embodiments of the present invention, the processing resources present should possess the specific capabilities necessary to perform the processing operations required by the system, an inventory of the capabilities of the resource pool as a whole 500 and/or of its constituent resources in particular 501, 502, 503, may be queried and listed in the course of operation. Capabilities thus inventoried and enumerated should likewise be understood. In a further embodiment, such capability inventory information may be coupled and integrated with error handling routines equipping said embodiment to gracefully recover from cases where an input media stream or processing task cannot be carried out by said embodiment for lack of able processing resources (including but not limited to absent media codecs), inadequate licensing, or due to any other limitation. In embodiments of the present invention in which a resource monitoring engine 400 is present, the result of such capabilities inquired of the resource pool 500 may be included within the load monitoring information 450 inquired by said resource monitoring engine 400, and subsequently provided by said engine 400 to said attribution module 600 as part of a resource utilization data query 640.
It will be further appreciated that the individual processing resources 501, 502, 503 that make up the resource pool 500 may, in various embodiments of the present invention, benefit from being mutually accessible resources. In such cases, mutual accessibility is beneficial, as it facilitates sharing or otherwise directing the outputs of one processing resource 502 toward the inputs of another 503. Such resource accessibility provides a wide array of potential resource sharing policies. In a further embodiment, a more complex resource accessibility policy may be defined or configured to variously restrict, encourage, optimize, or limit the use of any one or more processing resources 501, 502, 503 in accordance with the scenario in which said embodiment is deployed.
The deployment of certain resource sharing policies to the resource pool 500 may effect—whether implicitly or explicitly—a load balancing policy. Indeed load balancing may constitute, in at least certain embodiments of the present invention, a key benefit. More sophisticated load balancing policies may encourage the attribution of processing tasks to any one or several processing resources 501, 502, 503. In an embodiment where a processing task to perform is the result of several independent (or at least divisible) subtasks, such execution decisions as splitting, forking, or merging portions of data among multiple resources 501, 502, 503 may be envisioned. In a further embodiment, and particularly in cases where processing urgency, priority or other time-sensitive considerations are of primary importance, a processing load may be may be assigned, to the greatest possible extent, among multiple resources 501, 502, 503.
Once a result is yielded by a processing resource 501, 502, 503, said result may in some embodiments be made readily available externally 999. The possible nature of such results is to be appreciated here in a large and widely encompassing manner. Processed results may be understood as being, in a non-limiting enumeration, numerical values, images, or any portions of partially or fully decoded video.
In another embodiment, data generated or otherwise output by one or more individual processing resources may be in a raw output form relative to said data's final purpose. In such cases, said result data may likewise not be in a usable, contextually intelligible, or otherwise useful form. In such an embodiment, it may be preferable to await the availability (from within the resource pool 500 and/or without) of all contextually relevant results before undertaking an encapsulation process appropriate to the deployment scenario of said embodiment. Said encapsulation process (including but not limited to activities involving or relating to stream encoding and/or transfer encoding) may occur, in some embodiments, subsequent to the production of said raw outputs and entirely within said resource pool 500. In other embodiments, portions of said encapsulation may occur partially or entirely external to the resource pool 500. In a further embodiment, a variable approach to such raw output data form may be envisioned.
Individual processing resources 501, 502, 503, or indeed the resource pool 500 collectively, may communicate with the resource monitoring engine 400 in embodiments in which the latter is present. When the resource monitoring engine 400 is absent, the resource pool 500 communicates with the attribution module 600. Discussion of aspects regarding the assignment of a processing load is further made in the attribution module 600 section herein. Likewise, discussion of aspects regarding stream and/or transfer encoding is further made in the output module 800 section herein.
Before execution of the main processing steps can begin, embodiments of the present invention must bridge the input data received with the capabilities of processing resources present. As discussed in the previous sections, this coordination endeavor requires several steps.
Once the input media streams to process have been received by the input module 100, and their nature has been ascertained, as well as the specific system configuration and particular instructions with which to operate, in addition to a due characterization of the processing resources and information about their suitability for the processing tasks has been obtained, embodiments of the present invention may proceed to assign the necessary processing tasks to the processing resources present.
This dynamic sets forth the motivation for the major decision-making component central to embodiments of the present invention. The role of the latter component, more abstractly referred to as the attribution module 600, balances the operational and processing needs of incoming data with the system's ability to accommodate such processing with little perceivable delay. In embodiments of the present invention, the tangible result of such decision-making is the determination of one or more routing commands 650 which express the explicit association of available resources 501, 502, 503 to input streams 101, 102, 103 to process. The determination of said routing commands 650 is typically an ongoing exercise that in a further embodiment of the invention takes into account a collection of elements that includes knowledge of the precise nature 250 of the demand 100 and supply 640 of all available processing resources 500, in addition to externally articulated policy 350. Likewise, in a further embodiment of the invention, the generation and issue of a routing command is the result of the aforementioned elements effectively cooperating as a control system to both respond to and govern the system's processing needs.
As discussed previously herein, in various embodiments of the invention, the input module 100 typically contains a changing number of input media streams 101, 102, 103 as these are variously received and subsequently handled by the system over time. Determining the individual nature of said streams 101, 102, 103 (including, for example, type, framerate, and resolution) is one of several important elements in the generation of a routing command 650. This is particularly due to the fact that various resources may be present in embodiments of the present invention, with each of said resources being suited to a finite number of processing tasks typically required by the various streams in the input module 100. The aforementioned determination of stream type is one important piece of information to be considered when deciding which one or more available resources 500 are best suited to handle which stream 100. In a further embodiment of the present invention, the attribution module 600 may compile stream type information consisting of stream payload analysis results 250 received for streams in the input module 100 over a given period. While a format intelligible to said embodiment is sufficient, said compilation may in a further embodiment take the form of a list of stream type information 601. In another embodiment, such information may require standardization (including but not limited to XML validation) following collection prior to being rendered intelligible, standardized, and usable.
Likewise, the attribution of resources 500 to input streams 100 may be subject to a similar approach in which said resources' 500 capabilities—both in terms of technical suitability for a given processing task as well as their temporal availability to carry it out—are monitored. The recurrence or precise period of such monitoring may vary in accordance with the scenario and implementation requirements of embodiments of the present invention. Said suitability and availability are among the basic information requested and returned in a resource utilization data query 640. The attribution module 600 may likewise include a pair of modules to manage and implement said resource utilization data query 640; in a further embodiment, a resource query module 607 may manage all such queries 640 made to the resource monitoring engine 400, while information correspondingly received from such queries may be periodically collected in a resource table module 602 for further consideration by the attribution module 600. It will be further appreciated that, owing to the changing operational loads of various resources in the pool 500, the resource query module 607 may in various embodiments issue a query 640 in fairly regular time intervals.
While the resource monitoring engine 400 and its associated monitoring 450 and querying 640 (607) activities may be explicitly absent from some embodiments, the resource table module 602 may in such embodiments be adapted to include complementary information as the maximum number of streams that each resource 501, 502, 503 may be permitted to accept at any one time, with said resources' busy/free status gleaned not through polling or other observation, but instead deduced through calculation. This latter scenario may be encountered in a further series of embodiments, particularly mission critical environments in which conservative attribution of resources is a key consideration, or in other implementations in which resource utilization follows a fairly predictable pattern of input stream 100 traffic.
In a manner analogous to the standardization previously discussed with regard to the stream type information 601 listing, the input configuration and instructions 603 received, typically from the operator input module 300 when it is present, or from any other external source with requisite privileges, may likewise undergo a similar validation process to ensure that their contents as expressed are intelligible to the attribution module 600. In a further embodiment, the attribution module 600 may be equipped with an instructions parser 608 to ensure that any translation required, such as between a third-party human interface device and/or other user interface module which may form a part of the input module 300, is performed before input configurations and instructions 603 are provided to the comparator 604—the main decision-making module within the attribution module 600—whose components and operation will be discussed presently.
In various embodiments of the present invention, the comparator 604 receives the stream type info 601, in addition to data from the resource table module 602, and the input configuration and instructions 608 to formulate a decision as to which input stream 101, 102, 103 to assign to which resource 501, 502, 503, with said assignment ultimately formalized as a routing command 650. Each of the former elements 601, 602, 608 supplied to the comparator 604 originates in some form from the environment in which an embodiment of the invention operates, and for this reason such elements may collectively be appreciated as operational stimuli. To ensure that decisions by which input streams 100 are matched with processing resources 500 deemed optimal for a given implementation or scenario, embodiments of the invention apply a series of weightings to various raw operational stimuli which may vary with time. The routing command 650 to generate thus follows an ongoing statistical analysis of specific qualities and quantities contributed by said operational stimuli. In a further embodiment of the present invention, such statistical analysis may include the application of fuzzy logic to various operational stimuli. In a still further set of embodiments, said numerical values may additionally feature, for at least some parameters, basic or default set points or default/reference values.
In certain embodiments, the input configuration and instruction scores calculator 610 assigns numerical weights to certain input configuration and instructions 350, 603 received and duly interpreted (in scenarios and/or embodiments where such interpretation is necessary) for said embodiments by the instructions parser 608. Such numerical weightings provide a quantifiable measure of the importance, usefulness, relevance, severity, and/or priority to assign to specific types of configuration information or operational instructions. In a further embodiment, said weightings may be at least in part assigned as a consequence of a set of artificial intelligence algorithms. In an example embodiment, the severity and priority to provide to an emergency stop command issued by an operator may thus take precedence over another setting whereby said embodiment is set to normally prefer a hardware processing resource configuration, which may in turn have higher priority than a specific system preference for one of two otherwise identical resources during minimal load periods. In a further embodiment, the scores assigned are at least partially the result of a previously-specified qualitative or quantitative value (or a set thereof), said value(s) specified to said embodiment. Said value(s) may be specified through such channels as an operator-accessible interface or by personnel responsible for deployment or maintenance of an embodiment of the present invention. In a further embodiment, multiple aforementioned weightings may be combined via or during operation of said instructions scores calculator 610. In a still further embodiment, said values may be grouped into like-themed categories, either explicitly by deployment personnel, or through an algorithm present within the input configuration and instructions scores calculator 610 itself.
In various embodiments, a similar weighting, score calculation and attribution approach may be applied by the stream scores calculator and sorter 612 to the stream payload analysis result 250 subsequently converted into stream type information 601. Such quantification may be useful, for example, in cases where incoming streams to said embodiments require or otherwise benefit from ranking or similar quantification for purposes of determining or extrapolating operational considerations such as the respective priority to assign to an input stream 101, 102, 103, or an optimization heuristic to apply which may be a function of the amount of data to be processed (which may in turn depend on framerate and/or resolution). Said quantification is, in various embodiments, partially a result of scores—and which may specifically concern or otherwise incorporate weightings regarding input instructions/configurations and streams—the latter of which may be calculated, result from, and received from the input configuration and instructions scores calculator 610. For example, stream type info 601 supplied to the stream scores calculator 612 might reveal that a specific input stream 101 has a specific framerate which might exceed some maximum threshold permitted by previously specified input configuration and instructions 603. Accordingly, said maximum threshold may be weighed for relative importance and consideration and be given a particularly high score by the input configuration and instructions scores calculator 610. Furthermore, the stream scores calculator might provide a weighed value for said stream 101 indicating that said stream's framerate should be subsequently throttled or minimized in a later processing stage.
Likewise, the aforementioned weighting and quantification done by the stream scores calculator and sorter 612 may further be particularly valuable in the absence of explicit instructions, whether externally supplied 603 or parsed 608, regarding the handling of specific input streams. The stream scores calculator and sorter 612 may, in a further embodiment, calculate values which may complement or otherwise supplement those values originating from the stream type information 601. In a still further embodiment, said scores may be operatively combined with an artificial intelligence algorithm and be calculated, recalculated, and recombined to on an ongoing basis. Once weighted values have been assigned and associated to the supplied stream type information 601, the module's 612 further ability to prioritize the needs or one or more operationally relevant particularities may in a still further embodiment prove valuable. It will be appreciated that continual review, recalculation, and updating of said data 612 may be particularly useful in embodiments where rapid and/or numerous changes in the input module 100 are typically observed. Moreover, values for which a weighting is calculated or assigned may be grouped into like-themed categories, either explicitly by deployment personnel, or through an algorithm present within the input stream scores calculator and sorter 612 module itself.
In a manner analogous to the foregoing, a similar weighting, score calculation and attribution approach may in various embodiments be applied to results of a resource utilization data query 640 further and intelligibly converted and compiled in the resource table module 602. As was the case previously, the resource scores calculator and sorter 614 may receive scores calculated by the input configuration and instructions scores calculator 610 and which are of particular concern or consideration to the attribution of processing resources. For example, a previously-specified 603 preference for hardware processing resources received from the set of input configuration and instructions scores calculator 610 may be further combined with knowledge of the busy/free status of all (or specific) hardware resources by the resource scores calculator and sorter 614. Further to such combination, a further score may be complementarily calculated; in this case, the weighting expressed by said score would advantage specific hardware resources rather than otherwise equivalent software ones. It will likewise be appreciated that in accordance with the changing utilization and operational loads of the various resources in the pool 500, the determination of said scores by the resource scores calculator 614 would, in a manner analogous to aforementioned score modules 610, 612 be undertaken and accordingly refreshed at intervals compatible with deployment and operational scenarios of respective embodiments of the present invention.
Additionally, and in a manner akin to the two previously described modules, parameters for which a weighting is calculated or assigned may be grouped into like-themed categories, either explicitly by deployment personnel, or through an algorithm present within the resource stream scores calculator and sorter 614 module itself.
The two broad parallel weighting and score calculation processes just described—namely those involving, on the one hand, the input scores calculator FA and the stream scores calculator 612, and the input scores calculator 610 and the resource scores calculator FC on the other—implement a weighting attribution and first-round prioritization process involving their respective operational stimuli. This cascaded score computation follows the importance of determining the course of operation of embodiments of the present invention as a function of input supplied by human operators and/or deployment personnel while simultaneously subjecting said operation upon a current view of said embodiments' processing capabilities and raw processing needs. Once the scores and weightings based on operational stimuli for input streams 101, 102, 103 to handle 612 and available resources 501, 502, 503 with which to handle said streams 614 have been independently determined, the two sets of scores are brought together in the scores comparator and merger 616 module.
It will be appreciated that the two homologous sets of weightings provided 612, 614 represent statistically optimal rankings for the specific scenario in which an embodiment of the invention is deployed. Thus, the weightings and scores emphasizing the processing needs of input streams are blended with weightings and scores emphasizing the available processing capabilities with which to handle said streams. In a further embodiment of the invention, the scores comparator and merger 616 proceeds with a first attempt at matching the aforementioned stream scores 612 with the resource scores 614 by examining those with as many overlapping scores occurring in as many overlapping categories as possible.
Once the two aforementioned sets of numerical data 612, 614 are joined in the scores comparator and merger 616 module, a prototypical series of potential matches is assembled. In an embodiment of the invention, such assembly is typically carried out by iteratively attempting to combine or otherwise fit the highest-ranking respective scores of each of the aforementioned data sets together. Once a number of highly-ranking candidate fits or combinations of said merged data has been assembled in the scores comparator and merger 616, in which one or more streams 101, 102, 103 is thus provisionally matched for processing with one or more processing resources 501, 502, 503, said provisional combinations are made available to the routing command engine 618.
It will be appreciated that the time interval(s) with which the score calculators 610, 612, 614, 616 described herein operate, calculate, regenerate, and apply said scores and attendant sorting/priority information may likewise vary as a function of the needs dictated by operational scenarios of respective embodiments of the present invention. In a further embodiment, these respective time intervals may be set independently for each of the aforementioned score calculators. It will be appreciated that in all cases, it is advisable to ensure that potential vulnerabilities resulting from lack of data freshness during the respective collection and integration of such scores is sufficiently abated as to prevent inadvertent or otherwise harmfully erroneous operation of respective embodiments of the present invention. It will be further appreciated that the precise measures to take, including but not limited to the adjustment of the aforementioned time intervals, will vary in accordance with the needs and operational scenarios of said embodiments.
The routing command engine 618 receives all provisional combinations assembled by the scores comparator and merger 616 module. In an embodiment of the invention, additional error checking or validation may be integrated into the routing command engine 618 as it proceeds to discard any provisional combinations that may be considered invalid, undesirable, or impracticable in light of specific operational or policy-related conditions. In another embodiment, such discarding may be sidestepped by selecting the provisional combination having the highest calculated weightings in as many categories as possible for a given embodiment within a given scenario. Once any and all provisional combinations deemed non-optimal or not desirable are definitively eliminated, the routing command engine 618 generates a formal routing command FG in which the specific resource(s) 501, 502, 503 to process one or more specific streams 101, 102, 103 is expressed.
In an embodiment of the invention, the issuance of a routing command 650 by the routing command engine 618 marks the final operation step of the comparator 604, with said routing command 650 being received by the dispatching module 700.
The dispatching module 700 implements 750 the stream-to-resource attribution specified by routing commands 650 issued from the comparator 604. In a further embodiment, a plurality of routing commands 650 may be issued in a burst to the dispatching module 700, which may temporarily store said plurality of routing commands 650 in a routing command queue 605. Such temporary storage of routing commands 650 may in a further embodiment be governed by a control structure consisting of wait statuses applied to various routing commands 650 in the routing command queue 605 until a specific condition is satisfied and wherein bulk number of routing commands 650 may be issued. In another embodiment, no such queue 605 need be present, with routing commands 650 received by the dispatching module 700 being provided directly to the sorting network 701. The latter module 701 may in an embodiment be implemented by way of a switch fabric which in an embodiment of the invention allows routing of input module 100 contents to the resource pool 500. In another embodiment, a semaphore or other control system adapted to and commensurate with the operational scenario of said embodiment may likewise be envisioned for said sorting network 701 in lieu of a switch fabric.
Routing commands 650 are typically issued until either no additional input media stream 101, 102, 103 is received in the input module 100, or when all processing within the resource pool 500 has completed. It will be appreciated that a key effect of the dispatching module's 700 operation is the latter's ensuring that no contention for any processing resources 501, 502, 503 present in the resource pool 500 is encountered.
Although operational stimuli ultimately originating from the operator input module 300, the input module 100, and the resource pool 500 may be correctly conceived as forming the basis of routing commands 650 generally, in a further embodiment, a new routing command 650′ may be issued further to the fulfillment or completion of a previously-issued routing command 650. An amended routing command 650′ may be desirable or necessary, for example, in cases where a processing resource 501 deemed best suited for a particular processing task required for a specific input media stream 101 was not initially available (e.g. not present, not accessible to an embodiment of the invention, or accessible but occupied with processing tasks for a different media stream, etc.) for said stream 101, but later becomes available. In such cases, an embodiment of the present invention may deem it desirable to interrupt the processing resource 502 at the earliest opportune moment, relieve said resource 502 of its processing duties for said stream 101, and transfer the remainder of a specific processing operation for said stream 101 to the newly-available processing resource 501 best suited for said task. In such cases, the amended routing command 650′ is issued by the attribution module 600 using a process in many ways similar to that described previously. However, the generation of an amended routing command 650′ occurs as a result of the resource scores calculator and sorter 614 located within the attribution module's 600 comparator's 604 receiving an update 602 regarding the recent availability of the processing resource 501 determined in the present example to be better suited than the originally-attributed resource 502. Thus, the tenure of any one specific routing command 650 may be cut short by way of an amended routing command 650′.
In a different scenario, and in a further embodiment, an amended routing command 650′ may be issued to revoke the tenure of a particular processing resource 502—even if it is indeed the best suited resource available to perform a given task on a given input stream 102. This might be desirable in cases where a new input stream 101 appears in the input module 100 for processing and for which said processing resource 502 is likewise the best suited of all available resources. In such an example, the new input stream 101 may suspend the processing privileges 502 previously afforded to the previous stream 102 because the new stream 101 is specified as having (or otherwise known by said embodiment to have) higher priority or urgency than the input stream 102 previously being handled by said resource 502. Such “bumping” or reassignment of previously allocated resources by way of an amended routing command 650′ may be effected for any reasons and/or criteria deemed appropriate by the specific scenarios in which specific embodiments of the present invention are deployed.
In a further embodiment, a routing command 650 may incorporate any combination involving one or more input media streams 101, 102, 103 and any number of processing resources 501, 502, 503. This is particularly relevant in cases where multiple (but separate) processing operations must be carried out on one or more input media streams 101, 102, 103. A corollary to this, and an equally valid scenario, is one in which a specific processing operation to be carried out by a single resource 501 requires multiple input media streams 101, 102, 103.
In cases where the processing operation(s) to perform on one or more input streams 101, 102, 103 within an embodiment of the present invention would result in a modification to the content of said input streams, one or more new result streams are generated by said embodiment. This is done so as not to destroy (or otherwise destructively overwrite) the original input stream 101, 102, 103 received. Such generation is typically required when the result of said processing 500 is also a media stream. In a further embodiment, the latter new media stream, also known as an output stream 801, is typically packaged in a format identical or similar in composition or structure to the originating stream. In a further embodiment, and particularly in cases where the output stream is a combination of multiple types of streams, or a derivation into a more limited subset of data, an optimal output stream packaging, composition, and structure (including but not limited to transport type) into which to format the resulting data is determined and selected by said embodiment. In a still further embodiment, said selection is based on maintaining similarity with said target data's constituent data, which, it will be appreciated, is a function of modifications applied by the resource pool 500, which includes without limitation, any merging or stripping of various data performed on original data streams 101, 102, 103. In another embodiment, the format of the output media stream 801 may be a function of previously-specified configuration or instruction 350 data. In a still further embodiment, said formatting of output media streams 801 depends on the availability of target codecs as well as the embodiments overall capability to encode in said format.
In another embodiment, the input stream data integrity conservation principle described above may be waived either in whole or in part. In the latter case, said waving may be implemented on some conditional criteria, specified in said embodiment as part of the configuration data and instructions 350 supplied to said embodiment. In a further embodiment, no input stream 101, 102, 103 is conserved but rather is acted upon directly by one or more resources 501, 502, 503 in the resource pool without any prior data copying.
Once the output media stream 801 is generated or otherwise made available in the output module 800, it may be rendered available for use by being output to specific locations, including but not limited to storage media or to a URL accessible by an embodiment of the present invention 999. In another embodiment, said output module 800 may be entirely optional, with the raw results of processing pool 500 being made directly available to said storage media or accessible URL 999.
In a further embodiment, particularly one in which copies of the streams 101, 102, 103 are fetched and placed into the input module 100, such input stream data integrity conservation may be viewed as a still further option that is desirable in an even more limited series of scenarios.
A further module that may be optionally present in certain embodiments is the display management module 900. In certain settings, embodiments of the present invention may be accompanied by a screen or other visible area or surface that enables a human operator, such as a guard sitting at a security desk or video monitoring workstation, to view the one or more input 101, 102, 103 and/or output 801 media streams available to embodiments of the present invention. In a further embodiment, said display management module 900, 900′ may present additional media streams and/or other data fetched from an external source 099, but which have not required fetching and placing into the input module 100 for subsequent processing. The display of such streams may be implemented by way of a GPU dedicated for display purposes and set up to arrange said streams in a multi-tile or multi-window layout, said layout and arrangement selection being provided to the display management module 900 by way of one or more display signals received by the latter module 900 by said dedicated display GPUs. In a further embodiment, if said additional media streams require processing by said embodiment of the invention, said streams are fetched from their original locations 099 and made available to the input module 100, as discussed herein. Streams not requiring processing may likewise be fetched 099 separately and shown in the display management module 900.
In a further embodiment, the display management module 900 may allow one or more (typically human) operators to specify a particular set of instructions. Said instructions, otherwise known as operator input 950, may be issued by any HID or HID-like means, such as through a touch screen monitor on which aforementioned input 101, 102, 103 and output 801 is displayed. Said instructions 950 are transmitted to the operator input module 300, previously discussed herein, for handling by said embodiment of the present invention. In a still further embodiment, said operator input 950 may be limited to specific tasks, including, in a non-limiting enumeration, decreasing the framerate of a given video stream, pausing a stream, specifying a region of interest within a given stream for purposes such as zooming, specifying full-framerate decoding of one or more high-resolution streams, and other image processing tasks.
In a still further embodiment of the invention, the display management module 900 may be subject to various levels of user privilege and/or accessibility for various human user types. Without limiting the foregoing, levels might include such profile types as non-existent (where no access to any display content is granted, for example), spectator (with view privileges only), limited functionality, advanced functionality, superuser, wherein each successive user profile grants additional privileges and/or access to said profile holder, which in turn influences the extent of the operator input 950 that said profile holder might supply. In another embodiment of the invention, the display management module 900 may be associated with the aforementioned operator input module 300. In another embodiment, the two modules 300, 900 may be entirely (including physically) distinct from one another.
Further consideration of various aspects of modules involved in various embodiments of the present invention may be given. For example, various components of, and indeed the modules described herein and which make up embodiments of the present invention need not be physically contiguous nor be within a proximate geographic area. In some embodiments, various modules or aspects thereof described herein may be connected to or within said embodiments by way of some network connection, such as a LAN or the internet. In a similar vein, the implementation of various aspects, components, or functionalities described herein as being implemented within a specific module or submodule need not be interpreted as being necessarily limited, whether in whole or in part, or for any embodiment of the present invention, to said modules as disclosed herein. Likewise, any or all of the aforementioned modules may be implemented, in whole or in part, in one or more distinct computers. Finally, modules described herein and connected through a network may be connected among said one or more computers through any networking technology, topology, without limitation.
