Patent Application
20200272911
The present invention relates generally to systems, methods, and apparatuses that integrate cognitive processes into the management of physical automated systems.
Automation engineering systems are used for configuring, programming and commissioning automation systems comprising programmable logic controllers (PLCs), drives, human-machine interface (HMI) systems, distributed I/O, etc. Performing automation engineering tasks require a high level of technical expertise due to the complexity and criticality of modern automation systems, as well as domain expertise due to the specialty of the application areas when developing automation solutions (e.g., manufacturing and assembly, chemical, pharmaceutical, food and beverage, paper, electronics, etc.).
The growing complexity of automated systems, the increasing requirements for high productivity and quality of the engineering tasks as well as for the efficiency and safety of the automation solutions, produce a demand of automation engineers that is increasingly difficult to satisfy. Additionally, teams of automation engineers are faced with increasingly complex challenges that incur in higher costs and greater risks for providing the required levels of reliability, efficiency, and safety.
Traditional automation engineering, as described above, relies on the expertise of engineering teams. Engineering systems provide graphical user interfaces and support functions that assist engineers with their tasks, for example, by automating simple repetitive tasks, performing automatic conversion between different file formats, allowing comfortable editing of designs and programs through intuitive graphical user interfaces, performing validation and cross-checking of configurations, programs and designs, offering wizards that guide the engineer step-by-step, etc. Each of these features reduce the effort needed for performing engineering tasks, but do not reduce the requirement for expert users nor the responsibility of these users for developing correct automation solutions for increasingly complex systems.
Embodiments of the present invention address and overcome one or more of the above shortcomings and drawbacks, by providing methods, systems, and apparatuses related to a cognitive automation engineering system that may be used to control physical automated systems based on system knowledge. Cognitive systems are the result of many years of research and are now being integrated into commercial systems. By leveraging cognitive technologies, CAES can effectively transform the automation engineering discipline. For instance, an engineer can concentrate on higher level tasks and delegate configuration and programming tasks to CAES, thereby doing the work of an entire team of engineers. Apart from an increase in productivity, the automation of the cognitive processes behind engineering can improve the quality of the engineering work, producing more efficient and safer automation solutions.
According to some embodiments, a computer-implemented method for generating a deployment for an automated system based on system knowledge includes receiving user inputs comprising engineering documents and records of human-computer interactions. The exact technique for receiving the inputs depends on the type and content of the inputs. For example, where the engineering documents comprise one or more of e-mails, manuals, and webpages, the engineering documents may be retrieved from one or more databases. As an additional example, human-computer interactions with at least one human-machine interface in the automated system may be monitored to generate the records of human-computer interactions. Once the user inputs are received, knowledge data is extracted from them. A knowledge representation is generated based at least in part of the knowledge data. This knowledge representation comprises facts and rules related to the automated system. An automated reasoning engine is used to generate a set of actions executable by an automation engineering system based on the knowledge representation. An automated system deployment is generated based at least in part on the set of actions and the automated system deployment is implemented on the automated system.
Additions, refinements, or other modifications may be made to the aforementioned method in different embodiments of the present invention. For example, in some embodiments, a natural language processing model is applied to the user inputs to extract the knowledge data. In other embodiments, the knowledge data is extracted from the user inputs based at least in part on a user's manual identification of at least a portion of the knowledge data. In other embodiments, the knowledge representation described above is generated by applying one or more machines learning models to the knowledge data to generate the facts and rules related to the automated system.
Additionally, in other embodiments of the present invention, the knowledge representation is generated by applying an inductive programming model to generate one or more rules based on the knowledge data, generating executable code based on the one or more rules, and then using the executable code to generate the facts and rules related to the automated system. In some embodiments, pre-generated facts and rules related to the automated system are received. Then, prior to using the automated reasoning engine to generate the set of actions, the pre-generated facts and rules may be aggregated with the knowledge representation. In other embodiments, the automated reasoning engine is used to generate a suggested modification to automated system design parameters. Then, the suggested modification is transmitted to one or more engineers.
According to another aspect of the present invention, a system for generating a deployment for an automated system based on system knowledge includes a cognitive system. This cognitive system includes a user interface module configured to receive user inputs comprising engineering documents and records of human-computer interactions. The cognitive system further includes a knowledge extraction component and a knowledge representation component. The knowledge extraction component is configured to extract knowledge data from the user inputs. The knowledge representation component is configured to generate a knowledge representation from the data comprising facts and rules related to the automated system. The cognitive system further includes an automated reasoning engine. This automated reasoning engine is configured to generate a set of actions executable by an automation engineering system based on the knowledge representation. Aside from the cognitive system the aforementioned system includes an automation engineering system configured to generate an automated system deployment based at least in part on the set of actions, and implements the automated system deployment on the automated system.
In some embodiments, the aforementioned system further includes a plurality of engineering stations. Each engineering station executes an instance of the cognitive system and an instance of the automation engineering system. Each instance of the cognitive system may share the knowledge data locally extracted at the engineering station with instances of the cognitive system executed on other engineering systems. Alternatively (or additionally), the system may include a server computing system that executes an instance of the cognitive system and an instance of the automation engineering system. This server computing system may be configured to provide multiple users with simultaneous access to the cognitive system and the automation engineering system.
In other embodiments of the present invention, a system for performing cognitive tasks related to an automation system in a dispersed manner includes a plurality of computing devices. Each computing device includes a real-time execution system that generates input/output signals for controlling a physical system, and a cognitive system. The cognitive system is configured to extract knowledge data from records of human-computer interaction and acquire additional knowledge data from one or more other computing devices included in the plurality of computing devices. Each computing device also generates a knowledge representation comprising facts and rules related to the physical system based at least in part of the knowledge data and the additional knowledge data. Additionally, each computing device uses an automated reasoning engine to generate a set of actions for reconfiguring the real-time execution system based on the knowledge representation, and executes the set of actions. In some embodiments, the system further includes a real-time communication network connecting the plurality of computing devices and used to acquire the knowledge data from the other computing devices.
Various additions, refinements, or other modifications may be made to the aforementioned system in different embodiments. For example, in one embodiment, reconfiguration of the real-time execution system comprises providing executable instructions to one or more of a real-time scheduler and a resource manager included in the real-time execution system. In another embodiment, the records of human-computer interaction comprise one or more of parameter values, operation goals, commands, operational constraints, and priorities related to the physical system. In some embodiments, the additional knowledge data comprises an identification of a cyber-attack on the other computing devices. Then, the cognitive system may be further configured to determine that a particular computing device included in the other computing devices has been compromised by a cyber-attacked based on the additional knowledge data received from the particular computing device. A new set of actions can be generated for reconfiguring the real-time execution system on the compromised device based on the knowledge representation.
Additional features and advantages of the invention will be made apparent from the following detailed description of illustrative embodiments that proceeds with reference to the accompanying drawings.
The foregoing and other aspects of the present invention are best understood from the following detailed description when read in connection with the accompanying drawings. For the purpose of illustrating the invention, there is shown in the drawings embodiments that are presently preferred, it being understood, however, that the invention is not limited to the specific instrumentalities disclosed. Included in the drawings are the following Figures:
The following disclosure describes the present invention according to several embodiments directed at methods, systems, and apparatuses related to a Cognitive Automation Engineering System (CAES). The CAES comprises an Automation Engineering System (AES) that is augmented with a Cognitive System (CS). By incorporating the functionality of an AES, the CAES offers the conventional engineering functionality to engineers. Additionally, the CS can act as a “cyber-engineer” that can think, decide and act on its own, interacting with the human engineers and allowing them—in conjunction with the CAES—to perform automation workloads that would not be possible by the human engineers alone.
Modern-day cognitive systems such as IBM's Watson, are making their way into a multitude of application domains, and showing convincing results that further drive their adoption. For instance, medical diagnosis is gaining from vast data sets collected over the course of many years and across many different populations that can be used by cognitive systems in order to achieve accurate detection of diseases and health problems, as well as to predict future illnesses. Customer service systems are employing cognitive systems to offer effective automated support to customers over the phone, and online marketing applications—e.g. from Amazon—are based nowadays on cognitive systems and can provide suggestions to shoppers that closely match their interests.
The CAES discussed herein differs from existing cognitive systems such as IBM Watson™ in a variety of ways. For example, the CS is tailored to augment an engineering system. Thus, the CAES comprise a complete automation engineering system with all of its functionality: engineering object management, hardware and software component database, programming language models and compilers, communication components, etc.
An additional distinction from existing cognitive systems is that such systems produce results that assist users in making decisions, i.e. they provide information and advice that is deduced by the system from previously acquired knowledge. This function is achieved by the CAES through the “Feedback” link discussed below (see
Moreover, because the CAES is applied to the engineering of technical systems that will operate in critical environments, the “room for errors” needs to be eliminated, or at least reduced to a minimum. The CAES needs to reason about the degree of certainty that it has for all knowledge that it manages, and for all results from its internal learning and reasoning tasks. This degree of certainty will allow the CAES to decide between a proactive action, a suggestion to the human engineers, a warning, or ignoring a fact that is highly uncertain. The CAES also needs to reason about the criticality of the engineering actions—e.g. changing a parameter, changing a control program, or changing a comment field have different criticalities, and this will also help the CAES to decide upon its actions.
A Knowledge Extraction (KE) Component 115A in the Cognitive System 115 extracts relevant knowledge from the existing engineering documents 125 and the records of human-computer interaction 130 provided, directly or indirectly, by the Engineers 135. Examples of existing engineering documents 125 include function specifications, computer source code, and informal technical descriptions of components of the Automated System 140 (e.g., emails, engineering notes, etc.). The records of human-computer interaction 130 may include, for example, data logs from a human-machine interface (HMI) or devices within the Automated System 140 indicating how users are interacting with the device. It should be noted that the existing engineering documents 125 and the records of human-computer interaction 130 are meant to be exemplary of the type of the type of input data that can be used in various embodiments of the present invention. However, it should be understood that other types of input data can also be used.
The KE Component 115A performs context-sensitive knowledge extraction of the existing engineering documents 125 and the records of human-computer interaction 130. All this extracted knowledge is stored in the KR Component 115B. The KE Component 115A may use previously obtained knowledge from the KR Component 115B in order to perform context-sensitive knowledge extraction. For example, in one embodiment, the KE Component 115A comprises one or more machine learning models trained based on knowledge stored in the KR Component 115B. Such machine learning models may implement natural language processing algorithms, or other semantic extraction techniques to find knowledge within a large corpus of data.
The Knowledge Representation (KR) Component 115B stores knowledge extracted by the KE Component 115A as facts and rules that are encoded as data. Thus, as knowledge is extracted, the KE Component 115A applies a knowledge model to identify and encode the facts and rules. In some embodiments, the KR Component 115B may also receive User Defined Facts and Rules 120 which supplement those derived from the KE Component 115A.
The CS 115 learns from previous knowledge using a Machine Learning Component 115C and an Inductive Programming Component 115D. The Machine Learning Component 115C comprises one or more models which take extracted knowledge as an input and provide facts and rules as output. In one embodiment, the Machine Learning Component 115C comprises a deep learning model that applies artificial neural networks (ANNs) to process the extracted knowledge. The Inductive Programming Component 115D uses inductive learning techniques to automatically synthetize new rules for creating knowledge representations from the extracted knowledge and background knowledge of the particular automated system of interest. In some embodiments, the Inductive Programming Component 115D may compile a set of rules so that it may be evaluated directly by the AR Component 115E and the KE Component 115A. Thus, for example, the KR Component 115B can have highly customized rules that directly deduce new facts from input data. Inductive Programming techniques are generally known in the art and, thus, such techniques are not explained in detail herein.
Knowledge represented in the KR Component 115B may include, for example, the capabilities of a particular device, diagnostic knowledge, data layouts, and process-specific information on parameters, required functionalities, or process-specific analytical functionality. In some embodiments, the facts and rules are encoded as one or more knowledge models. These knowledge models may comprise ontologies expressed, for example, using the Web Ontology Language (OWL). In some embodiments, the knowledge models comprise a predictive model expressed using the Predictive Model Markup Language (PMML) standard and/or one or more rules expressed using the Rule Interchange Format (RIF) standard. Within the KR Component 115B, knowledge models may be connected syntactically via shared signature items, and semantically via assertions on the relation between elements of the different models (such as specialization/generalization, connectedness via relations, etc.).
The knowledge stored in the KR Component 115B is used by an Automated Reasoning (AR) Engine 115E in order to make decisions, derive solutions to problems and perform actions. The AR Engine 115E may be implemented, for example, as a rule engine, a deductive classifier, or a machine learning system. The AR Engine 115E can decide to perform engineering tasks on its own, analogously to what an engineer does. The AR Engine 115E may also assist engineers by providing feedback to them while they are working with the CAES 105, for instance by issuing warnings, providing advice and suggestions, pinpointing errors in designs, programs and configurations, suggesting to replace work done by the engineer with improved versions, etc.
In some embodiments, the CAES 105 can be used to deploy an automation solution to a physical system as with the case of an AES 110. Here, the KE Component 115A is able to obtain data from the physical system and produce additional knowledge for the KR Component 115B. This knowledge about the actual automated system can be further employed by the CS 115 in order to improve its operation as a cyber-engineer, for example, by providing better advice for the engineer, by generating optimized designs, programs and configurations, by producing tailored test cases, etc.
As shown in
Note that the arrow between the AES 110 and the Automated System 140 is bidirectional; this is meant to show that after a deployment is made, changes to requirements of the Automated System 140 may be transmitted back to the AES 110 so that they can be incorporated in future deployments. The CS 115 may also be used after the industrial automated system has been commissioned. For example, when new equipment is installed or replaced, engineers must re-program the system to enable the new functionality (see the “System Input” shown in
The Automated System 140 may include any number of components to perform a set of operations defined by the Engineers 135. Additionally, it should be noted that the techniques described herein may be applied in various types of automated systems. For example, as shown in
CAES can be employed in different system configurations that apply to specific application scenarios. For example, in some embodiments, an engineering office may be equipped with one or more high-end PCs—referred to herein “engineering stations”—that run instances of the engineering system. The engineering stations can be used by engineers in isolation or in a local area network for collaborative engineering. In this scenario, every engineering station contains a CAES as shown in
As an alternative to implementing the CAES across engineering stations, the CAES can be centrally hosted (e.g., in a cloud based computing environment) to allow use by multiple engineers at the same time. This architecture has the advantage of maintaining a single centralized repository for all engineering data and knowledge about the engineering process. The CAES reasons not only about the engineering tasks, but also about the assignment of tasks to engineers and itself, in order to avoid conflicts and to ensure a fluent engineering process.
To illustrate the applicability of CAES based in different system architectures, consider an example use case where image recognition is performed for hardware configuration. During automation engineering, a very important task is to design a hardware configuration of the automation elements (e.g., sensors, actuators, controllers, routers, firewalls, etc.). In conventional systems, the engineer uses schematics of the plant (e.g., piping and instrumentation diagrams) to create a hardware configuration that can be used to control the process. In the conventional practice, the relationship between the schematics of the plant and the hardware configuration of the automated system is implicit in the engineer's mind. Thus, depending on the engineer's expertise, the hardware configuration may be sub-optimal and does not fully capture the requirements. To improve this process, the CAES allows the engineers to provide also a photograph of the plan to be automated. The automated reasoning engine of the CAES extracts the components from the photograph and maps them to known components and hardware configurations acquired from other plants and stored in the knowledge representation. These mappings of photographs to components make an explicit connection between things that the engineer sees in real-life to hardware configuration objects. With this explicit understanding, the engineer can be aided by design recommendations from the CAES system to produce fewer errors in future deployments.
Starting at step 205, the computer system receives user inputs from engineers or other users. These user inputs may include, for example, engineering documents and records of human-computer interactions. In one embodiment, the engineering documents comprise one or more of e-mails, manuals, and webpages. The computer system may be configured to automatically retrieve the engineering documents from one or more databases. For example, in one embodiment, the computer system includes “crawler” software that systematically browses databases storing engineering documents to extract either the documents themselves or, in some instances, just the knowledge. In other embodiments, software may be installed on the system hosting the database to automatically push documents to the computer system as they are created or updated. In some embodiments, the computer system monitors human-computer interactions with one or more computers in an automated system (e.g., human-machine interfaces) to generate the records of human-computer interactions.
At step 210, the computer system extracts knowledge data from the user inputs. In some embodiments, a natural language processing model is applied to the user inputs to extract the knowledge data. In other embodiments, the knowledge data is extracted from the user inputs based at least in part on a user's manual identification of at least a portion of the knowledge data.
Next, at step 215, the computing system generates a knowledge representation based at least in part of the knowledge data. This knowledge representation includes facts and rules related to the automated system. In some embodiments, the knowledge representation is generated by applying a deep learning to the knowledge data to generate the facts and rules related to the automated system. In other embodiments, the computer system generates the knowledge representation at least in part by applying an inductive programming model to generate one or more programs based on the knowledge data and generating executed code based on the one or more programs. Once generated, the executable code is used to generate the facts and rules related to the automated system. In one embodiment, the computer system receives pre-generated facts and rules related to the automated system. These pre-generated facts and rules can then be aggregated with the knowledge representation.
At step 220, the computer system uses an automated reasoning engine to generate a set of actions that are executable by an automation engineering system based on the knowledge representation. These actions are similar to the activity that could be performed by an engineer. However, by using knowledge gathered about the system, the automated reasoning can automate the performance of these tasks. As discussed above, the automated reasoning engine itself is a software tool capable of making decisions based on a large corpus of facts and rules defining the automated system. In some embodiments, the automated reasoning engine is a rules engine, implemented using techniques generally known in the art. In other embodiments, the automated reasoning engine is a machine learning model (e.g., a deep learning network) that is trained to output actions based on input comprising the derived facts and rules.
At step, 225, the computing system generates an automated system deployment based at least in part on the set of actions. The deployment itself can take various forms, depending on the particular type of automated system being used. In general, the deployment includes configuration information and possible instructions which can be implemented on components of the system. Techniques for generating a deployment by an Automation engineering System are generally known in the art and, thus, the specifics of these techniques are not explained in detail herein. Finally, at step 230, the automated system deployment is implemented on the automated system. For example, configuration information may be applied to the various components in the system and any instructions for individual components can be delivered to those components.
In some embodiments, the computer system uses the automated reasoning engine to generate a suggested modification to automated system design parameters. These suggested modifications can then be transmitted to engineers or other users that would have interest in such suggestions. In general, any method may be used to transmit the suggestions. For example, in one embodiment, the suggestions are sent to the engineers as emails or postings on a webpage. In other embodiments, the suggestions may be integrated into design tools to provide suggestions for certain design options as they are selected by the user. For example, if an engineer begins designing a particular component of a system, the engineer may be presented with a list of suggested configurations derived from knowledge of the automated system that will be using the component.
In the example of
Each DCA Device includes the cognitive system (see
Within every DCA Device, the cognitive system controls the operation of the real-time execution system. The cognitive system decides on the usage of the DCA Device and uses this information to reconfigure and reprogram the real-time execution system, thereby adapting to changing conditions in the environment and operation goals. The real-time execution system then operates in a deterministic manner thanks to the use of an underlying real-time scheduler and resource manager. The cognitive system is able to monitor the execution of the real-time system, for example, by observing runtime values of variables.
The cognitive systems of the multiple DCA Devices can communicate directly with each other by exchanging knowledge. This knowledge may be encoded in a standard but generic format (e.g., OWL or RDF), thereby allowing the representation of domain-specific knowledge for arbitrary domains. The cognitive systems learn from the exchanged knowledge and use this knowledge to reason about the operation of the DCA Device. Such information sharing is important to reduce the overhead of reprogramming when deploying new control systems. In short, the DCAs learn, share their knowledge with other DCAs, and adapt to changing conditions in the environment. As noted above, cognitive systems also offer a human-machine interface for interacting with engineers and operators. Through this interface, the users can provide information such as parameter values, operation goals, commands, operational constraints, priorities, etc. The cognitive system of each cognitive system imports this information into its knowledge representation and uses it as described above. The cognitive systems are also able to provide feedback to the users, for example, by explaining the reasoning behind their actions, providing suggestions, issuing warnings and alarms, and providing current runtime information.
As in the current state-of-the-art, the real-time runtime systems can communicate with each other directly through real-time communication networks, thereby exchanging streams of data at runtime. Real-time communication networks are an integral part of the state of the art in many kinds of critical infrastructure (e.g., industrial and building automation, transportation systems, automotive, avionics, etc.). These network architectures are, in most cases, adaptations of the common IT networks, further enhanced for real-time operation, high availability and functional safety. And as in the case of IT networks, they are not laid out for dynamic modification of the protocol layers.
Also, the real-time execution systems are able to interact with the physical world via input/output interfaces that transmit signals. These signals are used for monitoring the physical system through sensors, and for controlling the physical system via actuators. The general system architecture for DCA encompasses all possible topologies and system configurations that can benefit from the new approach. Making an analogy to living organisms: if a DCA Device is an organism, then the cognitive system of the cognitive system is its brain, which performs though processes and stores knowledge in its memory. The real-time execution system is its set of reflexes, which are actions that are carried out instantaneously in reaction to external stimulus, and without involving the brain. A major difference in this analogy is that, unlike living organisms, a DCA can use its “brain” to “program” its “reflexes”.
In some embodiments, the cognitive system in each DCA Device shares knowledge with other DCA Devices via an exclusive communication channel, and is then able to reason about the current situation that surrounds the device and the engineering tasks that need to be carried out. An important characteristic of this architecture is that the cognitive system in each DCA only handles the engineering of its corresponding automation device (real-time execution system), while general cognitive systems can handle the engineering of multiple devices. In this regard, the cognitive system may add true intelligence to each device and the system's operation relies on the distribution of knowledge among DCAs and on local reasoning and decision making at each DCA.
To illustrate the applicability of the DCA architecture, consider the example use case of dispersed reengineering at runtime for cyber resilience. At each DCA Device, the cognitive system has already produced and deployed an engineering configuration and program for the real-time execution system that controls the physical system. Now imagine that a cyber-attack compromises one of the DCA Devices. By exchanging knowledge, the DCA Devices can detect this situation and issue a proper reaction of the system. One possibility is that the cognitive system of the affected DCA can determine that it has been compromised and shares this knowledge with the other DCA. Another possibility is that the cognitive system of the second DCA can determine that the other DCA has been compromised based on the knowledge that it receives from it. In any case, the cognitive system of the second DCA can reason about the situation and determine that its DCA must overtake and assume the critical control functions of the compromised DCA. The cognitive system in the second DCA then determines the proper configuration and control program code that need to be added to its existing configuration, performs the necessary reconfiguration and reengineering actions, and activates the new configuration. After this, the second DCA can control the plant while substituting the compromised DCA.
Many robust control systems today offer functionality for failover between controllers. However, this use case highlights a new approach for solving this problem, namely by applying cognitive systems for detecting the security breach, reasoning about the problem, deciding on a solution, and executing the necessary steps to overcome the security violation. The fact that a cognitive system performs these steps means that the solution to the problem can be expected to improve as the cognitive system increases its knowledge through learning. Traditional failover systems are programmed in a fixed manner and cannot adapt to new situations or improve its solution approach.
As noted above, various computing devices generally known in the art may be used to implement the techniques described herein. For example, in some embodiments, one or more desktop computers can be used to implement the CAES and, for DCA scenarios, multiple desktop computers can be connected to perform the techniques described herein. Also for cloud-based implementations, server computing systems may be employed. Additionally, the CAES and DCA may be implemented on more specialized computing architectures such as those used in automation systems. For example, in some embodiments, the cognitive system can be implemented on Critical Infrastructure Devices (CIDs) such as programmable logic controllers (PLC), smart sensors, remote terminal units (RTU) and industrial switches. As is generally understood in the art, CIDs are used to monitor and control critical infrastructure such as micro-grids, water management and treatment, energy generation and transmission, manufacturing, buildings, and traffic. CIDs generally include a general-purpose or embedded processor, high-speed memory, and networking capabilities. Thus, for example, CIDs can be readily adapted to act as DCA Devices as shown in the distributed system illustrated in
The computing devices described herein each include one or more processors that may include one or more central processing units (CPUs), graphical processing units (GPUs), or any other processor known in the art. More generally, a processor as used herein is a device for executing machine-readable instructions stored on a computer readable medium, for performing tasks and may comprise any one or combination of, hardware and firmware. A processor may also comprise memory storing machine-readable instructions executable for performing tasks. A processor acts upon information by manipulating, analyzing, modifying, converting or transmitting information for use by an executable procedure or an information device, and/or by routing the information to an output device. A processor may use or comprise the capabilities of a computer, controller or microprocessor, for example, and be conditioned using executable instructions to perform special purpose functions not performed by a general purpose computer. A processor may be coupled (electrically and/or as comprising executable components) with any other processor enabling interaction and/or communication there-between. A user interface processor or generator is a known element comprising electronic circuitry or software or a combination of both for generating display images or portions thereof. A user interface comprises one or more display images enabling user interaction with a processor or other device.
The various computing devices described herein may comprise at least one computer readable medium or memory for holding instructions programmed according to embodiments of the invention and for containing data structures, tables, records, or other data described herein. The term “computer readable medium” as used herein refers to any medium that participates in providing instructions to one or more processors for execution. A computer readable medium may take many forms including, but not limited to, non-transitory, non-volatile media, volatile media, and transmission media. Non-limiting examples of non-volatile media include optical disks, solid state drives, magnetic disks, and magneto-optical disks. Non-limiting examples of volatile media include dynamic memory. Non-limiting examples of transmission media include coaxial cables, copper wire, and fiber optics, including the wires that make up a system bus. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.
An executable application, as used herein, comprises code or machine readable instructions for conditioning the processor to implement predetermined functions, such as those of an operating system, a context data acquisition system or other information processing system, for example, in response to user command or input. An executable procedure is a segment of code or machine readable instruction, sub-routine, or other distinct section of code or portion of an executable application for performing one or more particular processes. These processes may include receiving input data and/or parameters, performing operations on received input data and/or performing functions in response to received input parameters, and providing resulting output data and/or parameters.
The functions and process steps herein may be performed automatically, wholly or partially in response to user command. An activity (including a step) performed automatically is performed in response to one or more executable instructions or device operation without user direct initiation of the activity.
In the present application, the terms “include” and “comprise,” as well as derivatives thereof, mean inclusion without limitation; the term “or” is inclusive, meaning and/or; the phrases “associated with” and “associated therewith,” as well as derivatives thereof, may mean to include, be included within, interconnect with, contain, be contained within, connect to or with, couple to or with, be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or with, have, have a property of, or the like; and the term “controller” means any device, system or part thereof that controls at least one operation, whether such a device is implemented in hardware, firmware, software or some combination of at least two of the same. It should be noted that the functionality associated with any particular controller may be centralized or distributed, whether locally or remotely. Definitions for certain words and phrases are provided throughout this patent document, and those of ordinary skill in the art will understand that such definitions apply in many, if not most, instances to prior as well as future uses of such defined words and phrases. While some terms may include a wide variety of embodiments, the appended claims may expressly limit these terms to specific embodiments.
The system and processes of the figures are not exclusive. Other systems, processes and menus may be derived in accordance with the principles of the invention to accomplish the same objectives. Although this invention has been described with reference to particular embodiments, it is to be understood that the embodiments and variations shown and described herein are for illustration purposes only. Modifications to the current design may be implemented by those skilled in the art, without departing from the scope of the invention. As described herein, the various systems, subsystems, agents, managers and processes can be implemented using hardware components, software components, and/or combinations thereof. No claim element herein is to be construed under the provisions of 35 U.S.C. 112, sixth paragraph, unless the element is expressly recited using the phrase “means for.”
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/US2017/048524 | 8/25/2017 | WO | 00 |
