Embodiments relate to a framework having two main components (e.g., operating modules) that work in tandem: a Mission Behavior Manager (MBM) module configured to translate mission scripts into actionable pieces of information; and an Autonomous Action Controller (AAC) module configured to interface with the MBM module to carry out commands with vendor-specific peripherals, sensors, and payloads.
With known technology in the field of robotics and autonomy, autonomy is very platform specific if it exists at all. There is a need to provide an autonomy framework that can facilitate the deploying and managing of autonomy at scale via reusability of autonomy algorithms, collaboration among network participants, and an increase in human-machine teaming. More specifically, there is a need to provide a framework that can operate with a set of predefined commands to allow a user to customize their mission with various unmanned platforms and payloads—i.e., allow a user to use predefined commands to build tailored missions in a manner that is agnostic to the desired mission and to the autonomy framework upon which the mission is to be carried out.
Embodiments relate to an autonomous action deployment system having one or more processors. The system includes a mission behavior manager (MBM) module and an autonomous action controller (AAC) module. A processor is configured to interface with one or more autonomous vehicle platforms. The MBM module is configured to receive a mission script that includes a text file, parse the text file into actions, create a data structure where each action exists as a node in a connected graph, validate that each node corresponds with a plug-in stored in a plug-in library specific to an autonomous vehicle platform, and generate an action request command. The AAC module is configured to receive the action request command and execute steps to accomplish the action request command for the autonomous vehicle platform via a corresponding driver.
Embodiments relate to a method of deploying an autonomous action. The method involves receiving a mission script that includes a text file. The method involves parsing the text file into actions. The method involves creating a data structure where each action exists as a node in a connected graph. The method involves validating that each node corresponds with a plug-in stored in a plug-in library specific to an autonomous vehicle platform. The method involves generating an action request command. The method involves executing steps to accomplish the action request command for the autonomous vehicle platform via a corresponding driver.
Other features and advantages of the present disclosure will become more apparent upon reading the following detailed description in conjunction with the accompanying drawings, wherein like elements are designated by like numerals, and wherein:
Referring to
The autonomous action deployment system 100 can include one or more processors 102. The one or more processors 102 can include a mission behavior manager (MBM) module 106 and an autonomous action controller (AAC) module 108. The MBM module 106 and the AAC module 108 are configured to allow or facilitate the one or more processors 102 to interface with one or more autonomous vehicle platforms 110. As will be explained in more detail, the system 100 includes a framework (may be referred to herein as Cross Platform Autonomy Stack (CPAS)) to translate mission scripts into actionable pieces of information that can be applicable to any autonomous vehicle platform 110, or to multiple autonomous vehicle platforms 110. The framework includes the MBM module 106 for the translation and the AAC module 108 for interfacing with the MBM module 106, executing plug-ins, and carry out commands with vendor-specific peripherals, sensors, and payloads of the autonomous vehicle platform(s) 110.
The MBM module 106 is configured to receive a mission script that includes a text file. The mission script(s) and text file(s) can be stored in memory 104 for later processing. This can include storing the mission script(s) and text file(s) in tables or other forms of storage representation. The MBM module 106 is configured to parse the text file into actions. The MBM module 106 is configured to create a data structure where each action exists as a node in a connected graph. The connected graph data structure can be stored in memory 104. The MBM module 106 can then validate that each node corresponds with a plug-in stored in a plug-in library 114. This can be achieved via standard validation methods. The plug-in library 114 and other libraries disclosed herein can be in the form of memory, such as a database for example. The corresponding plug-in can be specific to an autonomous vehicle platform 110. For instance, the MBM module 106 can validate that a node corresponds to a plug-in in the plug-in library 114, wherein that plug-in is specific to one of an autonomous vehicle platform 110 upon which the mission script will be used. For example, the system 100 can be used to receive a mission script and use that mission script to carry out commands with vendor-specific peripherals, sensors, and payloads associated with one or more autonomous vehicle platform(s) 110. There can be more than one autonomous vehicle platforms 110. In some embodiments, the plural autonomous vehicle platforms 110 can be a network of autonomous vehicle platforms 110. Thus, the MBM module 106 validates that each node corresponds to a plug-in for one or more of the autonomous vehicle platform(s) 110. After validation, the MBM module 106 can generate an action request command.
The action request command is transmitted to the AAC module 108. This is done via the MBM module 106 sending the action request command (a push operation).
The AAC module 108 is configured to receive the action request command and execute steps to accomplish the action request command for the autonomous vehicle platform 110. This can be achieved via a corresponding driver 112 (e.g., a software interface to hardware device(s) of the platform 110), for example.
It is understood that the framework (e.g., the CPAS)) of the system 100 resides on each autonomous vehicle platform 110 and connects each unique autonomous vehicle platform 110 via a standard means of data sharing. With reference to
The mission script is an action or behavior command signal or aggregate for one or more autonomous vehicle platforms 110. For instance, a mission script can be a command aggregate that, when implemented, causes a vehicle (or more than one vehicle) on an autonomous vehicle platform 110 (or more than one autonomous vehicle platforms 110) to move in a desired direction, moved to a desired location, perform a task (e.g., climb stairs, hover over a location, take a sample, etc.), etc. The mission script can be one or more action or behavior command aggregates that, when acted upon, achieve one or more goals. For instance, the goal can be to move to location X, take a sample, analyze the sample, move to location Y, and transmit analysis of the sample from location Y. The mission script can include multiple action or behavior command aggregates to accomplish this goal. A single mission script can be for one or more vehicles or one or more autonomous vehicle platforms 110.
The MBM module 106 can be configured to receive plural mission scripts. For each mission script, the MBM module 106 can parse the text file into actions. The MBM module 106 can create a data structure where each action exists as a node in a connected graph. The MBM module 106 can then validate that each node corresponds with a plug-in stored in a plug-in library 114. Each corresponding plug-in can be specific to an autonomous vehicle platform 110. After validation, the MBM module 106 can generate plural action request commands. The AAC module 108 can be configured to execute one or more action plug-ins for each node of each mission script. For example, the AAC module 108 can execute one or more plug-ins for each action request command.
Referring to
The MBM module 106 can include a script manager module 302 configured to determine whether the mission script received is a new mission script by comparing it to mission scripts stored in a mission script library 304. If it is determined that the mission script received is a new mission script, the script manager module 302 can add (e.g., store) the mission script to the mission script library 304. If the mission script received is not new, the script manager module 302 will update the local copy if it has a later timestamp than the stored version; otherwise, no action is necessary by the script manager module 302.
The MBM module 106 can include a mission graph translator (MGT) module 306 configured to parse the text file by giving a meaning to characters of the text file. The MGT module 306 can be configured to create the connected graph data structure by identifying and linking basic data structure units of the text file. The MGT module 306 can be further configured to create plural data structure nodes and link two or more data structure nodes.
The MGT module 306 can be configured to create plural data structure nodes for the mission script. Each data structure node can define an action for an autonomous vehicle platform 110. The MGT module 306 can arrange the plural data structure nodes in a connected graph manner such that a data structure node is required to be successfully accomplished or failed to be accomplished before moving on to another data structure node. This can facilitate execution of actions in a serial or parallel manner. In addition, the MGT module 306 can be configured to generate an error message when a node does not correspond with a plug-in stored in a plug-in library 114. This error message can be a signal sent to the one or more processors 102 that causes the one or more processors 102 to generate an audio and/or visual indication of the error. For instance, the one or more processors 102 can include a display 116 (e.g., a LCD monitor) that generates an audio and/or visual indication.
The MBM module 106 can include a behavior graph manager (BGM) module 308 configured to start the mission script. The BGM module 308 can further monitor results for each data structure node. As nodes are processing in the AAC module 108, feedback and result status are updated at a frequency determined by the developed plug-in. The feedback and status data points are utilized by the MBM module 106 to emit telemetry and mission status. Based on the results, the BGM module 308 can generate a corresponding plug-in command.
In some embodiments, autonomous action deployment system 100 includes the plug-in library 114—e.g., plug-in library 114 can be part of the system 100 or the autonomous action deployment system 100 includes the above-mentioned components in combination with plug-in library 114. In other embodiments, the plug-in library 114 can be a separate component (e.g., not controlled or operated by the system 100) from the system 100 but in communication with the system 100. In either case, the AAC module 108 is configured to execute plug-ins from the plug-in library 114. The AAC module 108 can be configured to utilize a set of pre-developed plug-ins for a particular autonomous vehicle platform 110 based on the mission script. For instance, the AAC module 108 can select an action plug-in from the plug-in library 114 for an autonomous vehicle platform 110 based on the action request commend generated by the MBM module 106. The AAC module 108 can then initialize the action plug-in for the autonomous vehicle platform 110 with a parameter and a goal set by the mission script. The initialization can occur via a dynamic library loader, for example. It should be noted that the pre-developed plugins can be action plug-ins used on any number of autonomous vehicle platforms 110, provided the capabilities of each are similar. Thus, the AAC module 108 can select one or more action plug-ins from the plug-in library 114 for one or more autonomous vehicle platforms 110 based on the action request commend(s) generated by the MBM module 106. The AAC module 108 can then initialize the one or more action plug-ins for the one or more autonomous vehicle platforms 110 with a parameter and a goal set by the mission script.
The autonomous vehicle platform 110 is for an autonomous vehicle. Thus, any one of the autonomous vehicle platforms 110 can include a hardware payload and a software payload for an autonomous vehicle.
The autonomous action deployment system 100 includes a single autonomous vehicle platform 100. The autonomous action deployment system 100 can communicate and otherwise interact with other autonomous action deployment systems 100 which are associated with their own autonomous vehicle platform 100.
Embodiments can relate to a method of deploying an autonomous action. The method can involve receiving a mission script that includes a text file. The method can involve parsing the text file into actions. The method can involve creating a data structure where each action exists as a node in a connected graph. The method can involve validating that each node corresponds with a plug-in stored in a plug-in library specific to an autonomous vehicle platform. The method can involve generating an action request command. The method can involve executing steps to accomplish the action request command for the autonomous vehicle platform via a corresponding driver.
The method can involve receiving plural mission scripts. The method can involve executing one or more action plug-ins for each node of each mission script.
Creating the connected graph data structure can involve identifying and linking basic data structure units of the text file. The method can involve creating plural data structure nodes for the mission script, each data structure node defining an action for an autonomous vehicle platform. The method can involve arranging the plural data structure nodes in a connected graph manner such that a data structure node is required to be successfully accomplished or failed to be accomplished before moving on to another data structure node.
As can be appreciated from the present disclosure, the CPAS framework is a full-stack autonomy solution that is deployable on any generalized unmanned system. It is vendor agnostic and easily integrated with other third party robotic systems. Today's autonomy ecosystem lacks the scalability and reusability necessary to allow for effective robot proliferation in support of mission objectives, with vendor lock-in creating duplicative development efforts and/or rigid systems. Currently, there is no framework that enables the connection of mission events with autonomous actions across multiple domains and hardware types.
The system 100 and mission agnostic autonomy framework enables users to design and deploy a generalized list of actions to accomplish high level missions. The framework also provides a standard language to support communication between the network of platforms and users. The framework is integratable onto any robotic asset and support non-OEM hardware and software payloads.
It is contemplated for the framework to be an open architecture and software product that enables rapid development, deployment, and reconfiguration of truly platform-agnostic autonomy. It can be ran on any operating system, but an exemplary operating system is Ubuntu OS. CPAS increases agility of unmanned technology deployment and usage, increases mission support capability and force effectiveness & safety, and answers the government's call for interoperability and technological unification. CPAS is comprised of two main components that work in tandem, the Mission Behavior Manager (MBM) and the Autonomous Action Controller (AAC). The AAC has both hardware and software interfaces to vendor specific peripherals, sensors, and payloads. The MBM is able to interact with other unmanned platforms in its surroundings for the purpose of information sharing and goal collaboration—i.e., MBM translates the mission scripts into actionable pieces of information and passes those commands to the autonomous action controller.
The current problem in robotics and autonomy is that autonomy is very platform specific, if it exists at all. CPAS is a software solution that allows for platform agnostic mission autonomy amongst various unmanned platforms to facilitate multi-agent collaboration and intuitive command and control. Unlike other solutions in the robotic autonomy space, CPAS increases human-machine teaming, is highly scalable and modular, and is network/communication protocol agnostic.
The software framework sits on top of the platform, the platform's command and control, low-level autonomy algorithms, and any non-OEM payloads. Due to its modular design, the framework simplifies the integration of autonomy and intuitive control onto robotic asset(s). The human centric design provides no-code drag and drop mission script builder and easy integration into familiar end-user devices.
The following is a description of an exemplary software flow and general process of sending a mission script to the MBM and then the running of the mission.
Exemplary process steps for mission script generation, reception, and processing can involve the following:
Exemplary process steps for operations within the ACC can involve the following:
Exemplary pseudo-code for the CPAS framework can include the following:
Referring to
One of the novel aspects of the CPAS framework lies in the implementation of the MBM, AAC, and Mission Scripts. Regarding the MBM and mission script interchange, the MBM runs a customized implementation of a hybrid Finite State Machine (FSM) and Behavior Tree (BT) to perform tasks that together define a mission. Below is a table of differences between traditional FSMs & BTs and the CPAS control structure.
One of the technical improvements of the system 100 is the interaction of the MBM and the AAC. Each node in a mission script defines an action that the platform needs to accomplish or fail before moving on to the next action. A particular platform must be loaded with the plugin libraries for each action that is contained within the mission script. The MBM reads from the available plugins in the libraries and uses that to scrub the current mission for action if it can complete a mission or not. This can be referred to as Mission Graph Introspection (see
Some embodiments of the mission scripts and the AAC themselves may not poses novel concepts. For instance, a mission script may be a series of user defined connected graphs using a CPAS defined schema for input, output, and data transition ports. The AAC can manage a set of CPAS plugins that contain hardware or software payloads so that they can be called from the currently running mission. An item to note is that once a plugin is developed for a particular hardware or software payload, it can be used on any CPAS system with minor modifications, if they are needed.
It will be understood that modifications to the embodiments disclosed herein can be made to meet a particular set of design criteria. For instance, any of the components of the system 100 can be any suitable number or type of each to meet a particular objective. Therefore, while certain exemplary embodiments of the system 100 and methods of using the same disclosed herein have been discussed and illustrated, it is to be distinctly understood that the invention is not limited thereto but can be otherwise variously embodied and practiced within the scope of the following claims.
It will be appreciated that some components, features, and/or configurations can be described in connection with only one particular embodiment, but these same components, features, and/or configurations can be applied or used with many other embodiments and should be considered applicable to the other embodiments, unless stated otherwise or unless such a component, feature, and/or configuration is technically impossible to use with the other embodiments. Thus, the components, features, and/or configurations of the various embodiments can be combined in any manner and such combinations are expressly contemplated and disclosed by this statement.
It will be appreciated by those skilled in the art that the present invention can be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The presently disclosed embodiments are therefore considered in all respects to be illustrative and not restrictive. The scope of the invention is indicated by the appended claims rather than the foregoing description and all changes that come within the meaning, range, and equivalence thereof are intended to be embraced therein. Additionally, the disclosure of a range of values is a disclosure of every numerical value within that range, including the end points.
This patent application is related to and claims the benefit of priority of U.S. provisional patent application No. 63/343,746, filed on May 19, 2022, the entire contents of which is incorporated herein by reference.
Number | Date | Country | |
---|---|---|---|
63343746 | May 2022 | US |