Patent Application
20240257648
Uncrewed aircraft systems (UAS) or drones are widely used in a huge variety of applications. As a result, drone activities have gradually become common scenes in everyday life. However, unauthorized drone activities such as those around airports, stadiums, borders, and prisons can have negative economic impacts and raise public safety and security concerns. Thus, drone activity monitoring and mitigation are of great importance to law enforcement and airport security, among others.
Aspects of this disclosure relate to systems and methods for providing drone activity cloud services to cloud consumers using cloud and edge computing. The drone monitoring service is rendered by drone sensors detecting and identifying drones, cloud and edge servers aggregating drone activity data from sensors and UAS traffic management systems, and cloud consumers monitoring drone activities using cloud and edge devices to access the cloud. The drone data analytics service reports drone activity statistics, predicted drone activities, and abnormal behaviors to cloud consumers based on the statistics and behavior models obtained by machine learning and federated learning techniques. The drone mitigation service, when initiated by cloud consumers, determines how to optimally configure sensors and collaboratively send signals to deactivate unauthorized drones. Moreover, data processing, artificial intelligence, mobility support, and traffic management functional units empower cloud and edge servers to support these cloud services.
An aspect is directed to a system for providing drone activity cloud services to cloud consumer devices. The system comprises a drone sensor configured to detect and identify a drone; a cloud server and an edge server configured to aggregate drone activity data from the drone sensor and an uncrewed aircraft systems (UAS) traffic management system; and a cloud consumer device configured to monitor drone activities using the cloud server and an edge device to access the drone activity data from the cloud server and the edge server, wherein the cloud server and the edge server are further configured to implement a drone data analytics service that reports one or more of drone activity statistics, predicted drone activities, and abnormal behaviors to the cloud consumer device based on statistics and behavior models obtained by machine learning and federated learning techniques, wherein a drone mitigation service, when initiated by the cloud consumer device, is configured to determine how to configure the drone sensor and collaboratively send signals to deactivate an unauthorized drone.
In some embodiments, the system further comprises: a cloud network comprising the cloud server and the cloud consumer device; and an edge network comprising the edge server and the edge device.
In some embodiments, the cloud consumer device comprises one or more of the following: a smartphone, a tablet, a virtual reality device, an augmented reality device, a mixed reality device, a laptop, a desktop computer, a sensor node, and/or a drone.
In some embodiments, the cloud server and the edge server each comprise: a data processing unit configured to filter, fuse, and/or aggregate the drone activity data for a drone monitoring service; and an artificial intelligence unit configured to implement the machine learning and/or the federated learning techniques for the drone data analytics service.
In some embodiments, the cloud server and the edge server each comprises: a sensor control unit configured to determine parameters for smart sensor configurations and/or intelligent jamming; a mobility support unit configured to handle a drone activity handoff at a boundary of a cloud or edge network when the drone moves from one network to another; and a traffic management unit configured to handle cooperative interactions with the UAS traffic management system.
In some embodiments, the drone data analytics service is further configured to generate an activity notification and a trajectory for the identified drone.
In some embodiments, the trajectory and the activity data are overlaid on a map with real-time updates.
An aspect is directed to a method for providing drone activity cloud services to cloud consumer devices. The method comprises detecting and identifying one or more drones using a drone sensor; aggregating, at a cloud server and an edge server, drone activity data from the drone sensor and an uncrewed aircraft systems (UAS) traffic management system; monitoring, at the cloud consumer device, drone activities using the cloud consumer device and the edge device to access the drone activity data from the cloud server and the edge server; implementing, at the cloud server and the edge server, a drone data analytics service that reports one or more of drone activity statistics, predicted drone activities, and abnormal behaviors to the cloud consumer device based on statistics and behavior models obtained by machine learning and federated learning techniques; and determining how to configure the drone sensor and collaboratively send signals to deactivate an unauthorized drone in response to initiating a drone mitigation service by the cloud consumer device.
In some embodiments, the cloud server and the cloud device are arranged to form a cloud network; and the edge server and the edge device are arranged to form an edge network.
In some embodiments, the cloud consumer device comprises one or more of the following: a smartphone, a tablet, a virtual reality device, an augmented reality device, a mixed reality device, a laptop, a desktop computer, a sensor node, and/or a drone.
In some embodiments, the cloud server and the edge server each comprise: a data processing unit configured to filter, fuse, and/or aggregate the drone activity data for a drone monitoring service; and an artificial intelligence unit configured to implement the machine learning and/or the federated learning techniques for the drone data analytics service, abnormal behavior detection, and/or collaborative learning.
In some embodiments, the cloud server and the edge server each comprise: a sensor control unit configured to determine parameters for smart sensor configurations and/or intelligent jamming; a mobility support unit configured to handle a drone activity handoff at a boundary of a cloud network or edge network when the drone moves from one network to another; and a traffic management unit configured to handle cooperative interactions with the UAS traffic management system.
In some embodiments, the method further comprises: generating, using the drone data analytics service, an activity notification and a trajectory for one of the identified drones.
In some embodiments, the trajectory and the activity data are overlaid on a map with real-time updates.
Another aspect is a hybrid network for providing drone activity services. The hybrid network comprises a cloud network comprising: a plurality of cloud servers and a plurality of cloud devices; and an edge network comprising: a plurality of edge servers and a plurality of edge devices, wherein the plurality of cloud devices and the plurality of edge devices comprise drone sensors configured to detect and identify drones, wherein the plurality of cloud servers and the plurality of edge servers are configured to aggregate drone activity data from the drone sensors and an uncrewed aircraft systems (UAS) traffic management system, and wherein the plurality of cloud devices and the plurality of edge devices are further configured to monitor drone activities and to access the aggregated drone activity data from the plurality of cloud servers and the plurality of edge servers.
In some embodiments, the plurality of cloud servers and the plurality of edge servers are further configured to implement a drone data analytics service that is configured to report one or more of drone activity statistics, predicted drone activities, and abnormal behaviors to the plurality of cloud devices based on statistics and behavior models obtained by machine learning and federated learning techniques, and wherein a drone mitigation service, when initiated by the plurality of cloud devices, is configured to determine how to configure the drone sensors and collaboratively send signals to deactivate an unauthorized drone.
In some embodiments, the plurality of cloud devices comprises one or more of the following: a smartphone, a tablet, a virtual reality device, an augmented reality device, a mixed reality device, a laptop, a desktop computer, a sensor node, and/or a drone.
In some embodiments, each of the plurality of cloud servers and of the plurality of edge servers comprises: a data processing unit configured to filter, fuse, and/or aggregate the drone activity data for a drone monitoring service; and an artificial intelligence unit configured to implement the machine learning and/or the federated learning techniques for the drone data analytics service, abnormal behavior detection, and/or collaborative learning.
In some embodiments, each of the plurality of cloud servers and of the plurality of edge servers comprises: a sensor control unit configured to determine parameters for smart sensor configurations and/or intelligent jamming; a mobility support unit configured to handle a done activity handoff at a boundary of the cloud network or of the edge network when a drone moves from one network to another; and a traffic management unit configured to handle cooperative interactions with the UAS traffic management system.
In some embodiments, the drone data analytics service is further configured to generate an activity notification and a trajectory for one of the identified drones.
Drone monitoring is typically provided by drone detection or counter-UAS (C-UAS) systems. These systems often operate independently and provide limited drone activity information and services. Moreover, even if connected to networks, they are typically not scalable and do not collaborate with other systems in the networks. Therefore, next-generation C-UAS systems are being designed to be part of the modern cloud infrastructure, and have artificial intelligence (AI) capabilities to provide top-notch drone activity services.
Recent advances in edge computing provide a suitable cloud infrastructure for large-scale drone monitoring and mitigation. First, since drone monitoring and mitigation are highly delay-sensitive, edge computing reduces the latency of the cloud service due to the close proximity of edge servers to drone sensors and customers. Second, compared to direct connections between drone sensors and cloud servers, edge computing can support large and dense distributed deployment of drone sensors, and facilitate the collaboration among sensors to achieve better detection and mitigation performance. Moreover, with edge computing, data analytics and machine learning can be more efficient and accurate, which offers more satisfying customized customer experiences.
Aspects of this disclosure relate to primary drone activity services for drone monitoring, drone data analytics, and drone mitigation based on the edge computing paradigm. In traditional drone monitoring systems, edge computing has been used to control authorized drones and manage their activities for performing certain tasks such as delivery and event monitoring. These solutions typically have the full control of those drones.
In contrast to traditional drone monitoring systems, aspects of this disclosure provide cloud services using edge computing to monitor both authorized and unauthorized drone activities, and mitigate unauthorized drone activities without requiring direct control over those drones, authorized or unauthorized. In certain embodiments, the system can implement monitoring and mitigation of drone activities using edge computing based on federated learning techniques for the data analytics of drone activities.
Cloud networks include cloud servers and cloud devices connected via wired (e.g. Ethernet, optical) and/or wireless connections (e.g. Wi-Fi, 4G LTE/5 G NR/6G cellular, satellite). The cloud infrastructure may include one or more edge networks connected to cloud networks. Nodes in the cloud infrastructure can be servers, gateways, mobile devices, sensors, or any device that has processors, storage, memory, and computing capabilities.
Cloud servers are typically located at data centers, which can be physically far away from edge networks and cloud consumers. Edge networks, on the other hand, are typically in close proximity to cloud consumers compared to cloud servers. Edge networks can also be densely deployed in the same or different geographical areas.
The one or more cloud devices 206 may include smartphones, tablets, virtual reality (VR)/augmented reality (AR)/mixed reality (MR) devices, laptops, desktop computers, sensor nodes, drones, and other devices. Cloud devices can be either connected to the one or more edge servers 210 in the one or more edge networks 208 to access the cloud, or directly connected to the cloud (e.g., via the one or more cloud servers 204) if no edge server 210 is in close proximity. In the former case, cloud devices in the edge networks 208 are called edge devices 212. Note that cloud devices may access the cloud via a satellite link if no terrestrial network infrastructure is available in the area.
The one or more edge networks 208 is formed by connecting edge devices 212 to edge servers 210/gateways via wired or wireless connections in a geographical area. The edge servers 210 perform edge computing, data processing, and data analytics, store/retrieve data to/from the databases in local storages, and send/receive data to/from the cloud and edge devices 212. In addition to typical network management functions, there can be at least five functional units in cloud and edge servers 210 to support cloud services.
The cloud or edge server 300 can also include:
A data processing unit 314 configured to handle drone activity data filtering, fusion, and aggregation for drone monitoring service;
An AI unit 316 configured to handle machine learning, deep learning, and federated learning tasks for data analytics, abnormal detection, and collaborative learning;
A sensor control unit 318 configured to compute and/or determine parameters for smart sensor configurations and intelligent jamming;
A mobility support unit 320 configured to handle the done activity handoff at the cloud or edge network boundary when drones move from one network to another; and/or
A traffic management unit 322 configured to handle the cooperative interactions with the UAS Traffic Management (UTM) system.
The cloud or edge server 300 can also be connected to a network management component 324.
Sensor nodes (or simply referred to as sensors) can be deployed as cloud/edge devices (e.g., cloud devices 206 and/or edge devices 212 of
There are at least two types of cloud consumers in this paradigm: C-UAS system providers (also referred to as providers) and C-UAS system customers (also referred to as customers). Providers can include cloud consumers that deploy multiple sensor nodes and edge gateways/servers 300 in each geographical location for detecting drones and collecting drone activity data. Customers can include cloud consumers that subscribe to drone monitoring and mitigation services. Customers can be provided with access to the drone activity data and visualize the drone activity data on a map as well as initiate drone mitigation through the cloud user interface (e.g. apps or web browsers). Customers' smart devices can also detect drones and report drone activity data. In this case, customers can also act as sensors.
There are at least three primary types of drone activity services: monitoring, data analytics, and mitigation.
When sensors detect the presence of one or more drones, the sensors can generate drone activity data, and send the raw data and/or generated data to edge and/or cloud servers. The drone activity data can be synchronously or asynchronously sent from sensors to servers. The data received by servers may contain duplicate or redundant information from multiple sensors regarding a particular drone target.
The cloud or edge servers can receive drone activity data from sensors and the UTM, filter redundant information, remove duplicates, combine data, generate data analytics, update statistics and learning models, store selected data locally, and/or send selected data to cloud servers for processing and storage at data centers. Thus, the duplicate and redundant information can be removed such that the redundant information is not presented to customers.
The drone activity data stored in the edge or the cloud can be retrieved by cloud consumers (both providers and customers) at any location with connections anytime through the cloud user interface using any cloud or edge device. The provider and consumer software in the servers and devices are capable of retrieving and showing drone activities on a 2D or 3D map (e.g. satellite map, street-view) with real-time updates. Customers can visualize the drone activities in real time such as drone trajectory and drone home points on a map in a selected area. Visualizations present a unified accounting of drone activities due to the data aggregation and collection performed by cloud or edge servers. Customers will also receive notifications and statistics of drone activities in the areas of interest.
Upon receiving the real-time drone constraints from the UTM, the cloud and servers in the areas can monitor the drone activities and analyze the activity data. The cloud and servers can also report any violation to the UTM, and notify customers (e.g. law enforcement) via the API on cloud and edge devices.
The data analytics can be performed regularly by incorporating new data with previously collected data in order to generate statistics of drone activities and build customer behavior profiles based on their usage of service. These models are established and adaptively updated using data sent from cloud and edge devices (e.g. sensors or customers), and federated learning techniques that facilitate collaborative machine learning.
The drone data statistics may be inquired by providers and customers. Some examples are:
The customer behavior profiles and models can show customers' preference regarding cloud service usages and drone activity inquiries. The profiles can provide customized information via cloud user interfaces for better customer service, prioritizing global/local data storage at cloud/edge servers to minimize the latency of data retrieval, and so on.
The drone activity prediction service can provide customers the prediction of drone activities based on drone data analytics, activity statistics, and customer inquiries. For example, law enforcement may be interested in the possible locations of unauthorized drone activity the next day, or the real-time predicted drone trajectory.
The anomaly detection can be used to detect the abnormal behavior of any drone target. The anomaly detection can generate warnings or alert customers for any concern (e.g. public safety) in real time. The anomaly detection can also detect the abnormal behavior of any server and device connected to the networks. The anomaly detection can notify providers to investigate any system failure or security issue. This service can detect malfunctioning sensors, compromised edge devices, and intruders to ensure system security, data integrity, and customers' privacy.
Customers can make requests via the cloud user interface to mitigate the activity of one or more unauthorized drones in real time. After receiving the requests from customers, the cloud or edge server sends commands to selected sensor nodes based on the mitigation strategy for the targets, configures the transmitters and antennas of these nodes for beamforming and collaborative jamming. The communications between the targets and their pilots are then disrupted. The deactivated unauthorized targets are either forced to land or return to their bases where their pilots are located.
In addition to intelligent jamming, the service can dynamically configure sensors to optimize sensing schedule, reduce power consumption, increase sensing range, and improve detection accuracy making drone detection more effective and efficient.
Customers may have a fleet of drones (authorized drones) for tasks or missions that can be configured, instructed, managed, and/or controlled by customers' authorized drone pilots. The cloud services provide drone activity information and visualization to authorized drone pilots, and assist them in their tasks and missions. For example, the service provides drone activities (both authorized and unauthorized drones) to law enforcement so that their authorized pilots can cooperate with each other, fly their drones in a seamless fashion, pursue unauthorized drones, and find the locations of unlawful pilots. Similarly, the service can assist first responders in their disaster relief missions by providing the authorized drone activities for their pilots to operate their drones collaboratively to survey the disaster areas, coordinate their operations, and monitor the progress.
Implementations disclosed herein provide systems, methods and apparatus for drone monitoring, data analytics, and mitigation cloud services using edge computing. It should be noted that the terms “couple,” “coupling,” “coupled” or other variations of the word couple as used herein may indicate either an indirect connection or a direct connection. For example, if a first component is “coupled” to a second component, the first component may be either indirectly connected to the second component via another component or directly connected to the second component.
The drone detection, analytics, and mitigation functions described herein may be stored as one or more instructions on a processor-readable or computer-readable medium. The term “computer-readable medium” refers to any available medium that can be accessed by a computer or processor. By way of example, and not limitation, such a medium may comprise RAM, ROM, EEPROM, flash memory, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. It should be noted that a computer-readable medium may be tangible and non-transitory. As used herein, the term “code” may refer to software, instructions, code or data that is/are executable by a computing device or processor.
The methods disclosed herein comprise one or more steps or actions for achieving the described method. The method steps and/or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is required for proper operation of the method that is being described, the order and/or use of specific steps and/or actions may be modified without departing from the scope of the claims.
As used herein, the term “plurality” denotes two or more. For example, a plurality of components indicates two or more components. The term “determining” encompasses a wide variety of actions and, therefore, “determining” can include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining and the like. Also, “determining” can include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like. Also, “determining” can include resolving, selecting, choosing, establishing and the like.
The phrase “based on” does not mean “based only on,” unless expressly specified otherwise. In other words, the phrase “based on” describes both “based only on” and “based at least on.”
The previous description of the disclosed implementations is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these implementations will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other implementations without departing from the scope of the invention. For example, it will be appreciated that one of ordinary skill in the art will be able to employ a number corresponding alternative and equivalent structural details, such as equivalent ways of fastening, mounting, coupling, or engaging tool components, equivalent mechanisms for producing particular actuation motions, and equivalent mechanisms for delivering electrical energy. Thus, the present invention is not intended to be limited to the implementations shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
This application claims the benefit of U.S. Provisional Application No. 63/481,693, filed Jan. 26, 2023, which is hereby incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63481693 | Jan 2023 | US |
