Patent Application
20240142978
An unmanned vehicle, or drone is a type of vehicle that can be powered or unpowered without a person directly operating it onboard. The vehicle can be operated remotely (e.g., by a human operator/pilot) or autonomously (e.g., using sensors and/or navigational programming). Unmanned vehicles can be designed for different environments, such as, but not limited to, unmanned aerial vehicles (UAV), unmanned ground vehicles (UGV), unmanned surface vehicles (USV), and unmanned underwater vehicles (UUV).
Drones can utilize any of a variety of sensors such as, but not limited to, cameras, infrared (thermal) sensors, LiDAR (Light Detection and Ranging), sonar, etc. At an elevated vantage point, UAVs having onboard sensors can often collect data at a greater range and with less influence from obstructions than if they were on the ground. UGVs, USVs, and UUVs with sensors can traverse and collect information in environments that are difficult or undesirable for a human.
Drones for information gathering over a large area are particularly useful in emergency and disaster situations. With little to no direct human supervision, they can obtain visual and other information at great speed and effectiveness to enhance planning and remediation by responders.
Systems and methods for fusing sensor data from drones in a virtual environment are described In one embodiment, a method for fusing sensor data from drones in a virtual environment includes obtaining geometry data describing a real-world landscape, drawing a map within a virtual environment using a 3-D visualization software and the geometry data, placing a plurality of projectors on the map within the virtual environment corresponding to sensors in the region of the real-world landscape, receiving sensor data and location data from the sensors; and projecting the sensor data onto the map at locations indicated by the location data using the projectors corresponding to the sensors that the sensor data is received from.
The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.
The description and claims will be more fully understood with reference to the following figures and data graphs, which are presented as exemplary embodiments of the invention and should not be construed as a complete recitation of the scope of the invention.
Turning now to the drawings, systems and methods for fusing sensor data from drones in a virtual environment are described. Drones can be used to collect information over large geographic areas via onboard sensors. As will be discussed further below, information from multiple drones can be merged or fused live or close to real-time into a virtual environment (a virtual representation within a computing system) that is representative of the physical real-world environment traversed by the drones. The virtual environment can be implemented in a computing system using 3-D visualization software such as a game engine (e.g., Unity, Unreal Engine, Godot, etc.). An initial map can be set up in the virtual environment using real-world geographic information concerning the area of interest (e.g., landscape, buildings, physical features, etc.), which can be referred to as geometry data. Information captured by sensors on the drones (e.g., a video feed) or other sensor systems (e.g., stationary cameras) can then be projected or superimposed onto the map built from geometry data. The drones can follow defined paths, navigate autonomously, or be manually controlled to cover as much of the area of interest as possible. A user interface displays the virtual environment and can provide controls for a user to direct a drone to a specific location. In this way, a user can visually review information over large areas live or in close to real-time via systematic navigation of the one or more drones.
Systems for Collecting Sensor Data from Drones
The drone command center 110 can include controller interfaces for the drones. In several embodiments of the invention, each drone has its own associated controller interface, e.g., Pixhawk Cube. The drone command center 110 may also have one or more computing systems that can coordinate the controller interfaces, execute a 3-D visualization software application (e.g., game engine) for the virtual environment, and/or generate information for a user interface on the one or more client devices 108 and 110 to display the virtual environment. Processes that may be performed at a drone command center 110 include those discussed further below.
The data center 112 can include one or more databases. Databases can store drone information/metadata and geometry data. As will be discussed further below, drone metadata includes information about the capabilities of each drone or information to configure each drone. Geometry information includes mapping data of some location in the real world that can be used to render a virtual environment. Notably, in some embodiments, separate data centers can house databases for different types of information.
A computing system that may be utilized at a command center in accordance with embodiments of the invention is conceptually illustrated in
Rendering a virtual environment in accordance with embodiments of the invention involves collecting initial geometry data for constructing a base landscape or map. Geometry data is information that gives a representation of the shape of bare ground (bare earth) topographic surface of the Earth. In some embodiments, geometry data can also include trees, buildings, and other surface objects. Geometry data can be obtained in any of a variety of ways, such as by retrieving from data sources that can be queried or have APIs. When the system has a network connection, Internet sources can include retrieving geometry data from a server such as, but not limited to, Esri ArcGIS, Google Earth, Google Maps, United States Geological Survey (USGS) digital elevation model (DEM), or Mapbox. When there is no network connection, or without needing to use a network connection, the system can accept locally generated geometry data. For example, drones or other devices can be used to collect geometry data using LiDAR.
While geometry data can be organized in any of a variety of ways, many embodiments utilize layers as logical collections of data for creating maps, scenes, and analysis. The data can include different aspects of an area, such as topography, elevation, natural features, buildings, etc.
In many embodiments of the invention, the virtual environment can be built using a 3-D visualization software application, such as a game engine, based on geometry data. Unity, Unreal Engine, and Godot are examples of game engines that may be utilized in accordance with embodiments of the invention.
The geographic background or structure of the virtual environment, which can be constructed from geometry data, can be referred to as a map. The map can be centered on a location as directed by a user or provided by the GPS of a device (e.g., a mobile device). For example, a user interacting with a graphical user interface may enter GPS coordinates (e.g., in WGS83 format) or click on a location in the interface. Alternatively, a location can be determined without user input from the GPS onboard a mobile device that the user is using to view the map. An example of a graphical user interface screen for obtaining input concerning a location for centering the map is shown in
Active drones can be placed on the map as projectors within the virtual environment. Many 3-D visualization software application, such as a game engines, include a projector component, which is a class that can be instantiated and used to project any material onto a scene. As will be discussed in greater detail further below, image, video, or other sensor data collected by each drone can be projected onto the map with proper placement given telemetry data of the drone. In many embodiments, information on how to attain and parse telemetry data of a drone may be stored in and retrieved from a database such as data center 112 or 160 above. The database includes information for each drone to be used such as, but not limited to, drone type, how to obtain and parse telemetry, command hash table, control type, and format of video or other sensor data captured by the drone. Collectively this type of information can be referred to as drone metadata. Telemetry data can include coordinates of the current location of the drone and position of the camera or sensor (e.g., gimbal orientation), and may be stored in JSON or XML format. In several embodiments of the invention, the sensor data is provided in XML.
Processes for Fusing Sensor Data from Drones in a Virtual Environment
A computing system at a command center or elsewhere can coordinate multiple drones to collect sensor data for display in a virtual environment. A process for collecting and displaying sensor data in accordance with an embodiment of the invention is illustrated in
The active drones are placed (304) on the map using their locations (e.g., each as provided by their GPS). Drones are active when they are contributing sensor data. In many embodiments of the invention, the drones are implemented as projector components as provided by the 3-D visualization software within the virtual environment. Additional detail of processes for placing drones on a map will be discussed further below with respect to
Sensor data from the active drones that have been placed on the map are projected (306) onto the map using the projectors corresponding to each active drone. Additional details of processes for projecting sensor data on a map will be discussed further below with respect to
User input captured (308) on a graphical user interface may instruct a drone to travel to a location or in a particular direction. The user input may be entered as coordinates (e.g., WGS83 format). Alternatively, the user input may be indicative of a direction (e.g., elevation up/down, slide left/right, forward, and reverse). Updating drone tasking from user input will be discussed in greater detail below with respect to
Although a specific process is described above with respect to
Processes in accordance with embodiments of the invention can retrieve information or metadata about the drones to be able to interface with them and facilitate conveying location and sensor data to the command center. Such processes maybe utilized, for example, in drone placement 304 of
The process 400 includes retrieving (402) the status and type of each live drone from the drone information database. The statuses can include active and inactive. Active status can indicate the drone is out in the real-world environment and ready to transmit data (e.g., a UAV is “launched”). Inactive status can indicate the drone is withdrawn from the field or powered down. In many embodiments of the invention, the area of interest is divided into sectors, and drones are assigned to sectors. The drones can be programmed using controllers such as those described further above to traverse their assigned sectors, for example, by providing waypoints.
A list can be created of drones having active status. Then for each of the drones in the list, telemetry and control information is retrieved (404) from the drone information database. Telemetry and control information can include, but is not limited to, a command hash table, drone control type, native sensor data format of sensor(s) on the drone, and/or information for converting the native sensor data format to a uniform format.
Telemetry is retrieved from one or more of the active drones. Telemetry can include, but is not limited to, location of the drone and sensor data captured by one or more sensors on the drone. In several embodiments of the invention, at least one sensor on a drone is a video camera providing a video feed or stream as sensor data.
The telemetry is parsed (406) and location coordinates for the drone(s) are converted into the coordinate system of the virtual environment. Telemetry may be in a suitable storage or messaging data format, such as JSON or XML. In some embodiments of the invention, drone location coordinates are provided as WGS83 format and converted into vector 3 coordinates (x, y, z) for the virtual environment. Furthermore, AGL (above ground level) altitude can be converted to MSL (mean sea level) altitude. An icon for the drone(s) is displayed (408) in the virtual environment at the vector 3 coordinates. A projector can be associated with the icon within the virtual environment for adding sensor data to be displayed on the map as will be described further below. Similarly, any new drones that are discovered or become active can be added to the map by an icon and projector. An example screen of a graphical user interface showing a map and several placed drones represented by icons in accordance with an embodiment of the invention is illustrated in
Although a specific process is described above with respect to
Processes for Projecting Sensor Data from Drones onto a Map
Once the location(s) of drone(s) are known, the sensor data captured by the drone(s) can be displayed on the map.
The process 500 includes retrieving (502) telemetry of a drone. As mentioned above, telemetry can be in JSON or XML format. Sensor data captured by a sensor on the drone is extracted (504) from the telemetry. In some embodiments of the invention, the sensor data is a video frame or portion of video. In other embodiments, sensor data can be multi-spectral, chemical, auditory, or thermal. As discussed further above, any of a variety of sensors may be utilized on a drone to capture different information about the real-world environment.
The sensor data is converted (506) to a visual format. In some embodiments utilizing video, the video frame is converted into an image (e.g., JPEG).
The orientation of the sensor (e.g., if it is on a gimbal) is determined (508). The orientation can be obtained, for example, from the telemetry, and can include the location or position of the sensor or gimbal; the degree of tilt, roll, and pan of the gimbal; and/or field of view of the camera. In some embodiments, when the location or position of the sensor or gimbal is not provided by the drone, it can be assumed to be at fixed default values or can be retrieved from the drone information database. Field of view may be provided in a measurement, such as degrees, or may be provided by calculating from lens size and sensor size of the camera.
A projector associated with the drone is rotated (510) within the virtual environment to match the sensor/gimbal orientation. The visual representation of the sensor data (e.g., image) is projected by the projector onto the map at the proper location and orientation previously determined to match the drone. An example screen of a graphical user interface showing a map and choices of available sensor data as standard (RGB) and thermal (IR) spectrum in accordance with an embodiment of the invention is illustrated in
In several embodiments of the invention, the projected visual sensor data remains on the map as the drone leaves the corresponding location in the real world. The map can be updated when new sensor data is available (e.g., from a different drone passing over the area or a next pass of the original drone).
Although a specific process is described above with respect to
A user of the system may wish to get up-to-date information in a specific location on the map. They may utilize a controller interface at the command center to task a drone to visit that area.
The process 600 includes receiving (602) user input. An example graphical user interface showing a drone management screen is illustrated in
The user input can be converted into instructions (604) for drone control. For example, if the user input is a set of WGS83 coordinates, it can be converted into directions for the drone to arrive at those coordinates. If the user input is selection of a location on a map, the coordinates of the location can be determined and then provided as directions for the drone. Alternatively, user input can be given as immediate controls (e.g., forward, reverse, slide or turn left/right, elevation up/down, rotate gimbal, etc.). One skilled in the art will recognize that other variations of drone control are possible. The drone is directed (606) using the instructions.
Updated sensor data is received (608) from the drone and displayed in the virtual environment (e.g., by the drone's associated projector).
Although a specific process is described above with respect to
Although the description above contains many specificities, these should not be construed as limiting the scope of the invention but as merely providing illustrations of some of the presently preferred embodiments of the invention. Various other embodiments are possible within its scope. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.
The current application claims the benefit of and priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 63/381,652, entitled “Systems and Methods for Fusing Sensor Data from Drones in a Virtual Environment” to Tyris Monte Audronis, filed Oct. 31, 2022, the disclosure of which is hereby incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63381652 | Oct 2022 | US |
