Patent Grant
11961407
The present technology is directed generally to methods and associated systems for generating, analyzing, and verifying a three-dimensional (3D) flight path for a moveable platform such as an unmanned aerial vehicle (UAV).
Traditionally, the flight path for a UAV is planned based on certain location points (waypoints) identified on a two-dimensional (2D) map. This approach can be inaccurate because it fails to account for the third dimension (e.g., the height) of the objects, such as buildings, structures, and other obstacles that may exist in the UAV's flight path. This approach also fails to accommodate the need to precisely control the UAV during delicate tasks (e.g., flying the UAV in a modern city with tall buildings to deliver a small item). In addition, it takes a long time and a significant amount of practice for the operator of a UAV to become familiar with the path planning tasks using the traditional approach. The traditional approach does not provide an operator of a UAV with intuitive user experiences when operating the UAV. Therefore, there exists a need for improved methods and systems for generating or planning 3D flight paths for a UAV.
The following summary is provided for the convenience of the reader and identifies several representative embodiments of the disclosed technology. Generally speaking, the present technology provides improved methods and associated systems that enable an operator to generate, analyze, and verify a 3D flight path of a UAV in a straightforward, easy-to-learn, and intuitive fashion. More particularly, the present technology enables an operator to create and observe a 3D flight path of a UAV via a virtual reality device. For example, the present technology enables an operator to observe and verify a generated 3D flight path from a first-person perspective via a virtual reality device. By doing so, the operator can verify whether the generated 3D flight path is exactly what he/she wants in order to perform a certain task (e.g., filming a movie or taking a picture of a target person or an object). In addition, the present technology enables an operator to generate an accurate 3D flight path and precisely control a UAV to conduct delicate or demanding tasks. Examples of demanding tasks include delivering a package to an east-facing window on a certain floor of a building, collecting an image of the face of an actor standing at a certain location, and filming a moving target from a particular view angle.
Representative embodiments of the present technology include a method and an associated system for generating a 3D path for a UAV. The method includes receiving a set of 3D information (e.g., a set of geographic information, or coordinates of objects) associated with an environment (e.g., a place where a UAV is operated in the real world, such as an area of a city, a defined space in a structure or a building, or an outdoor area) or a virtual reality environment (e.g., generated based on objects in the real world). The method further includes receiving a plurality of virtual locations in the virtual reality environment. In some embodiments, the method can receive physical locations in the real world environment and then transform the same into virtual locations in the virtual reality environment. This can be done by (1) a user input (e.g., a user enters the coordinates of particular locations), (2) a user's selection from a recommended list (e.g., an associated system provides a list of candidate locations from which the user makes a selection), or (3) retrieving data from a storage device (e.g., locations of a path that a UAV traveled previously, locations to which the UAV has frequently flown, and/or locations generated based on log or history files associated with an operator or a UAV). For individual virtual or physical locations, the system receives one or more corresponding action items. Representative action items include performing a pre-determined task at an individual location, such as camera aiming, stabilizing the UAV, collecting an image with a specific size or format, collecting information associated with the individual location (e.g., whether an object/individual can be seen by the UAV at the individual location; collecting/measuring virtual/real ambient information at the individual location), configuring a component of the UAV (e.g., adjusting a power output of a UAV power supply component), etc.
The system then receives a plurality of virtual locations (e.g., locations or coordinates in a virtual reality environment) corresponding to the plurality of locations. In some embodiments, the system can receive physical locations and then generate corresponding virtual locations. Once the virtual locations are determined, the system then generates a 3D path (e.g., a 3D trajectory) based on the set of 3D information. For example, the 3D path can be based on a requirement that it can be kept a certain distance from any object described by the set of 3D information. The 3D path is also based on the plurality of virtual locations (e.g., based on a requirement that the 3D path passes all virtual locations), and the plurality of action items (e.g., the action item can be a UAV flying around a target, and in such case the 3D path includes a path around the target). Details of the virtual reality environment will be discussed in Detailed Description below.
The system then generates a set of images associated with the 3D path based on at least one of the set of 3D information, the plurality of virtual locations, and the plurality of action items. For example, the generated images can be a set of images observed from a first-person perspective from a UAV in the virtual reality environment. The system then visually presents the set of images to an operator. In particular embodiments, the set of images can be presented via a virtual reality device. Accordingly, the system enables an operator to observe a proposed 3D flight path in an intuitive manner.
In particular embodiments, the system enables an operator to adjust the generated 3D path manually or automatically based on a user setting. For example, the operator can create additional locations (e.g., by input via a virtual reality device, a keypad, a touch screen, a control stick, and/or other suitable device) to be included in the 3D path in the virtual environment.
In particular embodiments, the system can include an image component coupled to a UAV configured to collect images based on pre-determined action item. In some embodiments, the image component can include a color-sensing camera that collects color images (e.g., those having red, green, and blue (RGB) pixels). In other embodiments, the image-collection component can be a camera (e.g., a thermal/infrared camera, or a night vison camera) that collects various other types of images.
Some embodiments of the present technology can be implemented as methods for configuring a system for planning flight paths or routes for a UAV. The methods can include programming a computer-readable medium with instructions that, when executed, receive a set of 3D information associated with a virtual reality environment (or a real-world environment, in some embodiments) and receive a plurality of virtual locations in the virtual reality environment (or physical locations in the real-world environment which will be transformed into virtual locations). For individual virtual locations, the instructions can include receiving one or more corresponding action items. The instructions can generate a 3D path based on the set of 3D information, the plurality of virtual locations, and the plurality of action items. The instructions can further generate a set of images associated with the 3D path based on the set of 3D information, the plurality of virtual locations, and the plurality of action items. The instructions can visually present the set of images to an operator. In particular embodiments, images are presented via a virtual reality device. In some embodiments, the instructions can adjust the 3D path in response to receiving an instruction from the operator via a virtual reality device. Methods and systems in accordance with embodiments of the present technology can include any one or a combination of any of the foregoing elements described above.
The present technology is directed generally to methods and associated systems for generating, analyzing, and verifying a 3D flight path of a UAV. A representative system configured in accordance with the present technology generates a 3D flight path in a virtual reality environment (e.g., created based on a real-world environment by measuring the dimensions of tangible/physical objects therein and then generating virtual data sets that correspond to the physical/tangible objects) at least partially based on (1) locations that the UAV is to pass, and (2) action items that the UAV is to perform at individual locations. In particular embodiments, the locations can include a real-world location and a virtual reality location. The real-world location can be a set of coordinates corresponding to the real-world environment, and the virtual reality location can be a set of coordinates corresponding to the virtual reality environment. In particular embodiments, the action items can include a task to be performed by the UAV (e.g., rotating to face a different direction) or by a component (e.g., a camera) of the UAV. Representative examples of the action items include (1) aligning an image component of a UAV with a target; (2) positioning an image component of a UAV at a horizontal level; (3) maintaining a view angle of an image component of a UAV; (4) aiming an image component of a UAV toward a target; (5) collecting an image associated with a target via an image component of a UAV; (6) instructing a UAV to fly around a target; and/or (7) instructing a UAV to rotate about an axis.
The system then generates a set of images associated with the 3D flight path. In particular embodiments, the set of images includes virtual reality images that the UAV would collect (according to the locations and corresponding action items) when it flies along the 3D flight path in the virtual reality environment. The flight path or portions of the flight path can be generated a shortest-distance algorithm, or other suitable algorithm, and can account for the expected endurance of the UAV, among other factors. The system then presents the set of images to an operator and provides the operator with an intuitive experience of how environment would look if the UAV were to fly along the generated 3D flight path in a corresponding real-world environment. The system provides the operator an opportunity to review the generated 3D flight path by reviewing of the set of images. In particular embodiments, the operator can adjust the 3D flight path by adding/removing an additional/existing location or action item to the existing 3D flight path via a virtual reality device.
In some embodiments, the system enables the operator to adjust the 3D flight path in a real-time manner. For example, a UAV can take off and fly based on a generated 3D flight path. The real-world images collected by an image component coupled to the UAV can be transmitted to the system and then presented to an operator of the UAV. The operator can then adjust (the not-yet-flown part of) the 3D flight path in the virtual reality environment. Via this arrangement, the system enables the operator to simultaneously monitor and precisely control the UAV to complete delicate, precise, and/or otherwise demanding tasks.
In particular embodiments, the system can generate a 3D flight path at least partially based on one or more rules provided by an operator. For example, these rules can be associated with various factors such as a minimum/maximum distance between a UAV and an obstacle or a target, algorithms for obstacle avoidance (e.g., distance-based, UAV-flight-time based, obstacle based algorithms, etc.), user preferences, and/or other suitable factors.
In some embodiments, an operator can provide the locations or action items to the system via a virtual reality device. In some embodiments, an operator can provide such information by one or more gestures. For example, an operator wearing a virtual reality device on his/her arm can position his/her arm toward a direction in a virtual reality environment, so as to indicate a direction that the operator wants the UAV to face or move toward. As another example, an operator wearing a virtual reality device in front of his/her eyes can blink his/her eyes at a particular location in the virtual reality environment, so as to instruct the system to add this particular location to a 3D flight path. In one example, an operator can input the location information via an input device or a controller. In yet another example, an operator wearing a virtual reality device on his/her hand can use particular hand gestures (e.g., gestures related to a rock-paper-scissors game) to indicate specific action items.
Unlike conventional systems, aspects of the present technology are directed to enabling an operator to generate, analyze, and verify 3D flight paths of a UAV that are suitable for delicate, high-precision and/or other demanding UAV flight tasks. Also, aspects of the present technology can improve the convenience of flight path planning and provide a better and more intuitive user experience than traditional methods. Several details describing structures or processes that are well-known and often associated with UAVs and corresponding systems and subsystems, but that may unnecessarily obscure some significant aspects of the disclosed technology, are not set forth in the following description for purposes of clarity. Moreover, although the following disclosure sets forth several embodiments of different aspects of the technology, several other embodiments can have different configurations or different components than those described in this section. Accordingly, the technology may have other embodiments with additional elements or without several of the elements described below with reference to
Many embodiments of the technology described below may take the form of computer- or controller-executable instructions, including routines executed by a programmable computer or controller. Those skilled in the relevant art will appreciate that the technology can be practiced on computer or controller systems other than those shown and described below. The technology can be embodied in a special-purpose computer or data processor that is specifically programmed, configured or constructed to perform one or more of the computer-executable instructions described below. Accordingly, the terms “computer” and “controller” as generally used herein refer to any suitable data processor and can include Internet appliances and handheld devices (including palm-top computers, wearable computers, cellular or mobile phones, multi-processor systems, processor-based or programmable consumer electronics, network computers, mini computers, a programmed computer chip, and the like). Information handled by these computers and controllers can be presented at any suitable display medium, including a CRT display or an LCD. Instructions for performing computer- or controller-executable tasks can be stored in or on any suitable computer-readable medium, including hardware, firmware or a combination of hardware and firmware. Instructions can be contained in any suitable memory device, including, for example, a flash drive, USB device, or other suitable medium. In particular embodiments, the term “component” can be hardware, firmware, or a set of instructions stored in a computer-readable medium.
As shown in
As illustrated in
The flight path generation component 105 is configured to generate a 3D path at least partially based on one or more (virtual or physical) locations provided by the operator 12 or suggested by the system 100a, In some embodiments, the locations can be provided as virtual reality locations in the virtual reality environment (e.g., the operator 12 can identify these virtual locations via the virtual reality component 103). In some embodiments, the locations can be provided as real-world locations (e.g., in a form of real-world coordinates) in the real-world environment. In such embodiments, the provided real-world locations can be transformed into corresponding virtual locations by the virtual reality component 103. For example, the system 100a can first determine the relationship between the coordinate systems used in the real-world environment and the virtual reality environment. Once the relationship is determined, the system 100a can then transform the provided real-world locations into the corresponding virtual locations (or vice versa, in other embodiments).
The flight path generation component 105 is also configured to generate the 3D path at least partially based on one or more action items corresponding to the provided/suggested locations. In particular embodiments, the action item includes performing a pre-determined task at a particular location. In some embodiments, for example, the action item can involve UAV movements, such as directing a UAV to fly around a target, or instructing a UAV to rotate about an axis. In some embodiments, the action item can involve an action performed by a component of a UAV. In such embodiments, for example, the action item can include: aligning an image component of a UAV with a target; positioning an image component of a UAV at a horizontal level; maintaining a view angle of an image component of a UAV; aiming an image component of a UAV toward a target; collecting an image associated with a target via an image component of a UAV; collecting a set of information by a sensor of a UAV; and/or instructing a communication component of a UAV to transmit a set of information to a remote device (e.g., a smartphone under the control of the operator 12). The information can include UAV information measured by a UAV sensor or images collected by a UAV image component.
When generating a 3D flight path, the flight path generation component 105 also considers the objects, targets, or obstacles in the virtual reality environment. In particular embodiments, the objects, targets, or obstacles in the virtual reality environment can be identified as a set of 3D information (e.g., in formats such as coordinates, lines, shapes, etc.). The flight path generation component 105 can generate a 3D flight path based on one or more pre-determined rules. In some embodiments, these rules can include rules of physics, such as that the 3D flight path cannot pass through a tangible object in the virtual reality environment, or through the ground of the virtual reality environment. In some embodiments, the rules can relate to the maneuverability of a UAV, such as the minimum turn radius of a UAV, the maximum/minimum speed of a UAV, and/or the maximum/minimum acceleration of a UAV.
After a 3D flight path is generated, the flight path analysis component 107 can then analyze the generated 3D flight path and perform a simulation in which a UAV flies along the generated 3D flight path in the virtual reality environment. In particular embodiments, the simulation includes generating a set of virtual reality images that the UAV can collect at each provided locations along the 3D flight path in the virtual reality environment. The flight path analysis component 107 then visually presents the set of images to the operator 12. In particular embodiments, the set of images can be visually presented to the operator 12 via the virtual reality component 103. By doing so, the system 100a enables the operator 12 to visually experience the 3D flight path from a first person perspective. By doing so, the operator 12 can have a clear and intuitive sense or understanding of how a UAV would travel in the real world environment. Meanwhile, the operator 12 can review and verify whether a UAV can perform an action item as he/she desires (e.g., filming a target from a particular view angle).
In the illustrated embodiments, the flight path verification component 109 is configured to further verify a generated 3D flight path so as to make sure that the 3D flight path meets certain pre-determined requirements. The requirements can be set by the operator 12 (e.g., based on the operator's preferences or level of skill when operating a UAV, and/or from a third party entity (e.g., a government regulation prohibiting a UAV from flying in a certain area). By verifying the generated 3D flight path, the system 100a can provide a safe and practicable 3D flight path to the operator 12.
The system 100a can also enable the operator 12 to adjust the generated 3D flight path. In particular embodiments, the operator 12 can add/cancel additional/existing locations to the generated 3D flight path or adjust the curvature of the generated 3D flight path, via the virtual reality component 103. In some embodiments, the operator 12 can adjust the 3D flight path manually (e.g., via a virtual reality device or an input device). In some embodiments, the operator 12 can adjust the 3D flight path in an automatic manner (e.g., based on a user preference that the system has learned from the operator's prior adjustments to generated 3D flight paths during prior tasks/projects). By so doing, the system 100a enables the operator 12 to precisely control the UAV 11 to complete desirable tasks.
As shown in
The airframe 210 can carry a payload 204, for example, an imaging device. In particular embodiments, the imaging device can include an image camera (e.g., a camera that is configured to capture video data, still data, or both). The image camera can be sensitive to wavelengths in any of a variety of suitable wavelength bands, including visual, ultraviolet, infrared or combinations thereof. In still further embodiments, the payload 204 can include other types of sensors, other types of cargo (e.g., packages or other deliverables), or both. In many of these embodiments, the payload 204 is supported relative to the airframe 210 with a gimbal 202 that allows the payload to be independently positioned relative to the airframe 210. Accordingly, for example when the payload 204 includes the imaging device, the imaging device can be moved relative to the airframe 210 to track a target. More particularly, for example, the imaging device can be rotated by an angle relative to the airframe 210 (or relative to another reference plane such as a horizontal plane). When the UAV 20 is not in flight, a landing gear can support the UAV 20 in a position that protects the payload 204.
In a representative embodiment, the UAV 20 includes a controller 208 carried by the UAV 20. The controller 208 can include an on-board computer-readable medium 203 that executes instructions directing the actions of the UAV 20, including, but not limited to, operation of the propulsion system and the imaging device. The on-board computer-readable medium 203 can be removable from the UAV 20.
At block 507, the system generates a 3D path based on the set of 3D information, the plurality of virtual locations, and the plurality of action items. The method 500 then continues at block 509 to generate a set of images associated with the 3D path based on the set of 3D information, the plurality of virtual locations, and the plurality of action items. In some embodiments, the set of images can be generated by a virtual reality system. At block 511, the system visually presents the set of images to an operator. In particular embodiments, the set of images is visually presented to the operator via a virtual reality device. The method 500 then returns to await for further instructions. In some embodiments, the system can further adjust the 3D path upon receiving an instruction from the operator via a virtual reality device.
From the foregoing, it will be appreciated that specific embodiments of the technology have been described herein for purposes of illustration, but that various modifications may be made without deviating from the technology. For example, particular embodiments were described above in the context of a UAV. In other embodiments, the present technology can be implemented by other suitable moveable devices, such as an unmanned ground vehicle (UGV), an unmanned surface vehicle (USV), or a robot.
Further, while advantages associated with certain embodiments of the technology have been described in the context of those embodiments, other embodiments may also exhibit such advantages, and not all embodiments need necessarily exhibit such advantages to fall with within the scope of the present technology. Accordingly, the present disclosure and associated technology can encompass other embodiments not expressly shown or described herein.
At least a portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.
The present application is a continuation of U.S. patent application Ser. No. 16/425,689, filed May 29, 2019 (now allowed), which is a continuation of International Patent Application No. PCT/CN16/108300, filed Dec. 1, 2016. The foregoing applications are incorporated herein by reference in their entireties.
| Number | Name | Date | Kind |
|---|---|---|---|
| 10191486 | Baer | Jan 2019 | B2 |
| 20050004723 | Duggan | Jan 2005 | A1 |
| 20090010495 | Schamp | Jan 2009 | A1 |
| 20090087029 | Coleman et al. | Apr 2009 | A1 |
| 20100073363 | Densham | Mar 2010 | A1 |
| 20100250022 | Hines et al. | Sep 2010 | A1 |
| 20100292868 | Rotem | Nov 2010 | A1 |
| 20110320068 | Lee | Dec 2011 | A1 |
| 20120256945 | Kidron | Oct 2012 | A1 |
| 20120280087 | Coffman et al. | Nov 2012 | A1 |
| 20150016667 | Algreatly | Jan 2015 | A1 |
| 20150341540 | Kim | Nov 2015 | A1 |
| 20160023602 | Krishnan | Jan 2016 | A1 |
| 20160271796 | Babu | Sep 2016 | A1 |
| 20160292869 | Hammond et al. | Oct 2016 | A1 |
| 20160307447 | Johnson et al. | Oct 2016 | A1 |
| 20170036771 | Woodman | Feb 2017 | A1 |
| 20170039764 | Hu | Feb 2017 | A1 |
| 20170080349 | Reveley | Mar 2017 | A1 |
| 20170251179 | Smolyanskiy | Aug 2017 | A1 |
| 20180025649 | Contreras | Jan 2018 | A1 |
| 20180157255 | Halverson | Jun 2018 | A1 |
| 20190253611 | Wang | Aug 2019 | A1 |
| 20190253626 | Li | Aug 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 104635579 | May 2015 | CN |
| 104702871 | Jun 2015 | CN |
| 204669545 | Sep 2015 | CN |
| 105739525 | Jul 2016 | CN |
| 105828062 | Aug 2016 | CN |
| 20160099917 | Aug 2016 | KR |
| Entry |
|---|
| K. Hsu, “Application of the Environmental Sensation Learning Vehicle Simulation Platform in Virtual Reality,” 2016 (Year: 2016). |
| G. Silano and L. Iannelli, “An educational simulation platform for GPS-denied unmanned Aerial Vehicles aimed to the detection and tracking of moving objects,” 2016 IEEE Conference on Control Applications (CCA), Buenos Aires, Argentina, 2016, pp. 1018-1023, doi: 10.1109/CCA.2016.7587947. (Year: 2016). |
| PCT International Search Report dated Aug. 29, 2017 for corresponding PCT Application No. PCT/CN2016/018300 (4 Pages). |
| Number | Date | Country | |
|---|---|---|---|
| 20220198942 A1 | Jun 2022 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16425689 | May 2019 | US |
| Child | 17653888 | US | |
| Parent | PCT/CN2016/108300 | Dec 2016 | US |
| Child | 16425689 | US |
