Patent Application
20220212792
This application generally refers to camera systems and networks of cameras systems. More specifically, the application relates to camera systems that can be used to supervise and control a drone or other device.
The U.S. Federal Aviation Administration (FAA) has very stringent requirements for drone operations. These requirements generally require a Certificate of Airworthiness (COA) or an exemption in order to operate drones that operate beyond the line of sight of the operator. Additionally, the systems and methods that are used to obtain a COA or an exemption are time consuming and often require the signature authority of multiple individuals within a management hierarchy. Accordingly, most drone operations are restricted to line of sight operations. In other words, they must be done in such a manner that requires the pilot or suitable surrogate to maintain visual contact with the drone throughout the entire flight.
Line of sight limitations can present a number of issues in the ever-expanding use of drones. For example, some companies are looking to utilize drones for last mile delivery. Last mile deliver typically refers to the delivery of packages to the final destination. The final destination can be anywhere from a few hundred yards from the point of origin to several miles. Some of these limitations are related to the range of the drone. Since unassisted human visual acuity quickly degrades beyond a few hundred yards, visual line of sight becomes difficult to achieve. Accordingly, the FAA is reluctant to grant COA's and/or exemptions to operators even when other requirements under 14 CFR part 107 are met.
Systems and methods for supervising and controlling a drone including:
Many embodiments are directed to a mesh network for controlling drones where the network is made up of a plurality of cameras making up a plurality of nodes within a specific geographical region. Each of the plurality of nodes has at least one of the plurality of cameras in a fixed position within the geographical region. Each of the plurality of nodes are configured to monitor a portion of the geographical region such that the plurality of nodes are capable of capturing image data from the entire geographical region. Additionally, the network of cameras are configured to control at least one drone with a transponder unit, where the transponder unit can transmit drone data to any of the plurality of cameras. Each of the plurality of cameras is configured to receive the drone data and combine the drone data with a visual image of the drone within the geographical region to determine a correct flight path for the drone within the network of nodes; and wherein each of the plurality of cameras is configured to transmit a new set of flight control data to the drone such that the drone can alter course as needed based on the new set of flight control data, where the latency between the drone and any of the plurality of cameras is lower than the human latency times.
In other embodiments, each of the plurality of cameras is a 5G enabled camera.
In still other embodiments, each of the nodes contains at least one camera.
In yet other embodiments, each of the plurality of nodes contains more than one camera.
In still yet other embodiments, at least one of the more than one cameras is an infrared camera.
In other embodiments, the system has a supervisory control system wherein the drone data is transmitted from the network of nodes to the supervisory control system for monitor.
In still other embodiments, the supervisory control system is a human based system.
In yet other embodiments, the drone is a VTOL drone, a fixed wing drone, and/or a hybrid drone between fixed and rotary wing.
Other embodiments are directed to a method for controlling moveable assets within an operational environment that include the following steps:
In other embodiments, the specific environment is an urban environment.
In still other embodiments, the continuous visual image of the drone is maintained by overlapping areas of interest between each of the cameras within the network of cameras.
In yet other embodiments, adjusting the flight path for the drone includes altering the flight path to avoid an obstruction selected from a group consisting of weather, building, construction, emergencies, and traffic.
In still yet other embodiments, the systems and methods include more than one drone.
In other embodiments, the drone(s) have a transponder for communication with and between the network of cameras.
Additional embodiments and features are set forth in part in the description that follows, and in part will become apparent to those skilled in the art upon examination of the specification or may be learned by the practice of the disclosure. A further understanding of the nature and advantages of the present disclosure may be realized by reference to the remaining portions of the specification and the drawings, which forms a part of this disclosure.
The description will be more fully understood with reference to the following figures, which are presented as exemplary embodiments of the invention and should not be construed as a complete recitation of the scope of the invention, wherein:
Turning now to the drawings, systems and methods for controlling drones based on a network of wireless cameras is illustrated. Many embodiments are directed to a network of cameras positioned at various locations within a desired operational environment. Each of the cameras are positioned at desired node locations and are configured to communicate with one or more drones within the operational environment. Each of the cameras are configured to obtain visual data of the drone and the drone's flight path. The cameras visual data can be used to confirm and/or improve the drone's location tracking. This can be especially helpful in areas where GPS positioning can be unreliable, such as urban environments. Accordingly, the network of cameras can help provide a true position of the drone in all environments. Additionally, each of the cameras within the operational environment are in communication with a transponder on each of the drones. The transponder transmits drone flight data to the cameras by which the cameras can then provide updated drone control information to the drones to ensure the drone(s) operate safely within the operational environment.
Urban cargo delivery systems are often referred to as “last mile” delivery systems. As previously discussed, such systems can operate anywhere from a few hundred yards to several miles to deliver goods and/or services to a desired location or customer. More and more companies are considering the use of drones to operate within the “last mile” delivery system due to their improved capabilities and distance. However, as previously discussed, the current governmental system that regulates the use of drones presents various obstacles by which potential users cannot operate efficiently. For example, the current 14 CFR part 107 requires that drone operators maintain a line of sight with the drone in order to safely operate and control the drone. This requirement can greatly reduce the operational area of a drone for “last mile” delivery systems. This is even with a drone that meets the requirements for air worthiness under part 107 and is configured to operate under the 400 foot above-ground-level (AGL) altitude.
The line of sight requirement combined with the lengthy and arduous process of obtaining permits to operate drones, can present various obstacles for the future of logistics chains and support. In response, the present disclosure proposes a system and method for drone control that maintains line of sight with a drone by way of a network of high frame rate, high definition, low latency and high transmission rate capable cameras. The efficient cameras work in conjunction with additional sensors positioned on or within the drones to continuously monitor the drone during flight. The continuous monitoring by way of the cameras and other sensors can allow the system to continuously maintain a visual line of sight with the drone and adjust the drone's functions as necessary to maintain safe and effective flight operations. The high definition network of cameras can enable higher performance and faster response time than a human operator. Additionally, human operators can serve a supervisory roll in monitoring the camera feed and drone data from a remote location and adjust as needed. However, many embodiments of the system are configured to operate with little feedback from the human due to the increased response time that humans typically have.
In accordance with many embodiments, drones can be operated with an artificial intelligence pilot that is enabled by a network of high-resolution cameras. For example,
As can be appreciated, the system 100 can be augmented by a remote supervisor 110. The remote supervisor can be a human operator that views the data transmitted 112 from the network of cameras 104 through a wireless transmission tower or system 114. The wireless transmission tower 114 can be a single tower or a network of towers that can communicate with a controller 110. The controller 116 can be a number of different configurations such as a human supervisor or operator that can send and receive signals to and from the network of cameras 104 and drones 102.
A more practical application illustration of an operational environment can be further illustrated in
In accordance with various embodiments, a drone 216 can have a flight path (218, 220) that is designated to travel from a location “A” in zone 1 and end at location “B” in zone 2. The drone can be provided with one or more flight paths (218, 220) from which it can operate. Additionally, in some embodiments the drone 216, in coordination with the network of cameras can adjust the flight path based on changing conditions such as weather, construction, traffic, emergencies such as fires in the flight path etc.
As can be appreciated, the network of cameras can communication with each other (also illustrated in
In accordance with many embodiments, the transponder communication between the drones and the cameras can be continuous such that any adjustments to the drone flight can be altered as needed. For example, some systems can be programmed to monitor various fault codes and/or data from the transponders and/or the cameras. Such codes and data could include one or more rotor failures, dramatic reduction in battery power, drift beyond predefined flight path, abnormal oscillations in the drone, rotor speed and temperature, and/or a unique ID number for the drone. The unique ID number can be similar to that of a tail number on a traditional aircraft that allows for that particular drone or moving asset to be identified as authorized to operate within the network of cameras. Additionally, drones can be configured with additional sensors that help monitor weather and the surrounding environment to notify the camera/control system when things have changed. These can include additional cameras that can work in conjunction with the transponder and the network of cameras to identify obstacles and navigate the operational environment. This can be highly beneficial because a high density of cameras or nodes can serve to help reroute the drone to avoid unforeseen problems. Furthermore, a dense network of nodes can be used to redirect drones in the event of a cancellation of a cargo delivery order. For example, in some embodiments the drone can be redirected between mesh networks and can be directed to a new supply depot and/or new delivery location.
The mesh network of nodes, illustrated in many embodiments, can represent what the FAA refers to as a “dedicated airspace”, which creates a type of local host model for flight operations. As can be appreciated, this type of model can be applied in any number of situations and in any number of locations such that FAA regulations can be met and still maintain a secure airspace. The secure airspace can be managed by the cameras and their ability to quickly identify the movement of any assets within the area. For example, much like traditional aircraft have identifying information that is transmitted to air traffic controllers, the network of cameras can be configured to receive similar transponder data from any moving asset in the area. If the moving asset is not identified as one that is authorized within the area, the cameras can be used to identify and control the unknown object and prevent undesired safety incidents. Likewise, the mesh network of nodes also addresses potential cyber security concerns that come with connections to the cloud by having a closed network for drone flight operations. By addressing potential security issues and creating a defined geographically dedicated airspace for drone operations can allow for FAA exemption approvals where they normally would not, such as night operations.
As briefly described above, the network of cameras can act as an artificial intelligence (AI) control system to help control moving assets within an operational area. In accordance with many embodiments, the AI system of cameras and/or node can act to control assets by combining camera image data, generated from maintaining a continual visual image of the moving assets and/or operational environment, with transponder data from the moving asset. The combined data can be used to identify what asset is moving in the area such as a drone or otherwise as well as identify any obstacles that could negatively affect the movement of the assets within the operational environment. This can serve as an AI control system for the moving assets because it can constantly be transmitting and receiving information that can be used to control the movement of the assets within the operational environment. An AI control system can be far faster and more efficient than humans. For example, a reasonable estimate for human response time in drone control is roughly 200 ms. This includes the time for the image to travel to the brain and the brain to process the image into action. In contrast an AI type control system, such as described in the embodiments herein, can combine high-definition camera image data with drone transponder data (relatively low in terms of data transfer times and size) within the 10 ms range or faster. This can result in a much faster response time to control drones within the network. Accordingly, the AI control of moveable assets within a network of cameras can offer improved systems for deliverables that would otherwise be unattainable with human control. In accordance with many embodiments, each camera can be configured with an internal processing system that can act as an internal AI. This internal system can help to improve or reduce latency of the data being transmitted between the drone and the camera(s). Some embodiments may have external computers or processors that serve as an additional AI unit to augment other computers or processors. The external computer can be located in local 5G towers such that they can operate to cover one or more cameras covering a particular area.
In accordance with many embodiments, the drone(s) can operate autonomously or semi-autonomously but still be machine supervised by the use of the network of cameras. The supervision of the drone movement can be handed off from one node of cameras to the next node to maintain constant visual contact with the drone. Additionally, the transponder data can be transmitted to one or more nodes within range to ensure a constant connection and analysis of the drone state within the network. Consequently, the network of cameras can identify and analyze the transponder data to direct and redirect the drone within the network. Additionally, with the low latency 5G network connection that numerous embodiments can have, it can be appreciated that the AI control system can operate continuously without the need for rest. Cameras can switch between operational modes and the nodes can have redundant cameras for continuous operation. This can be highly beneficial in aiding and maintaining supply chain networks that currently rely on human intervention. Systems described herein can operate beyond the capabilities of humans, thus allowing for better coverage in the supply chain as well as reduced risk.
In accordance with many embodiments, the drones used within the system can be any type of suitable drone for flight. For example, the drone can have a number of rotors and can be configured for Vertical Take Off and Landing (VTOL). Other drones can be fixed wing drones. Still other drones can be a hybrid between fixed and rotary wing drone. It should be well understood that many embodiments of the drones will be configured to meet FAA regulations for flight worthiness as well as be capable of communication with any number of systems for operational control. The drones, in accordance with numerous embodiments can be configured to house one or more transponders. The transponders, as previously described can be used to transmit drone vehicle data to the network of cameras which can then utilize the transponder data in combination with camera image data to direct or control the flight of the drone. The transponder can be any type of transponder that allows the drone to communicate continuously with one or more of the networked cameras.
The cameras that can be used in accordance with various embodiments should be high resolution cameras such that they are capable of producing high quality images similar to the human eye. Given the large amount of data can be generated through high resolution images, it can be appreciated that many embodiments of cameras are configured to be high bandwidth capable as well as have the ability to rapidly transmit data with little latency. As such, some embodiments of the cameras can be enabled with 5G capabilities. 5G wireless networks operate by sending signals directly between towers in sequence rather than bouncing signals to and from a remote hub, such as a geosynchronous satellite. This means speed of signal travel is much higher than older wireless systems and can match hardline systems like fiber networks. The 5G and any future generations of wireless network technology would be preferred for the system because of the speed at which such technology can transfer data. This presents the added benefit and augmentation to the system of cameras because of the large amount of data that high resolution images can generate and a 5G enabled network of cameras can offer low latency transfer times that offer a faster and more improved response time over human controlled devices A 5G network and beyond can be capable of transmitting such data rapidly between cameras and/or moving assets to allow for real time control of a moving asset. The 5G wireless network is only expected to improve over time, which would only improve the capabilities of the various embodiments described herein.
As can be appreciated, any number of cameras can be used within the system that are high resolution and configured with 5G or higher capabilities. Some embodiments, the cameras can produce 4K videos at a rate of 1 gigabit per second (Gbs) or higher. Additionally, the frame rate of the cameras can be upwards of 100 frames/second or higher. Some embodiments of cameras can be configured with infrared capabilities. Other embodiments can include cameras with additional sensors such as LED's or Spectral imaging capabilities. It should be understood that cameras can also be updated with improved imaging technology to allow for improved data capture for overall operational control. Accordingly, various embodiments of the system described above can utilize one or more types of cameras at the various nodes to produce multiple image types of the drones to be combined with the transponder data of the drone to control the movement of the drone within the network in a number of different flight conditions.
As can be appreciated, many embodiments of the system can be configured to use any number and type of cameras and/or drones and transponders such that the overall control of the moveable asset is continually maintained.
In some embodiments, the camera system 300 could be configured with a processor 312 that functions much like a powerful computer that can help to increase the range and capabilities of camera for processing image and transponder data. With the increasing prevalence of smaller processing systems seen in phones and cameras, it is reasonable to see how many embodiments of the camera 300 could function similar to that of a small laptop or cellular phone. This improved processing power combined with 5G and beyond capabilities can allow the cameras to be extremely efficient at processing data. Furthermore, the low latency 5G connection can also allow for the camera to be connected to a remote server that is much larger and capable of storing and managing larger amounts of data that can be used for future operations such that the system overall can continually be learning from each subsequent operation.
Although the term drone is used throughout, it should be appreciated that drones, in accordance with many embodiments, can vary in terms of their capabilities and functions. Essentially, many embodiments may define the drone to be a moving asset which could be any number of moving objects within the operational environment. For example, some embodiments may have unmanned aerial vehicles such as copters (tri, quad, etc.) fixed winged aircraft, hybrid aircraft. Other embodiments of drones may be wheeled vehicles that may be manned or unmanned. In manned embodiments, the network can be configured to communicate directly with the vehicle as described above, while offering a human interaction as a redundant control system if necessary. Accordingly, it should be understood that the term “drone” or “drones” can take on any reasonable meaning in terms of movable assets within the operational environment or ones that might come into the operational environment.
Referring now to
Likewise,
This description of the invention has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form described, and many modifications and variations are possible in light of the teaching above. The embodiments were chosen and described in order to best explain the principles of the invention and its practical applications. This description will enable others skilled in the art to best utilize and practice the invention in various embodiments and with various modifications as are suited to a particular use. The scope of the invention is defined by the following claims.
This application claims priority to U.S. Provisional Patent Application No. 63/134,905 filed on Jan. 7, 2021, the disclosure of which is herein incorporated by reference in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63134905 | Jan 2021 | US |
