Patent Grant
11385341
This application generally relates to unmanned aerial vehicles and more particularly to detecting and passively monitoring communications of an unmanned aerial vehicle.
Recent technological advancements and consumer demand have allowed for widespread, low-cost availability of unmanned aerial vehicles, such as quadcopters or other drones. Such technological advancements have also resulted in increased communication range, longer flight times, and heavier payload capacities. Gains have also be made in autonomous flight controls, on-board sensors, and video capture equipment. Yet these improvements continue to raise concerns that unmanned aerial vehicles, in particular consumer-grade drones, may be used for nefarious or harmful purposes. For example, a drone may enter restricted airspace at an airport and cause a plane to crash. Or a drone may be laden with explosives and used in a direct attack on a facility or group of people. Drones may also be used for corporate or industrial espionage, such as capturing images of classified vehicle prototypes at a car manufacturer's test facility. As yet another example, a drone operating near an outside concert venue may illegally record a musical performance. Due to the low cost and wide availability of consumer-grade drones, however, direct measures against an intruding drone, such as shooting down, are often insufficient deterrents.
Thus, what is desired in the art is a technique and architecture for detecting and passively monitoring communications of an unmanned aerial vehicle.
The foregoing needs are met, to a great extent, by the disclosed systems, methods, and techniques for detecting and passively monitoring communication of an unmanned aerial vehicle.
One aspect of the patent application is directed to generating an alert based on a characteristic of sensor data carried in a modulated and coded radio signal from a drone. In an example method, a radio frequency spectrum is scanned to detect one or more radio signals transmitted within a pre-defined area. A modulated radio signal of interest from the one or more radio signals is determined. The radio signal of interest is associated with a drone. The radio signal of interest is captured. The radio signal of interest is demodulated to determine coded sensor data carried by the radio signal of interest. The sensor data is determined by the drone. The coded sensor data is decoded to determine a characteristic of the sensor data. An alert is generated based on the characteristic of the sensor data.
Another aspect of the patent application is directed to determining a location associated with a drone based on video data carried in a modulated and coded radio signal from the drone. In an example method, a radio frequency spectrum is scanned to detect one or more radio signals transmitted within a pre-defined area. A modulated radio signal of interest from the one or more radio signals is determined. The radio signal of interest is associated with a drone. The radio signal of interest is captured. The radio signal of interest is demodulated to determine coded video data carried by the radio signal of interest. The video data is determined by the drone. The coded video data is decoded. A location associated with the drone is determined based on the decoded video data.
Yet another aspect of the patent application is directed to configuring a scanning receiver to detect radio signals from a drone located in a first reception range and reconfiguring the scanning receiver to also capture radio signals from a controller unit located in a second reception range. In an example method, a scanning receiver is caused to execute a first scan of a radio frequency spectrum to detect one or more radio signals transmitted within a first reception range of the scanning receiver. A modulated first radio signal of interest from the one or more radio signals transmitted within the first reception range is determined. The first radio signal of interest is transmitted by a drone. Based on the first radio signal of interest, the scanning receiver is caused to execute a second scan of the radio frequency spectrum to detect one or more radio signals transmitted within a second reception range of the scanning receiver. A modulated second radio signal of interest from the one or more radio signals transmitted within the second reception range is determined. The second radio signal of interest is transmitted by a controller unit associated with the drone. The first and second radio signals of interest are demodulated to determine first coded data carried by the first radio signal of interest and second coded data carried by the second radio signal of interest. The first and second coded data are decoded. An alert is generated based on at least one of the first coded data and the second coded data.
There has thus been outlined, rather broadly, certain embodiments of the application in order that the detailed description thereof herein may be better understood, and in order that the present contribution to the art may be better appreciated. There are, of course, additional embodiments of the application that will be described below and which will form the subject matter of the claims appended hereto.
To facilitate a fuller understanding of the application, reference is made to the accompanying drawings, in which like elements are referenced with like numerals. These drawings should not be construed to limit the application and are intended only for illustrative purposes.
Before explaining at least one embodiment of the application in detail, it is to be understood that the application is not limited in its application to the details of construction and to the arrangements of the components set forth in the following description or illustrated in the drawings. The application is capable of embodiments in addition to those described and of being practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein, as well as the abstract, are for the purpose of description and should not be regarded as limiting.
Reference in this application to “one embodiment,” “an embodiment,” “one or more embodiments,” or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure. The appearances of, for example, the phrases “an embodiment” in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Moreover, various features are described which may be exhibited by some embodiments and not by the other. Similarly, various requirements are described which may be requirements for some embodiments but not for other embodiments.
The area 110 may include any area that is susceptible to unwanted drone intrusion or surveillance. The border 112 defining the area 110 may form a secured perimeter around the area 110. As an example, the facility 114 may include a government building subject to potential terrorist attacks or interference. As another example, the area 110 may include a remote research, development, and test area for weapons or other technology. Such research, development, and test area may be associated with a government or with a private institution, such as the aforementioned automotive test track. The facility 114 may include an industrial facility associated with confidential materials, products, equipment, etc. Such a facility 114 may include a chemical processing plant, for example. The facility 114 may include an outdoor sports stadium or concert venue. The facility 114 may also include an office building. The facility 114 may comprise a military base or installation, as another example.
An unmanned aerial vehicle shall be referred to herein as a drone and includes any of various types of unmanned aerial vehicles, including miniature and micro unmanned aerial vehicles. For example, the drone 120 may comprise a rotary wing unmanned aerial vehicle, including a multirotor unmanned aerial vehicle. A multirotor unmanned aerial vehicle may comprise a tricopter, quadcopter, hexacopter, etc. Additionally or alternatively, the drone 120 may be configured with a fixed wing design. The drone 120 may operate inflight according to various proportions of autonomous and operator controls. For example, the drone 120 may operate inflight using only autonomous control and without any operator control. As another example, the drone 120 may operate according to both autonomous and operator control. Operator controls may include inputs for gross movement of the drone 120, video or image capture, and sensor data capture. Autonomous controls may include stability controls, collision avoidance controls, position/altitude hold controls, waypoint navigation controls, and return-to-home controls.
The drone 120 may be configured with one or more sensors to capture data while the drone 120 is inflight, particularly while in or near the area 110. For example, the drone 120 may be configured with a camera to capture video and/or image data. The drone 120 may be configured with a microphone to capture sound data. The drone 120 may be configured with a chemical sensor or similar to detect and measure various gases or air-born particulate. The drone 120 may be configured with a sensor to capture data (e.g., an emission spectrum) for use in spectroscopic analysis to identify a target substance or material.
As noted, the system 100 comprises one or more scanning receivers 102. Generally, a scanning receiver 102 scans the frequency spectrum to detect and capture any radio signals that may be associated with an unauthorized drone (e.g., the drone 120). A captured radio signal may be sent to the control system 106 and/or the control terminal 104 for analysis. If the radio signal is determined to be associated with the drone 120, the scanning receiver 102 may continue to capture the radio signal and send the captured radio signal to the control system 106 and/or the control terminal 104 for it to be decoded and further analyzed.
More particularly, a scanning receiver 102 comprises one or more antennas and a receiver unit. An antenna may be omni-directional or directional (e.g. a sector antenna). An omni-directional antenna may be appropriate in a system 100 configuration with a single scanning receiver 102 while a directional antenna may be appropriate in a system 100 configuration with multiple scanning receivers 102. The one or more antennas of a scanning receiver 102 may comprise one or more of a 443 MHz antenna, a 900 MHz antenna, a 1.2 GHz antenna, a 2.4 GHz antenna, and a 5.8 GHz antenna.
A scanning receiver 102, such as the receiver unit of the scanning receiver 102, may comprise a software-defined radio (SDR). The SDR may comprise one or more receiver channels. For example, the SDR may comprise one channel for scanning for possible radio signals associated with a drone and one or more other channels for capturing radio signals already identified as being associated with a drone. In this manner, one or more identified drones and transmissions thereof may be tracked while also scanning for other as-yet unidentified drones. The SDR may be controlled by the scanning receiver 102, the control system 106, the control terminal 104, or a combination thereof. A scanning receiver 102 may operate according to one or more modulation schemes or techniques, including CDMA, OFDM, WBFM, GFSK, PWM, PCM, and PPM. A scanning receiver 102 may operate according to one or more spread spectrum techniques, including DSSS and FHSS.
The one or more scanning receivers 102 may be positioned within the area 110 to maximize radio signal detection within the area 110, particularly with respect to the facility 114. In the system 100 illustrated in
The control system 106 and the control terminal 104 generally implement the back-end portions of the system 100. The control system 106 and the control terminal 104 may overlap, to varying degrees, in the functions performed by each. For ease of description, however, signal analysis and processing and communications with the scanning receiver 102 shall be generally described with reference to the control system 106. Operator interactions with the system 100 shall be generally described with reference to the control terminal 104. In neither case is the application so limited. Although not illustrated in
In some embodiments, the control system 106 and the control terminal 104 are separate systems. For example, the control system 106 may comprise one or more networked servers and a storage system. In this case, the control terminal 104 may comprise a separate computing device, such as a personal computer. The control terminal 104 may be configured for direct use by an operator. The control terminal 104 may comprise a display and one or more user inputs devices (e.g., a mouse or other pointing device and a keyboard). The system 100 may comprise multiple control terminals 104. A control terminal 104 may be located onsite, as in the case of the control terminal 104a. Additionally or alternatively, a control terminal 104 may be located offsite, as in the case of the control terminal 104b, and connected to the control system 106 via a network 116. In other embodiments, the control system 106 and the control terminal 104 are integrated as a single system or computing device, which may be located onsite or offsite. A control terminal 104 may provide a graphical user interface (GUI) 105 for system interaction by a system operator. The GUI 105 may display various data described herein to the system operator. The system operator may provide various inputs described herein to the GUI 105.
The control system 106 is configured to receive radio spectrum data captured by a scanning receiver 102 while performing radio frequency scanning. The radio spectrum data from the scanning receiver 102 may be sent as a sequence of frequency and amplitude (FA) packets. The radio spectrum data may be displayed to an operator via the GUI 105 of the control terminal 104. The control system 106 determines if a radio signal represented in the radio spectrum data is potentially associated with the unauthorized drone 120 or if a radio signal is otherwise of interest. For example, the captured radio spectrum data may be compared to a record of radio spectrum data (e.g., radio signals) that are known to be associated with drones. The control system 106 may determine the frequency or frequency range of the radio signal of interest. If the radio signal is of interest, the control system 106 sends a control message to the scanning receiver 102 to cause the scanning receiver 102 to dedicate an open channel for further capture and identification of the radio signal of interest. The control message may indicate an attenuation, center frequency, bandwidth, and/or bandwidth resolution for the open channel.
An alert may be generated if the radio signal is of interest, such as an alert presented on the control terminal 104 or sent to a designated email address or phone number. The alert and the frequency of the radio signal of interest may be recorded in a log. The radio spectrum data and/or the FA packets may be also recorded based on operator input to the control terminal 104.
In an embodiment in which the system 100 comprises two or more scanning receivers 102, such as is shown in
The control system 106 is further configured to receive radio signal data from the scanning receiver 102 captured via the scanning receiver's 102 dedicated open channel. The radio signal data may comprise the radio signal of interest associated with the drone 120. The radio signal data may comprise digital in-phase and quadrature (I&Q) data associated with the radio signal of interest. The control system 106 demodulates the radio signal data. Demodulating the radio signal data may comprise determining the modulation scheme used to initially modulate the radio signal data. Demodulating the radio signal data may also be accomplished by determining a radio signature associated with the radio signal data and comparing that to a set of radio signatures with known associations to available makes and models of drones and/or drone controller units or components thereof (e.g., a transmitter). Results of the demodulation may be displayed on the control terminal 104. For example, the demodulated radio signal from the radio signal data may be displayed on the control terminal 104 via the GUI 105. The digital I&Q data may be recorded automatically by the control system 106.
The control system 106 is further configured to decode the demodulated radio signal data. The control system 106 may determine the decoding algorithm to use in decoding the radio signal data. For example, the control system 106 may select a decoding algorithm from a set of potential decoding algorithms. Selecting the decoding algorithm may be based on the frequency range of the radio signal data and/or the modulation scheme of the radio signal data. Additionally or alternatively, an operator may determine the decoding algorithm via the GUI 105 of the control terminal 104. For example, the operator may select from a pull-down menu of possible drone manufacturers (e.g., those associated with the frequency range of the radio signal data and/or the modulation scheme of the radio signal data) until coherent data (e.g., video or tracking data) is displayed on the control terminal 104. The decoded radio signal data may be recorded by the control system 106, automatically or based on operator input.
The decoded radio signal data may comprise any of various types of data exchanged between the drone 120 and the drone's 120 controller unit 122. For example, the decoded radio signal data may comprise control instructions from the controller unit 122 to the drone 120 that control movement of the drone 120 and/or operation of a sensor of the drone 120. For example, a control instruction may cause a video camera on the drone 120 to capture video data. The decoded radio signal data may comprise control data from the drone 120 indicating that the drone 120 received and executed the control instructions sent from the controller unit 122. As another example, a control instruction may cause a sensor on the drone 120 to take a data reading, such as a data reading indicating the presence of a certain gas or a certain type of air-born particulates.
The decoded radio signal data may comprise telemetry data, such as GPS data indicating a current or past position of the drone 120. The decoded radio signal data may comprise video or image data captured by the drone 120. Video data may comprise a real-time or near real-time video stream from the drone 120. The decoded radio signal data may comprise audio data captured by the drone 120. The audio data may comprise a real-time or near real-time audio stream. The decoded radio signal data may comprise sensor data captured by a sensor on the drone 120. The sensor data may indicate one or more measurements or sensor readings taken by the sensor. The sensor data may also include metadata describing the sensor data, such as the time that the sensor data was captured, the type of sensor that captured the sensor data, or a quality indicator of the captured data (e.g., video resolution, frame rate, or bit rate).
The network device 108 is generally configured to implement data communications between the components of the system 100. For example, a scanning receiver 102 may communicate radio signal data to the control system 106 and/or the control terminal 104 via the network device 108. The network device 108 may also enable data communications with external systems and networks via the network 116. The network device 108 may be considered as part of or an extension of the network 116. As examples, the network device 108 may comprise one or more of a gateway, a router, a switch, or a bridge. A router may include a wireless router (e.g., a Wi-Fi router), a wired router (e.g., an Ethernet router), or a combination thereof.
The network 116 may be a fixed network (e.g., Ethernet, Fiber, ISDN, PLC, or the like) or a wireless network (e.g., WLAN, cellular, or the like) or a network of heterogeneous networks. For example, the network 116 may be comprised of multiple access networks that provide communications, such as voice, data, video, messaging, broadcast, or the like. For example, the network 116 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), and the like. Further, the network 116 may comprise other networks such as a core network, the Internet, a sensor network, an industrial control network, a personal area network, a fused personal network, a satellite network, a home network, or an enterprise network, as some examples.
The processor 32 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. In general, the processor 32 may execute computer-executable instructions stored in the memory (e.g., the memory 44 and/or the memory 46) of the scanning receiver 102 in order to perform the various required functions of the scanning receiver 102. The processor 32 may run radio access-layer (RAN) programs and/or other communications programs. The processor 32 may also perform security operations such as authentication, security key agreement, and/or cryptographic operations, such as at the access-layer and/or application layer for example.
The processor 32 is coupled to its communication circuitry (e.g., the transceiver 34, the transmit/receive element 36, the radio receiver 42, and the communication interface 40). The processor 32, through the execution of computer executable instructions, may control the communication circuitry in order to cause the scanning receiver 102 to communicate with other components of the system, such as the network device 108, the control system 106, and the control terminal 104. The processor 32 may further control the communication circuitry to detect and capture radio spectrum and radio signal data via the transmit/receive element 36 and the radio receiver 42. The radio receiver 42 may comprise a software-defined radio (SDR) receiver. The radio receiver 42 may define one or more channels, such as one or more channels to scan a frequency spectrum for any radio signals associated with a drone and one or more channels to capture identified radio signal data associated with a drone.
The transmit/receive element 36 may be configured to receive a signal sent from a drone to the drone's controller unit or vice versa. For example, in an embodiment, the transmit/receive element 36 may be an antenna configured to transmit and/or receive RF signals. The transmit/receive element 36 may support various networks and air interfaces, such as WLAN, WPAN, cellular, and the like. In an embodiment, the transmit/receive element 36 may be an emitter/detector configured to transmit and/or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit/receive element 36 may be configured to transmit and receive both RF and light signals. It will be appreciated that the transmit/receive element 36 may be configured to transmit and/or receive any combination of wireless or wired signals. The transceiver 34 and/or transmit/receive element 36 may be integrated with, in whole or in part, the communication interface(s) 40, particularly wherein a communication interface 40 comprises a wireless communication interface. The communication interface(s) 40 additionally or alternatively may comprise one or more wired communication interfaces.
The processor 32 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 44 and/or the removable memory 46. For example, the processor 32 may store captured radio signal data (e.g., FA packets and digital I&Q data) in its memory, as described above. The non-removable memory 44 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 46 may include a subscriber identity module (SIM) card, a memory stick, a USB drive, a secure digital (SD) memory card, and the like. In other embodiments, the processor 32 may access information from, and store data in, memory that is not physically located on the scanning receiver 102.
The processor 32 may receive power from the power source 48, and may be configured to distribute and/or control the power to the other components in the scanning receiver 102. The power source 48 may be any suitable device for powering the scanning receiver 102. For example, the power source 48 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like. The power source 48 may be additionally or alternatively configured to receive power from an external power source. The processor 32 may also be coupled to the GPS chipset 50, which is configured to provide location information regarding the location of the scanning receiver 102. It will be appreciated that the scanning receiver 102 may acquire location information by way of any suitable location-determination technique while remaining consistent with an embodiment
In operation, the CPU 91 fetches, decodes, executes instructions, and transfers information to and from other resources via the computer's main data-transfer path, system bus 80. Such a system bus connects the components in the computing system 90 and defines the medium for data exchange. The system bus 80 typically includes data lines for sending data, address lines for sending addresses, and control lines for sending interrupts and for operating the system bus 80. An example of such a system bus 80 may be the PCI (Peripheral Component Interconnect) bus or PCI Express (PCIe) bus.
Memories coupled to the system bus 80 include random access memory (RAM) 82 and read only memory (ROM) 93. Such memories include circuitry that allows information to be stored and retrieved. The ROMs 93 generally contain stored data that cannot easily be modified. Data stored in the RAM 82 may be read or changed by the CPU 91 or other hardware devices. Access to the RAM 82 and/or the ROM 93 may be controlled by the controller 92. The memory controller 92 may provide an address translation function that translates virtual addresses into physical addresses as instructions are executed. The memory controller 92 may also provide a memory protection function that isolates processes within the system and isolates system processes from user processes. Thus, a program running in a first mode may access only memory mapped by its own process virtual address space; it cannot access memory within another process's virtual address space unless memory sharing between the processes has been set up.
In addition, the computing system 90 may comprise a peripherals controller 83 responsible for communicating instructions from the CPU 91 to peripherals, such as a printer 94, a keyboard 84, a mouse 95, and a disk drive 85. A display 86, which is controlled by a display controller 96, is used to display visual output generated by the computing system 90. Such visual output may include text, graphics, animated graphics, and video. Visual output may further comprise a GUI. The display 86 may be implemented with a CRT-based video display, an LCD-based flat-panel display, gas plasma-based flat-panel display, or a touch-panel. The display controller 96 includes electronic components required to generate a video signal that is sent to the display 86.
Further, the computing system 90 may comprise communication circuitry, such as a network adaptor 97, that may be used to connect the computing system 90 to a communications network, such as the network device 108 or the network 116 of
The system 200 further comprises a first scanning receiver 202a and a second scanning receiver 202b, both located within the area 210. The first and second scanning receivers 202a, 202b may be the same or similar to, in at least some aspects, as the scanning receiver 102 of
In the example system 200, both the drone 220 and the controller unit 222 (as well as the controller unit 222 operator) are located within the reception range 230 of the first scanning receiver 202a and the reception range 234 of the second scanning receiver 202b. As such, the first scanning receiver 202a and the second scanning receiver 202b may detect and capture both the radio signals 238a from the controller unit 222 to the drone 220 and the radio signals 238b from the drone 220 to the controller unit 222. The radio signals 238a, 238b may be referred to generally as radio signal(s) 238. The radio signals 238 may be also variously referred to herein as radio transmissions or radio communications. The radio signals 238 may refer to one-way communications between the drone 220 and the controller unit 222 or two-way communications between the drone 220 and the controller unit 222.
The radio signals 238a from the controller unit 222 to the drone 220 may comprise control inputs for the drone 220. The control inputs, for example, may cause movements of the drone 220 or cause the drone 220 to capture sensor data or otherwise control the on-board sensor. The control inputs may cause the drone to execute a pre-defined operation, such as a hover or circling operation.
The radio signals 238b from the drone 220 to the controller unit 222 may comprise sensor data captured by a sensor of the drone 220 within the area 210 or telemetry data. The telemetry data may indicate a current location and/or recent locations of the drone 220. The telemetry data may include one or more coordinates (e.g., GPS coordinates) indicating a current location and/or recent locations of the drone 220. The telemetry data may be derived from the drone's 220 connection to a GPS system.
The sensor data may include a video stream, video data, one or more images, a sound stream, or sound data. The sensor data may include material detection data indicating detection of a particular (e.g., pre-determined or target) gas, chemical, air-born particulate, or other material. The sensor data may include material measurement data indicating a quantitative measure of a detected material, such as a density or concentration of a gas or air-born particulate. The sensor data, e.g., the material detection data, may include data (e.g., an emission spectrum) for use in spectroscopic analysis to identify a target substance or material.
The sensor data may have one or more characteristics. Characteristics of the sensor data may include a quality of video or sound data, such as video/image resolution, video frame rate, or sound bitrate. A characteristic of the sensor data may include the type of data of the sensor data. Types of data include a video stream, video data, an image, a sound stream, sound data, material detection data, or material measurement data. A characteristic of the sensor data may be indicated in metadata associated with the sensor data. The metadata may form part of the sensor data. Other characteristics that may be indicated in the metadata (or otherwise in the sensor data) include the time that the sensor data was captured, the location of the drone at the time that the sensor data was captured, an identifier of the drone that captured the sensor data, a brand of the controller unit, drone, sensor, transmitter, or receiver, or the type of sensor that captured the sensor data.
At step 310, a radio frequency spectrum is scanned to detect (e.g., initially capture) one or more radio signals (e.g., the radio signals 238b) transmitted within a pre-defined area (e.g., the area 210). The one or more radio signals may have been transmitted by a drone (e.g., the drone 220) within the area. The one or more radio signals may be detected, via radio frequency spectrum scanning, by one or more scanning receivers (e.g., the first scanning receiver 202a and/or the second scanning receiver 202b). The one or more radio signals may be transmitted to a controller unit (e.g., the controller unit 222). The controller unit and associated operator may be located outside of the area, as shown in
At step 320, a modulated radio signal of interest is determined from the one or more radio signals detected via the radio frequency spectrum scan. The radio signal of interest may be determined based on the radio frequency spectrum scan. The radio signal of interest may be additionally or alternatively determined based on system operator review via a control terminal (e.g., the control terminal 104 of
At step 330, the radio signal of interest is captured (e.g., further captured after initial detection). The captured radio signal may be modulated. The radio signal of interest may be captured by one or more scanning receivers, such as a scanning receiver that performed the radio frequency scan or another scanning receiver. The radio signal of interest may be captured based on a control message, such as a control message from a control system (e.g. the control system 106 of
At step 340, the radio signal of interest is demodulated to determine coded sensor data carried by the radio signal of interest. The sensor data may have been determined and/or captured by the drone associated with the radio signal of interest. Demodulating the radio signal of interest may comprise determining or selecting a demodulation scheme from a plurality of potential demodulation schemes. Determining or selecting the demodulation scheme may be performed automatically by the control system or control terminal or may be performed by the control system or control terminal based on system operator input to the GUI of the control terminal. Determining or selecting the demodulation scheme may be performed by the system operator. For example, the system operator may select the demodulation scheme from a pull-down menu on the GUI listing potential demodulation schemes. Determining the demodulation scheme may be additionally or alternatively based on one or more of spectral fingerprinting (e.g., a spectral signature), a brand name of the controller unit, a brand name of the drone's telemetry component(s), a brand name of the video transmitter of the drone, or a spectral mask associated with the radio signal of interest.
At step 350, the coded sensor data is decoded to determine a characteristic (e.g., one or more characteristics) of the sensor data. Decoding the coded sensor data may comprise determining a decoding algorithm to use in decoding the coded sensor data. Determining the decoding algorithm and decoding the coded sensor data may be executed by the control system or control terminal. Additionally or alternatively, the decoding algorithm may be selected by the system operator at the control terminal. For example, the system operator may select the decoding algorithm from a pull-down menu of possible decoding algorithms and/or associated manufacturers. The decoding algorithms and/or associated manufacturers in the pull-down menu may include those associated with the frequency range and/or modulation scheme of the radio signal of interest. The result of an attempted decoding of video data may be displayed on the GUI of the control terminal. If video data resulting from the attempt to decode is incoherent, the system operator may select another decoding algorithm to use in a subsequent attempt to decode. The decoded sensor data may be recorded and stored at the control system and/or the control terminal.
The sensor data may comprise data captured by one or more sensors of the drone. The sensor data may comprise a video stream, video data, one or more images, a sound stream, or sound data. The sensor data may comprise material detection data indicating that a particular (e.g., pre-determined or target) gas, chemical, air-born particulate, or other material has been detected by the drone. The sensor data may comprise material measurement data indicating a quantitative measurement associated with a detected material. For example, the sensor data may indicate a density or concentration of a gas or air-born particulate. The sensor data may comprise environmental data indicating one or more environmental variables associated with the area. For example, environmental variables may include a temperature or a sound level.
A characteristic of the sensor data may include a quality of video or sound data. For example, a characteristic may include a video/image resolution, video frame rate, or a sound bitrate. A characteristic may include the type of data (i.e., data type) of the sensor data. For example, types of data may include a video stream, video data, an image, a sound stream, sound data, material detection data, spectroscopy data, or measurement data. A characteristic or data type of the sensor data may refer to the subject material of material detection data or measurement data. Metadata associated with or forming part of the sensor data may indicate a characteristic of the sensor data. Other characteristics of the sensor data that may tend to be included in the metadata include the time that the sensor data was captured or determined, an identifier of the drone that captured the sensor data, the type of sensor that captured the sensor data, or a brand of the controller unit, drone, sensor, transmitter, or receiver.
At step 360, an alert is generated based on the characteristic of the sensor data. The alert may comprise a visual alert displayed to the system operator via the GUI of the control terminal. The alert may comprise a sound alert generated by the control terminal to notify the system operator. The alert may comprise a text message or an email message. The alert may indicate the characteristic of the sensor data and/or the sensor data itself if possible.
The alert may be generated, generally, if the characteristic of the sensor data indicates that the detected drone is performing unauthorized and/or undesirable activities, including sensor data capture. In an aspect, the alert may be generated based on comparing the characteristic of the sensor data to a pre-defined characteristic. The alert may be generated if there is a match between the two.
For example, a data type characteristic of the sensor data may be compared to a pre-defined data type. For instance, a sports facility (e.g., a stadium or ball park) operator may be concerned that a drone may capture an unauthorized video recording of a sports event. In this case, the pre-defined data type may be video data. If the data type of the sensor data is found to be video data, an alert may be generated. A similar scenario may occur with respect to a concert venue operator and audio data.
As another example, a pre-defined characteristic may relate to a subject material identified in material detection data or measurement data. If the sensor data indicates a detection of the subject material by the drone or a measurement for the subject material captured by the drone, an alert may be generated. For instance, a confidential industrial process may produce a tell-tale gas in the nearby area. Thus the pre-defined characteristic may be associated with this particular gas. If the sensor data indicates detection of the gas or comprises a measurement for the gas, an alert may be generated.
As yet another example, the pre-defined characteristic may relate to a quality metric of the sensor data, such as defining a threshold range for the quality metric. For instance, the pre-defined characteristic may relate to video resolution, such as a threshold video resolution. If the sensor data comprises video data with a video resolution above the video resolution threshold (e.g., High Definition (HD) video data), an alert may be generated because a facility operator may be concerned that confidential information may be discovered using HD video data. Conversely, if the sensor data comprises low resolution video data, no alert may be generated.
Additionally or alternatively, generating an alert may be based on determining if the sensor data is associated with a pre-defined activity in the area or at the facility. For example, the area may comprise a large parcel of rural land with a military research and development facility. If the sensor data is associated with the research and development activities, an alert may be generated. For instance, an alert may be generated based on the sensor data comprising image data of the research and development facility. Conversely, no alert may be generated if the sensor data comprises only image data of wildlife in the area.
Additionally or alternatively, generating the alert may be based on a location and/or movement of the drone, such as within the area. Determining the location and/or movement of the drone may be achieved in one or more ways. For example, the radio signal of interest discussed in relation to steps 320, 330, and 340 may comprise telemetry data generated by the drone. The telemetry data may be demodulated and decoded along with or in the same manner as the sensor data. The telemetry data may comprise one or more sets of location coordinates (e.g. GPS coordinates) of the drone. Movement of the drone may be determined based on sequential, timed sets of coordinates. Determining the location and/or movement of the drone may be achieved using a multilateration technique, particularly when the system comprises two or more scanning receivers that receive the radio signal from the drone. An example multilateration technique may be a TDOA technique. The location and/or movement of the drone may be also determined based on video data, such as video data from the sensor data. The location and/or movement of the drone may be shown to the system operator via a geographical display on the GUI of the control terminal.
Generating the alert may be based on a proximity of the drone (e.g., the drone's location) to the facility or other landmark in the area. If the drone is far away from the facility or a sensitive portion of the area, no alert may be generated. Conversely, if the drone is near the facility or the portion of the area, an alert may be generated. Generating the alert may be based on a directionality and/or speed of the drone (e.g., a movement vector), such as relative to the facility or aforementioned sensitive portion of the area. Fast movement towards the facility or the sensitive portion of the area may warrant an alert. But movement away from the facility or the sensitive portion of the area may not require an alert. Movement speed may also suggest the performance capabilities of the drone. If fast movement is determined, an alert may be generated because the drone may be a more sophisticated and expensive model that is more likely to be used by a bad-intentioned operator than an innocent hobbyist.
At step 410, a radio frequency spectrum is scanned to detect (e.g., initially capture) one or more radio signals (e.g., the radio signals 238a, 238b) transmitted within a pre-defined area (e.g., the area 210). The one or more radio signals may be detected, via radio frequency spectrum scanning, by one or more scanning receivers (e.g., the first scanning receiver 202a and/or the second scanning receiver 202b). The one or more radio signals may comprise one or more radio signals from the drone to the controller unit and/or one or more radio signals from the controller unit to the drone. The controller unit and associated operator may be located outside of the area, as shown in
At step 420, a modulated radio signal of interest is determined from the one or more radio signals detected via the radio frequency spectrum scan. The radio signal of interest may be determined based on the radio frequency spectrum scan. The radio signal of interest may be associated with a drone (e.g., the drone 220) within the area. The radio signals detected via the radio frequency spectrum scan may be sent to the control system as frequency and amplitude (FA) packets. Step 420 may be performed in the same or similar manner, in at least some aspects, as step 320 of
At step 430, the radio signal of interest is captured (e.g., further captured after initial detection). The captured radio signal may be modulated. The radio signal of interest may be captured by one or more scanning receivers, such as a scanning receiver that performed the radio frequency scan or another scanning receiver. The radio signal of interest may be captured based on a control message, such as a control message from a control system (e.g. the control system 106 of
At step 440, the radio signal of interest is demodulated to determine coded video data carried by the radio signal of interest. The video data may have been determined and/or captured by the drone associated with the radio signal of interest. Demodulating the radio signal of interest may comprise determining or selecting a demodulation scheme from a plurality of potential demodulation schemes. Step 440 may be performed in the same or similar manner, in at least some aspects, as step 340 of
At step 450, the coded video data is decoded. Decoding the coded video data may comprise determining a decoding algorithm to use in decoding the coded video data. Decoding the coded video data may be performed in the same or similar manner, in at least some aspects, as decoding the coded sensor data described in relation to step 350 of
The decoded video data may comprise a video stream or a portion of a video stream captured by the drone's on-board camera while in-flight, before taking flight, or after taking flight. The video data additionally or alternatively may comprise one or more images. The one or more images may be discontinuous with one another. That is, the one or more images may be images that are not part of a video. The video data may comprise video or images of the area, the facility, or other physical features of the area. The video data may comprise video or images of physical features outside of the defined area. Physical features may comprise, for example, a building, a parking lot, or a geographic landmark (e.g., a group of trees, a body of water, a hill, or even a far-off mountain range or other large-scale feature). The video data may comprise video or images of the controller unit operator, such as when the drone initially takes flight or when the drone lands and is recovered by the controller unit operator.
At step 460, a location associated with the drone is determined based on the decoded video data. The determined location may be a current location of the drone or a past (e.g., recent) location of the drone. The determined location may be a location of the controller unit and/or its operator. A location may be represented as a set of geographical coordinates. A location may be represented as a sub-region of the area, such as a grid sub-region. Additionally or alternatively, a location may be represented as an altitude, such as the altitude of the drone.
As an example of determining the location associated with the drone based on the decoded video data, the video data may be shown to a system operator via the GUI of the control terminal. The system operator may analyze the video data and, preferably, identify one or more physical features depicted in the video data. The system operator is preferably familiar with the area. The identified physical feature(s) may be used as a point of reference to determine the location associated with the drone. For example, the video data may comprise images of the west side of a known building in the area. Thus the system operator may determine that the drone is located west of the building. The scale of the building within the images may be also used in determining the location of the drone, with a larger scale indicating that the drone is relatively close to the building and a smaller scale indicating that the drone is relatively far from the building. The relative scales of the building or other physical feature over multiple frames of the video data may be used to determine movement of the drone and thus also the location of the drone. The system operator may also recognize a person depicted in the video data as the controller unit operator. The location of the drone at the time the person is depicted in the video data may be used as the location of the controller unit or operator thereof.
The system operator may input the location associated with the drone into the control terminal, such as via the control terminal's GUI. For example, the system operator may input the coordinates (e.g., geographical coordinates) of the location. As another example, the system operator may input a sub-region of the area. The area may be divided into a grid of sub-regions and the system operator may input the grid corresponding to the location. The system operator may input the location via a text input (e.g., “grid sub-region 3B”). The system operator may input the location by selecting an interface element corresponding to the location, such as via a pull-down menu of possible locations. The system operator may input the location via a geographical display (e.g., map) of the area in the GUI. For example, the system operator may click on or otherwise identify the location in the geographical display that corresponds to the location associated with the drone. As another example, the geographical display may be divided into a grid of sub-regions and the operator may click on or otherwise identify the grid sub-region that corresponds to the location associated with the drone.
Additionally or alternatively, the location associated with the drone may be determined by the control system or control terminal. For example, the control system or control terminal may apply image recognition techniques to the decoded video data to identify one or more physical features or a person (e.g., the controller unit operator) in the video data. The image recognition techniques may determine a facing or angle of the physical feature relative to the drone. For example, the image recognition techniques may identify which side of a building is depicted in the video data and this may be used to determine the location associated with the drone. The image recognition techniques may identify a relative scale of a physical feature shown in the video data, including any changes to the scale of the physical features over multiple frames or images of the video data. The scale of the physical feature may be used to determine the location associated with the drone (as well as any movement). For example, the scale of the physical feature may be used to determine a distance between the drone and the physical feature at the time of the video or image capture. The above-noted angle of the drone relative to the physical feature and the distance between the drone and the physical feature may be used in combination to determine the location of the drone.
Additionally or alternatively, the location associated with the drone may be determined based on auxiliary data associated with the video data. For example, the video data may comprise an on-screen display (OSD) superimposed on the video or image. The OSD may indicate the location of the drone at the time that the video, video frame, or image was captured. As another example, the video data may comprise an audio channel that indicates the location of the drone at the time the video data was captured. The drone may insert the OSD or audio channel based on telemetry data (e.g., GPS coordinates) received by or determined by the drone.
The location associated with the drone may be determined using other techniques in addition to or as an alternative to the video data techniques described herein. For example, the radio signal of interest may comprise telemetry data that indicates the location of the drone. The telemetry data may be demodulated and decoded along with, or in a similar manner as, the video data. As another example, the location associated with the drone may be determined using a multilateration technique, particularly when the system comprises two or more scanning receivers that receive the radio signal from the drone. An example multilateration technique may be a TDOA technique. When the system comprises three or more scanning receivers, the altitude of the drone may be also determined.
An alert may be generated based on the location associated with the drone. The alert may comprise a visual alert displayed to the system operator via the GUI of the control terminal. The alert may comprise a sound alert generated by the control terminal to notify the system operator. The alert may comprise a text message or an email message. The alert may indicate the location associated with the drone, such as the location of the drone, the location of the controller unit, or the location of the controller unit operator.
Generally, the alert may be generated if the location associated with the drone indicates or suggests that the presence or activity of the drone in the area may be harmful or done with ill intent. For example, an alert may be generated if the drone is found to be near (e.g., less than a threshold distance) the facility or other sub-region of the area associated with confidential activity. Conversely, no alert may be generated if the drone is found to be far away (e.g., greater than a threshold distance) from the facility or other sub-region of the area associated with confidential activity. Generating the alert may be additionally or alternatively based on a determined movement of the drone. The movement of the drone, including a directionality and speed, may be determined based on a timed sequence of locations of the drone. Generating the alert may be additionally or alternatively based on sensor data identified in the radio signal of interest, such as is described in relation to the method 300 of
The system 500 may be the same as or similar to, in at least some aspects, the system 100 of
The system 500 comprises a scanning receiver 502 within the area 510. The scanning receiver 502 may be the same as or similar to, in at least some aspects, the scanning receiver 102 of
The first and second reception ranges 540, 544 may be based on the sensitivity (e.g., gain) of the scanning receiver 502, in particular the sensitivity of the antenna of the scanning receiver 502. The first and second reception ranges 540, 544 may be based on a pre-defined sensitivity threshold. The sensitivity threshold may refer to a minimum signal strength or signal-to-noise ratio, for example, at which the scanning receiver 502 is able to extract usable information from the signal. The sensitivity threshold may be also purposefully set to effectuate a desired reception range, such as the first reception range 540 and the second reception range 544.
In an example, the scanning receiver 502 is initially configured with the first reception range 540. The drone 520 is detected in the first reception range 540. The drone's 520 controller unit 522 is outside of the first reception range 540 however. In this configuration, the scanning receiver 502 may capture radio signals 538b transmitted by the drone 520 to the controller unit 522. Yet the scanning receiver 502 is not able to capture radio signals 538a transmitted by the controller unit 522 to the drone 520 because the controller unit 522 is outside of the first reception range 540. To capture the radio signals 538a from the controller unit 522, the sensitivity threshold of the drone 520 may be adjusted to expand the initial first reception range 540 to the second reception range 544 which does encompass the controller unit 522. In this example, the first reception range 540 abuts the border 512 of the area 510 but does not extend beyond. This may be implemented so that alerts are not unnecessarily generated based on radio signals generated outside of the area 510. For example, a drone operating outside of the area 510 may be of no concern to the parties responsible for securing the area 510. The second reception range 544 does extend beyond the area 510 but may be done so to capture radio signals transmitted by the controller unit 522. The configuration of the scanning receiving 502 with the second reception range 544 may be temporary. For example, when the drone 520 leaves the area or when any operations associated with the drone 520 are deemed complete, the scanning receiver 502 may be reconfigured with the first reception range 540.
At step 610, a first scan of a radio frequency spectrum is performed by a scanning receiver (e.g., the scanning receiver 502) to detect (e.g., initially capture) one or more radio signals transmitted within a first reception range (e.g., the first reception range 540) of the scanning receiver. The first reception range may be defined by a first sensitivity threshold of the scanning receiver. Step 610 may comprise configuring the scanning receiver with the first sensitivity threshold. The first reception range may be limited to a pre-defined area (e.g., the area 510). That is, the first reception range may not extend beyond the pre-defined area.
At step 620, a modulated first radio signal of interest from the one or more radio signals transmitted within the first reception range is determined. Determining the modulated first radio signal of interest may comprise further capturing the first radio signal of interest. The one or more radio signals transmitted within the first reception range may have been transmitted by a drone (e.g., the drone 520). Determining the first radio signal of interest may be based on the first scan of the radio frequency spectrum. Step 620 may comprise determining that the first radio signal of interest was transmitted by a drone. For example, the first radio signal of interest may comprise sensor data that was captured by the drone or that is typically captured by a drone in general. As another example, the first radio signal of interest may comprise telemetry data. Step 620 may comprise determining that the first radio signal of interest was not transmitted by a controller unit (e.g., the controller unit 522), particularly a controller unit associated with the drone.
At step 630, a second scan of the radio frequency spectrum is performed by the scanning receiver to detect (e.g., initially capture) one or more radio signals transmitted within a second reception range (e.g., the second reception range 544) of the scanning receiver. The second reception range may be defined by a second sensitivity threshold of the scanning receiver. Step 630 may comprise configuring the scanning receiver with the second sensitivity threshold. Step 630 may comprise reconfiguring the scanning receiver from the first reception range to the second reception range. The second reception range may extend beyond the pre-defined area. The first reception range may be within the second reception range. The second scan of the radio frequency spectrum may be based on determining that other radio signals from the first scan of the radio frequency spectrum are not from a controller unit associated with the drone.
At step 640, a modulated second radio signal of interest from the one or more radio signals transmitted within the second reception range is determined. Determining the modulated second radio signal of interest may comprise further capturing the second radio signal of interest. The one or more radio signals transmitted within the second reception range may have been transmitted by a controller unit associated with the drone. Determining the second radio signal of interest may be based on the second scan of the radio frequency spectrum. Step 640 may comprise determining that the second radio signal of interest was transmitted by a controller unit, particularly the controller unit associated with the drone. For example, the second radio signal of interest may comprise control input data that is typically transmitted by a controller unit to a drone. Step 640 may comprise determining that the second radio signal of interest was not transmitted by a drone, particularly the drone that transmitted the first radio signal of interest.
At step 650, the first and second radio signals of interest are demodulated to determine first coded data carried by the first radio signal of interest and second coded data carried by the second radio signal of interest. The first coded data may comprise coded sensor data and/or coded telemetry data. The second coded data may comprise coded control input data. Demodulating the first and second radio signals of interest may comprise determining or selecting one or more demodulation schemes from a plurality of potential demodulation schemes. Step 650 may be performed in the same or similar manner, in at least some aspects, as steps 340, 440 of
At step 660, the first coded data and the second coded data are decoded. The first decoded data may comprise sensor data, such as is described, for example, in relation to steps 350, 450 of
At step 670, an alert may be generated based on at least one of the first coded data and the second coded data. For example, the alert may be generated based on the first coded data and the second coded data. The alert may be the same as or similar to, in at least some aspects, the alerts discussed in relation to the methods 300, 400 of
While the system and method have been described in terms of what are presently considered specific embodiments, the disclosure need not be limited to the disclosed embodiments. It is intended to cover various modifications and similar arrangements included within the spirit and scope of the claims, the scope of which should be accorded the broadest interpretation to encompass all such modifications and similar structures. The present disclosure includes any and all embodiments of the following claims.
| Number | Name | Date | Kind |
|---|---|---|---|
| 9805273 | Seeber | Oct 2017 | B1 |
| 10061018 | Naboulsi | Aug 2018 | B1 |
| 11022407 | Whitmarsh | Jun 2021 | B2 |
| 20080088508 | Smith | Apr 2008 | A1 |
| 20110267222 | Craig | Nov 2011 | A1 |
| 20150350914 | Baxley | Dec 2015 | A1 |
| 20160245907 | Parker | Aug 2016 | A1 |
| 20170039860 | Just | Feb 2017 | A1 |
| 20180129881 | Seeber | May 2018 | A1 |
| 20190180630 | Kleinbeck | Jun 2019 | A1 |
| Number | Date | Country |
|---|---|---|
| 2536043 | Sep 2016 | GB |
| 2546438 | Jul 2017 | GB |
| WO2018044231 | Mar 2018 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20210033715 A1 | Feb 2021 | US |
