The present disclosure generally relates to improved computer vision and image processing techniques and, more particularly, to systems and methods that may be used for improved detection of objects in images that are not rich in texture, such as corresponding to a region of an image depicting a water surface or an area of the sky.
Movable objects, such as unmanned aerial vehicles (UAV) (sometimes referred to as “drones”), include pilotless aircraft of various sizes and configurations that can be remotely operated by a user and/or programmed for automated flight. Movable objects can be used for many purposes and are often used in a wide variety of personal, commercial, and tactical applications. For instance, movable objects may find particular use in surveillance, national defense, and professional videography industries, among others, and are also popular with hobbyists and for recreational purposes.
In many applications, movable objects may be equipped with secondary devices to perform various tasks. For instance, secondary devices may include imaging equipment, such as one or more cameras, video cameras, etc., that captures images or video footage that is difficult, impractical, or simply impossible to capture otherwise. Movable objects may use computer vision or other image signal processing techniques to analyze these captured images to detect objects within the images and/or complete important navigational tasks, such as braking, hovering, avoiding objects, etc. As used herein, a “movable object” may be any physical device capable of moving in real space; an “object” in an image may correspond to at least one identifiable region or feature depicted in the image, such as, for example, an identifiable area in the image corresponding to a person, animal, inanimate object or group of objects, particular terrain or geography (e.g., mountain, river, sun, etc.), feature of a larger object, etc.
Movable objects often use conventional stereovision techniques to analyze the captured images. For example, a movable object may use two or more cameras to capture a first set of images of a scene at a first instance in time and capture a second set of images of the scene at a second instance in time. The scene may be any input that can be detected by the cameras and depicted in a captured image. The movable object may calculate a stereographic depth map for the scene based on a comparison of the first and second sets of images and known positions of the cameras. The movable object may use the calculated depth map to further calculate one or more status information parameters (e.g., speed, position, direction, etc.) corresponding to the movable object and/or objects in the captured images, for example, to facilitate navigational tasks performed by the movable object.
There are, however, some drawbacks when movable objects use conventional stereovision techniques to analyze captured images to complete navigational tasks. These drawbacks typically occur because conventional techniques operate under the assumption that the captured images are rich in texture, i.e., rich in colors, contrast, brightness, sharpness, etc., that provide a clear distinction between different objects in the images. Thus, movable objects that use conventional stereovision techniques to capture images of a scene with objects and features that are not rich in texture, such as a region of a water surface or a sky, may calculate inaccurate depth maps, resulting in calculations of inaccurate visual odometry parameters needed to complete navigation tasks. These inaccurate depth maps may lead to undesirable results, such as the movable object crashing, braking prematurely, hovering erratically, etc.
There is a need for improved computer-vision techniques that may be used, for example in movable objects, to detect objects and features that are not rich in texture within scenes of captured images. The computer-vision techniques should be suitable for use in a multitude of computer vision applications, including but not limited to UAVs, object and pattern recognition, machine learning, material analysis, agriculture analysis, food analysis, robotics, autonomous driving, and any other systems that would benefit from detecting objects in scenes and/or images that are not rich in texture.
The present disclosure overcomes the disadvantages of the existing technologies by providing systems and methods that may be used in computer vision systems, including but not limited to such systems in movable objects, such as aerial vehicles and platforms, UAVs, cars, boats, and robots. Unlike prior implementations, the disclosed embodiments can detect objects that are not rich in texture within scenes of captured images, such as objects in the images corresponding to regions of a water surface or a sky. The disclosed embodiments improve existing systems and techniques using stereovision, for example, by enabling a movable object to accurately calculate depth maps and successfully complete navigation techniques, such as braking, hovering, avoiding objects, etc. The disclosed systems and techniques also may reduce unsatisfactory navigation, such as crashing, premature braking, erratic hovering, etc.
In certain disclosed embodiments, systems and methods for processing image information may be used to detect a sky (or portion thereof) depicted in an image. Such embodiments, for example, may include one or memory devices storing instructions for execution by a processor, and one or more processors that are coupled to the memory devices and operative to execute the instructions. The disclosed embodiments may obtain image information or capture such image information using one or more cameras, whether internal or external to the system. In some embodiments, for example, the system may obtain the image information by retrieving it from a database. The obtained image information may include data that represents the contents of the image, such as pixel information indicating values, such as red-green-blue (RGB) values, indicating the color of each pixel in the image. Pixel information values may also include local binary pattern (LBP) values to provide the texture of an image. Other values also may be included in the image information, such as an intensity of each pixel, a number of pixels, a position of each pixel, etc.
In some embodiments, the system may be configured to determine whether the image information represents and/or includes a sky (or portion thereof) based on a classification model. The classification model may be constructed using machine learning principles, such as supervised learning, semi-supervised learning, and/or unsupervised learning. To train the classification model to detect the sky, the system may be configured using training parameters, such as RGB values corresponding to pixels in the image, local binary pattern values, intensity values, etc.
Further to the disclosed embodiments, systems and methods for processing image information may be used to detect a water surface (or portion thereof) in first and second images. Such embodiments, for example, may include one or memory devices storing instructions for execution by a processor, and one or more processors that are coupled to the memory devices and operative to execute the instructions. In some disclosed embodiments, the one or more processors may be configured to execute instructions to detect, in a first image, a first edge line based on first image information, and further detect, in a second image, a second edge line based on second image information. Each edge line may be a linear or curvilinear line, path, or other boundary that can be detected in the first and second images. In some embodiments, the first and second edge lines may correspond to the same object or feature depicted in both the first and second images, i.e., the first edge line detected in the first image may correspond to the same edge line as the second edge line detected in the second image. In some embodiments, to detect an edge line, the one or more processors may be configured to detect first and second edge points in an image (e.g., the endpoints of an edge line or other identifiable points that may be located on an edge line) and compare the relationship between the first and second edge points with a predetermined relationship. The relationship between the first and second edge points may be, for example, the distance between the first and second edge points and/or any other difference that may be determined between the first and second edge points. In some embodiments, the one or more processors may be configured to superimpose the first image onto the second image to compare the relationship between the first and second edge points with a predetermined relationship.
In accordance with certain disclosed embodiments, systems and methods may be used to process image information in an image to determine a movement for a movable object. Such embodiments, for example, may include one or memory devices storing instructions for execution by a processor, and one or more processors that are coupled to the memory devices and operative to execute the instructions. The one or more processors may be configured to execute instructions to detect whether an image includes a water surface or a sky based on image information in the image. In some embodiments, the one or more processors may be configured to determine a technique from a plurality of techniques for calculating a depth map in response to detecting the water surface or the sky in the image. In some embodiments, the determined technique may be configured to modify cost parameters or other metrics for pixels in a region of the image corresponding to a detected water surface or sky, e.g., to set the cost parameters or other metrics equal to a predetermined value in the region. The cost parameters or other metrics may be modified, for example, if the detected water surface or sky is determined to have less than a threshold amount of texture. The one or more processors may determine a movement parameter for the movable object using the generated depth map.
The above-noted plurality of techniques may include, for example, one or more types of global matching, semi-global matching, or other techniques that map similar neighboring pixels in constructing a depth map for the image. In some embodiments, the one or more processors may be configured to determine to use a first particular technique, such as global matching, to create the depth map if a sky is detected and use a second particular technique, such as semi-global matching, to create the depth map if a water surface is detected. In some implementations that using semi-global matching, the plurality of cost parameters may indicate the cost of a pixel along each path to a neighboring pixel. In some embodiments, setting the pixels' cost parameters to a value indicating the pixels are in an area with little or no texture may allow the systems and methods to ignore image information corresponding to a detected water surface or sky with insufficient texture when generating the depth map. In such embodiments, the technique for generating the depth map instead may use calculated depths of pixels surrounding, or otherwise in the same vicinity as, the pixels corresponding to regions of a detected water surface or sky with insufficient texture to interpolate or otherwise estimate depths for those pixels in the low-texture regions.
Further, in some embodiments in which the image contains low-texture regions, such as corresponding to regions showing clear sky, clear water, mirrored surfaces reflecting a clear sky, etc., the disclosed systems and methods may modify or change the movable object's landing strategy and/or visual odometry calculation strategy. For example, when detecting that a water surface is located underneath the movable object as it is flying or hovering, e.g., based on images captured from at least one sensor or camera on the movable object, the movable object may change its landing strategy to ensure that it will continue flying or hovering and/or warning the user of the underlying water surface. In some embodiments, the movable object may change at least one odometry calculation strategy, for example, so it may calculate at least one of its position and orientation without using image data received from sensors or cameras mounted underneath the movable object or otherwise directed below the movable object.
According to certain embodiments, the disclosed systems and methods may be used to process image information in an image to adjust a navigation strategy for the movable object. Such embodiments, for example, may include one or memory devices storing instructions for execution by a processor, and one or more processors that are coupled to the memory devices and operative to execute the instructions. The disclosed embodiments may detect whether the image includes a water surface or a sky based on image information in the image. In addition, in response to detecting that the image includes the water surface or the sky, the disclosed embodiments may select a navigation strategy based on whether the water surface and/or the sky has been detected. For example, if the disclosed embodiments detect a water surface, the movable object may employ a navigation strategy (e.g., a landing strategy) such that the movable object may keep flying or hovering over the water surface. The disclosed embodiments also may perform a visual odometry calculation without using a depth map in response to detecting a water surface or a sky in the image. For example, if the disclosed embodiments detect a water surface below the movable object, the disclosed embodiments may perform one or more visual calculations without using a depth map and/or may ignore odometry calculations derived from image information in regions of the detected water surface to increase flying and hovering stability.
Aspects of the disclosed embodiments may include a non-transitory tangible computer-readable medium that stores software instructions that, when executed by the one or more processors, are configured for and capable of performing and executing one or more of the methods, operations, and the like, in accordance with the disclosed embodiments. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only, and are not restrictive of the claims set forth herein.
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate disclosed embodiments and, together with the description, serve to explain the disclosed embodiments. In the drawings:
The following detailed description refers to the accompanying drawings. Wherever possible, the same reference numbers are used in the drawings and the following description to refer to the same or similar parts. While several illustrative embodiments are described herein, modifications, adaptations and other implementations are possible. For example, substitutions, additions, or modifications may be made to the components illustrated in the drawings, and the illustrative methods described herein may be modified by substituting, reordering, removing, or adding steps to the disclosed methods. Accordingly, the following detailed description is not limited to the disclosed embodiments and examples. Instead, the proper scope is defined by the appended claims.
Movable object 10 may include a housing 11, one or more propulsion assemblies 12, and a payload 14, such as one or more camera systems. In some embodiments, as shown in
Movable object 10 may include one or more propulsion assemblies 12 positioned at various locations (for example, top, sides, front, rear, and/or bottom of movable object 10) for propelling and steering movable object 10. Although only two exemplary propulsion assemblies 12 are shown in
Propulsion assemblies 12 may be configured to propel movable object 10 in one or more vertical and horizontal directions and to allow movable object 10 to rotate about one or more axes. That is, propulsion assemblies 12 may be configured to provide lift and/or thrust for creating and maintaining translational and rotational movements of movable object 10. For instance, propulsion assemblies 12 may be configured to enable movable object 10 to achieve and maintain desired altitudes, provide thrust for movement in all directions, and provide for steering of movable object 10. In some embodiments, propulsion assemblies 12 may enable movable object 10 to perform vertical takeoffs and landings (i.e., takeoff and landing without horizontal thrust). In other embodiments, movable object 10 may require constant minimum horizontal thrust to achieve and sustain flight. Propulsion assemblies 12 may be configured to enable movement of movable object 10 along and/or about multiple axes.
Payload 14 may include one or more sensory devices 19, such as the exemplary sensory device 19 shown in
Carrier 16 may include one or more devices configured to hold the payload 14 and/or allow the payload 14 to be adjusted (e.g., rotated) with respect to movable object 10. For example, carrier 16 may be a gimbal. Carrier 16 may be configured to allow payload 14 to be rotated about one or more axes, as described below. In some embodiments, carrier 16 may be configured to allow 360° of rotation about each axis to allow for greater control of the perspective of the payload 14. In other embodiments, carrier 16 may limit the range of rotation of payload 14 to less than 360° (e.g., ≤270°, ≤210°, ≤180, ≤120°, ≤90°, ≤45°, ≤30°, ≤15°, etc.), about one or more of its axes.
Communication device 20 may be configured to enable communications of data, information, commands (e.g., flight commands, commands for operating payload 14, etc.), and/or other types of signals between electronic control unit 22 and off-board entities. Communication device 20 may include one or more components configured to send and/or receive signals, such as receivers, transmitters, or transceivers that are configured to carry out one- or two-way communication. Components of communication device 20 may be configured to communicate with off-board entities via one or more communication networks, such as networks configured for WLAN, radio, cellular (e.g., WCDMA, LTE, etc.), WiFi, RFID, etc., and using one or more wireless communication protocols (e.g., IEEE 802.15.1, IEEE 802.11, etc.), and/or other types of communication networks or protocols usable to transmit signals indicative of data, information, commands, control, and/or other signals. Communication device 20 may be configured to enable communications with user input devices, such as a control terminal (e.g., a remote control) or other stationary, mobile, or handheld control device, that provide user input for controlling movable object 10 during flight. For example, communication device 20 may be configured to communicate with a second object 26, which may be a user input device or any other device capable of receiving and/or transmitting signals with movable object 10.
Second object 26 may be a stationary device, mobile device, or another type of device configured to communicate with movable object 10 via communication device 20. For example, in some embodiments, the second object 26 may be another movable object (e.g., another UAV), a computer, a terminal, a user input device (e.g., a remote control device), etc. Second object 26 may include a communication device 28 configured to enable wireless communication with movable object 10 (e.g., with communication device 20) or other objects. Communication device 28 may be configured to receive data and information from communication device 20, such as operational data relating to, for example, positional data, velocity data, acceleration data, sensory data (e.g., imaging data), and other data and information relating to movable object 10, its components, and/or its surrounding environment. In some embodiments, second object 26 may include control features, such as levers, buttons, touchscreen device, displays, etc. In some embodiments, second object 26 may embody an electronic communication device, such as a smartphone or a tablet, with virtual control features (e.g., graphical user interfaces, applications, etc.).
Power storage device 18 may be a device configured to energize or otherwise supply power to electronic components, mechanical components, or combinations thereof in the movable object 10. For example, power storage device 18 may be a battery, a battery bank, or other device. In other embodiments, power storage device 18 may be or include one or more of a combustible fuel, a fuel cell, or another type of power storage device.
Communication device 20 may be an electronic device configured to enable wireless communication with other devices and may include a transmitter 32, receiver 34, circuitry, and/or other components. Transmitter 32 and receiver 34 may be electronic components respectively configured to transmit and receive wireless communication signals. In some embodiments, transmitter 32 and receiver 34 may be separate devices or structures. Alternatively, transmitter 32 and receiver 34 may be combined (or their respective functions may be combined) in a single transceiver device configured to send (i.e., transmit) and receive wireless communication signals, which may include any type of electromagnetic signal encoded with or otherwise indicative of data or information. Transmitter 32 and receiver 34 may be connected to one or more shared antennas, such as the exemplary antenna in
Communication device 20 may be configured to transmit and/or receive data from one or more other devices via suitable means of communication usable to transfer data and information to or from electronic control unit 22. For example, communication device 20 may be configured to utilize one or more local area networks (LAN), wide area networks (WAN), infrared systems, radio systems, Wi-Fi networks, point-to-point (P2P) networks, cellular networks, satellite networks, and the like. Optionally, relay stations, such as towers, satellites, or mobile stations, can be used, as well as any other intermediate nodes that facilitate communications between the movable object 10 and second object 26. Wireless communications can be proximity dependent or proximity independent. In some embodiments, line-of-sight may or may not be required for communications.
Electronic control unit 22 may include one or more components, including, for example, a memory 36 and at least one processor 38. Memory 36 may be or include non-transitory computer readable media and can include one or more memory units of non-transitory computer-readable media. Non-transitory computer-readable media of memory 36 may be or include any type of volatile or non-volatile memory including without limitation floppy disks, hard disks, optical discs, DVDs, CD-ROMs, microdrive, magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory integrated circuits), or any other type of media or device suitable for storing instructions and/or data. Memory units may include permanent and/or removable portions of non-transitory computer-readable media (e.g., removable media or external storage, such as an SD card, RAM, etc.).
Information and data from sensory devices 19 and/or other devices may be communicated to and stored in non-transitory computer-readable media of memory 36. Non-transitory computer-readable media associated with memory 36 also may be configured to store logic, code and/or program instructions executable by processor 38 to perform any of the illustrative embodiments described herein. For example, non-transitory computer-readable media associated with memory 36 may be configured to store computer-readable instructions that, when executed by processor 38, cause the processor to perform a method comprising one or more steps. The method performed by processor 38 based on the instructions stored in non-transitory computer-readable media of memory 36 may involve processing inputs, such as inputs of data or information stored in the non-transitory computer-readable media of memory 36, inputs received from second object 26, inputs received from sensory devices 19, and/or other inputs received via communication device 20. The non-transitory computer-readable media may be configured to store data obtained or derived from sensory devices 19 to be processed by processor 38 and/or by second object 26 (e.g., via electronic control unit 30). In some embodiments, the non-transitory computer-readable media can be used to store the processing results produced by processor 38.
Processor 38 may include one or more processors and may embody a programmable processor, such as a central processing unit (CPU). Processor 38 may be operatively coupled to memory 36 or another memory device configured to store programs or instructions executable by processor 38 for performing one or more method steps. It is noted that method steps described herein may be embodied by one or more instructions and data stored in memory 36 and that cause the method steps to be carried out when processed by the processor 38.
In some embodiments, processor 38 may include, or alternatively may be operatively coupled to, one or more control modules, such as a communication module 40 and a flight control module 42 in the illustrative embodiment of
The components of electronic control unit 22 can be arranged in any suitable configuration. For example, one or more of the components of the electronic control unit 22 can be located on movable object 10, carrier 16, payload 14, second object 26, sensory device 19, or an additional external device in communication with one or more of the above. In some embodiments, one or more processors or memory devices can be situated at different locations, such as on the movable object 10, carrier 16, payload 14, second object 26, sensory device 19, or on an additional external device in communication with one or more of the above, or suitable combinations thereof, such that any suitable aspect of the processing and/or memory functions performed by the system can occur at one or more of the aforementioned locations.
Second object 26 may include the same or similar components as control system 23 in structure and/or function. For example, communication device 28 of second object 26 may include a transmitter 33 and a receiver 35. Transmitter 33 and receiver 35 may be the same or similar to transmitter 32 and receiver 34, respectively, in structure and/or function and therefore will not be described in detail. Electronic control unit 30 of second object 26 may be the same or similar to electronic control unit 22 in structure (e.g., may include memory, a processor, modules, etc.) and/or function and therefore will not be described in detail.
Control system 23 may receive information (“flight status information” or “status information”) relating to flight parameters of movable object 10. The status information may include information indicative of at least one of a movement and a position of the movable object, for example, while the movable object 10 is in flight or at rest. For example, the status information may include one or more of a speed, an acceleration, a heading, or a height (e.g., height above ground, altitude, etc.) of movable object 10, but is not limited thereto and may include other or additional information. Status information may be detected or collected via one or more sensors 44 included in, connected to, or otherwise associated with control system 23. For simplicity, only one exemplary sensor 44 is shown in
The water surface detection service 302, sky detection service 304, sky detector 306, classification model trainer 308, movement calculator service 310, visual odometry calculator 312, and positioning calculator 314 may be implemented in software for execution on processor 38 (
Water surface detection service 302 may allow movable object 10 to detect whether the movable object 10 encounters a water surface, for example, based on images captured by one or more cameras 19 in image system 25. Similarly, sky detection service 304 may allow movable object 10 to detect whether the movable object 10 encounters a sky based on images captured by one or more cameras. Water surface detection service 302 and sky detection service 304 may process one or more images captured by imaging system 25 to detect whether the movable object 10 encounters a water surface 404 or sky 402, such as shown in
In some embodiments, sky detection service 304 may include sky detector 306 and classification model trainer 308. Sky detector 306 may use a classification model that the classification model trainer 308 trains using training parameters or other data to allow movable object 10 to detect a sky. In some embodiments, sky detector 306 may provide the training parameters or other data to the classification model trainer 308, which uses the training parameters to train the classification model continuously, periodically, or on-demand during operation of movable object 10. Training parameters may include values, such as RGB values, local binary pattern values, intensity values, etc. In some embodiments, each value of the training parameters may correspond to one or more pixels in the image. The classification model trainer 308 may obtain the training parameters from an external resource, such as an application programmable interface (API) or database, via communication device 20. In addition, classification model trainer 308 may train the classification model using machine learning principles, such as supervised learning, semi-supervised learning, and/or unsupervised learning. Sky detection service 304 may process images captured continuously, periodically, or on-demand by imaging system 25 during operation of movable object 10 to detect a sky.
In some embodiments, movement calculator service 310 may include a visual odometry calculator 312 and a positioning calculator 314. Movement calculator service 310 may provide instructions to the movable object 10 to allow movable object 10 (e.g., via assemblies 12 and/or rotary components 24) to move or navigate through an environment without assistance from second object 26. Movement calculator service 310 may allow movable object 10 to complete navigational tasks, such as braking, hovering, avoiding objects, etc. In some embodiments, movement calculator service 310 may use one or more determinations made by the water surface detection service 302 and/or sky detection service 304 to complete navigational tasks. For example, in some embodiments, movement calculator service 310 may ignore determinations by visual odometry calculator 312 and positioning calculator 314 based on one or more determinations made by the water surface detection service 302 and/or sky detection service 304 to complete navigational tasks. In addition, in some embodiments, movement calculator service 310 may utilize outputs from visual odometry calculator 312 to verify outputs of positioning calculator 314, or vice-versa, to allow movable object 10 to navigate through an environment and/or complete navigational tasks. Movement calculator service 310 may calculate status information during the operation of movable object 10.
Visual odometry calculator 312 may calculate status information using one or more images from imaging system 25. In certain embodiments, visual odometry calculator 312 may calculate a depth map using the images from imaging system 25 to allow movable object 10 to perform navigational tasks. Visual odometry calculator 312 may use a variety of techniques to calculate the depth map, such as global, semi-global, or neighborhood matching.
Positioning calculator 314 may work with, or independently from, visual odometry calculator 312 to allow movable object 10 to calculate status information or perform navigational tasks. Positioning calculator 314 may obtain positional data from one or more sensors 44. For example, positioning calculator 314 may obtain GPS data from a GPS sensor 44. GPS data may allow positioning calculator 314 to calculate movable object 10's position (e.g., coordinate position) in the environment. Positioning calculator 314 may use an inertial measurement unit (IMU), not depicted, to obtain IMU data. An IMU may include one or more sensors 44, such as one or more accelerometers, gyroscopes, pressure sensors, and/or magnetometers. IMU data may allow positioning calculator 314 to calculate movable object 10's acceleration, velocity, angular rate, etc. about its various axes, magnetic field strength surrounding movable object 10, etc.
Body of water 404 may include a large accumulation of water, such as an ocean, pond, sea, lake, wetland, reservoir, etc., or a small accumulation of water, such as a puddle, pool, container of water, etc. In addition, body of water 404 may exist on Earth or on another planet and/or be natural or unnatural. While the disclosed embodiments refer to a body of water, those skilled in the art will appreciate the systems and methods described herein may apply to other types of liquids or materials that are not rich in texture when they are imaged by an imaging system. In the disclosed embodiments, body of water 404 includes a water surface 406. Water surface 406 may, at any given time, appear smooth or choppy. When water surface 406 appears choppy, water surface 410 may include a wave 408 (e.g., or a ripple on the water surface). Wave 408 may be produced by a variety of circumstances, such as but not limited to wind gusts or air turbulence resulting from operation of movable object 10.
As shown in
As shown in
To detect a first edge line based on the first image information in the set of first images at step 602, movable object 10 may employ the exemplary edge line detection process 700 illustrated in
At step 704, movable object 10 may detect a first edge point in the first image. An edge point may be comprised of one or pixels. Similarly, at step 706, movable object 10 may detect a second edge point in the first image. Using the first and second edge points, at step 708, movable object 10 may determine a first edge line based on comparing a relationship (e.g., distance) between the first and second edge points with a threshold relationship (e.g., threshold distance value). In some embodiments, movable object 10 may determine a first edge line in the image when the distance between the first and second edge points is less than a threshold value, or vice-versa.
In some embodiments, movable object 10 may loop through steps 702-708 before determining the first edge line. The threshold value may be, for example, a predetermined value or may be dynamically determined for different iterations of steps 702-708. In some embodiments, for example, movable object 10 may decrease the threshold value or start with a smaller threshold value than the initial threshold value if movable object 10 determines the threshold value requires refinement, e.g., based on prior edge-line calculations.
Returning to
At step 606, movable object 10 may determine a water surface based on comparing a difference between the first and second edge lines with a threshold value. Here, the threshold value may indicate a difference between the first and the second edge lines. For example, the threshold value may indicate a difference in position, area, rotation, etc. of the first and second edge lines. Thus, in some embodiments, movable object 10 may determine a water surface 406 based on determining that the change in orientation or rotation of the first and second edge lines is less than a threshold value. In some embodiments, movable object 10 may compare the difference between the first and second edge lines by superimposing the first edge line in the first image taken at Position A onto the second edge line in the second image taken at Position B. In some embodiments, movable object 10 may use GPS data from GPS sensor 44 or IMU data from an IMU unit. Movable object 10 may use conventional spatial reconstruction to superimpose the first image onto the second image to compare the difference between the first and second edge lines.
In addition to or in the place of detecting a first edge line based on first image information in a set of first images, detecting a second edge line based on second image information in a set of second images, and determining a water surface based on comparing a difference between the first and second edge lines with a threshold value; movable object 10 may determine a water surface based on a light spot on the water surface. For example, movable object 10 may detect a light spot based on first image information in a set of first images (similar to step 602), detect a light spot based on second image information in a set of second images (similar to step 604), and determine a water surface based on comparing a difference between the first and second light spots with a threshold value (similar to step 606).
However, detecting a first or second light spot may include additional steps, such as the exemplary steps illustrated in
At step 904, movable object 10 may train a classification model using a set of training parameters. In the disclosed embodiments, for example, the training parameters may include values, such as one or more of RGB values, local binary pattern values, intensity values, etc. Each value may correspond to one or more pixels in the image. The training parameters may be used to train the classification model so it may process an image's image information to distinguish which region(s) of the image corresponds to sky relative to other objects or features in the image. In certain embodiments, movable object 10 may obtain the training parameters from an external source, such as an API or database, via communication device 20. In addition, movable object 10 may train the classification model using machine learning principles, such as supervised learning, semi-supervised learning, and/or unsupervised learning. In some embodiments, movable object 10 may train the classification model using a support vector machine. Movable object 10 may also a support vector machine that uses a Gaussian kernel process to train the classification model.
At step 906, movable object 10 may determine whether the image information represents a sky based on the classification model. For example, movable object 10 may capture images continuously, periodically, or on-demand, and collect the captured image information from imaging system 25. Movable object 10 may provide the collected image information for a captured image to the classification model, which in turn uses the image information to determine one or more regions of the image corresponding to a sky. In some embodiments, steps 902-906 may be applied to multiple captured images, for example, that are averaged or otherwise combined before their image information is processed by the classification model to detect a sky in accordance with the exemplary steps of
At step 1002, movable object 10 may detect whether an image includes a water surface or sky based on image information in the image using techniques similar to those described above in relation to
In some embodiments, semi-global matching techniques may not produce optimized results, such as when there is an empty sky, e.g., planes, birds, etc. in the sky. Thus, movable object 10 may set a cost parameter, e.g., equal to a predetermined value, to indicate pixels in such an area of the image without texture. In such implementations, the cost parameter may indicate the cost of a pixel along each path to a neighboring pixel when matching with a corresponding pixel in another image. Setting the pixels' cost parameters to a value indicating the pixels are in an area with little or no texture may cause movable object 10 to ignore image information corresponding to a detected water surface or sky with little or no texture when generating the depth map, at step 1006. Those skilled in the art will appreciate that the depth map may be generated using other techniques in cases where there is little or no texture in the detected regions of the water surface or sky. In some embodiments, the depth map may be determined without using the pixel values in areas of detected water surface or sky with little or no texture, e.g., instead interpolating or otherwise estimating depths based on pixel values surrounding the area with little or no texture.
At step 1008, movable object 10 may determine a movement parameter, e.g., representing at least one of status information, navigational tasks, or a direction or rotation in which movable object 10 may move. By way of example, and not limitation, the movement parameter may correspond to a rotation about one or more axes, moving in a particular direction, hovering, accelerating, decelerating (braking), changing altitude, changing flight modes, changing flight paths, and so forth.
In addition, at step 1010, movable object 10 may adjust its navigation strategy based on whether a water surface and/or sky has been detected in the image and/or the location of the detected water surface or sky in the image. In some embodiments, for example, movable object 10 may decide to use only GPS data and/or IMU data when navigating over a detected water surface or sky. In other embodiments, movable object 10 instead may decide to turn off or limit use of its braking or hovering operations when traveling over a detected region of a water surface or sky. For example, in some embodiments, movable object 10 may adjust its navigation strategy if a water surface is detected so it may refuse to land until it has passed the water surface or no longer detects a water surface below it.
At step 1012, movable object 10 may adjust a visual odometry calculation by not using a depth map if a water surface or sky has been detected in the image. In such cases, the movable object 10 may determine visual odometry instead by relying on GPS data or IMU data. For example, movable object 10 may adjust a visual odometry calculation by not using the depth map when a body of water is detected while movable object 10 is hovering. Movable object 10 may not use the depth map in this example because the depth map may produce instability while hovering.
In some embodiments, movable object 10 may skip or otherwise not complete one or more of the steps in the exemplary process 1000. For example, in some embodiments where the detected water surface or sky has insufficient texture, the movable object 10 may not perform steps of calculating a depth map (step 1004) and/or generating the depth map (step 1006). In such embodiments, movable object 10 may not perform one or more of these steps, for example, to reduce unnecessary processing operations in situations where generating a depth map may not be necessary or useful for completing one or more of the steps 1010-1012. For example, when movable object 10 detects an underlying water surface, movable object may not decide to generate a depth map and instead may alter or change its landing strategy and/or visual odometry calculation strategy without using a depth map.
In the exemplary steps and figures described above, image information may be organized by regions of an image. Therefore, in some embodiments, movable object 10 may use only image information related one or more regions in the image, which may increase the processing speed for movable object 10. For example, movable object 10 may process an image only from a subset of image information based on prior experiences with the environment, prior experiences with the image data, the navigational task that the movable object is trying to complete, status information that the movable object is trying to calculate, etc. In some embodiments, movable object 10 may use only certain regions in an image, for example, representing the bottom portion of an image when the movable object is trying to hover or it previously detected a water surface at a particular position or GPS location.
Modifications and adaptations of the embodiments will be apparent from consideration of the specification and practice of the disclosed embodiments. For example, the described implementations include hardware, firmware, and software, but systems and techniques consistent with the present disclosure may be implemented as hardware alone. Additionally, the disclosed embodiments are not limited to the examples discussed herein.
Computer programs based on the written description and methods of this specification are within the skill of a software developer. The various programs or program modules may be created using a variety of programming techniques. For example, program sections or program modules may be designed in or by means of Java, C, C++, assembly language, or any such programming languages. One or more of such software sections or modules may be integrated into a computer system, non-transitory computer-readable media, or existing communications software.
While illustrative embodiments have been described herein, the scope includes any and all embodiments having equivalent elements, modifications, omissions, combinations (e.g., of aspects across various embodiments), adaptations or alterations based on the present disclosure. The elements in the claims are to be interpreted broadly based on the language employed in the claims and not limited to examples described in the present specification or during the prosecution of the application, which examples are to be construed as non-exclusive. Further, the steps of the disclosed methods may be modified in any manner, including by reordering steps or inserting or deleting steps. It is intended, therefore, that the specification and examples be considered as exemplary only, with the true scope and spirit being indicated by the following claims and their full scope of equivalents.
This application is a continuation of International Application No. PCT/CN2018/073864, filed Jan. 23, 2018, the entire content of which is incorporated herein by reference.
Number | Date | Country | |
---|---|---|---|
Parent | PCT/CN2018/073864 | Jan 2018 | US |
Child | 16900521 | US |