Patent Application
20210357664
A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent files or records, but otherwise reserves all copyright rights whatsoever. The following notice applies to the software and data as described below and in the drawings that form a part of this document: Copyright Raven Industries, Inc. of Sioux Falls, S. Dak. All Rights Reserved.
This document pertains generally, but not by way of limitation, to remote obstacle and hazard detection and identification for agricultural vehicles.
Agricultural vehicles (e.g., one or more of vehicles, implements or both) are operated in environments including crops, uneven terrain, fixed-position or moving obstacles and fixed-position or moving hazards. Vehicle operators learn the locations of known obstacles and hazards (collectively obstacles) through repetitive work in fields and actively navigate around these obstacles based on memory or field maps that are annotated with obstacle locations.
In other examples, technicians conduct drone flights over fields with cameras or video cameras (collectively cameras) to attempt to identify obstacles, Alternatively, various other sensors, including multi-spectral sensors, may he deployed either by satellite or aerial drone to map the topography of a field and to identify regions of concern. The identified obstacles are then manually input into a field map on a field computer to assist with navigation of the agricultural vehicle relative to the obstacles, for instance to interrupt application of an agricultural product or provide guidance around obstacles. Alternatively, obstacles identified in previous agricultural operations e.g., planting earlier in the season or harvesting from a prior season) are logged to field maps for use in future agricultural operations.
In still other examples, agricultural vehicles are equipped with extensive instrument packages including sensors directed in multiple directions (e.g., forward, backward, to the sides or under the vehicle) to identify obstacles. The agricultural vehicles include signal processing algorithms that attempt to identify obstacles from the signals of the various sensors and provide alerts regarding the identified obstacles.
Optionally, an agricultural vehicle may use a ground based scout drone that drives ahead of the vehicle and provides a forward look at the forthcoming path for the agricultural vehicle. The observation of the ground based scout drone is used at the agricultural vehicle to modulate an agricultural husbandry operation, such as the application of agricultural products (e.g., for a sprayer or spreader).
The present inventors have recognized, among other things, that a problem to be solved includes detecting and identifying obstacles along a path of an agricultural vehicle and within a localized vehicle zone proximate to the vehicle with a consolidated detection system configured to detect both forthcoming obstacles (along a path) and proximate obstacles relative to the vehicle. For instance, in other systems drone flights, satellite imagery or the like are conducted over a field prior to an agricultural operation. Optionally, obstacles identified in previous agricultural operations (e.g., planting, harvesting in the prior year or the like) are noted by an operator of the corresponding vehicle. Identified obstacles are manually input to a field map, for instance on a field computer associated with an agricultural vehicle. In this example, obstacles are detected and identified outside of a present agricultural operation and input to a system, such as a field computer having a field map. In some examples, each of detection, identification, and indexing obstacles to a field map for use by an operator or automated driving system are laborious.
Additionally, systems relying on field maps including previously logged obstacles fail to detect, identify and index intervening obstacles (e.g., obstacles and hazards) that have developed in a field in the intervening time between the last update of a field map and the present (to be conducted) agricultural operation. Such obstacles include, but are not limited to, fallen trees, accumulated brush, livestock, people, damaged fences, water, washouts, sink holes, muddy terrain or the like. Extensive updating of a field map is accordingly needed before conducting the agricultural operation, or alternatively an operator (e.g., driver) accompanies the vehicle to navigate the intervening obstacles. In some examples automated driving is not advisable because of the difficulty of consistent updating of the field map with obstacles, especially with a dynamic environment, such as a field, that may have livestock, people or difficult to detect obstacles (e.g., washouts, sink holes, water, mud or the like).
In still other examples, an agricultural vehicle is equipped with instrument packages, controllers and software that attempt to detect and identify obstacles. The instrument packages include sensors, such as cameras (e.g., video, still or the like), thermographic, spectrographic sensors or the like, mounted to the vehicle and aimed in multiple directions to attempt to detect obstacles in a manner similar to an in person operator. The controllers for these systems include signal processing algorithms that attempt to identify obstacles from the signals of the various sensors and provide alerts regarding the identified obstacles. These systems are technically intensive and expensive. In addition, the signals are provided from sensors that are affixed to the vehicle. The sensors have limited fields of view due to low mounting elevations and intervening crops. Additionally, the limited fields of view are further aggravated by turning, accelerating, shaking or vibrating of the vehicle or the like that frustrate accurate sensor operation. In some examples, multiple sensors are included to provide multi-direction sensing from the agricultural vehicle. Signal processing of a signal to detect and identify an obstacle from a single sensor is a computer intensive process that is further intensified when conducted for signals from multiple sensors directed in multiple directions.
Optionally, another agricultural vehicle may use a ground based scout drone (including a ground drone in combination with an airborne drone) that moves ahead of the vehicle and provides forward observation of the forthcoming path for the agricultural vehicle including identification of crop rows. The forward observation is relayed to the agricultural vehicle and processed to ascertain crop or field conditions to modulate agricultural husbandry and log forthcoming problem areas. The scout drone is used to look ahead of the agricultural vehicle (e.g., along the path or route the vehicle will follow) and facilitate operation of the vehicle along that path. In some examples, the scout drone is not configured to monitor a zone around the vehicle, and detect and identify obstacles (livestock, people including children or the like) that may be under or around the vehicle and at risk of a collision with the vehicle without otherwise being along the forthcoming path of the vehicle.
The present subject matter provides a solution to these problems, for instance with an autonomous obstacle monitoring and vehicle control system configured to operate a remote sensing device (e.g., a drone, boom, articulating arm or the like) including one or more sensors. The remote sensing device is movable relative to the agricultural vehicle, and configured to observe obstacles proximate to the agricultural vehicle or along a path of the vehicle, for instance based on an assigned mission for the remote sensing device.
The system includes an obstacle recognition module in communication with the remote sensing device. The obstacle recognition module is configured to identify and index obstacles along the path or proximate to the agricultural vehicle. For instance, an observed obstacle (e.g., corresponding information or signals representing the obstacle from the one or more sensors) is compared with one or more archived obstacles having associated archived characteristics, such as pixel attributes representing one or more shape, brightness, groupings or arrangement of pixels or the like. In another example, the observed article includes associated non-optical information including a heat signature, ultrasound profile, radar or LIDAR profile or the like that is compared with corresponding archived characteristics of the archived obstacles. The obstacle recognition module identifies the obstacle (e.g., optionally labels or appends indications to the obstacle) based on the comparison, such as an obstacle name and associated probability of identification, for instance, ‘truck’ and ‘90 percent’ probability. In other examples, the obstacle recognition module indexes the identified obstacle with its location (e.g., relative to the agricultural vehicle or other frame of reference) or vector (e.g., including a location, speed, acceleration, direction of movement including component vectors or the like). The obstacle recognition module, in one example, monitors the identified obstacles with continued observation through the remote sensing device, and updating of identification and indexing.
The system further includes a vehicle operation module that autonomously controls the agricultural vehicle based on the identified and indexed obstacles. For instance, the vehicle operation module includes one or more of autonomous steering, throttle and braking control. The system described herein modulates the autonomous control according to the identified and indexed obstacles. For example, the vehicle operation module receives one or more of locations, vectors, identity or the like of identified obstacles and refines or updates planned paths (to avoid obstacles or revise paths for enhanced efficiency, to meet another vehicle at a specified location or the like), varies vehicle navigation along planned paths (to avoid obstacles), varies planned agricultural operations or the like (e.g., spraying with one or more nozzles, the application of product, boom height, implement height or the like).
In still other examples, the obstacle recognition module assigns a priority to the identified obstacle to modify operation of the remainder of the system including a vehicle operation module that autonomously controls the agricultural vehicle. For example, categories of identified obstacles include a halt operation, modified operation or normal operation indication. Identified obstacles within the halt operation indication category interrupt operation of the agricultural vehicle, for instance the vehicle operation module prevents or arrests driving operation. One example of a halt operation indication includes identified humans or animals within a threshold range of the vehicle or having an intercepting vector with the vehicle. Another example of a halt operation indication includes a diagnostic obstacle, such as a seized wheel or faulty critical component of the vehicle. For safety or operation reasons the vehicle operation module observes the halt operation indication and arrests operation of the agricultural vehicle.
In another example, identified obstacles having the normal operation indication are monitored, however the vehicle operation module continues a planned (normal) operation without modification or halting. For instance, the identified obstacle is outside of a path (e.g., swath, guidance line or the like) of the vehicle, has a vector indicating no interception or a minimal likelihood of interception with the vehicle, is a diagnostic obstacle but has minimal impact on operation of the vehicle or the like.
In still another example, the identified obstacles have a modified operation indication and include identified obstacles such as animals or humans that are outside of a threshold range from the vehicle or have a vector directed away from the vehicle. Optionally, an identified obstacle having a modified operation indication include static obstacles that permit avoidance (e.g., fallen tree, damaged fence, unharvested crop, saturated or muddy soil, sink holes, washed out field zones or the like) or diagnostic obstacles including one or more plugged sprayer nozzles, a fault row section of a planter or the like. With the modified operation indication the vehicle operation module continues operation with one or more variations based on the identified obstacle including autonomously navigating around the obstacle, compensating for a diagnostic obstacle (logging the incomplete product application or planting operation, increasing droplet size to offset spray drift, increasing flow rate in an adjacent nozzle or the like).
The autonomous obstacle monitoring and vehicle control system facilitates the assignment of one or more missions to the remote sensing device for observation of both field and diagnostic obstacles in a global manner, in contrast to discrete systems that conduct husbandry evaluations or are statically mounted around a vehicle. Instead, the systems described herein use a remote sensing device to observe a variety of obstacles, identify and index those obstacles, and then cooperatively communicate the identified obstacles to enhance autonomous control of the agricultural vehicle including, but not limited to, driving operation of the agricultural vehicle.
This overview is intended to provide an overview of subject matter of the present patent application. It is not intended to provide an exclusive or exhaustive explanation of the invention. The detailed description is included to provide further information about the present patent application.
In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views. Like numerals having different letter suffixes may represent different instances of similar components. The drawings illustrate generally, by way of example, but not by way of limitation, various embodiments discussed in the present document.
One example of an autonomous obstacle monitoring and vehicle control system 110 (referred to herein as the system 110) is shown in
One example device 114 includes a drone (aerial or ground) having one or more sensors configured to observe an area proximate to the drone and obstacles therein. The drone including the associated sensors is guided along a determined path, proximate to the vehicle or the like and observes potential obstacles proximate to one or more of the path or the vehicle. Optionally, the remote sensing system 112 includes a docking station 116 configured to store, deploy and retrieve the drone. The docking station 116 facilitates recharging of the drone, downloading or uploading of information to and from the drone including observations by the sensors associated with the drone or the like. Additionally, the docking station 116 facilitates deployment and retrieval of the drone through anchors, fiducials or the like described herein. In another example, the drone includes a wireless transceiver and is configured to supply information (e.g., observations by sensors, kinematics for the drone, such as location, velocity, acceleration or the like) without docking of the drone to the docking station 116.
Another example of a remote sensing device 118 is shown in
The autonomous obstacle monitoring and vehicle control system 110 further includes a controller, such as an autonomous agricultural system controller 104 shown in
The controller 104 includes a mission administration module configured to operate the remote sensing devices 114, 118. For instance, the mission administration module includes a database of mission types, associated mission routes, known or archived obstacles associated with the mission type or the like.
The mission administration module in an example provides the remote sensing devices 114, 118 with instructions for conducting the various missions including one or more of, but not limited to, mission routes the devices will move along, mission time, locations for observation (e.g., proximate to a path, proximate to the agricultural system, proximate to another vehicle or implement or the like), obstacle characteristics, such as a archived obstacle characteristics to facilitate identification of observed obstacles or the like. Optionally, the mission administration module of the controller 104 actively controls the remote sensing devices during conduct of a mission. In another example, the controller 104 hands off control after providing instructions to the remote sensing devices (including control systems associated with the remote sensing devices).
The autonomous agricultural system controller 104 further includes a vehicle operation module that interconnects the controller with the operable features of the agricultural system 100 including, but not limited to, steering controls, throttle and brakes, transmission, agricultural implement controls or the like. The vehicle operation module is configured to control the agricultural system (e.g., vehicle, implement, combination of both or the like) based on the determined path and one or more obstacles, such as obstacles identified and indexed with the autonomous obstacle monitoring and vehicle control system 110 described herein.
As further discussed herein, the system 110 includes an obstacle recognition module in communication with the remote sensing device 114 (or 118). In one example, the obstacle recognition module is a component (circuit, processor configured to carry out instructions or the like) of the autonomous agricultural system control 104. In another example, the obstacle recognition module is provided with a different component of the agricultural system 100, a different vehicle (e.g., a companion vehicle), a server, a mobile device, with the remote sensing device 114 (or 118) or in a cloud based computing environment. The obstacle recognition module interprets observations conducted with the remote sensing device 114 and identifies obstacles from the observations. As discussed herein, in one example characteristics (e.g., of observed obstacles) detected with the remote sensing devices 114, 118 and their associated one or more sensors are compared with characteristics of archived obstacles. The obstacle recognition module identifies observed obstacles based on the comparison. The identified obstacles are indexed to enhance control of the agricultural system. For example, locations of the identified obstacles are indexed to field maps, relative to the agricultural system, relative to a coordinate system or the like. In another example, a vector of the obstacle (e.g., its direction of travel and magnitude such as speed, acceleration or the like) is indexed. The vehicle operation module controls the agricultural system based on the determined path (e.g., from the path module) and the identified and indexed obstacles. As discussed herein, vehicle operation with respect to the path and obstacles includes conducting one or more of normal operation (operation is not modified or is modified in a manner that does not affect the specified goal or result of an agricultural operation), modified operation (navigating around or relative to the obstacles) or halted operation (e.g., for high priority obstacles, like humans, or obstacles that are unnavigable by the system 110).
As shown in
In one example, the observations made with the remote sensing device 114 are interpreted by the obstacle recognition module, optionally a component of the autonomous agricultural system controller 104 or another component of the agricultural system 100 (e.g., onboard the agricultural system) or with one or more remote systems, for instance, associated with the autonomous obstacle monitoring and vehicle control system 110 such as a mobile device, cloud, a computing environment or the like. In still another example, obstacle recognition is conducted with the remote sensing device 114, for instance, with an onboard processer provided thereon,
Referring again to
In the example shown in
In still other examples, as described herein, the inspection route 210, in another example, includes turning or directing the sensors of the remote sensing device 118 in different directions relative to the agricultural system 100 including, but not limited to, directing the sensors along a determined path outwardly away from the agricultural system 100, for instance, to detect one or more of diagnostic or field obstacles proximate to the agricultural vehicle along or proximate to the determined path or the like. In a similar manner, the remote sensing device 114 shown in
As further shown in
In another example, the autonomous agricultural system controller 104 includes a path module 302. As previously described herein, the path module 302 receives one or more of determined or preplanned paths (guidelines, swaths, turn segments or the like) for the agricultural vehicle to provide initial automated control, direction and guidance of the agricultural vehicle, implement or the like, for instance, within a field. in another example, the path module 302 is configured to provide ongoing generation of paths such as guidelines, swaths, turn segments or the like for the agricultural vehicle to move through a field and conduct one or more agricultural operations. As described herein, the paths determined with the path module 302 (e.g., generated, retained, received or the like) are modified or updated, according to obstacles identified and indexed, for instance, with the obstacle recognition module 310.
As further shown in
In an example including the scouting mission, the mission administration module 304 guides the remote sensing device 114 along a determined path, for instance received from the path module 302, to accordingly observe one or more obstacles proximate to the determined path of the agricultural system 100. During the scouting mission, the remote sensing device 114 is configured to observe one or more of the path, the area proximate to the path or the like with one or more onboard sensors. The observations of the remote sensing device 114 are interpreted with the obstacle recognition module 310 to identify and index obstacles observed proximate to the determined path to facilitate modification of operation of the agricultural system 100, including navigation relative to the obstacles.
The mission administration module 104 operates the remote sensing device 114 through automated control of the remote sensing device 114 or uploading of a mission plan to the remote sensing device 114 for to facilitate conducting of the mission by the remote sensing device 114. In another example, the mission administration module 304 initiates performance of the mission, for instance, instructing the remote sensing device 114 to initiate movement relative to the agricultural system 100 to move the remote sensing device 114 along the determined path, along the mission route or the like. In another example, the mission administration module 304 initiates, conducts and terminates one or more of the missions stored within the mission administration module 304 with the remote sensing device 114 and optionally guides collection or retrieval of the remote sensing device 114, for instance, to the docking station 116 provided with the agricultural system 100.
As previously described, in one example, the autonomous agricultural system controller 104 includes a mission administration module. In another example, a cloud-based system 316 is also provided as a component of the autonomous obstacle monitoring and vehicle control system 300. In one example, the cloud-based computing system 316 provides an intermediate component or interface between the remote sensing device 114 and the autonomous agricultural system controller 104. In another example, the cloud-based system 316 provides further information to one or more of the remote sensing device 114 or the controller 104. For instance, the cloud-based system 316 provides one or more of the mission routes, mission types and associated mission routes or the like to the remote sensing device 114 by way of instruction from the autonomous agricultural system controller 104. In another example, one or more other components of the controller 104 or the obstacle recognition module 310 are provided by way of the cloud-based system 316. In another example, the obstacle recognition module 310, mission administration module 304 or, in some examples, the path module 302 are included as components of the cloud-based system 316 and accordingly are remote relative to (though in communication with) one or more of the remote sensing device 114, the autonomous agricultural system controller 104 or the like.
Optionally, an intermediate controller is interposed between the autonomous agricultural system controller 104 and the remote sensing device. In one example, the intermediate controller is a ‘dock brain’ associated with the dock station 116 shown in
As further shown in
In some examples, the user interface 308 includes, but is not limited to, one or more of a tablet computer, touchscreen, computer, smartphone, field computer or the like provided on or associated with the agricultural system 100. For instance, the user interface 308 is, in one example, a component of or installed within the cab of an agricultural vehicle such as a tractor, combine sprayer, truck or the like, In another example, the user interface 308 is optionally included as a component of another agricultural system, for instance, a companion agricultural system or one in communication with another agricultural system. In still another example, the user interface 308 is a component such as a smartphone, tablet computer or the like that is maintained separately relative to the agricultural system 100.
Referring again to
In another example, the obstacle recognition module 310 is incorporated with or included with the autonomous agricultural system controller 104. In still another example, the obstacle recognition module 310 is optionally included with one or more of the remote sensing devices 114, 118, for instance, to facilitate the identification and indexing of observed obstacles onboard the remote sensing devices. Further, the obstacle recognition module is, in another example, provided remote to the agricultural system 100, for instance through the cloud based system 316.
The obstacle recognition module 310 identifies and indexes obstacles. In one example, the obstacle recognition module 310 interprets sensory information received from one or more of the remote sensing devices 114, 118 by way of wireless communication between the remote sensing device 114 and the autonomous agricultural system controller 104 or direct (wired) communication, for instance, when the remote sensing device 114 docks with a docking station such as the docking station 116 shown in
As shown in
In still other examples, the obstacle recognition module 310 is configured to compare sensory data, for instance, from other non-visual sensors including one or more of chemical sensors, thermographic sensors, hyperspectral sensors, ground penetrating radar, radar, LIDAR or ultrasound sensors. The obstacle recognition module 310, for instance with the obstacle comparator 312, compares the observations with corresponding archived characteristics comparable to the observation type (e.g., radar characteristics are compared with radar observations or the like).
In one example, one or more of the obstacle comparator 312 or the identification module 314 includes an onboard or remote artificial intelligence module configured to identify obstacles from the sensor feeds of an observed area (e.g., proximate to a determined path, proximate to the vehicle or the like) with one or more neural networks. The networks receive sensor data, for instance, images, video, return signal ranging information, thermographic observations, spectrographic observations, radar observations. LIDAR observations, ultrasound observations or the like. The obstacle comparator 312 (for instance, an artificial intelligence neural network) processes that information through a plurality of layers that compare the observations with archived characteristics of archived obstacles. For instance, the one or more neural networks, in one example, look for pixel attributes representing shape, brightness and groupings that resemble image characteristics the neural network has previously been trained for, for instance, with previous comparisons. In another example, the one or more neural networks compare thermographic or spectragraphic archived characteristics with thermographic or spectragraphic information provided by the remote sensing device 114 (or 118) observations. This process is optionally repeated for other sensor observation formats (e.g., radar, LIDAR or the like) with comparison to corresponding archived characteristics.
Obstacles within the transmitted observation data from the remote sensing device 114 include image characteristics that match or approximate stored archived characteristics of the one or more neural networks, optionally within a probability. For example, where a truck drives into the field of view of an RGB camera sensor associated with the remote sensing device 114, the one or more neural networks of the obstacle comparator 312 compare observations of the truck with archived characteristics of a variety of obstacles. The identification of the obstacle includes, in one example, a category or label, for instance, truck, human, livestock, fence, washout, brush or the like with an optional probability or confidence of the identification (for instance, a 90 percent confidence, 70 percent confidence, 50 percent confidence or the like). The identification module 314 selects the closest (e.g., highest confidence) comparison and assigns a label to the obstacle, such as “truck” and an optional probability or confidence of identification, for instance, a 90 percent confidence based on the comparison conducted with the comparator 312. Where a probability is included, the probability facilitates the filtering of false positives once a baseline accuracy for obstacle identification, for instance, of one or more categories of obstacles or different types of obstacles is known.
In another example, the identification module 314 identifies a potential obstacle from the observations if the confidence of the obstacle comparison at the comparator 312 is greater than a specified threshold, such as a threshold confidence value or the like (e.g., 40, 50, 60, 70, 80 percent or more likelihood of identification or the like). In the example with the previously identified truck, if the highest confidence from the comparison conducted with the obstacle comparator 312 was below a threshold confidence, such as 50 percent, the identification module 314 may withhold identification and optionally tag the potential obstacle for further investigation such as further observation with the remote sensing device 114.
The pre-trained or ongoing trained neural network as the obstacle comparator 310 allows for rapid analysis of incoming signals, for instance, from the remote sensing device 114 to facilitate comparison to archived characteristics and identification of obstacles as provided herein above. Additionally, the pre-trained or ongoing trained neural networks also facilitate continued training of the neural networks (e.g., an example of deep learning), for instance, with the ongoing monitoring conducted with the remote sensing device 114 and ongoing comparisons. The continued training enhances identification of obstacles as the neural network (e.g., obstacle comparator 310) continues to refine archived characteristics, associated archived obstacles, and increase the confidence of identification while at the same time learning new obstacles from observations from the remote sensing device 114. Optionally, the comparisons and identifications (e.g., of the comparator 310 and identification module 312) are conducted across a plurality of systems 300 and the refined archived characteristics, associated archived. obstacles, and newly learned obstacles from other systems 300 are consolidated in a client network (e.g., as part of a cloud based system) that facilitates a global enhancement of identification of obstacles.
In other examples, the identification module 314 is configured to associate or match various sensor observations, for instance, from one or more different sensor types associated with the remote sensing device(s) and accordingly provide component comparisons between the archived characteristics and their associated sensor observations, for instance, one or more of optical or video archived characteristics are compared against corresponding optical or video observations from the remote sensing device 114. In another example, thermographic archived characteristics, hyper spectral archived characteristics, radar characteristics or the like are compared with corresponding sensory observations from the remote sensing devices 114. In various examples, the comparisons between these archived characteristics and the associated sensory observations from different sensors facilitate the enhanced identification of obstacles. The comparisons are conducted in concert or in tandem to enhance the identification of obstacles, for instance, to provide a higher confidence that the obstacle identified has been properly identified. In one example, the identification module 312 identifies an obstacle, such as a cow, with a 60 percent component confidence based on a comparison of thermal characteristics with thermal observations and a 90 percent confidence based on a comparison of visual characteristics with visual observations. Because the component identifications and associated confidences are similar (e.g., identifying an obstacle as the same) the confidence of the identification may be scaled or multiplied to a composite value greater than one or both of the component confidences (e.g., 95 percent).
In another example, the identified obstacle such as a human, livestock or the like has a lower confidence value where one or more of the comparisons, for instance, between thermographic archived characteristics and thermographic observations and visual archived characteristics and visual observations differs. For instance, if the comparison of visual archived characteristics with corresponding visual observations indicates the presence of livestock (e.g., greater than 50 percent) while the associated thermographic comparisons do not indicate livestock (e.g., less than 50 percent) the identification of the obstacle is accordingly assigned a lower composite confidence value, for instance, 50 percent confidence value or less relative to a 90 percent or 95 percent confidence value when the comparisons between various archived characteristics and the associated sensory observations correspond.
In another example, the indexing module 316 is configured to index the identified object, for instance, by way of location relative to the vehicle, relative to a coordinate system, relative to other obstacles or reference points in a field or the like. In one example, the indexing of the obstacle includes, but is not limited to, determining a vector or other kinematic characteristic associated with the obstacle. For instance, one or more of livestock, humans, vehicles within a field or the like in addition to having a location may also have one or more kinematic characteristics including acceleration, velocity or the like. The indexing module 314 is configured to determine associated kinematic characteristics and index them to the identified obstacle. For instance, an obstacle is monitored over time to assess kinematic characteristics, and where movement is detected the indexing module 314 appends the characteristics as a vector or other indicator corresponding to the detected movement. Optionally, the vector has an origin corresponding to a present location of the identified obstacle. In another example, comparisons of multiple observations of obstacles with corresponding archived characteristics are conducted, and changes in comparisons indicate movement including magnitude and direction. A corresponding vector is then appended to the obstacle. As described herein, the kinematic characteristics including, for instance, velocity, location, acceleration or the like are used to enhance the autonomous control of the agricultural system 100. For instance, control of the agricultural system 100 is conducted with the vehicle operation module 306 according to the identification of the obstacles and indexing of obstacles including predicted future paths (based on kinematic characteristics, such as appended vectors) of the obstacles.
In another example, the obstacle recognition module 310 includes a prioritizing module 318. The prioritizing module 318 is configured to assign priorities to obstacles based on a catalog set of priorities, priority rules, user input priorities or user input priority rules or the like. In one example, archived obstacles such as humans, livestock or the like have a higher priority relative to other identified obstacles including, for instance, brush, washouts, rocks, saturated or soaked areas of the field or the like. As described herein, the assigned priority of an identified object changes the operation of the agricultural system 100, for instance with the vehicle operation module 306 of the autonomous agricultural system controller 104.
In one example, the identification of a human proximate to a determined path of the agricultural system 100 is given a high priority while other obstacles such as livestock, brush, fence or rocks or the like proximate to the determined path are given a lower priority (and optionally scaled lower priorities with livestock higher than brush or similar inanimate obstacles). For instance, the prioritizing module 318 assigns a higher priority to particular types of obstacles, for instance, humans or the like. In other examples, the prioritizing module assigns a priority based on probability of the identification (e.g., confidence), for instance, a rock that is identified with a smaller probability is assigned a lower priority relative to a rock identified with a higher probability. In other examples, the proximity to the agricultural system 100 or the determined path of the agricultural system 100 triggers the assignment of a higher priority to an identified obstacle in comparison to the same identified obstacle that is not proximate to the determined path or is not proximate to the agricultural system 100. In other examples, the proximity to the determined path or the agricultural system is also triggered by the vector of the identified obstacle (where present) in addition to or instead of the index location of the obstacle.
The priorities of identified obstacles are provided to the vehicle operation module 306, and the vehicle operation module 306 accordingly conditions the operation of the agricultural system 100. In one example, a higher priority obstacle may trigger the halting of operation of the agricultural system 100 (e.g., halted operation). A lower priority obstacle may trigger modified operation of the agricultural system 100, for instance, by way of navigating the system 100 around the obstacle, moving the system 100 in a direction that guides the agricultural system 100 away from the vector of travel of a moving obstacle such as a secondary vehicle, human, livestock or the like.
In operation, the obstacle recognition module 310 cooperates with and communicates with other components of the autonomous obstacle monitoring and vehicle control system 300 to identify and track obstacles and facilitate enhanced guidance of the agricultural system 100 through the field while still allowing the agricultural system 100 to accomplish one or more agricultural processes. The remote sensing device 114 is deployed by the autonomous agricultural system controller 104 by way of a mission administration module 304 providing a mission route to the remote sensing device 114 or, in another example, guiding operation of the remote sensing device 114 along a mission route. The remote sensing device 114, while conducting the mission, observes the area proximate to a determined path, proximate to the agricultural system 100 or the like, for instance, along a corresponding mission route for the remote sensing device 114. In some examples, the mission route is provided around or proximate to the agricultural system 100 and, in other examples, is provided along the determined path of the agricultural system 100, for instance, along one or more guidance lines, turn segments or the like. The remote sensing device 114 observes the area proximate to the determined path proximate, to the agricultural system 100 or the like.
The observations (e.g., sensor data) form the remote sensing device are conveyed to the controller 104, for instance, when docked to the docking station 116 or in a wireless manner, for instance, by wireless communication with the controller 104. The autonomous agricultural system controller 104 passes the observations to the obstacle recognition module 310 for identification and indexing of obstacles with one or more of the obstacle comparator 312, the identification module 314, or the indexing module 316. In some examples, identification and indexing of obstacles includes tracking movement of the obstacles, for instance, by way of updated sensory information provided by the remote sensing device 114 continuing with additional mission controlled movement and continued or repeated observation of obstacles along the mission route.
As previously described, the obstacle recognition module 310 optionally includes a prioritizing module 318 that prioritizes the identified obstacles including prioritization based on one or more of the proximity of the object relative to the agricultural system 100 relative to the determined path of the agricultural system 100 or based on the identity of the object itself. For example, humans, livestock, other vehicles or the like may have a higher priority relative to lower priority obstacles, such as brush, washouts, fences, rocks or the like.
The identified obstacles including, for instance, their identification, confidence value of the identification, indexing of location or vectors or the like as well as optional prioritizations are passed to the autonomous agricultural system controller 104, for instance, to the vehicle operation module 306. For an agricultural operation, the vehicle operation module 306 analyzes the determined path of the agricultural system 100, such as initial guidance lines, swaths, turn segments or the like in combination with he identified and indexed obstacles proximate to the determined path. The vehicle operation module 306 generates an updated or modified path based on the identified and indexed obstacles, obstacle priorities or the like. The vehicle operation module 306 implements guidance of the agricultural system 100 according to the modified or updated path generated with the autonomous obstacle monitoring and vehicle control system 300.
The identification of obstacles, including tracking obstacles, is repeated in a continuous or ongoing manner. For example, the remote sensing device 114 repeatedly travels proximate to a determined path, a scouting route, diagnostic route or the like and observations of the device 114 update obstacles already identified (e.g., identity, confidence of the identification, location, movement or the like) or facilitate identification of previously undetected obstacles. The updated indexing and identification of obstacles, identification and indexing of new obstacles, as well as obstacle priority is communicated to the remainder of the system 300, for instance, the vehicle operation module 306 to facilitate enhanced and updated guidance of the agricultural system 100.
In one example, the assigned priorities trigger conditional responses with the autonomous agricultural system controller 104 including one or more of halted operation, for instance, with a high priority identified obstacle such as a human or livestock within proximity to the determined path or within proximity to the agricultural system 100. In another example, the autonomous agricultural system controller 104 includes other conditional responses in addition to a halt operation response. These example operations include, but are not limited to, a normal operation, modified operation or the like. In normal operation, the identified obstacle, in one example, has a lower priority and does not trigger modification of the operation of the agricultural vehicle, for instance, by the vehicle operation module 306. In other examples, the identified obstacle has a higher priority but does not otherwise trigger halted operation of the agricultural system 100. In this example, a modified operation is instituted by the vehicle operation module 306 including, but not limited to, navigating around the lower priority identified obstacle with an updated path (e.g., including real time navigation, a planned path based on the obstacle and the initial path or the like), alerting an operator that, intervention is needed or conducting additional missions, for instance, with the remote sensing device 114 such as a scout mission to further identify the obstacle or refine indexing of the obstacle location or vector proximate to the determined path or proximate to the agricultural system 100.
The obstacle recognition module 310 described herein is, in one example, a component of the system 300, such as the autonomous agricultural system controller 104. In this example, one or more of the archived characteristics of various archived obstacles as well as associated priorities are included with the controller 104 and accordingly provided in an onboard fashion with the remainder of the system 300 associated with the agricultural system 100. The obstacle recognition module 310 is optionally updated on a regular or semi-regular basis with updated archived obstacles and associated characteristics, for instance with a jump drive, download from a cloud based system 316 or the like. In another example, obstacle recognition conducted with the module 310 is conversely uploaded to the cloud based system to enhance obstacle recognition in other modules 310 of other systems 300 (e.g., on a client network or the like) by further populating archived characteristics to archived obstacles. A trained neural network (e.g., the obstacle comparator 312 discussed herein) is an example of a comparator that is enhanced over time through population of archived characteristics, archived obstacles, refinements to archived characteristics and entry of newly identified obstacles.
In still another example, the obstacle recognition module 310 is remote relative to the agricultural system 100 and provided as part of the cloud-based system 316 that is configured to receive sensory information (observations) from the remote sensing device 114 and then return identified obstacles, their priorities, as well as indexing of the obstacles (position, vector or the like) to the controller 104 for implementation of enhanced vehicle guidance by way of modification or updating of guidance of the agricultural system 100.
As previously shown and discussed herein, the various components of the autonomous obstacle monitoring and vehicle control system 300 are interconnected with one or more of wired or wireless interfaces. For instance, the remote sensing device 114, in one example, a drone is interconnected with the autonomous agricultural system controller 104 by way of a wireless connection. In another example, the remote sensing device 114 interconnects with the controller 104 upon docking with an associated docking station 116 shown, for instance, in
As further shown in
As further shown in
As further shown in
In another example, a global positioning system (GPS) unit or fiducial provided with the remote sensing device is another example of a sensor included with the sensor suite 400. The GPS unit is operated to track the remote sensing device 114 location and facilitate retrieval of the remote sensing device 114, for instance, at the docking station 116. In another example, the GPS unit is used for gross control or operation of the remote sensing device 114 to guide the remote sensing device 114 toward the docking station 116, and then one or more other sensors of the sensor suite 400 are operated (cameras, video cameras or the like) to detect the docking station 116 and enhance the precision of landing (retrieval) of the remote sensing device 114 at the docking station 116.
As further shown in
In other examples, the remote sensing device 114 includes a data transceiver 408. In one example, the data transceiver relays information with a wireless connection to one or more components of the autonomous obstacle monitoring and vehicle control system 300, 350. In one example, the data transceiver 408 is used instead of the power and data port 402 for information relaying, and handles information uploads and downloads between the remote sensing device 114 and the remainder of the system 300, 350. In another example, the data transceiver 408 works in concert with or in combination with the power and data port 402 to relay information while the remote sensing device 114 is deployed from the agricultural system 100. Upon landing, the remote sensing device 114 uploads and downloads data through the power and data port 402. The data transceiver 408 conducts transmissions with the remainder of the autonomous obstacle monitoring and vehicle control system with one or more wireless formats including, but not limited to, cellular communications, 900 megahertz or radio frequency communications, Wi-Fi networks or the like.
As further shown in
Referring again to
The docking station 116, in another example, includes one or more drone anchors 422 provided with the docking station 116. The one or more drone anchors 422 are configured to couple the remote sensing device 114 with the docking station 116 to securely dock the remote sensing device 114 when not deployed. When deployment is desired, the drone anchors 422 are released. For instance, one or more of electromagnets, mechanical latches or the like are released to free the remote sensing device 114 for deployment and operation in one or more of the missions described herein.
As further shown in
In another example, the docking station 116 optionally includes a dock brain, as described herein, including one or more of circuits, processers or the like configured to retain information, instructions or the like for use with the remote sensing device 114. In one example, the dock brain receives mission information including mission names, mission routes or the like and relays the information from the autonomous agricultural system controller 104 to the remote sensing device 114. Optionally, the dock brain facilitates the rapid uploading and downloading of information to and from the remote sensing device 114 including sensory observations made with the sensor suite 400. In another example, the dock brain provides remote control of the remote sensing device 114 including guiding the device 114 according to mission routes, deploying of the device and retrieval of the device.
An initial path 504, such as a proposed path or the like, of the second agricultural system 502 extends from the second agricultural system 502 to the first agricultural system 501. Optionally, the initial path 504 is a dynamic path that changes as the first agricultural system 501 moves in a field (e.g., conducts harvesting). The initial path 504 is one example of a determined path, for instance, provided by the field computer 352 shown in
In operation, a remote sensing device 114 is deployed from one or more of the first or second agricultural systems 501, 502 and conducts the scouting mission 500, for instance, along the scouting route 510 (e.g., proximate to the initial path 504). As shown in
The observations of the remote sensing device 114 are relayed to the remainder of the autonomous obstacle monitoring and vehicle control system 300 (or 350) to identify and index obstacles such as the field obstacles 506 along the initial path 504. Referring to
In another example, an indexing module 316 of the obstacle recognition module 310 indexes the obstacles 506 with one or more of locations, vectors or the like to track the obstacles and enhance operation of the agricultural system 502 relative to the obstacles 506 in
The identified and indexed obstacles 506 are relayed to the vehicle operating module 306 and the vehicle operating module 306 modifies the initial path to accordingly provide an updated path 512 for the second agricultural system 502. One example of a refinement or modification to the initial path conducted with the vehicle operating module 306 in
As shown in
As previously discussed,
In one example, the remote sensing device 114 in combination with the remainder of the autonomous obstacle monitoring and vehicle control systems 300, 350 identifies and indexes obstacles along the routes 612A-612C. The identified and indexed obstacles include the field obstacles 620 such as unharvested crops, bodies of water, humans, livestock, fences or the like. Optionally, the identified and indexed obstacles include the absence of previously detected obstacles, for instances, absent obstacles 622. In another example, the absent obstacles 622 correspond to other field obstacles 620 that are now absent (e.g., because of harvesting), and as the remote sensing device 114 conducts mission operations the absence of the obstacle 620 initiates removal of the previous obstacle 620 from further monitoring (e.g., by the obstacle recognition module 310 or the vehicle operating module 306), and thereby ends the effect the now absent obstacle 620 would have with the vehicle operating module 306.
By monitoring the obstacles in the field including field obstacles 620 and optionally absent obstacles 622 (or initiating the removal of previous obstacles 620, such as now harvested crops) the remote sensing device 114 in combination with the rest of the system 300 (or 350) provides updated field information to the systems 300 for corresponding modification of the operation (e.g., driving, implement operation or the like) of one or more of the agricultural systems such as the fourth agricultural system 608. In the context of
In operation, the remote sensing device 114, such as a drone, is deployed from the fourth agricultural system 608 or one or more of the other component first, second or third agricultural systems 602, 604, 606, for instance, having a docking station such as the station 116 previously shown and described in
The scouting missions conducted along the various scouting routes 612A-C allow the remote sensing device 114 to observe the area proximate to the determined path, for instance, along the determined path, adjacent to the determined path, within a specified distance relative to the determined path based on scanning ranges of the one or more sensors on the remote sensing device 114 or the like. The remote sensing device 114 then relays the observations to the remainder of the autonomous obstacle monitoring and vehicle control system 300, 350. Observations made by the remote sensing device 114 are interpreted by the obstacle recognition module 310 as discussed herein. The obstacle recognition module 310 including one or more of an obstacle comparator 312, identification module 314 and the like is configured to identify and index the locations of the identified obstacles provided along the scouting routes 612A-612C.
The identified and indexed obstacles including the field obstacles 620 (and optionally the absent obstacles 622) are provided in combination to a vehicle operating module 306 and accordingly the determined path (e.g. in one example a straight line from the system 608 to one or more of the other systems 602-606) is modified based on the intervening identified and indexed obstacles 620, 622 observed along each of the scouting routes 612A-612C. As shown in
As shown in
The unharvested crops are also examples of field obstacles 714 in
In another example, Harvested zones of the field are optionally identified and indexed as absent obstacles 716. In various examples, the absent obstacle 716 is identified and indexed in a similar manner to one or more of the other field obstacles previously described herein. For instance, the remote sensing device 114 observes the area proximate to one or more of a determined path or proximate to the agricultural systems 700, 702, and the obstacle recognition module 310 is configured to identify and index the absent obstacle 716 such as an area of harvested crops to facilitate operation or navigation of one or more of the agricultural system such as the second agricultural system 702 through the harvested portions of the field. Optionally, the absent obstacles 716 are interpreted by the system 300 as overwriting features in comparison to the previously present converse obstacles such as the field obstacles 714 (unharvested crops). The system 300, such as the vehicle operating module 306 or the obstacle recognition module 310, affirmatively initiates removal of the field obstacles 714 from further consideration by the system 300 based on the overwriting presence of the absent obstacle 716 (harvested crop zone) in a coincident indexed location to the previous field obstacle 714 (the unharvested crop).
As further shown in
In a similar manner, one or more missions are conducted proximate to the first agricultural system 700. In one example, an inspection mission 720B is conducted proximate to the first agricultural system 700 to identify obstacles proximate to the first agricultural system 700. In another example, a scouting mission 732B, is conducted along a determined path 740 (proposed, initial or undefined path) of the first agricultural system 700, for instance, while conducting harvesting operations in the field.
Referring again to
The obstacle recognition module 310 identifies and indexes one or more field obstacles including he field obstacles 708 (humans) in proximity to the first agricultural system 700. The obstacles 708 are indexed relative to the first agricultural system or another coordinate system to refine operation of the agricultural system 700 based on the obstacles and their locations. The obstacle recognition module 310 optionally assigns a priority to the identified field obstacles 708 based on one or more of the obstacle identity, indexing, proximity to a path or the system, confidence of the identification or the like. In this example, because the field obstacle 708 are humans, the obstacles are assigned a high priority and operation of the agricultural system 700 is modified to a halted operation thereby preventing operation of the first agricultural system 700.
As further shown in
In a similar manner to the inspection mission 720B, the remote sensing device 114 observes the area proximate to the scouting route 734B while conducting the scouting mission 732B. The observations of the remote sensing device 114 are relayed to the obstacle recognition module 310 to identify and index observed obstacles. For instance, in this example, the field obstacle 708, such as the human provided in front of the first agricultural system 700, as well as the field obstacle 706 (livestock) are observed with the remote sensing device 114 and the obstacle recognition module 310 conducts one or more of identification, indexing or prioritizing of the obstacles 706, 708.
In one example, the obstacles 706, 708 are relayed to the vehicle operation module 306 and the determined path 740 is modified or updated to adjust control of the system 700 according to the obstacles. One or more of driving control, implement control or the like of the first agricultural system 700 is implemented to navigate the system 700 relative to the obstacles (or alternatively halt operation depending on proximity, priority or the like) while attempting to accomplish the agricultural operation (e.g., harvesting). For instance, in one example, with the field obstacle 706 (livestock) in front of the first agricultural system 700, a modified operation and halted operation are conducted with the first agricultural system 700. For instance, the first agricultural system 700 travels to a stop location 744 proximate to the field obstacle 706 and thereafter halts further travel. In one example, while in halted operation, the autonomous obstacle monitoring and vehicle control system. 350 (or 300) sends an alert, for instance, through one or more of the user interface 308, field computer 352. or the like to request further interaction by way of an operator, remote operator or the like.
In another example, with the field obstacle 706 such as livestock having an obstacle vector 707 shown in
As further shown in
The vehicle operation module 306 receives the obstacles (e.g., identities, indexing, priorities) and determines because of the proximity of the field obstacles 706, 708 that system operation including travel to a position adjacent to the moving first agricultural system 700 (for loading of grain) should be halted until the field obstacles 706, 708 are moved to avoid collision with the obstacles. In another example, a notification is provided by way of a user interface 308, field computer 352 or the like to an operator to facilitate operation intervention, for instance, including manual guidance around the field obstacles 706, 708.
As further shown in
In a similar manner to the other scouting missions and inspection missions described herein, the observations of the remote sensing device 114 are relayed to the obstacle recognition module 310 for one or more of identification, indexing or prioritizing of obstacles proximate to the scouting route 734A. For instance, the field obstacle 710 (fallen tree), field obstacle 712, such as a washed out or muddy portion of the field, are identified, indexed and prioritized. In another example, the first agricultural system 700 is another example of an obstacle that is observed with the remote sensing device 114, and identified, indexed or prioritized with the obstacle recognition module 310. The indexing of the first agricultural system 700 includes a vector in one example.
The obstacles are relayed to the vehicle operation module 306 of the autonomous agricultural system controller 104 to facilitate updating or modification of the determined path 730. For instance, as shown in
In another example, the position of the first agricultural system 700 as well as its updated path 746 or vector are relayed to the vehicle operation module 306 of the first agricultural system 700 without recognition by the obstacle recognition module 310. Instead, the first agricultural system 700 provides position and travel information to the second system 702 vehicle operation module 306 to use in combination with obstacles 706. 708, 710 otherwise observed and identified with the remote sensing device 114.
As further shown in
As previously discussed, the diagnostic mission 820 is, in one example, conducted while the agricultural system is in operation (e.g., moving, engaging in an operation) or while stationary. In other examples, the diagnostic mission 820 is conducted in an automatic fashion, for instance, on a specified interval including, but not limited to, every two or four hours of operation of the agricultural system 800. In other examples, the diagnostic mission 820 and corresponding operation of the remote sensing device 114 is conducted on an as-needed basis. For instance, upon an alert of a potential technical or operational issue with the agricultural system 800 the autonomous obstacle monitoring and vehicle control system 300, 350 initiates the diagnostic mission 820 and automatically deploys the remote sensing device 114 to conduct observations. In another example, an operator such as a remote operator triggers the initiation of the diagnostic mission 820, for instance, upon notification of a potential technical or operational error of the agricultural system 800, and conducting of the diagnostic mission 820 including identification and indexing of the obstacles is then carried out automatically.
As shown in
The diagnostic obstacle 806, also shown in
The diagnostic obstacle 810 shown in
In operation, the agricultural system 800 is conducting an agricultural operation, such as spraying, in a field. The remote sensing system 112 is operated to conduct the diagnostic mission 820. In one example, the diagnostic mission 820 is triggered or initiated based on a schedule, interval timing or the like configured to deploy the remote sensing device 114 in an automatic fashion, for instance, when a scheduled diagnostic mission is scheduled to occur. In another example, the diagnostic mission 820 is conducted on an as-needed basis, for instance, upon detection or indication that a technical or operational error has occurred with the agricultural system 800 (e.g., with one or more other diagnostic systems). In another example, one or more diagnostic indicators are relayed to an operator, such as a remote operator, and the operator then initiates the diagnostic mission 820.
The remote sensing device 114 is deployed according to the implemented diagnostic mission 820. Referring to
As the remote sensing device 114 conducts the diagnostic mission 820, the one or more sensors associated with the remote sensing device 114 observe the agricultural system 800 and optionally the area proximate to the agricultural system 800 to assess one or more potential diagnostic obstacles. The observations of the remote sensing device 114 are relayed to the obstacle recognition module 310, for instance, a component of the autonomous obstacle monitoring and vehicle control system 300, 350 or a separate component in communication with the reminder of the system 300, 350.
As described herein, the obstacle recognition module 310 includes one or more submodules, circuits, processors, components or the like configured to identify obstacles and index obstacles from the observations made with the remote sensing device 114. For instance, one or more of the diagnostic obstacles 802-810 are identified by way of an obstacle comparator 312 configured to compare one or more observed characteristics of a potential obstacle with archived characteristics of archived obstacles. An identification module 314 identifies the obstacle based on the comparisons, for instance appending a designation (e.g., a name) to the obstacle corresponding to the comparison that generated the greatest confidence value of the identified obstacle 802, a blocked spray nozzle having a 90 percent confidence of identification (in contrast to lower confidence comparisons, such as spray drift with a 60 percent confidence). The indexing module 316 is configured to index one or more of the location, vector or the like of the obstacle. For instance, in one example, including one or more of the wheels, spray nozzles or the like, a location of the spray nozzle, a location of the respective wheel or the like is indexed to the identified obstacle. In another example, the prioritizing module 318 shown in
After identification of one or more obstacles, the identified obstacle (or obstacles) are relayed to the autonomous agricultural system controller 104 including the vehicle operation module 306. The vehicle operation module 306 based on the priority, identification, indexing or the like of the one or more diagnostic obstacles 802-810 is configured to control the operation of the agricultural system 800. For instance, one or more of driving control, implement control or the like are controlled (e.g., maintained, modified, modulated or the like) by way of the vehicle operation module 306 according to the identification of the various diagnostic obstacles. In one example where a diagnostic obstacle 808 has a relatively high priority including, but not limited to, a deflected or damaged sprayer boom the vehicle operation module 306 triggers a halted operation of the agricultural system 800 or optionally modified operation, for instance, to bring the agricultural system 800 to a garage, service center or the like for service.
In another example, with the diagnostic obstacle 810 including sprayer drift or the diagnostic obstacle 802, such as a fouled or partially blocked sprayer 802, one or more potential modified operations of the agricultural system 800 are initiated or conducted with the vehicle operation module 306. For instance, with the diagnostic obstacle 810 including sprayer drift the vehicle operation module 306 modifies the operation of the implement such as the sprayer boom to accordingly change the spray droplet size at the affected nozzles suffering from the spray drift diagnostic obstacle 810. The increased droplet sizes are less prone to sprayer drift and accordingly the diagnostic obstacle 810 is addressed with modified operation of the system 800 while continuing the agricultural operation (spraying). In another example with the diagnostic obstacle 802, such as the fouled spray nozzle, blocked spray nozzle or the like, the vehicle operation module 306 conducts modified operations, by compensating for the fouled or blocked spray nozzle with increased application rates through one or more unblocked or unfouled spray nozzles proximate to the blocked or fouled spray nozzle.
As shown in
Another example of a diagnostic obstacle 904 is also shown in
Another diagnostic obstacle 906 example is shown in
In operation, the autonomous obstacle monitoring and vehicle control system 300, 350 shown in
As shown in
As the remote sensing device 114 conducts the diagnostic mission 920, the one or more sensors of the remote sensing device 114 observe, the agricultural system 900 and optionally the area proximate to the agricultural system 900. The observations are forwarded in real time (or upon docking of the remote sensing device 114 to a docking station 116) to the obstacle recognition module 310. As previously discussed in other examples, the obstacle recognition module 310 includes the obstacle comparator 312, identification module 314, indexing module 316 and prioritizing module 318 to identify and index one or more obstacles including the diagnostic obstacles 902, 904, 906 shown, for instance, in
Upon identification, including one or more of identification, indexing or prioritizing of obstacles, the identified obstacles are forwarded on to the vehicle operation module 306 of the autonomous agricultural system controller 104 shown in
In another example, for instance, with the diagnostic obstacle 904 including an obstruction provided along the implement, the vehicle operation module 306 attempts to conduct a modified operation. Depending on the success or failure of the modified operation the vehicle operation module 306 (or obstacle recognition module) may raise the severity and corresponding priority of the diagnostic obstacle in a manner that triggers a secondary hatted operation. In one example, the agricultural system 900 begins rearward movement according to modified operation provided with the module 306 in order to back the implement away from the obstruction. If upon repetition of the diagnostic mission 920 it is determined that the obstruction is no longer engaged with the implement, normal operation is resumed by the vehicle operation module 306. In another example, should the modified operation intended to remove the diagnostic obstacle 904 not succeed, for instance, debris remains lodged within the implement, the vehicle operation module 306 institutes a halted operation and optionally call for service to have the debris removed. In still other examples, the vehicle operation module 306 triggers an alternative form of modified operation including, for instance, guidance or navigation of the agricultural system 900 around the debris or other diagnostic obstacle 904 otherwise frustrating operation of the implement or frustrating travel of the system 900.
As shown in
In another example, the diagnostic obstacle 1004 includes a damaged or misaligned agricultural implement. As shown in dashed line in
Two other examples of diagnostic obstacles 1006, 1010 are also shown in
Another example of a diagnostic obstacle 1008 includes an issue with one or more of the ground engaging element similar to one or more of the previously described ground engaging elements (e.g., in
In operation, the autonomous obstacle monitoring and vehicle control system 300, 350 (e.g., shown in
As the remote sensing device 114 conducts the diagnostic mission 1020, the one or more sensors of the remote sensing device 114 (or in another example the remote sensing device 118 shown in
The obstacle recognition module 310 includes one or more of an obstacle comparator 312, identification module 314, indexing module 316 and prioritizing module 318 configured to analyze observations including, but not limited to, visual (image or video), ultrasound, radar, ground penetrating radar, LIDAR, infrared, thermographic, spectrometric, RGB (red-green-blue), hyperspectral, or chemical observations. The obstacle recognition module is configured to conduct one or more of identification, indexing or prioritizing of one or more of the diagnostic obstacles 1002, 1004, 1006, 1008, 1010.
As with previous examples, the identified obstacles are relayed to the autonomous agricultural system controller 104 including a vehicle operation module 306. Depending on one or more of the priority, identification or indexing of an identified diagnostic obstacle, the vehicle operational module 306 conducts operations including, but not limited to, halted operation, modified operation (e.g., to attempt to address the diagnostic issue, compensate for the issue, call for service while conducting additional agricultural operations or the like). Another operation mode implemented with the vehicle operation module 306 includes normal operation, for instance, if a diagnostic obstacle is considered noncritical, such as a partially empty seed bin. Optionally, even with normal operation the system 300 (or 350) calls for service to add additional seed to the hoppers or seed bins or schedules driving to a loading zone to add seed.
Other diagnostic obstacles such as the diagnostic obstacle 1004, 1006, 1008 or 1010 are, in some examples, given a higher priority and accordingly prompt halted operation of the agricultural system 1000 to facilitate service of the vehicle or modified operation to navigate the agricultural system 1000 to a service location to facilitate servicing of the one or more issues. In another example, the diagnostic obstacle 1010 such as an obstruction prompts modified operation, for instance, to hack away from the obstruction and facilitate navigation of the agricultural system 1000 around the obstruction followed by continued normal operation, for instance, along an updated or modified path based on the original determined path while including navigation around the obstacle.
As further shown in
One or more example obstacles 1102, 1104 are shown in
In one example, the first and second obstacles 1102 (e.g., crop characteristic) corresponds to nitrogen content or another crop characteristic associated with the crops that are an indication of crop health. In another example, the first and second obstacle 1102, 1104 correspond to water content in the crop, in the soil or one or more other characteristics associated with the crop or the underlying soil. In yet another example, the first and second obstacles 1102, 1104 corresponds to other crop characteristics including foliage or canopy density, foliage or canopy color, crop height or the like.
In one example, the first and second obstacles 1102, 1104 correspond with variations in nitrogen content that are identified as distinct obstacles, related obstacles having different values or the like. As described herein, the obstacle recognition module 310 is configured to identify the first and second obstacles 1102, 1104 (in this example nitrogen content levels and index the obstacles to locations in the field.
In one example, the remote sensing device 114 such as a drone, articuable arm or the like includes one or more sensors including, but not limited to, a normalized difference vegetation index camera (NDVI) that calculates visible and near-infrared light reflected by vegetation, a multi-spectral camera or hyper-spectral camera configured to sense the difference in nitrogen content or other crop characteristics, for instance, through one or more differentiation of colors, differentiation of electromagnetic waves (outside of visible light), or the like. In this example, the upper portion of the field, for instance, corresponding to the first obstacle 1102 having less dense crop coverage indicates a lower concentration of nitrogen content therein while the lower portion corresponding to the second obstacle 1104 has a higher crop density and, in this example, a higher nitrogen content.
The obstacles 1102, 1104, in this example, correspond to variations in crop characteristics and are identified and indexed with the obstacle recognition module 310. The identified obstacles 1102, 1104 are relayed to the vehicle operation module 306 to control the application of one or more agricultural products to the zones of the field with the obstacles 1102, 1104 (variations in one or more characteristics). For instance, in the zone of the field having a higher nitrogen content corresponding to the second obstacle 1104 the vehicle operation module 306 initiates the application of a lesser quantity or concentration of agricultural product from one or more sprayer nozzles, spreaders or the like overlying the portion of the field with the second obstacle 1104. Conversely, the first obstacle 1102, corresponding to a lower nitrogen content in that portion of the field, prompts the vehicle operation module 306 to increase the quantity or concentration of agricultural product applied through spray nozzles, spreaders or the like overlying the portion of the field with the first obstacle 1102.
In operation, the remote sensing device 114 is deployed to conduct the scouting mission 1120, for instance, according to a scouting route 1122 provided by the mission administration module 304 shown in
The observations of the remote sensing device 114 are interpreted with the obstacle recognition module 310 including, but not limited to, the obstacle comparator 312, identification module 314, the indexing module 316 and the prioritizing module 318. In one example, the obstacle comparator 312 compares one or more archived characteristics, for instance, nitrogen content, water content, foliage density or color, crop height, reflectivity of visible and near-infrared light or other crop characteristics or the like with the observations made with the remote sensing device 114. Through this comparison, one or more of the first or second obstacles 1102, 1104 (and potentially graduated versions of the obstacles corresponding to varying characteristic levels) are identified and indexed to the corresponding portions of the field. For instance, as shown in
The identified and indexed obstacles, in this example the crop characteristics 1102, 1104, are relayed to one or more other components of the systems 300, 350 including, for instance, the vehicle operation module 306 of the autonomous agricultural system controller 104. The vehicle operation module 306 controls operation of the agricultural system 1100, including one or more implements associated with the agricultural system 1100. For instance, the agricultural system 1100 including an agricultural sprayer includes one or more sprayer booms extending from the vehicle and having a plurality of sprayer nozzles there along. In one example, the vehicle operation module 306 controls operation such as the concentration of agricultural product, volume of agricultural product or the like applied through associated sprayer nozzles overlying portions of the field having the identified and indexed obstacles 1102 or 1104. For instance, for sprayer nozzles traveling over the portion of the field having the identified and indexed obstacle 1102 and corresponding to a lower nitrogen content, a higher volume or concentration of the agricultural product such as a fertilizer is applied through the overlying spray nozzles. Conversely, the sprayer nozzles traveling over the portions of the field having the second crop characteristic 1104 corresponding to a higher nitrogen content apply a lower concentration or quantity of the agricultural product based on control provided with the vehicle operation module 306.
In still other examples, the obstacles 1102, 1104 correspond to water content, density of crop foliage, crop height, indicators of crop health, composite crop characteristics based observations from multiple sensor types or the like. The agricultural system 1100, for instance, including the autonomous obstacle monitoring vehicle control system 300 (or 350) is configured to control the operation of the associated implement according identified obstacles and the associated crop or soil characteristics.
As further shown in
In an example, the agricultural system 1200 such as a sprayer, spreader or the like carries one or more agricultural products such as herbicides, pesticides or the like that are administered from spray nozzles, spreading wheels or the like. These implements are controlled based on identification and indexing of the diagnostic obstacles 1202-1210 including, but not limited to, weeds, pests or the like (as a type of obstacle). The obstacle recognition module 310 and the vehicle operation module 306 of the systems 300 or 350 control the operation of the various implements to treat these diagnostic obstacles.
For instance, in the portion of the field having the diagnostic obstacle 1204 corresponding to a lower weed density (relative to the zone having the diagnostic obstacle 1202), a lower concentration, flow rate of agricultural product or quantity of granular product is applied. Conversely, in the portions of the field having the identified and indexed diagnostic obstacle 1202 a relatively higher concentration, flow rate of agricultural product or quantity of granular agricultural product is applied. Control of application is conducted by the vehicle operation module 306 based on the identified obstacles 1202, 1204, Accordingly, as the agricultural system 1200 or one or more applicators (e.g., spray nozzles, spreader wheels, or the like) enter zones indexed with the obstacles 1202, 1204 the vehicle operation module 306 controls operation of the applicators based on the obstacles. In one example, the concentration or flow rate of a liquid agricultural product is increased. by a specified quantity such as 10 percent or more to increase the mortality rate for the identified weed, pests, higher identified density of the same or the like(e.g., 1202 in contrast to 1204). In another example, the identification and indexing of weeds or density of weeds as the diagnostic obstacles 1202, 1204 is relayed to the vehicle operation module 306 of a cultivator. The vehicle operation module 306 selectively operates the cultivator shovels depending on their locations within the zones corresponding to the diagnostic obstacles 1202, 1204. For instance, in the zone of the diagnostic obstacle 1202 with higher weed density the vehicle operation module 306 operates the cultivator shovels more aggressively by cultivating over greater lengths, in close proximity to crop rows or the like).
In another example, the obstacle recognition module 310 is configured to identify and index pests, damage caused by pests, weeds, weed density or the like as the obstacles 1202, 1204, 1206, 1208, 1210 and discriminate between various pests, weeds, associated damage or the like. Archived characteristics for varied pests, weeds, damage caused to crops by pests or the like are included with or available to the obstacle recognition module 310 to directly or indirectly identify and index pests or weeds for instance, by shape of pests or weeds, color of pests or weeds, shape or color of damage to crops or the like. The obstacle recognition module 310 including the obstacle comparator 312, identification module 314 or the like are configured to compare the archived characteristics with associated characteristics observed with the remote sensing device 114.
The refined identification of obstacles (based on pest type, weed type or the like) facilitates varied husbandry. For instance, the vehicle operation module 306 is configured to modulate the agricultural product composition based on the identified obstacles. In one example, the vehicle operation module 306 prescribes a first composition of agricultural product concentration, constituents or the like) to address a first diagnostic obstacle (e.g., pest type, weed type or the like) and prescribes a second composition of agricultural product different from the first to address a second diagnostic obstacle corresponding a different pest, weed or the like. Variation in husbandry control is available for a spreader or cultivator (e.g., moving the shovels to different depths for different weed) with the systems 300, 350 described herein.
In another example, the remote sensing device 114 and obstacle recognition module 310 identify and index other plants, for instance, a different crop in a proximate zone of a field, such as wheat in a zone adjacent to corn. The proximate crop is identified as an obstacle, and the vehicle operation module 306 arrests application of an agricultural product such as a pesticide, herbicide or the like that is potentially harmful to the adjacent crop. In this example, the adjacent crop is identified as a diagnostic obstacle relative to the treatment provided by the agricultural system 1200, and accordingly the vehicle operation module 306 ceases application of the product until the agricultural system 1200 (or one or more of its applicators) is outside of the zone of the adjacent crop.
In still other examples, the autonomous obstacle monitoring and vehicle control system 300, 350 when with or without. the appropriate agricultural product, implement or the like optionally is configured to index obstacles (e.g., 1202-1210) or provide an alert with the indexed obstacles that is delivered to a user or logged for eventual review by a user that indicates the identified and indexed weed, pest or the like including the location within the field. At a future time, an agricultural system having the associated agricultural product, implement or the like configured to address one or more of the identified pest, weed or the like is readily dispatched to address the diagnostic obstacle 1202-1210. For instance, as the agricultural system arrives at the location of the previously identified and indexed obstacle 1202-1210 the appropriate husbandry operation is automatically conducted (e.g., with one of the systems 300, 350 having a vehicle operation module 306.
The obstacle recognition module 310, for instance, of the systems 300, 350 described herein identifies and indexes the soils observed with the remote sensing device 114. In one example, the obstacle recognition module 310 identifies and indexes zones in the field having varying alkalinities corresponding to the obstacles 1302, 1304, 1306 (with 1306 having the greatest alkalinity). In one example, the agricultural system 1300 includes the vehicle operating module 306, and controls the application rate of lime or another soil husbandry product based on the identified and indexed obstacles (alkalinities) in the field. For instance, in a portion of the field having the diagnostic obstacle 1306 corresponding to a relatively high alkalinity, a higher quantity of lime is applied. In the second zone, corresponding to the diagnostic obstacle 1304, a lower alkalinity (but greater than obstacle 1302) is identified and a moderate or lesser quantity of lime is applied with the system 1300. Conversely, in the portion of the field having the diagnostic obstacle 1302 corresponding to a relatively low alkalinity, a small quantity or zero quantity of lime is applied, for instance, with the agricultural system 1300 (e.g., a spreader).
In a similar manner, soil identification and indexing as described herein are conducted to assess nitrogen content in the soil to control fertilizer application, for instance, with a sprayer, cultivator or the like. In other examples, soil characteristics are identified and indexed with sensors associated with the remote sensing device 114 (or 118) and the obstacle recognition module 310 to differentiate between sandy, clay, black (good) top soil or the like to facilitate control of the agricultural system 1300 or a forthcoming agricultural system for independent and discrete husbandry of the field and differentiated zones of the field, for instance, corresponding to the diagnostic obstacles 1302, 1304, 1306.
At 1402 an obstacle monitoring mission is conducted with a remote sensing device, for instance the remote sensing device 114 or 118 shown in
At 1408 the method 1400 includes recognizing the one or more obstacles observed with the remote sensing device. Recognizing includes 1410 comparing the one or more obstacles with archived characteristics of archived obstacles. For example, with an image of the detected obstacle pixels, arrays of pixels, coloring or the like are compared with archived characteristics. At 1412, the one or more obstacles are identified based on the comparison between the obstacles and archived characteristics of archived obstacles. Optionally, at 1414 the method 1400 includes indexing one or more of locations or vectors of the one or more identified obstacles. For instance, a static obstacle (e.g., such as its virtual representation or indication) is indexed on field map, with coordinates relative to a coordinate system or coordinate system origin or the like. In another example, an obstacle (e.g., representation or indication) is indexed with a vector, for instance extending from its present location and having magnitude and directly corresponding to velocity, acceleration or the like.
At 1416 the agricultural vehicle is operated based on the identifying and indexing of the one or more identified. obstacles. For example, an obstacle within a planned path or proximate to the path of the vehicle prompts at least one of navigation modification around the obstacle (modified operation), halted operation of the vehicle (e.g., if the obstacle is impassable or has a sufficiently high priority that (riggers halting), or normal operation (e.g., if the obstacle has a low priority, is passable by an overhead sprayer boom or the like). In another example, an identified obstacle including a diagnostic obstacle prompts operation of the agricultural vehicle in a manner based on identifying of the identified obstacle. For instance, a faulty bearing (e.g., having a detectable thermal characteristic at the vehicle wheel) is considered a diagnostic obstacle that, may trigger halted operation or modified operation (e.g., immediate return to a base location for service). In one example, a greater thermal signature prompts halted operation while a comparatively lesser thermal signature (corresponding to failing as opposed to failed bearing) prompts modified operation such as decreased operating speeds, finishing of a field zone and return to a maintenance location or the like.
Several options for the method 1400 follow. In one example, the method 1400 includes selecting an obstacle monitoring mission from a mission database including a plurality of missions and respective mission routes. For instance, the plurality of missions and respective mission routes include an inspection mission having an inspection route proximate to the agricultural vehicle (e.g., around the vehicle, to view one or more components necessary for an operation or the like). A scout mission is another example missing having a scouting route along a determined path (including predetermined, or real time determined path) of the agricultural vehicle. For instance, the remote sensing device observes the field along the determined path, and identifies and indexes forthcoming obstacles (unharvested crops, livestock, humans, fallen trees, water or the like), absence of obstacles (harvested and ‘open’ field or the like) to facilitate operation of the agricultural vehicle including autonomous driving and implement operations. A diagnostic mission is another example mission having a diagnostic route proximate to the agricultural vehicle. Optionally, the remote sensing device is deployed and operated proximate to the agricultural vehicle to assess one or more potential issues with the vehicle including the implement or the vehicle itself. For instance, one or more of the vehicle components are diagnosed as running outside of specified parameters (including failing to run), and the autonomous obstacle monitoring and vehicle control system described herein deploys the remote sensing device to observe the vehicle component, and the system identifies the component through comparison with archived characteristics of the obstacle (e.g., in this example an archived component) and facilitates identification of an issue with the component for instance with comparison of the component, such as its heat signature, accumulated debris around the component or the like. The diagnostic mission is optionally conducted while the vehicle is stationary, conducting operations in a field or traveling between a field and a starting location (e.g., maintenance site, garage or the like).
In another example, the method 1400 includes a path module configured to determine a path of travel for the agricultural vehicle. For instance, the path module determines a proposed path for the agricultural vehicle. The proposed path is modified to an updated path based on the identification and indexing of the one or more identified obstacles.
In yet another example, Identifying the one or more obstacles as identified obstacles includes one or both of identifying field obstacles or identifying diagnostic obstacles. In one example, identifying the one or more obstacles as identified (field) obstacles includes identifying one or more of debris, field washouts, sink holes, water, saturated ground, humans, livestock, animals, fences, damaged fences, open gates, fallen trees, accumulated brush, harvested crops, unharvested crops, vehicles or rocks. In another example, identifying the one or more obstacles as identified (diagnostic) obstacles includes identifying a full grain bin, failed component, failing component, damaged component, trapped debris, failed implement, failing implement, damaged implement, fouled spray nozzle, or agricultural product drift.
Optionally, operating the agricultural vehicle based on the identification and indexing of the one or more identified obstacles includes prioritizing the one or more identified obstacles based on one or more of the identifying or indexing. For instance, an identified human is in one example prioritized higher than livestock or a water or mud zone. In another example, an identified vehicle that is indexed with a vector extending away from the vector (path) of the agricultural vehicle is prioritized lower than an identified vehicle having a vector intersecting with the vector of the agricultural vehicle. Operating the agricultural vehicle includes autonomously controlling the agricultural vehicle based on the prioritizing of the one or more identified obstacles.
In another example, prioritizing includes associating one of a halt operation, modified operation or normal operation indication with the identified obstacles based on one or more of the identifying or indexing. Optionally, autonomously controlling the agricultural vehicle based on the prioritizing (discussed herein) includes halting operation for a halt operation indication, modifying operation for a modified operation indication or conducting normal operation with the agricultural vehicle for a normal operation indication.
In still another example the method 1400 includes repeating recognizing of the one or more obstacles, and repeating operating the agricultural vehicle based on the repeated identifying and indexing of the one or more identified obstacles. Optionally, repeated recognition includes updated identification and indexing of obstacles to facilitate operation of the agricultural vehicle with the updated identified and indexed obstacles. For example, obstacle identification and indexing are refined to facilitate refined operation of the agricultural vehicle (e.g., with enhances obstacle identification, indexing or the like).
As discussed herein, the remote sensing device includes a drone in one example, and conducting the obstacle monitoring mission with the remote sensing device includes deploying the drone from a docking station for moving along the mission route and observing the one or more obstacles.
Aspect 1 can include subject matter such as an autonomous obstacle monitoring and vehicle control system comprising: a remote sensing device including one or more sensors configured to observe obstacles proximate to a path of an agricultural system or proximate to the agricultural system, wherein the remote sensing device is movable relative to the agricultural system; an obstacle recognition module in communication with the remote sensing device, the obstacle recognition module configured to identify and index the obstacles proximate to the path or the agricultural system; and an autonomous agricultural system controller configured for communication with the agricultural system, the autonomous agricultural system controller includes: a path module configured to determine a path of travel for the agricultural system; a mission administration module configured to operate the remote sensing device along one or more mission routes for observation of the obstacles proximate to the one or more mission routes; and a vehicle operation module configured to control the agricultural system based on the determined path and identified and indexed obstacles.
Aspect 2 can include, or can optionally be combined with the subject matter of Aspect 1, to optionally include wherein the remote sensing device includes a drone.
Aspect 3 can include, or can optionally be combined with the subject matter of one or any combination of Aspects 1 or 2 to optionally include a drone docking station, the drone docking station is configured for coupling with the agricultural system, and the drone docking station includes a power and data interface configured to couple with the drone in a docked configuration.
Aspect 4 can include, or can optionally be combined with the subject matter of one or any combination of Aspects 1-3 to optionally include wherein the drone includes: a data transceiver; and a global positioning system receiver.
Aspect 5 can include, or can optionally be combined with the subject matter of one or any combination of Aspects 1-4 to optionally include wherein the remote sensing devices includes one or more of a boom or articulating arm movable relative to the agricultural system.
Aspect 6 can include, or can optionally be combined with the subject matter of Aspects 1-5 to optionally include wherein the one or more sensors include one or more of chemical sensing, optical, video, spectrometric, RGB (red-green-blue), thermographic, hyperspectral, ground penetrating radar, radar, LIDAR or ultrasound sensors.
Aspect 7 can include, or can optionally be combined with the subject matter of Aspects 1-6 to optionally include wherein the obstacle recognition module includes a prioritizing module configured to prioritize obstacles according to one or more of the identification or indexing.
Aspect 8 can include, or can optionally be combined with the subject matter of Aspects 1-7 to optionally include wherein obstacles include one or more of field obstacles or diagnostic obstacles, and the obstacle recognition module is configured to identify one or more of field obstacles or diagnostic obstacles.
Aspect 9 can include, or can optionally be combined with the subject matter of Aspects 1-8 to optionally include wherein field obstacles include one or more of debris, field washouts, sink holes, water, saturated ground, humans, livestock, animals, fences, damaged fences, open gates, fallen trees, harvested crop zones, unharvested crops, vehicles or rocks.
Aspect 10 can include, or can optionally he combined with the subject matter of Aspects 1-9 to optionally include wherein diagnostic obstacles include one or more of a full grain bin, failed component, failing component, damaged component, trapped debris, failed implement, failing implement, damaged implement, fouled spray nozzle, or agricultural product drift.
Aspect 11 can include, or can optionally he combined with the subject matter of Aspects 1-10 to optionally include wherein the autonomous agricultural system controller includes a mission database having the one or more missions, each of the one or more missions having a respective mission route of the one or more mission routes.
Aspect 12 can include, or can optionally be combined with the subject matter of Aspects 1-11 to optionally include wherein the mission database includes one or more of an inspection mission, scout mission or diagnostic mission.
Aspect 13 can include, or can optionally he combined with the subject matter of Aspects 1-12 to optionally include wherein the mission database includes an inspection mission having an associated inspection route for the remote sensing device proximate to the agricultural system.
Aspect 14 can include, or can optionally be combined with the subject matter of Aspects 1-13 to optionally include wherein the mission database includes a scout mission having an associated scouting route for the remote sensing device proximate to the determined path.
Aspect 15 can include, or can optionally be combined with the subject matter of Aspects 1-14 to optionally include wherein the mission database includes a diagnostic mission having an associated diagnostic route for the remote sensing device proximate to the agricultural system.
Aspect 16 can include, or can optionally be combined with the subject matter of Aspects 1-15 to optionally include wherein the vehicle operation module is configured to control steering, throttle and braking of the agricultural system.
Aspect 17 can include, or can optionally be combined with the subject matter of Aspects 1-16 to optionally include the agricultural system.
Aspect 18 can include, or can optionally be combined with the subject matter of Aspects 1-17 to optionally include wherein the agricultural system includes one or more of the agricultural system, agricultural implement or towed agricultural implement.
Aspect 19 can include, or can optionally be combined with the subject matter of Aspects 1-18 to optionally include an autonomous obstacle monitoring and vehicle control system comprising: a remote sensing device including one or more sensors configured to observe obstacles, wherein the remote sensing device is movable relative to an agricultural system; an obstacle recognition module in communication with the remote sensing device, the obstacle recognition module is configured to identify obstacles observed with the remote sensing device; an autonomous agricultural system controller in communication with the obstacle recognition module and the remote sensing device, the autonomous agricultural system controller includes: a mission database including one or more missions, each mission having a respective mission route; a mission administration module configured to operate the remote sensing device along the respective mission route associated with the one or more missions for observation of the obstacles proximate to the respective mission route; a vehicle operation module configured to control the agricultural system based on the identified obstacles.
Aspect 20 can include, or can optionally be combined with the subject matter of Aspects 1-19 to optionally include wherein the one or more missions of the mission database include one or more of an inspection mission, scout mission or diagnostic mission.
Aspect 21 can include, or can optionally be combined with the subject matter of Aspects 1-20 to optionally include wherein the mission database includes an inspection mission having an associated inspection route for the remote sensing device proximate to the agricultural system.
Aspect 22 can include, or can optionally be combined with the subject matter of Aspects 1-21 to optionally include wherein the mission database includes a scout, mission having an associated scouting route for the remote sensing device along a determined path of the agricultural system.
Aspect 23 can include, or can optionally be combined with the subject matter of Aspects 1-22 to optionally include wherein the mission database includes a diagnostic mission having an associated diagnostic route for the remote sensing device proximate to the agricultural system.
Aspect 24 can include, or can optionally be combined with the subject matter of Aspects 1-23 to optionally include wherein the autonomous agricultural system controller includes a path module configured to determine a path of travel for the agricultural system.
Aspect 25 can include, or can optionally he combined with the subject matter of Aspects 1-24 to optionally include wherein the remote sensing device includes a drone.
Aspect 26 can include, or can optionally be combined with the subject matter of Aspects 1-25 to optionally include a drone docking station, the drone docking station is configured for coupling with the agricultural system, and the drone docking station includes a power and data interface configured to couple with the drone in a docked configuration.
Aspect 27 can include, or can optionally be combined with the subject matter of Aspects 1-26 to optionally include wherein the remote sensing devices includes one or more of a boom or articulating arm movable relative to the agricultural system, and the one or more sensors are coupled with the boom or articulating arm.
Aspect 28 can include, or can optionally be combined with the subject matter of Aspects 1-27 to optionally include wherein the obstacle recognition module includes: an obstacle comparator configured to compare obstacles observed with the remote sensing device with archived characteristics of one or more archived obstacles; an identification module configured to identify the observed obstacles as identified obstacles based on the comparison; and an indexing module configured to index one or more of location or vector of the identified obstacles.
Aspect 29 can include, or can optionally be combined with the subject matter of Aspects 1-28 to optionally include wherein the observed obstacles include one or more sensed characteristics, and the obstacle comparator is configured to compare the sensed characteristics of the observed obstacle with the archived. characteristics of the one or more archived obstacles.
Aspect 30 can include, or can optionally be combined with the subject matter of Aspects 1-29 to optionally include wherein the identification module is configured to assign one or more of an identification marker or probability to the identified obstacle.
Aspect 31 can include, or can optionally be combined with the subject matter of Aspects 1-30 to optionally include wherein the obstacle recognition module includes a prioritizing module configured to prioritize the identified obstacles according to one or more of the respective identification marker, probability or indexing assigned to each of the identified obstacles.
Aspect 32 can include, or can optionally be combined with the subject matter of Aspects 1-31 to optionally include wherein the vehicle operation module is configured to control the agricultural system in normal operation, modified operation or halted operation modes based on the prioritizing of the identified obstacles.
Aspect 33 can include, or can optionally be combined with the subject matter of Aspects 1-32 to optionally include wherein the vehicle operation module is configured to control steering, throttle and braking of the agricultural system.
Aspect 34 can include, or can optionally be combined with the subject matter of Aspects 1-33 to optionally include the agricultural system.
Aspect 35 can include, or can optionally he combined with the subject matter of Aspects 1-34 to optionally include wherein the agricultural system includes one or more of the agricultural system, agricultural implement or towed agricultural implement.
Aspect 36 can include, or can optionally be combined with the subject matter of Aspects 1-35 to optionally include a method for autonomous obstacle monitoring and vehicle control comprising: conducting an obstacle monitoring mission with a remote sensing device, conducting the obstacle monitoring mission includes: moving the remote sensing device relative to an agricultural system along a mission route; and observing one or more obstacles with the remote sensing device proximate to the mission route; recognizing the one or more obstacles observed with the remote sensing device, recognizing includes: comparing the one or more obstacles with archived characteristics of archived obstacles; identifying the one or more obstacles as identified obstacles based on the comparison; and indexing one or more of locations or vectors of the one or more identified obstacles; and operating the agricultural system based on the identifying and indexing of the one or more identified obstacles.
Aspect 37 can include, or can optionally be combined with the subject matter of Aspects 1-36 to optionally include selecting an obstacle monitoring mission from a mission database including a plurality of missions and respective mission routes.
Aspect 38 can include, or can optionally be combined with the subject matter of Aspects 1-37 to optionally include wherein the plurality of missions and respective mission routes include: an inspection mission having an inspection route proximate to the agricultural system; a scout mission having a scouting route along a determined path of the agricultural system; and a diagnostic mission having a diagnostic route proximate to the agricultural system.
Aspect 39 can include, or can optionally be combined with the subject matter of Aspects 1-38 to optionally include a path module configured to determine a path of travel for the agricultural system including: determining a proposed path for the agricultural system; and modifying the proposed path to an updated path based on the identification and indexing of the one or more identified obstacles.
Aspect 40 can include, or can optionally be combined with the subject matter of Aspects 1-39 to optionally include wherein identifying the one or more obstacles as identified obstacles includes identifying field obstacles or diagnostic obstacles.
Aspect 41 can include, or can optionally be combined with the subject matter of Aspects 1-40 to optionally include wherein identifying the one or more obstacles as identified obstacles includes identifying one or more of debris, field washouts, sink holes, water, saturated ground, humans, livestock, animals, fences, damaged fences, open gates, fallen trees, accumulated brush, harvested crop zones, unharvested crops, vehicles or rocks.
Aspect 42 can include, or can optionally be combined with the subject matter of Aspects 1-41 to optionally include wherein identifying the one or more obstacles as identified obstacles includes identifying a full grain bin, failed component, failing component, damaged component, trapped debris, failed implement, failing implement, damaged implement, fouled spray nozzle, or agricultural product drift.
Aspect 43 can include, or can optionally be combined with the subject matter of Aspects 1-42 to optionally include wherein operating the agricultural system based on the identification and indexing of the one or more identified obstacles includes: prioritizing the one or more identified obstacles based on one or more of the identifying or indexing; and autonomously controlling the agricultural system based on the prioritizing of the one or more identified obstacles.
Aspect 44 can include, or can optionally be combined with the subject matter of Aspects 1-43 to optionally include wherein prioritizing includes associating one of a halt operation, modified operation or normal operation indication with the identified obstacles based on one or more of the identifying or indexing; and autonomously controlling the agricultural system based on the prioritizing includes halting operation for a halt operation indication, modifying operation for a modified operation indication or conducting normal operation with the agricultural system for a normal operation indication.
Aspect 45 can include, or can optionally be combined with the subject matter of Aspects 1-44 to optionally include repeating recognizing of the one or more obstacles, and repeating operating the agricultural system based on the repeated identifying and indexing of the one or more identified obstacles.
Aspect 46 can include, or can optionally be combined with the subject matter of Aspects 1-45 to optionally include wherein the remote sensing device includes a drone, and conducting the obstacle monitoring mission with the remote sensing device includes: deploying the drone from a docking station for moving along the mission route and observing the one or more obstacles.
Each of these non-limiting aspects can stand on its own, or can be combined in various permutations or combinations with one or more of the other aspects.
The above description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show, by way of illustration, specific embodiments in which the invention can be practiced. These embodiments are also referred to herein as “examples.” Such examples can include elements in addition to those shown or described. However, the present inventors also contemplate examples in which only those elements shown or described are provided. Moreover, the present inventors also contemplate examples using any combination or permutation of those elements shown or described (or one or more aspects thereof), either with respect to a particular example (or one or more aspects thereof), or with respect to other examples (or one or more aspects thereof) shown or described herein.
In the event of inconsistent usages between this document and any documents so incorporated by reference, the usage in this document controls.
In this document, the terms “a” or “an” are used, as is common in patent documents, to include one or more than one, independent of any other instances or usages of “at least one” or “one or more.” In this document, the term “or” is used to refer to a nonexclusive or, such that “A or B” includes “A but not B,” “B but not A,” and “A and B,” unless otherwise indicated. In this document, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein,” Also, in the following claims, the terms “including” and “comprising” are open-ended, that is, a system, device, article, composition, formulation, or process that includes elements in addition to those listed after such a term in a claim are still deemed to fall within the scope of that claim. Moreover, in the following claims, the terms “first,” “second,” and “third,” etc. are used merely as labels, and are not intended to impose numerical requirements on their objects.
Geometric terms, such as “parallel”, “perpendicular”, “round”, or “square”, are not intended to require absolute mathematical precision, unless the context indicates otherwise. Instead, such geometric terms allow for variations due to manufacturing or equivalent functions. For example, if an element is described as “round” or “generally round,” a component that is not precisely circular (e.g., one that is slightly oblong or is a many-sided polygon) is still encompassed by this description.
Method examples described herein can be machine or computer-implemented at least in part. Some examples can include a computer-readable medium or machine-readable medium encoded with instructions operable to configure an electronic device to perform methods as described in the above examples. An implementation of such methods can include code, such as microcode, assembly language code, a higher-level language code, or the like. Such code can include computer readable instructions for performing various methods. The code may form portions of computer program products. Further, in an example, the code can be tangibly stored on one or more volatile, non-transitory, or non-volatile tangible computer-readable media, such as during execution or at other times. Examples of these tangible computer-readable media can include, but are not limited to, hard disks, removable magnetic disks, removable optical disks (e.g., compact disks and digital video disks), magnetic cassettes, memory cards or sticks, random access memories (RAMS), read only memories (ROMs), and the like.
The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more aspects thereof) may be used in combination with each other. Other embodiments can be used, such as by one of ordinary skill in the art upon reviewing the above description. The. Abstract is provided to comply with 37 C.F.R. § 1.72(b), to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Also, in the above Detailed Description, various features may be grouped together to streamline the disclosure. This should not be interpreted as intending that an unclaimed disclosed feature is essential to any claim, Rather, inventive subject matter may lie in less than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the Detailed Description as examples or embodiments, with each claim standing on its own as a separate embodiment, and it is contemplated that such embodiments can be combined with each other in various combinations or permutations. The scope of the invention should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.
This application claims the benefit of priority to U.S. Patent Application Ser. No. 63/024,979, filed May 14, 2020, which application is incorporated by reference herein in its entirety.
| Number | Date | Country | |
|---|---|---|---|
| 63024979 | May 2020 | US |
