The present application claims benefit of Nexus Robotics Inc., Canadian Patent Application Serial No. ______, filed on Jan. 27, 2022, (Robic, LLP Ref. No. 020662-0008), and entitled “Weed Picking Module and Method.” The entire contents of this application are incorporated herein by reference.
The technical field generally relates to picking implements for use on agricultural vehicles, and more specifically to modules and methods of optimizing weed or crop picking and removal with automated implements.
Weeds and plants compete with crops for resources, including water, nutrients, and sunlight. Picking of weeds from agricultural fields is a continuous process which improves crop yield by removing competition for resources. Weed picking is a labour-intensive process and there is a desire for automation, such as by use of an autonomous weeding vehicle. A key performance indicator of an autonomous weeding vehicle is its weed picking rate. There is naturally a desire for methods and devices which improve weed picking rates of autonomous weeding vehicles.
In accordance with one aspect, there is provided a method for picking weeds with an implement provided on an agricultural vehicle. The method comprises steps to maximize a number of weeds to be picked in a single take. The method may also comprises steps to maximize a number of immature crop to be picked in a single take, to thin the field, i.e. to selectively pick young crop to allow others to grow. The method may comprise steps of capturing images of a field traveled by the agricultural vehicle, the field comprising weeds and crops. The method may comprise a step of generating a map from the captured images, comprising coordinates of the weeds and crops. The method may comprise determining, based on the coordinates of the weeds and crops, at least one picking location that maximizes a number of weeds to be picked in a single take. The method comprises determining one offset for the implement to follow for arriving at the optimal location. The method comprises moving the implement along the offset; and picking weeds at the optimal location with the implement. When the method is used for thinning instead of, or in addition to weeding, the offset is determined based on the location of immature or selected crops to be removed in a single take.
In accordance with one embodiment, the implement avoids contact with the crops as it picks the weeds or selected immature crops.
In accordance with another embodiment, moving the implement comprises a combination of rotational and translational movements.
In accordance with another embodiment, determining the offset comprises first selecting an anchor weed which has to be picked. In other possible implementations, the anchor is an immature or undesired crop that is to be removed to provide more room for other crops.
In accordance with another embodiment, the anchor weed is selected based on proximity to the implement.
In accordance with another embodiment, the anchor weed is selected based on the maximum number of weeds that may be picked in the single take while avoiding contact with the crops.
In accordance with another embodiment, avoiding contact with the crops comprises maintaining a minimum lateral distance between the implement and the crops.
In accordance with another embodiment, determining the offset comprises a step of forming clusters of weeds comprising the anchor weed.
In accordance with another embodiment, the picking location is determined using a first loss function which is optimized by at least one of maximizing the number of weeds to be picked and avoiding the crops within a given space.
In accordance with another embodiment, the offset is determined using a second loss function which is optimized to center the weeds to be picked relative to a center of the implement.
In accordance with another embodiment, inputs to the second loss function comprise the coordinates of the anchor weed, the respective coordinates of additional weeds, and the coordinates of any crops within a footprint of the implement; determining the offset comprises optimizing the second loss function such that the footprint of the implement encompasses the coordinates of the anchor weed and at least one of the coordinates of the additional weeds while excluding the coordinates of any crops; and output of the second loss function comprises the offset to apply to a current position of the implement and an identification of the weeds picked.
In accordance with another embodiment, the step of moving the implement along the offset comprises first applying the offset to a current position of the implement.
In accordance with another embodiment, the offset is further selected from a plurality of potential offsets based on at least one of time and implement travel distance required to pick all weeds at each picking location.
In accordance with another embodiment, the implement is a gripper comprising jaws and wherein the step of picking weeds comprises controlling the jaws of the gripper to pinch stems or roots of the weeds to pick them from the ground.
In accordance with another embodiment, the step of picking weeds comprises adjusting an opening width of the gripper based on the number of weeds to be picked.
In accordance with another embodiment, the images are captured using at least one camera on the agricultural vehicle.
In accordance with another aspect, a weed picking module provided on an agricultural vehicle, the module comprising: an agricultural implement configured and adapted for weed picking, the implement being translatable along three degrees of freedom; a controller configured to control the movement of the agricultural implement; and a processing module comprising: input ports for receiving coordinates of weeds and crops determined from images; and a non-transitory memory and a processor, the non-transitory memory storing the coordinates and instructions for causing the processor to generate an optimal location for the implement that maximizes a number of weeds to be gripped in a single take; and output ports for sending positioning data to the controller, the controller moving the implement in accordance with the optimal location and positioning the implement over the weeds to be gripped based on the positioning data, and controlling operation of the implement to pick the weeds.
In accordance with another embodiment, the implement is also rotatable along at least one degree of freedom.
In accordance with another embodiment, the agricultural implement is a gripper comprising jaws configured and adapted to pinch and retrieve stems or roots of the weeds.
In accordance with another embodiment, one of a delta robot or a gantry system operatively connected to the agricultural implement, for translating the implement along the three axes.
In accordance with another embodiment, the controller comprises at least one first actuator for controlling movement of the gripper and a second actuator for actuation of the gripper.
In accordance with another embodiment, the memory comprises instructions to form clusters of weeds comprising an anchor weed.
In accordance with another embodiment, the memory stores a loss function which is optimized so at to maximize at least one of: number of weeds; and distance of the implement to the crop.
In accordance with another embodiment, the instructions stored in memory cause the processor to: input coordinates of the anchor weed, respective coordinates of additional weeds, coordinates of a crop, and a footprint of the implement to the loss function; optimize the loss function such that the footprint of the implement encompasses the coordinates of the anchor weed and at least one of the coordinates of the additional weeds while excluding the coordinates of the crop; and output, from the loss function, an offset to apply to a current position of the implement and an identification of the weeds picked.
All of the embodiments above can be implemented for field thinning, instead of, or in addition to, weeding.
Other features of advantages of the present invention will be better understood upon reading example embodiments thereof, with reference to the appended drawings. While the invention will be described in conjunction with example embodiments, it will be understood that it is not intended to limit the scope of the invention to such embodiments. On the contrary, it is intended to cover all alternatives, modifications and equivalents as defined in the present application.
Broadly described, the picking module can be used on or with an agricultural vehicle, such as an autonomous weeding machine, capable of navigating farm fields while identifying weeds and pulling them using a controllable implement, such as a robot gripper. The module focuses on picking and removing weeds or other undesired plant growing close to the crop where other means of weeding are not effective and manual labor is employed. The weeds are picked by physically gripping them and pulling them from the earth and then discarding them on the ground.
With reference to
With reference to
The picking module 100 may comprise frame 102 onto which the grippers 210 and other parts can be mounted, as well as a processing module 400 comprising a processor 414 and non-transitory memory 412 for storing and carrying out computer instructions (see
The picking module 100 additionally may include cameras 220 mounted thereon for identifying weeds, extra or immature plants and crops in the agricultural field. The cameras 220 may be coupled to lights mounted in proximity thereof for illuminating the field of vision of the cameras 220, providing improved images. In the illustrated implementation as shown in
With the implementation shown, when the autonomous weeding vehicle 10 passes in an agricultural field, the picking module 100 is activated. Once the picking module 100 is activated, the cameras 220 begin capturing images of the field. In one implementation, the cameras 220 are coupled to a transceiver module for transmitting images to a remote site where an artificial intelligence (AI) system identifies weeds and crops. In one implementation, the images transmitted by the cameras 220 are interpreted by an operator at the remote site who marks crops and weeds, and also optionally immature crops that are to be removed for thinning. The cameras 220 may transmit the images through any wireless communication system to the remote site. In yet other implementations, the AI module and image analysis can be conducted locally, in a processing device provided on the vehicle and in communication with a controller controlling the weed picking implement.
According to the images generated and marked with respect to weeds and crops, and possibly other plants, such as immature crops, a map 500 may be generated identifying the coordinates of weeds 520 and crops 510 in the area captured by the images (see
In one implementation, the map 500 may instead, or in addition to the cameras of the weeding module 100, be provided by another image capture system such as a satellite image. As the vehicle travels through the agricultural field, the map 500 may be continuously updated with images captured by the cameras. Additionally, a speed of the vehicle may be aligned with the camera capture rate and the processor. The speed of the vehicle may be calibrated so that the vehicle moves through the field at a rate that allows for the cameras and the processor to capture and process images, and to then effect weed picking by the gripper 210.
The processing module 400 is configured to receive inputs, such as the map 500 or the weed and crop coordinates, process the inputs, and output data for controlling the gripper. For example, the map 500 or map data may include coordinates of weeds 520 and crops 510. The coordinates can be provided or accessed through input ports 410 to the memory 412 and the processor 414, the processor 414 generating an offset for the gripper 210 to move and rotate to, based on the received coordinates, and sending the offset information via output ports 416 to a controller for controlling the gripper 210. The controller controls the movement of the grippers 210 based on the coordinates of the weeds and crops. In one implementation, the controller comprises a first actuator for controlling movement of the gripper and a second actuator for actuation of the jaws (opening and closing) of the gripper. In one implementation, the at least first actuator comprises four actuators, each one corresponding to translation along the x, y and z axes, as well as rotation around the z axis. In the illustrated implementation, the weed picking module 100 is installed underneath the autonomous weeding vehicle 10. In one implementation, the weed picking module 100 may be installed in a front, a side, or a rear portion of the vehicle 10 for carrying out weed picking. In the illustrated implementation, the grippers 210 of the weed picking module 100 are moveable via three mechanical arms which allow for translational motion of the grippers along three degrees of freedom, also known as a delta robot configuration. The grippers 210 are further rotatable along a vertical axis of the autonomous weeding vehicle 10. The grippers 210 thus have four degrees of freedom, three translational and one rotational. Alternatively, the grippers 210 may have fewer or greater degrees of freedom depending on design requirements, or they may use a movement configuration different to a delta robot configuration. For example, they may instead use a gantry system whereby the gripper is translatable along three axes. In preferred implementations, the gripper is translated along 3 degrees of freedom (DOF: x, y, z) and rotated about the z axis, allowing controlling the gripper with 4 DOF. While providing better results, rotation of the gripper 210 may be optional, and it can be considered only to translate the gripper 210 in accordance with the optimal location/offset that will maximize the number of weeds 530 that can be picked in a single take, while avoiding the crops 510. For thinning purpose, the offset can be calculated to maximize the number of immature crops to remove near the desired, stronger crops.
The processing module 400 is thus configured to maximize a weed picking rate by maximizing the number of weeds 520 to be picked in a single take. A single take, or picking event, involves deployment of the gripper 210 from a first position, such as proximate to the frame 102 of the picking module 100, to a second position proximate the ground for weed or immature crop picking. The map 500 produced based on the images is processed, and an anchor weed 530 is selected (see
The anchor weed 530 corresponds to a weed that must be picked in a weed picking event. The anchor weed 530 may, in one implementation, be the closest weed to the gripper 210 (selection based on proximity to the gripper). In one implementation, the anchor weed 530 may instead be selected based on other parameters, such as the number of weeds 520 in a cluster of weeds, for example allowing for a maximum number of weeds 520 to be picked in a single take. The cluster of weeds defines a group of weeds 520 that are removeable in a single take and includes the anchor weed 530. In this implementation, the memory 412 of the processing module 400 may include instructions for the processor 414 to form multiple clusters of weeds, then resolve and compare solutions for each cluster to find which solution allows for the greatest number of weeds 520 to be picked in a single take. AI clustering models can be used to form the clusters of weeds, such as k-means models. Alternate clustering methods can also be used, such as simpler Euclidian distance-based methods. Since several clusters are likely to be formed, the system can choose the cluster which optimizes a “loss function”. Parameters inputted in the clustering loss function can include the number of weeds in a given cluster, the presence of a crop 510, the distance of the cluster from the crop 510 and whether the shape of the loss function is similar to the footprint of the gripper 210. The loss function can be optimized at least one of maximizing the number of weeds to be picked and avoiding the crops within a given space. The space can be a 3D space or zone, such as the one represented by
The anchor weed 530 may also be selected based on constraints, for example being selected to avoid gripper 210 contact with the crop 510. In possible implementations, the processor 414 may find that an anchor weed 530 cannot be picked while respecting the constraints, such as crop 510 avoidance. The anchor weed 530 may therefore need to be changed to allow for a solution which respects the constraints. In one implementation, the constraints may comprise a maximum or minimum weed distance from a crop 510. A maximum weed distance from a crop 510 may be chosen to allow for weeds 520 that are determined to be too far to be competing with crops 510 to be left in the field. A minimum weed distance from a crop 510 may be chosen to reduce or eliminate the likelihood of contact of the crop 510 with the gripper. In one implementation, the constraints may comprise a maximum or minimum gripper distance from the crop 510. Similar to the weed distance previously mentioned, a maximum gripper distance may allow the gripper to ignore weeds too far away, while a minimum gripper distance may reduce or eliminate the likelihood of contact with the crop 510, to avoid damaging the crop while removing weeds. The minimum and maximum distances may be one of lateral distance, vertical distance, or a combination of the two to ensure that sufficient distance separates the gripper 210 from the crop 510.
Picking the maximum number of weeds 520 per take involves, firstly, processing of the map 500 with respect to the anchor weed 530. The footprint of the gripper 210 may, in effect, be modelled. In a possible implementation, it can be modeled as a function in a 3D space, whereby the 3D space has an x-axis, a y-axis, and a z-axis. The distance between gripper jaws may be the width of the 3D space, the length of the jaws may be the length of the 3D space, while the grippable height, extending from a bottom end of the gripper jaw to a top end of the gripper jaw, may be the height of the 3D space. For example, and with additional reference to
Where z defines a point in the 3D space. l defines the length of the gripper/a jaw (i.e., the length of the 3D space), w defines the space between the jaws (i.e., width of the 3D space), xc and yc define center points of the gripper in the x and y plane, and x and y represent coordinates in the x and y plane. With reference to
In possible implementations, given that the gripper footprint can be further rotated around the z axis, the gripper footprint may fully be modelled using the following functions accounting for rotation along the z axis, in addition to positioning along the x and y planes:
With the gripper footprint fully modelled, the first loss function (or cost function) which respects the constraints (e.g., crop avoidance) and the objectives of the weed picking device (e.g., maximum number of weeds per take), is then solved with respect to the picking space or zone defined above. As an example, the first loss function may be of the following form:
Where c is the number of crops and w is the number of weeds. For example, the crops 510 may each have a negative value (e.g., -3), while the weeds 520 may have a positive value (e.g., +1). The specific values may be selected by the operator. The function may be of the simplest form comprising the sum of c and w. The function may then be solved in the space and optimized. The processor optimizes for the value of the function with respect to the number of weeds 520 while respecting the constraints so that a maximum value is obtained. The optimized solution of the function, having the maximum value, represents a picking location where the maximum number of weeds 520 may be picked. In one implementation, even after applying a large penalty to a constraint, it is possible that the solution will still not respect a constraint. For example, it may be the case that five weeds 520 (with a value of +1 each) and one crop 510 (with a value of -3 each) are within the footprint of the gripper 210, having a sum of +2 which may suggest that the solution is optimal. Since this solution would also pick the one crop 510 along with the five weeds 520, it may not be acceptable. Accordingly, it may be desired to explicitly check a constraint to ensure that it is respected or to define the constraint with a very high value that cannot be overcome in any practical scenario.
In a next step, once the map 500 has been processed and the picking location selected in view of the optimized function, the positioning of the gripper 210 may be controlled accordingly. In a possible implementation, a second loss function may, therefore, additionally include optimization for positioning of the gripper:
Where Pw is the xy coordinates of the weeds (including the anchor weed 530), Pcr is the xy coordinates of the crops, xg is the offset of the x coordinate of the gripper, yg is the offset of the y coordinate of the gripper, and θg is the offset of the orientation of the gripper. The function is optimized by looking for the minimal values of xc,yc and θ with respect to the coordinates of the weeds 520 and the crops 510. That is, there is an offset that the gripper 210 must travel in the x and y directions and along the θ axis to arrive at the picking location which allows the gripper 210 to respect all the constraints, while maximizing the number of weeds 520 picked in a single take. The solution that results in the minimal value of the function is the optimized solution. The optimized solution comprises both an optimal offset, whereby the gripper moves by the offset amounts in the x and y directions and the θ axis, and an optimal weed picking location, where the greatest number of weeds may be picked in a single take. In one implementation, the optimal offset may instead be carried out in only one axis, only two axes or any other variation. Optimizing the second loss function allows centering the weeds to be picked relative to the center of the implement.
As an example, the second loss function ensures that that the gripper encompasses the coordinates of the anchor weed 530, at least one additional weed, while avoiding the crop 510. The processing module 400 may then control the gripper 210 to pick the weeds 520 in the optimal location, and then stores the information about the picked weeds 520 in the memory, so that the map 500 may be updated as the vehicle 10 moves through an agricultural field.
The second loss function may, in one implementation, be defined according to the following:
Where:
Where Wcost is the cost with respect to the weeds, Ccost is the cost with respect to the crops, Tcost is the total cost, α is a number greater than 1, and Ncr is the number of crops considered by the gripper. Ncr has a minimum value of 1, since the gripper considers at least one crop when deciding which weeds to pick. In resolving the above functions, only those weeds and crops falling in the gripper footprint are considered (where z≤1). Any neighboring weeds and crops may be ignored. Per the above, where the cost for crops Ccost is equal to or greater than 1, there are no crops in the gripper footprint and the constraint of crop avoidance is respected. In this case, the total cost Tcost is equal to the cost for weeds Wcost. If Ccost is smaller than 1, however, the crop is inside the gripper footprint and would be grabbed. Accordingly, this would not respect the crop avoidance constraint and the solution would not be acceptable. In such a case, Wcost is multiplied by an arbitrarily large number α to return a large loss.
The optimized solution may be found through any number of optimization tools. In one implementation, optimization may be achieved through Particle Swarm Optimization (PSO). PSO optimizes a function by iteratively improving a potential solution with regards to given variables so as to obtain the best solution. Other optimization tools may equally be used, such as constrained optimization by linear approximation (COBYLA). In accordance with the above, the second loss function finds an optimal location for the gripper to maximize the number of weeds while respecting constraints in a first step. In a next step, the second loss function finds an optimal offset, using both translational and rotational motion, for the gripper to move along and arrive at the optimal location for picking the weeds. In one implementation, it may be desired to not optimise a solution. For example, the first loss function alone may be sufficient without consideration of the second loss function. In such a case, the only constraint to be considered may be, for example, whether there is a crop 510 inside the footprint of the gripper 210. The first loss function may accordingly be resolved while accounting only for crop avoidance. so that no optimisation is carried out for either maximizing the number of weeds or for finding an optimal offset to the optimal location. In one implementation, constraints may be shared between the optimal location and the optimal offset, for example to ensure that the gripper maintains a minimum distance to the crops as it moves in accordance with the optimal offset to an optimal location for weed picking.
The specific optimization tool, such as PSO or COBYLA, may be selected based on case-specific data or through comparison. In one implementation, for example, it was found that PSO resulted in a higher rate of weed picking, while also having a slightly higher rate of failure which was acceptable in the present case:
Table 1 shows a comparison of two optimization tools, PSO and COBYLA, in one example based on 1,000 samples, a second example of 1,000 samples, and a third example of 10,000 samples, each sample corresponding to the number of times the algorithm was run based on given initialization conditions. The above is only provided for illustrative purposes. Other optimization tools may equally be preferred generally based on a compromise between accuracy and computational requirements.
In one implementation, the gripper 210 moves to the anchor weed 530 first, then determines the rotational and translational movement required to maximize the number of weeds in the take, while also grabbing the anchor weed 530. In one implementation, there may be other constraints/variables which the function can consider. For example, the function may additionally consider distance to the crop when determining an optimal offset or when determining the positioning of the gripper. In one implementation, the function may be capable of considering other factors, including time required to reach the weeds, where, for example, the minimum translation (implement travel distance) and rotation as defined in the second loss function do not result in a minimized or optimized gripper travel time.
With reference to
With reference to
As disclosed, it is possible that a solution cannot be found for picking of a weed, even after softening constraints or changing the anchor weed. In such cases, an operator may decide to either ignore the weed or to override the constraints, for example allowing the gripper to make contact with a crop while picking the weed. Additionally, although we have treated the 3D space of the jaws as a constant space, it may additionally be modified. For example, the distance between the two jaws may be modified by moving the jaws closer together or further apart. This may be useful in cases where a solution cannot be found while respecting constraints with a larger space between the jaws, where a smaller distance allows a solution to be obtained.
Although reference and examples have been made to weed removal in accordance with one embodiment of the present disclosure, it is equally envisaged that other uses may be possible. For example, the agricultural implement may, instead of removing weeds from a field, remove extra or immature crops. Crop thinning is the process of removing young crops from a field to make space for other, stronger crops. For example, carrots are often planted in a row in a field. It is sometimes necessary to remove some carrots from the row to allow other carrots to grow to full size. It may also be necessary to remove carrots in areas where growth has been less than desired, due to poor sunlight or other factors. The removed carrots may then be replanted elsewhere or disposed of. Accordingly, instead of identifying and removing weeds from a field, the present disclosure may instead be directed to identifying and removing immature or selected crops, such as young carrots.
Number | Date | Country | Kind |
---|---|---|---|
3146854 | Jan 2022 | CA | national |