Patent Application
20230413800
Invention applies to the field of optics, specifically to laser technologies, more specifically to industrial lasers, particularly to laser beam positioning. The invention has particular application for the laser-treatment of weeds or other unfavourable plants.
There are several methods to position lasers, for example, robotic arms with multiple degrees of freedom, manipulators, CNC type mechanisms, rotating prisms, or two perpendicular mirrors—galvanometer scanners (galvo scanners). There are also solutions available to track moving objects using image recognition software based on neural network algorithms, including for purpose to position lasers on them. A few examples of such technologies are presented in European patent publication No. 1098563, U.S. Pat. Nos. 6,795,568 and 10,681,9052. Nevertheless these technologies don't provide fast enough fulfilment of all tasks: recognition of defined objects, object segmentation, and treatment with a laser in the whole selected area, so it is necessary to have additional means to detect speed of the moving object, for example, radars, lidars or speed wheels. These additional components make the system too complicated and increase costs. For example, detection and segmentation of objects in an image which is 500×500 pixels or larger, usually takes several tens to several hundreds of milliseconds, so without speed control and speed correction it is practically impossible to make real-time laser treatment, because object would be in a different place after this time period.
To avoid the need to measure and compensate for the relative displacement of an object relative to the camera and/or laser, a technique has been developed that allows a laser beam to process a moving object or a stationary object with a moving device, i.e. laser and laser beam, respectively, with millimeter accuracy or better. An obvious solution to the above problem would be to significantly increase the moving speed of the laser, which is not cost-effective, as it would require more than 100 times more powerful laser to get the same treatment effect, which would be technologically complex and financially inefficient.
In accordance with a first aspect of the invention there is provided a method for positioning a laser, which includes following steps:
The above problems or work tasks are solved by creating a method for laser positioning and processing of objects of interest with a laser beam, which includes image acquisition step, followed by identification of objects of interest in the image. The object of interest may be weed or other unfavorable non-agricultural plant or part of a plant. The method is characterized by the division of the area to be processed into lines, followed by the calculation of the positions of the lines in space and finally by the processing of the lines with laser beam. As a result, one cycle with the above-mentioned steps is realized in up to 25 milliseconds. In some cases, one cycle of the steps (a) to (e) may take less than 25 milliseconds.
In this way, the present invention allows for laser treatment of objects that are moving relative to the laser providing the laser-generated beam, with a precision of the order of ˜1 mm at acceptable movement speeds. Preferably, the method comprises (e.g. iteratively) performing a plurality of cycles of the steps (a) to (e) as the laser and objects of interest are in motion relative to each other. For instance, the method may comprise moving the laser relative to the object(s) of interest during and/or between each cycle of steps (a) to (e). In other words, on completion of one cycle (a) to (e), the process starts again with an acquisition of a further image. Within each cycle, identified treatment line(s) are treated by the laser beam. In this way, a plurality of images may be acquired and processed in order to provide laser treatment of objects of interest whilst the objects of interest and the laser are in motion relative to each other. For example, the method of the invention may be implemented on an apparatus that is towed by a vehicle (e.g. a tractor) in an agricultural field in order to provide laser treatment of undesirable weeds in the field while the apparatus moves relative to the objects of interest. In such examples the method may comprise moving the laser (and any camera or further components of the apparatus) during and/or between each cycle of steps (a) to (e). It is also envisaged that the invention may be used in a “static” mode in which the objects of interest and the laser are not in relative motion.
The laser-treated area is typically the area, or portion, of the image (or in some instances the whole image) that corresponds to the region that is desired to be treated by the laser, dependent on the particular application. For example, the laser-treated area may be a whole object of interest or multiple objects of interest (e.g. weeds), or only part of one or more objects of interest. In some examples, the laser-treated area may be a narrow “single line” (for example 5 to 10 pixels wide, or even in some cases less than 5 pixels wide) spanning across the width of the image, corresponding to approximately the same line width of the laser beam.
Preferably, in step e, only a subset of lines (e.g. treatment lines) are treated with the laser beam. In particularly preferred embodiments, in step (e) only a single (e.g. exactly one) line is treated with the laser beam. Thus, in contrast to state of the art processes which may attempt to treat the whole of each identified object of interest in the acquired image, the present invention preferably treats sections of object(s) of interest arranged along a single line with the laser beam within one cycle of the steps a to e. This advantageously reduces the time taken to perform a cycle of steps a to e, typically to 25 milliseconds or less. In this way, a plurality of cycles of the steps a to e can be performed in order to treat the objects of interest as the laser and objects of interest are in motion relative to each other.
The line or lines treated by the laser in step (e) may comprise one or more segments (which may also be referred to as “laser treatment regions”) dependent on the position of the objects of interest in the acquired image. Each segment is typically a region of the single line that overlaps with an identified object of interest for laser treatment. The treatment line(s) are typically orientated in a cross direction to a direction of relative motion between the laser (e.g. an apparatus carrying the laser) and objects of interest, preferably being arranged substantially perpendicular relative to the direction of relative motion (e.g. angled at at least 60 degrees relative to the direction of relative motion, more preferably at least 75 degrees). Typically, the treatment line(s) have a dimension (e.g. along the direction of relative motion) smaller than a dimension of the objects of interest in the image (e.g. less than 10 pixels). However, in general the treatment line(s) may have any orientation, for example along the direction of motion
The completion of one cycle of the steps (a) to (e) within 25 milliseconds advantageously allows for the present invention to treat the objects of interest (e.g. weeds) while continuously moving at a speeds of up to 300 m/hr (a typical average speed for manual weeding) with a ˜2 mm spatial error, which is deemed acceptable. Thus, preferably the method comprises moving the laser relative to the object(s) of interest by approximately 300 m/hr (e.g. between 100 to 3000 m/hr, more preferably between 200 and 500 m/hr, more preferably between 200 and 400 m/hr).
Step b comprises the identification of objects of interest (e.g. weeds) in the acquired image. The image is typically an image of the ground (e.g. in an agricultural field). The acquired image typically represents a size of 50 mm×50 mm of the ground surface or larger. The image is preferably a two-dimensional image. As will be explained in more detail below, the identification of the objects of interest typically comprises detection of green pixels in the acquired image and instance segmentation of objects within the acquired image. The instance segmentation is typically performed using a machine learning algorithm such as a neural network. A preferred machine learning algorithm is a convolutional neural network.
The machine learning algorithm is configured to differentiate between objects that are not intended for laser treatment (e.g. agricultural crops) and the objects of interest that are desired to be treated by the laser (e.g. weeds or other unfavourable non-agricultural plants or parts of a plant). In particular, the machine learning algorithm is typically configured to identify objects that are not intended for laser treatment. The machine learning algorithm is typically trained using training data comprising tagged images. For example, the training data may comprise a plurality of images that are tagged as agricultural crops or weeds.
By using a machine learning algorithm to identify objects that are not intended for laser treatment, and detection of green pixels in the acquired image, it is possible to infer that the identified green pixels that do not correspond to the identified objects that are not intended for laser treatment (e.g. agricultural crops) correspond to objects of interest for laser treatment (e.g. weeds).
Step d comprises calculation of the coordinates of the lines (e.g. treatment lines) in physical space. In other words, the coordinates of the treatment lines determined in step c are transformed from the image space (e.g. pixel coordinates) into the physical “real-world” space (e.g. in Cartesian or polar coordinates). This may be performed based on knowledge of the geometry of the camera used to acquire the image relative to the ground.
In step e, the lines (e.g. treatment lines) are treated with a laser-generated laser beam. The thermal energy provided by the laser beam preferably kills or otherwise damages the object of interest for laser treatment (e.g. weed). The laser beam is generated by a laser, with the direction and movement of the laser beam preferably being controllable by means of galvanometer. In other words, the laser beam is preferably guided by means of a galvanometer scanner. The inventors have found that the use of galvanometer scanner is particularly advantageous as this allows for the fast and precise control of the laser beam. This is especially beneficial when using high-powered lasers that are typically heavy and unwieldy to physically move.
Moreover 50 mm×50 mm or greater picture is captured. Laser beam is moved by the help of galvo-scanner, where laser output beam is aligned with the camera objective at a distance of 15 cm or closer. Typically, the exit point of the laser beam and the centre of the camera lens are positioned at a distance of 15 cm or closer from each other. Minimising these distances increases accuracy and avoids errors (e.g. parallax errors) in the positioning of the laser-beam.
The identification of objects of interest in the image typically includes the following sub-steps:
As discussed, the instance segmentation of objects within the image is typically performed using a machine learning algorithm such as a convolutional neural network. The instance segmentation in step b2 is typically instance segmentation of objects of interest not intended for laser treatment (e.g. agricultural crops). In other words, preferably the machine learning algorithm only identifies objects of interest that are not intended for laser treatment (e.g. crops such as carrots). By identifying only one class, the cycle speed is increased. However, in alternative embodiments, a multiple class machine learning algorithm (e.g. CNN) may be used to identify both objects of interest intended for laser treatment and those not intended for laser treatment.
In steps b3 and b4, the coordinates of polygons of objects not intended for laser treatment (e.g. crops) and optionally objects of interest for laser treatment (e.g. weeds) are acquired and marked on a binary image while waiting for the result of the green pixel detection. In other words, the detection of green pixels in step b1 and the steps of b2 to b4 are typically performed substantially simultaneously, for example in parallel CPU processing threads.
In step b5, a binary image of objects of interest for laser treatment (e.g. weeds) is acquired. This is achieved by combining the result of the green pixel detection in step b1 (e.g. a binary mask of green pixels in the image) with the binary image obtained from the instance segmentation. The identified objects not intended for laser treatment may be subtracted from the binary mask of green pixels in the image, thereby obtaining a binary image of objects of interest for laser treatment (e.g. weeds). In step b6 the coordinates (e.g. polygon coordinates) of the objects of interest for laser treatment may then be acquired.
In a preferred embodiment, the identification of objects of interest in the image comprises: b1. detecting green pixels in the image and acquiring a binary mask of the green pixels; b2. performing instance segmentation of objects of interest not intended for laser treatment and obtaining a binary mask of the objects of interest not intended for laser treatment; and b3 generating a binary mask of the objects of interest for laser treatment by subtracting the binary mask obtained in step b2 from the binary mask obtained in step b1.
In preferred embodiments, the detection of green pixels is performed in a portion of the acquired image (rather than across the entire image). Although the whole acquired image may be analysed for instance segmentation and identification of objects of interest, only the green pixels of the objects of interest for laser treatment within the identified treatment lines are required to be treated by the laser beam. This advantageously increases processing speed. In some embodiments, only a portion of the image may be analysed by the CNN in order to further increase processing speed. However, typically such a portion is typically required to be at least 1.5×, preferably at least 2× larger than the size of an object not intended for laser treatment (e.g. a crop).
Division of the area intended for laser treatment in to treatment lines typically includes the following sub-steps
In step c1, one or more theoretical treatment lines (which may also be referred to as “laser processing lines” herein) are constructed with predefined distance in pixels. The theoretical treatment line(s) typically have a predetermined relationship with the acquired image, dependent on the geometrical relationship between the camera and the laser. In preferred embodiments, a theoretical treatment line spans the width of the image. The theoretical treatment lines are typically orientated in a cross direction to a direction of relative movement between the laser and objects of interest. In preferred embodiments, a theoretical treatment line has a dimension (e.g. along the direction of relative motion) smaller than a dimension of the objects of interest. This may be a dimension of 10 image pixels or less, preferably 5 image pixels or less and more preferably 1 pixel.
The theoretical treatment line(s) are typically located within the laser-treated area. In other words, the number of theoretical treatment lines is typically dependent on the size of the laser-treated area. In alternative embodiments, the treatment lines may be determined using techniques such as detection of pixel brightness, for example with the treatment lines corresponding to areas of pixel brightness above a predetermined threshold.
In some embodiments, the division of the area intended for laser treatment comprises constructing one or more theoretical treatment lines with predefined distance in pixels, and determining region(s) of overlap between the theoretical treatment line(s) and identified object(s) of interest for laser treatment.
In addition, calculation of each treatment line coordinates in the physical space include transformation of each line from coordinate system in pixels to coordinate system in metric units in physical space. Line treatment with laser generated laser beam include reception of converted line in metric coordinates and treatment of object of interest. Angular coordinate systems (e.g. polar coordinates) in physical space may be used, as well as Cartesian coordinate systems.
The laser beam is preferably guided by means of a galvanometer scanner. This is particularly advantageous as it allows fast and precise control of the laser beam, in contrast to alternative mechanisms such as gimbals or turntables. The laser beam is typically a visible light laser beam with a wavelength of 400-500 nm and a power density of at least 15 kW/cm{circumflex over ( )}2.
Preferably, the images are acquired by a visible light camera with a speed at least 70 frames per second, even better 200 frames per second, even more better 500 frames per second with a resolution of 0.3 Megapixels or greater. Object of relevance is preferably a plant or part of the plant and it is typically detected using object detection based on neural network algorithms.
Moreover in each step, preferably only a single line is treated by a visible light laser beam, typically in the wavelength of the 400-500 nm and preferably having a power density 15 kW/cm{circumflex over ( )}2 or more. A power density of 15 Kw/cm{circumflex over ( )}2 or more has been found to provide enough energy to kill or otherwise damage the objects of interest for laser treatment such as weeds.
In accordance with a second aspect of the invention there is provided an apparatus for positioning a laser beam, comprising:
Preferably, the laser beam positioning device is a galvanometer scanner.
In accordance with a further aspect of the invention there is provided an apparatus for positioning a laser beam, comprising:
Thus, the apparatus of the present invention provides all of the advantages discussed above in relation to the method of the first aspect of the invention.
The apparatus contains several basic elements. It contains a camera configured to obtain an image, a computing device configured to identify the object of interest in the acquired image, and a laser configured to generate a laser beam.
The apparatus also comprises a laser beam positioning device appropriately coupled to the laser and configured to position the laser, thereby positioning the laser beam itself. The apparatus is characterized in that a galvo scanner is used as the laser beam positioning device. In addition, the computing device is connected to the laser and the laser beam positioning device in order to give start command for the generation of the laser beam according to the identified object of interest and to position the laser beam through the laser positioning device so that the laser beam reaches the object of interest. In addition, the computing device includes a neural network node configured to recognize an object of interest, such as a plant or part of a plant.
Typically, the laser is configured to generate a laser beam with a wavelength of 400-500 nm and a power density of at least 15 kW/cm{circumflex over ( )}2. These are suitable parameters to damage and/or kill weeds and other undesirable plants.
Preferably, the laser is configured to emit laser beam in the wavelength of 445 nm and power density of 16.7 kW/cm{circumflex over ( )}2.
Innovative approach described in the previous sections eliminates the need for movement speed measurement and/or movement measurement and compensation ensuring treatment for the whole working area. Treatment region for each treatment step is reduced significantly by this approach. By this we mean that the laser receives a treatment command only for a single line (e.g. a single theoretical treatment line) (or a limited number of multiple lines to maintain claimed accuracy) instead of all lines in array as it would be if standard treatment approach would be used for the whole working area at once. Single line can be treated in a few milliseconds not exceeding 20 milliseconds (depending on the total line length and laser power), which introduces only a millimeter or less (or a different predetermined maximum allowed spatial error displacement) as relative displacement measured from starting position. Next image is acquired after the line is treated by the laser, objects are identified and segmentation polygons are calculated and next treatment coordinates are sent to the laser. Thus dividing the working area in a single line tasks ensuring ability to perform high accuracy treatment of one millimeter or less while objects are in motion relative to the laser module.
An additional light source, such as a flash or other light source, may be installed to provide a sufficient amount of light.
To eliminate the need for stereo disparity compensation between the laser beam exit point and the camera lens, the camera center and the laser beam exit point should be located 15 cm or closer.
Typically, the apparatus according to the invention is configured to be mounted on or integrated in a vehicle (e.g. a tractor or any other suitable manned or autonomous vehicle) or a unit (e.g. a trailer) configured to be attached to a vehicle. Preferably the apparatus (and any vehicle to which the apparatus is mounted or attached) is configured to move the laser relative to the object(s) of interest by approximately 300 m/hr (e.g. between 100 to 3,000 m/hr, more preferably between 200 and 500 m/hr, more preferably between 200 and 400 m/hr).
In accordance with a third aspect of the present invention, there is provided a method, performed in a laser treatment apparatus, comprising: (a) acquiring an image; (b) identifying one or more objects of interest for laser treatment within the acquired image; (c) selecting a laser treatment area of the acquired image and dividing the laser treatment area into one or more laser processing lines; (d) determining one or more laser treatment regions within the laser processing line(s) based on the identified object(s) for laser treatment; and (e) treating the laser treatment region(s) with a laser beam.
Thus, the method of the third aspect provides all of the advantages discussed above. The method of the third aspect may comprise any of the preferable or optional features discussed above with reference to the preceding aspects of the invention and offer corresponding advantages. In particular, by dividing the laser treatment area into one or more laser processing lines, and treating the laser treatment region(s) with the laser beam, the method allows objects of interest to be laser-treated while the laser apparatus and objects of interest are in relative motion, with a precision of the order of approximately 1 mm at acceptable movement speeds.
The laser treatment area is typically the area, or portion, of the image (or in some instances the whole image) that corresponds to the region that is desired to be treated by the laser, dependent on the particular application, as has been described above.
Typically, each laser processing line has a dimension that is smaller than a dimension of the objects of interest for laser treatment (e.g. in image coordinates). In this way, the method allows for faster processing of the acquired image. In some examples, the laser processing lines have a dimension in the direction of relative motion of 10 pixels or less, preferably 5 pixels or less, more preferably 1 pixel.
Preferably, the laser treatment region(s) within a subset of the laser processing lines are treated with the laser beam in step (e). In this way, the speed with which the cycle of steps (a) to (e) is performed is increased, as only a subset, or portion, of the area corresponding to the image is treated with the laser beam. This is in contrast to prior art methods which as described above may attempt to treat each part of the image. In particularly preferred embodiments, the laser treatment regions within exactly one (e.g. only a single) processing line are treated with the laser beam in step (e). In this way, the processing speed may be maximised. In preferred embodiments the sequence of steps (a) to (e) may be performed in 25 ms or faster.
Typically, each laser processing line is elongate and orientated in a cross direction to a direction of relative movement between the apparatus and the objects of interest. Preferably, the positions of the laser processing lines are predetermined for each acquired image.
The method preferably comprises (e.g. iteratively) performing a plurality of the cycles of the steps (a) to (e) as the laser treatment apparatus and the objects of interest for laser treatment move relative to each other. For instance, the method may comprise moving the laser treatment apparatus relative to the object(s) of interest during and/or between each cycle of steps (a) to (e). Preferably, the method comprises moving the apparatus relative to the object(s) of interest by approximately 300 m/hr (e.g. between 100 to 3000 m/hr, more preferably between 200 and 500 m/hr, more preferably between 200 and 400 m/hr).
The method preferably comprises converting the determined laser treatment region(s) into physical space coordinates for treatment by the laser beam.
In preferred embodiments, the laser beam is generated by a laser and is directed by a galvanometer scanner.
Preferably, the identifying one or more objects for laser treatment within the acquired image comprises identifying objects of interest within the acquired image using a machine learning algorithm, preferably a neural network. The machine learning algorithm may be configured to identify objects of interest that are not intended for laser treatment (e.g. agricultural crops). Typically, the method also comprises identifying green pixels within the image or a portion of the image. By also identifying green pixels within the image, it can be inferred that the green pixels that do not correspond to the objects of interest that are not intended for laser treatment instead correspond to objects of interest that are intended for laser treatment (e.g. weeds). A binary mask of objects of interest for laser treatment may be generated as described above.
Step (d) of the method comprises determining one or more laser treatment regions within the laser processing line(s) based on the identified object(s) for laser treatment. Typically, this step comprises determining the region(s) where the object(s) of interest for laser treatment overlap with the laser processing line(s). A single laser processing line may comprise exactly one laser treatment region, or could comprise two or more laterally spaced laser treatment regions collinearly arranged along the laser processing line dependent on the relative arrangement of the laser processing lines and the objects of interest.
In accordance with a fourth aspect of the invention there is provided a laser treatment apparatus, comprising: a camera configured to acquire an image; a laser configured to generate a laser beam; a laser beam positioning device configured to direct the laser beam; and an image processing unit configured to perform the steps of: (a) receive an image acquired by the camera; (b) identify one or more objects of interest for laser treatment within the acquired image; (c) select a laser treatment area of the acquired image and divide the laser treatment area into one or more processing lines; (d) determine one or more laser treatment regions within the laser processing line(s) based on the identified object(s) for laser treatment; and (e) control the laser and laser beam positioning device so as to treat the laser treatment regions(s) with the laser beam.
The laser treatment apparatus therefore provides all of the advantages discussed above. The image processing unit may be configured to perform any of the steps discussed above. As such, the image processing unit may comprise or be configured to perform any of the preferable or optional features discussed above with reference to the preceding aspects of the invention and offer corresponding advantages.
Preferably, the image processing unit is further configured to control the laser and laser beam positioning device such that the laser treatment region(s) within a subset of the processing lines are treated in step (e). Preferably, the image processing unit is further configured to control the laser and laser beam positioning device such that the laser treatment region(s) within (e.g. only) exactly one processing line are treated within step (e).
Preferably, the laser beam positioning device is a galvanometer, although other devices are envisaged such as gimbals or turntables to which the laser is mounted. Typically, the laser is configured to generate a laser beam with a wavelength in the region of 400-500 nm and a power density of at least 15 kW/cm{circumflex over ( )}2.
Typically, the laser treatment apparatus is configured to be mounted on or integrated in a vehicle (e.g. a tractor or any other suitable manned or autonomous vehicle) or a unit (e.g. a trailer) configured to be attached to a vehicle. Preferably the apparatus (and any vehicle to which the apparatus is mounted or attached) is configured to move the laser relative to the object(s) of interest by approximately 300 m/hr (e.g. between 100 to 3,000 m/hr, more preferably between 200 and 500 m/hr, more preferably between 200 and 400 m/hr).
In accordance with a first example there is provided technology for laser positioning consisting of the following stages: a. acquisition of an image; b. identification of the objects of interest in the image; c. division of the area to be processed into processing lines; d. calculation of the positions of the lines to be processed in the space; e. processing of the lines with a laser; differs by the fact that the whole process takes less than 25 milliseconds.
Preferably, the technology is characterised by the fact that the size of the obtained image in nature is 50 mm×50 mm or more.
Preferably, the technology is characterised by the fact that the laser beam is directed by means of the mechanism of a galvoscanner.
Preferably, the technology is characterised by the fact that the output point of the laser and the center of the camera are at a distance of 15 cm or less.
Preferably, the technology is characterised by the fact that a. To obtain an image a visible light camera is used at a speed of at least 500 frames per second and with a resolution of 0.3 megapixels or more; b. The object of interest is a plant or a part of a plant and is identified using a recognition algorithm based on neuron network technologies; c. only one line is processed in a single session; d. The line is processed using a visible light laser with a wavelength of 400 to 500 nm and a power density of at least 15 kW/cm{circumflex over ( )}2.
In accordance with a second example there is provided equipment for processing an object with a laser, consisting of the following parts: a. camera; b. computer; c. laser; differs by the fact that it can acquire an image, identify the object of interest in the image, and process one line with a laser so that the whole process takes less than 25 milliseconds.
The invention will be described with reference to the appended drawings, in which:
In one example, the technology consists of several basic elements shown in
An example will now be described. In order to exclude the need to use the measurement and compensation of the displacement speed and/or displacement, while ensuring the required processing area, the area to be processed in a single laser run is significantly reduced. This means that a command is sent to the laser to process only one line (or some other maximum permissible number of lines to fit within the predefined fault limits), not an entire array of lines, as it would be in the case of the standard approach. Processing of one line may be performed in a unit of time ranging from a few milliseconds to teen milliseconds (depending on the length of the line and the power of the laser), which results in a relative difference with respect to the original position within only a millimeter or fraction of a millimeter (or some other predefined maximum permissible error). After completing processing of the line, the next image is obtained, the site is identified and segmented and the coordinates of the next line to be processed are sent to the laser. Thus, by dividing the area to be processed into tasks of single lines, it is possible to process objects in motion with a laser with the accuracy of a millimeter or even more precisely.
In one example, the equipment consists of several main elements shown in
In order to ensure a sufficient amount of light, an additional light source, such as a flash or other lighting fixture, may be installed.
In order to exclude the need to compensate for the perspective between the output point of the laser and the camera lens, the center of the camera and the output point of the laser should be located at a distance of 15 cm from each other or less.
In another example of application, the invention may be realized in the following manner. First, the camera acquires an image of the furrow area and then it is saved in RAM or other fast memory on the on-board computing device. The main thread of the computing device program crops a narrow horizontal area across the width of the acquired image and passes this area to open source program for detecting green pixels in the on-board computing device's parallel CPU thread. Then the main thread of the computing device program passes the original acquired image to a pre-trained convolutional neural network model for agricultural crop instance segmentation. The obtained instance segmentation masks are converted into polygon coordinates in parallel CPU threads. The main thread of the computing device program waits for the result from the green pixel detection program and obtains a binary mask with all green vegetation, including agricultural crops, marked. The coordinates of the agricultural crop instance polygons are used to subtract the green areas of the agricultural crops in the binary mask of the green plants and leave only the green areas of the weeds, thus obtaining the binary mask of the weeds. The main thread of the computing device program then obtains the coordinates of the weed area bounding polygons from the weed binary image and constructs laser processing path lines with a certain step in pixels and determines the intersections with the weed bounding polygon. From the obtained intersection pairs, the main program thread constructs the laser processing lines and each line is converted from the image coordinate system in pixels to the metric coordinate system of the galvo scanner area to be processed on furrow. The main thread of the computing device program puts one converted line to a queue so that the parallel CPU thread can send it to the galvo scanner using the User Datagram Protocol UDP. The galvo scanner receives the treatment line in metric coordinates and processes the weed area by positioning the laser beam as requested.
The above example may be described as follows. First the image of the area to be processed in the furrow is obtained with a camera and saved in RAM or other faster access memory of the board computer of the equipment. The software flow of the main board computer from the acquired image cuts a narrow horizontal bar across the entire width of the image and transfers this pixel bar to the open code software for determining the green pixels in the parallel CPU flow of the board computer. The main software of the board computer passes the original image to a previously trained convolution neural network model for segmenting crop instances. The resulting instance segmentation masks are converted to the coordinates of the site in parallel CPU flows. The main board computer software flow waits for the result from the detection program of green pixels and obtains a binary image marking all green plants, including crops. The site coordinates of crop instances are used to conceal green areas of crops in the binary image of green plants, and to leave only the green areas of weeds, thus obtaining a binary image of the weeds. The software flow of the main board computer retrieves from a binary image of the weeds the coordinates of the outline sites of weeded areas. The software flow of the main board computer designs imaginary processing lines with a specified step in pixels and determines intersections with outline sites of weeds. From the pairs of resulting intersections the main software flow designs the processing lines and converts each line of the image from pixels of the coordinate system to the metric coordinate system of the area to be processed on site. The software flow of the main board computer places one converted line in a row so that the parallel CPU flow sends it to a laser executing device using the user software protocol UDP. The laser receives the metric coordinates of the lines to be processed and processes the area of weeds.
It is possible to use described technology for different purposes, for example but not limited to: engraving, marking (e.g. labelling), cutting, elimination (e.g. destruction) of weeds or other harmful organisms. Technology allows to treat with laser moving objects or treat stationary objects while the laser is moving, with a precision of millimeter or even more precise.
In this example, the apparatus 50 is mounted on a trailer 100 that is towed by a tractor 1000. However, in other embodiments, it is envisaged that the apparatus may be mounted on a vehicle that may be either manned or autonomous. The apparatus is towed by the tractor along the indicated direction of motion, with laser treatment of undesirable objects (e.g. weeds) occurring as the apparatus is in motion relative to the undesirable objects.
Referring back to
The image 11 preferably represents a portion of the ground having a size of 50 mm×50 mm or greater and typically has a resolution of 0.3 Megapixels or greater. In one preferred example the image has a resolution of 1920×1200 pixels. The image is saved in RAM or other fast memory (although this is not essential). The following steps are then performed by the image processing unit (e.g. computing device).
At step S102, a portion of the image 11 is selected for green pixel identification (although in other embodiments green pixels may be identified across the whole image). This is schematically shown in
In parallel with the detection of the green pixels, at step S108 the original full image 11 is processed to identify agricultural crops 1 in the image by instance segmentation. This is performed by a convolutional neural network that has been pre-trained for agricultural crop instance segmentation. The training data for training the convolutional neural network typically comprises a plurality of annotated images of agricultural crops of interest. At step S110, the obtained instance segmentation masks are converted into a single binary mask.
As is schematically shown in the flow diagram of
At step S112, the binary mask of the agricultural crop instances (obtained in step S110) are subtracted from the binary mask of green pixels obtained in step S106. This therefore generates a binary mask of the weeds (i.e. the green pixels that have not been identified as agricultural crops and are inferred to be objects of interest for laser treatment in the manner discussed above) in the selected portion P1 of the image.
In step S114, the polygon coordinates of the weeds within the selected portion of the image are obtained from the binary mask of the weeds generated in step S112.
In step S116, laser treatment regions(s) 12 are constructed, which will now be explained with reference to
Each laser processing line 10 extends across the full width W of the image and has a height dimension (perpendicular to the image width, i.e. along the direction of motion) that may be selected based on the desired precision and the resolution of the image. Typically, the height of the laser processing line is 10 image pixels or less, preferably 1-5 pixels. In preferred embodiments, each laser processing line has a height of 1 image pixel. In general, the dimension of each laser processing line 10 along the direction of motion is smaller than a dimension of the objects of interest for laser treatment (in image space).
The intersection(s) of the laser processing lines 10 with the polygon coordinates of the weed(s) are then determined in order to construct the laser treatment regions 12. The intersection points of the laser processing lines 10 with the polygon coordinates of the weeds is illustrated in
Thus, the laser processing lines 10 are divided into laser treatment region(s) 12 across which a laser is to be applied. This is schematically illustrated in
The laser treatment regions 12 are typically defined as coordinate pairs where the laser processing line 10 intersects with the weed-bounding polygons. For example, laser treatment regions 12a and 12b may be defined by an array of two intersection coordinate pairs [(x0, y0; x1, y1) (x0, y0; x1, y1)].
Following the determination of the laser treatment regions 12, at step S118 each laser treatment region 12 is converted from the image coordinate system (e.g. pixels) to the physical space coordinate system. This may be performed based on knowledge of the geometry of the camera 22 used to acquire the image relative to the ground. Laser treatment regions 12 within a subset of the laser processing lines 10 may then be processed by the laser at step S120.
Referring back to
The laser beam is generated by the laser 24, with the direction of the laser beam controlled by a galvanometer scanner 26. Thus, the physical space coordinates are sent to the galvanometer scanner 26 with the one or more mirrors of the galvo scanner adjusted so as to treat the calculated treatment lines 12. The coordinates may be sent using a User Datagram Protocol (UDP) or other suitable communication protocol. If the laser processing line (or subset of laser processing lines) is determined to not intersect with any objects of interest for laser treatment, the laser is not activated for that image cycle.
In this example, within one cycle of steps S100 to S120, only one laser processing line 10a is processed and treated with the laser, as shown in
The selected portion P2 of the second image is shown in
Therefore, by iteratively performing the cycle of steps S100 to S120 as described above, with only one laser processing line (or a subset of laser processing lines) being treated within a cycle (“per image”), the objects of interest may be laser treated whilst in continuous relative motion compared to the laser. Consequently, this allows for the present invention to provide laser treatment of the weeds while the apparatus 50 is continuously moving at speeds of up to 300 m/hr (a typical average speed for manual weeding) with a ˜2 mm spatial error, which is deemed acceptable.
Although in the presently described embodiment the laser treatment regions for a single laser processing line are treated with the laser beam in each cycle, in other embodiments, a subset of the laser processing lines (e.g. a subset of the laser processing lines shown in
Further preferred aspects and embodiments of the present invention are set out in the following numbered clauses:
[Numbered Clause 1]: The technology for laser positioning consisting of the following stages:
[Numbered Clause 2]: The technology according to Numbered Clause 1 is characterised by the fact that the size of the obtained image in nature is 50 mm×50 mm or more.
[Numbered Clause 3]: The technology according to Numbered Clause 1 is characterised by the fact that the laser beam is directed by means of the mechanism of a galvoscanner.
[Numbered Clause 4]: The technology according to Numbered Clause 1 is characterised by the fact that the output point of the laser and the center of the camera are at a distance of 15 cm or less.
[Numbered Clause 5]: The technology according to any Numbered Clause from 1 to 4 is characterised by the fact that
[Numbered Clause 6]: The equipment for processing an object with a laser, consisting of the following parts:
| Number | Date | Country | Kind |
|---|---|---|---|
| LVP2020000076 | Nov 2020 | LV | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/EP2021/081396 | 11/11/2021 | WO |
