Patent Application
20240060275
The present disclosure generally relates to a method and system for configuring a machine control unit of a construction machine to control an earth-moving operation that comprises a multitude of phases that are to be performed consecutively, for instance for controlling a motor grader or crawler for ditch or road construction. The controlling comprises using 3D sensors to detect a current state of the earth-moving operation in order to configure the machine control unit to at least partially control the construction machine during the next phase of the earth-moving works.
It is known to use automation systems for construction machines to facilitate earth-moving operations for the operator of the machine by fully or partially controlling the machine. Existing automation systems focus primarily on a final grading pass required to cut the soil surface to the desired profile. There are two primary applications that differ based on whether the desired profile is localized relative to world coordinates (“3D system”), or the desired profile is localized relative to vehicle coordinates (“2D system”). 2D systems are simpler to work with. A soil profile is designed relative to the vehicle system and the operator is fully responsible for navigating the vehicle so that the system generates the correct surface. 3D systems require a surface design file that is created using a complex computer system. The machine can then be localized relative to the 3D plan and the surface cut correctly with a variety of possible navigation motion.
There are various measuring systems known for the specific needs in the field of construction, especially for earth-moving machines, such as excavators, graders or dozers. Such measuring systems can be used with a construction machine to build a digital 3D model of the surroundings of the machine, to detect obstacles in the surroundings of the machine, and/or to support a control of the construction machine. For example, EP 3 086 196 A1 suggests an environment detection by means of a camera system arranged on a bulldozer. The camera system uses a SLAM or SfM algorithm to generate a 3D point cloud of the environment. JP 2019 167 719 A2 discloses a 2D laser scanner by means of which a 3D model of the environment is generated when an excavator, on which the 2D scanner is mounted, is pivoting. WO 2019/197064 A1 shows a device for monitoring a construction machine environment with a ToF camera. Images generated in such a way are used for guiding an operator of the construction machine in order to control the operation of the construction machine on the construction site according to a plan.
Advances in perception sensors allow for a new interactive workflow that enables a 3D design surface to be generated in a simpler manner, resembling that of existing 2D systems. These same perception sensors allow for the machine to localize itself relative to coordinates of a design file and enable advancements in the automation of the machine so that the automation systems can be used in a wider range of operator conditions. WO 2020/126123 A2 discloses an example of a compact “reality capture device” comprising a laser scanner and at least one camera. With this reality capture device, an environment can be optically scanned and measured by means of the laser scanner emitting a laser measurement beam, e.g. using pulsed electromagnetic radiation, wherein an echo is received from a backscattering surface point of the environment and a distance to the surface point is derived and associated with an angular emission direction of the associated laser measurement beam. This way, a three-dimensional point cloud is generated. For example, the distance measurement may be based on the time of flight, the shape, and/or the phase of the pulse. For additional information, the laser scanner data is combined with camera data, in particular to provide high-resolution spectral information, e.g. by means of an RGB camera or an infrared camera. The reality capture device may be mobile and configured to provide surveying data and referencing data at the same time, e.g. wherein at least trajectory data of the device, e.g. position and/or pose data, are provided with the probing data, e.g. laser scanner data and/or camera data, such that probing data of different positions of the reality capture device can be combined into a common coordinate system. The reality capture devices can be configured to autonomously create a 3D map of a new environment, e.g. by means of a simultaneous localization and mapping (SLAM) functionality. The European patent application No. 21176387.5 discloses a similar reality capture device comprising an arrangement of time-of-flight (ToF) cameras instead of a laser scanner, wherein the ToF cameras are configured for jointly capturing 3D point cloud data of the surrounding.
EP 3 865 895 A1 discloses a measurement system for construction machines comprising one or more reality capture devices for capturing surroundings of the construction machine before and during earthmoving works at a construction site.
Some earth-moving operations comprise a multitude of phases that are to be performed consecutively, wherein the phases are at least partly iterative. One example of such an operation is “V-ditching”, which is used for construction a ditch with a V-shaped profile. During consecutive passes of the construction machine, the tool is continuously lowered to perform several cuts.
It would be desirable to provide a construction site measuring system for a construction machine that facilitates performing such earth-moving operations for the operator of the construction machine, particularly by allowing a higher grade of automation in the control of the machine.
It is therefore an object of the present disclosure to provide an improved system and method for configuring a machine control unit of a construction machine for performing an earth-moving operation.
It is a further object of the disclosure to provide such a system and method that allow configuring the machine control unit to perform the earth-moving operation at least partially automatically.
It is a further object of the disclosure to provide such a system and method that allow configuring the machine control unit to perform at least a subset of phases of an earth-moving operation that comprising a multitude of phases that are to be performed consecutively.
A first aspect pertains to a system for configuring a machine control unit of a construction machine for performing an earth-moving operation, the earth-moving operation comprising a multitude of phases that are to be performed consecutively. Said system comprises a measuring system configured for capturing three-dimensional measuring data of a terrain in a surrounding of the construction machine in at least a first detection range, a user interface configured for receiving user input from an operator of the construction machine, and a computing unit operatively coupled at least with the measuring system, the user interface and the machine control unit. According to this aspect, the computing unit is configured for detecting elements of a previous phase of the earth-moving operation based on the three-dimensional measuring data, and, based at least on the detected elements, for configuring the machine control unit to perform the next phase of the earth-moving operation at least partially automatically.
According to some embodiments of the system, the measuring system comprises at least one measuring unit at the construction machine, each measuring unit being configured for capturing 3D point cloud data, i.e. as the three-dimensional measuring data. Each measuring unit comprises:
According to some embodiments, the system comprises a context camera having a known position relative to the measuring system and/or the three-dimensional measuring data and being configured for capturing context image data of the terrain within the first detection range. In this case, the user interface is configured for displaying at least one context image based on the context image data to an operator of the construction machine, the computing unit is configured for overlaying the detected elements on the displayed context image and for receiving input from the operator related to a next phase of the earth-moving operation, and configuring the machine control unit is also based on the input from the operator. Optionally, the user interface may comprise a touch-sensitive display on which the context image is displayed and on which the user input is received. The input from the operator for instance may comprise a selection of one or more of the detected elements.
According to some embodiments, the construction machine is a motor grader and the earth-moving operation comprises construction a ditch, wherein the phases are passes of the motor grader along a ditch line, including a plurality of cutting passes, and the detected elements comprise an edge and/or surface produced by a previous cutting pass. For instance, the earth-moving operation may comprise V-ditching. Optionally, the input from the operator may comprise one or more of the following:
In one embodiment, the input from the operator comprises selecting an edge and a surface produced by a previous cutting pass and selecting to adjust a cross slope of the surface for the next pass while maintaining the selected edge as a vertex of a plane to be shaped during the next pass.
According to some embodiments of the system, the computing unit is configured for
Optionally, the computing unit also may be configured for steering the construction machine to the desired line.
In some embodiments, the operator input comprises an offset of the desired line from a given line and the computing unit is configured to calculate the desired line based on the offset. For instance, if the given line is a spline, the computing unit may be configured to calculate another spline as the desired line based on the offset. In one example, the given line may be a line belonging to an existing ditch on a first side of the road, and the desired line is for another ditch to be constructed on the other side of the road, e.g. with the same distance to the road as the ditch on the first side.
According to some embodiments of the system, the measuring system is configured for continuously capturing the 3D measuring data and for continuously providing the captured 3D measuring data to the computing unit, wherein the computing unit is configured for continuously evaluating the received 3D measuring data for continuously detecting the elements and for continuously configuring the machine control unit while at least partially automatically performing a phase of the earth-moving operation.
A second aspect pertains to a construction machine, e.g. a grader, a dozer or an excavator, the construction machine comprising
A third aspect pertains to a computer-implemented method for configuring a machine control unit of a construction machine (e.g. the construction machine according to the second aspect), for performing an earth-moving operation, the earth-moving operation comprising a multitude of phases that are to be performed consecutively. The method comprises the following steps that are performed by a computing unit (e.g. the computing unit of the system according to the first aspect):
According to some embodiments, the method further comprises:
According to some embodiments of the method, the construction machine is a motor grader and the earth-moving operation comprises constructing a ditch, for instance by V-ditching. In this case, the phases are passes of the motor grader along a ditch line, including a plurality of cutting passes, and the detected elements comprise an edge and/or surface produced by a previous cutting pass. Optionally, the input from the operator may then comprise one or more of the following:
In one embodiment, the input from the operator comprises selecting an edge and a surface produced by a previous cutting pass and selecting to adjust a cross slope of the surface for the next pass while maintaining the selected edge as a vertex of a plane to be shaped during the next pass.
According to some embodiments, the method further comprises:
Optionally, the method may further comprise automatically steering the construction machine to the desired line. In one embodiment, the operator input comprises an offset of the desired line from a given line and the method comprises calculating the desired line based on the offset.
According to some embodiments of the method, the 3D measuring data is captured continuously and evaluated continuously for detecting the elements, and the machine control unit is configured continuously while at least partially automatically performing a phase of the earth-moving operation.
According to some embodiments of the method, the earth-moving operation comprises constructing a ditch, the desired line is a ditch line, e.g. a ditch edge line, the first phase is a marking pass along the ditch line, and the produced elements produced by the marking pass comprise an edge and/or surface. Optionally, the construction machine may be a motor grader, the earth-moving operation may comprise V-ditching, and/or the ditch line may be a ditch edge line, a ditch spline or a ditch edge spline.
A fourth aspect pertains to a computer programme product comprising programme code which is stored on a machine-readable medium, or being embodied by an electromagnetic wave comprising a programme code segment, and having computer-executable instructions for performing, particularly when executed on a computing unit of a system according to the first aspect, the method according to the third.
By way of example only, preferred embodiments will be described more fully hereinafter with reference to the accompanying figures, wherein:
In the construction machines 1 shown here, i.e. in graders and crawlers, current automation systems often comprise grade control systems that can be configured to maintain a desired cross slope. This is a simple functionality that can be used by operators during road construction and ditch construction. Typical cross slope systems only control one axis of a tool motion. They act as kinematic constraints, requiring the operator to control a depth of the tool 11, navigate the machine 1, and control other tool motions, e.g. associated with material management.
The operator of a motor grader starts with a light marking cut (see also
As shown in
As shown in more detail in
In context, an RCD in particular comprises some sensor arrangement to capture a 3D point cloud of the environment using a non-contact method. Typically, this includes determining distances to points in the environment using time-of-flight (ToF) measurements based on light waves (e.g. in the infrared spectrum), radio waves, or ultrasonic sound waves. However, stereoscopic vision sensors can provide point-cloud measurements of the environment using disparity, or the difference in position of data in the left and right images. For instance, each RCD may comprise at least two cameras and optionally other sensors, such as a laser scanner or an arrangement of ToF cameras. Reality-capture devices 3a,b of one or more measuring units 2 may be configured to contribute to generating the same 3D point cloud of the terrain, to detect obstacles and/or to track persons or animals entering a danger zone around the machine. Measuring units 2 may be provided on many parts of the construction machine, for instance connected to the tool 11, on the chassis 12 and/or on the cab 14.
The system for configuring the machine control unit comprises a computing unit and a user interface. Preferably, the user interface may be provided at or inside the cab 14 so that it may be used by an operator of the excavator 1 during operation. The user interface comprises a display for displaying live images and/or a graphical user interface (GUI). In the exemplary embodiment shown in
The user interface comprises input devices that preferably are provided inside the cab 14—e.g. comprising a touch-sensitive display (touchscreen) and optionally a stylus for use with the touchscreen. The computing unit can use the measuring data generated by the measuring unit (RCD data)—e.g. LiDAR data from a LiDAR scanner and image data from a multitude of cameras—for generating a 3D model, e.g. of the construction site or parts thereof, and optionally also for obstacle detection.
The construction site measuring system additionally may comprise at least one of the following components, which optionally may be provided in a common housing together with the measuring unit 2 or in a common housing together with the computing unit and/or the user interface:
For instance, if the measuring data comprises LiDAR data and image data, the image data can be used for colouring the LiDAR data and/or for optimizing a referencing of the LiDAR data by matching them with an image-based generated point cloud (e.g. generated by a visual simultaneous localization and mapping (VSLAM) algorithm). Also a feature tracking and/or feature recognition algorithm can help combining the LiDAR data to a consistent and well-referenced global point cloud. Similarly, the position data gained with the GNSS-antenna and the IMU data from the IMU can be used for a sensor fusion to obtain a higher precision when building a 3D model of the terrain. A VSLAM point cloud generation can also be supported on LiDAR data, in particular in such a way that the LiDAR data introduce scale and thereby increase stability of the algorithm. The LiDAR scanners may be configured for generating the LiDAR data while the two rotation axes of each scanner rotate faster than 0.1 Hz, particularly faster than 1 Hz, with a point acquisition rate of at least 300,000 points per second, particularly at least 500,000 points per second.
LiDAR scanners—as well as ToF cameras—may be capable to capture a 3D representation of the surrounding at a very fast pace. Therefore, with a moving construction machine it is possible to generate a coherent 3D point cloud based on a SLAM (Simultaneous Localization and Mapping) algorithm that uses the LiDAR data or the ToF data—either alone or in combination with image data from the cameras. Such localization and mapping is specifically advantageous if the construction machine is operating under a bridge or some other place shadowed from GNSS signals. The SLAM algorithm may be supported by at least one IMU providing IMU data that may be processed to stabilize the algorithm. In particular, all such fused sensor data can be processed by a Kalman filter.
Optionally, the system for configuring the machine control unit may also comprise at least one context camera, e.g. as a part of the construction site measuring system. Said context camera is any camera device that generates an image that provides context of the environment. In particular, the context camera need not provide any data for use in a sensor system. However, images provided by a stereoscopic camera are suitable for use as a context camera in addition to their use as a sensor through the calculation of a disparity map. The context camera may be provided in a common housing together with the RCD 3a,b of the measuring unit 2, at the semi-transparent display 4 or in a common housing together with the computing unit and the user interface. The context image data may be related to the RCD data of the construction site measuring system through the use of extrinsic parameters, which characterize the relative position of the context camera and RCDs as they are mounted on the construction machine 1. Images taken by the context camera are displayable to the operator of the machine on the semi-transparent display 4, on a touchscreen and/or on any other display of the user interface, particularly as a live video.
The use of one or more RCDs 3a,b mounted to the construction machine enables the entire work plan to be localized to the site based on the RCD data. If the RCDs are used for navigation or mapping (e.g. some variant of SLAM), then a design file can be localized to that navigation frame. If the machine is outfitted with GNSS or another global localization system, then the RCD data can be localized to global coordinates and the design file can be localized to the world frame as well. The design file can then be saved and output for reference or later use.
The system may further comprise a machine control unit 75 operatively coupled with the computing unit 71. The machine control unit 75 is configured to aid the operator in performing earth-moving operations. In particular, this includes at least partially automatically supervising coordinates of a tool of the construction machine, for instance so that the blade of the motor grader of
The system 70 may also comprise one or more reality capture devices (RCD) 82 and at least one context camera 83 that are operatively coupled with the computing unit 71. An image of the context camera 83 is displayable on a screen, e.g. a touchscreen, of the user interface 73. Alternatively, the RCD 82 and context camera 83 may be provided separately on the construction machine and be connectable to the system 70 so that they are operatively coupled with the computing unit 71.
The computing unit 71 is configured to receive, from the RCD 82, point cloud data of a surface of a construction site, particularly of a surface of the surrounding of the construction machine. The computing unit 71 is configured to receive, from the user interface 73, user input regarding a planned earth-moving operation on the construction site. The computing unit 71 determines 3D coordinates of the planned earth-moving operation and programmes the machine control unit 75 accordingly.
In order to construct a road with this cross section using the described system, it is necessary for the system to provide for spline offsetting. This allows the operator to use the inner edge of the ditch 30 to define the working direction of the road and then offset the shoulder edge and crown 40 of the road laterally in order to ensure they are parallel to the ditch line. The system should also be capable of generating and following curved splines for steering. This is illustrated in
A similar methodology can be used when slot grading with a crawler (as shown in Figure lb). The operator performs an initial marking cut and then configures the system to maintain the desired slope and navigate along the desired trench. The operator is then only responsible for managing vehicle load and shifting it from forward to reverse and vice versa until a windrow spreading operation is necessary. In a slot grading approach the operator sets a marking line and then uses the system to generate a windrow to either side of the crawler in order to develop a trench. The system then steers between the two windrows.
Aside from V-ditching, other embodiments of the method and system are can also be used for other applications that comprise iterative passes over the same route to construct a structure, such as road building or slot grading. Aside from motor graders, the disclosed method and workflow can be used with other construction machines, such as crawlers and wheel loaders.
During performing the method, a three-dimensional point cloud of a 3D surface is captured, for instance using one or more reality-capture devices (RCD) mounted on a construction machine to capture the surface surrounding the machine. This allows localizing the construction machine as well as recognizing features in the surrounding. In particular, the 3D point cloud may be captured continuously. Optionally, a context camera having a known position relative to the 3D point cloud and/or to a measuring system capturing the 3D point cloud captures context image data of the surrounding of the machine.
The illustrated method starts with performing a marking or registration pass, where the soil is cut just enough to provide a registration line to aid steering on subsequent passes. The operator can either perform 101 the marking pass manually, or start an assistance procedure 110 to be assisted by the machine control unit of the construction machine and/or a system.
In the latter case, i.e. during the assistance procedure 110, the operator indicates a desired line for the marking pass (“ditch line”), for instance a straight line or a spline. This indication may comprise an input of coordinates or a line drawn in a context image displayed on a touchscreen. Also, the operator may select a feature in the context image, e.g. a road or similar structure, and indicate an offset for the desired line from the feature. Alternatively or additionally, a data file comprising the coordinates can be uploaded to the system.
This indication of the desired ditch line is received 111 as input by the system. Since the RCDs mounted to the construction machine enable localization 112 of the construction machine, optionally the automation system can be set to steer 113 to the desired ditch line and/or set to position 114 the tool, e.g. a toe of a mould board, at the designated ditch line and maintain this position, e.g. at least a height and inclination, while the construction machine follows the ditch line—either steered by the operator, or controlled by the automation system. The marking pass is thus performed 115 fully or partly automatically. In the latter case, for instance, the automation system actuates blade side shift in order to keep the tool, e.g. the toe of the mould board, at the desired position while the operator controls the steering, circle shift and circle angles.
After the marking pass has been made (manually 101 or using the assistance procedure 110), a following pass can begin, e.g. as shown here a cutting procedure 120 that includes making a first deep cut. Using the RCDs mounted to the construction machine, the construction site measuring system can automatically detect 121 edges and surfaces of the previous pass, i.e. of the marking pass or a previous cutting pass. These edges and surfaces may be detected 121, e.g., in image data and/or point cloud data captured by the RCDs.
The system can then present these detected elements in a display, for instance by overlaying 122 them in a context image displayed on a touchscreen, and allow the operator to select previously shaped edges and/or surfaces in order to configure the automation systems for a present cutting pass. For instance, the operator can select the edge of the previous cut and configure the system to cut 25 mm below it for the next pass. The operator can select the previous surface and configure the system to maintain a parallel cut for the following pass. The cut might also be intended to change the cross slope of the surface but use the previously sloped edge as a vertex of the plane being shaped on the subsequent cutting pass. These selections are received 123 as operator input on the user interface, e.g. on a touchscreen showing a context image, to configure 124 the automation system for fully or partly automatically performing 125 the cutting pass. This cutting procedure 120 can be repeated until the last pass has been made and the construction of the ditch is completed.
Optionally, if a user interface displays a context image on a touchscreen that is also used for receiving the operator input (e.g. in steps 111 and 123), the input may include a selection of pixels of the context image or be interpreted as such a selection of pixels. Since the context camera capturing the context image data has a known position relative to the measuring system and/or the 3D measuring data, the selection of pixels (e.g. representing a desired ditch line) can be mapped to the 3D measuring data, e.g. to a surface of a 3D terrain model that is generated based on the 3D measuring data. Based on the mapping, it is then possible to determine 3D coordinates on the surface that can be provided to the machine control unit for at least partially controlling the earth-moving operation. This is described in detail in the European patent application No. 22180149.1.
The data captured by the RCDs can also be used to monitor the shed material as it forms a windrow. The volume of material in the windrow or the height of the windrow can be used to indicate to the operator when it is necessary to perform a spreading pass (see e.g. steps 7 and 8 in
Optionally, the method may include storing information regarding previously identified lines and surfaces in a memory and retrieving this information in a subsequent pass. For instance, the method may comprise passes for cleaning a bottom of a ditch (see e.g. steps 9 and 10 in
In some embodiments, the system may be configured to self-configure. In this mode, the operator may set the cross-slope and height for the marking pass, incrementing the height until the correct depth is reached for a marking pass. The localization capability of the RCDs enables a globally consistent height to be maintained as the vehicle navigates the site. The operator can then use an increment button to tell the system the desired depth of each cutting pass. The construction site measuring system can use its localization capabilities to determine when it was at the beginning position of the job. It can then combine the localization data with the configuration for the previous cutting pass in order to automatically detect and identify the edges and surfaces cut by the machine. It can then automatically set the desired steering spline or mould-board toe line based on the previous pass and operator indicated depth.
According to some embodiments, a method and system can also be used to monitor the construction of a V-ditch. Since the system knows the workflow, it can be used to track the steps used in the construction as well as the system configuration and work product for each pass. This data provides a real-time status for the construction job and can also be used to train new operators, using the system to designate the desired workflow for inexperienced operators.
Optionally, the system may include a mechanism for sensing a load of the construction machine. This may include sensing engine torque, watching engine speed changes, hydrostatic loop pressures and or displacements, transmission output torque, torque converter torque, etc. In this case, a depth control of the control unit may be set to a load dependent state where the system can automatically adjust a height of the tool based on the detected load.
Although aspects are illustrated above, partly with reference to some preferred embodiments, it must be understood that numerous modifications and combinations of different features of the embodiments can be made. All of these modifications lie within the scope of the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 22190684.5 | Aug 2022 | EP | regional |
