Patent Application
20240385327
The present disclosure relates to processing of measured data.
Conventionally, there is known a laser radar device configured to radiate a pulse of laser light into a target space of detection and detect an object in the target space based on the level of the reflection light. For example, Patent Literature 1 discloses a lidar configured to scan the peripheral space by appropriately controlling the emitting direction (scanning direction) of the light pulse to be repeatedly emitted and measure the return light thereby to generate point cloud data indicating peripheral object information such as distance information and reflectance information.
When generating three-dimensional data of a target object of measurement using a plurality of measurement devices such as a lidar, holes (missing parts) could occur in the three-dimensional data due to the presence of the blind spot of the measurement devices or the presence of a part to be sparsely measured depending on the measurement angle.
The present disclosure has been made to solve the above issues, and it is an object of the present invention to provide an information processing device capable of suitably generating interpolation data for interpolating the three-dimensional data of a target object of measurement.
One invention is an information processing device including:
Another invention is a control method executed by a computer, the control method including:
Still another invention is a program causing a computer to:
According to a preferred embodiment of the present invention, an information processing device includes: an acquisition unit configured to acquire three-dimensional data of a target object of measurement generated based on measurement data, which is generated by plural measurement device configured to measure a space where the target object of measurement is placed; a detection unit configured to detect an interpolation target spot which is missing in the three-dimensional data; a candidate position determination unit configured to determine, based on a gradient angle of the target object of measurement with respect to each of plural directions at the interpolation target spot, a candidate position to interpolate the interpolation target spot with respect to each of the plural directions; and an interpolation data generation unit configured to generate interpolation data corresponding to the interpolation target spot based on the candidate position. According to this aspect, the information processing device can suitably generate interpolation data of the three-dimensional data of the target object of measurement measured by plural measurement devices.
In one aspect of the information processing device, the candidate position determination unit is configured to estimate the gradient angle in an arbitrary direction from the plural direction, based on measured positions of the target object of measurement based on the measurement data at positions along the arbitrary direction. According to this aspect, the information processing device can accurately estimate the gradient angle for each direction. In some embodiments, the candidate position determination unit is configured to calculate the gradient angle based on any one of a principal component analysis, a linear approximation, or a polynomial approximation.
In another aspect of the information processing device, the candidate position determination unit is configured to determine the gradient angle based on an inclination of an inclined plane of the target object of measurement stored in a storage unit. According to this aspect, the information processing device can accurately determine the gradient angle for each direction.
In still another aspect of the information processing device, the interpolation data generation unit is configured to generate the interpolation data based on a candidate position which is a highest position among the candidate positions for each interpolation target spot. According is to this aspect, the information processing device generate highly accurate interpolation data even if the interpolation target spot is a valley or the like of the target object of measurement.
In still another aspect of the information processing device, the information processing device further includes a division unit configured to divide a horizontal plane of a space indicated by the three-dimensional data into grids, wherein the detection unit is configured to detect, as the interpolation target spot, a grid in which a measured position indicated by the measurement data does not exist. According to this aspect, the information processing device can accurately detect the interpolation target spot. In some embodiments, the plural directions are directions on the horizontal plane, and are determined based on directions in which the grids are arranged vertically, horizontally, or diagonally.
In still another aspect of the information processing device, the target object of measurement is a pile in which plural mounds are formed. When a pile in which a plurality of mounds are formed is used as a target object of measurement, a blind spot of a measurement device occurs in a valley part between mounds. In this case, the information processing device can suitably generate interpolation data accurately representing the blind spot. In some embodiment, the target object of measurement is a pile of powder substances.
According to another preferred embodiment of the present invention, there is provided a control method executed by a computer, the control method including: acquiring three-dimensional data of a target object of measurement generated based on measurement data, which is generated by plural measurement device configured to measure a space where the target object of measurement is placed; detecting an interpolation target spot which is missing in the three-dimensional data; determining, based on a gradient angle of the target object of measurement with respect to each of plural directions at the interpolation target spot, a candidate position to interpolate the interpolation target spot with respect to each of the plural directions; and generating interpolation data corresponding to the interpolation target spot based on the candidate position. By executing this control method, the computer can suitably generate the interpolation data of the three-dimensional data of the target object of measurement measured by plural measurement devices.
According to still another preferred embodiment of the present invention, there is provided a program causing a computer to: acquire three-dimensional data of a target object of measurement generated based on measurement data, which is generated by plural measurement device configured to measure a space where the target object of measurement is placed; detect an interpolation target spot which is missing in the three-dimensional data; determine, based on a gradient angle of the target object of measurement with respect to each of plural directions at the interpolation target spot, a candidate position to interpolate the interpolation target spot with is respect to each of the plural directions; and generate interpolation data corresponding to the interpolation target spot based on the candidate position. By executing the program, the computer can suitably generate the interpolation data of the three-dimensional data of the target object of measurement measured by plural measurement devices. In some embodiments, the program is stored in a storage medium.
Hereinafter, a preferred embodiment of the present invention are described below with reference to drawings.
The information processing device 1 is electrically connected to the sensor group 2 and processes data outputted by various sensors included in the sensor group 2. In this example, the information processing device 1 generates, as three-dimensional data representing the three-dimensional shape of the target object of measurement, point cloud data (also referred to as “integrated point cloud data”) obtained by integrating (accumulating) the point cloud data outputted by a plurality of installed lidars 3. For example, the integrated point cloud data is data which expresses measured positions of the target object of measurement as points in a three-dimensional coordinate system (also referred to as “reference coordinate system”) which uses a predetermined position in the space of the warehouse as its origin. When integrating data outputted from the plurality of lidars 3, the information processing device 1 transforms the point cloud data outputted by each lidar 3 based on the installation position and posture of each lidar 3 measured in advance and then adds up each data in the reference coordinate system after the transformation. Hereafter, the X-axis and Y-axis of reference coordinate system are parallel to the horizontal plane and the Z-axis of the reference coordinate system is along with the vertical height direction perpendicular to the horizontal plane. In some embodiments, the information processing device 1 may perform the integration (synthesis) of the point cloud data outputted by the lidars 3 based on any technique which is not limited to the above-described example.
A lidar 3 discretely measures the distance to an external object by emitting a pulse laser which is an infrared laser while changing the emitting angle within a predetermined angle range (field of view) in the horizontal and vertical directions. In this case, the lidar 3 is equipped with a radiation unit for radiating a laser light while changing the irradiation direction (i.e., the scanning direction), a light receiving unit for receiving the reflected light (scattered light) of the radiated laser light, and an output unit for outputting data based on the received light signal outputted by the light receiving unit. The measurement data generated by the lidar 3 for each irradiation direction of the pulsed laser is generated based on the irradiation direction corresponding to the laser light received by the light receiving unit and the response delay time (i.e., Time of Flight) of the laser light which is identified by the received light signal described above. Then, for each scanning cycle, the lidar generates generates a frame of point cloud data that is data cloud corresponding to irradiated points in the field of view of the lidar 3 (i.e., the irradiation range of the pulse laser). The lidar 3 is an example of the “measurement device” in the present invention. The lidar 3 is not limited to the above-described scan type lidar and may be a flash type lidar configured to generate three-dimensional data by diffusing and radiating a laser light to the field of view of the two-dimensional sensor array. Hereafter, the point (and its measurement data) measured through irradiation with the pulse laser emitted by the radiation unit is also referred to as “measured point”.
In this example, a required number of the lidars 3 are installed so as to include the entire pile 4 in the entire field of view obtained by combining the field of view of each lidar 3. Each lidar 3 is provided at a position to overview the pile 4.
The sensor group 2 may include various external sensors or/and internal sensors in addition to the lidars 3. For example, the sensor group 2 may include a GNSS (Global Navigation Satellite System) receiver or the like required to generate the position data.
The interface 11 performs an interface operation related to the transfer of data between the information processing device 1 and any external device. In this example, the interface 11 acquires the output data from the sensor group 2 such as the lidars 3, and supplies it to the controller 13. Examples of the interface 11 include a wireless interface, such as a network adapter, for is performing wireless communication, and a hardware interface, such as a cable, for connecting to an external device. The interface 11 may also perform interface operations with various peripheral devices such as an input device, a display device, a sound output device, and the like.
The memory 12 is configured by various volatile and non-volatile memories such as a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk drive, a flash memory, and the like. The memory 12 stores a program for the controller 13 to perform a predetermined process. The program executed by the controller 13 may be stored in a storage medium other than the memory 12. Further, the memory 12 stores integrated point cloud data and interpolation data which are three-dimensional data of the pile 4 generated by the controller 13.
The controller 13 includes one or more processors, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a TPU (Tensor Processing Unit), and controls the entire information processing unit 1. In this case, the controller 13 executes a program stored in the memory 12 or the like to perform various processing described later. In this example, based on the point cloud data outputted by the lidars 3, the controller 13 generates the integrated point cloud data which is three-dimensional data of the pile 4. Furthermore, the controller 13 detects a spot (also referred to as “interpolation target spot”) where the data is missing in the generated integrated point cloud data as an interpolation target, and generates interpolation data representing the position of the pile 4 to be estimated at the detected interpolation target spot.
The controller 13 functions as the “acquisition unit”, “detection unit”, “candidate position determination unit”, “interpolation data generation unit”, and a computer configured to execute the program.
The process performed by the controller 13 is not limited to being implemented in software by a program, but may be implemented by any combination of hardware, firmware, and software. The process performed by the controllers 13 may also be implemented using any user programmable integrated circuit such as a FPGA (Field-Programmable Gate Array) and a microcomputer, for example. In this case, the integrated circuit may be used to implement a program that the controller 13 executes in this example.
First, a description will be given of the interpolation target spot with reference to
As described above, when the lidars 3 are fixedly installed, missing parts (holes) in the integrated point cloud data obtained by integrating the point cloud data each lidar 3 outputs could occur. Therefore, the information processing device 1 detects such a part as an interpolation target spot and generates interpolation data corresponding to the interpolation target spot.
Next, a description will be given of a specific method of generating interpolation data. after dividing the X-Y plane into predetermined-size grids, the information processing device 1 calculates a height with respect to each grid based on the integrated point cloud data. Then, the information processing device 1 detects a grid, in which any measured point of the integrated point cloud data does not exist, as an interpolation target spot, and then estimates the height for each interpolation target spot, on the basis of the gradient angles of the pile 4 estimated in respective directions. In the following, a grid to be an interpolation target spot is referred to as the “interpolation target grid”. In addition, the height of the pile 4 with respect to each grid specified based on the integrated point cloud data is referred to as the “measurement height”, and the estimated height of the interpolation target grid in the pile 4 is referred to as the “estimated height”.
First, the information processing device 1 divides the X-Y plane (i.e., the horizontal plane) of the space in which the pile 4 as the target object of measurement exists into predetermined-size grids by placing a mesh at predetermined intervals for the X-axis and Y-axis. The size of the grids in this instance is, for example, 20 cm or 10 cm, and is determined according to the required resolution of the three-dimensional data of the pile 4.
Next, the information processing device 1 divides the integrated point cloud data into data in units of grids, and calculates the measurement height with respect to each grid based on the integrated point cloud data. Here, the information processing device 1 calculates the measurement height obtained by averaging the height of each measured point in the each grid.
Next, the information processing device 1 detects a grid (i.e., a grid for which the measurement height cannot be calculated) in which there is no measured point as an interpolation target grid. Then, the information processing device 1 calculates the estimated height of the interpolation target grid, based on gradient angles of the pile 4 in a plurality of directions and the heights estimated from the respective gradient angles. The height of the pile 4 estimated from the gradient angle with respect to each direction is a candidate for the estimated height, and is also referred to as “candidate height” hereafter.
For example, in the case of determining the candidate height in the direction along the arrow A1, the information processing device 1 acquires the measurement heights and X-Y coordinate positions of a predetermined number of neighboring grids (grids including the grid Gg) which exist in the direction along the arrow A1. Then, based on acquired measurement heights and X-Y coordinate positions (e.g., the grid center position) of the grids, the information processing device 1 estimates the gradient angle and the candidate height in the direction along the arrow A1 according to the geometric calculation method.
Then, the information processing device 1 calculates the candidate heights corresponding to the directions along the respective arrows A1 to A8, and selects a candidate height indicating the highest position among the calculated candidate heights as the estimated height in the interpolation target grid. According to this method, the information processing device 1 can estimate the height with high accuracy even for such an interpolation target grid near a valley between mounds of the pile 4.
In this instance, the information processing device 1 calculates the gradient angle (see the line 70) of the pile 4 estimated based on the measurement heights and X-Y coordinate positions of the grids 1 to 3 for the first direction in which the grids 1 to 3 exist, and calculates the candidate height 81 of the grids 4 and the candidate height 82 of the grids 5 in the first direction based on is the gradient angle. Similarly, the information processing device 1 calculates the gradient angle (see the line 71) of the pile 4 estimated based on the measurement heights and X-Y coordinate positions of the grids 6 to 8 for the second direction in which the grids 6 to 8 exist, and calculates the candidate height 83 of the grids 5 and the candidate height 84 of the grids 4 in the second direction based on the gradient angle. In this way, the information processing device 1 calculates the candidate heights of the grid 4 and the grid 5 for all the directions including the first direction and the second direction.
Then, the information processing device 1 determines a candidate height (the candidate height 81 in
As described above, the information processing device 1 calculates the candidate heights of the interpolation target grid in a plurality of directions based on the measurement heights of the other grids, and determines the highest candidate height among them as the estimated height of the interpolation target grid. Thus, it is possible to accurately calculate the estimated high of the interpolation target grid existing in the vicinity of the valley of the pile 4.
First, the information processing device 1 acquires each point cloud data from a plurality of lidars 3 (step S11). Then, the information processing device 1 generates the integrated point cloud data obtained by integrating the point cloud data acquired from the plurality of lidars 3 (step S12). In this case, the information processing device 1 transforms each point cloud data outputted from the lidars 3 into data in the reference coordinate system, and acquires the data obtained by integrating the transformed point cloud data as the integrated point cloud data.
Next, the information processing device 1 divides the space of the reference coordinate system into predetermined-size grids on the horizontal plane, and then calculates a measurement height with respect to each grid based on the integrated point cloud data, and detects a grid for which the measurement height cannot be calculated as the interpolation target grid (step S13).
Then, for each interpolation target grid, the information processing device 1 calculates the candidate height with respect to each direction (step S14). In this case, the information processing device 1 sets a candidate height for each of the eight directions shown in
Then, for each interpolation target grid, the information processing device 1 determines the highest candidate height as the estimated height (step S15). Then, the information processing device 1 generates interpolation data indicating the estimated high for each interpolation target grid (step S16).
Next, specific examples (the first calculation example to the third calculation example) of a method of calculating the gradient angle will be specifically described.
The first calculation example is a method based on a pre-measurement, in which the inclination of the inclined plane of the mound formed when the target object of measurement is naturally stacked (such as when dropping the target object of measurement with a device or the like from above) is measured in advance, and the measured value is stored in advance in the memory 12 of the information processing device 1. In this case, pre-measurement for each type of the target object of measurement is conducted, and the measurement results of the gradient angles are stored in the form of a table or the like in the memory 12. In this case, by referring to the memory 12, the information processing device 1 can suitably calculate the gradient angle.
The second calculation example is a method using principal component analysis, in which the information processing device 1 sets an interpolation target spot as an attention grid in order, and then sets an analysis area (an area having a width and a thickness in the scan direction) which is elongated along the vertical direction with respect to the direction of scanning whether or not there is a grid, and performs principal component analysis on a set of grid points included in the analysis area. Then, the information processing device 1 calculates, as the gradient angle at the scan direction in the attention grid, the gradient when the eigenvalue of the second principal component and eigenvalue of the third principal component become small (i.e., the distribution of the grid points can be determined to be linear). In this case, when the distribution of grid points is not determined to be linear, it is excluded from the target of interpolation calculation. In some embodiments, upon determining that the calculated gradient is up-slope with respect to the attention grid, it may be excluded from the target of interpolation calculation.
The third calculation example is a method using a linear approximation or a polynomial approximation. In this method, for the attention grid set in the same way as the second calculation example, the information processing device 1 sets an analysis area (an area having a width and a thickness in the scan direction) which is elongated along the vertical direction with respect to the direction of scanning whether or not there is a grid, and then applies linear approximation or polynomial approximation to a set of grid points included in the analysis area. It calculates the slope of the straight line as the gradient angle at the scan direction in the attention grid in the case of linear approximation, while it calculates the slope of the tangent at the end point of the approximate curve in the polynomial approximation as the gradient angle at the scan direction in is the attention grid in the case of polynomial approximation. In the case of up slope with respect to the target grid, it may be excluded from the interpolation calculation. In some embodiments, upon determining that the calculated gradient is up-slope with respect to the attention grid, it may be excluded from the target of interpolation calculation.
Here, the second calculation method will be supplemented. In the second calculation method, the information processing device 1 calculates the covariance matrix for the set of grid points distributed in the space by principal component analysis, and then applies eigenvalue analysis to the covariance matrix to thereby obtain three eigenvectors (vectors orthogonal to each other) and three eigenvalues corresponding to the respective three eigenvectors. The magnitude of the eigenvalue represents the distribution state of points with respect to the direction of the corresponding eigenvector. When the points are linearly distributed, only the first component protrusively becomes a large value (i.e., the second component and the third component become small values). Therefore, the information processing device 1 calculates the angle formed between the direction of the eigenvector corresponding to the first component and the X-Y plane as the gradient angle.
Next, the third calculation method will be supplementally described. In the third calculation method, based on the linear approximation or the polynomial approximation, the information processing device 1 approximates a set of grid points distributed in the space by a linear equation (i.e., a primary expression) or a polynomial equation using the least squares method. Then, the information processing device 1 calculates the angle formed between the direction vector of the straight line and the X-Y plane as the gradient angle in the case of the linear approximation, and calculates the angle formed between the direction vector of the tangent at the end point of the approximate curve and the X-Y plane as the gradient angle in the case of the polynomial approximation.
The directions in which the candidate heights are calculated for the interpolation target grid are not limited to the eight directions indicated by the arrows A1 to A8 shown in
As described above, the controller 13 of the information processing device 1 according to the present embodiment is configured to acquire integrated point cloud data which is three-dimensional data of a target object of measurement generated based on point cloud data, which is generated by a plurality of lidars 3 configured to measure a space where the target object of measurement is placed. Then, the controller 13 detects an interpolation target grid which is missing in the integrated point cloud data. Then, based on a gradient angle of the target object of measurement with respect to each of plural directions at the interpolation target spot, the controller 13 is configured to determine a candidate height to interpolate the interpolation target grid with respect to each of the plural directions. Then, the controller 13 is configured to generate interpolation data corresponding to the interpolation target grid based on the candidate height. Thus, the information processing device 1 can suitably generate interpolation data to interpolate the holes of the three-dimensional data of the target object of measurement generated due to the blind spot of the lidar 3 or the like.
In the embodiments described above, the program is stored by any type of a non-transitory computer-readable medium (non-transitory computer readable medium) and can be supplied to a control unit or the like that is a computer. The non-transitory computer-readable medium include any type of a tangible storage medium. Examples of the non-transitory 5 computer readable medium include a magnetic storage medium (e.g., a flexible disk, a magnetic tape, a hard disk drive), a magnetic-optical storage medium (e.g., a magnetic optical disk), CD-ROM (Read Only Memory), CD-R, CD-R/W, a solid-state memory (e.g., a mask ROM, a PROM (Programmable ROM), an EPROM (Erasable PROM), a flash ROM, a RAM (Random Access Memory)).
While the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. Namely, the present invention includes, of course, various modifications that may be made by a person skilled in the art according to the is entire disclosure including claims and technical ideas. In addition, all Patent and Non-Patent Literatures mentioned in this specification are incorporated by reference in its entirety.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2022-007785 | Jan 2022 | JP | national |
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2023/000778 | 1/13/2023 | WO |
