Patent Application
20230043666
The present invention relates to an information processing device.
In the above-mentioned technical field, Patent Literature 1 discloses a technique for detecting a place where chips generated in processing adhere and accumulate.
However, since the technique described in the above document detects chips by a heat detector instead of a visual sensor such as a camera, the detection accuracy of chips is not sufficient.
It is an object of the present invention to provide a technique for solving the above problem.
In order to achieve the above object, the present invention has the configuration described in the claims.
According to the present invention, objects in the machine tool can be detected with high accuracy.
Embodiments will now be described in detail by way of example with reference to the drawings. However, the constituent elements described in the following embodiments are merely illustrative, and the scope of the present invention is not intended to be limited thereto.
An information processing device 100 according to a first embodiment will be described with reference to
The information processing device 100 includes a processing unit 101. The processing unit 101 detects the object 130 from the inside of the machine tool 110 by using image data 120 obtained by imaging the inside of the machine tool 110.
The processing unit 101 associates the image data 120 with grid information that divides the image data into a plurality of grids including a first grid, a second grid, a third grid, and a fourth grid, and also to area information that divides the inside of the machine tool into a plurality of area including a first area (area B) including the first grid and the third grid and a second area (area A) including the second grid and the fourth grid.
A plurality of grids are used to detect the presence degree of an object in one grid (object presence or absence, object amount, and object existence probability, among others). The plurality of areas are areas divided according to the structures in the machine tool, and are used to move the object in an area out of the area by discharging fluid in each area. Therefore, when the grid information and the area information are superimposed on the image, a plurality of grids are included in one area. For example, in
For example, when an object is detected in the first grid, and the first grid in which the object is detected is included in the first area (area B), the processing unit 101 transmits a signal to the machine tool for instructing the discharge of fluid in order to move object remaining on a structure in the machine tool corresponding to the first area (area B).
According to the above configuration, since the object can be accurately detected from the image obtained by imaging the inside of the machine tool, the object in the machine tool can be accurately moved. As a result, it is possible to reduce the time for removing the object in the machining area of the machine tool. Furthermore, this eliminates the necessity of the removal work of the object by a human operator or reduces the number of times of the removal work of the object by a human operator, thereby increasing the operation rate of the machine tool.
A machine tool system 200 according to the second embodiment will be described with reference to
The machine tool 202 of this embodiment is a machine for performing machining process such as cutting and grinding on a workpiece 221 made of metal, wood, stone, resin, or the like by using a tool 223 attached to a tool spindle 222. However, the present invention is not limited to this form, and a machine tool for other removal processing or a machine tool for additional processing may be used. Further, the machine tool may be a combined machine tool in which these functions are combined. A coolant discharging unit 224 discharges coolant 225 to cool the tool 223 and the workpiece 221. Further, the coolant discharging unit 224 of the present embodiment may discharge the coolant 225 to effectively move the object. The object is, e.g., chips 226 generated during cutting the workpiece 221 with the tool 223. It should be noted that a liquid such as coolant or a gas such as air may be used, i.e., any fluid may be used. Therefore, instead of the coolant discharging unit 224, a fluid discharging unit such as an air blow unit for blowing air may be used. The fluid discharging unit may include a nozzle or the like capable of swinging in the X- and Z-directions.
The machine tool 202 further includes a camera 227, a control unit 228, and a coolant pump 229.
The camera 227 is an imaging unit that acquires image data by imaging the inside of the machine tool 202. Here, the camera 227 is provided as an example of the imaging unit.
The information processing device 201 includes a chip processing unit 211, an image reliability determination unit 212, and a notification unit 213.
The chip processing unit 211 detects the chips 226 which are removal objects to be removed from the inside of the machine tool 202 by using image data 210 obtained by imaging the inside of the machine tool 202 (in particular, the machining area) by the camera 227. Specifically, the image data 210 is divided into small areas (made into grids), and the presence or absence of the chips 226, e.g., the presence degree of chips is determined for each grid which is a small area. In the present embodiment, the image is divided for each grid, but the present invention is not limited thereto. The image data may not be divided, and the image and grid information divided into grids may be associated with each other.
Apart from the grid information, the chip processing unit 211 associates the image data with area information to be divided into a plurality of areas. The plurality of areas are areas divided according to the structures in the machine tool 202, and are areas A, B, and the like shown in
The chip processing unit 211 detects the presence degree of chips. In other words, the chip processing unit 211 performs chip detection. The concept of the presence degree of the chip includes, in addition to the presence or absence of the chips 226 in one grid (small area), chip size, chip amount, chip shape, chip type, and chip existence probability, among others in one grid (small area). The information on the size of chips, which presupposes the existence of chips, is an example of the presence degree of chips. The chip processing unit 211 may detect a plurality of pieces of information such as a chip size and a chip type.
The chip processing unit 211 extracts the feature of the image data 210. The feature extracted by the chip processing unit 211 includes the information amount, frequency component, contrast, and luminance distribution of the captured image. The chip processing unit 211 has a function of improving detection accuracy by performing machine learning with a neural network on a large number of training images about which the presence or absence of chips is known.
The control unit 228 controls imaging with the camera 227 while controlling the coolant pump 229. For example, the control unit 228 controls the coolant pump 229 to stop the discharge of the coolant 225, controls the camera 227 to perform imaging, and controls the coolant pump 229 to discharge the coolant again after imaging. With respect to the coolant discharge operation, the processing unit 301 generates a signal that instructs the discharge of fluid in order to move objects remaining on a structure (e.g., the pallet) in the machine tool 202 corresponding to an area (e.g., area A) that is divided according to the structures in the machine tool 202. The control unit 228 performs polling process for periodically inquiring a signal to the processing unit 301 and controls the coolant pump 229 to cause discharge of the coolant 225 upon acquiring a signal of the fluid discharge command. In addition, the control unit 228 controls the fluid discharging unit including the nozzle and the swinging mechanism in accordance with the fluid discharge command, and causes the nozzle to swing toward the area to be cleaned. Similarly in the case of the first embodiment, the processing unit 101 generates a signal that instructs the discharge of fluid in order to move objects remaining on a structure (e.g., the pallet) in the machine tool 110 corresponding to an area (e.g., area A) that is divided according to the structures in the machine tool 202.
Further, the control unit 228 determines the timing of imaging for chip detection according to a machining program (G code) or a user operation. The control unit 228 may turn off the coolant pump 229 to image the chips, e.g., at a brake in machining during running (a timing at which another surface of the workpiece is cut), at a timing of replacing a tool by an ATC, or at a timing instructed by a user.
The control unit 228 controls the coolant discharging unit 224 on the basis of the position of the chips detected by the chip processing unit 211 (the grid on which the chips are detected), the amount of chips in one grid, and the amount of chips in one area, among others to effectively discharge the coolant.
To move the chips 226 toward a shooter 230, the control unit 228 controls the coolant discharging unit 224 so that the coolant flushes the chips 226 as if the chips 226 are swept out by a broom.
The chip processing unit 211 might make an erroneous determination (inference) due to brightness/darkness of lighting, contrast, and mist, among other causes. If the inference is wrong, chips might not be removed even though they are present, or conversely, wasteful removal might be attempted even though chips are absent.
Therefore, the image reliability determination unit 212 calculates the reliability of the image (image reliability) from the characteristic factors of the image data that affect the inference result as an indicator indicating whether the inference by the chip processing unit 211 is appropriate. In other words, the image reliability determination unit 212 determines the image reliability indicating whether the image data 210 is not suitable for detecting the chip 226 on the basis of the feature of the image data 210.
The notification unit 213 provides a notification indicating that the image data 210 is not suitable for the detection of the chips 226 on the basis of the determination result of the image reliability determination unit 212 to the user. A display control unit 320 performs processing for displaying image data (e.g., the upper portion of
The image reliability determination unit 212 uses three indicators of brightness/darkness, blur level, and density of mist as the image reliability. Specifically, the image reliability determination unit 212 includes a brightness/darkness determination unit 231, a blur determination unit 232, and a mist determination unit 233, and determines whether the estimation accuracy of chip detection is decreased due to the effects of brightness/darkness, blur, and mist. Here, the threshold values for the brightness/darkness determination, the blur level determination, and the mist determination may be changed according to the environment.
Each image reliability is calculated by image characteristic factors such as luminance value, frequency distribution, contrast, and combinations thereof. Then, it is determined whether the chip processing unit 211 functions appropriately on the basis of the magnitude of the image reliability value (0 to 1). When the image reliability of the image data to be determined is low, inappropriate chip removal behavior can be avoided by not using the image data. If the image reliability is decreased due to sudden and transient factors only, the image reliability is expected to improve over time.
Although the present embodiment uses the above three indicators as the image reliability, the present invention is not limited thereto, and other indicators (e.g., sharpness, contrast, number of gradations, noise amount, number of pixels with white out, number of pixels with black out, and peak signal-to-noise ratio) may be used.
If the image reliability has been decreased for a certain period of time, a notification indicating this is provided to the user by, e.g., displaying an alert. Thus, the user can recognize that the chip processing unit 211 does not function appropriately, and can attempt to improve the environment. Specifically, the chip processing unit 211 can be made to function appropriately by performing, e.g., maintenance of the camera 227, review of imaging conditions, cleaning of the lens, change of lighting, installation and operation of the mist collector.
The brightness/darkness determination unit 231 determines that the image data is not suitable for chip detection because the image data is too bright or too dark. Specifically, the brightness/darkness determination unit 231 uses the luminance of the image to monitor the brightness/darkness of the image. More specifically, the average value of the luminance is used as an indicator value of brightness/darkness and compared with a threshold value. When the state in which the indicator value of brightness/darkness is smaller or larger than the threshold value continues for a certain period of time, the notification unit 213 is made to provide a notification indicating that the image data is not suitable for the detection of chips due to being too bright or too dark. In this case, the notification unit 213 provides a notification such as “The image is too dark for chip detection. Check the operation status of the light.” or “The image is too bright for chip detection. Check the aperture of the camera.”
The blur determination unit 232 determines that the image data is not suitable for detection of chips due to blur of the image data. Specifically, the blur level of an image is monitored by using a value obtained by normalizing the maximum value (maximum gradient) of the image data subjected to a differential filter (Sobel filter) to a range of 0 to 1. Alternatively, blur may be determined by acquiring an optical transfer function (OTF, so-called response function) of an image data, and determining how much contrast can be maintained at what spatial frequency (amplitude characteristics, MTF: modulation transfer function).
When the indicator of the blur level of the image remains below a predetermined threshold value for a certain period of time, the notification unit 213 provides a notification indicating that the image is not suitable for detection of the removal object due to lack of focus. In this case, the notification unit 213 provides a notification such as “Chip detection is impossible due to blur. Check the lens of the camera.”
The mist determination unit 233 determines that an image captured in this state is unsuitable for detecting chips due to the density of mist generated when a workpiece or a tool is cooled by coolant discharged in the machine tool.
Specifically, the mist determination unit 233 calculates an indicator of the density of the mist from a combination of the luminance value, frequency distribution, and contrast of the image. In particular, in the present embodiment, the product of three features of “darkness degree”, “edge amount”, and “contrast” is used as a mist indicator. This is because any one of these features decreases as the degree of mist increases. The “darkness degree”, “edge amount”, and “contrast” are calculated from the original image data before grid division.
Increased mist will lead to erroneous determination that there is no chip so that when the indicator value of the mist is smaller than the threshold value for a certain period of time, the notification unit 213 provides a notification indicating that the image is not suitable for detection of chips due to the density of the mist of the coolant. For example, the notification unit 213 provides a notification such as “Chip detection is impossible due to dense mist. Check the operation status of the mist collector.”
Here, as an example, three features of “darkness degree”, “edge amount”, and “contrast” are defined as follows.
“Darkness degree”=a value calculated by normalizing average luminance in pixels to a range of 0 to 1 and then subtracting the obtained value from 1.
“Edge amount”=a value calculated by normalizing Sobel max, which is the maximum value (maximum difference) of image data subjected to a differential filter (Sobel filter), to a range of 0 to 1 by setting the theoretical maximum value to 1140
“Contrast”=a value obtained by normalizing the work amount of contrast correction (luminance histogram flattening) to a range of 0 to 1 by setting the maximum value to 255
The work amount of contrast correction (luminance histogram flattening) is obtained by flattening the luminance histogram and quantifying the lack of gaps in the histogram after the processing (the number of luminance values whose frequency is not zero) (
It is also possible to use the variance, standard deviation, or average value of the differential filter output as the “edge amount”. As the “contrast”, it is also possible to use luminance dispersion, luminance standard deviation, and Michelson contrast (a value calculated by using maximum luminance and minimum luminance of all pixels).
Furthermore, three features of “darkness degree”, “edge amount”, and “contrast” may be weighted and then integrated.
By performing the brightness/darkness determination, blur determination, and mist determination in the manner described above, it is possible to obtain a robust determination result by suppressing the influence of image variations.
The processing unit 301 executes various kinds of operational processing to implement various kinds of functions. The storage 302 stores various data and functions as a working area when the processing unit 301 performs operational processing. Specifically, the storage 302 stores a chip detection program 321, determination parameters 323, determination results 324, cleaning conditions 325 (including cleaning path information according to the number of times of cleaning and the area), an in-machine image 327, an image reliability determination program 328, and notification messages 329.
The touch panel 303 has both an input function for accepting an instruction input from a user and a display function for displaying a chip determination result or the like.
The chip detection program 321 uses a chip determination model 322 formed by applying the determination parameters 323 to a learning/inference model. The determination parameters 323 are parameters for detecting chips, and are obtained by using a partial image (or an image portion corresponding to one grid) and information on chips as training data and making the learning/inference model learn the data in advance. The determination parameters 323 are stored for each of a plurality of different mesh sizes. Further, learning parameters which affect characteristics such as learning efficiency of the chip determination model 322 may be separately set in the storage 302.
As shown in
Referring back to
The processing unit 301 is implemented by, e.g., a central processing unit (CPU). By executing the chip detection program 321, the processing unit 301 can function as an area dividing unit 311, an imaging control unit 312, a grid dividing unit 313, a chip processing unit 314, and a coolant control unit 318. The processing unit 301 can also function as the image reliability determination unit 212 and the notification unit 213 by executing the image reliability determination program 328.
The imaging control unit 312 outputs an imaging control signal to the camera 227 at a preset imaging timing, acquires the image 327, and stores the image 327 in the storage 302. The imaging timing is, e.g., a timing between operational steps, a timing of a tool exchange, a certain time interval, and a timing at which a predetermined machining amount is reached.
The grid dividing unit 313 divides the image data of the in-machine image into grids so as to be handled as a partial image, and associates the image data with the grid information. Thus, it is possible to detect chips for each grid, thereby accurately detect chips. The grid dividing unit 313 divides the image into partial images of a size determined on the basis of each size of the chip, workpiece, and jig. As a result, the edge shapes of the workpiece and the jig can be easily distinguished from the chips, and erroneous detection can be reduced. The grid is not limited to a square grid pattern (a square mesh), but may be a diamond, triangle, and honeycomb, among other shapes. Further, it is not necessary to divide all of the images into partial images, and only areas in which chips need to be detected may be divided.
As shown in
By efficiently moving the chips toward the shooter, it is possible to reduce the accumulation of chips in a short time. The chips moved toward the shooter are discharged and removed to the chip box outside the machine tool via the chip conveyor. For example, the area B in
The chip processing unit 314 executes preprocessing for improving the determination accuracy, such as noise removal processing and image size conversion processing, for each partial image. Thereafter, the chip processing unit 314 executes the chip detection program 321 for each of the partial images to determine the presence or absence of chips by using the chip determination model 322, and stores the determination result 324.
The chip processing unit 314 uses, e.g., a convolutional neural network (CNN) which is a model specialized in image classification among deep learning techniques relatively resistant to environmental changes. However, other machine learning algorithms may be used.
When a machine learning algorithm such as a support vector machine (SVM) is used, filtering process or the like is performed on the partial images to calculate a shape feature of each partial image. The class may be determined for each of the partial images by inputting the shape feature into a learned chip determination model. In addition, non-machine learning algorithms such as deterministic algorithms may be used. For example, as the number of chips increases, partial images become more complex and tend to have higher image frequency components. Therefore, the chip processing unit 314 may perform frequency analysis such as fast Fourier transform (FFT) and compare the spectral statistics of each partial image with the determination parameters to determine the class. An entire area image without chips may be prepared in advance, and the difference image between this and the entire area image to be determined may be meshed and subjected to a fast Fourier transform to eliminate the environment-dependent components.
The chip processing unit 314 reads the determination result 324 of each partial image from the storage 302 and displays it on the touch panel 303.
Specifically, each partial image is displayed in a manner in which the determination result can be identified (e.g., with a different color for each different determination result).
For example, each grid is classified into the following four groups according to the values of the determination probability P0 of class 0 (no chip exist) and the determination probability P1 of class 1 (chips exist), and is masked with different colors.
Group 1: P1 is 0 to 30% (P0 is 70 to 100%): red
Group 2: P1 is 30 to 50% (P0 is 50 to 70%): blue
Group 3: P1 is 50 to 70% (P0 is 30 to 50%): green
Group 4: P1 is 70 to 100% (P0 is 0 to 30%): yellow
Thus, e.g., when the determination probability of detecting chips in the first grid is 0.98, P1 is 98%, and the grid is displayed in yellow (group 4). When the determination probability of detecting chips in the second grid is 0.12, P1 is 12%, and the second grid is displayed in red. It should be noted that although the determination probability of detecting chips in the second grid is 0.12, it may be expressed as that P0 is 88%.
The user confirming the determination result from the image can touch and select any partial image that has been erroneously determined and input a correct determination result (another color). For example, the user can change a partial image that is colored as group 2 to group 1, and a partial image that is colored as group 3 to group 4. A teaching processing unit 316 corrects the determination result 324 of the storage 302 according to the change input of the determination result by the user.
The coolant control unit 318 controls the coolant pump 229 via the control unit 228. Specifically, the coolant control unit 318 tunes off the coolant pump 229 in an observation mode in which the state of the inside of the machine is imaged as it is and displayed on the touch panel 303, or in a chip detection mode in which the chip processing unit 211 tries to detect chips. Otherwise, the coolant control unit 318 controls the coolant discharging unit 224 via the control unit 228 in accordance with the determination result by the chip processing unit 314, and discharges the coolant to the target position. Whether each area should be cleaned is determined by whether the accumulation amount of each area has exceeded a threshold value (set value). The accumulation amount of each area is calculated from the determination result of each grid included in the area.
No chip: 0 points
Few chips: 3 points
Many chips: 8 points
Examples: Imaging of area A (consisting of 200 grids) and AI judgment result at a certain time point:
No chip×165, Few chips×30, Many chips×5->Accumulation score=30×3+8×5≈130>area A threshold value
Then, the entire area A is cleaned.
That is, the processing unit 301 calculates a score on the existence probability of chips for each of a plurality of grids obtained by dividing a part of the image data. Then, the processing unit 301 defines an area including a predetermined number of grids, and determines, in an area by area manner, whether cleaning should be performed according to the total score in the whole area. In other words, when the total score of the areas divided according to the structures in the machine tool 202 exceeds a threshold value, the processing unit 301 generates a signal for controlling the discharge of fluid from the fluid discharging unit so that fluid flows through the areas, and transmits the signal to the control unit 228 of the machine tool 202. The above-described total score (also referred to as the “accumulation amount of the area” or “accumulation score”) is an example of a “score relating to accumulation of the object in the area”.
For example, it is assumed that chips are detected from the first grid and the third grid corresponding to the inclined surface B2 of
When image data is acquired in step S501, the image reliability determination unit 212 performs the brightness/darkness indicator calculation processing (S503), the blur indicator calculation processing (S505), and the mist indicator calculation processing (S507) in parallel. Further, in steps S509 to S513, the image reliability determination unit 212 compares the indicators with each of the threshold values in parallel processing, and if any indicator is equal to or less than the threshold value, reads the corresponding notification message 329 from the storage 302 and displays the notification message on the touch panel 303 (S515 to S519).
The above-described embodiments are summarized as follows. The imaging control unit 312 controls the camera 227 at predetermined timing to acquire image data. The grid dividing unit 313 divides image data into partial images by grids. The area dividing unit 311 divides the image data into a plurality of areas each composed of a plurality of predetermined partial images according to the structures in the machine tool. As shown in
The chip processing unit 314 determines the existence probability of chips for each partial image divided into the grids. Further, the chip processing unit 314 calculates a total score related to the accumulation of the object in the entire area for each area corresponding to the structures in the machine tool on the basis of the existence probability of chips in a plurality of partial images included in the area. When the total score for accumulation in any area exceeds a threshold value, the coolant control unit 318 performs control for discharging coolant along the cleaning path of the area. When the total score of area A exceeds the threshold value, coolant is discharged along the zigzag cleaning path for area A. When the total score of the area B exceeds the threshold, coolant is discharged along the zigzag cleaning path of the horizontal surface B1, and then coolant is discharged along the linear cleaning path of the inclined surface B2.
First, as shown in
The display control unit 320 of the information processing device 201 may perform processing for displaying the score graph screen illustrated in
As shown in
In this example, as shown by the threshold line 402, the threshold value is set to “100”. For example, since the total score of the area from the first time to the fourth time does not exceed the threshold value, no cleaning is performed. Therefore, chips increase during that time, and the total score of the areas increases. When the total score of the areas exceeds the threshold at the fifth time, cleaning is performed. Cleaning reduces the chips and reduces the total score of the area at the sixth time. Thus, an increase in the total score of the area during the period in which no cleaning is performed and a decrease in the total score of the area due to cleaning are repeated. The user can easily grasp the state change of the chips in the areas corresponding to the structures in the machine tool by viewing the score graph screen.
According to the present embodiment, it is possible to know whether an image obtained by imaging the inside of the machine tool is suitable for detecting a removal object. The reliability of the detection result of the removal object can be determined, and if the reliability is low, means for solving the problem can be examined. Consequently, the removal object in the machine tool can be detected with higher accuracy.
The present invention may be applied to a system comprising a plurality of devices or to a single device. Further, the present invention is applicable to a case where an information processing program for implementing the function of the embodiment is supplied to a system or a device and executed by a built-in processor. To implement the functions of the present invention by a computer, a program installed in the computer, a medium storing the program, a server for downloading the program, and a processor for executing the program are also included in the technical scope of the present invention. In particular, at least a non-transitory computer readable medium containing a program that causes a computer to perform the processing steps included in the above-described embodiments is within the scope of the present invention.
This application is a continuation application of International Application No. PCT/JP2021/030012, filed on Aug. 17, 2021, which claims priority to and the benefit of Japanese Patent Application No. 2020-141250, filed on Aug. 24, 2020. The contents of these applications are incorporated herein by reference in their entirety.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2020-141250 | Aug 2020 | JP | national |
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2021/030012 | Aug 2021 | US |
| Child | 17974692 | US |
