Patent Application
20240192663
The present disclosure relates to a control device, and more particularly relates to a control device for automating tasks in a computing device included in a manufacturing device, tasks in paperwork, and the like.
With the development of information processing technology in recent years, digitization, such as automation of various tasks and automation of manufacturing processes in factories have been in progress. For such automation, software that automatically performs a task in a computer, such as Robotic Process Automation (RPA), is used. Currently, a plurality of RPA software is commercially available, and most of the RPA software is installed in a computer to be controlled and operated in the computer to be controlled, so that an operation performed by a person is automated on the basis of a specified procedure. However, many of these pieces of software have been developed in recent years, and need to be executed by a relatively new computer.
On the other hand, in the manufacturing industry, an expensive manufacturing device cannot be frequently replaced, and manufacturing conditions and manufacturing recipes need to be rederived along with the replacement of the manufacturing device. Therefore, a large amount of operation is required. That is, in manufacturing a certain product, if the product can be smoothly manufactured without problems, motivation for renewal of the manufacturing device is not high.
In addition, such a manufacturing device is often controlled by an old computer. As described above, since software such as RPA needs to be executed by a relatively new computer, it is difficult to apply the latest RPA software to an old computer. Furthermore, since these old computers are not assumed to be connected to a network, it is often difficult to be controlled from an external computer.
These signals may be digital signals or may be analog signals via a digital-to-analog converter. Further, in the conventional automation, the control signal line 13 and the measurement signal line 14 are drawn out and connected to an external computer 15 separately installed. The external computer 15 controls the manufacturing device main body 11 in place of the existing computer 12. In addition, the signal line may be connected to a programmable logic controller or the like incorporated in the existing computer 12.
However, in such a conventional method of processing a signal flowing between the computer 12 and the manufacturing device main body 11, it is necessary to analyze a communication protocol between the control unit 121 and the manufacturing device main body 11 and processing in the control unit 121 to newly develop a control circuit that generates a control signal. In addition, since the manufacturing device 10 itself is old, there are many cases where the automation development of the task is hindered, such as no technical material at the time of device development remains, the person in charge of development is retired, and the device control technical information is proprietary.
The present disclosure has been made to solve the above problems. That is, if an operation method is known for a manufacturing device or office device, there is provided a means for automating at least a part of tasks for control which has been conventionally performed manually, without analyzing a communication protocol between a control unit and a device main body or a control method in the device. Furthermore, the present disclosure applies a signal processing technology to a human interface (for example, the GUI) displayed on a display, which can be operated using a keyboard, a mouse, or the like, and controls the human interface from an external computer, thereby automating tasks for control which has been conventionally performed manually.
In order to resolve the above-described problems, according to the present disclosure, there is provided a control device connected to a control unit of a computer as an external device in order to automate at least a part of a task in the computer mounted in a device, in which the device includes a main body and the computer, and the computer includes key input and a GUI, the control device including an image acquisition unit that acquires an image signal of the GUI transmitted from the computer, an image processing unit that processes the image signal acquired by the image acquisition unit, a storage unit that stores a control sequence defining a sequence for controlling the main body of the device, a sequence processing unit that generates a control signal from the image signal processed by the image processing unit and the control sequence stored in the storage unit, and a key signal generation unit that transmits the control signal generated by the sequence processing unit as a key signal, in which processing the image signal includes identifying a keyword in the GUI, and generation of the control signal includes generation of the key signal for controlling the main body of the device based on the keyword identified as the control sequence.
The following is a description of an embodiment of the present invention, with reference to the drawings. A control device for a computer according to an embodiment of the present invention is an external computer installed separately from a computer installed in an existing manufacturing device or office device, and is a device that automates an operation for control in an existing computer that is conventionally performed manually. Then, this external computer is different from the conventional technique on the point that the external computer automatically outputs a control signal for controlling the device main body of the manufacturing device or the office device on the basis of the image signal acquired from the existing computer.
Hereinafter, a first embodiment of the present invention will be described. In the present embodiment, for example, task for controlling condition input and the like in an existing computer is automated on a manufacturing device such as a film forming device based on image processing by an external computer.
In addition, the manufacturing device 10 may include an external sensor 24 that monitors external factors such as temperature and humidity, for example, in the vicinity of the manufacturing device main body 11. The output of the external sensor 24 is connected to the measurement signal line 14, and the external sensor 24 transmits the monitored data of the external factor as the external sensor signal 25. The external sensor signal 25 is transmitted to the automatic control unit 201 via the control unit 121 of the 12. Then, the external sensor signal 25 existing computer transmitted to the automatic control unit 201 is reflected in generation of a control sequence to be described later. Alternatively, the external sensor 24 and the automatic control unit 201 of the external computer 20 may be connected by an external sensor signal line 24a, and the external sensor signal 25 may be directly fetched via the external sensor signal line 24a.
In the control device according to the present embodiment configured as described above, the image signal 21 of the GUI output from the existing computer 12 is acquired, and the acquired image signal 21 is processed in the automatic control unit 201 included in the external computer 20. Next, in the automatic control unit 201, a control signal is generated based on the processed image signal 21 and the control sequence of the manufacturing device main body 11. The generated control signal is transmitted to the control unit 121 in the existing computer 12 as a key signal 22 or a pointer signal 23. By automatically performing this series of operations, it is possible to automate the task for controlling the manufacturing device main body 11 which has been conventionally performed manually. In the present embodiment, the UI displayed by the display 122 is a GUI, however, it may be a CUI.
The control signal transmitted from the existing computer 12 to the manufacturing device main body 11 is, for example, a pulse signal or the like to the stepping motor of the moving stage built in the manufacturing device 10. Furthermore, in a case where the manufacturing device 10 is an etching apparatus or the like used for semiconductor manufacturing, the control signal is an input signal to the RF signal generator or the like. On the other hand, the measurement signal transmitted from the manufacturing device main body 11 to the computer 12 is read from an encoder of the stage in the former case, and is read by a mass flow controller in the latter case.
In the present embodiment, the computer 12 is a workstation based on a general personal computer Unix (registered trademark), but may be an embedded workstation using a PC98 (registered trademark) series or a compatible machine thereof, a proprietary HP-UX (registered trademark), Solaris (registered trademark), or the like as an operating system. A manufacturer of the manufacturing device 10 may control the manufacturing device 10 by incorporating the same proprietary control software into the computer 12.
On the other hand, the external computer 20 in the present embodiment is a general personal computer or microcomputer, but the latest personal computer or microcomputer in that period may be used.
The image signal 21 acquired by the image acquisition unit 31 is processed by the image processing unit 32, and recognizes the input keyword, the position of the mouse pointer, and the like in the GUI or CUI on the display 122 in the existing computer 12. In addition, the sequence processing unit 34 processes a task to be performed next acquired from the control sequence. For example, the task is like the one in which the mouse pointer is moved onto a button at a specific coordinate on the screen and clicked. Here, the sequence processing unit 34 determines whether the position of the keyword or the mouse pointer processed by the image processing unit 32 is equal to the position of the keyword or the mouse pointer in the image output of the computer 12. If it is determined that they are not equal, the previous sequence is corrected and the determination is repeated again, and if it is determined that they are equal, the process proceeds to the next sequence.
The control sequence stored in the storage unit 33 is a file describing a procedure of tasks in the computer 12 which has been conventionally performed manually, and may be generated by a person or may be generated mechanically. However, since the control sequence is a complicated task that requires setting of a plurality of parameters, manually inputting the control sequence and the parameters to the computer 12 leads to a decrease in throughput in manufacturing. For example, in a glass film forming device in optical device manufacturing, a refractive index, an internal stress, and the like of a glass film fluctuate due to external factors such as environmental temperature and humidity. In order to correct these fluctuations, it is desired to change the manufacturing recipe and parameters of the film forming device each time film formation is performed, but it is not preferable to manually perform the change from the viewpoint of operation and errors at work.
From such a viewpoint, the automatic control unit 201 may include a control sequence update unit 37 that updates the control sequence in order to automate input task of a parameter that is changed by an external factor such as a manufacturing condition and needs to be appropriately updated. The control sequence update unit 37 is a processor connected to an output of the storage unit 33, and has a function of reflecting the above-described external sensor signal 25 on a control sequence stored in advance in the storage unit 33 to automatically generate a new control sequence. The new control sequence generated by the control sequence update unit 37 is transmitted to the sequence processing unit 34 to generate a control signal. Although the control sequence can be automatically generated by the existing computer 12, in this case, it is often difficult to automatically generate the control sequence due to limitations of the performance and function of the computer. Therefore, as illustrated in
Hereinafter, as an example of creation of a control sequence, an outline will be described by taking formation of a glass film in manufacturing of an optical device as an example. In manufacturing an optical device called a quartz-based planar lightwave circuit, a flame hydrolysis deposition (FHD) method suitable for forming a thick film glass is used, but film formation on a silicon wafer is performed in a normal pressure gas phase exposed to outside air (e.g., refer to Non Patent Literature 2). Therefore, due to the influence of environmental humidity, water molecules in the air adhere to the soot of the formed glass, and it becomes difficult to obtain a desired glass refractive index. Therefore, it is preferable to monitor the environmental humidity during the deposition of the glass to automatically correct the supply amount of the raw material. More specifically, the refractive index of glass nglass can be assumed to have a linear change in a region where the amount of absorbed moisture is small, and
d
OH
=β×d
OH,
In order to control the manufacturing device main body 11 by the external computer 20, it is necessary to identify the position of the keyword or the mouse pointer on the image displayed on the display 122 of the computer 12. The keyword identification is realized by using a general OCR function. On the other hand, the identification of the position of the mouse pointer is realized by moving the mouse pointer by a minute amount and analyzing a difference image between two images before and after the movement, as described later.
u=v×255/(M−m) (Formula 1)
Here, u is a value of each pixel after normalization, M is a luminance value of a pixel having the maximum luminance, m is a luminance value of a pixel having the minimum luminance value, and v is a luminance value of each pixel. Finally, in step S406, the image processing unit 32 identifies coordinates at which the luminance value obtained by (Formula 1) is maximized as the position of the mouse pointer 51, and outputs the identified position of the mouse pointer 51.
In the above-described minute amount movement of the mouse pointer 51, it is preferable to move in both the positive direction and the negative direction of the X component of the screen or in both the positive direction and the negative direction of the Y component. This is because, when the mouse pointer 51 is at the screen end, the mouse pointer does not move in the direction of moving to the outside of the screen.
In this way, the loop of moving and determining the pointer position is repeated for the following reasons. In general, a mouse pointer as a human interface moves with an acceleration with respect to a movement amount of a mouse. That is, when a person moves a mouse, a mouse pointer also moves by a minute amount in the case of a minute movement, but in the case of a large movement, the pointer moves with acceleration more than the movement pulse amount the mouse actually counts. This acceleration is individually set for each conventional control unit that is a controlled object, and thus cannot be commonly applied to all controlled objects. Therefore, it is preferable to repeat the loop as described above.
In addition, the above-described allowable positional deviation range is preferably a range that falls within the size of the button 52 of the control software provided by the manufacturer of the device. For example, the allowable positional deviation is desirably within the size range of the button 52 described as “LOGIN” in
By setting such a control sequence in advance, it is possible to automate tasks for controlling the manufacturing device 10 having a legacy computer that cannot directly use the RPA. In addition, it is also possible to automatically generate an appropriate recipe according to a situation such as an environmental change by utilizing input from an external sensor. According to the example of the glass film formation described above, the sensor device #4 is a humidity sensor, and the character string on the screen (x, y) is a reading of the mass flow controller and represents the current supply amount of GeO2. Furthermore, the variable r is a setting value of the new mass flow controller, and is calculated using f (v) expressed by the above-described three formulas. The movement and clicking of the mouse cursor corresponding to the procedures 9 and 10 described in
As described above, in the control device according to the present embodiment, the automatic control unit 201 of the external computer 20 processes the image signal from the computer 12 and transmits the control signal based on the image signal. Therefore, it is possible to automate the control of the manufacturing device main body 11 without analyzing the communication protocol and the control method between the computer 12 and the manufacturing device main body 11 as in the conventional technique.
Furthermore, in the present embodiment, since the control sequence is automatically generated in the automatic control unit 201, it is possible to automatically set the control recipe and parameters of the manufacturing device main body 11, which have been conventionally performed manually. Therefore, it is possible to improve the manufacturing throughput and the product yield as compared with the conventional technique.
Hereinafter, a second embodiment of the present invention will be described. In the present embodiment, unlike the first embodiment, an external computer 20 is configured to perform remote control from an external terminal via a network.
In addition to the above configuration, the present embodiment has a configuration in which the external computer 20 is connected to an external terminal 82 via the network 81. As the network 81, for example, a communication technology such as Ethernet (registered trademark), Worldwide Interoperability For Microwave Access (WiMAX (trademark)), 3G, 4G, or Digital Subscriber Line (DSL) can be used. Furthermore, the external terminal 82 may be a thin client or a fat client. In addition, in a case where the RPA software is installed in the fat client itself, the external computer 20 may be controlled therefrom.
Furthermore, a network camera 83 may be installed in the vicinity of the manufacturing device main body 11, and the network camera 83 may also be connected to the external terminal 82 via the network 81. As a result, the user who performs remote control can monitor the state of the manufacturing device main body 11 controlled.
The control device according to the present embodiment configured as described above can automate the task for controlling the manufacturing device main body 11 similarly to the first embodiment by the user remotely operating the external computer 20 via the external terminal 82. Therefore, it is possible to achieve the same effects as those of the first embodiment even remotely. The place where the external terminal 82 is located may be a room other than the room where the manufacturing device 10 is located, or may be a house other than the factory where the manufacturing device 10 is installed. In a case where the manufacturing device 10 is installed in an isolated place such as a clean room, for example, it takes time and effort to go in and out. However, according to the present embodiment, for example, remote control such as control monitoring in an office or control monitoring at the time of working at home can be performed.
As described above, the present disclosure can be applied to a computer having at least one of key input and pointer input and image output in order to automate the task. Furthermore, the present invention can also be applied to control of an old computer in which RPA software does not operate. That is, the present invention can be applied not only to automate manufacturing devices but also to automate normal paperwork and general operations.
Application as a control device for automating operations in a manufacturing device or an office device is expected.
| Filing Document | Filing Date | Country | Kind |
|---|---|---|---|
| PCT/JP2021/019453 | 5/21/2021 | WO |
