Patent Application
20230106601
The present invention relates to a management apparatus, a lithography apparatus, a management method, and an article manufacturing method.
Japanese Patent Laid-Open No. 2009-205641 describes a position control apparatus including an iterative learning control circuit. The position control apparatus includes a detection device that detects the position of a controlled object, a subtraction device that generates an error obtained by subtracting the output of the detection device from the target value, an iterative learning control circuit that includes a filter to which the error is input, and a calculation means for calculating the parameter variation of the controlled object. The characteristic of the filter is changed in accordance with the parameter variation of the controlled object.
A control apparatus using a neural network can decide the parameter values of the neural network by performing reinforcement learning. However, since the state of a controlled object can change over time, even the neural network optimized at a given time is no longer optimum since the state of the controlled object has changed thereafter. Therefore, the control accuracy of the control apparatus may deteriorate due to the change in the state of the controlled object.
The present invention provides a technique advantageous in suppressing deterioration in control accuracy caused by a change in the state of a controlled object.
One aspect of the present invention is related to a management apparatus, and the management apparatus comprises a learning device configured to, in a case where a reward obtained from a control result of a controlled object by a controller configured to control the controlled object using a neural network, for which a parameter value is decided by reinforcement learning, does not satisfy a predetermined criterion, redecide the parameter value by reinforcement learning.
Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.
Hereinafter, embodiments will be described in detail with reference to the attached drawings. Note, the following embodiments are not intended to limit the scope of the claimed invention. Multiple features are described in the embodiments, but limitation is not made an invention that requires all such features, and multiple such features may be combined as appropriate. Furthermore, in the attached drawings, the same reference numerals are given to the same or similar configurations, and redundant description thereof is omitted.
The processing apparatus 1 can include a controlled object and control the controlled object using a neural network for which parameter values are decided by reinforcement learning. The control apparatus 2 can be configured to send a driving command to the processing apparatus 1 and receive a driving result or a control result from the processing apparatus 1. The management apparatus 3 can perform reinforcement learning of deciding a plurality of parameter values of the neural network of the processing apparatus 1. More specifically, the management apparatus 3 can decide the plurality of parameter values of the neural network by repeating an operation of sending a driving command to the processing apparatus 1 and receiving a driving result from the processing apparatus 1 while changing all or some of the plurality of parameter values. The management apparatus 3 may be understood as a learning apparatus.
All or some of the functions of the control apparatus 2 may be incorporated in the management apparatus 3. All or some of the functions of the control apparatus 2 may be incorporated in the processing apparatus 1. The processing apparatus 1, the control apparatus 2, and the management apparatus 3 may be formed physically integrally or separately. The processing apparatus 1 may be controlled by the control apparatus 2 as a whole, or may include components controlled by the control apparatus 2 and those not controlled by the control apparatus 2.
The first compensator 82 can be, for example, a PID compensator but may be another compensator. When, for example, L represents the number of inputs, M represents the number of intermediate layers, and N represents the number of outputs (L, M, and N are all positive integers), the second compensator 83 can be, for example, a neural network defined by the product of an L×matrix and an M×N matrix. The plurality of parameter values of the neural network can be decided or updated by reinforcement learning executed by the management apparatus 3. The first compensator 82 is not always necessary, and only the second compensator 82 may generate the command value to be given to the driver 7.
The management apparatus 3 can function as a learning device or a relearning device that executes a learning sequence when a reward obtained from the control result of the stage ST by the controller 8 of the processing apparatus 1 does not satisfy a predetermined criterion. In the learning sequence, a parameter value set constituted by the plurality of parameter values of the second compensator (neural network) 83 can be decided or redecided by reinforcement learning.
In step S103, the management apparatus 3 can acquire, from the controller 8 of the processing apparatus 1 via the control apparatus 2, driving data indicating the driving state of the stage ST as the controlled object in step S102. The driving data can include, for example, at least one of the output from the sensor 6 and the output from the subtracter 81. In step S104, the management apparatus 3 can calculate a reward based on the driving data acquired in step S103. The reward can be calculated based on a predefined formula. For example, in a case where the reward is calculated based on the control error, the reward can be calculated in accordance with a formula that gives the reciprocal of the control error, a formula that gives the reciprocal of the logarithm of the control error, a formula that gives the reciprocal of the quadratic function of the control error, or the like, but may be calculated in accordance with another formula. In one example, as the value of the reward is larger, the second compensator (neural network) 83 is more superior. Conversely, as the value of the reward is smaller, the second compensator (neural network) 83 may be more superior.
In step S105, the management apparatus 3 generates a new parameter value set by changing at least one of the plurality of parameter values of the second compensator (neural network) 83, and sets the new parameter values in the second compensator (neural network) 83. Steps S106, S107, and S108 can be the same as steps S102, S103, and S104, respectively. In step S106, the management apparatus 3 can send a command to the processing apparatus 1 to drive the stage ST. More specifically, in step S106, the management apparatus 3 can send a driving command to the controller 8 of the processing apparatus 1 via the control apparatus 2. In response to this, the controller 8 of the processing apparatus 1 can cause the driver 7 to drive the stage ST in accordance with the driving command, thereby controlling the position of the stage ST. In step S107, the management apparatus 3 can acquire, from the controller 8 of the processing apparatus 1 via the control apparatus 2, driving data indicating the driving state of the stage ST in step S106. In step S108, the management apparatus 3 can calculate a reward based on the driving data acquired in step S107.
In step S109, the management apparatus 3 determines whether the reward calculated in step S108 is improved, as compared with the reward calculated in step S104. Then, in a case where the reward calculated in step S108 is improved, as compared with the reward calculated in step S104, the management apparatus 3 adopts, in step S110, as the latest parameter values, the parameter value set obtained after the change operation is executed in step S105. On the other hand, in a case where the reward calculated in step S108 is not improved, as compared with the reward calculated in step S104, the management apparatus 3 does not adopt, in step S111, the parameter value set obtained after the change operation is executed in step S105, and returns to step S105. In this case, in step S105, a new parameter value set is set in the second compensator (neural network) 83.
If step S110 is executed, the management apparatus 3 determines in step S112 whether the reward calculated in step S108 immediately precedingly executed satisfies the predetermined criterion. In a case where the reward satisfies the predetermined criterion, the processing shown in
The processing apparatus 1 can operate, in a sequence (to be referred to as an actual sequence hereinafter) of executing processing for the processing target object, as an apparatus including the learned model (second compensator 83) obtained in the above-described learning sequence. In one example, the processing apparatus 1 can execute the actual sequence under management of the management apparatus 3. However, in another example, the processing apparatus 1 can execute the actual sequence independently of management of the management apparatus 3.
In step S204, the management apparatus 3 determines whether the reward calculated in step S203 satisfies a predetermined criterion. In a case where the reward satisfies the predetermined criterion, the management apparatus 3 returns to step S201. In a case where the reward does not satisfy the predetermined criterion, the management apparatus 3 advances to step S205, and executes the learning sequence (that is, relearning) shown in
The learning sequence in step S205 can be executed starting from the current learned model. Alternatively, the learning sequence in step S205 can be executed after the neural network is returned to the initial state or an arbitrary state in the learning process.
An example in which the above-described manufacturing system MS is applied to a scanning exposure apparatus 500 will be described below with reference to
The controller 25 controls the illumination optical system 23, the original stage mechanism 12, the projection optical system 13, the substrate stage mechanism 15, the first position measurement device 17, the second position measurement device 18, the substrate mark measurement device 21, and the substrate conveyer 22. The controller 25 controls processing of transferring a pattern of an original 11 to the substrate 14. The controller 25 can be formed by, for example, a PLD (the abbreviation of a Programmable Logic Device) such as an FPGA (the abbreviation of a Field Programmable Gate Array), an ASIC (the abbreviation of an Application Specific Integrated Circuit), a general-purpose computer installed with a program, or a combination of all or some of these components. The controller 25 can correspond to the controller 8 in the processing apparatus 1 shown in
The original stage mechanism 12 can include an original stage RST that holds the original 11, and a first actuator RAC that drives the original stage RST. The substrate stage mechanism 15 can include a substrate stage WST that holds the substrate 14, and a second actuator WAC that drives the substrate stage WST. The illumination optical system 23 illuminates the original 11. The illumination optical system 23 shapes, by a light shielding member such as a masking blade, light emitted from a light source (not shown) into, for example, band-like or arcuate slit light long in the X direction, and illuminates a portion of the original 11 with this slit light. The original 11 and the substrate 14 are held by the original stage RST and the substrate stage WST, respectively, and arranged at almost optically conjugate positions (on the object plane and image plane of the projection optical system 13) via the projection optical system 13.
The projection optical system 13 has a predetermined projection magnification (for example, 1, ½, or ¼), and projects the pattern of the original 11 on the substrate 14 by the slit light. A region (a region irradiated with the slit light) on the substrate 14 where the pattern of the original 11 is projected can be called an irradiation region. The original stage RST and the substrate stage WST are configured to be movable in a direction (Y direction) orthogonal to the optical axis direction (Z direction) of the projection optical system 13. The original stage RST and the substrate stage WST are relatively scanned at a velocity ratio corresponding to the projection magnification of the projection optical system 13 in synchronism with each other. This scans the substrate 14 in the Y direction with respect to the irradiation region, thereby transferring the pattern formed on the original 11 to a shot region of the substrate 14. Then, by sequentially performing such scanning exposure for the plurality of shot regions of the substrate 14 while moving the substrate stage WST, the exposure processing for the one substrate 14 is completed.
The first position measurement device 17 includes, for example, a laser interferometer, and measures the position of the original stage RST. For example, the laser interferometer irradiates, with a laser beam, a reflecting plate (not shown) provided in the original stage RST, and detects a displacement (a displacement from a reference position) of the original stage RST by interference between the laser beam reflected by the reflecting plate and the laser beam reflected by a reference surface. The first position measurement device 17 can acquire the current position of the original stage RST based on the displacement. In this example, the first position measurement device 17 may measure the position of the original stage RST by a position measurement device, for example, an encoder instead of the laser interferometer. The substrate mark measurement device 21 includes, for example, an optical system and an image sensor, and can detect the position of a mark provided on the substrate 14.
The second position measurement device 18 includes, for example, a laser interferometer, and measures the position of the substrate stage WST. For example, the laser interferometer irradiates, with a laser beam, a reflecting plate (not shown) provided in the substrate stage WST, and detects a displacement (a displacement from a reference position) of the substrate stage WST by interference between the laser beam reflected by the reflecting plate and the laser beam reflected by a reference surface. The second position measurement device 18 can acquire the current position of the substrate stage WST based on the displacement. In this example, the second position measurement device 18 may measure the position of the substrate stage WST by a position measurement device, for example, an encoder instead of the laser interferometer.
The scanning exposure apparatus 500 is required to accurately transfer the pattern of the original 11 to the target position of the substrate 14. To achieve this, it is important to accurately control the relative position of the original 11 on the original stage RST with respect to the substrate 14 on the substrate stage WST during scanning exposure. Therefore, as a reward, a value for evaluating the relative position error (synchronous error) between the original stage RST and the substrate stage WST can be adopted. To improve the detection accuracy of the mark of the substrate 14, it is important to accurately position the substrate stage WST under the substrate mark measurement device 21. Therefore, as a reward, a value for evaluating the control error of the substrate stage WST while the mark is imaged can be adopted. To improve the throughput, it is important to increase the conveyance speed of the substrate. At the time of loading and unloading the substrate, it is important that the control errors of the substrate conveyer 22 and the substrate stage WST converge to a predetermined value or less in a short time after the completion of driving. Therefore, as a reward, a value for evaluating the convergence times of the substrate conveyer 22 and the substrate stage WST can be adopted. Each of the substrate stage mechanism 15, the original stage mechanism 12, and the substrate conveyer 22 is an example of an operation unit that performs an operation for the processing of transferring the pattern of the original 11 to the substrate 14.
In step S302, the controller 25 controls the substrate conveyer 22 to load (convey) the substrate 14 to the substrate stage WST. More specifically, in step S302, the controller 25 can control the substrate stage mechanism 15 so that the mark of the substrate 14 falls within the field of view of the substrate mark measurement device 21, and control the substrate mark measurement device 21 to detect the position of the mark of the substrate 14. This operation can be executed for each of the plurality of marks of the substrate 14. In step S304, the controller 25 controls the substrate stage mechanism 15, the original stage mechanism 12, the illumination optical system 23, and the like so that the pattern of the original 11 is transferred to each of the plurality of shot regions of the substrate 14. In step S305, the controller 25 controls the substrate conveyer 22 to unload (convey) the substrate 14 on the substrate stage WST. In steps S302, S303, S304, and S305, the driving data required to calculate the reward for the control in steps S302, S303, S304, and S305 can be provided from the controller 25 (controller 8) to the management apparatus 3 via the control apparatus 2, respectively. These driving data may be collectively provided to the management apparatus 3 from the controller 25 (controller 8) via the control apparatus 2 after step S305 is complete.
In step S306, the management apparatus 3 calculates, based on the driving data, the reward for the control in each of the plurality of sub-sequences, that is, steps S302, S303, S304, and S305. For example, for the control in each of steps S302 and S305, the value for evaluating the time required for the control error of the substrate stage or holder holding the substrate to converge to a predetermined value or less can be calculated as the reward. For the control in step S303, the value for evaluating the control error of the substrate stage (holder) during measurement of the alignment error between the substrate and the original can be calculated as the reward. For the control in step S304, the value for evaluating the synchronous error between the substrate and the original during exposure of the substrate can be calculated as the reward.
In step S307, the management apparatus 3 determines whether the reward calculated in step S306 satisfies a predetermined criterion. In a case where the reward satisfies the predetermined criterion, the management apparatus 3 terminates the actual sequence shown in
In a case where the reward to be calculated is the value for evaluating the time required for the control error of the substrate stage or holder holding the substrate to converge to the predetermined value or less, the corresponding criterion is also given as the time required for the control error to converge to the predetermined value or less. In a case where the reward to be calculated is the value for evaluating the control error of the substrate stage during measurement of the alignment error between the substrate and the original, the corresponding criterion can also be given as the control error of the substrate stage during measurement of the alignment error. In a case where the reward to be calculated is the value for evaluating the synchronous error between the substrate and the original during exposure of the substrate, the corresponding criterion can also be given as the synchronous error between the substrate and the original during exposure of the substrate.
Examples of the controlled object for which a neural network is formed are the substrate stage mechanism 15, the original stage mechanism 12, and the substrate conveyer 22 but a neural network may be incorporated in another component. For example, a plurality of components such as the substrate stage mechanism 15, the original stage mechanism 12, and the substrate conveyer 22 may be controlled by one neural network or the plurality of components may be controlled by different neural networks, respectively. Furthermore, as a learned model, the same learned model or different learned models may be used for the conveyance sequence, the measurement sequence, and the exposure sequence. In calculation of a reward, the same formula or different formulas may be used for the conveyance sequence, the measurement sequence, and the exposure sequence.
With reference to
The timing of executing learning in step S308 can be, for example, immediately after the execution of the sequence ends, between the processing for a given substrate and processing for the next substrate, or after the processing operations for substrates using the same original end. Alternatively, learning in step S308 may be executed, for example, in parallel with maintenance of components of the light source.
The example in which the manufacturing system MS is applied to the scanning exposure apparatus 500 has been explained above. However, the manufacturing system MS may be applied to an exposure apparatus (for example, a stepper) of another type or a lithography apparatus of another type such as an imprint apparatus. In this case, the lithography apparatus is an apparatus for forming a pattern on a substrate, and the concept includes an exposure apparatus, an imprint apparatus, and an electron beam drawing apparatus.
An article manufacturing method of manufacturing an article (for example, a semiconductor IC element, a liquid crystal display element, or a MEMS) using the above-described lithography apparatus will be described below. The article manufacturing method can be a method that includes a transfer step of transferring a pattern of an original to a substrate using the lithography apparatus, and a processing step of processing the substrate having undergone the transfer step, thereby obtaining an article from the substrate having undergone the processing step.
When the lithography apparatus is an exposure apparatus, the article manufacturing method can include a step of exposing a substrate (a substrate, a glass substrate, or the like) coated with a photosensitive agent, a step of developing the substrate (photosensitive agent), and a step of processing the developed substrate in other known steps. The other known steps include etching, resist removal, dicing, bonding, and packaging. According to this article manufacturing method, a higher-quality article than a conventional one can be manufactured. When the lithography apparatus is an imprint apparatus, the article manufacturing method can include a step of forming a pattern made of a cured product of an imprint material by molding the imprint material on a substrate using a mold, and a step of processing the substrate using the pattern.
While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2020-111910 | Jun 2020 | JP | national |
This application is a Continuation of International Patent Application No. PCT/JP2021/023323, filed Jun. 21, 2021, which claims the benefit of Japanese Patent Application No. 2020-111910, filed Jun. 29, 2020, both of which are hereby incorporated by reference herein in their entirety.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2021/023323 | Jun 2021 | US |
| Child | 18060667 | US |
