Patent Application
20230274539
The present invention relates to an inference processing system that is capable of reducing load when executing inference processing, an edge device, a method of controlling the inference processing system, a method of controlling the edge device, and a storage medium.
There is known an inference processing apparatus that performs inference processing using a neural network. Particularly, in an inference processing apparatus configured to perform image recognition, a neural network, such as the Convolutional Neural Network (CNN), is used.
In the neural network, processing in intermediate layers and processing in an output layer are sequentially performed on an image input to an input layer, whereby a final inference result in which an object included in the image is recognized can be obtained. In the intermediate layers, a plurality of feature amount extraction-processing layers are hierarchically connected, and in each layer, convolution arithmetic operation processing, activation processing, and pooling processing are executed on input data input from the preceding layer. The intermediate layers perform high-dimensional extraction of a feature amount included in an input image by thus repeating the processing operations in the respective processing layers. In the neural network, if the number of intermediate layers is increased, it is possible to perform higher dimensional extraction of the feature amount, but on the other hand, for example, in an apparatus that has a relatively low computational power, such as an image capturing apparatus, larger computational load is applied to inference processing performed by the neural network, which increases the processing time. As a solution to this problem, it is envisaged, for example, that processing operations in processing layers up to a predetermined intermediate layer are executed in an inference apparatus, and intermediate data obtained by the processing operations is transmitted to a server, where processing operations in processing layers after the predetermined intermediate layer are executed using the received intermediate data as an input to the processing layers after the predetermined intermediate layer (see e.g. PCT International Patent Publication No. WO 2018/011842). With this, it is possible to reduce the computational load on the inference apparatus by distributing the load required for the inference processing performed by the neural network, and further, since the intermediate data which is not the original data itself is transmitted to the server, it is possible to ensure the secrecy of information concerning privacy.
However, in the technique disclosed in PCT International Patent Publication No. WO 2018/011842, in a case where inference processing is performed using a plurality of neural networks that output respective different inference results based on the same data input thereto, the inference apparatus is required to execute processing operations up to a predetermined intermediate layer in each neural network, which increases the computational load.
The present invention provides an inference processing system that is capable of reducing load on a device that executes inference processing while keeping the secrecy of information concerning privacy when inference processing is performed using a plurality of neural networks that output respective different inference results based on the same data input thereto, an edge device, a method of controlling the inference processing system, a method of controlling the edge device, and a storage medium.
In a first aspect of the present invention, there is provided an inference processing system that includes a first terminal and a second terminal and performs inference processing using a plurality of neural networks, wherein the first terminal executes inference processing by a first neural network using acquired data as an input thereto, and outputs intermediate data to the second terminal, the intermediate data being obtained by executing processing operations in intermediate layers, up to a predetermined intermediate layer, of the first neural network, which are commonized with a second neural network, and wherein the second terminal executes processing operations in intermediate layers, after the predetermined intermediate layer, of the second neural network using the intermediate data as an input thereto.
In a second aspect of the present invention, there is provided an edge device that communicates with a server, including at least one processor, and a memory coupled to the at least one processor, the memory having instructions that, when executed by the processor, perform the operations as: an execution unit configured to execute inference processing by a first neural network using acquired data as an input thereto, an output unit configured to output intermediate data to the server, the intermediate data being obtained by executing processing operations in intermediate layers, up to a predetermined intermediate layer, of the first neural network, which are commonized with a second neural network, and an acquisition unit configured to acquire an inference result obtained by the server that executes processing operations in intermediate layers, after the predetermined intermediate layer, of the second neural network using the intermediate data as an input thereto.
According to the present invention, it is possible to reduce load on the device that executes inference processing while keeping the secrecy of information concerning privacy when inference processing is performed using the plurality of neural networks that output respective different inference results based on the same data input thereto.
Further features of the present invention will become apparent from the following description of exemplary embodiments (with reference to the attached drawings).
The present invention will now be described in detail below with reference to the accompanying drawings showing embodiments thereof.
Referring to
The CPU 201 performs a variety of controls by executing programs stored in the ROM 202. The ROM 202 stores the programs that are executed by the CPU 201, and the like. Note that the storage device for storing the programs executed by the CPU 201 is not limited to the ROM but may be a hard disk or the like. The memory 203 is e.g. a RAM and is used as a work memory for the CPU 201.
The input section 204 receives a user operation and sends a control signal corresponding to the received user operation to the CPU 201. For example, the input section 204 includes physical operation buttons, a touch panel, and so forth, as input devices each for receiving a user operation. The touch panel outputs coordinate information indicating a position where a user touches the touch panel to the CPU 201. The CPU 201 controls the display section 205, the image capturing section 206, and the communication section 207, based on control signals and coordinate information received from the input section 204. Thus, each of the display section 205, the image capturing section 206, and the communication section 207, performs an operation responsive to a user operation.
The display section 205 is e.g. a display and displays a variety of images. Note that in the present embodiment, the touch panel of the input section 204 and the display section 205 are integrally formed. For example, the touch panel is formed such that its transmittance does not prevent the display section 205 from displaying an image or information, and is affixed to an upper layer of the display surface of the display section 205. Further, input coordinates on the touch panel and display coordinates on the display section 205 are associated with each other.
The image capturing section 206 includes lenses, a shutter having a diaphragm function, an image sensor, such as a CCD or CMOS device, that converts an optical image to electrical signals, and an image processor that performs a variety of image processing, such as exposure control and ranging control, based on the electrical signals output from the image sensor. The image capturing section 206 is controlled by the CPU 201 to perform image capturing according to a user operation received by the input section 204. The communication section 207 is controlled by the CPU 201 to communicate with the server 103 via the communication network 102.
The server 103 includes a CPU 209, a memory 210, a communication section 211, and a GPU 212. These components are interconnected via a system bus 213. Note that GPU is an abbreviation of Graphics Processing Unit.
The CPU 209 performs a variety of controls by executing programs stored e.g. in a hard disk or a ROM, not shown. The memory 210 is e.g. a RAM and is used as a work memory for the CPU 209 and the GPU 212. The communication section 211 is controlled by the CPU 209 to communicate with the image capturing apparatus 101 via the communication network 102. In the present embodiment, when a communication request is received from the image capturing apparatus 101, the CPU 209 generates a control signal responsive to this communication request and causes the GPU 212 to operate based on the control signal. Note that details of the communication between the image capturing apparatus 101 and the server 103 will be described hereinafter with reference to
The GPU 212 is an arithmetic unit that performs processing specialized for computation of computer graphics. In computation generally required for a neural network, such as a matrix computation, the GPU 212 is capable of processing the computation within a shorter time than a time required by the CPU 209. Note that although in the present embodiment, the configuration in which the server 103 includes the CPU 209 and the GPU 212 will be described, this is not limitative. For example, the server 103 may be configured to have a processor, such as a TRU, specialized for matrix calculation. Note that TPU is an abbreviation of Tensor Processing Unit. Further, the server 103 may be configured to have a plurality of GPUs 212.
Next, inference processing performed by the inference processing system 100 in the present embodiment will be described.
In the inference processing system 100, a neural network A and a neural network B that output respective different inference results based on the same data input thereto are used. The neural network A performs simple cluster classification for classifying objects into approximately 10 clusters. As shown in
In the inference processing system 100, the image capturing apparatus 101 which is relatively low in computational power uses the neural network A that performs simple cluster classification. With this, the image capturing apparatus 101 can perform inference processing using the neural network A without depending on a communication state of the communication network 102, and therefore, the image capturing apparatus 101 is capable of always performing inference necessary for image capturing. The inference processing using the neural network A is executed, for example, when auto focus control and control for changing a shutter speed are performed.
Further, in the inference processing system 100, the server 103 that is higher in computational power than the image capturing apparatus 101 uses the neural network B that performs detailed cluster classification. This makes it possible to obtain classification result data of detailed cluster classification while reducing the computational load on the image capturing apparatus 101 which is relatively low in computational power. The inference processing using the neural network B is executed, for example, when an image obtained through photographing is tagged.
In the present embodiment, in the neural network A and the neural network B, the processing layers from the input layer to a predetermined intermediate layer are commonized. That is, parameters used in the processing layers from the input layer to the predetermined intermediate layer of the neural network A are the same as those used in the corresponding layers of the neural network B. With this configuration, in the inference processing system 100, the image capturing apparatus 101 executes inference processing by the neural network A, using acquired data as an input, and transmits intermediate data to the server 103, which is obtained by executing processing operations in the processing layers up to the predetermined intermediate layer, which are commonized with the neural network B. The server 103 executes processing in processing layers after the above-mentioned predetermined intermediate layer of the neural network B, using the received intermediate data as an input thereto. This makes it possible to reduce the computational load on the image capturing apparatus 101, which is related to the neural network B, and further, since the intermediate data which is not the original data itself is transmitted to the server 103, it is possible to ensure the secrecy of information concerning privacy. As a result, when performing inference processing using a plurality of neural networks that output respective different inference results based on the same data input thereto, it is possible to reduce load on a device that executes the inference processing while keeping the secrecy of information concerning privacy.
Then, the image capturing apparatus 101 executes processing operations in predetermined intermediate layers commonized with the neural network B, more specifically, a first intermediate layer 402 and a second intermediate layer 403. Note that the processing operations in the input layer 401, the first intermediate layer 402, and the second intermediate layer 403 are realized by the CPU 201 of the image capturing apparatus 101, which executes programs stored e.g. in the ROM 202. Then, the image capturing apparatus 101 transmits second intermediate layer output data 413 (intermediate data) obtained by executing the processing in the second intermediate layer 403 to the server 103 via the communication network 102.
The server 103 inputs the received second intermediate layer output data 413 to an input layer 408, executes processing operations in a third intermediate layer 409, a fourth intermediate layer 410, . . . , an N-th intermediate layer 411 of the neural network B, and further, executes processing in an output layer B 412 of the same. These processing operations are realized by the CPU 209 and the GPU 212 of the server 103, which execute programs stored e.g. in the ROM of the server 103. Execution of the processing in the output layer B 412 causes classification result data 414 obtained by the neural network B to be output. The server 103 transmits the classification result data 414 to the image capturing apparatus 101 via the communication network 102.
On the other hand, the image capturing apparatus 101 inputs the second intermediate layer output data 413 obtained by executing the processing in the second intermediate layer 403 to a third intermediate layer 404 of the neural network A, executes processing operations in the third intermediate layer 404, a fourth intermediate layer 405, . . . , and an M-th intermediate layer 406, and further executes processing in an output layer A 407. As a result, classification result data 415 obtained by the neural network A is output. Note that the number of intermediate layers of the neural network A need not be the same as the number of intermediate layers of the neural network B, and the respective numbers of intermediate layers of the neural network A and the neural network B can be set as desired. Further, in the intermediate layers other than the layers (the input layer 401, the first intermediate layer 402, and the second intermediate layer 403) commonized between the neural network A and the neural network B, the number of nodes of each intermediate layer may be set as desired.
In the present embodiment, as shown in
On the other hand, for a third intermediate layer 504, a fourth intermediate layer 505, . . . , an M-th intermediate layer 506, and an output layer A 507 in
First, the inference control process performed by the image capturing apparatus 101 will be described.
Referring to
In the step S603, the CPU 201 inputs an image captured by the image capturing section 206 to the input layer 401 of the neural network A and executes processing operations in the first intermediate layer 402 and the second intermediate layer 403, which are the predetermined intermediate layers commonized with the neural network B. Then, in a step S604, the CPU 201 transmits the second intermediate layer output data 413 obtained by executing the processing in the second intermediate layer 403 to the server 103 via the communication section 207. Then, in a step S605, the CPU 201 inputs the second intermediate layer output data 413 to the third intermediate layer 404 and sequentially executes processing operations in the third intermediate layer 404, the fourth intermediate layer 405, . . . , and the M-th intermediate layer 406. Then, in a step S606, the CPU 201 executes processing in the output layer A 407 of the neural network A. As a result, the classification result data 415 obtained by the neural network A is output.
Then, in a step S607, the CPU 201 executes a variety of processing operations based on the classification result data 415. These variety of processing operations include e.g. auto focus control processing and control processing for changing the shutter speed. With this, it is possible to change the photographing settings to the optimum settings. Then, in a step S608, the CPU 201 waits until the classification result data 414 output by the neural network B based on the second intermediate layer output data 413 is received from the server 103. When it is determined by the CPU 201 that the classification result data 414 has been received from the server 103, the process proceeds to a step S609.
In the step S609, the CPU 201 executes a variety of processing operations based on the received classification result data 414. These variety of processing operations include e.g. processing for adding information indicating a success or failure of photographing to an image as a tag and processing for sorting images into folders based on a success or failure of photographing. After that, the present process is terminated.
Next, the inference control process performed by the server 103 will be described.
Referring to
In the step S612, the CPU 209 transmits a communication availability notification to the image capturing apparatus 101 as a response to the received communication request. Then, in a step S613, the CPU 209 waits until second intermediate layer output data is received from the image capturing apparatus 101. Note that this second intermediate layer output data is the second intermediate layer output data 413 transmitted from the image capturing apparatus 101 to the server 103 in the above-described step S604. If it is determined by the CPU 209 that second intermediate layer output data 413 has been received from the image capturing apparatus 101, the process proceeds to a step S614.
In the step S614, the GPU 212 inputs the received second intermediate layer output data 413 to the input layer 408 according to a command from the CPU 209 and sequentially executes the processing operations in the third intermediate layer 409, the fourth intermediate layer 410, . . . , and the N-th intermediate layer 411 of the neural network B. Then, in a step S615, the GPU 212 executes the processing in the output layer B 412 of the neural network B. As a result, the classification result data 414 obtained by the neural network B is output.
Then, in a step S616, the CPU 209 transmits the classification result data 414 obtained by the neural network B to the image capturing apparatus 101 via the communication section 211. Then, the present process is terminated.
The present invention has been described based on the above-described embodiment, but the present invention is not limited to the above-described embodiment. For example, the server 103 that is relatively high in computational power may perform inference processing using a plurality of neural networks. The following description will be given of a configuration in which the server 103 performs inference processing using two neural networks.
The server 103 acquires the above-mentioned second intermediate layer output data 413 from the image capturing apparatus 101 and inputs the acquired second intermediate layer output data 413 to the input layer 408. The server 103 executes processing operations in the third intermediate layer 409, the fourth intermediate layer 410, . . . , the N-th intermediate layer 411, and the output layer B 412 of the neural network B. As a result, the classification result data 414 obtained by the neural network B is output. Further, the server 103 executes processing operations in a third intermediate layer 701, a fourth intermediate layer 702, . . . , an I-th intermediate layer 703, and an output layer C 704 of the neural network C. As a result, classification result data 705 obtained by the neural network C is output. These processing operations are realized by the CPU 209 and the GPU 212 of the server 103 that execute programs stored e.g. in the ROM of the server 103.
The server 103 transmits the classification result data 414 obtained by the neural network B and the classification result data 705 obtained by the neural network C to the image capturing apparatus 101 via the communication network 102. Note that the number of intermediate layers of the neural network C need not be the same as the number of intermediate layers of the neural network A or the number of intermediate layers of the neural network B, but these numbers can be set as desired. Further, in the intermediate layers other than the layers commonized between the neural network A, the neural network B, and the neural network C, the number of nodes of each intermediate layer may be set as desired.
The learning of the neural network C is also performed by commonizing the input layer 401, the first intermediate layer 402, and the second intermediate layer 403, as described above. More specifically, for the learning of the neural network C, parameters of the input layer 401, the first intermediate layer 402, and the second intermediate layer 403 are fixed to the same parameters as used for the learning of the neural network A and the neural network B. Further, in the inference processing system 100, the image capturing apparatus 101 is caused to perform the processing operations up to the second intermediate layer 403, and the server 103 is caused to perform, by using the second intermediate layer output data 413 obtained by the processing operations performed by image capturing apparatus 101, the processing operations in the third intermediate layer 409 and following intermediate layers of the neural network B and the processing operations in the third intermediate layer 701 and following intermediate layers of the neural network C. With this, it is possible to obtain the respective inference results of the neural network A, the neural network B, and the neural network C, while reducing the computational load on the image capturing apparatus 101 which is related to the inference processing.
Further, in the above-described embodiment, there may be employed a configuration that can control which neural networks are to be used by the image capturing apparatus 101 and the server 103, respectively. For example, the image capturing apparatus 101 stores a table shown in
For example, in a case where a user sets an operation mode of the image capturing apparatus 101 to a mode for performing normal image capturing, the image capturing apparatus 101 performs inference processing using the neural network A, and the server 103 performs inference processing using the neural network B and the neural network C.
Further, when continuous photographing is performed, the auto focus control and the control for changing a shutter speed are performed only at the start of photographing and are not performed during photographing. That is, the neural network A is used only at the start of photographing. For this reason, in a case where the user sets the operation mode of the image capturing apparatus 101 to a mode for performing continuous photographing, the image capturing apparatus 101 performs inference processing using the neural network A and the neural network C, and the server 103 performs inference processing using the neural network B. With this, the image capturing apparatus 101 can determine a best shot during continuous photographing using the neural network C and display the best-shot image immediately after completion of the continuous photographing. Note that in the above-mentioned case as well, the second intermediate layer output data 413 obtained by executing the processing operations in the intermediate layers up to the predetermined intermediate layer, which are commonized, is also transmitted from the image capturing apparatus 101 to the server 103.
Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a ‘non-transitory computer-readable storage medium’) to perform the functions of one or more of the above-described embodiment(s) and/or that includes one or more circuits (e.g., application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiment(s), and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s) and/or controlling the one or more circuits to perform the functions of one or more of the above-described embodiment(s). The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.
While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.
This application claims the benefit of Japanese Patent Application No. 2022-029316 filed Feb. 28, 2022, which is hereby incorporated by reference herein in its entirety.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2022-029316 | Feb 2022 | JP | national |
