INFORMATION PROCESSING METHOD AND APPARATUS, AND STORAGE MEDIUM

TECHNICAL FIELD

The present disclosure relates to the field of information processing, and in particular, to an information processing method and apparatus, an electronic device, and a storage medium.

BACKGROUND

With the help of powerful performance advantages, a convolutional neural network promotes significant progress in the fields of computer vision, natural language processing, etc., and becomes a research upsurge in industry and academia. However, because a deep convolutional neural network is limited by a large number of matrix operations, massive storage and computing resources are often required. Reducing the redundancy of a convolution unit in the neural network is one of the important ways to solve this problem. Group convolution is a mode of channel group convolution and is widely applied to various networks.

SUMMARY

The present disclosure provides the technical solution of executing information processing of input information by means of a neural network.

According to one aspect of the present disclosure, an information processing method is provided, applied to a neural network and including:

inputting received input information into a neural network;

processing the input information by means of the neural network, where in the case that convolution processing is executed by means of a convolution layer of the neural network, a convolution kernel of the convolution layer is updated by using a transformation matrix configured for the convolution layer, so that the convolution processing of the convolution layer is completed by means of the updated convolution kernel; and

outputting a processing result of the processing of the neural network.

In some possible implementations, updating the convolution kernel of the convolution layer by using the transformation matrix configured for the convolution layer includes:

acquiring a space dimension of the convolution kernel of the convolution layer;

executing duplication processing on the transformation matrix corresponding to the convolution layer based on the space dimension of the convolution kernel, where the number of times of duplication processing is determined by the space dimension of the convolution kernel; and

executing dot product processing on the transformation matrix after the duplication processing and the convolution kernel to obtain the updated convolution kernel of the corresponding convolution layer.

In some possible implementations, before executing convolution processing by means of the convolution layer of the neural network, the method further includes:

determining a matrix unit constituting the transformation matrix corresponding to the convolution layer, where the matrix unit includes a first matrix and a second matrix, or only includes the second matrix, where in response to the number of channels of an input feature and the number of channels of an output feature of the convolution layer being different, the transformation matrix corresponding to the convolution layer includes the first matrix and the second matrix; and in response to the number of channels of the input feature and the number of channels of the output feature of the convolution layer being identical, the transformation matrix corresponding to the convolution layer includes the second matrix, where the first matrix is formed by connecting unit matrixes, and the second matrix is obtained by inner products of function transformations of a plurality of sub-matrixes; and

forming the transformation matrix of the convolution layer based on the determined matrix unit.

In some possible implementations, determining the second matrix constituting the transformation matrix of the convolution layer includes:

acquiring a gate control parameter configured for each convolution layer;

determining the sub-matrixes constituting the second matrix based on the gate control parameter; and

forming the second matrix based on the determined sub-matrixes.

In some possible implementations, acquiring the gate control parameter configured for each convolution layer includes:

acquiring the gate control parameter configured for each convolution layer according to received configuration information; or

determining the gate control parameter configured for the convolution layer based on a training result of the neural network.

In some possible implementations, forming the transformation matrix of the convolution layer based on the determined matrix unit includes:

acquiring the number of first channels of the input feature and the number of second channels of the output feature of each convolution layer;

in response to the number of the first channels being greater than the number of the second channels, forming the transformation matrix as a product of the first matrix and the second matrix; and

in response to the number of the first channels being less than the number of the second channels, forming the transformation matrix as a product of the second matrix and the first matrix.

In some possible implementations, determining the sub-matrixes constituting the second matrix based on the gate control parameter includes:

performing function processing on the gate control parameter by using a sign function to obtain a binaryzation vector; and

obtaining a binaryzation gate control vector based on the binaryzation vector, and obtaining the plurality of sub-matrixes based on the binaryzation gate control vector, a first basic matrix, and a second basic matrix.

In some possible implementations, obtaining the binaryzation gate control vector based on the binaryzation vector includes:

determining the binaryzation vector as the binaryzation gate control vector; or

determining a product of a permutation matrix and the binaryzation vector as the binaryzation gate control vector.

In some possible implementations, obtaining the plurality of sub-matrixes based on the binaryzation gate control vector, the first basic matrix, and the second basic matrix includes:

in response to an element in the binaryzation gate control vector being a first numerical value, obtaining a sub-matrix of an all-ones matrix; and

in response to the element in the binaryzation gate control vector being a second numerical value, obtaining a sub-matrix of the unit matrix.

In some possible implementations, the first basic matrix is the all-ones matrix, and the second basic matrix is the unit matrix.

In some possible implementations, forming the second matrix based on the determined sub-matrixes includes:

performing an inner product operation on the plurality of sub-matrixes to obtain the second matrix.

In some possible implementations, the input information includes at least one of text information, image information, video information, or voice information.

In some possible implementations, the dimension of the transformation matrix is a product of the number of the first channels and the number of the second channels, the number of the first channels is the number of channels of the input feature of the convolution layer, the number of the second channels is the number of channels of the output feature of the convolution layer, and an element of the transformation matrix includes at least one of 0 or 1.

In some possible implementations, the method further includes a step of training the neural network, which includes:

acquiring a training sample and a real detection result for monitoring;

performing processing on the training sample by using the neural network to obtain a prediction result; and

feeding back and adjusting a network parameter of the neural network based on a loss corresponding to the prediction result and the real detection result, until a termination condition is satisfied, where the network parameter includes the convolution kernel of each network layer and the transformation matrix.

According to the second aspect of the present disclosure, an information processing apparatus is provided, including:

an input module configured to input received input information to a neural network;

an information processing module configured to process the input information by means of the neural network, where in the case that convolution processing is executed by means of a convolution layer of the neural network, a convolution kernel of the convolution layer is updated by using a transformation matrix configured for the convolution layer, so that the convolution processing of the convolution layer is completed by means of the updated convolution kernel; and

an output module configured to output a processing result of the processing of the neural network.

In some possible implementations, the information processing module is further configured to: acquire a space dimension of the convolution kernel of the convolution layer;

execute duplication processing on the transformation matrix corresponding to the convolution layer based on the space dimension of the convolution kernel, where the number of times of duplication processing is determined by the space dimension of the convolution kernel; and

execute dot product processing on the transformation matrix after the duplication processing and the convolution kernel to obtain the updated convolution kernel of the corresponding convolution layer.

In some possible implementations, the information processing module is further configured to determine a matrix unit constituting the transformation matrix corresponding to the convolution layer, and form the transformation matrix of the convolution layer based on the determined matrix unit, where the matrix unit includes a first matrix and a second matrix, or only includes the second matrix, where in response to the number of channels of an input feature and the number of channels of an output feature of the convolution layer being different, the transformation matrix corresponding to the convolution layer includes the first matrix and the second matrix; and in response to the number of channels of the input feature and the number of channels of the output feature of the convolution layer being identical, the transformation matrix corresponding to the convolution layer includes the second matrix, where the first matrix is formed by connecting unit matrixes, and the second matrix is obtained by inner products of function transformations of a plurality of sub-matrixes.

In some possible implementations, the information processing module is further configured to acquire a gate control parameter configured for each convolution layer;

determine the sub-matrixes constituting the second matrix based on the gate control parameter; and

form the second matrix based on the determined sub-matrixes.

In some possible implementations, the information processing module is further configured to acquire the gate control parameter configured for each convolution layer according to received configuration information; or

determine the gate control parameter configured for the convolution layer based on a training result of the neural network.

In some possible implementations, the information processing module is further configured to acquire the number of first channels of the input feature and the number of second channels of the output feature of each convolution layer;

in response to the number of the first channels being greater than the number of the second channels, form the transformation matrix as a product of the first matrix and the second matrix; and

in response to the number of the first channels being less than the number of the second channels, form the transformation matrix as a product of the second matrix and the first matrix.

In some possible implementations, the information processing module is further configured to perform function processing on the gate control parameter by using a sign function to obtain a binaryzation vector; and

obtain a binaryzation gate control vector based on the binaryzation vector, and obtain the plurality of sub-matrixes based on the binaryzation gate control vector, a first basic matrix, and a second basic matrix.

In some possible implementations, the information processing module is further configured to determine the binaryzation vector as the binaryzation gate control vector; or

determine a product of a permutation matrix and the binaryzation vector as the binaryzation gate control vector.

In some possible implementations, the information processing module is further configured to obtain a sub-matrix of an all-ones matrix in the case that an element in the binaryzation gate control vector is a first numerical value; and

obtain a sub-matrix of the unit matrix in the case that an element in the binaryzation gate control vector is a second numerical value.

In some possible implementations, the first basic matrix is the all-ones matrix, and the second basic matrix is the unit matrix.

In some possible implementations, the information processing module is further configured to perform an inner product operation on the plurality of sub-matrixes to obtain the second matrix.

In some possible implementations, the input information includes at least one of text information, image information, video information, or voice information.

In some possible implementations, the information processing module is further configured to train the neural network, where the step of training the neural network includes:

acquiring a training sample and a real detection result for monitoring;

performing processing on the training sample by using the neural network to obtain a prediction result; and

According to the third aspect of the present disclosure, an electronic device is provided, including: a processor; and a memory configured to store processor-executable instructions; where the processor is configured to call the instructions stored in the memory, so as to execute the method according to any one in the first aspect.

According to the fourth aspect of the present disclosure, a computer-readable storage medium is provided, having computer program instructions stored thereon, where when the computer program instructions are executed by a processor, the method according to any one of the first aspect is implemented.

In embodiments of the present disclosure, input information is input to a neural network to execute corresponding operation processing, where when convolution processing of a convolution layer of the neural network is executed, a convolution kernel of the convolution layer is updated based on a transformation matrix determined for each convolution layer, and corresponding convolution processing is completed by using the new convolution kernel. By means of the method, a corresponding transformation matrix is individually configured for each convolution layer, and a corresponding group effect is formed, where the group is not limited to a group of adjacent channels; moreover, the operation precision of a neural network can be further improved.

It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the present disclosure.

The other features and aspects of the present disclosure can be described more clearly according to the detailed descriptions of the exemplary embodiments in the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings here incorporated in the description and constituting a part of the description describe the embodiments of the present disclosure and are intended to explain the technical solutions of the present disclosure together with the description.

FIG. 1 shows a flow chart of an information processing method according to embodiments of the present disclosure;

FIG. 2 shows a flow chart of updating a convolution kernel in an information processing method according to embodiments of the present disclosure;

FIG. 3 shows a schematic diagram of an existing conventional convolution operation;

FIG. 4 shows a schematic diagram of an existing convolution operation of group convolution;

FIG. 5 shows a schematic structural diagram of different transformation matrixes according to embodiments of the present disclosure;

FIG. 6 shows a flow chart of determining a transformation matrix in an information processing method according to embodiments of the present disclosure;

FIG. 7 shows a flow chart of a method for determining a second matrix constituting a transformation matrix of a convolution layer in an information processing method according to embodiments of the present disclosure;

FIG. 8 shows a flow chart of step S1012 in an information processing method according to embodiments of the present disclosure;

FIG. 9 shows a flow chart of step S103 in an information processing method according to embodiments of the present disclosure;

FIG. 10 shows a flow chart of training a neural network according to embodiments of the present disclosure;

FIG. 11 shows a block diagram of an information processing apparatus according to embodiments of the present disclosure;

FIG. 12 shows a block diagram of an electronic device according to embodiments of the present disclosure;

FIG. 13 shows another block diagram of an electronic device according to embodiments of the present disclosure.

DETAILED DESCRIPTION

Various exemplary embodiments, features, and aspects of the present disclosure are described below in detail with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements having the same or similar functions. Although the various aspects of the embodiments are illustrated in the accompanying drawings, unless stated particularly, it is not required to draw the accompanying drawings in proportion.

The special word “exemplary” here means “used as examples, embodiments, or descriptions”. Any “exemplary” embodiment given here is not necessarily construed as being superior to or better than other embodiments.

The term “and/or” as used herein is merely the association relationship describing the associated objects, indicating that there may be three relationships, for example, A and/or B, which may indicate that A exists separately, both A and B exist, and B exists separately. In addition, the term “at least one” as used herein means any one of multiple elements or any combination of at least two of the multiple elements, for example, including at least one of A, B, or C, which indicates that any one or more elements selected from a set consisting of A, B, and C are included.

In addition, numerous details are given in the following detailed description for the purpose of better explaining the present disclosure. It should be understood by persons skilled in the art that the present disclosure may still be implemented even without some of those details. In some examples, methods, means, elements, and circuits that are well known to persons skilled in the art are not described in detail so that the principle of the present disclosure becomes apparent.

It should be understood that the foregoing various method embodiments mentioned in the present disclosure may be combined with each other to form a combined embodiment without departing from the principle logic. Details are not described herein again due to space limitation.

In addition, the present disclosure further provides an information processing apparatus, an electronic device, a computer-readable storage medium, and a program, which can all be used to implement any of the information processing methods provided by the present disclosure. For the corresponding technical solutions and descriptions, please refer to the corresponding content in the method section. Details are not described herein again.

An execution subject of the information processing apparatus in the embodiments of the present disclosure may be any electronic device or server, for example, an image processing device having an image processing function, a voice processing device having a voice processing function, and a video processing device having a video processing function, or the like, which may be mainly determined according to information to be processed. The electronic device may be a User Equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a Personal Digital Assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, or the like. In some possible implementations, the information processing method may also be implemented by a processor by invoking computer-readable instructions stored in a memory.

FIG. 1 shows a flow chart of an information processing method according to embodiments of the present disclosure. As shown in FIG. 1, the information processing method includes the following steps.

At S10, received input information is input into a neural network.

In some possible implementations, the input information may include at least one of a number, an image, a text, an audio, or a video, or other information may also be included in other implementations, which is not specifically defined in the present disclosure.

In some possible implementations, the information processing method provided in the present disclosure may be implemented by means of the neural network, and the neural network may be a trained network that can execute corresponding processing of the input information and satisfies the precision requirements. For example, the neural network in the embodiments of the present disclosure is a convolutional neural network, which may be a neural network having functions of target detection and target identification, so that detection and identification of a target object in a received image may be implemented, where the target object may be any type of object such as pedestrian, human face, vehicle, and animal, and may be specifically determined according to application scenes.

When processing of the input information is executed by means of the neural network, i.e., the input information is input to the neural network, a corresponding operation is executed by means of each network layer of the neural network. The neural network may include at least one convolution layer.

At S20, the input information is processed by means of the neural network, where in the case that convolution processing is executed by means of the convolution layer of the neural network, a convolution kernel of the convolution layer is updated by using a transformation matrix configured for the convolution layer, so that the convolution processing of the convolution layer is completed by means of the updated convolution kernel.

In some possible implementations, after the input information is input to the neural network, operation processing may be performed on the input information by means of the neural network, for example, operations such as vector operation or matrix operation, or addition, subtraction, multiplication and division operations may be executed for a feature of the input information. A specific operation type may be determined according to the structure of the neural network. In some embodiments, the neural network may include at least one convolution layer, a pooling layer, a full connection layer, a residual network, and a classifier, or other network layers may also be included in other embodiments, which is not specifically defined in the present disclosure.

When convolution processing in the neural network is executed, the embodiments of the present disclosure may update the convolution kernel of convolution operation of each convolution layer according to the transformation matrix configured for each convolution layer of the neural network. Different transformation matrixes may be configured for each convolution layer, the same transformation matrix may also be configured for each convolution layer, and the transformation matrix may also be a parameter matrix obtained by training and learning of the neural network, and may be specifically set according to requirements and application scenes. The dimension of the transformation matrix in the embodiments of the present disclosure is a product of the number of first channels of an input feature and the number of second channels of an output feature of the convolution layer, and may be, for example, C_in×C_out, where C_inis the number of channels of the input feature of the convolution layer, and C_outindicates the number of channels of the output feature of the convolution layer, and the transformation matrix may be constructed as a binaryzation matrix, where an element in the binaryzation matrix includes at least one of 0 or 1, i.e., the transformation matrix in the embodiments of the present disclosure may be a matrix consisting of at least one element of 0 or 1.

In some possible implementations, the transformation matrix corresponding to each convolution layer may be a matrix obtained by the training of the neural network, where when the neural network is trained, the transformation matrix may be introduced, and the transformation matrix that satisfies training requirements and is adapted to a training sample is determined in combination with a feature of the training sample. That is, the transformation matrix configured for each convolution layer in the embodiments of the present disclosure may enable a convolution mode of the convolution layer to adapt to a sample feature of the training sample, for example, different group convolutions of different convolution layers may be implemented. In order to improve the application precision of the neural network, in the embodiments of the present disclosure, the type of the input information is the same as that of the training sample used for training the neural network.

In some possible implementations, the transformation matrix of each convolution layer may be determined according to received configuration information, where the configuration information is information on the transformation matrix of the convolution layer. Furthermore, each transformation matrix is a set transformation matrix adapted to the input information, i.e., a transformation matrix that can obtain an accurate processing result. A method for receiving the configuration information may include receiving configuration information transmitted by other devices, or reading pre-stored configuration information and the like, which is not specifically defined in the present disclosure.

After obtaining the transformation matrix configured for each convolution layer, i.e., obtaining a new convolution kernel based on the configured transformation matrix, i.e., updating of the convolution kernel of the convolution layer is completed, where the convolution kernel is a convolution kernel determined by a convolution mode used in convolution processing in the prior art. When the neural network is trained, a specific parameter of the convolution kernel before updating may be obtained by means of training.

At S30, a processing result of the processing of the neural network is output.

After processing of the neural network, i.e., the processing result of the input information by the neural network may be obtained, the processing result may be output.

In some possible implementations, the input information may be image information, the neural network may be a network that detects the type of an object in the input information. In this case, the processing result may be the type of an object included in the image information. Alternatively, the neural network may detect a positional area where an object of a target type in the input information is located. In this case, the processing result is the positional area of the object of the target type included in the image information, where the processing result may also be a matrix form, which is not specifically defined in the present disclosure.

The steps of the information processing method in embodiments of the present disclosure are respectively described in detail below with reference to the accompanying drawings, where after the transformation matrix configured for each convolution layer is obtained, the convolution kernel of the corresponding convolution layer may be correspondingly updated according to the configured transformation matrix. FIG. 2 shows a flow chart of updating a convolution kernel in an information processing method according to embodiments of the present disclosure, where updating the convolution kernel of the convolution layer by the transformation matrix configured for the convolution layer includes the following steps.

At S21, a space dimension of the convolution kernel of the convolution layer is acquired.

In some possible implementations, after acquiring the transformation matrix configured for each convolution layer, an updating procedure of the convolution kernel may be executed, where the space dimension of the convolution kernel of each convolution layer may be acquired. For example, the dimension of the convolution kernel of each convolution layer in the neural network may be represented as k×k×C_in×C_out, where k×k is a space dimension of the convolution kernel, k is an integer greater than or equal to 1, may be, for example, a numerical value such as 3 or 5, and may be specifically determined according to the structure of the neural network; C_inis the number of channels of the input feature of the convolution layer (the number of first channels), and C_outindicates the number of channels of the output feature of the convolution layer (the number of second channels).

At S22, duplication processing is executed on the transformation matrix corresponding to the convolution layer based on the space dimension of the convolution kernel, where the number of times of duplication processing is determined by the space dimension of the convolution kernel.

In some possible implementations, the duplication processing may be executed on the transformation of the convolution layer based on the space dimension of the convolution kernel of the convolution layer, i.e., k×k transformation matrixes are duplicated, and a new matrix is formed by using the duplicated k×k transformation matrixes, where the formed new matrix has the same dimension as the convolution kernel.

At S23, dot product processing is executed on the transformation matrix after the duplication processing and the convolution kernel to obtain the updated convolution kernel of the corresponding convolution layer.

In some possible implementations, an updated convolution kernel may be obtained by using a dot product of the new matrix formed by the duplicated k×k transformation matrixes and the convolution kernel.

In some possible implementations, an expression for executing the convolution processing by using the updated convolution kernel in the present disclosure may include:

f
_i,j′=Σ_m=0^k−1(U·ω_m,n)f_(i+m,j+n)+b (1),

where f_(i+m,j+n)represents a feature unit in the (i+m)^−throw and the (j+n)^−thcolumn in an input feature F_inof the convolution layer, the size of F_inmay be represented as N×C_in×H×W, N represents a sample amount of input features of the convolution layer, C_inrepresents the number of channels of the input feature, H and W respectively represent the height and width of the input feature of a single channel, and f_(i+m,j+n)∈R^N×Cⁱⁿ; f_i,j′ represents a feature unit in the i-th row and the j-th column in an output feature F_outof the convolution layer, F_out∈R^N×C^out^×H′×W′, c_outrepresents the number of channels of the output feature, H′×W′ represents the height and width of the output feature of a single channel, ω_m,nrepresents a convolution unit in the m-th row and nth column in the convolution kernel of the convolution layer, the space dimension of the convolution kernel is k row and k column, U is the transformation matrix configured by the convolution layer (having the same dimension as the convolution unit), and b represents an optional bias term, which may be a numerical value greater than or equal to 0.

By means of the method above, an updating procedure of the convolution kernel of each convolution layer may be completed. Because the transformation matrix configured for each convolution layer may be in a different form, any convolution operation may be implemented.

In the prior art, when group convolution of convolution processing is implemented in the neural network, several important defects still exist in previous group convolution:

(1) a convolution parameter is determined depending on an artificial design mode, and an appropriate group num needs to be found by means of tedious experimental verification, so that said mode is not easy to popularize in practical application;

(2) existing applications all use the same type of group convolution strategy for all convolution layers of the whole network, such that on the one hand, it is difficult to manually select the group convolution strategy suitable for the whole network, and on the other hand, such an operation mode may not make the performance of the neural network reach an optimal state; and

(3) moreover, some group methods only divide the convolution features of adjacent channels into the same group, and such an easy-to-implement mode greatly ignores the relevance of feature information of different channels.

However, according to the embodiments of the present disclosure, individual meta convolution processing of each convolution layer is implemented by means of the adaptive transformation matrix configured for each convolution layer. In the case that the transformation matrix is a parameter obtained by the training of the neural network, independent learning of any group convolution scheme can be implemented for a deep neural network convolution layer without human intervention. Respective different group strategies are configured for different convolution layers of the neural network. A meta convolution method provided in the embodiments of the present disclosure may be applied to any convolution layer of a deep neural network, so that the convolution layers having different depths of the network all can independently select the optimal channel group scheme adapted to the current feature expression by means of learning. The convolution processing of the present disclosure has diversity. The meta convolution method is represented by a transformation matrix form, so that not only the existing adjacent group convolution technology may be expressed, but any channel group scheme can be expanded, the relevance of feature information of different channels is increased, and the cutting-edge development of a convolution redundancy elimination technology is promoted. In addition, the convolution processing provided in the embodiments of the present disclosure is further simple. The network parameter is decomposed by using Kronecker (Kronecker product) operation, and the meta convolution method provided in the present disclosure has the advantages such as small computation, small number of parameters, and easy implementation and application by means of a differentiable end-to-end training mode. The present disclosure further has versatility and is applicable to different network models and visual tasks. The meta convolution method may be easily and effectively applied to various convolutional neural networks to achieve excellent effects on various vision tasks, such as image classification (CIFAR10, ImageNet), target detection and identification (COCO, Kinetics), and image segmentation (Cityscapes, ADE2k).

FIG. 3 shows a schematic diagram of an existing conventional convolution operation. FIG. 4 shows a schematic diagram of an existing convolution operation of group convolution. As shown in FIG. 3, for an ordinary convolution operation, each channel of output features of C_outchannels is obtained by performing a convolution operation of input features of all C_inchannels together. As shown in FIG. 4, conventional group convolution relates to performing grouping on dimension of channel, so as to arrive at the purpose of reducing the number of parameters. FIG. 4 intuitively indicates the group convolution operation having the group num of 2, i.e., the input features of every C_in/2 channels is one group, which is convoluted with the weight of the dimension

$\frac{C_{in}}{2} \times \frac{C_{out}}{2} \times k \times k,$

so that a group of output features of

$\frac{C_{out}}{2}$

channels is obtained. In this case, the total weight dimension is

$2 \times \frac{C_{in}}{2} \times \frac{C_{out}}{2} \times k \times k,$

and the number of parameters is 2 times less than the ordinary convolution. Usually, the group num of the mode is manually set, and can be exactly divided by C_in. When the group num equals the number of channels C_inof the input feature, it is equivalent to respectively performing the convolution operation on the feature of each channel.

To understand a procedure of updating the convolution kernel by means of the transformation matrix to implement a new convolution mode (meta convolution) provided in the embodiments of the present disclosure more clearly, description is provided below by means of examples.

As stated in the foregoing embodiments, a transformation matrix U∈{0,1}^Cⁱⁿ^×C^outis a binaryzation matrix capable of learning, in which each element is either 0 or 1, and the dimension is the same as ω_m,n. In the embodiments of the present disclosure, performing dot product on a transformation matrix U and a convolution unit ω_m,nof the convolution layer is equivalent to performing sparse expression on the weight. Different Us represent different convolution operation methods, for example: FIG. 5 shows a schematic structural diagram of different transformation matrixes according to the embodiments of the present disclosure.

(1) When U is in the form of matrix a in FIG. 5, U is an all-ones matrix, where when a new convolution kernel is formed by using the transformation matrix, equivalent to changing the convolution kernel of the convolution operation, meta convolution represents an ordinary convolution operation, which corresponds to the convolution mode in FIG. 3. In this case C_in=8, C_out=4, and the group num is 1.

(2) When U is in the form of matrix b in FIG. 5, U is a block diagonal matrix, where when a new convolution kernel is formed by using the transformation matrix, the meta convolution represents the group convolution operation, which corresponds to the convolution mode in FIG. 4. In this case, C_in=8, C_out=4, and the group num is 2.

(3) When U is in the form of matrix c in FIG. 5, U is a block diagonal matrix, where when a new convolution kernel is formed by using the transformation matrix, the meta convolution represents the group convolution operation having the group num of 4, and similarly, C_in=8, C_out=4.

(4) When U is in the form of matrix d in FIG. 5, U is a unit matrix, where when a new convolution kernel is formed by using the transformation matrix, the meta convolution represents a group convolution operation that individual convolution is respectively performed on the feature of each channel. In this case, C_in=C_out=8, and the group num is 8.

(5) When U is a matrix of matrix g in FIG. 5, the meta convolution represents a convolution operation mode that has never happened before, where the output feature of each C_outchannel is not obtained by the input features of fixed adjacent C_inchannels. In this case any channel group scheme is possible. Matrix g may be a matrix obtained by means of matrixes e and f, and f in FIG. 5 represents a convolution form corresponding to matrix g.

It can be known from the foregoing exemplary descriptions that a method for updating the convolution kernel by means of the transformation matrix to implement meta convolution provided in the present disclosure achieves the sparse representation of the weight of the convolution layer, so that not only the existing convolution operation can be expressed, but also any channel group convolution scheme that has never happened before can be expanded. The method has richer expression capability than the previous convolution technology. Meanwhile, different from the previous convolution method in which the group num is artificially designed, the meta convolution can independently learn and adapt to the convolution scheme of the current data.

If the meta convolution method provided in the embodiments of the present disclosure is applied to any convolution layer of the deep neural network, the meta convolution method may be that the convolution layers having different depths of the network independently select the optimal channel group scheme adapted to the current feature expression by means of learning, where a corresponding binarization diagonal block matrix U is configured for each convolution layer, that is to say, in a deep neural network having L hidden layers, the meta convolution method brings a learning parameter of dimensional C_in×C_out×L. For example, in a 100-layer deep network, if the number of channels of each layer of a feature map is 1,000, millions of parameters are brought.

In some possible implementations, a configured transformation matrix may be directly obtained according to received configuration information, and the transformation matrix of each convolution layer may be directly determined by means of training of the neural network. In addition, in order to further reduce the optimization difficulty of the transformation matrix and reduce the amount of operation parameters, the embodiments of the present disclosure divide the transformation matrix into two matrixes multiplied by each other. That is to say, the transformation matrix in the embodiments of the present disclosure may include a first matrix and a second matrix, where the first matrix and the second matrix may be acquired according to the received configuration information, or the first matrix and the second matrix are obtained according to a training result. The first matrix is formed by connecting unit matrixes, and the second matrix is obtained by inner products of function transformations of a plurality of sub-matrixes. The transformation matrix may be obtained by means of a product of the first matrix and the second matrix.

FIG. 6 shows a flow chart of determining a transformation matrix in an information processing method according to embodiments of the present disclosure. Before executing the convolution processing by means of the convolution layer of the neural network, the transformation matrix corresponding to the convolution layer is determined. The step includes the following steps.

At S101, a matrix unit constituting the transformation matrix corresponding to the convolution layer is determined, where the matrix unit includes a second matrix, or includes a first matrix and the second matrix, where in response to the number of channels of an input feature and the number of channels of an output feature of the convolution layer being different, the transformation matrix corresponding to the convolution layer includes the first matrix and the second matrix; and in response to the number of channels of the input feature and the number of channels of the output feature of the convolution layer being identical, a binaryzation matrix corresponding to the convolution layer includes the second matrix, where the first matrix is formed by connecting unit matrixes, and the second matrix is obtained by inner products of function transformations of the plurality of sub-matrixes.

At S102, the transformation matrix of the convolution layer is formed based on the determined matrix unit.

In some possible implementations, for the case that the numbers of channels of the input feature and the output feature of the convolution layer are the same or different, the matrix unit constituting the transformation matrix may be determined in different modes. For example, in the case that the number of channels of the input feature and the number of channels of the output feature of the convolution layer are the same, the matrix unit constituting the transformation matrix of the convolution layer is the second matrix, and in the case that the number of channels of the input feature and the number of channels of the output feature of the convolution layer are different, the matrix unit constituting the transformation matrix of the convolution layer may be the first matrix and the second matrix.

In some possible implementations, the first matrix and the second matrix corresponding to the transformation matrix may be obtained according to the received configuration information, and related parameters of the first matrix and the second matrix may also be trained and learned by means of the neural network.

In the embodiments of the present disclosure, the first matrix constituting the transformation matrix is formed by connecting the unit matrixes, and in the case that the number of first channels of the input feature and the number of second channels of the output feature of the convolution layer are determined, the dimensions of the first matrix and the second matrix may be determined. In the case that the number of the first channels is greater than the number of the second channels, the dimension of the first matrix is C_in×C_out, and the dimension of the second matrix is C_out×C_out. In the case that the number of the first channels is less than the number of the second channels, the dimension of the first matrix is C_in×C_out, and the dimension of second matrix Ũ is C_in×C_in. In the embodiments of the present disclosure, the dimension of the first matrix may be determined based on the number of the first channels of the input feature and the number of the second channels of the output feature of the convolution layer, and a plurality of unit matrixes forming the first matrix by means of connection may be determined based on the dimension, where the form of the first matrix may be easily obtained because the unit matrix is a square matrix.

For the second matrix forming the transformation matrix, the embodiments of the present disclosure may determine the second matrix according to an obtained gate control parameter. FIG. 7 shows a flow chart of a method for determining a second matrix of a transformation matrix of a convolution layer in an information processing method according to the embodiments of the present disclosure, where determining the second matrix constituting the transformation matrix of the convolution layer includes the following steps.

At S1011, a gate control parameter configured for each convolution layer is acquired.

At S1012, the sub-matrixes constituting the second matrix are determined based on the gate control parameter.

At S1013, the second matrix is formed based on the determined sub-matrixes.

In some possible implementations, the gate control parameter may include a plurality of numerical values, which may be floating point type decimals near 0, such as a float 64-bit or 32-bit decimal, which is not specifically defined in the present disclosure. The received configuration information may include the continuous numerical values, or the neural network may also learn and determine the continuous numerical values by training.

In some possible implementations, the second matrix may be obtained by means of the inner product operation of the plurality of sub-matrixes, the gate control parameter obtained by means of step S1011 may form the plurality of sub-matrixes, and then, the second matrix is obtained according to an inner product operation result of the plurality of sub-matrixes.

FIG. 8 shows a flow chart of step S1012 in an information processing method according to embodiments of the present disclosure, where determining the sub-matrixes constituting the second matrix based on the gate control parameter may include the following steps.

At S10121, function processing is performed on the gate control parameter by using a sign function to obtain a binaryzation vector.

In some possible implementations, each parameter numerical value of the gate control parameter may be input to the sign function, a corresponding result may be obtained by means of processing of the sign function, and the binaryzation vector may be constituted based on an operation result of the sign function corresponding to each gate control parameter.

The expression of the binaryzation vector may be represented as:

g=sign({tilde over (g)}) (2);

where {tilde over (g)} is a gate control parameter, and g is a binaryzation vector. For sign function f(a)=sign(a), if a is greater than or equal to zero, sign(a) equals 1, and if a is less than zero, sign(a) equals 0. Therefore, after the processing of the sign function, an element in the obtained binaryzation vector may include at least one of 0 or 1, and the number of elements is the same as the number of continuous numerical values of the gate control parameter.

At S10122, a binaryzation gate control vector is obtained based on the binaryzation vector, and the plurality of sub-matrixes is obtained based on the binaryzation gate control vector, a first basic matrix, and a second basic matrix.

In some possible implementations, the element of the binaryzation vector may be directly determined as the binaryzation gate control vector, i.e., no processing is performed on the binaryzation vector, where the expression of the binaryzation gate control vector may be: {right arrow over (g)}=g, where {right arrow over (g)} represents the binaryzation gate control vector. Furthermore, the plurality of sub-matrixes constituting the second matrix may be formed according to the binaryzation gate control vector, the first basic matrix, and the second basic matrix. In the embodiments of the present disclosure, the first matrix may be the all-ones matrix, and the second basic matrix is the unit matrix. A mode of a convolution group formed by the second matrix determined by means such a mode may be any group mode, such as a convolution form of g in FIG. 5.

In some other possible implementations, in order to implement the form of block group convolution of the convolution layer, the binaryzation gate control vector may be obtained by using a product of a permutation matrix and the binaryzation vector, where the permutation matrix may be an ascending sort matrix, in which the binarization vectors are ranked so that 0 in the obtained binarization gated vector is before 1, where the expression of the binaryzation gate control vector may be: {right arrow over (g)}=Pg, and P is a permutation matrix. Furthermore, the plurality of sub-matrixes constituting the second matrix may be formed according to the binaryzation gate control vector, the first basic matrix, and the second basic matrix.

In some possible implementations, obtaining the plurality of sub-matrixes based on the binaryzation gate control vector, the first basic matrix, and the second basic matrix may include: in response to an element in the binaryzation gate control vector being a first numerical value, obtaining a sub-matrix of an all-ones matrix; and in response to the element in the binaryzation gate control vector being a second numerical value, obtaining a sub-matrix of the unit matrix, where the first numerical value is 1, and the second numerical value is 0. That is to say, the sub-matrixes obtained in the embodiments of the present disclosure may be the all-ones matrix or the unit matrix, where a corresponding sub-matrix is the all-ones matrix when the element in the binaryzation gate control vector is 1, and a corresponding sub-matrix is the unit matrix when the element in the binaryzation gate control vector is 0.

In some possible implementations, the corresponding sub-matrix may be obtained for each element in the binaryzation gate control vector, where a mode for obtaining the sub-matrix may include:

obtaining a first vector by multiplying the element in the binaryzation gate control vector by the first basic matrix;

obtaining a second vector by multiplying the element in the binaryzation gate control vector by the second basic matrix; and

obtaining the corresponding sub-matrix by using a difference between a sum result of the first vector and the second basic matrix and the second vector.

The expression of obtaining the plurality of sub-matrixes may be:

Ũ
_i
=g
_i1+(1−g_i)I,∀g_i∈{right arrow over (g)} (3).

The i-th element g_iin binaryzation gate control vector {right arrow over (g)} may be multiplied by a first basic matrix 1 to obtain the first vector; the i-th element g_iis multiplied by a second basic matrix I to obtain the second vector, and a sum operation is performed on the first vector and the second basic vector to obtain a sum result; and the i-th sub-matrix Ũ_iis obtained by using a difference between the sum result and the second vector, where i is an integer greater than 0 and less than or equal to K, and K is the number of elements of the binaryzation gate control vector.

Based on the foregoing configuration of the embodiments of the present disclosure, the sub-matrixes may be determined based on the obtained gate control parameter, so as to further determine the second matrix. In the case of training and learning by means of the neural network, the learning of a second matrix Ũ of C×C dimension may be converted to the learning of a series of sub-matrixes Ũ_i, and the number of parameters is also reduced to

$\sum_{i} C_{i}^{2}$

from CXC, where i represents the number of sub-matrixes. For example, the second matrix may be decomposed to three sub-matrixes of 2×2 to perform the Kronecker inner product operation, i.e.:

$\begin{matrix} \tilde{U} = I \otimes I \otimes 1 = [\begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix}] \otimes [\begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix}] \otimes [\begin{matrix} 1 & 1 \\ 1 & 1 \end{matrix}] = [\begin{matrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{matrix}] \otimes [\begin{matrix} 1 & 1 \\ 1 & 1 \end{matrix}] = [\begin{matrix} 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\ 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\ 0 & 0 & 1 & 1 & 0 & 0 & 0 & 0 \\ 0 & 0 & 1 & 1 & 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 & 1 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 1 & 1 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 \\ 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 \end{matrix}] . & (4) \end{matrix}$

In this case, the number of parameters is reduced to 3×2{circumflex over ( )}2=12 from 8{circumflex over ( )}2=64. Obviously, the amount of operation of convolution processing may be reduced by means of the mode in the embodiments of the present disclosure.

As stated above, after obtaining the sub-matrixes, the second matrix may be obtained based on the inner product operation of the sub-matrixes, where the expression of the second matrix is:

Ũ=Ũ
₁
⊗Ũ
₂
⊗ . . . ⊗Ũ
_K;

where Ũ represents the second matrix, ⊗ represents the inner product operation, and Ũ_irepresents the i-th sub-matrix.

The inner product operation represents an operation between any two matrixes, and may be defined as:

$\begin{matrix} [\begin{matrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{matrix}] \otimes [\begin{matrix} b_{11} & b_{12} \\ b_{21} & b_{22} \end{matrix}] = [\begin{matrix} a_{11} b_{11} & a_{11} b_{12} & a_{12} b_{11} & a_{12} b_{12} \\ a_{11} b_{21} & a_{11} b_{22} & a_{12} b_{21} & a_{12} b_{22} \\ a_{21} b_{11} & a_{21} b_{12} & a_{22} b_{11} & a_{22} b_{12} \\ a_{21} b_{21} & a_{21} b_{22} & a_{22} b_{21} & a_{22} b_{22} \end{matrix}] . & (5) \end{matrix}$

By means of the foregoing configuration, the embodiments of the present disclosure may determine that the sub-matrixes of the second matrix are formed. If the number of the first channels of the input feature and the number of the second channels of the convolution layer are the same, the second matrix may be the transformation matrix; if the number of the first channels and the number of the second channels are different, the transformation matrix may be determined according to the first matrix and the second matrix. In this case, a long matrix (the transformation matrix) of C_in×C_outdimension is represented by using the first matrix formed by connecting the unit matrixes and a square matrix Ũ (the second matrix) of C×C dimension, where C is the smaller numerical value in the number of channels of the input feature C_inand the number of channels of the output feature C_outof the convolution layer, i.e., C=min(C_in,C_out).

FIG. 9 shows a flow chart of step S103 in an information processing method according to embodiments of the present disclosure. Forming the transformation matrix of the convolution layer based on the determined matrix unit includes the following steps.

At S1031, the number of first channels of the input feature and the number of second channels of the output feature of each convolution layer are acquired.

At S1032, in response to the number of the first channels being greater than the number of the second channels, the transformation matrix is formed as a product of the first matrix and the second matrix.

At S1033, in response to the number of the first channels being less than the number of the second channels, the transformation matrix is formed as a product of the second matrix and the first matrix.

As stated above, the embodiments of the present disclosure may acquire the first matrix and the second matrix constituting the transformation matrix, where the first matrix and the second matrix may be obtained based on the received configuration information as stated in the embodiments above, and may also be obtained by means of the training of the neural network. When the transformation matrix corresponding to each convolution layer is formed, a mode of forming the first matrix and the second matrix may be first determined according to the number of channels of the input feature and the number of channels of the output feature in the convolution layer.

If the number of channels (the number of first channels) of the input feature is greater than the number of channels (the number of second channels) of the output feature, the transformation matrix is a result of multiplying the first matrix by the second matrix. If the number of channels of the input feature is less than the number of channels of the output feature, the transformation matrix is a result of multiplying the second matrix by the first matrix. If the numbers of channels of the input feature and the output feature are the same, the transformation matrix may be determined by multiplying the first matrix by the second matrix or multiplying the second matrix by the first matrix.

In the case that C_inand C_outare equal, the second matrix in the embodiments of the present disclosure may serve as the transformation matrix. Descriptions are not made herein specifically. The determining of the first matrix and the second matrix that constitute the transformation matrix is described for the case that C_inand C_outare unequal below.

When C_inis greater than C_out, the transformation matrix equals a product of a first matrix Ĩ_dand a second matrix Ũ. In this case, the dimension of the first matrix Ĩ_dis C_in×C_out, the expression of the first matrix is Ĩ_d∈{0,1}^Cⁱⁿ^×C^out, the dimension of the second matrix Ũ is C_out×C_out, and the expression of the second matrix is Ũ∈{0,1}^C^out^×C^out. The first matrix and the second matrix each are a matrix consisting of at least one element of 0 or 1, and correspondingly, the expression of the transformation matrix U is: U=Ĩ_d×Ũ, where the first matrix Ĩ_dis formed by connecting unit matrixes I, the dimension of I is C_out×C_out, and the expression of the unit matrix I is I∈{0,1}^C^out^×C^out. For example, when the transformation matrix is a fringe matrix shown in g of FIG. 4, C_in=8 and C_out=4, the first matrix Ĩ_dhaving the dimension of 8×4 and the second matrix Ũ having the dimension of 4×4 may be constituted.

When C_inis less than C_out, the transformation matrix equals a product of a second matrix Ũ and a first matrix Ĩ_u, where the dimension of the first matrix Ĩ_uis C_in×C_out, the expression of the first matrix is Ĩ_u∈{0,1}^Cⁱⁿ^×C^out, the dimension of the second matrix Ũ is C_in×C_in, and the expression of the second matrix is Ũ∈{0,1}^Cⁱⁿ^×Cⁱⁿ. The first matrix and the second matrix each are a matrix consisting of at least one element of 0 or 1, and correspondingly, the expression of the transformation matrix U is: U=Ũ×Ĩ_u, where the first matrix Ĩ_uformed by connecting unit matrixes I, the dimension of I is C_in×C_in, and the expression of the unit matrix I is I∈{0,1}^Cⁱⁿ^×Cⁱⁿ.

By means of the mode above, the first matrix and the second matrix constituting the transformation matrix may be determined, where, as stated above, the first matrix is formed by connecting the unit matrixes, and after the number of channels of the input feature and the number of channels of the output feature are determined, the first matrix is also determined, accordingly. In the case that the dimension of the second matrix is known, an element value in the second matrix may be further determined. The second matrix in the embodiments of the present disclosure may be obtained by inner products of function transformations of the plurality of sub-matrixes.

In some possible implementations, the gate control parameter {tilde over (g)} of each convolution layer may be learnt when performing training by means of the neural network. Alternatively, the received configuration information may include the gate control parameter configured for each convolution layer, so that the transformation matrix corresponding to each convolution layer may be determined by means of the mode above, and the number of parameters of the second matrix Ũ is also reduced to merely i parameters from

$\sum_{i} C_{i}^{2} .$

Alternatively, the received configuration information may also merely include the gate control parameter {tilde over (g)} corresponding to each convolution layer, and the sub-matrixes and the second matrix may be further determined by means of the mode above.

The specific steps of training the neural network are described for examples of implementing the information processing method in the embodiments of the present disclosure by means of the neural network below. FIG. 10 shows a flow chart of training a neural network according to the embodiments of the present disclosure. The step of training the neural network includes the following steps.

At S41, a training sample and a real detection result for monitoring are acquired.

In some possible implementations, the training sample may be sample data of the foregoing type of the input information, such as at least one of text information, image information, video information, or voice information. The real detection result for monitoring is a real result of a training sample to be predicted, such as an object type in an image and a position of a corresponding object, which is not specifically defined in the present disclosure.

At S42, processing is performed on the training sample by using the neural network to obtain a prediction result.

In some possible implementations, sample data in the training sample may be input to the neural network, and a corresponding prediction result is obtained by means of the operation of each network layer in the neural network. The convolution processing of the neural network may be executed based on the information processing mode, i.e., updating the convolution kernel of the network layer by using a pre-configured transformation matrix, and convolution operation is executed by using a new convolution kernel. A processing result obtained by the neural network is a prediction result.

At S43, a network parameter of the neural network is fed back and adjusted based on a loss corresponding to the prediction result and the real detection result, until a termination condition is satisfied, where the network parameter includes the convolution kernel of each network layer and the transformation matrix (including the continuous values in the gate control parameter).

In some possible implementations, a loss value corresponding to the prediction result and the real detection result may be obtained by using a preset loss function. If the loss value is greater than a loss threshold, the network parameter of the neural network is fed back and adjusted, and the prediction result corresponding to the sample data is re-predicted by using the neural network having the adjusted parameter, until the loss corresponding to the prediction result is less than the loss threshold, i.e., it indicates that the neural network satisfies the precision requirements, and training may be terminated in this case. The preset loss function may be a subtraction operation between the prediction result and the real detection result, i.e., the loss value is a difference between the prediction result and the real detection result. In other embodiments, the preset loss function may also be other forms, which is not specifically defined in the present disclosure.

The training of the neural network may be completed by means of the mode above, and the transformation matrix configured for each convolution layer of the neural network may be obtained, so that the meta convolution operation of each convolution layer may be completed.

In summary, in embodiments of the present disclosure, the input information may be input to the neural network to execute corresponding operation processing, where when convolution processing of the convolution layer of the neural network is executed, the convolution kernel of the convolution layer may be updated based on the transformation matrix determined for each convolution layer, and corresponding convolution processing is completed by using a new convolution kernel. By means of the mode, a corresponding transformation matrix may be individually configured for each convolution layer, a corresponding group effect is formed, where the group is not limited to a group of adjacent channels, and the operation precision of the neural network may be further improved.

In addition, compared with the defects of the previous technologies in artificially setting the group num for a specific task, the technical solutions of the embodiments of the present disclosure may implement independent learning of any group convolution scheme for the deep neural network convolution layer without human intervention. Furthermore, the embodiments of the present disclosure may not only express the existing adjacent group convolution technologies, but also expand any channel group scheme, the relevance of feature information of different channels is increased, and the cutting-edge development of the convolution redundancy elimination technology is promoted. The meta convolution method provided in the present disclosure is applied to any convolution layer of the deep neural network, so that the convolution layers having different depths of the network can all independently select the channel group scheme adapted to the current feature expression by means of learning. Compared with the traditional strategy that the whole network uses single type group convolution, the optimal performance model can be obtained. In addition, in the present disclosure, the network parameter is decomposed by using Kronecker operation, and the meta convolution method provided in the embodiments of the present disclosure has the advantages such as small computation, small number of parameters, and easy implementation and application by means of the differentiable end-to-end training mode.

It can be understood by a person skilled in the art that, in the foregoing methods of the specific implementations, the order in which the steps are written does not imply a strict execution order which constitutes any limitation to the implementation process, and the specific order of executing the steps should be determined by functions and possible internal logics thereof.

FIG. 11 shows a block diagram of an image processing apparatus according to embodiments of the present disclosure. As shown in FIG. 11, the image processing apparatus includes:

an input module 10 configured to input received input information to a neural network;

an information processing module 20 configured to process the input information by means of the neural network, where in the case that convolution processing is executed by means of a convolution layer of the neural network, a convolution kernel of the convolution layer is updated by using a transformation matrix configured for the convolution layer, so that the convolution processing of the convolution layer is completed by means of the updated convolution kernel; and

an output module 30 configured to output a processing result of the processing of the neural network.

In some possible implementations, the information processing module is further configured to acquire a space dimension of the convolution kernel of the convolution layer;