Patent Application
20220164666
The disclosed technology relates to designing and implementing artificial neural networks as well as their practical applications.
Computer vision has become ubiquitous in society, with applications in search, image recognition, mobile apps, mapping, medicine, drones, and self-driving cars. Core to many of these applications are visual recognition tasks such as image classification, localization, and detection. Recent developments in artificial neural networks (ANN) have greatly advanced the performance of visual recognition systems. Deep learning is part of a broader family of machine learning methods that are based on ANNs. Deep learning architectures such as deep neural networks, deep belief networks, recurrent neural networks and convolutional neural networks (CNNs) have been applied to fields including computer vision. The practical applications for ANNs are far and wide, encompassing finance, personal communication, industry, education, and so on.
Typically, a computing device that runs an application using an ANN needs to be connected to the Internet to have access to a deep neural network. For example, a mobile phone collects data locally but the processing for a deep neural network is performed on remote servers. Thus, deep neural networks are not implemented in off-the-shelf devices for use without consistent Internet access. Accordingly, the availability of on-device applications with artificial intelligence remains limited.
An efficient mixed-precision search (EMS) technique enables automatic and proxy-less mixed-precision network design on large-scale datasets. Unlike existing techniques that execute all bit-width options to search for an optimized bit-width, the EMS technique uses a folding quantizer module that only executes a sampled bit-width option together with a full-precision pass. First, by executing a sampled bit-width option, the EMS technique reduces computational cost in terms of throughput and memory usage during network training. Second, by executing a full-precision pass, network parameters in the unsampled bit-width options are updated in every training iteration, and thus the EMS technique achieves a reasonable rate of convergence. Moreover, the EMS technique selects a mixed-precision network with quantization errors most effective for loss minimization. For example, on CIFAR-10 and ImageNet datasets, the EMS technique outperforms both 8-bit and 4-bit uniform quantization results as well as other mixed-precision search techniques.
Embodiments of the present technology will be described and explained through the use of the accompanying drawings.
Various features of the technologies described herein will become more apparent to those skilled in the art from a study of the Detailed Description in conjunction with the drawings. Embodiments are illustrated by way of example and not limitation in the drawings, in which like references may indicate similar elements. While the drawings depict various embodiments for the purpose of illustration, those skilled in the art will recognize that alternative embodiments may be employed without departing from the principles of the technologies. Accordingly, while specific embodiments are shown in the drawings, the technology is amenable to various modifications.
Introduced here is an efficient mixed-precision search (EMS) technique to automate the design for an optimal artificial neural network (ANN). An ANN is a widely used model in the field of machine learning. Network architecture search (NAS) is a common technique for automating the design of an ANN. NAS has been used to design networks that are on par or outperform hand-designed architectures. Efforts to design efficient ANNs generally fall into three groups: design efficient operators, identify efficient network architectures, and convert full-precision networks to use low bit-widths. A “bit-width” refers to a number of bits used to represent a number.
The disclosed EMS technique automates a search for “optimal” mixed-precision parameters that offer improved accuracy and reduced computational costs compared to existing mixed-precision and single-precision search methods. A single-precision ANN uses the same size bit-width for all layers whereas a mixed-precision ANN uses different bit-widths for different layers. Mixed-precision search methods have been previously explored to find suitable bit-widths for allocation on network layers and achieve a mixed-precision network with lossless accuracy and higher computational efficiency. However, existing mixed-precision methods suffer from slow convergence or high training costs, especially with a large-scale training dataset. In contrast, an ANN with “optimal” mixed-precision bit-widths offers a best or favorable balance between accuracy and performance.
There are numerous benefits to using low-precision bit-width search methods, which are also referred to here as “reduced-precision” because the precision is less than a full-precision counterpart. First, reduced-precision formats require less memory, thereby enabling training and deployment of larger neural networks. Second, reduced-precision formats require less memory bandwidth, thereby increasing data transfer operations. Third, math operations run faster in reduced-precision formats compared to full-precision formats. Mixed-precision training achieves all these benefits while ensuring that task-specific accuracy is not lost compared to full-precision training, or there is only a negligible or acceptable amount of loss.
More breakthrough technologies are enabled by improving computational efficiencies while maintaining task-specific accuracies. For example, ANNs that implement deep learning have led to breakthroughs in several areas, including image processing and understanding, language modeling, language translation, speech processing, game playing, and many others. ANN complexity has increased to achieve these breakthroughs, which in turn has increased the computational resources required to train networks. One way to lower the required computational resources is to use lower-precision arithmetic with mixed-precision operations.
In a particular example, computer vision is a field of artificial intelligence that trains computers to interpret and understand the visual world. Computer vision tasks include methods for acquiring, processing, analyzing and “understanding” digital images, and extraction of high-dimensional data from the real world in order to produce numerical or symbolic information, e.g., in the forms of decisions. The “understanding” refers to the transformation of visual images (the input) into descriptions of the world. A convolutional neural network (CNN) is an example of an ANN that is commonly used for object detection and classification, with millions of images and hundreds of object classes.
However, a computing device that runs a computer vision application using a CNN needs to be connected to the Internet to have access to a deep neural network. For example, a smartphone that collects visual data typically processes the collected data on a deep neural network on remote servers. Thus, the availability of on-device applications with artificial intelligence (AI) is limited due to computational costs and power consumption. A trade-off to reduce computational costs and power consumption includes reducing task-specific accuracies of an on-device application, which is undesirable.
The EMS technique enables the design of efficient neural networks for mobile and internet-of-things (IoT) applications by addressing challenges such as computational cost, memory usage, battery usage, and model size. Implementing the EMS technique to run deep learning models in mobile environments will make Al technology readily accessible to the billions of smartphone users across the globe today. Thus, overall, the EMS technique makes a significantly positive impact to on-device Al development. Moreover, by removing cloud dependencies, the EMS technique allows for direct interaction between end-users and applications, and better protects user privacy.
Thus, the users of the smart devices can benefit from using on-device technologies that are computationally light-weighted and energy efficient because the EMS technique enables the design of efficient neural networks that can run on mobile environments. For example, assuming that implementing the EMS technique saves a small amount of energy per smartphone (per day), an enormous amount of energy is saved in aggregate when considering the billions of smartphones across the globe today.
Examples of algorithms in NAS include reinforcement learning-based NAS (RL-NAS), differentiable NAS (DNAS), and single-path-one-shot (SPOS). The RL-NAS is an end-to-end framework that considers a NAS as a Markov decision process (MDP) and can discover a model with high accuracy. However, training cost is prohibitively high due to its nature of delay rewards, which hinders its applications on a large-scale dataset. For example, RL-NAS requires thousands of GPU-days of searching/training to achieve state-of-the-art computer vision.
The DNAS searches for an optimal neural network architecture by firstly creating a super-network with microstructures parameterized by network and architecture parameters. An optimal neural network architecture is determined by sampling from the category distribution indicated by the architecture parameters. A deterministic attention (DA) mechanism can incorporate with architecture parameters to learn connections between data nodes by back-propagation. The DNAS methods are less complex and more stable than RL-NAS for training but consume a larger amount of computing resources and have a larger memory footprint.
The SPOS methods have better training efficiency than DNAS. Like DNAS, the SPOS methods firstly construct a super network with various candidate operations. Instead of executing all operators in one iteration, the super network in an SPOS framework is trained by sampling and optimizing a single path. The sampling can be uniform or follow specific strategies. A “super-network” based NAS provides a more computationally-efficient solution. The essential idea is to train one super-network that spans many options for the final design rather than generating and training thousands of networks independently. In addition to the learned parameters, a set of architecture parameters learn to prefer one module over another. These algorithms are differentiable, allowing the use of gradient descent for optimization.
A mixed-precision NAS is closely related to DNAS and SPOS methods. Under either DNAS or SPOS, the mixed-precision NAS is realized by implementing a candidate operator with paralleled candidate quantizers. With large amounts of quantizers inserted into a base model, however, the DNAS-based mixed-precision methods run slow due to high training complexity. Although SPOS is faster than DNAS, optimization converges more slowly due to an asynchronous parameter update and highly-varying gradient during training. Therefore, considering resource consumption and training time, both DNAS and SPOS are undesirable when training with a large-scale dataset.
Neural network quantization is a tool used to satisfy performance requirements while conforming with memory and battery usage limits. Quantization is a process of approximating an ANN that uses floating-point numbers with an ANN of low bit-width numbers. Quantization dramatically reduces the memory requirement and computational cost of using ANNs. For example, weights and activations of an ANN can be represented using 8-bit integers (or INT8) rather than 32-bits, without incurring significant loss in accuracy.
Neural network quantization can take a trained full-precision network and quantize its weights into a pre-set bit-width, which reduces runtime computational cost while sacrificing accuracy in, for example, image classification of a computer vision system. To avoid a significant drop in accuracy, neural networks include quantization modules and jointly train weights and quantization parameters (“quantizers”). By learning quantizers, a NAS can successfully produce a uniform 4-bit residual neural network (ResNet) that nearly matches the accuracy of a full-precision counterpart in image classification. However, a drop in accuracy that is caused by quantization differs across various network architectures, and retaining accuracy is more challenging when quantizing compact networks such as mobile architectures (e.g., MobileNetV2, MobileNetV3). As a result, uniform low-bit networks of a compact architecture that implement quantization approaches have a deficiency in accuracy relative to their full-precision counterparts.
Quantization generally approximates 32-bit full-precision weights and activations with binary representation. The computation complexity is reduced by replacing floating-point operations with bit-wise operations. Although these methods save memory and computation, they result in a dramatic degradation in accuracy. Some techniques approximate an ANN with more bits. In one example, both ternary values and scaled gradients are learned to approximate 32-bit weights. An example includes DoReFa-Net quantized activation, where weights and gradients use different bit-widths. In another example, a technique uses an activation quantizer with a learnable clipping parameter. In another example, a quantizer adaptively adjusts its quantization levels during training. The quantization error is obtained by minimizing the mean square difference between full-precision data and quantized data.
In yet another example, quantization intervals of a quantizer are learned. The input data is firstly passed to a non-linear transform with learnable parameters and clipped into the range of [0, 1] before feeding into a uniform quantizer. These learnable quantizers Consistently retain the accuracy of a full-precision network with 4-bit precision on neural network architectures. Uniform quantization results on compact neural networks such as MobileNetV1, V2, V3 are rarely reported.
Uniform quantization requires pre-set and fixed numerical precision during training of the ANN. However, as different layers exhibit different sensitivity to quantization error, a uniform numerical precision is sub-optimal in most cases. To address this problem, an ANN can provide for optimal bit-width allocation for particular layers of the ANN. That is, the quantization is not fixed and a suitable low-precision format is selected during training.
A mixed-precision network provides improved accuracy compared to a single-precision (SP) network while reducing computational costs. Mixed-precision training offers significant computational savings by, for example, performing operations in half-precision format while storing minimal information to retain as much information as possible in critical parts of the network. However, manual design of a mixed-precision network is impractical due to enormous design choices.
A suitable mixed-precision design can be identified by using NAS. In one example, a super network is constructed by replacing each full-precision operator with a set of mixed-precision operators and optimizing for the bit-width used in each operator for weights and activations. However, implementing this solution is not trivial because, for example, quantization functions introduce computational cost. For example, existing quantization functions that offer improved accuracy such as LQ-Net and QIL cost significantly more than a linear function that scales the input into a fixed range. Therefore, searching for an optimal mixed-precision network takes a long time, especially when the training dataset is large as with ImageNet.
In another example, a hardware-aware automated quantization (HAQ) framework is used for mixed-precision network design. The HAQ adopts RL to search a quantization policy with direct hardware feedback, which performs better in latency, energy and model size compared with rule-based approaches. However, HAQ trains slowly and the exploration step includes a fine-tuning that is time-consuming and requires class sub-sampling to reduce complexity. In yet another example, mixed-precision training is based on DNAS to construct a deterministic or stochastic super-network parameterized by network parameters and architecture parameters (e.g., probabilities). Recent mixed-precision search techniques are motivated by DNAS and developed based on a DA mechanism. Optimal bit-width allocation is resolved by training the super-network under complexity-constrained classification loss and then sampling a sub-network with highest architecture probabilities.
In each training iteration, a forward function proceeds as a weighted sum of a selected quantizer output and an identity transform, which is a transformation that copies source data to a destination without making any change to the source data. The design reduces computational cost compared to unfolding quantizers, which execute all candidate quantizers concurrently. The quantizer selection is controlled by a learned probability distribution tied to architecture parameters. An implementation uses probability-based sampling, where a quantizer with a higher probability is selected for a given training iteration. However, although the quantizer with a higher probably has a greater chance of being sampled, the higher probability is not determinative of being selected. During training, a candidate quantizer with its quantization error direction aligned with the gradient decent direction gets a higher probability. By introducing the FQ module including an identity connection, the gradient descend becomes more stable compared with SP-based approaches and thus maintains a fast convergence.
Implementations on CIFAR-10 and ImageNet datasets, which are each a collection of images that are commonly used to train machine learning and computer vision algorithms, show that the EMS technique achieves 72% and 42% complexity reduction on ResNet-18 and MobileNet-v2 models, respectively, compared with 8-bit uniform quantization results. Compared to other mixed-precision networks, the EMS technique reduces training time by 3X and memory usage by 1.5×n, with better accuracy and computation efficiency.
y
j
l
=p
j
l
·{circumflex over (x)}
j
l+(1−pjl)·xl (1)
where {circumflex over (x)}jl is the low-precision data and pjl is defined as a sampling probability of j-th candidate quantizer. The output yjl can be regarded as the weighted sum of all paralleled edges with only one quantizer activated and the others replaced by skip connections. Moreover, the output yjl can be reformulated by:
y
j
l
=x
l
+p
j
l
·Δx
j
l (2)
where Δxjl={circumflex over (x)}jl−xl represents the quantization error of j-th candidate quantizers. Therefore, the super network is constructed by injecting a random quantization noise to the input of every operator. The quantizer distribution Pl also controls gain of the injected quantization noise. As will be shown later, Pl potentially reveals the importance of the candidate quantizers for minimizing the loss function. Therefore, the optimal bit-width for l-th operator is determined by Pl, and the problem of optimal bit-width allocation is cast to learning a super network parameterized by P:={P1, . . . , PL}.
Given a super network N(D,O,E|P), the optimal bit-width allocation is regarded as learning the architecture parameters P by minimizing a classification loss under complexity constraints, which is formulated as:
where Lcls(x|θ, P) and Lcpx(P) represent the classification loss and complexity loss. β is the Lagrange multiplier. The classification loss is defined as the cross-entropy loss between ground-truth labels and predicted class distribution. The complexity loss is computed by:
where C(oi) is denoted as the bitwise operations (BitOps) of an operator illustrated in Eq. 7.
In Eq. 7, bw
The following describes the statistical behavior of architecture parameter Pl of every FQ. Here, the statistic behavior of Pl is characterized by its expected gradient, which can be represented by:
As shown in Eq. 8, the expected gradient with respect to architecture parameters Pl is related to the inner product between quantization error of i-th candidate quantizer Δxil and corresponding gradient respect to quantizer output
Those quantizers with quantization errors most homodromous to their output gradient among the candidates are encouraged during the training. As shown, the architecture parameters P of the super network is evolving toward a distribution that is beneficial to minimizing the objective L(x|θ, P).
The FQ modules can use shared or independent weights.
The following examples include settings and implementation details, followed by implementation results that demonstrate effectiveness and efficiency of network training, and the network accuracy and complexity of the EMS technique.
The EMS technique is compatible with various quantization approaches. In one example, there are four quantizers with bit-widths of {1,2,4,8} in an FQ. The candidate quantizers are implemented using the LQ-Net approach, which is one method in neural network quantization. In some layers, weights and activations are quantized using different bits, but the same bit-width is commonly used in weight and activation quantization in a common layer to improve operability for software and hardware design. This technique uses the same Pl to control FQs of the same downstream operator.
The EMS technique improves over baseline methods (e.g., DA, SP, LQ-Net). Regarding DA, an HWGQ-Net is re-implemented and replaced with the LQ-Net quantization approach for a fair comparison. Regarding SP, the j-th quantizer qj is activated by probability sampling according to pj. The gradient with respect to pj is computed by the straight-through estimator (STE) approach
The LQ-Net is used as a baseline of uniform quantization and compared with both the 4-bit and 8-bit LQ-Net quantization results.
The performance of the EMS technique can be evaluated by using two base networks, ResNet18 and MobileNetV2 on the CIFAR-10 and the ImageNet datasets. ResNet18 is commonly used in quantization, while MobileNetV2 as MobileNetV2 adopt a light-weighted network architecture that is commonly used in mobile applications due to their efficiencies. Quantization on MobileNetV2 is challenging and rarely used in quantization because of limited computation resources on mobile devices.
In this example, for CIFAR-10, the input images have a size of 32×32. The four borders of an image are padded with four zero-pixels and an image is randomly cropped into an image with size of 32×32. The image is then horizontally flipped randomly. Finally, the augmented image is normalized with a channel mean of (0.491, 0.482, 0.447) and standard deviation of (0.247, 0.243, 0.262).
For ImageNet, a raw image can have any size. In an example, raw images are resized into images with shortest edge length of 224 and a center image is cropped using a window with size of 224×224. The brightness, contrast, saturation and lighting are randomly adjusted. For example, for an input image x, the brightness is adjusted using the equation shown below:
x=a·x (9)
where a is a scaling factor randomly chosen from [0.6, 1.4]. The contrast of the input image is adjusted by:
x=x·r+(1−r)·
where
Evaluation metrics include accuracy, average bit-widths (Bits), computed BitOps, and GPU hours. The average bit-width is defined as the square root of BitOps normalized by the BitOps of a 1-bit quantized network. The GPU hours is computed as the product of training time and number of GPUs.
In one example, for the CIFAR-10 image set, a super net is trained for 1200 epochs with a batch size of 400. A standard stochastic gradient descent (SGD) optimizer is used for optimization with an initial learning rate of 0.1, momentum of 0.9, and weight decay of 4e-5. The learning rate is reduced to 0.01 after 58800 iterations. For ImageNet, a model is trained for 100 epochs with a batch size of 256. The SGD optimizer is adopted for network optimization with the same settings as those used in CIFAR-10 except that the learning rate decays every 150K iterations by a factor of 10. After network training, the quantizer with the highest pl of every FQ is selected and the resulted network is fine-tuned. The network weights from the training are used to initiate the network in the fine-tuning stage.
The effectiveness and efficiency of network training is evaluated by comparing the proposed approach and the DA and SP on the rate of convergence, throughput and memory usage. In the example, the MobileNetV2 is used as the base model.
In the example implementations, the FQ modules use shared or independent weights. The impact of using the shared weight option versus using the independent weight option is evaluated. A comparison is performed of the SP, DA and EMS approach with the shared weight option and the independent weight option (e.g., SP-Shared, SP-Independent, DA-Shared, DA-Independent, EMS-Shared, and EMS-Independent).
As shown, the SP-Shared converges slower than the DA-Shared and EMS-Shared. The EMS-Shared converges faster than DA-Shared in the initial iterations. In the later iterations, curves of the EMS-Shared and the DA-Shared are overlapping and converge to a similar validation error. This indicates that the EMS approach has a similar rate of convergence with DA and significantly outperforms SP in terms of rate of convergence. The rate of convergence depends on parameter update mechanism in quantizer candidates. In DA, parameters in all quantizer candidates get updated in each training iteration. In the EMS approach, parameters in all quantizer candidates get updated in each training iteration regardless of which candidate is sampled. In SP, the parameters in a quantizer candidate only gets updated when that candidate is sampled.
In terms of training throughput and memory usage, these metrics are evaluated on an Nvidia RTX 2080 Ti™ GPU. The training throughput is defined as the number of iterations per second and the memory usage is observed by GPU monitor Nvidia System Management Interface.
The EMS technique improves over the DA mixed-precision search approach and the uniform quantization approach LQ-Net in terms of accuracy, inference complexity (BitOps) and training time cost (GPU hours). Table I shows results that compare these different approaches.
The EMS approach achieves higher accuracy than the DA approach with lower inference complexity and shorter training time. With a similar accuracy, the mixed-precision ResNet-18 produced by the EMS technique has lower complexity than that produced by the DA approach. The mixed-precision MobileNetV2 produced by the EMS technique achieves higher accuracy (65.1%) and lower complexity than that by the DA approach. In terms of model size, 37% complexity reduction results in a model with 25% smaller than the DA one. In terms of training time, the EMS technique is 3× and 1.75× faster than the DA approach on the CIFAR-10 and the ImageNet datasets. The reason why the acceleration ratio is higher on the Cifar10 than the ImageNet is that the GPU machine suffers from a cooling problem when multiple GPU cards are used. The cooling problem drives the machine's temperature high, which slows down the computation. The training on CIFAR-10 are conducted with a single GPU, which does not cause the overheating. Therefore, the actual speedup on the ImageNet using the EMS technique is higher than that reported.
Compared with uniform quantization, LQ-Net, on the CIFAR-10, the EMS technique produces mixed-precision models based on ResNet-18 and MobileNetV2 with a better accuracy and much higher efficiency compared with the 8-bit LQ-Net. The ResNet-18 and MobileNetV2 models produced by the EMS approach are 4.6X and 1.7X more efficient than the 8-bit networks in terms of inference complexity, respectively. Regarding ImageNet, the EMS technique produces an accurate model (labeled with t) with accuracy of 65.5 and complexity of 55.80 GBitOps. Compared with the 8-bit LQ-Net, the technique achieves a better accuracy and is 2× more efficient. Meanwhile, the EMS technique (labeled with t) has a similar complexity (27.75 GBitOps) as the 4-bit LQ-Net network (27.13 GBitOps) and is 0.9% more accurate.
The results in Table I show that the accuracy and complexity of MobileNetV2 are more sensitive to quantization. In the example, the accuracy of a quantized MobileNetV2 is 0.7%-1.3% worse than that of a quantized ResNet-18 on both datasets. The EMS technique effectively improves the performance of quantized MobileNetV2 together with a considerable efficiency boost. The training process of a quantized network will more easily get trapped at a poor local-minima than a full-precision network due to the noisy gradient approximation. In the EMS approach, the gradient noise is suppressed by maintaining a full-precision path during training, which prevents being trapped at a sub-optimal local minima.
The EMS technique is profiled for accuracy and complexity, compared with the DA.
Thus, the EMS technique can efficiently search for a suitable mixed-precision network and outperform other mixed-precision search algorithms in terms of both rate of convergence and throughput during network training. This property makes the EMS approach more feasible when training with a large-scale dataset. Moreover, the EMS approach outperforms other mixed-precision methods and the uniform quantization approach, in terms of network accuracy and complexity.
A goal of searching for an optimized mixed-precision neural network is to improve performance of a network while reducing memory and battery consumption. The EMS approach improves over existing mixed-precision technologies in those areas, which improves on-device Al development that would benefit mobile phone users. This becomes impractical with existing approaches that require a long time for network training, especially with a limited number of GPUs (e.g., on mobile devices). Taking ResNet-18 architecture as an example, it takes 140 GPU hours to achieve acceptable results using a NVIDIA RTX-2080 Ti card, which is one of the latest mixed-precision search approaches. As such, if a device only has a single GPU card, the same operation will take almost 6 days to complete on the ImageNet-scale dataset. In contrast, implementing the EMS technique under the same condition, the ResNet-18 experiment only cost about 3 days to achieve a similar accuracy using a significantly lower BitOps (e.g., 17% lower power consumption, 5% smaller model size and potentially a much faster inference), which reduces the experimental cycle by half.
At 602, a data node of a layer of the ANN receives data (e.g., activation input, a weight). A layer of the ANN has a network architecture including a quantization edge and a parallel full-precision edge, which are each between the data node and an operator node (see, e.g.,
At 604, the ANN is trained iteratively to search for optimal quantizers in a mixed-precision ANN. At 604(a) a candidate quantizer of the folding quantizer is selected for sampling. The sampled quantizers are selected using one or more techniques including probabilistically, randomly, pseudo-randomly, deterministically, algorithmically, etc. In one example, a candidate quantizer is selected for sampling based on a probability distribution for the candidate quantizers. Generally, a candidate quantizer is selected for sampling to reduce (e.g., minimize) a classification loss, which is bounded by a complexity constraint for the ANN.
At 604(b), the received data is quantized based on the sampled quantizer. The quantized data has a low-precision format (a low bit-width) compared to the full-precision format of the original received data. In an implementation, a number of candidate quantizers available for sampling in each next iteration is reduced, which allows convergence to an optimal quantizer (which is then fixed for the layer).
At 604(c), a weighted sum is computed of the quantized data and the original (unquantized) data. That is, the quantized data and the original data are each weighted and summed. In one example, all the candidate quantizers use common weights (have the same weight values). In another example, the candidate quantizers use independent weights, that are different from each other and/or independently adjustable.
At 604(d), an output of the operator node is computed based on the weighted sum. In one example, the weighted sums of each layer are processed with respective operators. In an example where the data corresponds to activation inputs, each layer can include weights that are similarly processed to obtain quantized weights that are used to generate weighted sums of the weights. The weighted sums of the weights and activation inputs are processed as linear combinations for the operators of the respective layers.
At 604(e), the network parameters of the layers are updated for the next iteration based on outputs computed in the current iteration. Examples of network parameters include the weights for candidate quantizers. For example, the iterative training includes computing a classification loss, computing a gradient loss of the network parameters, and updating the network parameters to reduce the classification loss. In one example, the network parameters of unsampled candidate quantizers are updated in every iteration. In another example, the network parameters of all the candidate quantizers are updated in each iteration regardless of which candidate quantizer is sampled.
At 606, the folding quantizer of the layer converges to an optimal quantizer representing a low-precision bit-width. The folding quantizers of different layers converge to different low-precision bit-widths (hence, a multi-precision network results). In one example, the received data includes an activation input for a layer, which uses another folding quantizer to similarly quantize a weight for the operator node. The respective folding quantizers converge to obtain optimal quantizers for the activation input and the weight.
In some embodiments, the platform 702 may reside in a network environment 700. Thus, the platform 702 may be connected to one or more networks 706a-b. The network(s) 706a-b can include personal area networks (PANs), local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), cellular networks, the Internet, etc. Additionally or alternatively, the editing platform 1102 can be communicatively coupled to computing device(s) over a short-range communication protocol, such as Bluetooth® or near-field communication (NFC).
The interface 704 is accessible via a web browser, desktop application, mobile application, and/or over-the-top (OTT) application. Accordingly, the interface 704 may be viewed on a personal computer, tablet computer, mobile phone, game console, music player, wearable electronic device (e.g., a watch or a fitness accessory), network-connected (“smart”) electronic device, (e.g., a television or a home assistant device), virtual/augmented reality system (e.g., a head-mounted display), or some other electronic device.
Some features of the platform 702 can be hosted locally. That is, the search platform 702 may reside on the computing device used to access the interface 704. For example, the platform 702 may be embodied as a mobile application executing on a mobile computer. Other embodiments of the platform 702 are executed by a cloud computing service operated by Amazon Web Services® (AWS), Google Cloud Platform™, Microsoft Azure®, or a similar technology. In such embodiments, the platform 702 may reside on a host computer server that is communicatively coupled to one or more content computer servers 708. The content computer server(s) 708 can include different types of data (e.g., images, query logs), user information (e.g., profiles and credentials), and other assets. Such information could also be stored on the host computer server.
Certain embodiments are described in the context of network-accessible interfaces. However, those skilled in the art will recognize that the interfaces need not necessarily be accessible via a network. For example, a computing device may be configured to execute a self-contained computer program that does not require network access. Instead, the self-contained computer program may cause necessary assets (e.g., graphics, matching algorithms, and processing operations) to be downloaded at a single point in time or on a periodic basis (e.g., weekly, daily, or hourly).
The computing system 800 may include one or more central processing units (also referred to as “processors”) 802, main memory 806, non-volatile memory 810, network adapter 812 (e.g., network interface), video display 818, input/output devices 820, control device 822 (e.g., keyboard and pointing devices), drive unit 824 including a non-transitory computer-readable storage medium 826, and signal generation device 830 that are communicatively connected to a bus 816. The bus 816 is illustrated as an abstraction that represents one or more physical buses and/or point-to-point connections that are connected by appropriate bridges, adapters, or controllers. The bus 816, therefore, can include a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), IIC (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 1194 bus (also referred to as “Firewire”).
The computing system 800 may share a similar computer processor architecture as that of a personal computer, tablet computer, mobile phone, game console, music player, wearable electronic device (e.g., a watch or fitness tracker), network-connected (“smart”) device (e.g., a television or home assistant device), virtual/augmented reality systems (e.g., a head-mounted display), or another electronic device capable of executing a set of instructions (sequential or otherwise) that specify action(s) to be taken by the computing system 800.
While the main memory 806, non-volatile memory 810, and storage medium 826 (also called a “machine-readable medium”) are shown to be a single medium, the term “machine-readable medium” and “storage medium” should be taken to include a single medium or multiple media (e.g., a centralized/distributed database and/or associated caches and servers) that store one or more sets of instructions 828. The term “machine-readable medium” and “storage medium” shall also be taken to include any medium that is capable of storing, encoding, or carrying a set of instructions for execution by the computing system 800.
In general, the routines executed to implement the embodiments of the disclosure may be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions (collectively referred to as “computer programs”). The computer programs typically comprise one or more instructions (e.g., instructions 804, 808, 828) set at various times in various memory and storage devices in a computing device. When read and executed by the one or more processors 802, the instruction(s) cause the computing system 800 to perform operations to execute elements involving the various aspects of the disclosure.
Moreover, while embodiments have been described in the context of fully functioning computing devices, those skilled in the art will appreciate that the various embodiments are capable of being distributed as a program product in a variety of forms. The disclosure applies regardless of the particular type of machine or computer-readable media used to actually effect the distribution.
Further examples of machine-readable storage media, machine-readable media, or computer-readable media include recordable-type media such as volatile and non-volatile memory devices 810, floppy and other removable disks, hard disk drives, optical disks (e.g., Compact Disk Read-Only Memory (CD-ROMS), Digital Versatile Disks (DVDs)), and transmission-type media such as digital and analog communication links.
The network adapter 812 enables the computing system 800 to mediate data in a network 814 with an entity that is external to the computing system 800 through any communication protocol supported by the computing system 800 and the external entity. The network adapter 812 can include a network adaptor card, a wireless network interface card, a router, an access point, a wireless router, a switch, a multilayer switch, a protocol converter, a gateway, a bridge, bridge router, a hub, a digital media receiver, and/or a repeater.
The network adapter 812 may include a firewall that governs and/or manages permission to access/proxy data in a computer network and tracks varying levels of trust between different machines and/or applications. The firewall can be any number of modules having any combination of hardware and/or software components able to enforce a predetermined set of access rights between a particular set of machines and applications, machines and machines, and/or applications and applications (e.g., to regulate the flow of traffic and resource sharing between these entities). The firewall may additionally manage and/or have access to an access control list that details permissions including the access and operation rights of an object by an individual, a machine, and/or an application, and the circumstances under which the permission rights stand.
The techniques introduced here can be implemented by programmable circuitry (e.g., one or more microprocessors), software and/or firmware, special-purpose hardwired (i.e., non-programmable) circuitry, or a combination of such forms. Special-purpose circuitry can be in the form of one or more application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), etc.
The foregoing description of various embodiments of the claimed subject matter has been provided for the purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise forms disclosed. Many modifications and variations will be apparent to one skilled in the art. Embodiments were chosen and described in order to best describe the principles of the invention and its practical applications, thereby enabling those skilled in the relevant art to understand the claimed subject matter, the various embodiments, and the various modifications that are suited to the particular uses contemplated.
Although the Detailed Description describes certain embodiments and the best mode contemplated, the technology can be practiced in many ways no matter how detailed the Detailed Description appears. Embodiments may vary considerably in their implementation details, while still being encompassed by the specification. Particular terminology used when describing certain features or aspects of various embodiments should not be taken to imply that the terminology is being redefined herein to be restricted to any specific characteristics, features, or aspects of the technology with which that terminology is associated. In general, the terms used in the following claims should not be construed to limit the technology to the specific embodiments disclosed in the specification, unless those terms are explicitly defined herein. Accordingly, the actual scope of the technology encompasses not only the disclosed embodiments, but also all equivalent ways of practicing or implementing the embodiments.
The language used in the specification has been principally selected for readability and instructional purposes. It may not have been selected to delineate or circumscribe the subject matter. It is therefore intended that the scope of the technology be limited not by this Detailed Description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of various embodiments is intended to be illustrative, but not limiting, of the scope of the technology as set forth in the following claims.
