Patent Application
20240036824
A Block Floating-Point (BFP) number system represents a block of floating-point (FP) numbers by a shared exponent (typically the largest exponent in the block) and right-shifted significands of the block of FP numbers. Computations using BFP can provide improved accuracy compared to integer arithmetic and use fewer computing resources than full floating. However, the range of numbers that can be represented using a BFP format is limited, since small numbers are replaced by zero when the significand are right-shifted too far.
In some applications, such as computational neural networks, input data may have a very large range. The use of BFP in such applications can lead to inaccurate results.
The accompanying drawings provide visual representations which will be used to more fully describe various representative embodiments and can be used by those skilled in the art to better understand the representative embodiments disclosed and their inherent advantages. In these drawings, like reference numerals identify corresponding or analogous elements.
The various apparatus and devices described herein provide mechanisms for data processing using and enhanced block floating point data format.
While this present disclosure is susceptible of embodiment in many different forms, there is shown in the drawings and will herein be described in detail specific embodiments, with the understanding that the embodiments shown and described herein should be considered as providing examples of the principles of the present disclosure and are not intended to limit the present disclosure to the specific embodiments shown and described. In the description below, like reference numerals are used to describe the same, similar or corresponding parts in the several views of the drawings. For simplicity and clarity of illustration, reference numerals may be repeated among the figures to indicate corresponding or analogous elements.
A number may be represented as (−1)s×m×be, where s is a sign value, m is a significand, e is an exponent and b is a base. In some binary (b=2) floating-point representations, such as the 32-bit IEEE format, the significand is either zero or in the range 1≤m<2. For non-zero values of m, the value m−1 is referred to as the fractional part of the significand. The 32-bit IEEE format stores the exponent as an 8-bit value and the significand as a 23-bit value.
A Block Floating-Point (BFP) number system represents a block of floating-point (FP) numbers by a shared exponent (typically the largest exponent in the Block) and right-shifted significands of the block of FP numbers. The present disclosure improves upon BFP by representing small FP numbers (that would ordinarily be set to zero) by the difference between the exponent and the shared exponent. A tag bit indicates whether the EBFP number represents a shifted significand or the exponent difference.
Some data processing applications, such as Neural Network (NN) processing, require very large amounts of data. For example, a single network architecture can use millions of parameters. Consequently, there is great interest in storing data as efficiently as possible. In some applications, for example, 8-bit scaled integers are used for inference but data for training requires the use of floating-point numbers with a greater exponent range than the 16-bit IEEE half-precision format, which has only 5 exponent bits. A 16-bit “Bfloat” format has been used successfully for NN training tasks. The Bfloat format has a sign bit, 8 exponent bits, and 7 fraction bits (denoted as s,8e,7f). Other FP formats have been proposed recently, including “DLfloat” which has 6 exponent bits and 9 fraction bits (s,6e,9f) as well as other 8-bit formats having more exponent bits than fraction bits (such as s,4e,3f and s,5e,2f). Block Floating-Point (BFP) representation has been used in a variety of applications, such as NN and Fast Fourier Transforms. In BFP, a block of data shares a common exponent, typically the largest exponent of the block to be processed. The significands of FP numbers are right-shifted by the difference between their individual exponents and the shared exponent. BFP has the added advantage that arithmetic processing can be performed on integer data paths saving considerable power and area in NN hardware implementation. BFP appears particularly well-suited to computing dot products because numbers with smaller exponents will not contribute many bits, if any, to the result. However, a difficulty with using BFP for processing Convolutional Neural Networks (CNNs) is that output feature maps are derived from multiple input feature maps which can have widely differing numeric distributions. In this case, many or even most of the numbers in a BFP scheme for encoding feature maps could end up being set to zero. By contrast, the weights employed in CNNs are routinely normalized to the range −1 . . . +1. Given that successful training and inference is usually dependent on the highest magnitude parameter of each filter, blocks of weights need exponents to sit only within a relatively small range.
TABLE 1 shows an example dot product computation for vector operands A and B. The number are denoted by hexadecimal significands with radix 2 exponents. Corresponding decimal significands and exponents are shown in brackets. The maximum of each vector is shown in bold font.
−0 × 1.ccp
+
20
(−1.80
×
2
20
)
+0 × 1.dep
+
19
(1.87
×
2
19
)
Result
+0 × 1.5d1bp
+
29
TABLE 2 shows the same dot product computation for vector operands A and B performed using Block Floating Point arithmetic. In this example, the dot product is calculated as zero because a number of small operands are represented by zero in the Block Floating Point format.
0
This example illustrates that conventional Block Floating Point arithmetic is not well suited for use where the data has a large range of values.
The present disclosure uses a number format, referred to as Enhanced Block Floating Point (EBFP). The format may be used in applications such as convolutional neural networks where (i) individual feature maps have widely differing numeric distributions and (ii) filter kernels only require their larger parameters to represented with higher accuracy.
In accordance with various embodiments, the exponent of a floating number to be encoded is compared with the shared exponent: when the difference is large enough that the BFP representation would be zero due to all the significand bits being shifted out of range, the exponent difference is stored; otherwise, the suitably encoded significand is stored.
In accordance with an embodiment of the disclosure, a number in floating-point format is converted to a number in EBFP format in a data processor. An input value having a sign, an exponent and a significand is encoded by determining an exponent difference between a base exponent and the exponent, setting one or more tag bits of an output value based on the exponent difference. When the exponent difference is less than a first threshold, the significand and exponent difference are encoded to a payload of the output value. When the exponent difference is not less than the first threshold, only the exponent difference is encoded to the payload of the output value. A sign bit in the output value is set corresponding to the sign of the input value, and the output value is stored.
The EBFP format is described in more detail below with reference to an apparatus for converting a floating-point (FP) number to an EBFP. In addition to the encoding scheme, two other aspects of EBFP are described: (a) rounding, and (b) special values. Rounding can be employed when converting a floating-point number into EBFP to preserve as much accuracy as possible. In one embodiment, a round-to-nearest scheme is used (ties away; i.e. round up when the guard bit is set) so that the upper fraction bits of 8-bit and 16-bit EBFP numbers are the same for all numbers. Other schemes may be used, such as IEEE round-to-nearest (ties nearest even) or performing a logic OR operation between the guard bit and the significand least significant bit (lsb). Rounding can occur across the boundary between the two EPFP representations. The largest exponent difference that can be represented with 5 bits is 31. In one embodiment of EBFP, this value represents zero when the sign bit is 0 or (optionally) Not a Number (IEEE NaN or unsigned Infinity) when the sign bit is 1.
First word 204 includes sign bit 210, 1-bit tag 212, and a payload consisting of fields 214, 216, 218 and 220. The tag bit 212 is set to zero to indicate that the payload is associated with a significand. Fields 214, 216 and 218 indicate a difference between the shared exponent 202 and the exponent of the number being represented. Field 214 contains L zeros, where L may be zero. Field 216 contains a “one” bit, and field 218 contains an R-bit integer, where R is a designated integer. The factor 2(R+1) is called the “radix” of the representation, so the radix is 2 when R=0 , 4 when R=1, and 8 when R=2. Field 218 is omitted when R=0. The exponent difference is given by 2R×L+P . Field 220 is a rounded and right-shifted fractional part of the significand. The total number of bits in the payload is fixed. Since the number of zeros in field 214 is variable, the number of bits, T, in fraction field varies accordingly. When the integer value of field 220 is F, the significand is given 1+2−T×F, which may be denoted by 1.fff . . . f. Thus, when the shared exponent is se, the number represented is case:
x=2se×2−(2
In one embodiment, the designated number R is zero and the radix is two. In this
x=2se×2−L(1+2−T×F),
and the payload is simply the right-shifted significand. The exponent difference may be determined by counting the number of leading zeros in the EBFP number.
In second payload 206, the payload 222 is set to zero. When the tag bit is zero, the payload represents the number zero. When the tag bit is one, the payload represents an exponent difference of −1. This can occur when rounding causes the maximum value to overflow. Thus, the number represented is 2se+1.
In payload 208, the tag bit is set to one to indicate that the payload 224 relates only to the exponent difference. When the payload is an integer E, the number represented is 2se+E+bias, where bias is an offset or bias value. The bias value is included since some small values of exponent difference can represented by payload 204.
TABLE 3 shows how output values are produced based on an exponent difference for an example implementation where the payload has 8 bits and includes a sign bit, a tag bit and 6 payload bits. In this example, R=0, so the radix is 2. The format is designated “8r2”. In the table below, “f” denotes fractional bit of the input value and “e” denotes one bit of the biased exponent difference.
For zero tag, the bits indicated in bold font indicate the encoding of the exponent difference. In this example, the payload is equivalent to a right-shifted significand, including an explicit leading bit. Note that for an exponent difference greater than 5, the right-shifted significand is lost because of the limited number of bits. For an exponent difference greater than 5, only the exponent difference is encoded with a bias of 6.
In the coding of the tag and exponent difference, each bit has two states indicated by 1 and 0. It will be apparent to those of skill in the art that, herein, the states may equivalently be represented by 0 and 1.
In the embodiment shown in TABLE 3, the exponent difference can be decoded from the EBFP number by counting the number of leading zeros in the payload. This operation is denoted as CLZ(payload).
TABLE 4 shows the result of the example dot product computation described above. The exponents and signs of FP values with smaller exponents are retained. The resulting error compared to the true result is 13%. This is much improved compared to conventional BFP, which gave the results as zero. The accuracy of the EBFP approach is sufficient for many applications, including training convolutional neural networks.
TABLE 5 shows how output values are produced based on an exponent difference for an example implementation where the payload has 8 bits and includes a sign bit, two tag bits and 5 payload bits. In this example, R=0. In the table below, “f” denotes fractional bit of the input value and “e” denotes one bit of the biased exponent difference. Is this embodiment, the exponent difference can be decoded from the EBFP number by counting the number of leading zeros in the tag and payload. This operation is denoted as CLZ(tag, payload).
TABLES 1 and 2 above, illustrate how an output payload can be obtained from an exponent difference and a significand.
TABLE 6 shows how output values are produced based on an exponent difference for an example implementation where the payload has 8 bits and includes a sign bit, a tag bit and 6 payload bits. In this example, R=1, so the radix is 4. In the table below, “f” denotes fractional bit of the input value and “e” denotes one bit of the biased exponent difference.
In the examples above, the significand is stored to the right of the encoded exponent difference. It will be apparent to those of ordinary skill in the art that alternative arrangements may be used without departing from the present disclosure. For example, in one embodiment, the significand is stored to the left of the encoded exponent difference, and the encoded exponent difference includes L trailing zeros. This is shown in TABLE 7A below. For the encoded exponent in this embodiment the use of one and zeros is reversed. The exponent difference can be decoded by counting the number of trailing zeros in the tag and payload. The exponent difference is decoded as 2×CTZ(tag, payload)+p−1.
The payload is made up an encoded exponent difference (shown in bold font) concatenated with a number (possibly 0) of fraction bits (ff . . . f), where the encoded exponent difference includes a number (possibly 0) of bits set to zero, at least one bit set to one, and a number (possibly 0) of additional bits (p).
TABLE 7B, below, shows an example encoding using storage 304′ in
The payload is made up an encoded exponent difference concatenated with a number (possibly 0) of fraction bits (ff . . . f), where the encoded exponent difference includes a number (possibly 0) of bits set to zero, at least one bit set to one, and a number (possibly 0) of additional bits (p).
In accordance with various representation embodiments, a digital processor includes a subtraction unit, a tag unit, a positional encoder, an exponent unit and output storage. The subtraction unit is configured to determine an exponent difference between a base exponent and an exponent of an input value, the input value having a sign, an exponent and a significand. The tag unit is configured to produce a tag value based, at least in part, on the exponent difference. The positional encoder is configured to produce a first payload based on the significand of the input value and the exponent difference. The exponent unit is configured to produce a second payload from the exponent difference. The output storage is configured to store, as an output value, the tag value, a sign bit indicating the sign of input value and a payload.
The first payload is stored when the exponent difference is less than a first threshold, and the second payload is stored when the exponent difference is not less than the first threshold.
When the exponent difference is not less than the first threshold, as depicted by the negative branch from decision block 710, flow continues to decision block 718. When the exponent difference of the input FP number is less than a second threshold value, as depicted by the positive branch from decision block 718, the exponent difference is encoded to the output EBFP number at block 720. For example, the output payload may be a biased exponent difference. When the exponent difference of the input FP number is not less than a second threshold value, as depicted by the negative branch from decision block 718, the output payload is set, at block 722, to a designated value to indicate underflow. The resulting EBFP number represents zero. Flow continues to decision block 714.
By this method, the payload in the resulting EBFP number may represent an exponent-difference, an exponent-difference and a significand, or a special value such as zero. The one or more tag bits indicate how the payload is to be interpreted.
The example number formats described above use 8-bit words. This enables computations to be made using shorter word lengths. This is advantageous, for example, when a large number of values is being processed for when memory is limited. In some applications, such as accumulators, more precision is needed. An EBFP format using 16-bit words is described below. In general, the format using M-bit words, where M can be any number (e.g., 8, 16, 24, 32, 64 etc.).
In one embodiment using 16-bit words, all EBFP16 numbers have an additional eight fraction bits than in EBFP8, while the range of exponent differences is the same as in EBFP8. EBFP16 may be used where a wider storage format is needed and provides better accuracy and a wider exponent range than the “Bfloat” format.
TABLE 8 below gives an example of an EBFP16r2 (radix 2) format with two tag bits. Note that for exponent differences in the range 7-37, the last eight bits of the payload contain the fractional part of the number, while the first 5 bits contain the exponent. In this case, the payload is similar to floating point representation of the input, except that the exponent is to be subtracted from the shared exponent.
TABLE 9 below gives an example of an EBFP16r4 (radix 4) format with two tag bits.
In one embodiment, an EBFP number is encoded in a first format of the form “s:tag:P:1:F” or second format of the form “s:tag:D”. where “s” is a sign-bit, “tag” is one or more bits of an encoding tag, “P” is R encoded exponent difference bits, “F” is a fraction and “D” is an exponent difference. Except for a subset of tag values, the floating-point number represented has significand 1.F and exponent difference 2R×(tag+CLZ)+P, where CLZ is the number of leading zeros in the fraction F. For a first special tag value (e.g., all ones), the second format is used where the exponent difference is D plus a bias offset.
Some example embodiments for an 8-bit EBFP number are given below in TABLE 10.
In contrast with the embodiments discussed above, the positions of the one or more “p” bits are fixed as the leading bits in the payload. With an 8-bit data, R may be in the range 0-5. Some examples are listed below in TABLES 11-15.
In TABLE 15, “xxx” is any 3-bit combination except for the special values “111” and“110”.
Still further embodiments are given in TABLES 16-18.
TABLE 18 is equivalent to TABLE 17 and illustrates how the use of zero and one in the part of the encoding shown in bold font may be reversed.
To improve accuracy when the number of fraction bits is reduced, rounding is used. Examples of rounding a 16-bit floating point number into EBFP8r2 and EBFP16r2 formats are now described. Bits shown in bold font are encoded in both EBFP8 and EBP16 formats. For clarity, these nits are separated by a space from the 8 trailing bits.
Example 1: Floating-point number=+1.11010 10011111 01×2sh-exp.
For upper bits, the guard bit is G=1, while for the lower bits the guard bit is G=0. Thus, the EBFP8 format is: 0 10 11011, and the EBFP16 format is: 0 10 11011 10011111. In the EBFP format, 1 denotes a negative, 2′s-complement, most significant bit of the lower bits.
Example 2: Floating-point number=+1.1101 01001111 101×2(sh-exp−2).
For the upper bits, the guard bit is G=0, while for lower bits the guard bit is G=1. Thus, the EBFP8 formatted number is: 0 00 11101, and the EBFP16 formatted number is: 00 11101 01010000.
Rounding to Nearest (Ties Away) generally results in the same most significant bits for both EBFP8 fraction bits as for EBFP16. However, there are some ‘corner’ cases.
Example 3: Floating-point number=+1.1111 0111111 111×2(sh-exp−2).
In this example, rounding the lower bits causes G=1 for upper bits. Thus, the EBFP8 formatted number is: 0 00 11111, and the EBFP16 formatted number is: 0 01 00000 10000000. However, this is equivalent to 0 00 11111 10000000 (but with positive most significant bit in lower 8 bits). In this case, the EBFP8 and EBFP16 MSB's do not match but are numerically equal. In one embodiment, when rounding from EBFP16 to EBFP8, the EBFP8 payload is decremented if the bottom 8 bits of EBFP16==0×80. Otherwise, the payload is truncated.
A method for rounding FP32 to EBFP8-r2 is described in
{Zeros(exp−diff): “1”: FP32-frac [22:23−exp−diff]}
Finally, when the exponent difference is less than 2, the initial EBFP code is set at block 1114 to:
{(2−exp−diff): FP32-frac[22:18]}.
At block 1116, the rounded EBFP code is set as the initial code plus the round-up bit.
When the exponent difference is 38, 7 or 0, and the round-up bit is one, the rounding operation may cause the tag value to change. In this case, the rounded EBFP, tag, and payload may be adjusted, as depicted by block 1118.
TABLE 19 shows conversions from FP32 into EBFP8-r2 for some example numbers, in accordance with various embodiments of the disclosure. The shared exponent is sh-exp=+4. For cases where the tag value changes when rounding is applied, the tag values are shown in bold font.
10
11
00
In this document, relational terms such as first and second, top and bottom, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms “comprises,” “comprising,” “includes,” “including,” “has,” “having,” or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element preceded by “comprises . . . a” does not, without more constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.
Reference throughout this document to “one embodiment,” “certain embodiments,” “an embodiment,” “implementation(s),” “aspect(s),” or similar terms means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Thus, the appearances of such phrases or in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments without limitation.
The term “or,” as used herein, is to be interpreted as an inclusive or meaning any one or any combination. Therefore, “A, B or C” means “any of the following: A; B; C; A and B; A and C; B and C; A, B and C.” An exception to this definition will occur only when a combination of elements, functions, steps or acts are in some way inherently mutually exclusive.
As used herein, the term “configured to,” when applied to an element, means that the element may be designed or constructed to perform a designated function, or that is has the required structure to enable it to be reconfigured or adapted to perform that function.
Numerous details have been set forth to provide an understanding of the embodiments described herein. The embodiments may be practiced without these details. In other instances, well-known methods, procedures, and components have not been described in detail to avoid obscuring the embodiments described. The disclosure is not to be considered as limited to the scope of the embodiments described herein.
Those skilled in the art will recognize that the present disclosure has been described by means of examples. The present disclosure could be implemented using hardware component equivalents such as special purpose hardware and/or dedicated processors which are equivalents to the present disclosure as described and claimed. Similarly, dedicated processors and/or dedicated hard wired logic may be used to construct alternative equivalent embodiments of the present disclosure.
Dedicated or reconfigurable hardware components used to implement the disclosed mechanisms may be described, for example, by instructions of a hardware description language (HDL), such as VHDL, Verilog or RTL (Register Transfer Language), or by a netlist of components and connectivity. The instructions may be at a functional level or a logical level or a combination thereof. The instructions or netlist may be input to an automated design or fabrication process (sometimes referred to as high-level synthesis) that interprets the instructions and creates digital hardware that implements the described functionality or logic.
The HDL instructions or the netlist may be stored on non-transitory computer readable medium such as Electrically Erasable Programmable Read Only Memory (EEPROM); non-volatile memory (NVM); mass storage such as a hard disc drive, floppy disc drive, optical disc drive; optical storage elements, magnetic storage elements, magneto-optical storage elements, flash memory, core memory and/or other equivalent storage technologies without departing from the present disclosure. Such alternative storage devices should be considered equivalents.
Various embodiments described herein are implemented using dedicated hardware, configurable hardware or programmed processors executing programming instructions that are broadly described in flow chart form that can be stored on any suitable electronic storage medium or transmitted over any suitable electronic communication medium. A combination of these elements may be used. Those skilled in the art will appreciate that the processes and mechanisms described above can be implemented in any number of variations without departing from the present disclosure. For example, the order of certain operations carried out can often be varied, additional operations can be added or operations can be deleted without departing from the present disclosure. Such variations are contemplated and considered equivalent.
The various representative embodiments, which have been described in detail herein, have been presented by way of example and not by way of limitation. It will be understood by those skilled in the art that various changes may be made in the form and details of the described embodiments resulting in equivalent embodiments that remain within the scope of the appended claims.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2211212.2 | Aug 2022 | GB | national |
