MULTI-OBJECT DETECTION WITH SINGLE DETECTION PER OBJECT

Abstract

Systems and methods for classification of data comprise optimizing a neural network by minimizing a rhino loss function, including receiving a training batch of data samples comprising a plurality of samples for each of a plurality of classifications, extracting features from the samples to generate a batch of features, processing the batch of features using a neural network to generate a plurality of classifications to differentiate the samples, computing a rhino loss value for the training batch based, at least in part, on the classifications, and modifying weights of the neural network to reduce the rhino loss value.

Description

TECHNICAL FIELD

The present application, in accordance with one or more embodiments, relates generally to classification systems and methods and, more particularly, for example, to systems and methods for training and/or implementing multi-object classification systems and methods.

BACKGROUND

Object detection is often implemented as a computer vision technique for locating instances of objects in images or videos. Object detection algorithms typically leverage machine learning or deep learning to produce meaningful results. When humans look at images or video, they can recognize and locate objects of interest within a matter of moments. A goal of object detection is to replicate this intelligence using a computer. In some systems, objects are detected in an image by an object detection process and a bounding box is defined surrounding each detected object with an identification of an object class. For example, an image of a neighborhood may include a dog, a bicycle and a truck, that are each detected and classified.

Object detection is used in a variety of real-time systems, such as advanced driver assistance systems that enable cars to detect driving lanes or perform pedestrian detection to improve road safety. Object detection is also useful in applications such as video surveillance, image retrieval, and other systems. Object detection problems are often solved using deep learning, machine learning, and other artificial intelligence systems. Popular deep learning-based approaches use convolutional neural networks (CNNs), such as regions with convolutional neural networks (R-CNN), You Look Only Once (YOLO), and other approaches that automatically learn to detect objects within images.

In one approach for object detection through deep learning, a custom object detector is created and trained. To train a custom object detector from scratch, a network architecture is designed to learn the features for the objects of interest, using a large set of labeled data to train the CNN. The results of a custom object detector are acceptable for many applications. However, these systems may require a lot of time and effort to set up the layers and weights in the CNN. In a second approach, a pretrained object detector is used. Many object detection workflows using deep learning leverage transfer learning, an approach that enables the system to start with a pretrained network and then fine-tune it for a particular application. This method can provide faster results because the object detectors have already been trained on thousands, or even millions, of images, but has other drawbacks in terms of complexity and accuracy.

In view of the forgoing, there is a continued need in the art for improved object detection and classification systems and methods.

SUMMARY

The present disclosure is directed to systems and methods for object detection and classification. In various embodiments, improved systems and methods are described that can be used for a variety of classification problems including object detection and speech recognition tasks. In some embodiments, improved training methods incorporate a “rhino” loss function to force the model to activate one time for each object. These approaches reduce the complexity of full system solutions, including eliminating the need in many embodiments for conventional post-processing that is typically applied after the classification step. For example, in some object detection systems, a post-processing step called Non-Maximum suppression is used to reject redundant detections per object. This post-processing not only increases the computational complexity, it also decreases the performance. The single-detection systems and methods disclosed herein provide advantages over such systems.

Various embodiments disclosed herein can be used without conventional post-processing, greatly reducing the amount of computational complexity in run-time and increasing effectiveness to accurately estimate small objects. In addition, the training system can converge faster than other, state of art methods. In a speech recognition task, for example, a system of the present disclosure is configured to apply a heavy decoding algorithm in order to decode the speech letters from the input data. In practice, the decoding can be less than optimal due to a trade-off between the amount of processing and the performance using the search algorithm. The techniques disclosed herein can greatly simplify the decoding part of speech recognition and it can improve the performance while reducing the computational complexity.

The scope of the present disclosure is defined by the claims, which are incorporated into this section by reference. A more complete understanding of the present disclosure will be afforded to those skilled in the art, as well as a realization of additional advantages thereof, by a consideration of the following detailed description of one or more embodiments. Reference will be made to the appended sheets of drawings that will first be described briefly.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the disclosure and their advantages can be better understood with reference to the following drawings and the detailed description that follows. It should be appreciated that like reference numerals are used to identify like elements illustrated in one or more of the figures, where showings therein are for purposes of illustrating embodiments of the present disclosure and not for purposes of limiting the same. The components in the drawings are not necessarily to scale, emphasis instead being placed upon clearly illustrating the principles of the present disclosure.

FIG. 1 illustrates an example backbone network for use in an object detection process, in accordance with one or more embodiments of the present disclosure.

FIG. 2 illustrates an example object detection process, in accordance with one or more embodiments of the present disclosure.

FIG. 3 illustrates an example object detection process, in accordance with one or more embodiments of the present disclosure.

FIG. 4 illustrates an example object detection process including images of a detected car, in accordance with one or more embodiments of the present disclosure.

FIG. 5 illustrates an example object detection process, including combination of feature representations to produce activations in the grid cell responsible for detecting a car, in accordance with one or more embodiments of the present disclosure.

FIG. 6 illustrates an example object detection process for an image including a person riding a motorcycle, in accordance with one or more embodiments of the present disclosure.

FIG. 7 illustrates an example bounding box and cell grid, in accordance with one or more embodiments of the present disclosure.

FIG. 8 illustrates an example bounding box and cell grid, in accordance with one or more embodiments of the present disclosure.

FIG. 9 illustrates an example object detection process using a bounding box and cell grid, in accordance with one or more embodiments of the present disclosure.

FIG. 10 illustrates an example bounding box and cell grid used in an example object detection process, in accordance with one or more embodiments of the present disclosure.

FIGS. 11A-C illustrate an example object detection and classification process, in accordance with one or more embodiments of the present disclosure.

FIGS. 12A-B illustrate an example object detection and classification process, in accordance with one or more embodiments of the present disclosure.

FIG. 13 illustrates an example neural network, in accordance with one or more embodiments of the present disclosure.

FIG. 14 illustrates and example object detection system, in accordance with one or more embodiments of the present disclosure.

DETAILED DESCRIPTION

The present disclosure is directed to improved systems and methods for object detection and/or classification. The techniques disclosed herein can be applied generally to classification problems, including voice detection and authentication in audio, object detection and classification in an image, and/or other classification problems. For example, a two-dimensional classification problem may include an object detection process directed to identifying and locating objects of certain classes in an image. Object localization can be done in various ways, including creating a bounding box around the object. A one-dimensional classification problem, for example, may include phoneme recognition. In phoneme recognition, unlike object detection in an image, the system receives a sequence of data. The detection of classes in a sequence is often important when detecting speech. In the present disclosure, improved techniques are described that can be applied to various classification systems, including an object detection problem (as an example of a 2-D classification problem) and a phoneme recognition problem (as an example of a 1-D classification problem with a sequential data).

Whether a classification system includes a custom object detector or uses a pretrained one, the system designer decides what type of object detection network to use (e.g., a two-stage network or a single-stage network). The initial stage of two-stage networks, such as R-CNN and its variants, identifies region proposals, or subsets of the image that might contain an object. The second stage classifies the objects within the region proposals. Two-stage networks can achieve accurate object detection results; however, they are typically slower than single-stage networks.

In single-stage networks, such as YOLO v2, the CNN produces network predictions for regions across an image using anchor boxes, and the predictions are decoded to generate the final bounding boxes for the objects. Single-stage networks can be much faster than two-stage networks, but they may not reach the same level of accuracy, especially for scenes containing small objects. However, single-stage networks are simpler, faster and memory and computationally efficient object detectors and more practical to be used in many end-user products.

Many conventional object detector techniques require the use of a post processing stage, such as non-max suppression, in order to disregard redundant detections for each object. For example, an object detector may detect a single object (e.g., a car) three different times and place three different bounding boxes around the object. After using the non-max suppression, the highest confident estimation will be retrieved while the others will be rejected allowing each object to be identified using a single bounding box. This post processing stage can impose additional computational complexity especially when the number of objects per image is high. Embodiments of the deep learning-based techniques disclosed herein include a single-stage object detector that does not require the post processing stage such as non-max suppression, which can improve the performance of estimation for multi-class object detection.

Referring to the figures, embodiments of the present disclosure will now be described. The present disclosure introduces a novel network that can recognize the multi-class objects and localize them with one bounding box per object. The proposed technique is a pretrained object detector that leverages the transfer learning in order to build a single-stage object detector.

In order to understand what is in an image, the input image is fed through a convolutional network to build a rich feature representation of the original image. This part of the architecture may be referred to herein as the “backbone” network, which is pre-trained as an image classifier to learn how to extract features from an image. In this approach, it is recognized that image classification may be easier and cheaper to label than a full image as it only requires a single label as opposed to defining bounding box annotations for each image. The training can be conducted on a large labeled dataset (e.g., ImageNet) in order to learn good feature representations.

An example of a backbone network is illustrated in FIG. 1 and will now be described in accordance with one or more embodiments. A convolutional neural network 100 (for example, a VGG network) may be implemented using an architecture configured to receive and process an input image 110 from a training dataset for image classification. The input image 110 is converted to a fixed size and image format 120 and then passed through a plurality of convolutional layers 140, which include rectified linear activation functions, max pooling layers, a softmax output layer 150, and/or other processing steps.

Referring to FIG. 2, after pre-training the backbone architecture 100 as an image classifier, the last few layers of the network are removed so that the backbone network 100 outputs a collection of stacked feature maps 130 which describe the original image in a low spatial resolution albeit a high feature (channel) resolution. In the illustrated example, a 7×7×512 representation of the image observation includes 512 feature maps describing different characteristics of the original image 110.

Referring to FIG. 3, the 7×7 grid 130 can be related back to the original input image 110 in order to understand what each grid cell represents relative to the original image. From this data, the system can also determine roughly where objects are located in the coarse (7×7) feature maps by observing which grid cell contains the center of the bounding box annotation. This grid cell can be identified as being “responsible” for detecting that specific object. Referring to FIG. 4, for example, a car is identified in a bounding box 112, the center of the bounding box identified and a corresponding grid cell 114 is identified at the cell “responsible” for detecting the car. Referring to FIG. 5, the feature representations from the grid 130 are combined to produce an activation in the grid cell 114 responsible for the detecting the car.

If the input image contains multiple objects, then multiple activations can be identified on the grid denoting that an object is in each of the activated regions. For example, as illustrated in the example of FIG. 6, two objects are detected in an image, namely a “person” and a “motorbike”. In the first image 600A, a first bounding box 610 bounds the detected person, and a second bounding box 620 bounds the motorbike. In the next image 600B, the center 610A of bounding box 610 and the center 620A of bounding box 620 are identified. The corresponding grid cells 610B and 620B, respectively, are illustrated in image 600C. The output of the last layer of the network has two activations, 610C and 620C of image 600D, related to the two objects.

In various embodiments disclosed herein, the network learns to find the responsible grid cell to be used for detection of the object. In other words, the network will choose all the grid cells that are inside the ground truth bounding box of the object, such as the grid cells marked with an “X” in FIG. 7, to be the target grid cells and will be used to detect the car in the bounding box 700. Then the network will train to choose one of these target grid cells to activate and use it to detect the object.

In some embodiments, the last layer generates an N*N output probability for each class (here it is assumed N=7 in a 7×7 grid). If we assume the number of classes is C, then there will be N*N*C output probabilities (y⁽¹⁾, . . . , y^(C)). For each of N*N grid cells, it also generates four coordinates c_x¹, c_y¹, c_x², c_y² corresponding to the estimated four outputs which are related to the position of the x-axis and y-axis of the two corners on the top left and bottom right of the rectangular bounding box as it is shown in FIG. 8. The outputs of the network are obtained after using a sigmoid function resulting in numbers between zero and one. The reference point for each grid cell is the center of the grid cell as it is shown by the circle inside the grid cell 810. The center of the grid cell corresponds to c_x¹=c_y¹=c_x²=c_y²=0. c_x¹, c_y¹ moves the left corner of the rectangular bounding box to the upper left region of the image and c_x², c_y² moves the right corner of the rectangular bounding box to the right bottom region of the image. So c_x¹ moves along the horizontal arrow 820 in the image when the value changes from zero to one, c_x² moves along the horizontal arrow 830 in the image when the value changes from zero to one while c_y¹ moves along the vertical arrow 840 in the image 800 when the value changes from zero to one, c_y² moves along the vertical arrow 850 in the image 800 when the value changes from zero to one. The estimated coordinates using c_x, c_y¹, c_x², c_y² will be mapped to the x and y axis of the image by considering the left corner of the image to have (x,y)=(0,0) and the right bottom corner of the image to have (x,y)=(1,1) as it is shown in the image below. The estimated mapped coordinate for each grid cell will be named as m_x¹, m_y¹, m_x², m_y².

The likelihood that a grid cell contains the object of class i is defined as y⁽ⁱ⁾ and it is assumed that the number of classes is C. If all the y⁽ⁱ⁾ for all grid cells are close to zero, then there is a determination of no object detected in the image.

Four bounding box descriptors are used to describe the x-y coordinate of the upper left corner of the bounding box (c_x¹, c_y¹) and bottom right corner of the bounding box (c_x², c_y²). These values will be mapped to get the corresponding values (m_x¹, m_y¹, m_x², m_y²) considering the reference point of (x=0,y=0) on the upper left corner of the image and (x=1,y=1) on the bottom right corner of the image.

Thus, the network is configured to learn a convolution filter for each of the above attributes such that it produces 4+C output channels to describe a single bounding box at each grid cell location. This means that the network will learn a set of weights to look across all feature maps (in above example it is assumed to be 512) to evaluate the grid cells.

It is possible that we increase the size of model by introducing new parameters to learn for each class to estimate the bounding box. In other words, there will be 5*C output for each grid cell instead of 4+C as it is shown in the figure below. This will enlarge the model size at the output layer, and it may improve the performance of the model for the objects that have different aspect ratio or shape. In this embodiment we assume we have 4+C output for each grid cell unless it says otherwise.

Now we will describe a proposed rhino loss function that enforces the network to detect each object using only one grid cell activation. Without loss of generality, it is assumed the number of class is one (C=1) and the object of interest is “car”. So, we have “car” confidence score y_n⁽¹⁾ and bounding box coordinate m_xn¹, m_yn¹, m_xn², m_yn² for n-th grid cell. In each image, each object is shown with a rectangular bounding box around it as its ground truth. All the grid cells inside the bounding box will be considered as target grid cells that will be used to detect the object. For example, a car object in image 900 of FIG. 9 and image 1000 of FIG. 10 has a bounding box with twelve grid cells as the target grid cells. Slices of network output corresponding to each object are extracted. When all the slices are extracted and there is no more object left in the image, the remaining image will belong to background objects. For example, there is only one object in the image of FIG. 10 and the slice corresponding to this object is extracted from the image and the remaining image with background objects is shown in image 1020 on the right. For each slice of the image one mask can be generated in order to generate the slice. The mask is one when the grid cell in the slice of the object and it is zero elsewhere. For example, as illustrated in FIG. 10, an image 1000 includes a mask for the of a “car” object has value equals to one inside the bounding box and it is zero elsewhere. An example mask of the slice is illustrated in FIG. 10, which shows a slice 1010 representing the car extracted from the image 1000, and the remaining image 1020.

The rhino loss function for ith sample of the data (L_rhino(i)) and the total detection loss for a batch data of size D (L_rhino^total) is given below.

• The total number of grid cells=N²
• The number of objects or slices for jth class of ith sample of the data=S_ij
• The number of classes=C
• The binary mask of sth object for jth class of ith sample of the data=msk_n,s^(j)(i)

$\begin{array}{cc}{d}_{n,s}^{\left(j\right)}\left(i\right)=y_n^{\left(j\right)}\left(i\right)*{\mathrm{msk}}_{n,s}^{\left(j\right)}\left(i\right),i=0,\dots ,D-1,s=0,\dots ,S_{\mathrm{ij}}-1,j=0,\dots ,C-1,n=0,\dots ,N^2-1& \left(1\right)\\ L_{\det }^{\mathrm{obj}}\left(i\right)=-\sum _{j=0}^{C-1}\sum _{s=0}^{S_{\mathrm{ij}}-1}\log \left(P_s^{\left(j\right)}\left(i\right)\right)& \left(2\right)\\ P_s^{\left(j\right)}\left(i\right)=\sum _{n=0}^{N^2-1}{p}_{n,s}^{\left(j\right)}\left(i\right)& \left(3\right)\\ p_{n,s}^{\left(j\right)}\left(i\right)=\left(1-d_{0,s}^{\left(j\right)}\left(i\right)\right)\left(1-d_{1,s}^{\left(j\right)}\left(i\right)\right)\dots \left(d_{n,s}^{\left(j\right)}\left(i\right)\right)\dots \left(1-d_{N^2-1,s}^{\left(j\right)}\left(i\right)\right)& \left(4\right)\\ {\mathrm{Msk}}_n^{\left(j\right)}\left(i\right)=\max \left(\sum _{s=0}^{S_{\mathrm{ij}}-1}{\mathrm{msk}}_{n,s}^{\left(j\right)}\left(i\right),1\right)& \left(5\right)\\ {\mathrm{mskg}}_n^{\left(j\right)}\left(i\right)=1-{\mathrm{Msk}}_n^{\left(j\right)}\left(i\right)& \left(6\right)\\ g_n^{\left(j\right)}\left(i\right)=y_n^{\left(j\right)}\left(i\right)*{\mathrm{mskg}}_n^{\left(j\right)}\left(i\right),i=0,\dots ,D-1,j=0,\dots ,C-1,n=0,\dots ,N^2-1& \left(7\right)\\ L_{\det }^{\mathrm{no}-\mathrm{obj}}\left(i\right)=-\sum _{j=0}^{C-1}\sum _{n=0}^{N^2-1}{\left(g_n^{\left(j\right)}\left(i\right)\right)}^{\gamma }\log \left(1-g_n^{\left(j\right)}\left(i\right)\right)& \left(8\right)\end{array}$

γ≥0 is a hyperparameter that needs to be tuned for the training

$\begin{array}{cc}{L}_{\mathrm{rhino}}\left(i\right)=L_{\det }^{\mathrm{obj}}\left(i\right)+L_{\det }^{\mathrm{no}-\mathrm{obj}}\left(i\right)& \left(9\right)\\ L_{\mathrm{rhino}}^{\mathrm{total}}=\frac{1}{D}\sum _{i=1}^D{L}_{\det }\left(i\right)& \left(10\right)\end{array}$

Embodiments of rhino loss with overlapping bounding boxes using reassignment will now be described with reference to FIGS. 11A, 11B and 11C. As illustrated bounding boxes of the same class (e.g., bounding boxes 1100A and 1100B) may have overlaps and so the binary mask corresponding to each object may also have overlaps. For example, there are three classes “cat”, “dog” and “duck” in the images of FIGS. 11A-C, and all of the bounding boxes (1100A, 1100B, 1100C, and 1100D) of these all objects have overlapping regions. As a result, the mask corresponding to these overlapped objects are modified in various embodiments.

In one embodiment, the mask of each overlapped object is modified at each update of the training. The modified mask is called _n,s^(j)(i). To do this, the following rhino soft target score (rhino_n,s^(j)(i)) is computed for each object (∀s).

$\begin{array}{cc}{\mathrm{rhino}}_{n,s}^{\left(j\right)}\left(i\right)=\frac{p_{n,s}^{\left(j\right)}\left(i\right)}{P_s^{\left(j\right)}\left(i\right)},i=0,\dots ,D-1,s=0,\dots ,S_{\mathrm{ij}}-1,j=0,\dots ,C-1,n=0,\dots ,N^2-1& \left(11\right)\end{array}$

The rhino soft target score will be computed for each grid of all the objects. Then the object of any class that has the maximum metric value will have its mask to be one. For example, in the image of FIG. 11B, the two cats have 6 grid cells in an overlap area (e.g., the area 1120 where the bounding boxes 1100A and 1100B overlap). After computing the rhino metric, the system makes a determination assigning the grid cells of the overlap region to one of the objects. For example, in the illustrated embodiment, the system decided that the first two black grid cell belong to the right object represented by bounding box 1100B and the other four white grid cells in the overlap region belong to the left object represented by bounding box 1100A.

In one embodiment, the system replaces the mask in (1) with the modified mask computed using the method below to address the problem of overlapping area of the bounding boxes:

for each i compute rhino_n,s^(j)(i) for all j, s and n

for each n find s* and j* that has maximum value of rhino_n,s^(j)(i)

among all s and j. Set _n,s*^(j*)(i)=1 and _n,s^(j)(i)=0 for other j, s

If we increase the number of parameters by having one set of coordinates for each class, then there is no need to modify the mask for overlapping area of objects belonging to different classes. In this case the number of outputs will be changed from (4+C)*N² to 5*C*N². This can increase the number of parameters of the object detector model and it may also improve the performance when the classes do not have similar shapes or aspect ratio (e.g. person and car).

In another embodiment, an alternative approach to address the problem of overlapping bounding box when the two bounding boxes belong to different classes is provided. Note that if the overlapped bounding box belong to the same class, then the method that is proposed above with respect to FIGS. 11A and 11B may be used. In this embodiment, we assume that overlapped classes for the sth object of jth class of sample i at the grid cell n is overlapClass_n,s^(j)(i). For example, in image of FIG. 11C, which is assumed to be ith sample of data, there are three classes of “cat” (j=0), “dog” (j=1) and “duck” (j=2). If n is the shared grid cell 1130 that has overlap with all the three classes as it is shown in the figure, overlapClass_n,s=0^(j=2)(i)={0,1}. This is because “duck” object which is the first object of class “duck” (s=0) has two overlaps with two classes of “cat” and “dog” at grid cell n and so overlapClass_n,s=0^(j=2)(i) contains the index of these two classes which is zero and one.

Equation (4) can be revised as follows to address the problem of overlapped bounding boxes of different class:

$\begin{array}{cc}{h}_{n,s}^{\left(j\right)}\left(i\right)=\left(\prod _{k\in {\mathrm{overlapClass}}_{n,s}^{\left(j\right)}\left(i\right)}{\mathrm{msk}}_{m,s}^{\left(j\right)}\left(i\right)\left(1-y_n^{\left(k\right)}\left(i\right)\right)\right)& \left(12\right)\\ p_{n,s}^{\left(j\right)}\left(i\right)=\left(1-d_{0,s}^{\left(j\right)}\left(i\right)\right)\left(1-d_{1,s}^{\left(j\right)}\left(i\right)\right)\dots \left(h_{n,s}^{\left(j\right)}\left(i\right)\right)\left(d_{n,s}^{\left(j\right)}\left(i\right)\right)\dots \left(1-d_{N^2-1,s}^{\left(j\right)}\left(i\right)\right)& \left(13\right)\end{array}$

As shown one additional term (h_n,s^(j)(i)) is added to the multiplications in (4) in order to address the problem of overlapping bounding boxes when the objects belong to different classes. As it is mentioned above, if there is mix of intra class and inter class objects, the grid cells of intra class objects may be reassigned using the method previously discussed and the inter class objects will have their rhino loss function modified as it is given in (12)-(13).

The bounding box loss function is designed to estimate the boundary box around the estimated object. The total bounding box loss is defined as follows:

$\begin{array}{cc}{\mathrm{LoC}}_{n,s}^{\left(j\right)}\left(i\right)={n,s}_{\left(j\right)}^{}\left(i\right)*{\mathrm{rhino}}_{n,s}^{\left(j\right)}\left(i\right)*\left(1-{\mathrm{IoU}}_{n,s}^{\left(j\right)}\left(i\right)+R_{n,s}^{\left(j\right)}\left(B,\mathrm{Bgt}\right)\right)& \left(14\right)\\ L_{\mathrm{loc}}\left(i\right)=\sum _{j=0}^{C-1}\sum _{s=0}^{S_{\mathrm{ij}}-1}\sum _{n=0}^{N^2-1}{\mathrm{LoC}}_{n,s}^{\left(j\right)}\left(i\right)& \left(15\right)\\ L_{\mathrm{loc}}^{\mathrm{total}}=\frac{1}{D}\sum _{i=1}^D{L}_{\det }\left(i\right)& \left(16\right)\end{array}$

Where IoU_n,s^(j) and R_n,s^(j) are the Intersection over Union (IoU) loss and the penalty term defined in [7] for predicted box B and target box Bgt for each grid cell n of sth object of the jth class of image i. Both IoU_n,s^(j) and R_n,s^(j) will be computed using (m_x¹(i), m_y,n¹(i), m_x,n²(i), m_y,n²(i)) outputs for each grid cell n. Please note that the losses are defined in [7] using the height and width of bounding box and center point of the bounding boxes. So (m_x,n¹(i), m_y,n¹(i), m_x,n²(i), m_y,n²(i)) which are the x-y coordinate of the upper left and bottom right of the bounding box will be translated to the height/width with center point and then the losses will be computed. In one embodiment, the loss may be computed as described in Zhaohui Zheng1, Ping Wang1, Wei Liu2, Jinze Li3, Rongguang Ye1, and Dongwei Ren “Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression” AAAI 2020, which is incorporated by reference herein.

The total loss function L^total is sum of bounding box loss and the rhino loss:

L ^total =L _rhino ^total +β*L _loc ^total   (17)

β>0 is a hyperparameter that needs to be tuned to balance the loss values of the two losses namely rhino loss and localization loss.

Phoneme Recognition

A phoneme recognition task involves recognizing the phoneme of speech (C classes) in a sequence of audio data. This is often the initial step for a speech recognition system. The backbone for phoneme recognition can be a recurrent neural network or CNN. Each output is the confidence score for the probability of detecting the jth class and it is obtained after applying the sigmoid function. Like object detection, a marking window is defined for each phoneme to be classified in a sequence. Note that the marking window is a 1-D array unlike the bounding box of object detector which is a 2-D array. So the rhino loss function for ith sample of the data (L_rhino(i)) and the total detection loss for a batch data of size D (L_rhino^total) can be obtained as (10).

Embodiments of applying rhino loss with an overlapping marking window in a sequence of data with reassignment will now be described. As previously explained, if two marking windows have overlap area, the system will reassign the overlap area to either of the two classes using a rhino score which such as defined in (11). For example, in FIG. 12A, class A and B have an overlap area in the middle which is shown by the shaded region in the sequence of audio frames 1200. Unlike object detection, each frame of data isn't assigned individually based on rhino score. Instead the rhino score is computed for the overlap area for both classes of A and B. Then a max value of rhino score over the overlapped area is obtained. Depending on whether this max value belongs to class A or B, the left area plus max position or right area plus max position is reassigned to class A (frames 1210) or B (frames 1220). Note that this reassignment may affect a marking window and so it will update a binary mask at each update of the training.

Embodiments of rhino loss with an overlapping marking window in a sequence of data without reassignment will now be described with reference to FIG. 12B. The overlapping window marking for a sequence of data can be solved by modifying the rhino loss function. However, there is at least one difference between the method proposed for object detection and the one that can be used for a sequence of data. The difference is that each detection for a sequence of data is sequential. In other words, the order of detection at each time frame of sequence is important. For example, if there is a sequence of data with labels ABC, only the ABC detection order is the correct estimation and all other estimations including BAC or ACB are incorrect. But in object detection the order of detection does not make any difference and so there is a modification in the method proposed in some embodiments.

Similar to overlapClass_n,s^(j)(i) discussed herein, overlapClass_preceding,s^(j)(i) and overlapClass_suceeding,s^(j)(i) are defined to include the index of overlapped classes that comes before and after the sth phoneme. For example if the sequence ABC has overlaps, overlapClass_preceding,s^(j)(i) for class B would be the index of class A and overlapClass_suceeding,s^(j)(i) for class B would be the index of class C. Also, we define n_preceding,s^(j)(i)n_suceeding,s^(j)(i) to be the end time frame of preceding class (here class A) in overlap area and start time frame of succeeding class (here class C) in overlap area. This is shown in the example of FIG. 12B.

The modified rhino loss can be written as follows:

$\begin{array}{cc}{u}_{n,s}^{\left(j\right)}\left(i\right)=\prod _{k\in {\mathrm{overlapClass}}_{\mathrm{suceeding},s}^{\left(j\right)}\left(i\right)}\prod _{m=n}^{n_{\mathrm{preceding},s}^{\left(j\right)}\left(i\right)}{\mathrm{msk}}_{m,s}^{\left(j\right)}\left(i\right)\left(1-y_m^{\left(k\right)}\left(i\right)\right)& \left(18\right)\\ v_{n,s}^{\left(j\right)}\left(i\right)=\prod _{k\in {\mathrm{overlapClass}}_{\mathrm{suceeding},s}^{\left(j\right)}\left(i\right)}\prod _{m=n_{\mathrm{suceeding},s}^{\left(j\right)}\left(i\right)}^n{\mathrm{msk}}_{m,s}^{\left(j\right)}\left(i\right)\left(1-y_m^{\left(k\right)}\left(i\right)\right)& \left(19\right)\\ p_{n,s}^{\left(j\right)}\left(i\right)=\left(1-d_{0,s}^{\left(j\right)}\left(i\right)\right)\left(1-d_{1,s}^{\left(j\right)}\left(i\right)\right)\dots \left(u_{n,s}^{\left(j\right)}\left(i\right)\right)\left(d_{n,s}^{\left(j\right)}\left(i\right)\right)\left(v_{n,s}^{\left(j\right)}\left(i\right)\right)\dots \left(1-d_{N^2-1,s}^{\left(j\right)}\left(i\right)\right)& \left(20\right)\end{array}$

Note that it is assumed that n_preceding,s^(j)(i)≥n nd n_suceeding,s^(j)≤n for each time frame n. If either of these two conditions is not met, then there is no need to compute the multiplications in (18) or (19).

The techniques described herein provide a general solution for any classification problem and so it can be applied to many problems including object detection, keyword spotting, acoustic event detection, speech recognition. The disclosure can provide an opportunity to solve many practical problems in which high accuracy with low computation complexity is an important requirement.

Referring to FIG. 13, an example a neural network and training process that may be used to generate trained artificial intelligence training models for use with the rhino loss function disclosed herein for object detection, speaker identification, and other classification will now be described, in accordance with one or more embodiments. The neural network 1300 may be implemented as any neural network configured to receive the input data samples and generate classifications as taught herein, such as a recurrent neural network, a convolutional neural network (CNN), or other neural network.

The neural network 1300 is trained using a supervised learning process that compares input data to a ground truth (e.g., expected network output). For a speaker verification system, for example, a training dataset 1302 may include sample speech input (e.g., an audio sample) labeled with a corresponding speaker ID. The input data 1302 may comprise other labeled data types, such as a plurality of images labeled with object classification data, audio data labeled for phoneme recognition, etc. In some embodiments, the input data 1302 is provided to a feature extraction process 1304 to generate a batch of features for input to the neural network 1300. The input batch is compared against the output of the neural network 1300, and differences between the generated output data and the ground truth output data are determined using a rhino loss function 1340 as discloses herein and fed back into the neural network 1300 to make corrections to the various trainable weights and biases. The loss may be fed back into the neural network 1300 using a back-propagation technique (e.g., using a stochastic gradient descent algorithm or similar algorithm). In some examples, training data combinations may be presented to the neural network 1300 multiple times until the overall rhino loss function converges to an acceptable level.

In some examples, each of input layer 1310, hidden layers 1320, and/or output layer 1330 include one or more neurons, with each neuron applying a combination (e.g., a weighted sum using a trainable weighting matrix W) of its inputs x, adding an optional trainable bias b, and applying an activation function f to generate an output a as shown in the equation a=f(Wx+b). In some examples, the activation function f may be a linear activation function, an activation function with upper and/or lower limits, a log-sigmoid function, a hyperbolic tangent function, a rectified linear unit function, and/or the like. In some examples, each of the neurons may have a same or a different activation function.

After training, the neural network 1300 may be implemented in a run time environment of a remote device to receive input data and generate associated classifications. It should be understood that the architecture of neural network 1300 is representative only and that other architectures are possible, including a neural network with only one hidden layer, a neural network with different numbers of neuron, a neural network without an input layer and/or output layer, a neural network with recurrent layers, and/or the like.

In other embodiments, the training dataset 1302 may include captured sensor data associated with one or more types of sensors, such as speech utterances, visible light images, fingerprint data, and/or other types of biometric information. The training dataset may include images of a user's face for a face identification system, fingerprint images for a finger print identification system, retina images for a retina identification system, and/or datasets for training another type of biometric identification system.

FIG. 14 illustrates an example system 1400 configured to implement a generalized negative log-likelihood loss for speaker verification, in accordance with one or more embodiment of the present disclosure. Not all of the depicted components in the example system 1400 may be required, however, and one or more embodiments may include additional components not shown in the figure. Variations in the arrangement and type of the components may be made without departing from the scope of the disclosure, including additional components, different components, and/or fewer components. While the example system of FIG. 14 is configured for speaker verification, it will be appreciated that the methods disclosed herein may be implemented through other system configurations

The system 1400 includes an authentication device 1420 including processing components 1430, audio input processing components 1440, user input/output components 1446, communications components 1448, and a memory 1450. In some embodiments, other sensors and components 1445 may be included to facilitate additional biometric authentication modalities, such as fingerprint recognition, facial recognition, iris recognition, etc. Various components of authentication device 1420 may interface and communicate through a bus or other electronic communications interface.

The authentication device 1420, for example, may be implemented on a general-purpose computing device, as a system on a chip, integrated circuit, or other processing system and may be configured to operate as part of an electronic system 1410. In some embodiments, the electronic system 1410 may be, or may be coupled to, a mobile phone, a tablet, a laptop computer, a desktop computer, an automobile, a personal digital assistant (PDA), a television, a voice interactive device (e.g., a smart speaker, conference speaker system, etc.), a network or system access point, and/or other system of device configured to receive user voice input for authentication and/or identification.

The processing components 1430 may include one or more of a processor, a controller, a logic device, a microprocessor, a single-core processor, a multi-core processor, a microcontroller, a programmable logic device (PLD) (e.g., field programmable gate array (FPGA)), a digital signal processing (DSP) device, an application specific integrated circuit, or other device(s) that may be configured by hardwiring, executing software instructions, or a combination of both, to perform various operations discussed herein for audio source enhancement. In the illustrated embodiment, the processing components 1430 include a central processing unit (CPU) 1432, a neural processing unit (NPU) 1434 configured to implement logic for executing machine learning algorithms, and/or a graphics processing unit (GPU) 1436. The processing components 1430 are configured to execute instructions stored in the memory 1450 and/or other memory components. The processing components 1430 may perform operations of the authentication device 1420 and/or electronic system 1410, including one or more of the processes and/or computations disclosed herein.

The memory 1450 may be implemented as one or more memory devices or components configured to store data, including audio data, user data, trained neural networks, authentication data, and program instructions. The memory 1450 may include one or more types of memory devices including volatile and non-volatile memory devices, such as random-access memory (RAM), read-only memory (ROM), electrically-erasable programmable read-only memory (EEPROM), flash memory, hard disk drive, and/or other types of memory.

Audio input processing components 1440 include circuits and digital logic components for receiving an audio input signal, such as speech from one or more users 1444 that is sensed by an audio sensor, such as one or more microphones 1442. In various embodiments, the audio input processing components 1440 are configured to process a multi-channel input audio stream received from a plurality of microphones, such as a microphone array, and generate an enhanced target audio signal comprising speech from the user 1444.

Communications components 1448 are configured to facilitate communication between the authentication device 1420 and the electronic system 1410 and/or one or more networks and external devices. For example, the communications components 1448 may enable Wi-Fi (e.g., IEEE 802.11) or Bluetooth connections between the electronic system 1410 and one or more local devices or enable connections to a wireless router to provide network access to an external computing system via a network 1480. In various embodiments, the communications components 1448 may include wired and/or other wireless communications components for facilitating direct or indirect communications between the authentication device 1420 and/or other devices and components.

The authentication device 1420 may further include other sensor and components 1445, depending on a particular implementation. The other sensor components 1445 may include other biometric input sensors (e.g., fingerprint sensors, retina scanners, video or image capture for face recognition, etc.), and the user input/output components 1446 may include I/O components such as a touchscreen, a touchpad display, a keypad, one or more buttons, dials, or knobs, loudspeaker and/or other components operable to enable a user to interact with the electronic system 1410.

The memory 1450 includes program logic and data configured to facilitate speaker verification in accordance with one or more embodiments disclosed herein, and/or perform other functions of the authentication device 1420 and/or electronic system 1410. The memory 1450 includes program logic for instructing processing components 1430 to perform voice processing 1452, including speech recognition 1454, on an audio input signal received through the audio input processing components 1440. In various embodiments, the voice processing 1452 logic is configured to identify an audio sample comprising one or more spoken utterances for speaker verification processing.

The memory 1450 may further includes program logic for implementing user verification controls 1462, which may include security protocols for verifying a user 1444 (e.g., to validate the user's identity for a secure transaction, to identify access rights to data or programs of the electronic system 1410, etc.). In some embodiments, the user verification controls 1462 includes program logic for an enrollment and/or registration procedure to identify a user and/or obtain user voice print information, which may include a unique user identifier and one or more embedding vectors. The memory 1450 may further include program logic for instructing the processing components 1430 to perform a voice authentication process 1464 as described herein, which may include neural networks trained for speaker verification using generalized negative log-likelihood loss processes, feature extraction components for extracting features from an input audio sample, processes for identifying embedding vectors and generating centroid or other vectors and confidence scores for use in speaker identification.

The memory 1450 may further include other biometric authentication processes 1466, which may include facial recognition, fingerprint identification, retina scanning, and/or other biometric processing for a particular implementation. The other biometric authentication processes 1466 may include feature extraction processes, on or more neural networks, statistical analysis modules, and/or other processes. In some embodiments, the user verification controls 1462 may process confidence scores or other information from the voice authentication process 1464 and/or one or more other biometric authentication processes 1466 to generate the speaker identification determination. In some embodiments, the other biometric authentication processes 1466 include a neural network trained through a process using a batch of biometric input data and a rhino loss function as described herein.

The memory 1450 includes program logic for instructing processing components 1430 to perform image processing 1456, including object detection 1456, on images received through one or more components (e.g., other sensors/components 1445 such as image capture components, communications components 1448, etc.).

In various embodiments, the authentication device 1420 may operate in communication with one or more servers across a network 1480. For example, a neural network server 1490 includes processing components and program logic configured to train neural networks (e.g., neural network training module 1492), for use in speaker verification as described herein. In some embodiments, a database 1494 stores training data 1496, including training datasets and validation datasets for used in training one or more neural network models. Trained neural networks 1498 may also be stored in the database 1494 for downloading to one or more runtime environments, for use in the voice authentication processes 1464. The trained neural networks 1498 may also be provided to the one or more verification servers 1482, which provide cloud or other networked speaker identification services. For example, the verification server 1482 may receive biometric data from an authentication device 1420, such as voice data or other biometric data, and upload data to the verification server 1482 for further processing. The uploaded data may include a received audio sample, extracted features, embedding vectors, and/or other data. The verification server 1482, through a biometric authentication process 1484 that includes one or more neural networks (e.g., trained neural network 1488 stored in a database 1486) trained in accordance with the present disclosure, and system and/or user data 1489 to compare the sample against known authentication factors and/or user identifiers to determine whether the user 1444 has been verified. In various embodiments, the verification server 1482 may be implemented to provide authentication for a financial service or transaction, access to a cloud or other online system, cloud or network authentication services for used with an electronic system 1410, etc.

Where applicable, various embodiments provided by the present disclosure may be implemented using hardware, software, or combinations of hardware and software. Also, where applicable, the various hardware components and/or software components set forth herein may be combined into composite components comprising software, hardware, and/or both without departing from the scope of the present disclosure. Where applicable, the various hardware components and/or software components set forth herein may be separated into sub-components comprising software, hardware, or both without departing from the scope of the present disclosure. In addition, where applicable, it is contemplated that software components may be implemented as hardware components and vice versa.

Software, in accordance with the present disclosure, such as program code and/or data, may be stored on one or more computer readable mediums. It is also contemplated that software identified herein may be implemented using one or more general purpose or specific purpose computers and/or computer systems, networked and/or otherwise. Where applicable, the ordering of various steps described herein may be changed, combined into composite steps, and/or separated into sub-steps to provide features described herein.

The foregoing disclosure is not intended to limit the present disclosure to the precise forms or particular fields of use disclosed. As such, it is contemplated that various alternate embodiments and/or modifications to the present disclosure, whether explicitly described or implied herein, are possible in light of the disclosure. Having thus described embodiments of the present disclosure, persons of ordinary skill in the art will recognize that changes may be made in form and detail without departing from the scope of the present disclosure. Thus, the present disclosure is limited only by the claims.

Claims

1. A method comprising: receiving a training batch of data samples comprising a plurality of labeled classifications;extracting features from the data sample to generate a batch of features;processing the batch of features using a neural network to generate one or more classifications for each data sample;computing a rhino loss value for the training batch; andmodifying weights of the neural network to reduce the rhino loss value.

2. The method of claim 1, wherein the training batch includes a plurality of speech utterances and computing the rhino loss value further comprises generating the rhino loss value for a plurality of speakers.

3. The method of claim 1, wherein processing the batch of features using a neural network to generate one or more classifications for each data sample, further comprises identifying one or more objects in each sample with a single-classification per object.

4. The method of claim 1, wherein the training batch comprises a plurality of audio samples comprising a first number of speakers and a second number of audio samples per speaker.

5. The method of claim 4, wherein the classification comprises phoneme recognition in a stream of audio samples.

6. The method of claim 1, further comprising a speaker authentication process comprising: receiving a target audio signal comprising speech from a target speaker; extracting target features from the target audio signal;processing the target features through the neural network to generate one or more user classifications; anddetermining whether the target speaker is associated with a user identifier based at least in part on the one or more user classifications;wherein determining whether the target speaker is associated with a user identifier comprises calculating a confidence score measuring a strength of a classification determination.

7. The method of claim 1, wherein the training batch comprise a plurality of images including object classification labels.

8. The method of claim 7, wherein processing the batch of features using a neural network to generate one or more classifications for each data sample comprises producing an object detection classification activation in one grid cell determined responsible for detecting the classified object.

9. The method of claim 7, wherein computing the rhino loss value further comprises generating the rhino loss value for a plurality of object classifications.

10. The method of claim 7, wherein processing the batch of features using a neural network to generate one or more classifications for each data sample comprises detecting and localizing an object in an image with one bounding box per object using a single-stage object detector.

11. A system comprising: a logic device configured to train a neural network using a rhino loss function, the logic device configured to execute logic comprising: receiving a training batch of labeled data samples;extracting features from the data samples to generate a batch of features;processing the batch of features using a neural network to generate classifications configured to classify the data samples;computing a rhino loss value for the training batch based, at least in part, on the classifications; andmodifying weights of the neural network to reduce the rhino loss value.

12. The system of claim 11, wherein computing the rhino loss value further comprises calculating the rhino loss value for a plurality of speakers, based at least in part on the classifications.

13. The system of claim 14, wherein processing the batch of features using a neural network to generate one or more classifications for each data sample, further comprises identifying one or more objects in each sample with a single-classification per object.

14. The system of claim 11, wherein the logic device is further configured to execute logic comprising a backbone network comprising a pre-trained image classifier configured to learn how to extract features from the image.

15. The system of claim 11, wherein the logic device is further configured to execute logic comprising a backbone network configured for phoneme recognition; wherein each output is a confidence score for a probability of detecting a class and it is obtained after applying a sigmoid function.

16. A system comprising: a logic device configured to train a neural network for a classification task by executing logic comprising: receiving a training dataset comprising labeled training data samples;pre-training a backbone architecture as a classifier using the training dataset;extracting feature maps from an intermediate layer of the backbone architecture; andidentifying a portion of each data sample that relates to the extracted feature maps.

17. The system of claim 16, wherein the training dataset comprises a plurality of images and wherein the logic device is further configured to execute logic comprising subdividing each image into a plurality grid cells and identifying which of the plurality of grid cells relates to a center of a bounding box annotation for the image.

18. The system of claim 17, wherein the image includes a plurality of objects, and wherein the logic device is further configured to execute logic comprising generating a single activation for each of the detected objects.

19. The system of claim 16, wherein the training dataset comprises a plurality of audio samples comprising a plurality of frames, and wherein the logic device is further configured to execute logic comprising identifying a phenome by identifying frames that relate to a phenome activation.

20. The system of claim 16, wherein identifying a portion of each data sample that relates to the extracted feature maps comprises using a neural network to generate one or more classifications for each data sample comprises detecting and localizing an object in an image with one bounding box per object using a single-stage object detector.