Patent Application
20170364757
This application claims the benefit under 35 U.S.C. §119(a) of European Patent Application EP 16175330.6, filed Jun. 20, 2016, the entire disclosure of which is hereby incorporated herein by reference.
This disclosure relates to image processing methods, and in particular to vehicle image processing where objects are identified from camera images, such as pedestrians, by image processing and a candidate selection process.
Automatic self-driving vehicles are being developed which include sophisticated image processing means adapted to process camera images to elicit information regarding the surrounding environment. In particular it is necessary to identify objects in the environment such as pedestrians.
Pedestrian detection is a canonical case of object detection with a significant relevance in advanced driver assistance systems (ADAS). Due to the diversity of the appearance of pedestrians including clothing, pose, and occlusion, as well as background clutter, pedestrian detection is considered as one of the most challenging tasks of image understanding. The current application relates to solving problems in pedestrian detection, but it can be also applied to other object detection problems such as traffic sign recognition (TSR), vehicle detection, animal detection and similar.
One of the fastest and most popular approaches in general object detection, specifically for pedestrian detection, uses a technique which extracts aggregated channel features (ACF) in a very efficient manner and then learns from training data a constant soft cascade for fast detection.
This methodology has been extensively studied and significantly improved by either applying filters over the channel features or extending them with new feature types.
In recent years, convolutional neural networks (CNN) have brought breakthroughs in many computer imaging tasks. Embedding CNN in system processing has been considered a standard strategy. For example, in object detection, the common practice is to generate candidate windows through an efficient approach and then use a CNN for finer classification. The candidate windows can be either category independent (e.g. for general object detection) or category specific (e.g. for pedestrian detection). In case of the latter, a detector (e.g. ACF) is always used to generate a significant reduced number of high quality proposals.
Then a CNN process evaluates each proposal by resizing the original window without reusing the features extracted by the candidate generator. Although the CNN is able to learn good features, its high computational cost (including image resizing and feature extraction from pixels) often blocks its usage for real-time applications. It is an object of the invention to overcome these drawbacks.
In one aspect of the invention is provided a method of detecting objects of interest in a vehicle image processing system. The method includes: a) capturing an image on a camera; b) providing a plurality of potential candidate windows by running a detection window at spatially different locations along said image, and repeating this at different image scaling relative to the detection window size; c) for each potential candidate window applying a candidate selection process adapted to select one or more candidates from said potential candidate windows; d) forwarding the candidates determined form step c) to a convolutional neural network (CNN) process; and e) processing the candidates to identify objects of interest, wherein the candidate input into the convolutional neural network (CNN) process have been resized by step b).
The candidate selection process may include a cascade. After step d) the process preferably does not include any further processing of the original image from step a). Preferably in step e) the candidates are not resized.
The method may include the additional step after step a) of: converting said image into one or more feature planes (channelized images) and step b) comprises providing a plurality of potential candidate windows by running a detection window at spatially different locations along said one or more of said channelized (feature plane) images, and repeating this at different channel image scaling relative to the detection window size.
Step b) may comprise for said image from step a) or for one or more channelized images (feature planes), converting said image or channelized image(s) into a set (pyramid) of scaled images, and for each of these applying a fixed size detection window at spatially different locations, to provide potential candidate windows.
The convolutional neural network process may not include a regularization layer and includes a dropout layer.
The convolutional neural network process preferably does not include a sub sampling layer.
The convolutional neural network process may not include the last two non-linearity layers and include sigmoid layers which enclose the fully connected layer.
The said object of interest may be a pedestrian.
Further features and advantages will appear more clearly on a reading of the following detailed description of the preferred embodiment, which is given by way of non-limiting example only and with reference to the accompanying drawings.
The present invention will now be described, by way of example with reference to the accompanying drawings, in which:
The following terms used will now be described and defined:
ACF Methodology and channels: ACF stands for Aggregated Channel Features. In order to increase the classification performance a common practice relies on first computing richer features over the original images. A channel is a registered map of a given input image, where the output pixels are computed from corresponding patches of input pixels. For instance, in a color image each color channel can serve as a channel, e.g. red, green and blue (RGB) channels. Other channels can be computed using linear or non-linear transformation of the given input image. A typical ACF detector uses instead of the raw RGB image (3 channels) 10 channels: 3 channels: 1 uv color space, 7 channels: 6 channels Histogram of oriented gradients (HOG)+1 channel Magnitude.
HOG: The histogram of oriented gradients (HOG) is a feature descriptor used in computer vision and image processing for the purpose of object detection. The technique counts occurrences of gradient orientation in localized portions of an image. What makes the ACF detector unique is that it does not use Haar features (differences between integral subparts computed in a rectangular area) to build weak classifiers but instead the channels are divided into 4×4 blocks and pixels in each block are summed (aggregated). This means features are single pixel lookups in the aggregated channels.
Cascade: A cascade is linear sequence of one or more or several (e.g. weak) “classifiers”. Classifiers are for example tests applied on a potential candidate (window) i.e. a particular image (which is usually a processed scaled sub image) to see if it has e.g. a characteristic of an object of interest i.e. a pedestrian. Weak classifiers are often decision stumps or trees for binary classification. The cascade may consist of several stages. For the problem of pedestrian detection, each stage can be considered a binary classification function (linear combination of weak classifiers) that is trained to reject a significant fraction of the non-pedestrians, while allowing almost all the pedestrian to pass to the next stage.
Soft cascade: The soft cascade architecture allows for monotonic accumulation of information. It trains a one stage cascade (monolithic) and is able to reject negative candidates (no pedestrians) after each weak classifier (instead of after each stage). This is done by calculating a rejection trace (every weak classifier gets a threshold).
Constant soft cascade: The constant variant of the soft cascade uses instead of a rejection trace one constant rejection threshold as soon as the confidence of a candidate falls below this threshold it will rejected. This allows for quickly calibrating the detector for a target detection rate, false positive rate or speed.
In step (b) a pyramid 2 or set of feature planes/scales is provided. There may be e.g. 27 scales per channel in an example. There are two parts to this process:
First, the image is channelized, meanings converting the image to a series of feature planes hereinafter referred to as channelized images. See above for the definition of channels.
Second, for each channel a set or “pyramid” of images are provided, where this set of the channelized images are provided with different sizes (this could be regarded as different magnifications). This is because in common techniques, the detection window may be of a fixed size, are used to spatially encapsulate the object of interest e.g. pedestrian. Thus a set of one or more channelized images of different sizes is determined for one or more original channelized image (i.e. channel).
In step c) a detection window is then run along each channelized image in the pyramid (i.e. each image produced by the processing in step b)) to provide a potential candidate windows, and on each of these potential candidate windows a candidate selection process is implemented e.g. by cascading. Reference numeral 3 represents the cascading process.
So in other words, regarding image pyramid/scales; the later processing step (in ACF) of performing cascade (generating candidates) from potential candidate windows in common methodology can only be applied to the content of a so called (fixed) detection window, i.e. a patch of the image (of a channel) with a constant size. Hence, in order to encapsulate objects such as pedestrians in an (e.g. channelized) image, the detection window is moved spatially (i.e. pixel wise) aiming at localizing the object/pedestrian in any location at different image scales (by scaling the original image).
Thus, to recap, a set of differently scaled images (similar to magnification levels) for each of the (channelized images), is computed—this is referred to as an image pyramid. A fixed size detection window is then shifted at different locations in each of the scaled (channelized) images (with respect to one or more channel sets), which can be regarded as “potential candidate windows”, and a candidate selection process is applied at each instant. In
It should be noted that alternatively, different detection window sizes can be used on the original (i.e. non-resized) channelized image of each channel. So for each detection window size, the detection window is run along the original channelized images to identify potential candidate windows, and at each instant, the cascade is applied (i.e. for multiple locations of the detection window, for each detection window size).
In the example particular, an image pyramid with three scales (7a, 7b, 7c) is created to detect the pedestrian 2. The detection window is run along successive portions of each scale (image), the candidate selection process (e.g. cascade) implemented at each stage. As shown in
So in essence in step b) a plurality of differently scaled images is effectively provided. This may be done for one or more channels.
Referring back to
At each detection window location (for each potential candidate window), cascading is performed whereby the content of the detection window (representing a possible object of interest i.e. candidate) goes through the cascade to identify actual candidates. This is performed by processes well known in the art. If the confidence of the candidate falls below the threshold of the cascade, it will be rejected, otherwise it would be accepted and passed to the next CNN stage.
The confidence of a candidate may be calculated by summing up the confidences given by each weak classifier included in the cascade. So in summary in step c) candidate for objects such as pedestrian, are processed. This may be performed by cascading. The output of step c) may be a set of one or more candidates manifested as images comprising specific portions of the images of step b) or step a) so candidate windows of different sizes.
In step d) once one or more candidates have been selected, for each candidate, newly determined windows 4 from the raw image of step a) containing the candidate are (and have to be resized either as a prerequisite or part of the CNN process. The CNN process then determines object of interest by refinement processes as known in the art. The CNN thus is used to recalculate (overwrite) the original confidences of the candidates that passed the cascade. CNN is a known techniques for this application. The reference numeral 5 represents that portion of the original image (determined from cascading) which is input to the CNN process.
CNN/convolutional layer: A CNN architecture is formed by a stack of distinct layers where an image is given to the first layer and the probability/confidence of its class (pedestrian, non-pedestrian) is given as an output. The convolutional layer is the core building block of a CNN. The layer's parameters consist of a set of learnable filters, which extend through the full depth of the input image. During training the network learns filters that activate when they see some specific type of features at some spatial position of the input image.
In a general aspect of the invention, the initial steps are similar to the procedure (e.g. steps a) b) and c) above; i.e. an ACF procedure. However, rather than input an resized candidate window from the raw image to the CNN process, the input to the CNN classification process are processed images (e.g. via ACF) e.g. from one or more of the steps a), b), and c) instead of raw image pixels, i.e. instead of portion of the original image. In order to do this the CNN process (layers) are amended appropriately. Thus in this fashion there is no need to repeat any re-sizing and/or channelization in the CNN process.
So, in other words, the input to the CNN according to one aspect are candidates (selected from each potential candidate window) which have already been channelized (i.e. have channel features), and which have been already effectively resized, by virtue of running a fixed detection window along a set (for each channel) of images of different sizes (pyramid), or as mentioned running different sized detection windows along the channelized image.
Thus the input to the CNN process means the images (candidates) do not have to be resized and/or channelized. So according to one general aspect, the original image is not used again to determine or derive any input to the CNN process. It should be noted that an advantage is that the image input to the CNN do not have to be resized. Furthermore, the method (process) does not include any further processing of the original image (channelization/resizing).
It should be noted that in some instances there may be no channelization. In this instance scaled images are determined from the original image, and the detection window run along each scaled images and at each instant the cascade (candidate selection process) performed—or as mentioned the detection window may be run along the original image, and repeated for different detection window sizes. Again at each instant the cascade process is applied.
As resizing/channelization to produce suitable candidates (i.e. via ACF) is already computed in detection stage b) c), this saves processing. By doing this, the first convolutional layer of CNN which is the most costly step is avoided. The total number of multiplications is reduced to about 1/30: from approximately 90 million for networks using an input of 128×64×3 to about 3 million using examples with an input of 16×8×10. Furthermore, as CNN takes the input from ACF scale pyramid, the resizing operation is also avoided.
In order to combine the CNN with the methodology (ACF) to provide methods according to examples of the invention, network architecture can be adapted. In the prior art, input to the CNN process may comprises RGB images. In one examples of the invention, ACF data (e.g. 10 channels) are n input to the CNN. First, the subsampling layers of the CNN may be removed so that the network can still have a sufficient depth. Subsampling is a form of non-linear down-sampling. There are several non-linear functions to implement Pooling among whose Max-Pooling is the most common one. It partitions the input image into a set of non-overlapping rectangles and, for each such sub-region, outputs the maximum. Further explanation of this can be found in references related to Convolutional neural networks.
The input to a CNN may have a size of 16×8×10, applying a subsampling would shrink this size to 8×4×10. This would not allow to have a sufficient number of convolutional layers (sufficient depth), which is needed for CNN to be able to learn more complex patterns.
Second, the regularization layers may be replaced (i.e. contrast normalization layers and batch normalization layers) by a computationally much cheaper dropout layer. There are a variety of methods that can be used to do regularization. This is used for preventing the network from “over-fitting” (i.e. contrast normalization layers and batch normalization layers). Contrast normalization will now be explained. With reference to
A dropout (layer) is a kind of regularization layer that prevents the network from overfitting. At each training stage, individual nodes are randomly “dropped out” of the net (with probability 1−p) or kept with probability p, so that a reduced network is left. Probability p is an input parameter.
Finally, the last two non-linearity layers are replaced by sigmoid layers which enclose the fully connected layer and change the objective function of the CNN from softmax and cross-entropy to square error.
To explain with regard to non-linearity layers: without a nonlinear activation function, the neural network is calculating linear combinations of values or linear combinations of lines.
One of the most common non-linear activation functions is the Rectified Linear Units (ReLU) layer ƒ(x)=max(0, x). Compared to other functions the usage of ReLU is preferable, because it results in the neural network training several times faster, without making a significant difference to generalization accuracy.
Sigmoid function is also a form of non-linear function ƒ(x)=(1+e−x)−1 shown as Eq. 1,
where, ijk is the height, width and the depth (number of channels), D is the total number of channels. This is the softmax operator. It is applied across feature channels and in a convolutional manner. Softmax can be seen as the combination of an activation function (exponential) and a normalization function. Log loss/cross-entropy is 1(x, c)=−log xc, where xc is the predicted probability of class c.
The square error is defined as Eq. 2,
E=½Σi=1n(ti−Oi)2 Eq. 2,
where E is the squared error, n is the number of the input samples, ti is the label of i-th sample and Oi is its corresponding network output. With regard to the definition of “label”, every class gets an integer number as a label. For example, pedestrians get the label 2 (10 in binary representation) and non-pedestrians get label of 1 (01). We have two neurons at the end of the network. Labels 1 (01 in binary representation) means, that the first neuron shall return 1 and second neuron return 0. The opposite happens for label 2. The output, Oi, is the output for the given input i. Oi is a real number between 0 and 1. Based on that the error E is calculated and used for training the network (backpropagation).
| Number | Date | Country | Kind |
|---|---|---|---|
| 16175330.6 | Jun 2016 | EP | regional |
