Patent Application
20140169688
In many scenarios, it is as important to analyze images and perform object detection quickly as it is to perform object detection rapidly. A common application where it is desirable to perform object detection both rapidly and accurately is face detection. For instance, many mobile computing devices include cameras, wherein it is desirable to quickly locate faces in frames captured by a camera (e.g., for the purposes of auto-focusing).
Conventional techniques for analyzing images and detecting certain objects therein, however, remain relatively slow. Specifically, detection time for most detection systems is best measured in seconds per frame, rather than frames per second.
Speed at which objects can be detected by in images has been improved in object detection systems through utilization of cascaded detectors/processing. Cascaded detectors refer to a sequence of detectors, wherein accuracy of a detector in the cascaded sequence of detectors increases as position in the sequence advances, while computation time to execute a detector in the cascaded sequence of detectors likewise increases as position in the sequence increases. Accordingly, a first detector in the cascaded sequence of detectors is the fastest and least accurate detector, while the last detector in the sequence is the slowest and most accurate detector. Cascaded processing refers to a similar approach, where processing is undertaken in stages over respective subwindows of an image, and wherein subwindows are rejected as candidates during such stages. Accordingly, a subwindow rejected during a first stage will have a relatively small amount of processing undertaken thereover, while a subwindow rejected during a later stage will have been analyzed with a relatively large amount of processing undertaken thereover.
When performing object detection in an image, it is desirable to minimize an amount of processing required to identify an object, without sacrificing accuracy. Thus, in conventional staged detection systems, a first detector in the cascaded sequence of detectors (or a first process executed over a subwindow in a cascade of processes) is tasked with analyzing each subwindow rejecting subwindows that certainly do not include the object of interest, and doing so relatively quickly. Subsequently, a second detector in the cascaded sequence of detectors (or a second process in a cascade of processes) is executed over all subwindows not rejected in earlier stages; this process continues until all subwindows in the image have been rejected (and thus, the object does not exist in the image) or until the last detector in the cascaded sequence of detectors (or the last process in the cascade of processes) indicates that one or more subwindows in the image include the object of interest. While the process set forth above has increased speed in which object identification can be undertaken, the process remains relatively slow, due to the large number of subwindows in a given image (e.g., several thousand).
The following is a brief summary of subject matter that is described in greater detail herein. This summary is not intended to be limiting as to the scope of the claims.
Described herein are various technologies pertaining to detecting specified objects in images. In an example, an object that is desirably detected in images may be a human face or pedestrian, although the technologies described herein are not so limited. Object detection can accomplished relatively quickly through employment of crosstalk cascades, which refers to a combination of and communication between a plurality of object detection cascades. A cascade, as the term is used herein, generally refers to the employment of cascaded processing over subwindows of an image over several stages of a multi-stage object detection approach. The cascaded processing is undertaken to iteratively reduce candidate subwindows in an image, wherein a candidate subwindow is a subwindow that may include an object desirably detected. A crosstalk cascade includes an excitatory cascade, a soft cascade, and an inhibitory cascade.
In operation, an image is received and a plurality of subwindows of the image are defined, wherein subwindows may overlap, may be of varying sizes, and may be of varying shapes. Accordingly, a single image may include tens of thousands, hundreds of thousands, or even millions of subwindows. Features of the image for each subwindow may then be extracted utilizing a relatively lightweight feature extraction algorithm, wherein such features of respective subwindows are analyzed iteratively during cascaded processing of the subwindow in connection with ascertain whether the subwindow can be rejected as a candidate.
In an excitatory cascade, at a first stage of the object detection process, first processing can be undertaken over a sampled (sparse) set of subwindows (e.g., one of every X subwindows can be selected). In an exemplary embodiment, each stage of the object detection process can use a detector that is independent of detectors utilized in previous stages of the object detection process. In another embodiment, a boosted classifier that comprises K weak classifiers can be employed, wherein a kth stage of the object detection process corresponds to aggregated output of k weak classifiers. Other embodiments are also contemplated. Thus, cascaded processing can refer to any form of progressive processing that may be undertaken over a subwindow. If a score assigned to a subwindow from the sampled set from the first processing is below a excitatory cascade threshold score for the first stage, then the subwindow and other subwindows in a predefined neighborhood of the subwindow (e.g., 7 pixels in an X direction, 7 pixels in a Y direction, and 4 pixels in scale) are rejected. Thus, for example, several subwindows can be rejected as candidate subwindows based upon a score assigned to a single subwindow.
Thereafter, in a soft cascade, the first processing can be executed over subwindows identified as candidates from the excitatory cascade (excluding, optionally, subwindows over which the first processing has been undertaken in the excitatory cascade to reduce computation). For each subwindow, its respective score is compared with a pre-defined soft cascade threshold score for the first stage, and if a score is below the threshold, then the subwindow is rejected (resulting in a reduction in candidate subwindows).
Subsequently, in an inhibitory cascade, for each remaining candidate subwindow (every subwindow not rejected during the excitatory cascade or the soft cascade in the first stage), its score is compared with scores assigned to candidate subwindows in a predefined neighborhood of the subwindow. If the ratio of the score of the subwindow to a score of any of the subwindows in the predefined neighborhood is below an inhibitory cascade threshold for the first stage, then the subwindow is rejected. Thus, if the neighbor is getting a much higher response as including the object that is desirably detected, then further processing on the subwindow is not necessary.
As noted above, a crosstalk cascade is the combination of excitatory, soft, and inhibitory cascades, wherein the output of at least one of such cascades is dependent upon the output of another one of the cascades in each stage of a multi-stage object detection process. After the first stage of the detection process is completed, the second stage is commenced and the next processing (e.g., the next detector in the cascaded sequence of detectors or the next classifier in the k weak classifiers) is selected. The second processing is then undertaken over remaining candidate subwindows in the manner described above (excitatory cascade, soft cascade, and inhibitory cascade are executed at each stage with different thresholds, depending on the stage). The employment of these cascades, in the manner set forth above, greatly increases computational efficiency such that object detection can be undertaken quickly and accurately.
Other aspects will be appreciated upon reading and understanding the attached figures and description.
Various technologies pertaining to object detection will now be described with reference to the drawings, where like reference numerals represent like elements throughout. In addition, several functional block diagrams of exemplary systems are illustrated and described herein for purposes of explanation; however, it is to be understood that functionality that is described as being carried out by certain system components may be performed by multiple components. Similarly, for instance, a component may be configured to perform functionality that is described as being carried out by multiple components. Additionally, as used herein, the term “exemplary” is intended to mean serving as an illustration or example of something, and is not intended to indicate a preference.
As used herein, the terms “component” and “system” are intended to encompass computer-readable data storage that is configured with computer-executable instructions that cause certain functionality to be performed when executed by a processor. The computer-executable instructions may include a routine, a function, or the like. It is also to be understood that a component or system may be localized on a single device or distributed across several devices.
With reference now to
The system 100 includes a receiver component 102 that receives an image, wherein the image may be a digital image received from a camera, a digital image received from a data repository that comprises images, etc. The system 100 further comprises an object detector system 104 that is in communication with the receiver component 102. The object detector system 104 is configured to relatively quickly detect a specified object, such as a face or pedestrian, in the image received by the image receiver component 102. As will be described in greater detail below, the object detector system 104 employs a crosstalk cascade, which itself includes plurality of different types of cascades, in connection with detecting the object in the image.
The object detector system 104 includes a feature generator component 106 that receives the image and extracts a plurality of features therefrom. In an exemplary embodiment, the feature generator component 106 can comprise a relatively lightweight algorithm that quickly extracts features from the image that can be used to detect the object of interest. The particular features that can be employed will depend upon the object that is desirably detected, and can be learned at a pre-processing stage.
The object detector system 104 additionally comprises a subwindow definer component 108 that defines numerous subwindows in the image, such that the features extracted by the feature generator component 106 are associated with subwindows defined by the subwindow definer component 108. The subwindow definer component 108 can define subwindows of the image as being of different shapes, sizes, overlapping, etc. Accordingly, the image may include several thousand subwindows, tens of thousands of subwindows, hundreds of thousands of subwindows, millions of subwindows, and so on.
The object detector system 104 additionally comprises a cascaded classifier 110, which is configured to generate scores for subwindows of images that are indicative of likelihood/probability of presence of the object in the respective subwindows in multiple stages of an image detection process. While the cascaded classifier 110 will be described below as a classifier that comprises multiple, weak classifiers, it is to be understood that other embodiments are contemplated. For example, the cascaded classifier 110 can comprise a plurality of independent classifiers that are arranged in a sequence, the classifiers becoming increasingly accurate (and computationally expensive) as the sequence progresses.
In an exemplary embodiment, the cascaded classifier 110 may be a boosted classifier H that comprises K weak classifiers and has the following form:
H(x)=HK(x)=Σi=1Kαihi(x), (1)
where each hi is a weak classifier (with output −1 or 1), αi is its associated weights, and x is a subwindow of the image being analyzed by the cascaded classifier 110. x can be classified as positive (as being a candidate for including the object in the image) if H(x)>0 and H(x) serves as the confidence. Accordingly, a sequence of score functions Hk(x) can be defined for k<K in an analogous manner.
The object detector system 104 further comprises a stage selector component 112 that increments/selects a stage (i) corresponding to the cascaded detector. In an example, a first stage selected by the stage selector component 112 is associated with a first weak classifier of the cascaded classifier 110, which performs a relatively small amount of processing on any given subwindow (with a relatively low level of accuracy). A later stage in the multi-stage object detection process aggregates output of previous stages and executes causes a next week classifier in the cascaded classifier 110 to be executed (and the output aggregated with output of previous stages), thereby involving further computation/processing but a higher level of certainty with respect to the output. Initially, the stage selector component 112 selects the first stage, and thus the first, baseline weak classifier, such that the first, baseline weak classifier of the cascaded detector 110 initially is executed over subwindows of the image defined by the subwindow definer component 108.
The object detector system 104 further comprises a crosstalk cascade component 113, which utilizes multiple cascades, at least some of which are in communication with one another, to reduce a number of candidate subwindows (subwindows that are candidates as including the object that is desirably detected) over stages of the multi-stage detection process. The crosstalk cascade component 113 comprises an excitatory cascade component 114, a soft cascade component 116 (also known as a rejection cascade), and an inhibitory cascade component 118, at each stage of the multi-stage detection process to reduce candidate subwindows. Operation of the crosstalk cascade component 113 is now provided.
Initially, at the first stage, the first weak classifier of the cascaded classifier 110 is executed over a (sparsely) sampled set of subwindows of the image received by the receiver component 102. For instance, the first weak classifier can be executed over one of every set number (such as nine) of subwindows in the image. The first weak classifier h1(x) is executed over the sampled subwindows, and the first weak classifier outputs a score for each subwindow in the sampled subwindows. If the score for a subwindow output by the first weak classifier is below a predefined excitatory cascade threshold score for the first stage, then the subwindow and all subwindows in a predefined neighborhood of the subwindow are rejected by the excitatory cascade component 114. Otherwise, the subwindow in the sampled subwindows and its neighbors are identified as being candidate subwindows, and are retained for further processing. In an exemplary embodiment, the neighborhood may be seven pixels in the X direction, seven pixels in the Y direction, and three pixels in scale, and the step size may be three pixels by three pixels by one pixel. It is to be understood that the size of the neighborhood may depend upon the object that is desirably detected, and can be selected/tuned.
Still referring to the first stage, the first weak classifier can be utilized to generate a score for each candidate subwindow (subwindows not rejected by the excitatory cascade component 114). To reduce computation, scores for subwindows in the sampled set can be cached by the excitatory cascade component 114, such that recomputation of such scores is unnecessary. The soft cascade analyzer component 116 can compare the scores to a predefined soft cascade threshold score for the first stage, and can reject subwindows that respective have scores below the predefined soft cascade threshold score.
The object detector system 104 also comprises an inhibitory cascade component 118. The inhibitory cascade component 118 compares, for each subwindow, the score assigned to a respective subwindow with scores assigned to all candidate subwindows in the predefined neighborhood of the subwindow. Specifically, in an exemplary embodiment, the inhibitory cascade component 118 can compute a ratio between the score of the subwindow and the score of the subwindow in its predefined neighborhood, and can reject the subwindow if the ratio is below a predefined inhibitory cascade threshold for the first stage. In other words, based upon the ratio, the inhibitory cascade component 118 can determine that a subwindow in the neighborhood of the subwindow being analyzed is much more likely to include the object desirably detected than the subwindow.
After the first stage has been completed, the stage selector component 112 selects the second stage, such that the second weak classifier in the cascaded classifier 110 executes over remaining candidate subwindows (subwindows not rejected in the first stage). The excitatory cascade component 114, the soft cascade component 116, and the inhibitory cascade component 118 perform the actions described above at the second stage (using thresholds for the second stage), and at each subsequent stage, until there are no remaining candidates (the object does not exist in the image) or until all weak classifiers of the cascaded classifier 110 have executed, in which case remaining candidate subwindows can be identified as including the object.
Additional details pertaining to the excitatory cascade component 114, the soft cascade component 116, and the inhibitory cascade component 118 are now provided. As noted above, the excitatory cascade component 114 executes an excitatory cascade at each stage of a multi-stage object detection process. The goal of an excitatory cascade is to generate a set of candidate subwindows Xc that comprise all quasi-positive subwindows. Xg can denote the samples set of all subwindows x not rejected in previous stages, wherein, in an exemplary embodiment, the sampling is undertaken with a step size half the defined size of a neighborhood . For instance, if
=7×7×3, the step size is (3, 3, 1), meaning that one in nine x is in Xg. At each stage k, initially Xc=0. For each x ∈ Xg, and for each stage k≦k*, a determination is made regarding Hk(x)>θkE, where θkE is the excitation cascade threshold for stage k. k* is the maximum stage where the excitation component 114 is to execute; in stages above k*, the excitation component 114 is idle. If Hk(x)>θkE, then x and all subwindows in the neighborhood of x (all x′ ∈
(x)) are added to the set of candidate subwindows Xc.
Again at each stage, the soft cascade component 116 can receive Xc from the excitatory cascade component 114. For each x in Xc, Hk(x) can be computed. It is to be ascertained that Hk(x) previously computed in stage k for the excitatory cascade component 114 can be cached to avoid recomputation. Hk(x) is then compared with a soft cascade threshold for stage k, θkR. If the soft cascade component determines that Hk(x)<θkR, then x is rejected as a candidate (e.g., x is removed from Xc).
The inhibitory cascade component 118, at each stage, can receive Xc from the soft cascade component 116, and can compute the ratio of the score of x with respective scores assigned to subwindows x′ in the neighborhood of x. Formally, the inhibitory cascade component 118 can, for each subwindow x in Xc, compute the ratio Hk(x)/Hk(x′), for x′ ∈(x) in Xc, and can compare such ratio with an inhibitory cascade threshold θkl for stage k. If
then x is removed from Xc. Subsequently, if Xc is not empty, and stage k is not the last stage in the multi-stage process, the stage selector component 112 increments the stage. The excitatory cascade component 114 is provided with Xc, samples from Xc, and generates an updated Xc based upon the next weak classifier. If stage k is the last stage, and Xc is not empty, x in Xc are output as subwindows that include the object of interest. If Xc is empty, regardless of whether stage k is the last stage, then the crosstalk cascade component 113 can output an indication that the object does not exist in the image.
While the soft cascade component 116 is described as receiving input from the excitatory cascade component 114 and providing output to the inhibitory cascade component 118, it is to be understood that the inhibitory cascade component 118 can receive input from the excitatory cascade component 114, and can provide its output to the soft cascade component 116.
In summary, the crosstalk cascade component 113 combines the excitatory cascade component 114, the soft cascade component 116, and the inhibitory cascade component 118. The excitatory cascade component 114 generally reduces computation for small k, the inhibitory cascade component 118 reduces computation for large k, and the soft cascade component 116 reduces computation across all k. Combining the excitatory cascade component 114 with the inhibitory cascade component 118 includes provision of the sparse set of candidates Xc from the excitatory cascade component 114 as input to the inhibitory cascade component 118 (or the soft cascade component 116).
With reference now to
Now referring to
The soft cascade component 116 can compare each score for the subwindows 204-220 with the soft cascade threshold for stage k. For each subwindow where the score falls below the aforementioned threshold, the soft cascade component 116 can reject such subwindow, and remove the subwindow from the set of candidate subwindows. Here, for example, the soft cascade component 116 can remove subwindows 204 and 218 from the set of candidate subwindows, as their scores are below the threshold.
Turning now to
With reference now to
The system 500 comprises a soft cascade threshold setter component 506 that sets soft cascade thresholds for each stage of the multi-stage object detection process. An exemplary technique that can be undertaken by the soft cascade threshold setter component 506 is described below; it is to be understood, however, that other techniques are contemplated and are intended to fall under the scope of the hereto-appended claims. The soft cascade threshold setter component 506 can employ an unsupervised approach in connection with learning θkR, based upon the cascaded classifier 110 H, quasi-positives xi, and a target quasi miss rate (QMR) γ, which is the fraction of quasi-positives with Hk(x)>θ* rejected by a cascade, where θ* is less than or equal to zero (e.g., θ*=−1).
First, γ′=1−(1−γ)1/K. If at each stage k the QMR is ≦γ′, the overall QMR of the soft cascade with be ≦γ. X1 can denote the set of quasi-positives, and 1={H1(xi)|x1 ∈ X1}. The rejection threshold for the first stage can be set as follows:
θ1R=[1]r−∈, where r=γ′·|
1| (2)
Here []r denotes the rth order statistic of
(e.g., the rth smallest value in
), and ∈=10−5. For each remaining stage 1<k<K, Xk={xi ∈ Xk−1|Hk−1(xi)>θk−1R} and
k={Hk(xi)|xi ∈ Xk}, and the following can be computed as the soft cascade threshold for stage k:
θkR=[k]r−∈, where r=└γ′·|
k|┘ (3)
When the soft cascade component 116 is configured with the soft cascade threshold for stages of the multi-stage process, the QMR is at most γ. θkR can be further optimized by performing a second pass with a target QMR γ=0 and XK+1 as the initial set of quasi-positives.
The system 500 additionally comprises an excitatory cascade threshold setter component 508 that sets excitatory cascade thresholds for respective stages of the multi-stage object detection process. The excitatory cascade threshold setter component 508 learns such thresholds, in an exemplary embodiment, in an unsupervised manner. Additionally, the maximum excitation stage k* can be learned in an unsupervised manner. The thresholds θkE and the maximum excitation stage k* can be learned given H, xi, and target QMR γ. For purposes of explanation, only the two-dimensional case (single scale) is described, and θkE is initially learned for a single excitation stage k*=k. Xgo denotes a sampling grid offset by o relative to xi. Given a step size (sx,sy), there are sxsy possible offsets o for Xgo (including one case where xi ∈ Xgo). Every sampling grid Xgo can be treated as a separate, equally likely possibility.
hio can be the maximum score Hk(x′i) of all the x′i ∈(xi) ∩ Xgo. In an exemplary embodiment, as noted above, the step size can be half the size of
, such that, for example, at least n≧4 neighbors of xi will be in Xgo regardless of the offset o; however, the number of neighbors of xi may be fewer. Subsequently, if all the x′i ∈
(xi) ∩ Xgo were rejected by the soft cascade, then hio=−∞, otherwise hio=max(Hk(x′i)). Then, the following can be computed:
θkE=[k]r−∈, where r=└γ·|
k|┘ and
k={hio}. (4)
If θkE≠−∞, then by construction on average 1−γ quasi-positives will end up on Xc. Unfortunately, θkE=−∞ implies that no such threshold exists. The above procedure fails if the excitation stage is too late. It can be recognized that a region of support (ROS) of the cascaded classifier 110 decreased with increasing k while its discriminative power increases. This causes a tension between performing excitation early, when discriminability is low, and performing excitation late when the ROS is small. Accordingly, the last valid excitation can be identified by starting with k=K and working backwards until θkE valid. Finally, θkE, for earlier stages, can be set to the smallest value such that |Xc| does not increase.
The system 500 further comprises an inhibitory cascade threshold setter component 510 that sets thresholds to be utilized by the inhibitory cascade component 118 for respective stages of the multi-stage detection process. The inhibitory cascade threshold setter component 510 can set such thresholds θkI as follows, given H, xi, and target QMR γ. The procedure is somewhat similar to the approach undertaken by the soft cascade threshold setter component 506 described above. First, the following can be defined: Rk(x)={Hk(x)/H)k(x′)}. Further, y′=1−(1−γ)1/K, X′k denotes the set of surviving quasi-positives in stage k, and
k={Rk(xi)|xi ∈ X′k}. Thereafter, θkI can be set through employment of the following algorithm:
θkI=min{1,[k]r/(1+∈)}, where r=└γ′·|
k|┘. (5)
X′1 comprises all quasi-positives, and X′k={xi ∈ X′k−1|Rk−1(xi)>θk−1I}. An inhibitory cascade with θkI defined above has a QMR of at most γ.
With reference now to
Moreover, the acts described herein may be computer-executable instructions that can be implemented by one or more processors and/or stored on a computer-readable medium or media. The computer-executable instructions may include a routine, a sub-routine, programs, a thread of execution, and/or the like. Still further, results of acts of the methodologies may be stored in a computer-readable medium, displayed on a display device, and/or the like. The computer-readable medium may be any suitable computer-readable storage device, such as memory, hard drive, CD, DVD, flash drive, or the like. As used herein, the term “computer-readable medium” is not intended to encompass a propagating signal.
Turning now to
At 608, the object in the image is identified based at least in part upon the rejecting of the subwindow. For example, when all classifiers in the cascaded classifier have executed, the cascaded classifier can identify subwindows that include the object. The methodology 600 completes at 610.
Turning now to
At 708, subwindows of the image in a candidate set of subwindows (subwindows not previously rejected) are sampled. For instance, a sparse sampling of subwindows can be selected at 708. At 710, a weak classifier corresponding to the current stage is executed over each subwindow sampled at 708. Accordingly, each sampled subwindow is assigned a score by the weak classifier. For subsequent stages, the score output by a respective weak classifier for such stages is aggregated with previous scores generated by other weak classifiers. At 712, a determination is made regarding whether the score computed at 710 is less than an excitatory cascade threshold for the current stage. If it is determined that the score is less than this threshold, then at 714, the subwindow and subwindows in a predefined neighborhood of the subwindow are rejected. If the score greater than the aforementioned threshold, then the subwindow and subwindows in the neighborhood are retained in a candidate set.
Thereafter, at 716, the weak classifier corresponding to the current stage is executed over subwindows in the candidate set. Accordingly, each subwindow in the candidate set will have a respective score assigned thereto for the stage corresponding to the weak classifier.
At 718, for each subwindow, a determination is made regarding whether the score computed at 716 for a respective subwindow is less than a soft cascade threshold for such stage. If, for the respective subwindow, the score is less than such threshold, then at 720, the subwindow is rejected. Otherwise, the respective subwindow is retained in the candidate set.
The methodology 700 then continues to 722, where an inhibitory score is computed for each subwindow in the candidate set. As described above, such inhibitory score is indicative of a score for a subwindow relative to a score for a subwindow in its predefined neighborhood. At 724, for each subwindow in the candidate set, a determination is made regarding whether the inhibitory score for a respective subwindow is below an inhibitory cascade threshold. If the inhibitory score for the subwindow is less than such threshold, then at 726 the subwindow is rejected. Otherwise, the subwindow is retained as a candidate for including the object that is desirably detected.
Thereafter, at 728, a determination is made regarding whether the current stage is the final stage in the object detection process. If it is not the final stage, then the methodology 700 returns to 706, where the stage is incremented, and the methodology 700 proceeds as described above. If it is the last stage, then all candidate subwindows (subwindows not rejected) are indicated as including the object that is desirably detected at 730. The methodology 700 completes at 732.
Now referring to
The computing device 800 additionally includes a data store 808 that is accessible by the processor 802 by way of the system bus 806. The data store may be or include any suitable computer-readable storage, including a hard disk, memory, etc. The data store 808 may include executable instructions, images, etc. The computing device 800 also includes an input interface 810 that allows external devices to communicate with the computing device 800. For instance, the input interface 810 may be used to receive instructions from an external computer device, from a user, etc. The computing device 800 also includes an output interface 812 that interfaces the computing device 800 with one or more external devices. For example, the computing device 800 may display text, images, etc. by way of the output interface 812.
Additionally, while illustrated as a single system, it is to be understood that the computing device 800 may be a distributed system. Thus, for instance, several devices may be in communication by way of a network connection and may collectively perform tasks described as being performed by the computing device 800.
It is noted that several examples have been provided for purposes of explanation. These examples are not to be construed as limiting the hereto-appended claims. Additionally, it may be recognized that the examples provided herein may be permutated while still falling under the scope of the claims.
