Patent Application
20070098220
The present application is based on, and claims priority from, Great Britain Application Serial Number 0522218.7, filed Oct. 31, 2005, the disclosure of which is hereby incorporated by reference herein in its entirety.
The present invention relates to a method of triggering a detector to detect a moving feature within a video stream. It finds particular although not exclusive application in the field of face detection and tracking.
There has been much recent work on optimizing face detection technology for real-time operation. Although the increasing speed and accuracy of face detectors can now provide real-time or near real-time performance on a powerful computing platform such as a PC, this cannot easily be translated to smaller devices such as PDAs or digital cameras.
Since face and other moving feature detectors are often too slow, with current technology, to be operated in real-time, it is convenient to provide some sort of trigger, generated from a real-time video stream, which notifies the detector of possible hits, which the detector can the investigate in more detail. Typically, the trigger may generate a large number of “false alarms”, but that is not a serious problem unless the number of triggers being generated is so large that the detector is unable to keep up. Since the detector will not start us unless triggered, it is more important that the triggering mechanism does not miss interesting frames which merit more detailed investigation. A trigger of this type is sometimes known as a “preattemptive trigger”.
Current preattemptive trigger systems may conveniently be divided into two categories, namely those that are based on the “blind” trigger, and those that are based on a “local trigger”.
A “blind” trigger based only on then maximum frame rate of the face or feature detector. That is, whenever the face detector has concluded a pass on the image, the current frame is selected for processing.
A “local” trigger depends on the tracking mechanism. In this case the tracker isolates a localised patch which results from tracking a previously detected face or other feature. The patch is periodically sent to the face detector for confirmation and re-location of the face, but the decision to request confirmation depends only on the level of certainty in the tracking algorithm.
A specific example of the “local” trigger system, is described in:
U.S. patent US20040186816 A in which an object tracking algorithm (such as a Kalmann filter) is initialized using an object detector. After initialization the object detector is only triggered if the tracker fails, at which point the object detector is run on the whole frame to reinitialize the tracking algorithm.
None of the main systems are in practice particularly satisfactory, the face or object detector at an inappropriate point.
If inappropriate frames or locations are chosen by the triggering system, then the face detector may fail to find present faces and the face tracking performance will deteriorate.
According to the present invention there is provided a method of trigger a detector to detect a moving feature within a video stream the method comprising:
The characteristic which is used to trigger the detector may be any convenient local measure of the sequence, and may be a probabilistic measure. In one embodiment, the characteristic (distinctive pattern) may be a local extremum, such as a maximum or a minimum. Preferably, the detector may be triggered when the sequence of values is determined to have peaked or troughed.
The invention further extends to a computer program carrying out such a method, and also to a computer readable media and to an electronic signal holding or representative of such a computer program.
In the preferred embodiment, the method is used to trigger a sub real-time face detector algorithm. Typically, face detectors/trackers make use of complex algorithms which the present technology is not always operable in real-time. The present invention conserves resources by triggering or authorising the face detector to run only when it is most likely to be successful, in other words when a fast operating preattemptive trigger has determined that there is likely to be a full frontal face in the frame. In a typical stretch of video including a head turning sequence, prior art techniques may often fail whereas the present invention succeeds by triggering the face detector to operate only on the most advantageous frame, namely the frame containing a full frontal face.
In a preferred embodiment, by simultaneously monitoring some specific measures of a video stream, we can determine both specific frames and sub frame locations likely to give the highest hit rate for the desired feature detector (for example for a frontal face detector). This enhances the usefulness of sub real-time feature detectors in real-time systems.
The invention may be carried into practice in a number of ways and one specific embodiment will now be described, by way of example, with reference to the accompanying drawings, in which:
The preferred embodiment is concerned with providing a pre-attemptive trigger for a face detector/tracker, for example within a digital camera or video system.
As shown schematically in
As the video frames are captured, one by one in real-time, each frame is analysed by a rapid software procedure which provides a measure of the symmetry of the image. This measure of symmetry is shown by the line 18 in
It will of course be understood that for a human or animal face, the level of symmetry is greatest at the full face position 16, when α equals 0. In operation, the value of the symmetry measure is monitored, frame by frame, and when the system determines that the measure has peaked, as shown in point 20 in the drawing, a more sophisticated face detector/tracker algorithm is triggered. The peak value of the symmetry measure may be determined in any convenient manner using a conventional peak detection mechanism. It will of course be understood that in a real-time environment it will not be possible to detect a peak until several frames after the peak has occurred, but the necessary short delay in real-time peak determination is not in practice normally significant. Once the peak has been satisfactorily been detected, the face detector is triggered, and it is instructed to commence operations on the frame 16 at which the peak has occurred.
As the face detection/tracking algorithm attempts to detect and track the facial image starting from the frame 16, in sub real-time, the real-time video continues in parallel, with the system repeatedly looking for peaks in the signal. Each time a peak is found, a new trigger is generated, and another instance of the face detection tracking algorithm started. Provided that the trigger infrequency is not so rapid that it overloads the sub real-time face detector, the overall result of running the face detector periodically only when triggered is to mimic very effectively through real-time face detection.
In a more sophisticated system, a symmetry measure may be calculated not only for each entire frame, but also for a variety of smaller locations or sub frames, each of which will generate its own symmetry measure. All of these measures are monitored simultaneously, and when a peak is detected in a given sub frame output, the face detector algorithm is triggered just for that sub frame. In other words, whenever a peak is detected, the system passes to the face detector not only an identification of the particular frame at which the peak was detected, but also the location or area within the frame. By ensuring that the face detector concentrates only on specific locations within the frame which have been identified as interesting by the pre-emptive trigger, the resources can be conserved. Another possibility which may be used alone or in conjunction with the above is to repeat the calculation of the measure of symmetry at a variety of different scales or resolutions within the image, and where appropriate also to pass scale information onto the face detector.
By ensuring that the system triggers only for full face images, we can conserve resources by instructing the face detector to operate only on frames where there is reasonable possibility of success. In the video sequence of
The algorithm used to determine the measure of symmetry 16 will typically operate on a frame by frame basis, and has to be relatively fast since it will normally need to operate in real-time. Any convenient method of determining symmetry may be used, including methods which are specially designed to identify faces such as any or all of the following: Marola, G., “On the detection of the axes of symmetry of symmetric and almost symmetric planar object”, IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 11, No. 1, January 1989; Reisfeld, D. and Yeshurun, Y. “Robust Detection of Facial Features by Generalized Symmetry, conf. on Pattern Recognition” (I:117-120); Daniel Reisfeld, Haim Wolfson, Yehezkel Yeshurun “Context-free attentional operators: the generalized symmetry transform” International Journal of Computer Vision Volume 14, Issue 2 (March 1995), pp 119-130.
Where faces are to be detected, other methods could of course be used to generate the pre-attentive triggers, for example the tracking of eye candidates frame by frame, with the static measure being the distance between the eyes. An alternative static measure could be the evolution of the candidate patches of skin within the image (this being particularly useful when the facial image may not be particularly symmetric, such as a woman having long hair which partially obscures the face). Triggering could also be designed for non-frontal faces, such as full profile, by employing appropriate static measures.
It will be understood that the present invention may also find application in other fields in which preattemptive triggers are used. For example, similar methods could be applied to detect turning cars, vehicles, or generally any moving objects that has a plane of symmetry. Both the algorithm for generating the preattemptive triggers, and the detector which runs as a result of those triggers, may be optimised in any particular application for the specific object or type of object to be detected or tract.
Even more generally, as shown in
Although in the embodiment above, it is proposed that a trigger be generated on the detection of a peak or a maximal value of the measure, other mathematical algorithms achieving similar ends could equally well be substituted, for example the detection of a trough or minimal value of the measure.
One method embodying the invention is shown schematically in
The invention may find practical application in such areas as digital video photography, where it may be useful to display on an LCD screen in near real-time, an overlaid box or other indication of the position of a face which has been identified and is being tracked. In a still digital camera system, the invention could be used to follow the movement of one or more faces within an image and to automatically trigger the shutter release at an appropriate point to ensure that a full face portrait was taken.
In practical a embodiment, the method described above may be implemented within some suitable apparatus or hardware such as a camera 40 (
| Number | Date | Country | Kind |
|---|---|---|---|
| 0522218.7 | Oct 2005 | GB | national |
