Research and development on a region extraction technique with which a moving object is detected by extracting a region of the moving object in an image from the image including an image of the moving object (hereinafter referred simply to as “moving object”) have been widely carried out. The technique of extracting the region of a moving object is a basic technique commonly used for: focus control and image quality improvement processing in a digital video camera or a digital still camera; a driving safety support system for vehicles; or a collision avoidance control or an alarm for avoiding collision which are used for a robot to avoid collision against a moving object.
Among techniques for detecting a moving object in an image, a method generally used is learning information on a shape of the moving object in advance; performing matching between the learnt information on a shape and a region in the image while scanning the image; and extracting a region with a high degree of matching as an object to be detected, as described in Non Patent Literature 1, for example.
In addition, Non Patent Literatures 2 to 5 disclose a method of tracking a moving object in a crowded scene in which occlusion is likely to occur, Unlike Non Patent Literature 1, these methods extract trajectories from video without using the information on a shape, and perform clustering on each of the trajectories based on the similarity of the trajectories, thereby allowing extracting, as classes, moving objects each having a different motion and tracking the extracted moving objects. It is effective, in particular, for tracking of an occluded object which is difficult to be extracted with the moving object extracting method in which information on a shape is used.
However, according to the method described in above-described Non Patent Literature 1, the information on a shape of a moving object is only partly visible in such a scene as in a crowded environment, and so on, in particular, in which occlusion is likely to occur, leading to decrease in the degree of matching, and thus it is difficult to detect an object to be detected.
In addition, since the focus is not on region extraction but on tracking a moving object, the technique of tracking a moving object in a crowded scene as represented by Non Patent Literatures 2 to 5 described above is based on the assumption of a scene in which part of the moving object is not occluded for a long period of time. This means that this method is available only when it is possible to track the moving object without occlusion of part of the moving object, Therefore, the technique can be used limitedly for a panoramic scene captured by a camera located in a high position, and the like.
In view of the above, one non-limiting and exemplary embodiment provides a moving object detection apparatus and a moving object detection method which solve the difficulty of a capturing position with limitation and an erroneous detection of a camera, and are capable of properly detecting and extracting a moving object even under the circumstance in which occlusion is likely to occur.
In one general aspect, the techniques disclosed here feature a moving object detection apparatus which detects a moving object in video by performing segmentation to identify an entire or partial region of the moving object in the video, the moving object detection apparatus including: an image input unit configured to receive at least three pictures included in the video; a trajectory calculating unit configured to (i) extract, from among the at least three pictures, image subsets each including at least two of the pictures, and (ii) calculate, for each of the image subsets, trajectories each of which is a trajectory of motion of a block, between the pictures included in the image subset, the block including one or more pixels and being included in each of the pictures; a subclass classification unit configured to classify, for each of the image subsets, the trajectories into subclasses, the trajectories being included in the image subset and calculated by the trajectory calculating unit; an inter-subclass similarity calculating unit configured to (i) calculate a trajectory share ratio which indicates a degree of sharing a same trajectory between two of the subclasses, and (ii) calculate a similarity between the two of the subclasses based on the trajectory share ratio; and a moving object detection unit configured to detect the moving object in the video by classifying the subclasses into classes such that the subclasses between which the similarity calculated by the inter-subclass similarity calculating unit is higher are more likely to be classified into the same class, to identify, as the region of a moving object, the blocks which correspond to the trajectories included in a same class.
These general and specific aspects may be implemented using a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium, or any combination of systems, methods, integrated circuits, computer programs, or computer-readable recording media.
Additional benefits and advantages of the disclosed embodiments will be apparent from the Specification and Drawings. The benefits and/or advantages may be individually obtained by the various embodiments and features of the Specification and Drawings, which need not all be provided in order to obtain one or more of such benefits and/or advantages.
One or more exemplary embodiments or features disclosed herein provide the moving object detection apparatus and the moving object detection method which solve the difficulty of limited capturing position and an erroneous detection of a camera, and are capable of properly detecting and extracting a moving object even under the circumstance in which occlusion is likely to occur.
These and other advantages and features will become apparent from the following description thereof taken in conjunction with the accompanying Drawings, by way of non-limiting examples of embodiments disclosed herein.
(Underlying Knowledge Forming Basis of the Present Disclosure)
In relation to the Non Patent Literatures disclosed in the Background Art section, the inventors have found the following problem.
Many on-vehicles cameras or mobile cameras are used, in many cases, for capturing images from a relatively low position. When capturing a crowded scene from a low position, in particular, a moving object positioned behind is more likely to be occluded by a moving object positioned in front, compared to a panoramic scene captured from a high position. Therefore, tracking a moving object for a long period of time is found to be difficult under the circumstance in which occlusion by guardrails or other objects is likely to occur, as shown in
In order to solve the above-described points, according to an exemplary embodiment disclosed herein, a moving object detection apparatus which detects a moving object in video by performing segmentation to identify an entire or partial region of the moving object in the video, the moving object detection apparatus includes: an image input unit configured to receive at least three pictures included in the video; a trajectory calculating unit configured to (i) extract, from among the at least three pictures, image subsets each including at least two of the pictures, and (ii) calculate, for each of the image subsets, trajectories each of which is a trajectory of motion of a block, between the pictures included in the image subset, the block including one or more pixels and being included in each of the pictures; a subclass classification unit configured to classify, for each of the image subsets, the trajectories into subclasses, the trajectories being included in the image subset and calculated by the trajectory calculating unit; an inter-subclass similarity calculating unit configured to (i) calculate a trajectory share ratio which indicates a degree of sharing a same trajectory between two of the subclasses, and (ii) calculate a similarity between the two of the subclasses based on the trajectory share ratio; and a moving object detection unit configured to detect the moving object in the video by classifying the subclasses into classes such that the subclasses between which the similarity calculated by the inter-subclass similarity calculating unit is higher are more likely to be classified into the same class, to identify, as the region of a moving object, the blocks which correspond to the trajectories included in a same class.
According to the configuration described above, trajectories are calculated for each of the image subsets. As described above, the trajectories are calculated at a short time interval, and thus it is possible to calculate dense trajectories in each of the image subsets. Such dense trajectories are classified into subclasses, and the similarity between the subclasses is calculated based on a share ratio, thereby making it possible to properly detect a moving object even when occlusion occurs. For example, even when the trajectories classified into the first subclass in the first image subset are separated into the second subclass and the third subclass in the second image subset in the next time period due to the effect of occlusion, the share ratio between the first subclass and the second subclass and the share ratio between the first subclass and the third subclass are both high. Accordingly, by performing the classification on the subclasses using the similarity between the subclasses which is calculated based on the share ratio, the first to the third subclasses are classified into the same class and detected as the same moving object. Thus, it is possible to solve the difficulty of limited capturing position and an erroneous detection of a camera, and properly detect and extract a moving object even under the circumstance in which occlusion is likely to occur.
For example, the subclass classification unit may include a subclass label assigning unit configured to (i) classify, for each of the image subsets, the trajectories which are included in the image subset and calculated by the trajectory calculating unit, into the subclasses, and (ii) assign, to each of the trajectories, a subclass label that is an identifier of a corresponding one of the subclasses into which the trajectory is classified, the inter-subclass similarity calculating unit may include a label propagation unit configured to propagate the subclass label between the subclasses by (i) calculating the trajectory share ratio which indicates a degree of sharing a trajectory indicating motion of a same block between a first subclass and a second subclass which are arbitrarily selected from among all of the subclasses, and (ii) reassigning the subclass label such that the first subclass and the second subclass between which the trajectory share ratio is higher are more likely to be reassigned with the same subclass label, and the moving object detection unit may detect the moving object in the video by classifying, into the same class, the subclasses to which the same subclass label is assigned, to identify, as the region of the moving object, the blocks which correspond to the trajectories included in the same class.
According to the configuration described above, trajectories are calculated for each of the image subsets. As described above, the trajectories are calculated at a short time interval, and thus it is possible to calculate dense trajectories in each of the image subsets. Such dense trajectories are classified into subclasses, and the subclass label is propagated between the subclasses based on the share ratio, thereby making it possible to properly detect a moving object even when occlusion occurs. For example, even when the trajectories classified into the first subclass in the first image subset are separated into the second subclass and the third subclass in the second image subset in the next time period due to the effect of occlusion, the share ratio between the first subclass and the second subclass and the share ratio between the first subclass and the third subclass are both high. The process of propagation of the subclass label allows the same subclass label to be reassigned to the first to the third subclasses, and thus the first to the third subclasses are detected as the same moving object. Thus, it is possible to solve the difficulty of limited capturing position and an erroneous detection of a camera, and properly detect and extract a moving object even under the circumstance in which occlusion is likely to occur.
More specifically, compared to the conventional trajectory clustering methods, trajectories which are temporally shorter are classified into subclasses, the subclass label obtained using, as a trajectory share ratio, the strength of temporal relation between the subclasses is propagated, and the subclass label having a temporally strong relation is updated to the same class label, thereby allowing region extraction while temporally maintaining coherence. It is therefore possible to determine a moving object which is temporarily separated into two objects as one moving object, and assign the same label, thereby producing an advantageous effect that erroneous detection is prevented in detecting a moving object, In addition, since temporally short trajectories are used, it is possible to reduce the risk that a corresponding point disappears in a scene in which occlusion frequently occurs and trajectories cannot be calculated. It is therefore possible to extract a moving object with higher density and improve the accuracy in detection. In addition, it is possible not only to detect a moving object but to extract a moving object from an image,
In addition, the trajectory calculating unit may (i) extract the image subsets from among the at least three pictures such that some of the pictures are shared between temporally adjacent image subsets, and (ii) calculate, for each of the image subsets, trajectories each of which is a trajectory of motion of a block, between the pictures included in the image subset, the block including one or more pixels and being included in the pictures, and the label propagation unit may propagate the subclass label between the subclasses by (i) determining that the trajectory share ratio between the first subclass and the second subclass is higher as a ratio of a trajectory passing a same position in the pictures shared between the first subclass and the second subclass to the trajectories included in the first subclass and the second subclass is higher, and (ii) reassigning the subclass label such that the first subclass and the second subclass between which the trajectory share ratio is higher are more likely to be reassigned with the same subclass label.
The image subsets are extracted such that pictures are shared between the temporally adjacent image subsets, With this, the trajectories for the pictures shared between temporally adjacent image subsets are the same. It is therefore possible to easily calculate the share ratio between the first subclass and the second subclass,
In addition, the label propagation unit may further propagate the subclass label between the subclasses by performing reassigning such that the same subclass label as the subclass label assigned by the subclass label assigning unit is more likely to be reassigned to the subclass which is higher in reliability that is a value corresponding to the number of trajectories included in the subclass or a spatial size of a region including trajectories included in the subclass.
It is possible to perform segmentation having temporal coherence with higher accuracy, by using the subclass reliability. It is therefore possible to correctly perform region extraction on the moving object, and thus the moving object in an image can be reliably detected.
In addition, the moving object detection apparatus may further include: a holding unit configured to hold, from among the trajectories calculated by the trajectory calculating unit, trajectories which are calculated using fewer pictures than the pictures included in the image subset; and a trajectory recovery unit configured to classify the trajectories held in the holding unit into a same class as a class to which other trajectories continuous to the trajectories held in the holding unit belong, wherein the moving object detection unit may further detect the moving object in the video by identifying, as the region of the moving object, blocks which correspond to the trajectories included in the same class, the trajectories including the trajectories classified by the trajectory recovery unit.
The trajectories which are not used in generating the subclass are classified into the class to which other trajectories continuous to the trajectories which are not used belong, thereby allowing extraction of a moving object with higher density.
In addition, the subclass label assigning unit may classify, for each of the image subsets, the trajectories which are included in the image subset and calculated by the trajectory calculating unit, into subclasses by classifying similar trajectories from among the trajectories into the same subclass, and assign, to each of the trajectories, a subclass label which is an identifier of the subclass into which the trajectory is classified. More specifically, the subclass label assigning unit may classify, for each of the image subsets, the trajectories into the subclasses, by performing a process of classifying, into a same subclass, a combination of most similar trajectories from among the trajectories which are included in the image subset and calculated by the trajectory calculating unit, repeatedly for predetermined times or until the trajectories are classified into a predetermined number of subclasses, and assign, to each of the trajectories, a subclass label which is an identifier of the subclass into which the trajectory is classified.
With this, it is possible to classify similar trajectories into the same subclass.
In addition, the subclass label assigning unit may (i) classify the trajectories calculated by the trajectory calculating unit into subclasses, by (a) calculating a distance that indicates the similarity between the trajectories, (b) combining, from among the calculated distances, distances each of which is shorter than a predetermined threshold, to transform the calculated distances into geodetic distances, and (c) calculating, for each of the image subsets, a geodetic distance of two arbitrary trajectories from among the trajectories which are included in the image subset and calculated by the trajectory calculating unit, detecting a discontinuity point in a distribution of the obtained geodetic distances, sorting out, as one cluster, trajectories which are spaced apart from each other by the geodetic distance shorter than the detected discontinuity point, to classify the trajectories into subclasses, and (ii) assign, for each of the trajectories, a subclass label that is an identifier of the subclass into which the trajectory is classified, the geodetic distance being a distance of a path that passes through, as a relay point, a trajectory other than the arbitrary two trajectories, from one of the two trajectories to the other.
As described above, the geodetic distance between trajectories is used in order to indicate the similarity between the trajectories, and thus it is possible to detect and extract a moving object with higher accuracy for an articulated object such as a person.
In addition, as to the method of generating a subclass, another example is that the subclass label assigning unit is configured to divide, for each of the image subsets, at least one image into a plurality of regions, using at least one of edge, color, and brightness of the at least one image, to generate subclasses, and assign a subclass label to each of the trajectories that pass through the regions resulting from the division, the at least one image being included in the image subsets.
As described above, it is also possible to classify the trajectories into subclasses, using spatial information of an image, such as edge, brightness, color, and so on.
In addition, the moving object detection apparatus may further include an output unit configured to perform image processing on at least one of the pictures received by the image input unit, such that each of moving object region classified into the same class by the moving object detection unit is displayed in a display mode different from a display mode of other moving object regions, and output the picture on which the image processing is performed.
With this, it is possible to display the extracted moving object on an image, using AV (Audio/Visual) devices, image monitoring devices, and so on, This allows a user to easily recognize the moving object.
In addition, the moving object detection apparatus may further include a recording and transmitting unit configured to write or transmit a result of detecting a moving object performed by the moving object detection unit, into a recording device or to an external device via a transmission path, respectively.
With this, by selecting only a necessary moving object image from among the moving object images, to store or output the selected image to an external device, it is possible to efficiently store and output images as pictorial elements, for example. Thus, this is effective in the case of storing and transmitting a moving object image to a device having a limited processing capacity such as a cellular phone.
In addition, the moving object detection apparatus may further include a motion prediction unit configured to calculate, from among the trajectories which are identified by the moving object detection unit and included in the class, a representative trajectory that represents the class, and predict, according to the calculated representative trajectory, motion of the moving object by predicting that a region of the moving object corresponding to the class moves between the pictures.
By predicting the motion of the moving object using the representative trajectory of a plurality of blocks, it is possible to perform motion prediction that is highly resistant to noise.
In addition, the inter-subclass similarity calculating unit may calculate a similarity between the subclasses by repeatedly multiplying a matrix W in which each of elements indicates the trajectory share ratio between the subclasses, by a matrix Z in which each of the elements indicates the similarity between the subclasses, to update the matrix Z in which each of the elements indicates the similarity between the subclasses.
Since the similarity between the subclasses can be calculated by performing multiplication between matrices, it is possible to calculate the similarity between the subclasses at high speed.
In addition, the inter-subclass similarity calculating unit may further calculate the similarity between the subclasses by weighting the updated matrix Z with a matrix Y in which each of the elements indicates a subclass reliability that is a value corresponding to the number of trajectories included in the subclass or a spatial size of a region including trajectories included in the subclass.
The similarity between the subclasses is calculated by adding the subclass reliability, and thus it is possible to increase the reliability of the similarity between the subclasses.
These general and specific aspects may be implemented using a system, a method, an integrated circuit, a computer program, or a computer-readable recording medium, or any combination of systems, methods, integrated circuits, computer programs, or computer-readable recording media.
Hereinafter, certain exemplary embodiments are described in greater detail with reference to the accompanying Drawings.
Each of the exemplary embodiments described below shows a general or specific example. The numerical values, shapes, materials, structural elements, the arrangement and connection of the structural elements, steps, the processing order of the steps etc. shown in the following exemplary embodiments are mere examples, and therefore do not limit the scope of the appended Claims and their equivalents. Therefore, among the structural elements in the following exemplary embodiments, structural elements not recited in any one of the independent claims are described as arbitrary structural elements,
The following describes, with reference to the drawings, exemplary embodiments according to the present disclosure.
The moving object detection apparatus 200 is an apparatus which detects a moving object in video by performing classification, using trajectories, to identity an entire or partial region of the moving object in the video. More specifically, the moving object detection apparatus 200 is an apparatus which detects a moving object in the video by performing segmentation to identify an entire or partial region of the moving object in the video.
The image input unit 201 is a processing unit which receives input of temporally different pictures included in the video, and is, for example, a video camera or a communication interface connected to the video camera, or the like. More specifically, the image input unit 201 receives at least three pictures included in the video. Hereinafter, the picture is also referred to as the image.
The trajectory calculating unit 202 divides, into image subsets, the at least three pictures which are received by the image input unit 201. Here, each of the image subsets is assumed to include at least two of the pictures. Then, the trajectory calculating unit 202 calculates a trajectory for each of the image subsets. The trajectory calculating unit 202 calculates a trajectory by detecting motion of an image between at least two pictures, for each block that (i) includes one or more pixels and (ii) is included in each of the pictures, and concatenating the detected motions over more than one picture. To be specific, a block is a unit for calculating a trajectory and a group of one or more pixels. More specifically, the trajectory calculating unit 202 (i) extracts, from among pictures, image subsets each composed of at least two of the pictures, and (ii) calculates, for each of the image subsets, trajectories each of which is a trajectory of the motion, between the pictures included in the image subset of a block including one or more pixels and included in the pictures.
The subclass classification unit 213 classifies, for each of the image subsets, the trajectories which are calculated by the trajectory calculating unit 202 and included in the image subset, into subclasses. The subclass classification unit 213 includes a subclass label assigning unit 203. The subclass label assigning unit 203 (i) classifies, for each of the image subsets, the trajectories which are calculated by the trajectory calculating unit 202 and included in the image subset, into subclasses, and (ii) assigns, to each of the trajectories, a subclass label that is an identifier of a corresponding one of the subclasses into which the trajectory is classified.
The inter-subclass similarity calculating unit 214A (i) calculates a trajectory share ratio which indicates the degree of sharing of the same trajectory between arbitrary two subclasses, and (ii) calculates the similarity between the subclasses based on the share ratio. The inter-subclass similarity calculating unit 214A includes a label propagation unit 204A. The label propagation unit 204A (i) calculates the trajectory share ratio which indicates the degree of sharing of the same trajectory (trajectory that indicates a motion of the same block) between the first subclass and the second subclasses which are arbitrarily selected from among all of the subclasses, and (ii) propagates the subclass label between the subclasses, by performing reassigning such that the same subclass label is more likely to be reassigned to the first subclass and the second subclass as the calculated share ratio is higher. Thus, the label propagation unit 204A uses the trajectory share ratio as the similarity (the strength of temporal relation) of an arbitrary subclass pair, to propagate label information of the subclass between the subclasses, it is to be noted that the trajectory share ratio is a ratio of holding the same trajectory between the subclasses. The details thereof will be described later. Here, the label information of the subclasses is updated based on the results of propagating the label information of the subclasses. This means that subclasses to which the same subclass label is assigned indicate a high similarity between the subclasses,
The moving object detection unit 204B identifies, as a region of a moving object, blocks which correspond to the trajectories included in the same class, by classifying the subclasses into classes such that the subclasses between which a higher similarity is present which is calculated by the inter-subclass similarity calculating unit 214A are more likely to be classified into the same class, thereby detecting the moving object in the video.
The same subclass label is assigned to two subclasses between which a high similarity is present. Thus, the moving object detection unit 204B identifies, as a region of a moving object, blocks which correspond to the trajectories included in the same class, by classifying, into the same class, the subclasses to which the same subclass label is assigned, thereby detecting the moving object in the video.
More specifically, the moving object detection unit 204B determines that subclasses to which the same subclass label is assigned as a result of the update of the label information are classified into the same class. As described above, the moving object detection unit 2048 determines whether or not the subclasses obtained from each of the image subsets are classified into the same class, thereby performing clustering having temporal coherence. This allows the same class label to be assigned to the trajectories which belong to the subclasses that are determined as being classified into the same class. As a result, since one class corresponds to one moving object, it is possible to identify a region of the moving object when the trajectories can be classified into the respective classes, and thus it is possible to detect the moving object. Here, a class is a group of one or more subclasses, and the subclasses which belong to the same class correspond to the same moving object.
The output unit 205 outputs the result of detecting the moving object in the video performed by the moving object detection unit 240B, More specifically, the output unit 205 performs image processing on at least one picture among the pictures received by the image input unit 201, such that each of moving object region classified into the same class by the moving object detection unit 204B is displayed in a display mode different from a display mode of other moving object regions, and outputs the picture on which the image processing is performed. It is to be noted that the output unit 205 outputs the picture on which the image processing is performed, to a display apparatus or the like.
It is to be noted that the term “segmentation” as referred to in the present specification includes both of (i) a detection technique of extracting an image region in which a particular target object is present, and (ii) a classification technique of classifying image regions (trajectories) for each object irrespective of the target. It is to be noted that the detection technique and the classification technique have much in common, and therefore are not distinguished from each other in the present specification.
It is to be noted that each of the structural elements included in the above-described moving object detection apparatus 200 (the image input unit 201, the trajectory calculating unit 202, the subclass label assigning unit 203, the label propagation unit 204A, the moving object detection unit 204B, and the output unit 205) may be realized by software such as a program executed on a computer, or may be configured in the form of a hardware product, such as an electronic circuit and an integrated circuit.
It is to be noted that the computer program may be stored on, for example, an optical disc, without being limited to the ROM 306 or HDD 308 that is a semiconductor. In addition, the computer program may also be transmitted via a wired or wireless network, broadcasting, and so on, and may be retrieved into the RAM 307 in the computer,
The following describes an operation of the moving object detection apparatus 200 according to this exemplary embodiment, with reference to
In
In the image inputting step S401, the image input unit 201 obtains, from the camera 301, a plurality of pictures included in video, Here, it is assumed that T (T≧3) pictures are received.
Next, in the trajectory calculating step S402, the trajectory calculating unit 202 divides T pictures received by the image input unit 201 into temporally sequential D image subsets, each of which includes F (F≧2) pictures, Here, it is to be noted that T>F, For example, the trajectory calculating unit 202 divides the pictures into image subsets 502 such that a picture of the same time is shared between adjacent image subsets as shown in
A detailed explanation for the specific method of calculating the corresponding point between the plural pictures described above is omitted here as described in detail in Non Patent Literature 6, Non Patent Literature 7, and so on.
It is to be noted that it is sufficient that T is three or more and F is two or more in this exemplary embodiment. In addition, as to the image subsets, T pictures need not necessarily be divided such that a picture of the same time is shared between adjacent image subsets as in the example of
Then, in the trajectory calculating step S402, the trajectory calculating unit 202 generates, for each of the image subsets, I trajectories corresponding to the respective pixels, from sets of pixel coordinates for corresponding points which indicate motion information of pixels at I points over F pictures, as shown in Expression 1 below.
The following describes an example of calculating trajectories for one image subset.
[Math. 1]
x
i=(xli,yli, . . . ,xfi,yfi, . . . , xFi,yFi) Expression 1
Here, F denotes the number of pictures included in the image subset; that is, the number of pictures used in calculating the trajectories.
Next, in the subclass label assigning step S403, the subclass label assigning unit 203 classifies I trajectories represented in Expression 1 into S subclasses, for each of image subsets D.
A various methods are available because the classification into subclasses only requires classifying similar trajectories into the same subclass. For example, by setting the number of subclasses to S, with input of I trajectories as shown in Expression 1, and using the k-means method as shown in Non-Patent Literature 8, pp. 526-528, it is possible to classify I trajectories into S subclasses in such a manner that similar trajectories are classified into the same subclass.
Furthermore, with input of I trajectories, it is possible to use a clustering algorithm based on a dendrogram as shown in Non-Patent Literature 8, pp. 550-555. Here, processing of sequentially determining trajectory pairs having highest similarity (having a shortest distance) as the same subclass is repeatedly performed either predetermined times or until the number of subclasses becomes S. It is to be noted here that, to represent the similarity between trajectories, a Euclidian distance between trajectory vectors may be used, or a distance obtained by normalizing the Euclidian distance using the number of trajectories belonging to the same subclass may be used.
For example, when calculating the similarity (distance) between a pair of trajectories, as shown in
By the processing using the clustering algorithm as described above, each of the trajectories x which belongs to a corresponding one of the image subsets D belongs to a corresponding one of subclasses D-s as shown in Expression 2. To be more specific, each of the trajectories is assigned with a label of the subclass D-s (hereinafter referred to as “subclass label D-s”).
[Math. 2]
x
iεD-2=(xli,yli, . . . ,xfi,yfi, . . . ,xFi,yFi) Expression 2
It is to be noted that the classification into subclasses may be performed such that the size of each of the subclasses is equal to or smaller than the smallest size of a moving object that is to be detected. Alternatively, the number of subclasses may be set to a greater number than the number of moving objects which is previously assumed. Here, one moving object may be divided into plural portions as subclasses, Accordingly, since the number of subclasses S need not necessarily the same as the number of moving objects which exist in the image, and it is sufficient to set the number of subclasses S as a value larger than the number of moving objects, strict setting is not required even when a specific number of moving objects in a captured scene is unknown. Since each of the image subsets is independently processed at this time, it is not necessary that the number of subclasses S is the same between the image subsets, and the number of pictures F included in the image subset may be different for each of the image subsets.
Next, in the label propagation and update step S404, the label propagation unit 204A propagates, between the subclasses, label information of the subclass D-s obtained from each of the image subsets D, based on a trajectory share ratio between the subclasses. Then, the label propagation unit 204A determines whether or not to update the label of the subclass, by determining whether or not an arbitrary subclass pair, among the subclasses obtained from different image subsets, belongs to the same class, to assign a subclass label to each of the trajectories. The moving object detection unit 204B determines that the subclasses belong to the same class when the labels of the subclasses are updated to the same label. As a result, since one class corresponds to one moving object, classification of the trajectories into the respective classes leads to identifying a region of the moving object, and thus it is possible to detect the moving object. In addition, this allows label propagation using the trajectory share ratio, between subclasses that belong to temporally different image subsets, thereby enabling determining that subclasses with a high trajectory share ratio belong to the same class. Thus, there is an advantageous effect that occlusion has less effect compared to the case where a long-term trajectory is used, and that it is possible to perform segmentation having temporal coherence in the same manner as the case where a long-term trajectory is used.
The specific processing will be described below. As shown in Expression 2,
As shown in
Here, the label propagation unit 204A calculates the trajectory share ratio W for an arbitrary subclass pair. At this time, the subclass labels Da and Db are subclasses obtained from different image subsets. In addition, N_Da is the number of trajectories included in the subclass Da. It is to be noted that the processing of assigning a subclass label in the subclass label assigning step S403 is performed independently on each of the image subsets, and a similarity relation (the strength of temporal relation) between the subclasses is represented as the trajectory share ratio, sing the independently processed subclasses.
More specifically, in the example of
In addition, the trajectory share ratio W can be expressed as an asymmetric matrix as shown in Expression 4. However, the trajectory share ratio W may be expressed as a symmetric matrix. More specifically, an average value of the trajectory share ratio WDa, Db of the subclass Db to the subclass Da and the trajectory share ratio WDb, Da of the subclass Da to the subclass Db may be reset as each of the trajectory share ratios WDa,Db and WDb,Da. Alternatively, a value larger between the trajectory share ratios WDa,Db and WDb,Da may be reset as each of the trajectory share ratios WDa,Db and WDb,Da. Such resetting allows the trajectory share ratio W to be expressed as a symmetric matrix.
Next, the label propagation unit 204A propagates label information as below, using the trajectory share ratio W calculated by Expression 3. Non Patent Literature 9 discloses a method of label propagation; however, there is no description as to the method of calculating similarity between subclasses from independently calculated subclasses, nor an example representing the similarity by the trajectory share ratio.
In this exemplary embodiment, the similarity between the subclasses calculated independently is represented by the trajectory share ratio, thereby performing the label propagation as below. To be more specific, the label propagation is performed by repeatedly calculating the similarity between subclasses, according to this exemplary embodiment. Here, the similarity between subclasses is indicated as an element of a subclass matrix Z described below.
[Math, 5]
Z
l+1
=αWZ
l+(1−α)Y Expression 5
Here, the first term in the right-hand side represents label propagation using the similarity between subclasses, The repeated calculation of the first term in Expression 5 using a trajectory share ratio corresponds here to the label propagation using the trajectory share ratio as weight. In addition, the second term represents subclass reliability described below. The subclass matrix is Z=[z1T, z2T, . . . , zpT, . . . zΓT], and zpT is a subclass label vector of the subclass p. zpT is a vector which has the number of elements of the number of subclass labels and of which only an element corresponding to each of the subclasses is denoted as 1 and other elements are denoted as 0 in an initial state (I=1). For example, z1T=[1, 0, 0, . . . , 0] and z2T=[0, 1, 0, 0]. More specifically, the subclass matrix Z is, in the initial state, a unit matrix exemplified in Expression 6.
The symbol I represents the number of repetition. It is to be noted that α≦1, and α=1 may be specified when the subclass reliability is assumed to be equal between all of the subclasses, or α=0.99 and the like may be specified when the subclass reliability is taken into consideration.
The subclass reliability matrix is Y=[y1T, y2T, . . . , ypT, . . . , YΓT], and Y is a diagonal matrix. ypT is a reliability vector of the subclass p, which has the number of elements of the number of subclass labels as with the subclass label vector zpT, and of which only an element corresponding to each of the subclass labels has a value greater than 0 and other elements are represented as 0. For example, yiT=[2, 0, 0, . . . , 0] and y2T=[0, 3, 0, . . . , 0]. The numeral “2” in yiT represents the reliability for the subclass 1, and the numeral “3” in y2T represents the reliability for the subclass 2. To be more specific, the subclass reliability matrix Y is a diagonal matrix exemplified in Expression 7.
The reliability may be set as below. For example, the reliability may be set such that the subclass reliability increases as the number of trajectories that belong to each of the subclasses are larger, or such that the subclass reliability increases as an average temporal length of trajectories that belong to each of the subclasses are greater. Alternatively, the reliability may be set such that the subclass reliability increases as a spatial size of a region including trajectories that belong to each of the subclasses are greater.
The label propagation unit 204A performs the process of repeating the calculation of Expression 5 for the predetermined number of times. It is to be noted that such a process of performing repeated operation is referred to as a label propagation process in Non Patent Literature 9, Then, whether or not to update the subclass label is determined by Expression 8 below.
[Math. 8]
z
p
Final=argmaxqZpq Expression 8
Here, Expression 8 indicates the row number having the largest value for each column of the subclass matrix Z having p columns and q rows. For example, it is assumed that the subclass matrix Z is represented as Expression 9. In this case, in the first column, the value in the third row is 1.5 which is the largest in the first column, Accordingly, 3 is obtained from Expression 8 as the row number having the largest value in the first column. This means that 3 is reassigned as a subclass label to the subclass having the subclass label 1 in the initial state.
More specifically, the column number of the subclass label vector z calculated by Expression 5 indicates the subclass label in the initial state, and the label propagation unit 204A updates, as a new subclass label, the row number having the largest value in each of the columns. With this process, the plural subclasses belong to the same class, and thus it is possible to determine, as the same class, different subclasses including subclasses calculated from independent image subsets. In addition, with the processes of Expression 5 and Expression 8, the relationship between the subclasses obtained from the same image subset is automatically taken into consideration. Accordingly, in such a case as indicated in
The following describes a specific advantageous effect, with reference to
It is to be noted that calculation of the trajectory share ratio between subclasses generated from different image subsets need not necessarily be performed on the image of the same time, but can also be calculated from the number of trajectories shared on two images of time (t+F) and time (t+F+n) as shown in
Next, in the image output step S405, the output unit 205 outputs the result of detecting the moving object in the video, which is performed in the label propagation and update step S404. More specifically, image processing is performed on the video received in the image inputting step S401 so as to be displayed, for example, in a different mode per moving object region θ detected in the label propagation and update step S404, and the image on which the image processing is performed is output to a display apparatus or the like. Through the processes described above, it is possible to perform display while temporally maintaining coherence and distinguishing the respective moving objects.
The following describes the method of determining the number of images F of image subsets in the case where the moving object is a walking person. A walking pace which indicates how many steps can be proceeded for in a minute is approximately 110 to 120 for general adult human, approximately 70 to 80 in the case of slow walk, and approximately 140 in the case of walk in a quick pace. Accordingly, time required for one step can be assumed as 0.43 to 0.86 seconds. Meanwhile, when an image of a walking person is captured from a lateral direction as shown in
As described above, it is possible to properly detect and extract a moving object even under the circumstance in which occlusion is likely to occur, while solving occlusion that occurs due to an imaging position of a camera and erroneous detection of which one moving object is separated into plural portions.
Thus, today, with video capturing apparatuses such as a digital video camera becoming widely used, the present disclosure provides an extremely high practical value as technology applied to: focus control and image quality improvement processing in digital cameras; a driving safety support system for vehicles; or collision avoidance control to avoid collision against a person and alarming using a robot.
Next, a moving object detection apparatus according to Embodiment 2 of the present disclosure will be described.
Here, an example is described in which the subclass label assigning process is performed by the subclass label assigning unit 203 in a different manner from the method described in Embodiment 1.
Such a moving object detection apparatus 200a according to Embodiment 2 has the same configuration as the moving object detection apparatus 200 according Embodiment 1, except for the configuration of the subclass label assigning unit 203a, and therefore the description of the same constituent elements will be omitted.
It is to be noted that, although the subclass label assigning unit 203a is included in the subclass classification unit 213 illustrated in
The subclass label assigning unit 203a, for each of the image subsets, uses at least one of edge, color and brightness of at least one of the images included in the image subset, to divide the image into plural regions, generates subclasses, and assigns a subclass label to each of the trajectories that pass through the regions resulting from the division.
More specifically, the subclass label assigning unit 203a, for each of the image subsets, performs a process for classifying similar image regions into the same subclass, from at least one of the images received by the image input unit 201, using the brightness, the edge information, the color, and so on. Next, the subclass label assigning unit 203a assigns, to the trajectory calculated by the trajectory calculating unit 202, a subclass label of the image region through which the trajectory passes. More specifically, the subclass label assigning unit 203a classifies trajectories passing through the image regions classified into the same subclass, by assigning the same subclass label to the trajectories. The difference from Embodiment is in performing the classification into subclasses not based on trajectories but based on information of the image, and assigning subclass labels to the trajectories based on the information. It is to be noted that the subclass classification process performed on the image region by the subclass label assigning unit 203a can be performed without using the information on the trajectories. Thus, the subclass classification process need not necessarily be performed after the process performed by the trajectory calculating unit 202, and both processes may be performed in parallel, or the subclass classification process on the image region may be performed before the process performed by the trajectory calculating unit 202.
The following describes an operation of the moving object detection apparatus 200a according Embodiment 2, with reference to
Since steps S401 and S402 are the same as those in Embodiment 1, the description thereof will be omitted.
Next, in a subclass label assigning step S403a, the subclass label assigning unit 203a performs image segmentation using at least one of the images received in the image input step S401. Here, any method may be used as long as segmentation is performed, in the method, using input such as a pixel (brightness) value, color information, edge information or the like and position information thereof, and for example, the method disclosed in Non Patent Literature 10 can be used.
According to the method disclosed in Non Patent Literature 10, an image region is segmented into a plurality of small regions using edge information or smoothness of a pixel value near the space. For example, when the target is a person, as shown in
Alternatively, for example, the pixel values and pixel positions may be arranged into a three-dimensional vector, and with an input of the three-dimensional vector, the image region may be segmented using a clustering algorithm such as the k-means method, or of course, color information such as RGB and pixel position information may be arranged into a five-dimensional vector to be used instead of the above-described three-dimensional vector. Thus, any method may be used as long as the method allows spatially segmenting the image.
Then, for each of the image subsets, a subclass label is assigned to each trajectory based on a relationship between the time of the image selected for the segmentation and the trajectory calculated in the trajectory calculating step S402, with the regions resulting from the segmentation being as subclasses.
More specifically, as shown in
Since the processes in and subsequent to the label propagation and update step S404 are the same as those in Embodiment 1, the description thereof will be omitted.
As described above, it is possible to detect a moving object in an image based on the trajectories even under the crowded circumstance in which occlusion is likely to occur, according to Embodiment 2. Furthermore, compared to Embodiment 1, since it is possible to explicitly use information such as color and brightness, for example, it is possible to perform moving object detection with higher accuracy when detecting an moving object dressed in a single color or the like.
As described above, it is possible to properly detect and extract a moving object even under the circumstance in which occlusion is likely to occur, while solving occlusion occurring due to an imaging position of a camera and erroneous detection of which one moving object is separated into plural portions.
Thus, today, with video capturing apparatuses such as a digital video camera becoming widely used, the present disclosure provides an extremely high practical value as technology applied to: focus control and image quality improvement processing in digital cameras; a driving safety support system for vehicles; or collision avoidance control to avoid collision against a person and alarming using a robot.
Next, a moving object detection apparatus according to Embodiment 3 of the present disclosure will be described. Here, an example is described in which the subclass label assigning process is performed by the subclass label assigning unit 203 in a different manner from the methods described in Embodiment 1 and Embodiment 2.
It is to be noted that, although the subclass label assigning unit 203b is included in the subclass classification unit 213 illustrated in
The subclass label assigning unit 203b (a) calculates, for the trajectories calculated by the trajectory calculating unit 202, a distance that indicates the similarity between the trajectories, (b) combines, from among the calculated distances, distances each of which is smaller than a predetermined threshold, to transform the calculated distances into geodetic distances, (c) calculates, for each of the image subsets, a geodetic distance of two arbitrary trajectories from among the trajectories which are included in the image subset and calculated by the trajectory calculating unit 202, the geodetic distance being a distance of a path that passes through, as a relay point, a trajectory other than the two trajectories, from one of the two trajectories to the other, detects a discontinuity point in a distribution of the obtained geodetic distances, sorts out, as one cluster, trajectories which are spaced apart from each other by the geodetic distance shorter than the detected discontinuity point, to classify the trajectories into subclasses, and assigns, for each of the trajectories, a subclass label that is an identifier of the subclass into which the trajectory is classified.
This means that the subclass label assigning unit 203b, in the same manner as in Embodiment 1, classifies the trajectories of each of the image subsets which are calculated by the trajectory calculating unit 202 into subclasses such that similar trajectories belong to the same class, and assigns a label to the subclass that is a result of classifying each of the trajectories. The difference from Embodiment 1 is in that a geodetic distance is used as the similarity between the trajectories when performing the classification into subclasses.
The following describes an operation of the moving object detection apparatus 200b according to Embodiment 3, with reference to
Next, in the subclass label assigning step S403b, the subclass label assigning unit 203b calculates, for each of the mage subsets, a distance matrix including a similarity in motion of a pixel using the trajectory i calculated by Expression 2. A linear distance f j) between the trajectory i and the trajectory j can be calculated as below.
[Math. 10]
f(i,j)=ptnij+w·mtnij Expression 10
Here, w is a weighting factor and a parameter to be set by a designer.
In addition, ptnij and mtnij will be described below.
As shown in Expression 10, in addition to a temporal average value of the distance between trajectories that is expressed by Expression 11, temporal variation between trajectories that is expressed by Expression 12 is an element of the linear distance f(i, j). The temporal variation between trajectories expressed by Expression 12 shows, in particular, similarity in pixel motion, and thus allows capturing of not only a rigid body of which the distance relationship between pixels does not temporally change but also shape change of an articulated object or the like. It is to be noted here that the above-described distance calculation need not necessarily be performed, but any distance may be used as long as it is the distance measurement that indicates the similarity between the trajectories.
Next, a threshold R is applied to the linear distance f(i, j) calculated by Expression 10, and the process of nonlinearization is performed on the threshold as below to calculate f′(i, j).
When the focus is on the trajectories i, R trajectories j are selected in ascending order of the linear distance from the trajectory i, and changes the distances from unselected trajectories j into an infinite value without changing the distances from the selected trajectories. It is to be noted that, although the trajectories j are selected in ascending order of the linear distance f(i, j), the process of nonlinearization may be performed by setting the threshold R as the expression below
More specifically, the subclass label assigning unit 203b may select, for each of the image subsets calculated by the trajectory calculating unit 202, a predetermined number of trajectories in ascending order of distance for each of the trajectories, and after distances from the unselected trajectories are nonlinearized into the infinite value, each of the distances may be converted into a geodetic distance. Alternatively, as shown in Expression 15, the subclass label assigning unit 203b may select, when the focus is on the trajectories i, trajectories j each having a distance smaller than or equal to the predetermined threshold R for each of the trajectories j calculated by the trajectory calculating unit 202, perform the process of non-linearization for changing the distances from unselected trajectories into the infinite value, and then convert each of the distances into the geodetic distances.
It is to be noted that the technique of nonlinearizing distances is not limited to that using functions described above, but may be anything that performs nonlinearization on the distance related to the trajectories i and j.
Next, the subclass label assigning unit 203b calculates the geodetic distance as in the following expression, using the nonlinearized distance f′(i, j).
[Math. 16]
g(i,j)=min(f′(i,j),f′(i,s)+f′(s,j)) Expression 16
It is to be noted that min(x, y) is a function which returns a smaller one of the value x and the value y. In addition, the trajectory s is a relay point between the trajectory i and trajectory j. Here, the relay point s in f′(i, s)+f′(s, j) is not limited to a single point. This method is a shortest path search method known as Dijkstra method and is described in the following Non Patent Literature 11.
Here, the advantageous effect of the nonlinearization expressed in Expressions 14 to 16 is described with reference to
Next, the trajectories are classified into subclasses by detecting a discontinuity point using g(i, j) resulting from geodetic distance conversion and corresponding to the threshold R, to assign the subclass labels. Here, the discontinuity point is where g(i, j) is infinite between the trajectory i and the trajectory j. An example of a result of the geodetic distance conversion which is obtained with respect to the threshold R is described with reference to
c) is a diagram which shows a result of clustering. Here, in
With the processes described above, it is possible to determine that the groups of trajectories not having an infinite geodetic distance are continuous and thus belong to the same subclass, and to separate the trajectories into subclasses based on the discontinuity point by determining that the groups of trajectories having infinite distance are discontinuous. In other words, each trajectory is assigned with a label of the subclass D-s as with the process in the subclass label assigning step S403 according to Embodiment 1.
Since the processes in and subsequent to the label propagation and update step S404 are the same as those in Embodiment 1, the description thereof will be omitted.
As described above, it is possible to detect a moving object in an image based on the trajectories even under the crowded circumstance in which occlusion is likely to occur, according to Embodiment 3. It is to be noted that, in this exemplary embodiment, since similar trajectories are classified into the same subclass based on the geodetic distance, the difference between the motion of a head region and the motion of arms and the difference between the motion of upper thighs and the motion of lower thighs are reflected in the example shown in
As described above, it is possible to properly detect and extract a moving object even under the circumstance in which occlusion is likely to occur, while solving occlusion that occurs due to an imaging position of a camera and erroneous detection of which one moving object is separated into plural portions.
Thus, today, with video capturing apparatuses such as a digital video camera becoming widely used, the present disclosure provides an extremely high practical value as technology applied to: focus control and image quality improvement processing in digital cameras; a driving safety support system for vehicles; or collision avoidance control to avoid collision against a person and alarming using a robot.
Next, a moving object detection apparatus according to Embodiment 4 of the present disclosure will be described.
The moving object detection apparatus 200c according to Embodiment 4 has the same configuration as the configuration of the moving object detection apparatuses according to Embodiment 1 to Embodiment 3 except for the recovery trajectory holding unit 1701 and the trajectory recovery unit 1702, and thus the description of the same constituent elements will be omitted It is to be noted that, although the description will be given as Embodiment 1 being the basic form, this exemplary embodiment can also be applied to Embodiment 2 and Embodiment 3 without loss of generality.
It is to be noted that, although the subclass label assigning unit 203 is included in the subclass classification unit 213 illustrated in
The recovery trajectory holding unit 1701 is a storage device which holds, from among the trajectories calculated by the trajectory calculating unit 202, trajectories that are calculated using fewer pictures than the pictures included in an image subset
The trajectory recovery unit 1702 classifies the trajectories held in the recovery trajectory holding unit 1701 into the same class to which other trajectories continuous to the trajectories held in the recovery trajectory holding unit 1701 belong.
It is to be noted that, the moving object detection unit 204B identifies, as a region of a moving object, blocks which correspond to the trajectories included in the same class, among the trajectories classified by the trajectory recovery unit 1702, in addition to the processes described in Embodiment 1, thereby detecting the moving object in video.
More specifically, the trajectory recovery unit 1702, from among trajectories in subclass pairs between temporally adjacent image subsets which are determined as being in the same class by the moving object detection unit 204B, assigns the same label as a label of the subclass pairs determined as being in the same class, to trajectories of F pictures on which motion detection cannot be performed due to occlusion or the like. Then, the recovery trajectory holding unit 1701 holds the trajectories which cannot be calculated for F pictures which are temporally short and which are used in the process performed by the trajectory recovery unit 1702. Through the processes described above, the moving object detection unit 2048 assigns labels to more trajectories, thereby detecting and extracting a moving object with higher accuracy. In general, when the similarity or a distance between trajectories is calculated, it is necessary that the vectors have the same length. Meanwhile, in this exemplary embodiment, the similarity is not calculated for trajectories of which the number of elements of the vector is less than a certain value, and class information obtained through the process performed by the moving object detection unit 204B is used instead to assign a label. With this, more trajectories are used to detect and extract a moving object.
Hereinafter, an operation performed by the moving object detection apparatus 200c according to Embodiment 4 will be described with reference to
In a trajectory calculating step S402c, the trajectory calculating unit 202 calculates a trajectory as expressed in Expression 1 as described in Embodiments 1 to 3. Here, there is the case where a trajectory in F images included in an image subset cannot be calculated due to occlusion or the like. In view of such a case, a trajectory in images fewer than F is separately held, as a recovery trajectory, in the recovery trajectory holding unit 1701. In addition, although the example of calculating a trajectory in the forward direction as in the direction from time (t+1) to time (t+F) is explained in Embodiments 1 to 3, the trajectory here is calculated, for example, separately in the reverse direction from time (t+F) to time (t+1).
Next, in a trajectory recovery step S1801, the trajectory recovery unit 1702 assigns class label information to the recovery trajectories held in the recovery trajectory holding unit 1701, using information on the subclass calculated from temporally different image subsets which are integrated into the same class in the label propagation and update step S404. It is to be noted that, the moving object detection unit 204B identifies, as a region of a moving object, blocks which correspond to the trajectories included in the same class, among the trajectories classified by the trajectory recovery unit 1702, thereby detecting the moving object in video.
In the same manner as above, the recovery trajectories connected to the trajectories obtained from an immediately subsequent image subset are assigned with the class labels obtained from the immediately subsequent image subset. For example, the recovery trajectory 1902 is connected to a trajectory in a class assigned with the class label θ1 obtained from the immediately subsequent image subset. Therefore, the recovery trajectory 102 is assigned with the class label θ1.
Accordingly, labels can be assigned to the trajectories over fewer than F images because the labels are assigned in the label propagation and update step S404 such that a class label obtained from temporally adjacent image subsets and a class label from a current image subset temporally maintain coherence. As described above, the labels can be assigned to the trajectories with higher density compared to Embodiments 1 to 3, and as a result, it is possible to detect and extract a moving object with higher accuracy. In addition, there is an advantageous effect that the moving object can be extracted with higher precision.
Next, in the image outputting step S405c, the output unit 205 performs output including the result of detecting the moving object in the video performed in the trajectory recovery step S1801. More specifically, image processing is performed on the video received in the image inputting step S401 so as to be displayed, for example, in a different mode per moving object region detected in the label propagation and update step S404 and the trajectory recovery step S1801, and the image on which the image processing is performed is output to a display apparatus or the like. Through the processes described above, it is possible to perform display while temporally maintaining coherence and distinguishing the respective moving objects.
As described above, it is possible to properly detect and extract a moving object even under the circumstance in which occlusion is likely to occur, while solving occlusion that occurs due to an imaging position of a camera and erroneous detection of which one moving object is separated into plural portions. In addition, since it is difficult to calculate a trajectory in a long time period under the circumstance in which occlusion is likely to occur, labels are also assigned to trajectories in a short time period as described above, thereby enabling the detection of a moving object with higher precision,
Thus, today, with video capturing apparatuses such as a digital video camera becoming widely used, the present disclosure provides an extremely high practical value as technology applied to: focus control and image quality improvement processing in digital cameras; a driving safety support system for vehicles; or collision avoidance control to avoid collision against a person and alarming using a robot.
[Modification 1 of Embodiments 1 to 4]
Next, a moving object detection apparatus according to Modification 1 of Embodiments 1 to 4 will be described. Here described is an example of adding a function of recording and transmitting of the result of detecting a moving object for each detected region after the class determination performed in Embodiments 1 to 4. Here, a modification example of Embodiment is described, but the modification is also applicable to Embodiments 2 to 4. The moving object detection apparatus 200e according to Modification 1 can be implemented by including a recording and transmitting unit 2001 in addition to the image input unit 201, the trajectory calculating unit 202, the subclass label assigning unit 203, the label propagation unit 204A, and the moving object detection unit 204B, as illustrated in
It is to be noted that, although the subclass label assigning unit 203 is included in the subclass classification unit 213 illustrated in
The recording and transmitting unit 2001 writes, in a recording device, the result of detecting a moving object performed by the moving object detection unit 204B, or transmits the result to an external device via a transmission path. More specifically, the recording and transmitting unit 2001 is a processing unit which identifies a moving object region in the picture received by the image input unit 201 based on the result of detecting a moving object performed by the moving object detection unit 204B, and, for each of the identified moving object regions, records information that indicates the moving object region on a recoding medium such as a memory, hard disk, as so on, or transmits the information to an external device via a communication interface, and the like. In other words, the recording and transmitting unit 2001 separately records or transmits the information that indicates the image resulting from detecting the moving object, according to the class label θ, as with the case of the image display. In addition, by averaging motion information that belongs to each of the classes as below, it is also possible to compress the motion information. Although it is normally necessary to hold motion information for of the each pixels, holding one motion for one moving object is sufficient if the following process is performed. For example, when the motion vector (uti, vti) of the pixel i is used as motion information, it is possible to calculate averaged motion information for each of the regions resulting from segmentation as below.
Here, Cm is the number of trajectories that belong to the class θm. The utcm and vtcm are the x component and the y component, respectively, of the motion vector of a pixel that belongs to the class θm.
[Modification 2 of Embodiments 1 to 41]
The following describes, as Modification 2 of Embodiments 1 to 4, the moving object detection apparatus that additionally has a function to predict the motion of a moving object based on the result of detecting the moving object. Here, the description will be given according to Embodiment 1, but the moving object detection apparatus can also be implemented according to Embodiments 2 to 4.
As shown in
It is to be noted that, although the subclass label assigning unit 203 is included in the subclass classification unit 213 illustrated in
The motion prediction unit 2201 calculates a representative trajectory from trajectories of pixels included in each of the classes, based on the result of detecting a moving object, and predicts motion of the moving object based on the representative trajectory, More specifically, the motion prediction unit 2201 calculates a representative trajectory that represents the class, from among the trajectories included in the class that is identified by the moving object detection unit 204B, and predicts that a region of the moving object which corresponds to the class moves between pictures according to the calculated representative trajectory, thereby predicting the motion of the moving object.
Next, in step S2301, the motion prediction unit 2201 calculates, from the trajectories that belong to the respective classes, a representative point of the class and a representative trajectory of the representative point, based on the result of detecting a moving object performed by the moving object detection unit 2048.
Here, trajectories of pixels that belong to the class θm are represented as xc
Here, Cm is the number of pixels that belong to the class θm or the number of trajectories of the pixels.
x
m
Each of the denotations “x” in the diagram indicates a pixel posit on that is an element of the representative trajectory corresponding to time t According to this method, as compared to the method of simply calculating a temporal average of trajectories of an adjacent pixel, it is possible to calculate the representative trajectory using only the trajectory of the pixel that is similar in motion. According to this, it is possible to calculate the representative trajectory with higher accuracy. As described above, by calculating the representative trajectory for each of the classes, it is possible to represent, accurately and easily, the motion of each imaging object or the motion of each part.
Next, in Step S2302, the motion prediction unit 2201 predicts the position of the moving object at a time earlier than time T, from the representative trajectory calculated in Step S2301. First, an example of calculating acceleration from the representative trajectory, and predicting the position of the moving object subsequent to time (T+1) is described. When three or more time-series images are inputted, it is possible to obtain an acceleration vector sm for each representative trajectory as expressed in the following Expression 20.
x
m
[Math. 22]
s
t+1
m
={u
t+1
m
−u
t
m
}−{u
t
m
−u
t−1
m} Expression 20
Here, utm is a motion vector of a pixel m at time t, and can be represented as Expressions 21 below.
[Math. 23]
u
t
m=(utm,vtm) Expression 21
As shown by dashed arrows and circles in
Then, the output unit 2202 outputs the position of the moving object or the position of the part of the moving object that is predicted in step S2301. With this, it is possible to perform motion prediction taking acceleration into consideration. In such a case where the motion suddenly becomes quick or suddenly stops, there is an advantageous effect of allowing predicting the position of the moving object by reflecting the acceleration. In addition, an affine parameter may be used instead of the motion vector. Since the affine parameter allows representing motion which includes rotation and is appropriate for representing rotational movement of an arm or a leg, it is possible to correctly predict, in particular, the position of an articulated object.
In addition, instead of the motion vector and acceleration described above, it is also possible to fit an Nth-order function directly to the representative trajectory xm. When temporally sequential T images are input, the Nth-order function can be fit to T items of position information on the images, which are included in the representative trajectory.
x
m
With this, it is possible to estimate the position posm (T+t′) on the image at time (T+t′) according to the fitted function value. More specifically, since fitting of the function allows representing smoother movement, it is possible to perform motion estimation with higher accuracy, In particularly, since this allows motion prediction according to the fitted function, the motion prediction is more accurate when the fitted function is closer to an original motion. Naturally, the position prediction on these images is also applicable to temporal interpolation.
As described above, according to this modification example, the trajectories of the pixels which have similar motions can be calculated as the same class, thus it is possible to obtain the representative trajectory with high accuracy.
Thus far, the moving object detection method and the moving object detection apparatus according to the present disclosure have been described based on the exemplary embodiments and the exemplary modifications thereof, the present disclosure is not limited to these exemplary embodiments and exemplary variations. Those skilled in the art will readily appreciate that many modifications are possible in the exemplary embodiments without departing from the novel teachings and advantages of this disclosure. Accordingly, all such modifications are intended to be included within the scope of this disclosure.
In addition, the present disclosure also includes a mode realized by an arbitrary combination of characteristic constituent elements in each of the embodiments.
In addition, the moving object detection apparatus in the embodiments above includes the image input unit 201 and the trajectory calculating unit 202, but these constituent elements are not essential according to the present disclosure. In other words, when trajectories of images in each of the plurality of blocks included in the video are previously calculated, the moving object detection apparatus 200 may obtain such trajectories from an external device, and perform the processing in steps S403 and S406 on the obtained trajectories.
In addition, each of the moving object detection apparatus according to the embodiments described above includes the output unit 205 or 205a, however, the output unit is not essential according to the present disclosure. To be more specific, the result of detecting a moving object may be used for other processing instead of displaying the result on a display or the like.
In addition, the present disclosure has been realized as a moving object detection apparatus, but it goes without saying that the present disclosure can be realized as an image processing device which extracts or divide a region of an object having motion in video.
Each of the structural elements in each of the above-described embodiments may be configured in the form of an exclusive hardware product, or may be realized by executing a software program suitable for the structural element. Each of the structural elements may be realized by means of a program executing unit such as a CPU and a processor, as shown in
The program causes a computer to execute each of the steps included in a moving object detection method. The moving object detection method is a method of detecting a moving object in video by performing segmentation to identify an entire or partial region of the moving object in the video, and includes: receiving at least three pictures included in the video; extracting, from among the at least three pictures, image subsets each including at least two of the pictures, and calculating, for each of the image subsets, trajectories each of which is a trajectory of motion of a block, between the pictures included in the image subset, the block including one or more pixels and being included in each of the pictures; classifying, for each of the image subsets, the calculated trajectories into subclasses, the calculated trajectories being included in the image subset; calculating a trajectory share ratio which indicates a degree of sharing a same trajectory between two of the subclasses, and calculating a similarity between the two of the subclasses based on the trajectory share ratio; and detecting the moving object in the video by classifying the subclasses into classes such that the subclasses between which the calculated similarity is higher are more likely to be classified into the same class, to identify, as the region of a moving object, the blocks which correspond to the trajectories included in a same class.
The present disclosure is applicable as a moving object detection apparatus which detects, based on motion in a plurality of pictures, a moving object in video by performing region extraction on an image that includes the moving object that moves changing shape such as a person, for example, as a moving object detection apparatus and the like that is to be incorporated in, for example, a motion analysis apparatus, a monitoring apparatus, and an audio-visual device such as a video camera or a television set.
Number | Date | Country | Kind |
---|---|---|---|
2011-063297 | Mar 2011 | JP | national |
This is a continuation application of PCT Patent Application No PCT/JP2012/001731 filed on Mar. 13, 2012, designating the United States of America, which is based on and claims priority of Japanese Patent Application No. 2011-063297 filed on Mar. 22, 2011. The entire disclosures of the above-identified applications, including the specifications, drawings and claims are incorporated herein by reference in their entirety. One or more exemplary embodiments disclosed herein relate generally to image processing techniques of detecting a moving object by identifying a region of the moving object in an image, and particularly to a moving object detection apparatus and the like which detect and extract a moving object based on motion information in video under the circumstance in which occlusion is likely to occur.
Number | Date | Country | |
---|---|---|---|
Parent | PCT/JP2012/001731 | Mar 2012 | US |
Child | 13665255 | US |