Patent Application
20110199461
The present invention relates to a motion locus creation system, motion locus creation apparatus, and motion locus creation method that create a motion locus that is a path of movement of an object, and a three-dimensional motion locus display apparatus.
Heretofore, many technologies have been proposed that display a motion locus (path of movement) of an object (person, article, or the like) positioned using a wireless tag, surveillance camera, or the like. Displaying such a motion locus makes it possible to monitor a suspicious person, look for abnormal behavior by a person and warn that person, improve work efficiency through worker behavior analysis, implement layout design based on consumer behavior analysis, and so forth.
Motion locus creation apparatuses of this kind have heretofore been disclosed in Patent Literature 1 and Patent Literature 2.
Patent Literature 1 discloses a technology whereby a path of a moving object in an image is found by means of image processing, and this path is displayed superimposed on a moving image.
Patent Literature 2 discloses a technology whereby positioning data for a moving object is obtained using a wireless ID tag attached to the moving object, and this path is displayed superimposed on a moving image.
A deficiency of the technology disclosed in Patent Literature 1 is that a moving object that enters a concealed position as viewed from a camera cannot be tracked, and therefore an accurate motion locus cannot be created while a moving object is in a concealed position. Also, since tracking is no longer possible after a moving object enters a concealed position, it is difficult to determine the sameness of a moving object that enters a concealed position and a moving object that emerges from a concealed position.
Also, a deficiency of the technology disclosed in Patent Literature 2 is that, although tracking is possible even if a moving object enters a concealed position as viewed from a camera, it is not possible to determine whether or not a moving object is in a concealed position, and therefore even if a moving object enters a concealed position a motion locus continues to be drawn unchanged, and it is extremely difficult for a user to ascertain a path of movement.
One example of a technology that detects whether or not a moving object has entered a concealed position is the Z buffer method described in Non-Patent Literature 1. The Z buffer method uses a 3D model of imaged space. Here, a combination of technology of Patent Literature 2 and the Z buffer method can be conceived of. That is to say, performing shade line processing using path-of-movement information obtained from a wireless ID tag and 3D model data can be conceived of.
However, in order to implement the Z buffer method, it is necessary to obtain 3D model information of an imaged space (depth information from a camera) beforehand, and this is a complicated procedure. More particularly, this is impractical when a 3D model changes over time.
It is an object of the present invention to provide a motion locus creation system, motion locus creation apparatus, and three-dimensional motion locus display apparatus that are capable of displaying a path of movement of an object to be tracked in an understandable way without using 3D model information.
One aspect of a motion locus creation system of the present invention is provided with: an imaging section that obtains a captured image of an area including an object to be tracked; a positioning section that positions the object to be tracked and outputs positioning data of the object to be tracked; a motion locus type selection section that selects a display type of a motion locus corresponding to each point in time according to whether or not the object to be tracked is shown in the captured image of each point in time; a motion locus creation section that forms motion locus data based on the positioning data and a motion locus display type selected by the motion locus type selection section; and a display section that displays an image based on the captured image and a motion locus based on the motion locus data in an overlapping manner.
One aspect of a motion locus creation apparatus of the present invention is provided with: a motion locus type selection section that selects a display type of a motion locus corresponding to each point in time according to whether or not an object to be tracked is shown in a captured image of each point in time; and a motion locus creation section that forms motion locus data based on positioning data of the object to be tracked and a motion locus display type selected by the motion locus type selection section.
One aspect of a three-dimensional motion locus display apparatus of the present invention is provided with: an imaging section that obtains a captured image including an object; a position detection section that obtains positioning data of the object having three-dimensional information composed of a horizontal-direction component, a depth-direction component, and a height-direction component; a motion locus generation section that is a section that generates a motion locus that is a path of movement of the object using the positioning data, and that generates a rounded motion locus for which a predetermined coordinate component relating to the positioning data is fixed at a constant value; and a display section that performs combined display of the captured image and the rounded motion locus on a two-dimensional display.
The present invention enables a motion locus creation system, motion locus creation apparatus, and three-dimensional motion locus display apparatus to be implemented that are capable of displaying a path of movement of an object to be tracked in an understandable way without using 3D model information.
Now, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following embodiments, cases are described in which an object to be tracked is a person, but an object to be tracked is not limited to a person, and may also be a vehicle or the like, for example.
Camera section 101 has imaging section 101-1 and image tracking section 101-2. Imaging section 101-1 captures an image of an area including an object to be tracked, and sends captured image S1 to display section 103 and image tracking section 101-2. Image tracking section 101-2 uses captured image S1 obtained at each point in time by imaging section 101-1 to track a person who is an object to be tracked. In this embodiment, for an image of each point in time, image tracking section 101-2 forms detection flag S2 indicating whether or not a person is being detected, and sends this detection flag S2 to data holding section 104.
Tag reader section 102 has a radio receiving section that receives a radio signal from a wireless tag, a positioning section that finds wireless tag position coordinates based on a received radio signal, and a coordinate conversion section that converts found position coordinates to XY coordinates on a display image. Tag reader section 102 sends converted wireless tag coordinate data S3 to data holding section 104.
An existing technology such as a three-point measurement method based on the field intensity of a radio signal received from a wireless tag, an arrival time/arrival direction estimation method, or the like, can be used by the positioning section of tag reader section 102 as a method of finding position coordinates. It is also possible to use a configuration in which a wireless tag itself incorporates a GPS or suchlike positioning function, and transmit its own positioning result to the radio receiving section of input interface section 120 as a radio signal. In this case, tag reader section 102 need not have a positioning section. Also, the above coordinate conversion section may be provided in data holding section 104 instead of being provided in tag reader section 102.
Data holding section 104 outputs detection flag S2-1 and coordinate data S3-1 of each point in time for an object to be tracked, together with timing. Detection flag S2-1 is input to motion locus type selection section 105, and coordinate data S3-1 is input to motion locus creation section 106.
Motion locus type selection section 105 determines whether or not an object to be tracked is in a concealed position at each point in time based on detection flag S2-1. Specifically, if detection flag S2-1 is ON (if an object to be tracked is being detected by camera section 101—that is, if an object to be tracked is shown in a captured image), it is determined that the object to be tracked is not in a concealed position. On the other hand, if detection flag S2-1 is OFF (if an object to be tracked is not being detected by camera section 101—that is, if an object to be tracked is not shown in a captured image), it is determined that the object to be tracked is in a concealed position.
Motion locus type selection section 105 forms motion locus type command signal S4 based on the determination result, and sends this to motion locus creation section 106. In this embodiment, motion locus type command signal S4 is formed that gives a “solid line” command if an object to be tracked is shown, and gives a “dotted line” command if an object to be tracked is not shown.
Motion locus creation section 106 forms motion locus data S5 by connecting coordinate data S3-1 of each point in time. At this time, motion locus creation section 106 forms motion locus data S5 by selecting a motion locus type for each line segment based on motion locus type command signal S4. Motion locus data S5 is sent to display section 103.
Display section 103 performs overlapping display of an image based on captured image S1 input from camera section 101 and a motion locus based on motion locus data S5 input from motion locus creation section 106. By this means, a motion locus that is a path of an object to be tracked is displayed superimposed on an image captured by camera section 101.
The operation of this embodiment will now be described.
In step ST13, image tracking section 101-2 determines whether or not a person has been able to be detected. If a person has been able to be detected, the processing flow proceeds to step ST14, and tracking status data with detection flag S2 ON is output. On the other hand, if a person has not been able to be detected, the processing flow proceeds to step ST15, and tracking status data with detection flag S2 OFF is output.
Next, camera section 101 waits for a predetermined time by performing timer processing in step ST16, and then returns to step ST11. The wait time in the timer processing in step ST16 can be set according to the speed of movement of an object to be tracked, for instance. For example, the imaging interval can be shortened by setting a shorter wait time for a faster speed of movement of an object to be tracked.
The wait time set in step ST34 may be matched to a wireless tag positioning time interval (an interval at which coordinate data S3 of each point in time is output from tag reader section 102), or may be made a fixed time set beforehand. Normally, the imaging interval of camera section 101 is shorter than a wireless tag positioning interval, and therefore it is desirable for the wait time to be set to a fixed time greater than or equal to a wireless tag positioning time interval.
As explained above, according to this embodiment, camera section 101 forms detection flag (tracking status data) S2 indicating whether or not an object to be tracked has been able to be detected from captured image S1, motion locus type selection section 105 decides a motion locus display type based on detection flag S2, and motion locus creation section 106 creates a motion locus based on coordinate data S3 obtained by tag reader section 102 and motion locus type command signal S4 decided by motion locus type selection section 105. By this means, a motion locus can be displayed that clearly indicates whether an object to be tracked has moved in front of object 110 or has moved behind object 110, and an easily understandable path of movement can be displayed, without using 3D model information.
In this embodiment, a case has been described in which a path of movement is formed by means of only coordinate data obtained by tag reader section 102, but a path of movement may also be found using coordinate data obtained by image tracking section 101-2 in a complementary manner.
In this embodiment, a preferred aspect is presented for a case in which the basic components of the configuration described in Embodiment 1 are maintained and there are additionally a plurality of objects to be tracked.
Camera section 201 has imaging section 201-1 and imaging coordinate acquisition section 201-2. Imaging section 201-1 captures an image of an area including an object to be tracked, and sends captured image S10 to image holding section 210 and imaging coordinate acquisition section 201-2. Image holding section 210 temporarily holds captured image S10, and outputs captured image S10-1 whose timing has been adjusted to display section 203.
Imaging coordinate acquisition section 201-2 acquires the coordinates of a person who is an object to be tracked using captured image S10 obtained at each point in time by imaging section 201-1. Imaging coordinate acquisition section 201-2 sends coordinate data of a person detected in an image of each point in time to data holding section 204 as imaging coordinate data S11. If there are a plurality of detected persons, imaging coordinate acquisition section 201-2 tracks a plurality of persons, and outputs imaging coordinate data S11 for a plurality of persons.
Tag reader section 202 has a radio receiving section that receives information by radio from a wireless tag. Tag reader section 202 has a positioning function that finds wireless tag position coordinates based on a received radio signal, and a tag ID receiving function. In the same way as described in Embodiment 1, a configuration may also be used in which a positioning function is incorporated in a wireless tag itself, and tag reader section 202 receives a positioning result. Tag reader section 202 sends a wireless tag's tag coordinate data S12 and tag ID data S13 as a pair to data holding section 204.
By this means, imaging coordinate data S11, tag ID data S13, and tag coordinate data S12 corresponding to a tag ID, are stored in data holding section 204. If there are a plurality of persons who are objects to be tracked, a plurality of imaging coordinate data S11, a plurality of tag ID data S13, and a plurality of tag coordinate data S12 corresponding to the respective tag IDs, are stored at each point in time.
Data integration section 211 reads data stored in data holding section 204, and performs person integration and coordinate integration. Person integration means integrating corresponding persons' imaging coordinates and tag coordinates from among imaging coordinates and tag coordinates of a plurality of persons. At this time, data integration section 211 can integrate corresponding persons' imaging coordinates and tag coordinates by, for example, identifying a person corresponding to each set of imaging coordinates using a person image recognition method, and linking together an identified person and a tag ID. Also, items with mutually close coordinates between imaging coordinates and tag coordinates may be integrated as imaging coordinates and tag coordinates of a corresponding person.
Data integration section 211 integrates imaging coordinates and tag coordinates as XY plane coordinates by further normalizing imaging coordinates and tag coordinates. Here, normalization includes processing for interpolation using tag coordinates if imaging coordinates are missing, using both imaging coordinates and tag coordinates of a corresponding person. Integrated and normalized coordinate data S14 of each person is sent to motion locus creation section 206 via data holding section 204.
Motion locus creation section 206 creates motion locus vector data S15 indicating tracking results up to the present time by sequentially connecting a vector from coordinates of a previous point in time to coordinates of the next point in time, and sends this motion locus vector data S15 to motion locus type selection section 205.
Motion locus type selection section 205 has motion locus vector data S15 and imaging coordinate data S11-1 as input. Motion locus type selection section 205 performs motion locus vector division on a fixed section basis, and determines a motion locus type indicated by a motion locus vector for each section according to whether or not there is imaging coordinate data S11-1 corresponding to each section. Motion locus type selection section 205 sends motion locus data S16 including a motion locus vector and section-specific motion locus vector type information to display section 203.
Specifically, if there is imaging coordinate data S11-1 corresponding to a motion locus vector, motion locus type selection section 205 determines that an object to be tracked is present in front of an object, and outputs motion locus data S16 directing that a motion locus indicated by the motion locus vector is to be displayed as a “solid line.” On the other hand, if there is no imaging coordinate data S11-1 corresponding to a motion locus vector, motion locus type selection section 205 determines that an object to be tracked is present behind an object, and outputs motion locus data S16 directing that a motion locus indicated by the motion locus vector is to be displayed as a “dotted line.”
The above-described processing by motion locus creation section 206 and motion locus type selection section 205 is performed for each person who is an object to be tracked.
In step ST42, it is determined whether or not there are imaging coordinates, using imaging coordinate data S11-1 of a period corresponding to the set motion locus vector section. If there are no imaging coordinates, the processing flow proceeds to step ST45-4, it is determined that a person is in a concealed position, and in step ST46-4 a motion locus indicated by the relevant motion locus vector is displayed as a “dotted line.” On the other hand, if there are imaging coordinates, the processing flow proceeds from step ST42 to step ST43.
In step ST43, it is determined whether or not a proportion for which imaging coordinates have been able to be acquired is greater than or equal to a threshold value, using imaging coordinate data S11-1 of a period corresponding to the set motion locus vector section. If a proportion for which imaging coordinates have been able to be acquired is greater than or equal to the threshold value, the processing flow proceeds to step ST45-3, it is determined that a person can be seen in the video, and in step ST46-3 a motion locus indicated by the relevant motion locus vector is displayed as a “solid line.” On the other hand, if a proportion for which imaging coordinates have been able to be acquired is less than the threshold value, the processing flow proceeds from step ST43 to step ST44.
In step ST44, it is determined whether or not imaging coordinates are missing consecutively, using imaging coordinate data S11-1 of a period corresponding to the set motion locus vector section. Here, “imaging coordinates are missing consecutively” means a case in which captured images in which an object to be tracked is not shown continue for at least threshold value th (where th≧2). If imaging coordinates are missing consecutively, the processing flow proceeds to step ST45-2, it is determined that a person is in a concealed position, and in step ST46-2 a motion locus indicated by the relevant motion locus vector is displayed as a “dotted line.” On the other hand, if imaging coordinates are not missing consecutively, the processing flow proceeds from step ST44 to step ST45-1, it is determined that a person can be seen in the video (it is determined that imaging coordinate data S11-1 has not been obtained due to an imaging failure or a person detection (tracking) failure), and in step ST46-1 a motion locus indicated by the relevant motion locus vector is displayed as a “solid line.”
After the processing in steps ST46-1 through ST46-4, motion locus type selection section 205 proceeds to step ST47, sets the next section as a motion locus vector section for determination (sets section=section+1), and returns to step ST42.
Thus, by determining a motion locus type comprehensively based on the presence or absence of imaging coordinates for each section and the proportion of missing imaging coordinates in steps ST42, ST43, and ST44, motion locus type selection section 205 can avoid erroneously determining that a section for which there has been an imaging coordinate acquisition failure is a “concealed position” section. By this means, accurate motion locus type selection can be performed.
In this embodiment, a case has been described in which a motion locus type is selected by means of three-step processing comprising steps ST42, ST43, and ST44, but a motion locus type may also be selected by means of two-step processing comprising any two of steps ST42, ST43, and ST44, or by means of one-step processing using one of steps ST43 and ST44.
As described above, according to this embodiment, even if there are a plurality of objects to be tracked at the same point in time, the provision of data integration section 211 enables a motion locus to be created for each object to be tracked.
Also, by determining that an object to be tracked is not shown in a captured image only if captured images in which an object to be tracked is not shown continue for at least threshold value th (where th≧2), it is possible to avoid erroneously determining that a section for which there has been an imaging coordinate acquisition failure is a “concealed position” section. Similarly, by determining that an object to be tracked is not shown in a captured image only if the ratio of the number of captured images in which an object to be tracked is not shown, to the total number of a temporally consecutive plurality of captured images, is greater than or equal to a threshold value, it is possible to avoid erroneously determining that a section for which there has been an imaging coordinate acquisition failure is a “concealed position” section.
In this embodiment and following Embodiments 4 through 8, three-dimensional motion locus display apparatuses are presented that show a user a three-dimensional motion locus in an understandable way by improving visibility when a motion locus having three-dimensional information is displayed on a two-dimensional image.
The present inventors considered visibility when a motion locus having three-dimensional information is displayed on a two-dimensional image.
In Patent Literature 1, for example, a technology is disclosed whereby a path of movement of an object detected using image recognition processing is displayed combined with a camera image.
Three-dimensional coordinates of an object are assumed to be represented by the coordinate axes shown in
The technology disclosed in Patent Literature 1 displays a two-dimensional path of movement in an object camera image (screen) combined with the camera image, and does not display a three-dimensional path of movement that includes depth-direction movement viewed from a camera. Therefore, if an object is hidden in a concealed position or objects overlap each other, for instance, a path of movement is cut off midway, and a path of movement of an object cannot be adequately ascertained.
On the other hand, Patent Literature 3 discloses a display method devised so that a path of movement of an object can be seen three-dimensionally. Specifically, in Patent Literature 3, depth-direction movement of an object is represented by displaying a trajectory of motion of an object (particle) in ribbon form and performing hidden-surface removal processing.
If a path of movement of an object having three-dimensional information is displayed combined with a camera image, a user can be shown a more detailed path of movement, and therefore implementation of such a display apparatus is desirable.
However, heretofore, the visibility of a display image on a two-dimensional display when displaying a three-dimensional path of movement combined with a camera image has not been sufficiently considered.
The present inventors investigated traditional problems associated with displaying a three-dimensional path of movement combined with a camera image on a two-dimensional display. The results of this investigation are described below using
In the case of such an example, it is difficult to see from the path of movement display image whether movement of object OB1 is in the depth direction or in the height direction.
That is to say, when only motion locus L0 is displayed, as shown in
Furthermore, when a positioning result includes height-direction error (error occurs, for example, according to the attaching position or radio wave environment of a wireless tag in positioning that uses a wireless tag), a user cannot discern whether displacement of a motion locus in the vertical direction of the screen is due to a movement of object OB1 in the height direction, a movement of object OB1 in the depth direction, or height-direction positioning error, and it becomes still more difficult to ascertain the movement of an object from a path of movement.
Incidentally, although the technology disclosed in Patent Literature 3 does not originally presuppose display of a path of movement combined with a camera image, if superimposition of a path of movement on a camera image by means of a ribbon is assumed, the image will be hidden by the ribbon, and there is a possible problem of simultaneous recognition of a camera image and motion locus being impeded.
This embodiment and following Embodiments 4 through 8 are based on the above considerations.
Before describing the configuration of this embodiment, display images created and displayed by a three-dimensional motion locus display apparatus of this embodiment will first be described.
(ii)
(iii)
(iv)
(v) In this embodiment, it is proposed, as a preferred example, that threshold value based determination should be performed for an amount of fluctuation per unit time of a horizontal-direction component or height-direction component in positioning data, rounded motion locus L1 should be displayed if the amount of fluctuation is greater than or equal to the threshold value, and original motion locus L0 should be displayed if the amount of fluctuation is less than the threshold value. By this means, it is possible to display rounded motion locus L1 only if visibility actually degrades when original motion locus L0 is displayed.
(vi) In this embodiment, it is proposed, as a preferred example, that rounded motion locus L1, original motion locus L0 on which rounding processing is not performed, and lines connecting corresponding points on rounded motion locus L1 and original motion locus L0 (dotted lines in the drawings) should be displayed simultaneously, as shown in
The configuration of a three-dimensional motion locus display apparatus that creates and displays an above-described display image will now be described.
Imaging apparatus 310 is a video camera comprising a lens, imaging element, moving image encoding circuitry, and so forth. Imaging apparatus 310 may be a stereo video camera. There are no particular restrictions on the encoding method, and MPEG2, MPEG 4, MPEG4/AVC (H.264), or the like can be used, for example.
Position detection section 320 obtains positioning data of an object having three-dimensional information comprising a horizontal-direction component, depth-direction component, and height-direction component, by measuring by means of a radio wave a three-dimensional position of a wireless tag attached to the object. If imaging apparatus 310 is a stereo camera, position detection section 320 may measure a three-dimensional position of an object from stereoscopic parallax of captured images obtained by imaging apparatus 310. Position detection section 320 may also measure a three-dimensional position of an object using radar, infrared radiation, ultrasound, or the like. Essentially, position detection section 320 may be any kind of apparatus as long as it can obtain object positioning data having three-dimensional information comprising a horizontal-direction component, depth-direction component, and height-direction component.
Image receiving section 331 receives a captured image (moving image data) output from imaging apparatus 310 in real time, and outputs moving image data to image playback section 333 in accordance with a request from image playback section 333. Image receiving section 331 also outputs received moving image data to image storage section 332. If there is a restriction on the storage capacity of image storage section 332, for instance, image receiving section 331 may initially decode received moving image data, and output moving image data that has been re-encoded by means of an encoding method with higher compression efficiency to image storage section 332.
Image storage section 332 stores moving image data output from image receiving section 331, and also outputs moving image data to image playback section 333 in accordance with a request from image playback section 333.
Image playback section 333 decodes moving image data obtained from image receiving section 331 or image storage section 332 in accordance with a user command (not shown) from input apparatus 340 received via input receiving section 338, and outputs decoded moving image data to display apparatus 350.
Display apparatus 350 is a two-dimensional display that performs combined display of an image based on moving image data and a motion locus based on motion locus data obtained from motion locus generation section 337.
Position storage section 334 stores position detection results (positioning data) output from position detection section 320 as a position history. A time, an object ID, and position coordinates (x, y, z) are stored as one record. That is to say, position coordinates (x, y, z) of each time are stored in position storage section 334 for each object.
Imaging condition acquisition section 336 acquires imaging apparatus 310 PTZ (pan/tilt/zoom) information from imaging apparatus 310 as imaging condition information. If imaging apparatus 310 is movable, imaging condition acquisition section 336 receives changed imaging condition information each time imaging conditions change, and holds changed imaging condition information together with change time information as a history.
Position fluctuation determination section 335 is used when selecting whether or not a rounded motion locus is to be displayed according to an amount of fluctuation, as in (v) above. In response to an inquiry from motion locus generation section 337, position fluctuation determination section 335 extracts a plurality of records relating to the same ID within a fixed time from a position history stored in position storage section 334, calculates a height-direction (z-direction) component fluctuation range (difference between a maximum value and minimum value) in the screen, and determines whether or not the fluctuation range is greater than or equal to a threshold value. At this time, position fluctuation determination section 335 first converts position history coordinates (x, y, z) to a camera field-of-view coordinate system using imaging conditions (information relating to imaging apparatus 310 PTZ) acquired from imaging condition acquisition section 336, and then calculates an object's height-direction (z-direction) fluctuation range, and performs threshold value based determination for the calculation result. In the same way as when performing horizontal-direction (x-direction) determination, position fluctuation determination section 335 can calculate a horizontal-direction fluctuation range using a horizontal-direction (x-direction) coordinate converted to a camera field-of-view coordinate system, and perform threshold value based determination for the calculation result. It goes without saying that the above-described coordinate conversion is unnecessary if a coordinate system height-direction (z-direction) or horizontal-direction (x-direction) coordinate axis represented by a position detection section 320 positioning result matches a coordinate system height-direction or horizontal-direction coordinate axis in the camera field-of-view coordinate system.
Input apparatus 340 is an apparatus such as a mouse or suchlike pointing device, a keyboard, or the like, that inputs user operations.
Input receiving section 338 receives a user operation input signal from input apparatus 340, acquires user apparatus information such as a mouse (pointing device) position, drag amount, wheel rotation amount, or click event, a number of keyboard (arrow key) depressions, or the like, and outputs this information.
Motion locus generation section 337 receives an event corresponding to the start of motion locus generation (period specification information specifying a period for which motion locus display is to be performed from among past images, or a command event specifying that real-time motion locus display is to be performed, by means of a mouse click, menu selection, or the like) from input receiving section 338.
Motion locus generation processing by motion locus generation section 337 differs according to the motion locus display method, and therefore motion locus generation section 337 motion locus generation processing for each display method is described separately below. In this embodiment, a method whereby a rounded motion locus resulting from fixing a height-direction component of an object at a constant value is displayed, such as shown in
[1] When the display described in (v) above is performed (that is, when a rounded motion locus is generated only if an amount of fluctuation per unit time of a horizontal-direction component or height-direction component is greater than or equal to a threshold value)
In this case, motion locus generation processing is broadly divided into processing when a motion locus corresponding to a past image is displayed, and processing when a motion locus corresponding to a real-time image is displayed, and therefore these two cases are described separately below.
Motion locus generation section 337 issues an inquiry to position fluctuation determination section 335 as to whether or not a fluctuation range in period T specified by period specification information is greater than or equal to a reference value, and receives the determination result as input. If a determination result indicating that the fluctuation range is greater than or equal to the threshold value is input from position fluctuation determination section 335, motion locus generation section 337 converts position history data (x(t), y(t), z(t)) of period T read from position storage section 334 to motion locus coordinate data for displaying a rounded motion locus. On the other hand, if a determination result indicating that the fluctuation range is less than the threshold value is input from position fluctuation determination section 335, motion locus generation section 337 uses position history data (x(t), y(t), z(t)) of period T read from position storage section 334 directly as motion locus coordinate data.
That is to say, if it is determined by motion locus generation section 337 that a z-direction fluctuation range (height-direction fluctuation range) is greater than or equal to the threshold value, motion locus generation section 337 obtains motion locus coordinate data for displaying a rounded motion locus by converting coordinate data (x(t), y(t), z(t)) so that (x(t), y(t), z(t))→(x(t), y(t), A), and tεT, where A is a predetermined value. If A=0 is set at this time, rounded motion locus L1 fixed to the floor can be generated as shown in
Lastly, motion locus generation section 337 generates motion locus data by connecting coordinate points indicated by the motion locus coordinate data, and outputs this to display apparatus 350. Motion locus generation section 337 may also generate motion locus data by performing curve interpolation of a polygonal line by means of spline interpolation or the like.
Motion locus generation section 337 reads the latest record for time T1 for which a command event has been received from the position storage section 334 position history, and starts motion locus generation. Provision may also be made for motion locus generation section 337 not to perform coordinate conversion processing according to a fluctuation range, but to generate motion loci sequentially in real time by issuing an inquiry to position fluctuation determination section 335 as to a fluctuation range for period T1 to R2 at point in time T2 after the elapse of a fixed period, and performing the same kind of processing as “when a motion locus corresponding to a past image is displayed” described above according to the determination result.
[2] When the display described in (vi) above is performed
Motion locus generation section 337 generates rounded motion locus data connecting coordinate points at which a horizontal-direction component (x-direction component) or height-direction component (z-direction component) is fixed at a constant value, original motion locus data connecting position history data coordinate points directly, and link segment data linking corresponding points on a rounded motion locus and original motion locus, and outputs these data to display apparatus 350.
Furthermore, motion locus generation section 337 varies the height of a rounded motion locus by varying the value of A in (x(t), y(t), z(t))→(x(t), y(t), A), tεT, in proportion to a degree of user operation, such as an amount of movement of a mouse wheel, acquired from input receiving section 338. By this means, the fluctuation range of the height of a rounded motion locus in the screen is greater toward the front (that is, toward the camera) and decreases progressively toward the rear (that is, farther away from the camera), so that an observer recognizing that a rounded motion locus is fixed in a plane can obtain a sense of pseudo-stereoscopic parallax (a sense of parallax increasing nearer the observer and decreasing as the distance from the observer increases), and can more accurately grasp the nature of a rounded motion locus extending in the depth direction.
Also, if motion loci of a plurality of objects are displayed at this time, a user may move the height of only the motion locus of an object specified using a GUI (Graphical User Interface) or the like. By this means, which motion locus is the motion locus of the specified object can be easily recognized.
As described above, according to this embodiment, by performing combined display of a rounded motion locus for which a predetermined coordinate component relating to positioning data of object OB1 is fixed at a constant value, and a captured image, motion loci can be presented in which height-direction (z-direction) movement of object OB1 and depth-direction (y-direction) movement of object OB1 are separated, enabling a user to distinguish between height-direction (z-direction) movement of object OB1 and depth-direction (y-direction) movement of object OB1 by means of rounded motion locus L1. By this means, according to this embodiment, three-dimensional motion locus display apparatus 300 can be implemented that enables an observer to easily grasp three-dimensional movement of an object, and enables visibility to be improved for an observer.
In this embodiment, selection of whether or not the motion locus rounding processing described in Embodiment 3 is to be performed is based on the relationship between a line-of-sight vector of imaging apparatus (camera) 310 and a movement vector of object OB1.
It is difficult to discern whether an original motion locus close to parallel to camera 310 line-of-sight vector CV is a movement in the depth direction (y direction) or a movement in the height direction (z direction). Focusing on this point, in this embodiment this kind of rounding processing described in Embodiment 3 is performed on an original motion locus close to parallel to line-of-sight vector CV.
If the absolute value of the inner product of vector Ucv resulting from normalizing line-of-sight vector CV and vector Uv resulting from normalizing movement vector V is greater than or equal to a predetermined value, line-of-sight vector CV and an original motion locus are determined to be close to parallel. A value such as 1/√2, for example, can be used as the predetermined value.
That is to say, when Ucv=CV/|CV| and Uv=V/|V|, if |Ucv·Uv|≧α (where α is a predetermined value), line-of-sight vector CV and an original motion locus are determined to be close to parallel, and a rounded motion locus is generated and displayed.
On the other hand, if the absolute value of the inner product of vector Ucv resulting from normalizing line-of-sight vector CV and vector Uv resulting from normalizing movement vector V is smaller than a predetermined value, line-of-sight vector CV and an original motion locus are determined to be close to perpendicular.
That is to say, when Ucv=CV/|CV| and Uv=V/|V|, if |Ucv·Uv|<α (where α is a predetermined value), line-of-sight vector CV and an original motion locus are determined to be close to perpendicular, rounding processing is not performed, and the original motion locus is generated and displayed.
Movement vector determination section 411 receives an inquiry from motion locus generation section 412 (motion locus generation period information, or the like), and acquires imaging condition information (imaging apparatus 310 PTZ information) from imaging condition acquisition section 336 according to this inquiry. Movement vector determination section 411 calculates an imaging apparatus 310 line-of-sight vector (taking the vector magnitude as 1). Movement vector determination section 411 also acquires position history data for the relevant period from position storage section 334, and calculates a movement vector that is a vector between position coordinates (taking the vector magnitude as 1). As described above, movement vector determination section 411 performs threshold value based determination of the absolute value of the inner product of the line-of-sight vector and movement vector, and outputs the determination result to motion locus generation section 412.
If the absolute value of the inner product is greater than or equal to a threshold value, motion locus generation section 412 generates a rounded motion locus for which a height-direction component of position history data is fixed at a constant value, whereas if the absolute value of the inner product is less than the threshold value, motion locus generation section 412 does not perform rounding processing, and generates an original motion locus using position history data directly.
As described above, according to this embodiment, a motion locus for which rounding processing should be performed can be determined accurately.
In this embodiment, whether or not rounding processing is to be performed has been determined by performing threshold value based determination of the absolute value of the inner product of imaging apparatus 310 line-of-sight vector CV and object OB1 movement vector V, but whether or not rounding processing is to be performed may also be determined by performing threshold value based determination of the absolute value of the angle between a straight line parallel to line-of-sight vector CV and a straight line parallel to movement vector V. Specifically, if this angle is less than a threshold value, a rounded motion locus is generated for which a height-direction component in positioning data, or a height-direction component when direction components in positioning data are converted to the field-of-view coordinate system of imaging apparatus 310, is fixed at a constant value, whereas if the angle is greater than or equal to the threshold value, an original motion locus for which rounding processing is not performed is generated.
If auxiliary plane F1 is made semi-transparent and hidden-surface processing is executed on an imaging object as shown in
Auxiliary plane generation section 511 generates auxiliary plane F1 as a plane in which a rounded motion locus is present in accordance with rounded motion locus position information output from motion locus generation section 337. At this time, auxiliary plane generation section 511 issues an inquiry to environmental data storage section 512 and acquires three-dimensional position information relating to environmental objects (walls, pillars, furniture and fixtures, and so forth), and issues an inquiry to imaging condition acquisition section 336 and acquires imaging apparatus 310 PTZ information. Then auxiliary plane generation section 511 determines the anteroposterior relationship between auxiliary plane F1 and environmental objects, and performs auxiliary plane F1 hidden-surface processing.
Environmental data storage section 512 stores three-dimensional position information such as position information on walls, pillars, and suchlike architectural structures present within the object detection and imaging ranges of position detection section 320 and imaging apparatus 310, information on the layout of furniture and fixtures within these ranges, and so forth. Environmental data storage section 512 outputs this three-dimensional environmental information in response to an auxiliary plane generation section 511 inquiry.
In response to an inquiry (specification period) from head position fluctuation determination section 612, head position detection section 611 acquires moving image data from image receiving section 331 or image storage section 332, detects a head position of an object when the object is a person by analyzing this data, and outputs the detection result to head position detection section 611. This head position detection can be implemented by means of known image recognition technology such as described in Non-Patent Literature 2, for example, and therefore a description thereof is omitted here.
In response to an inquiry (specification period) from motion locus generation section 613, head position fluctuation determination section 612 issues an inquiry to head position detection section 611 and acquires head positions for the relevant period, and calculates the fluctuation range of the head position z-coordinate (height direction in the screen). Specifically, the fluctuation range is calculated from the average height of the head position. Head position fluctuation determination section 612 determines whether or not the head position fluctuation range in the relevant period is greater than or equal to a predetermined threshold value, and outputs the determination result to motion locus generation section 613.
If a determination result indicating that the head position fluctuation range is greater than or equal to the threshold value is input from head position fluctuation determination section 612, motion locus generation section 613 converts period T position history data (x(t), y(t), z(t)) read from position storage section 334 so that, when the average head position for that period T is designated H, (x(t), y(t), z(t))→(x(t), y(t), H), and tεT. On the other hand, if a determination result indicating that the head position fluctuation range is less than the threshold value is input from head position fluctuation determination section 612, motion locus generation section 613 performs conversion so that (x(t), y(t), z(t))→(x(t), y(t), A), and tεT, where, for example, A is the floor height (A=0).
Abnormal section extraction section 711 detects abnormal behavior of an object from a position history stored in position storage section 334, a captured image captured by imaging apparatus 310, or the like, extracts a position history record relating to the section in which abnormal behavior was detected, and outputs this record to motion locus generation section 713. Three examples of an abnormal section extraction method are given below, but abnormal section extraction methods are not limited to these.
(1) A standard motion locus of an object is set and held beforehand, and an abnormality is detected by comparison with the standard motion locus. (2) An Off Limits section to which entry by an object is prohibited is set and held beforehand, and whether or not an object has entered the Off Limits section is detected. (3) An abnormality is detected by performing image recognition using an image captured by imaging apparatus 310.
Operating screen generation section 712 generates an auxiliary operating screen that includes person icons for setting the heights of motion loci of each object (person) and check boxes for performing display/non-display switching. Operating screen generation section 712 generates an auxiliary operating screen in which a position of a person icon is moved and/or a check box on/off status is switched according to a mouse position, click event, mouse drag amount, or the like, output from input receiving section 338. Processing by this operating screen generation section 712 is similar to known GUI operating window generation processing.
According to the method proposed in this embodiment, a motion locus can be presented that gives a pseudo-sense of depth without obscuring a captured image, even if a rounded motion locus is not used. That is to say, it is sufficient to generate original motion locus L0 and an auxiliary motion locus that performs a circular motion around original motion locus L0 with a moving radius perpendicular to object movement vector V, and to display these. Provision may also be made for a rounded motion locus and an auxiliary motion locus that performs a circular motion around the rounded motion locus with a moving radius perpendicular to object movement vector V to be generated, and for these to be displayed.
Such an auxiliary motion locus can be displayed by generating an auxiliary motion locus that performs a circular motion with a moving radius perpendicular to a movement vector (a vector from a certain motion locus coordinate point toward the next motion locus coordinate point) when motion locus data is generated by motion locus generation section 337, 412, 613, or 713, outputting this auxiliary motion locus to display apparatus 350. If motion locus interpolation is performed by means of a spline curve or the like, an auxiliary motion locus may be made a spline curve that performs a circular motion with a moving radius perpendicular to a spline curve after interpolation.
In above Embodiments 1 and 2, cases have been described in which coordinate data from a wireless tag is acquired using tag reader sections 102 and 202 respectively, but the present invention is not limited to this, and various positioning means capable of positioning an object to be tracked can be applied instead of tag reader section 102/202. Possible positioning means for replacing tag reader section 102/202 include radar, an ultrasound sensor, a camera, or the like, provided in a position allowing an object to be tracked to be positioned when an object to be tracked enters a concealed position as viewed from camera section 101/201. Also, in an indoor situation, an object to be tracked may be positioned by providing numerous sensors on the floor. The essential point is that, when an object to be tracked enters a concealed position as viewed from camera section 101/201, the positioning means should be able to pinpoint that position.
Also, in above Embodiments 1 and 2, cases have been described in which camera sections 101 and 201 have image tracking section 101-2 and imaging coordinate acquisition section 201-2 respectively, and motion locus type selection sections 105 and 205 determines a motion locus display type based on tracking status data (a detection flag) and imaging coordinate data obtained by image tracking section 101-2 and imaging coordinate acquisition section 201-2 respectively, but the present invention is not limited to this, the essential point being that the display type of a motion locus corresponding to each point in time should be selected according to whether or not an object to be tracked is shown in a captured image at each point in time.
Furthermore, in above Embodiments 1 and 2, cases have been described in which “solid line” is selected as a motion locus type when an object to be tracked is determined not to be in a concealed position, and “dotted line” is selected as a motion locus type when an object to be tracked is determined to be in a concealed position, but the present invention is not limited to this, and provision may also be made, for example, for a “thick line” to be selected as a motion locus type when an object to be tracked is determined not to be in a concealed position, and for a “thin line” to be selected as a motion locus type when an object to be tracked is determined to be in a concealed position. Alternatively, the color of a motion locus may be changed according to whether an object to be tracked is determined not to be in a concealed position or is determined to be in a concealed position. The essential point is that different motion locus display types should be selected when an object to be tracked is not in a concealed position, and when an object to be tracked is in a concealed position, as viewed from the camera section.
A motion locus display type may also be changed according to the status of a wireless tag. For example, if the color of a motion locus is changed when information indicating a low battery level is received from a wireless tag, a user can learn that the battery level is low from the motion locus color, and can take this as a guideline for a battery change.
It is possible for image tracking section 101-2, imaging coordinate acquisition section 201-2, motion locus type selection sections 105 and 205, and motion locus creation sections 106 and 206, used in Embodiments 1 and 2 respectively, to be implemented by means of a general-purpose computer such as a personal computer, with the processing included in image tracking section 101-2, imaging coordinate acquisition section 201-2, motion locus type selection section 105/205, and motion locus creation section 106/206 being implemented by reading software programs corresponding to the processing of each processing section stored in computer memory, and having these programs executed by a CPU.
Similarly, it is possible for display motion locus generation apparatuses 330, 410, 510, 610, and 710 used in Embodiments 3 through 8 respectively to be implemented by means of a general-purpose computer such as a personal computer, with the processing included in motion locus generation apparatuses 330, 410, 510, 610, and 710 being implemented by reading software programs corresponding to the processing of each processing section stored in computer memory, and having these programs executed by a CPU. Motion locus generation apparatuses 330, 410, 510, 610, and 710 may also be implemented by means of a dedicated device incorporating LSI chips corresponding to each processing section.
The disclosures of Japanese Patent Application No. 2008-268687, filed on Oct. 17, 2008, and Japanese Patent Application No. 2009-018740, filed on Jan. 29, 2009, including the specifications, drawings and abstracts, are incorporated herein by reference in their entirety.
The present invention is suitable for use in a system that displays a path of movement of a person or object by means of a motion locus, such as a surveillance system, for example.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2008-268687 | Oct 2008 | JP | national |
| 2009-018740 | Jan 2009 | JP | national |
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/JP2009/004293 | 9/1/2009 | WO | 00 | 4/12/2011 |
