Patent Application
20250157177
The present disclosure relates to a search assistance device, a search assistance system, a search assistance method, and a program.
In the related art, there are known technologies for searching for objects or people in remote locations. For example, Patent Document 1 discloses a technology for imaging a subject using a multi-lens camera and applying a vector median filter to remove unwanted objects from an image in which characters are partially obstructed. Patent Document 2 discloses a technology for extracting a target person from an image and using images taken at a plurality of points to specify the person even in a case where part of the person's image is missing due to an obstacle. Non Patent Document 1 discloses a technology for improving the visibility of a target object by integrating images from a visible light camera and an invisible light camera.
The above-mentioned technologies use images captured by a multi-lens camera, images taken at a plurality of points, or a plurality of types of images to search for target objects in remote locations, but there are cases where only images captured in specific directions can be acquired due to limitations on the number of cameras and the like in remote locations. In a case where a search target object is hidden by a peripheral object in an image captured in a specific direction, there is a likelihood that the search target object cannot be detected.
The present disclosure has been made to solve the above-mentioned problems, and an object of the present disclosure is to provide a search assistance device, a search assistance system, a search assistance method, and a program that are capable of detecting a search target object with a high probability even if the search target object is hidden by a peripheral object.
According to a first aspect, there is provided a search assistance device including: a captured video acquisition unit configured to acquire a captured video; an object information acquisition unit configured to acquire object information indicating an object included in the captured video; a setting unit configured to set a search target image including a search target object; a determination unit configured to determine whether or not part of a search target candidate is hidden by a peripheral object using the search target image or the object information; and a video creation unit configured to create a comparison video to be compared to search for the search target object using an image that shows the peripheral object in the captured video in a case where the determination unit determines that part of the search target candidate is hidden by the peripheral object.
According to a second aspect, there is provided a search assistance method including: a step of setting a search target image including a search target object; a step of acquiring a captured video and object information indicating an object included in the captured video; a step of determining whether or not part of a search target candidate is hidden by a peripheral object using the search target image or the object information; and a step of creating a comparison video to be compared to search for the search target object using an image that shows the peripheral object in the captured video in a case where it is determined that part of the search target candidate is hidden by the peripheral object.
According to a third aspect, there is provided a program causing a computer mounted on a search assistance device to execute: a step of setting a search target image including a search target object; a step of acquiring a captured video and object information indicating an object included in the captured video; a step of determining whether or not part of a search target candidate is hidden by a peripheral object using the search target image or the object information; and a step of creating a comparison video to be compared to search for the search target object using an image that shows the peripheral object in the captured video in a case where it is determined that part of the search target candidate is hidden by the peripheral object.
According to a fourth aspect, there is provided a search assistance system including: an imaging device configured to generate a captured video; an object information generation device configured to generate object information indicating an object included in the captured video; and a search assistance device including a setting unit configured to set a search target image including a search target object, a determination unit configured to determine whether or not part of a search target candidate is hidden by a peripheral object using the search target image or the object information, and a video creation unit configured to create a comparison video to be compared to search for the search target object using an image that shows the peripheral object in the captured video in a case where the determination unit determines that part of the search target candidate is hidden by the peripheral object.
According to the present disclosure, a search target object can be detected with a high probability even if the search target object is hidden by a peripheral object.
Embodiments of the present disclosure will be described below with reference to the drawings. The scope of the present disclosure is not limited to the following embodiments and can be changed in any way within the scope of technical ideas of the present disclosure. For example, images in the embodiments may be replaced with videos, and videos in the embodiments may be replaced with images. In addition, in the following drawings, the scale and the number in each structure may be different from the scale and the number in the actual structure to facilitate understanding of each configuration.
The search assistance device 100 performs processing to assist the search of a search target object. The search assistance device 100 may be configured, for example, to be connected to a plurality of camera systems 200 and a plurality of searcher's terminal devices 300 and to provide a cloud service that assists the search of a search target object. The search target object may be, for example, an item lost in a store, but the present disclosure is not limited thereto and the search target object may be a person such as a lost child.
The searcher's terminal device 300 is an information processing device such as a personal computer or a smartphone that is operated by a searcher who searches for a search target object. The searcher's terminal device 300 includes an operation interface that receives operations by the searcher, a display interface that presents various types of information to the searcher, a storage device, and a control unit such as a central processing unit (CPU) that performs various processes.
The camera system 200 is provided in the search area. The search area is, for example, a store such as a shopping mall or a restaurant, but the present disclosure is not limited thereto. The camera system 200 includes a plurality of camera devices 210, an object information generation device 216, and a control unit 220. The object information generation device 216 is, for example, an infrared camera device 212 and a radar device 214. The camera device 210 generates a captured video of the search area. The camera device 210 generates a captured video by emitting visible light. For example, one or more camera devices 210 are installed in one search area. Each of the camera devices 210 generates a captured video of the search area viewed from each installation location. In the present embodiment, the camera device 210 generates a video that is temporally continuous, but the present disclosure is not limited thereto, and the camera device 210 may generate images that are not temporally continuous. Further, in the present embodiment, the object information generation device 216 includes the infrared camera device 212 and the radar device 214 but is not limited thereto and may have another configuration as long as it is capable of generating object information as described below.
The object information generation device 216 acquires object information indicating an object included in the captured video. The infrared camera device 212 generates an infrared video as object information by emitting infrared light. The radar device 214 irradiates a search area with pulses of laser light and measures the reflected light to measure distance information to an object as object information.
The object information is information indicating a two-dimensional or three-dimensional shape of the search target object. Specifically, the object information is information indicating a two-dimensional or three-dimensional shape of the search target object or a peripheral object group including one peripheral object or a plurality of peripheral objects that hide part of the search target object. For example, in a case where the object information generation device 216 is an infrared camera device 212, the object information is information indicating a two-dimensional shape. In a case where the object information generation device 216 is a radar device 214, the object information is information indicating a three-dimensional shape acquired by, for example, swinging the transmission direction of a radar transmission signal vertically and horizontally. For example, in a case where the object information generation device 216 is an infrared camera device 212, an object information acquisition unit 108 of the search assistance device 100 to be described later acquires object information of a search target by distinguishing between the search target and peripheral objects that hide the search target from a perspective image. In a case where the object information generation device 216 is a radar device 214, the object information acquisition unit 108 of the search assistance device 100 to be described later acquires object information of a search target by distinguishing between the search target and peripheral objects that hide the search target based on differences in distance information.
The control unit 220 controls the camera device 210, the infrared camera device 212, and the radar device 214. The control unit 220 operates the camera device 210, the infrared camera device 212, and the radar device 214 in time synchronization. Accordingly, the control unit 220 acquires a captured video and object information that are synchronized in time. The control unit 220 transmits the captured video and object information synchronized in time to the search assistance device 100. The control unit 220 may perform camera control such as changing an imaging direction and zooming an imaging range of the camera device 210, the infrared camera device 212, or the radar device 214 based on the operation of the searcher, for example.
The search assistance device 100 includes, for example, a search area side communication unit 102, a terminal side communication unit 104, a captured video acquisition unit 106, an object information acquisition unit 108, a setting unit 110, a conversion unit 112, a determination unit 114, a video creation unit 116, and a search unit 118. The search area side communication unit 102 is a communication interface such as a network interface card (NIC) or a wireless communication module for connecting to the network NW. The terminal side communication unit 104 is a communication interface that performs communication via, for example, a LAN line. Each unit, such as the captured video acquisition unit 106, the object information acquisition unit 108, the setting unit 110, the conversion unit 112, the determination unit 114, the video creation unit 116, and the search unit 118, is realized by a computer, such as a CPU mounted on the search assistance device 100, executing a program stored in a program memory.
The captured video acquisition unit 106 acquires a captured video P10. The captured video P10 is supplied via the search area side communication unit 102. The captured video acquisition unit 106 stores, for example, search area information indicating the search area, camera information that uniquely specifies the camera device 210, and a captured video in a storage device (not shown) in association with each other.
The object information acquisition unit 108 acquires object information indicating an object included in the captured video P10. The object information is supplied via the search area side communication unit 102.
The object information includes object information of the search target and object information of peripheral objects. The object information acquisition unit 108 stores, for example, search area information indicating the search area, device information that uniquely specifies the infrared camera device 212 or the radar device 214, and object information in a storage device (not shown) in association with each other. The object information acquisition unit 108 may store the acquired object information in association with the search area information or the device information.
The setting unit 110 sets a search target image P12 including a search target object input via the terminal side communication unit 104. The search target image P12 is, for example, a still image viewed from a specific viewpoint and provided by a person (search requester) requesting a search for a search target, such as an image of a lost item or an image of the face of a lost child. The setting unit 110 may acquire, as the search target image P12, an image uploaded to the search assistance device 100 from another device, such as a smartphone of a search requester, for example.
The conversion unit 112 converts the search target image P12 set by the setting unit 110 into viewpoint converted images P14 viewed from optional viewpoints. The conversion unit 112 may generate one viewpoint converted image P14 from the search target image P12 or may generate a plurality of viewpoint converted images P14. The conversion unit 112 may, for example, cut out an area including the search target included in the search target image P12 and perform image processing such as angle correction, rotation, and size change on the area including the search target to create a viewpoint converted image P14 in which the viewpoint of the search target is converted. The conversion unit 112 may, for example, predict a portion of the search target image P12 where the search target does not appear and create a viewpoint converted image P14 in which the viewpoint of the search target is converted by combining the predicted portion with an area including the search target included in the search target image P12. The conversion unit 112 may predict the shape or the like of the search target by searching a search site or the like for objects similar to the search target based on the product name information of the search target, for example.
The determination unit 114 determines whether or not part of a search target candidate is hidden by a peripheral object by using the search target image P12 or the object information. The search target candidate may be any object included in the captured video P10. The determination unit 114 may compare the viewpoint converted image P14 obtained by converting the search target image P12 with an object included in the captured video P10, thereby detecting an object included in the captured video P10 that matches one of the viewpoint converted images P14 as a search target candidate and determine whether or not part of the detected search target candidate is hidden by a peripheral object. The determination unit 114 may determine whether or not the search target candidates overlap based on the object information, thereby determining whether or not part of an object included in the captured video P10 is hidden by a peripheral object. The determination unit 114 may determine whether or not part of the object information is missing, thereby determining whether or not part of an object included in the captured video P10 is hidden by a peripheral object. The search target candidate is, for example, an object included in the captured video P10 that matches at least one of the search target image P12 and one or more viewpoint converted images P14.
In a case where the determination unit 114 determines that part of the search target candidate is hidden by a peripheral object, the video creation unit 116 creates a comparison video to be compared in order to search for the search target object using an image showing the peripheral object in the captured video P10. The video creation unit 116 creates, for example, an entire video P16 representing the entirety of the search target candidate as a comparison video. The video creation unit 116 interpolates the portions hidden by peripheral objects by, for example, viewing through the peripheral objects using infrared images acquired by the infrared camera device 212. The video creation unit 116 uses the distance image acquired by the radar device 214 to specify portions of peripheral objects and interpolates the portions of the peripheral objects. Accordingly, the video creation unit 116 creates the entire video P16.
The search unit 118 searches for the search target object by comparing the entire video P16 created by the video creation unit 116 with the viewpoint converted image P14 converted by the conversion unit 112. The search unit 118 transmits the search results to the searcher's terminal device 300 via the terminal side communication unit 104. For example, in a case where a search target candidate is found, the search unit 118 may transmit the captured video P10 including the search target candidate, position information of the search target candidate, and the like as search results. Accordingly, the searcher can visually recognize the search target by displaying the captured video P10 and by checking the video from the camera device 210 which captures the position of the search target candidate, thereby enabling the searcher to search for the search target.
In a case where the determination unit 114 has detected a search target candidate that matches the viewpoint converted image P14 (step S106: YES), it determines whether or not part of the search target candidate is hidden by a peripheral object (step S108). In a case where the determination unit 114 determines that part of the search target candidate is not hidden by a peripheral object (step S108: NO), the determination unit 114 provides a notification of the search result (step S116). In a case where it is determined that part of the search target candidate is hidden by a peripheral object (step S108: YES), the video creation unit 116 cuts out and extracts an image including an image showing the peripheral object (peripheral object image) (step S110) and creates an entire video P16 in which the entire search target candidate can be checked using the peripheral object image (step S112). The video creation unit 116 recognizes portions of the image showing the search target candidate that are hidden by the peripheral objects based on the peripheral object images and interpolates the portions of the image showing the search target candidate that are hidden by the peripheral objects using object information.
Next, the search unit 118 determines whether or not the viewpoint converted image P14 matches the entire video P16 (step S114). In a case where the search unit 118 does not detect the viewpoint converted image P14 that matches the entire video P16 (step S114: NO), it returns the process to step S104. In a case where the search unit 118 has detected the viewpoint converted image P14 that matches the entire video P16 (step S114: YES), the search unit 118 provides a notification of the search result (step S116). In step S116, the search unit 118 transmits, for example, an image showing the search target candidate that was the source of the entire video P16, position information of the search target candidate, and the like, to the searcher's terminal device 300 as the search result.
As described above, with the search assistance device 100 according to the first embodiment, the captured video P10 and object information are acquired, and it is determined whether or not part of the search target candidate is hidden by a peripheral object. In a case where it is determined that part of the search target candidate is hidden by a peripheral object, the entire video P16 representing the entirety of the search target candidate can be created using the object information. In a case where a search target candidate is hidden by a peripheral object, it is highly possible that the true search target cannot be searched for with a high probability. Furthermore, in a case where the number of camera devices 210 is limited, it is highly possible that the true search target cannot be searched for. In contrast, with the search assistance device 100 according to the first embodiment, in a case where a search target candidate detected based on a comparison between the captured video P10 and the viewpoint converted image P14 is hidden by a peripheral object, the entire video P16 can be created using the object information. Accordingly, the search assistance device 100 can transmit, for example, the entire video P16 to the searcher's terminal device 300 to show the entire video P16 to the searcher, and thus, the searcher can search whether the search target candidate is the true search target. As a result, with the search assistance device 100 according to the first embodiment, the search target object can be detected with a high probability even if the search target object is hidden by a peripheral object.
In the first embodiment, the search assistance system includes the infrared camera device 212 or the radar device 214 to acquire object information and can create an entire video P16 using the object information acquired by the infrared camera device 212 or the radar device 214. Accordingly, the search assistance system and the search assistance device 100 can create an entire video P16 by performing processing such as interpolating portions of the search target candidates that are hidden by peripheral objects.
With the search assistance device 100 according to the first embodiment, the search assistance device 100 may convert the search target image P12 into viewpoint converted images P14 viewed from optional viewpoints and compare the viewpoint converted images P14 with the object included in the captured video P10, thereby detecting an object included in the captured video P10 that matches one of the viewpoint converted images P14 as a search target candidate. With the search assistance device 100 according to the first embodiment, the search unit 118 that searches for a search target object by comparing the entire video P16 with the viewpoint converted image P14 is provided. Therefore, the search target object can be searched for even if the search target object is hidden by peripheral objects.
A second embodiment will be described below.
In the search assistance device 100B, as shown in
As shown in
In a case where the determination unit 114 has detected an object (search target candidate) that matches the viewpoint converted image P14 (step S206: YES), it determines whether or not the search target candidate is hidden by a peripheral object (step S208). In a case where the search target candidate is not hidden by a peripheral object (step S208: NO), the determination unit 114 provides a notification of the search result (step S216). In a case where the search target candidate is hidden by a peripheral object (step S208: YES), the combining unit 120 cuts out and extracts an image including an image showing the peripheral object (peripheral object image) (step S210) and creates a composite video P20 by combining the peripheral object image with the viewpoint converted image P14 (step S212).
Next, the search unit 122 determines whether or not the composite image P20 matches the captured video P10 (step S214). In a case where the composite image P20 does not match the captured video P10 (step S214: NO), the search unit 122 returns the process to step S204. In a case where the composite image P20 matches the captured video P10 (step S214: YES), the search unit 122 provides a notification of the search result (step S216). The search unit 122 transmits, for example, an image showing the search target candidate that matches the composite image P20, position information of the search target candidate, and the like, to the searcher's terminal device 300 as the search result.
The search assistance device 100B according to the second embodiment includes the combining unit 120 that creates a composite image P20 by combining an image of a peripheral object that is hiding part of the search target candidate with the viewpoint converted image P14 and the search unit 122 that searches for the search target object by comparing an object included in the captured video P10 with the composite image P20 created by the combining unit 120. Accordingly, with the search assistance device 100B, the search target object can be detected with a high probability even if the search target object is hidden by a peripheral object.
A third embodiment will be described below.
The video creation unit 130 creates an entire video P16 from the captured video P10 and the object information, in which the entire search target candidate can be checked, and further creates a composite image P20a by combining an image of a peripheral object included in the captured video P10 with the viewpoint converted image P14. The search unit 132 compares the entire video P16 with the viewpoint converted image P14 as in the first embodiment and also compares the composite image P20a with the captured video P10 as in the second embodiment. The search unit 132 searches for a search target object based on a comparison result between the entire video P20a and the viewpoint converted image P14 and a comparison result between the composite image P20a and the captured video P10. The search unit 132 may determine that the search target object has been found, for example, in a case where both a similarity between the entire video P20 and the viewpoint converted image P14 and a similarity between the composite image P20a and the captured video P10 are greater than or equal to a reference. The search unit 132 may provide a notification indicating that the search target object may have been found, for example, in a case where one of the similarity between the entire video P20 and the viewpoint converted image P14 and the similarity between the composite image P20a and the captured video P10 is greater than or equal to a reference. Accordingly, the search assistance device 100C can search for a search target object with higher accuracy.
Next, the search unit 132 determines whether or not the composite image P20 matches the captured image P10 and determines whether or not the viewpoint converted image P14 matches the entire video P16 (step S302). In a case where the composite image P20 does not match the captured image P10 and the viewpoint converted image P14 does not match the entire video P16 (step S302: NO), the search unit 132 returns the process to step S204. In a case where the composite image P20 matches the captured image P10 or the viewpoint converted image P14 matches the entire video P16 (step S302: YES), the search unit 122 provides a notification of the search result (step S304). In a case where the composite image P20 matches the captured video P10, the search unit 132 transmits an image showing the search target candidate, position information of the search target candidate, and the like, to the searcher's terminal device 300 as the search result. In a case where the viewpoint converted image P14 matches the entire video P16, the search unit 132 transmits an image showing the search target candidate that was the source of the entire video P16, position information of the search target candidate, and the like, to the searcher's terminal device 300 as the search result.
Modification examples of the first embodiment, the second embodiment, and the third embodiment will be described below.
The mobile robot 230 may move based on the operation of the searcher and transmit the captured video P10 acquired while moving to the search assistance device 100. Accordingly, the mobile robot 230 can provide the captured video P10 to the search assistance device 100 instead of the camera device 210. As a result, the mobile robot 230 can assist in searching for the search target even in a location where a camera device such as the camera device 210 or the infrared camera device 212 is not installed.
The search assistance device 100 may acquire the captured video P10 by controlling the movement of the mobile robot 230 in order to create, for example, a 360-degree free viewpoint video. The search assistance device 100 detects a search target candidate using a fixed camera device 210, and in a case where it is desired to check the captured video P10 of the search target candidate in more detail, the search assistance device 100 transmits control information to the mobile robot 230 to move the mobile robot 230 to the location where the search target candidate has been found. In a case where the mobile robot 230 moves to the vicinity of a search target candidate, the search assistance device 100 can acquire a video for creating a free viewpoint video by controlling the zoom and the imaging direction of the mobile robot 230.
A fourth embodiment will be described below. A search assistance device 100D according to the fourth embodiment differs from the above-described embodiments in that a search target is searched for based on an object designated by a searcher in a captured video P10.
A conversion unit 112 # creates a viewpoint converted image P32 by performing viewpoint conversion on the object image P30 acquired by the target image acquisition unit 110a as the search target image (step S406). Next, the captured video acquisition unit 106 acquires the captured video P10, and the object information acquisition unit 108 acquires object information (step S408). Next, a determination unit 114 # compares viewpoint converted images P32 with objects included in the captured video P10 to detect an object included in the captured video P10 that matches one of the viewpoint converted images P32 as a search target candidate. In a case where the determination unit 114 # has detected a search target candidate by comparing the viewpoint converted image P32 with the captured video P10 (step S412: YES), it determines whether or not the search target candidate is hidden by a peripheral object (step S414). In a case where the determination unit 114 # does not detect an object that matches the viewpoint converted image P32 (step S412: NO), it returns the process to step S408.
In a case where part of the search target candidate is hidden by a peripheral object (step S414: YES), an entire video creation unit 116 # cuts out and extracts an image including a peripheral object image (step S418) and creates an entire video P34 in which the entire search target candidate can be checked from the viewpoint converted image P32 using the peripheral object image (step S420). The terminal side communication unit 104 # transmits the entire video P34 to the searcher's terminal device 300 (step S422). In a case where part of the search target candidate is not hidden by a peripheral object (step S414: NO), the search assistance device 100D transmits the viewpoint converted image P32 that matches the object image P30 to the searcher's terminal device 300 via the terminal side communication unit 104 # (step S316). Accordingly, the search assistance device 100D can search for a search target based on an object designated by the searcher in the captured video P10.
A modification example of the fourth embodiment will be described below.
The search assistance device 100D may acquire the captured video P10 from the camera device 210 and acquire a captured video acquired from the mobile robot 240 as object information. The search assistance device 100D uses the captured video received from the mobile robot 240 to create an entire video of the search target candidate included in the captured video P10 received from the camera device 210. Accordingly, the search assistance device 100D can create the entire video using a captured video captured in an imaging direction different from the imaging direction of the camera device 210, for example.
The search assistance device 100D may acquire the captured video by controlling the movement of the mobile robot 240 in order to create, for example, a 360-degree free viewpoint video. The search assistance device 100D detects a search target candidate using a fixed camera device 210, and in a case where it is desired to check the captured video of the search target candidate in more detail, the search assistance device 100D transmits control information to the mobile robot 240 to move the mobile robot 240 to the location where the search target candidate has been found. In a case where the mobile robot 240 moves to the vicinity of a search target candidate, the search assistance device 100D can acquire a video for creating a free viewpoint video by controlling the zoom and the imaging direction of the mobile robot 240.
Another modification example of the fourth embodiment will be described below. The searcher's terminal device 300 may have a goggle-type display device (not shown) that is worn on the searcher's head and may display the captured video P10 on the goggle-type display device. It is desirable that the captured video P10 to be displayed on the goggle-type display device is a 360-degree free viewpoint video so that the searcher can freely change the viewpoint. However, the present disclosure is not limited thereto, the captured video P10 may be a captured video P10 captured by controlling the imaging direction and the imaging range of the camera system 200 in accordance with the viewpoint of the searcher, may be the captured video P10 captured by controlling the movement of the mobile robot 240 in accordance with the viewpoint of the searcher, or may be a virtual reality video in which a video of the search range is virtually created.
The searcher's terminal device 300 includes a detection device that detects the searcher's gestures, detects finger movements as the searcher's gestures, generates designation information, and transmits the designation information to the search assistance device 100D. The terminal side communication unit 104 # of the search assistance device 100D receives the designation information, and the target image acquisition unit 110a acquires the object image P30 designated by the searcher based on the designation information as a search target image. The conversion unit 112 # creates a viewpoint converted image P32 by performing viewpoint conversion on the object image P30 designated by the searcher. The determination unit 114 # has detected a search target candidate by comparing the viewpoint converted image P32 with the captured video P10 and determines whether or not the search target candidate is hidden by a peripheral object. In a case where the search target candidate is hidden by a peripheral object, the entire video creation unit 116 # creates an entire video P34 from the viewpoint converted image P32 using object information of the search target object and object information of the peripheral objects. The entire video creation unit 116 # transmits the entire video P34 to the searcher's terminal device 300 via the terminal side communication unit 104 #. In a case where the search target candidate is not hidden by a peripheral object, the search assistance device 100D transmits the viewpoint converted image P32 that matches the object image P30 to the searcher's terminal device 300 via the terminal side communication unit 104 #. Accordingly, the search assistance device 100D can search for the search target using the entire video P34 or the viewpoint converted image P32. In addition, the search assistance device 100D allows a searcher to wear a goggle-type display device, giving the searcher the feeling as if he or she were actually searching at the site.
Although the embodiments of the present disclosure have been described in detail with reference to the drawings, the specific configurations are not limited to the above-described embodiments and also include design and the like within the scope of the gist of the present disclosure. The configurations described in the above-described embodiments can be combined in any manner.
| Number | Date | Country | |
|---|---|---|---|
| Parent | PCT/JP2022/044520 | Dec 2022 | WO |
| Child | 19021643 | US |
