Patent Grant
11749112
The present invention relates to a warning device, a warning method, and a warning program.
An imaging device such as a digital camera generally takes an image of an area to be imaged by using a filter that transmits light with a specific wavelength. The imaging device can improve the image quality of an image taken by use of a filter that is suitable for the amount of light or the like of an area to be imaged. Thus, in the daytime with a large amount of visible light, a camera is used that employs a filter for daytime which transmits more light in the visible spectrum, and in the nighttime with a small amount of visible light, a camera is used that employs a filter for nighttime which transmits more light in the infrared spectrum.
Japanese Unexamined Patent Application Publication No. H07-160957 discloses a technique related to a monitoring device that includes an infrared sensor and a near-infrared camera. The monitoring device disclosed in Japanese Unexamined Patent Application Publication No. H07-160957 takes an image of an area monitored by the infrared sensor by using the near-infrared camera, and detects a change in the monitored area based on the image taken by the near-infrared camera. The monitoring device then determines an abnormal state based on the detected change and the output of the infrared sensor.
Recently, a warning device has been developed that takes an image of the area around a vehicle by using an in-vehicle camera, performs image recognition of the taken image and detects a person, a vehicle or the like as an object in the image, and issues a warning to a driver. Japanese Unexamined Patent Application Publication No. 2011-146049 discloses a technique related to a human detection device for detecting a person from a taken image. The human detection device disclosed in Japanese Unexamined Patent Application Publication No. 2011-146049 detects the appearance of an object based on an image acquired from an image sensor, determines whether a parameter related to the detected appearance of the object and a comparative parameter related to a human match or not, calculates a human probability indicating the probability that the object is a human according to the degree of matching, and determines that the object is a human when the human probability is equal to or more than a preset human threshold.
In the case of the warning device described above, there are circumstances where it is difficult to determine the outside environment of a vehicle. The circumstances where it is difficult to determine the outside environment of a vehicle include the circumstances where weather conditions vary, the circumstances where a vehicle travels through a place where the surrounding brightness suddenly changes before and after entering a tunnel and the like. Therefore, there is a problem that it is sometimes difficult to issue a warning based on detection of an object under the circumstances where it is difficult to determine the outside environment of a vehicle.
Further, in the warning device described above, when an object exists in the traveling direction of a vehicle, it is necessary to issue a warning to a driver because there is a possibility of a collision. The possibility of a collision can be calculated by predicting the relative moving speed, the moving direction or the like of an object from analysis results of images taken in succession while a vehicle is traveling. However, the accuracy of detecting the movement of an object can decrease under the circumstances where it is difficult to determine the outside environment of a vehicle, and there is a problem that it is sometimes difficult to issue an appropriate warning.
Note that, in the monitoring device disclosed in Japanese Unexamined Patent Application Publication No. H07-160957, an area to be monitored is a specific area in facility, and it does not assume the circumstances where it is difficult to determine the outside environment. Further, in the human detection device disclosed in Japanese Unexamined Patent Application Publication No. 2011-146049, a target to be analyzed is a single image. Therefore, in the case where an appropriate filter cannot be used due to the circumstances where it is difficult to determine the outside environment, for example, the image quality is degraded. When the image quality of an image to be analyzed is low, the accuracy of detecting an object that can interfere with the traveling of a vehicle can decrease. This causes a failure to issue an appropriate warning. Further, such a problem occurs in a warning device that is mounted on movable bodies in general, not limited to vehicles.
A first aspect of the embodiment provides a warning device that includes at least one memory storing instructions, and a processor configured to execute the instructions to: acquire an image group including a plurality of images taken in succession; perform detection of an object on each of images in the image group; compare detection results of the object in the image group in chronological order and determine a degree of movement of the object; and issue a warning in accordance with the determined degree of movement, wherein the processor is further configured to execute the instructions to acquire a plurality of image groups respectively based on a plurality of filter characteristics and compare corresponding images between the plurality of image groups and thereby select an image with high detection accuracy for each time period, compare the selected images in chronological order, and determine the degree of movement of the object.
A second aspect of the embodiment provides a warning method that includes acquiring an image group including a plurality of images taken in succession; performing detection of an object on each of images in the image group; comparing detection results of the object in the image group in chronological order and determining a degree of movement of the object; and issuing a warning in accordance with the determined degree of movement, wherein the acquiring the image group acquires a plurality of image groups respectively based on a plurality of filter characteristics and the comparing the detection results compares corresponding images between the plurality of image groups and thereby selects an image with high detection accuracy for each time period, compares the selected images in chronological order, and determines the degree of movement of the object.
A third aspect of the embodiment provides a non-transitory computer readable medium storing a warning program that causes a computer to execute a processing of acquiring an image group including a plurality of images taken in succession; a processing of performing detection of an object on each of images in the image group; a processing of comparing detection results of the object in the image group in chronological order and determining a degree of movement of the object; and a processing of issuing a warning in accordance with the determined degree of movement, wherein the processing of acquiring the image group acquires a plurality of image groups respectively based on a plurality of filter characteristics and the processing of comparing the detection results compares corresponding images between the plurality of image groups and thereby selects an image with high detection accuracy for each time period, compares the selected images in chronological order, and determines the degree of movement of the object.
Specific exemplary embodiments of the present invention will be described hereinafter in detail with reference to the drawings. It is noted that in the description of the drawings, the same elements will be denoted by the same reference symbols and redundant description will be omitted to clarify the explanation.
The warning device 10 includes an image acquisition means 11, a detection means 12, and a warning means 13. The image acquisition means 11 takes images of the area around the movable body 1 by a plurality of different filter characteristics while the movable body 1 is traveling, and acquires them as a plurality of images. In other words, the image acquisition means 11 acquires a plurality of images based on the plurality of filter characteristics, respectively. The image acquisition means 11 includes imaging devices 111 and 112. The imaging devices 111 and 112 respectively correspond to the plurality of filter characteristics. Each of the imaging devices 111 and 112 takes images by the corresponding filter characteristics. Note that the number of imaging devices included in the image acquisition means 11 is not limited to two. For example, the image acquisition means 11 may include three or more imaging devices respectively corresponding to three or more filter characteristics. Note that the imaging device is a camera, for example.
The detection means 12 performs detection of a specified object on each of the plurality of acquired images. The detection of an object is the processing of trying to detect the appearance of an object by image recognition.
For example, there is a case where a person who is crossing a road is shown, together with the shape of the road, in an image taken by a camera in the traveling direction of a vehicle. In such a case, the area showing the appearance of the person who is crossing the road can be detected. The detection means 12 generates the detection result as an index value. Further, as described in Japanese Unexamined Patent Application Publication No. 2011-146049, the detection means 12 may detect a parameter related to the appearance of an object and generate the degree of matching between a comparative parameter and the detected parameter as a detection result. Note that the processing of generating a detection result when detecting an object from each image is not limited thereto. Further, the object is not limited to a human, and it includes any movable body that can interfere with the traveling of the movable body 1, such as an animal and another vehicle. Furthermore, the object is not necessarily a movable body, and it may be a person standing on the road, a stopped vehicle, a road closed sign, an obstacle left on the road and the like. Note that the parameter related to the appearance may be luminance, saturation, edge, or a combination of those.
The warning means 13 issues a specific warning when an object is detected from at least any one of the plurality of acquired images. Particularly, the warning means 13 according to an embodiment of the present invention issues a higher level of warning when an object is detected from all of the plurality of images than when an object is detected from only some of the plurality of images. Issuing a higher level of warning means changing the operation of a warning means so that an object is more clearly recognizable, as described later.
The processor 110 is a control device such as a CPU (Central Processing Unit). The IF unit 120 is an interface for inputting and outputting data to and from the outside of the warning device 10. Particularly, the IF unit 120 outputs information indicating a warning to the display device 21 or the speaker 22 under control of the processor 110.
The imaging devices 131 and 132 respectively correspond to the imaging devices 111 and 112 described above. The imaging device 131 includes a first filter 1311 and a first image sensor 1312. The first filter 1311 makes selection of the wavelength of light to be incident on the first image sensor 1312. Thus, the first filter 1311 is a filter that is applied when taking an image by the imaging device 131. The first filter 1311 has the filter characteristics that block light in the near-infrared spectrum and transmit light in the visible spectrum, for example. It is assumed that the first filter 1311 is at least a filter that is suitable in the case where the area around the movable body 1 is bright such as during daytime hours. The first image sensor 1312 is an element that converts the intensity of light that has passed through the first filter 1311 into an electrical signal.
The imaging device 132 includes a second filter 1321 and a second image sensor 1322. The second filter 1321 makes selection of the wavelength of light to be incident on the second image sensor 1322. Thus, the second filter 1321 is a filter that is applied when taking an image by the imaging device 132. The second filter 1321 has the filter characteristics that block light in the visible spectrum and transmit light in the near-infrared spectrum, for example. It is assumed that the second filter 1321 is at least a filter that is suitable in the case where the area around the movable body 1 is dark such as during nighttime hours. The second image sensor 1322 is an element that converts the intensity of light that has passed through the second filter 1321 into an electrical signal.
The storage unit 140 is a storage device such as a memory or a hard disk, for example. The storage unit 140 stores a warning program 141, taken images 1421 and 1422, and index values 1431 and 1432. The warning program 141 is a computer program in which object detection and warning processing, which is a warning method, of the warning device 10 according to this embodiment is implemented.
The taken image 1421 is image data that is taken by the imaging device 131 with use of the first filter 1311 and the first image sensor 1312. The taken image 1422 is image data that is taken by the imaging device 132 with use of the second filter 1321 and the second image sensor 1322. The index value 1431 is a value indicating the level of detection of an object from the taken image 1421 by the detection means 12. The index value 1432 is a value indicating the level of detection of an object from the taken image 1422 by the detection means 12. The index values 1431 and 1432 are, in other words, index values of the accuracy of detecting an object when image recognition is performed on taken images.
The processor 110 reads the warning program 141 from the storage unit 140 and executes it. The warning device 10 thereby operates as the image acquisition means 11, the detection means 12, the warning means 13 and the like according to this embodiment by using the IF unit 120, the imaging devices 131 and 132 as appropriate.
First, the imaging device 111 takes an image of the area around the movable body 1 by the first filter characteristics while the movable body 1 is traveling, and acquires it as a first image (S111). At this time, the warning device 10 stores the acquired first image into the storage unit 140. Then, the detection means 12 performs detection of a specified object on the first image (S112).
In parallel with Step S111, the imaging device 112 takes an image of the area around the movable body 1 by the second filter characteristics while the movable body 1 is traveling, and acquires it as a second image (S121). At this time, the warning device 10 stores the acquired second image into the storage unit 140. Then, the detection means 12 performs detection of a specified object on the second image (S122).
After Steps S112 and S122, the warning means 13 determines whether an object is detected from one of the first image and the second image (S131). When an object is detected from one image, the warning means 13 performs first warning processing (S132). For example, the warning means 13 outputs the image from which the object is detected to the display device 21 and displays the image on the screen. For example, in the case where the amount of light in the outside environment is not sufficient, while an object is clearly shown in the first image, the object cannot be recognized and detected in the second image. In such a case, the warning means 13 outputs the first image to the display device 21. With the first image displayed on the screen, a driver can recognize the danger.
When the determination in Step S131 results in No, the warning means 13 determines whether an object is detected from both of the first image and the second image (S133). For example, in the case of the taken images 30a and 30b in
On the other hand, the case where the level of warning is not high compared with the above (for example, when an object is detected from one of the first image and the second image) is also described as an example. In this case, one of the first image and the second image in which an object is shown is output to the display device 21 and displayed on the screen as described above. At this time, the warning means 13 performs processing such as displaying the first image on the display device 21 without any modification, or, when modifying the image and displaying it on the display device 21, performs processing such as drawing a thinner line around an object, placing a different mark above an object, or changing the color of an object to be modified compared with the case where the level of warning is high. Further, in the case where the warning means 13 outputs a warning tone through the speaker 22, it performs processing such as changing the volume or changing the warning tone compared with the case where the level of warning is high.
When the determination in Step S133 results in No, the process of
In Steps S112 and S122 in
Then, the detection means 12 stores the calculated index values into the storage unit 140. An example of warning processing from Steps S131 to S134 in
First, the warning means 13 determines whether the index value exceeds a specified value in one image or not (S131a). For example, the warning means 13 determines whether the index value 1431 read from the storage unit 140 exceeds a specified value is one or not. Likewise, the warning means 13 determines whether the index value 1432 read from the storage unit 140 exceeds a specified value is one or not. When one of the index values 1431 and 1432 exceeds a specified value and the other one of them falls below the specified value, the warning means 13 displays the corresponding image, which is the image in which the index value exceeds the specified value, on the screen (S132a). Step S132a is one example of first warning processing.
When the determination in Step S131a results in No, the warning means 13 determines whether the index value exceeds a specified value in both images or not (S133a). When the index value exceeds a specified value in both images, the warning means 13 selects the image with the highest index value (S134a). For example, in the case of the taken images 30a and 30b, the taken image 30a is selected. Then, the warning means 13 displays, on the screen, the selected image with the object highlighted (S134b). For example, the warning means 13 generates an image in which a line is drawn around the area of the object 41 in the taken image 30a, outputs the image to the display device 21 and displays it on the screen.
As described above, the first embodiment has the following advantages.
First, in the case of taking an image by using any one of two filter characteristics as before, it is possible to detect an object with high accuracy by use of appropriate filter characteristics for the outside environment. However, the accuracy of detecting an object is degraded if inappropriate filter characteristics are used. Further, when a movable body is traveling, the circumstances where it is difficult to determine the outside environment can occur. Therefore, in the first embodiment, images of the surrounding area are taken by using each of a plurality of filter characteristics, thereby increasing the probability of detecting an object from at least any one of the images. It is thus possible to issue a warning to a driver about a dangerous situation in more cases and thereby enhance the safety.
An example of the circumstances where it is difficult to determine the outside environment is the early evening when it is difficult to distinguish between the daytime and the nighttime from the amount of light. In such circumstances, each of the images taken by use of a filter for daytime and a filter for nighttime does not have sufficient quality, and there is a possibility that accurate detection cannot be achieved by analyzing a single image only. Thus, in the first embodiment, it is determined as a dangerous situation when an object is detected at a certain level of accuracy from each of a plurality of images obtained by taking the same object by a plurality of different filter characteristics. Particularly, in such circumstances, a driver is likely to have difficulty recognizing an object compared with the case where it is clearly daytime or nighttime. Therefore, the level of warning is increased in such circumstances, and it is thereby possible to issue an appropriate warning to a driver.
Further, in the case of second warning processing in which the level of warning is higher than in first warning processing, one of two images in which an object is detected at higher accuracy is used. This is because, when an object is detected from two or more taken images, a warning is issued with use of the image with higher detection accuracy, so that a driver can more easily recognize the object.
A second embodiment is a modified example of the first embodiment described above.
The storage unit 16 further includes map information 161, specific area 162, position information 163, first mode 164 and second mode 165, in addition to the elements in the storage unit 140 of
The first mode 164 is setting information that defines that the image acquisition means 11 takes an image by one filter characteristics. The second mode 165 is setting information that defines that the image acquisition means 11 takes an image by a plurality of filter characteristics.
The position information acquisition means 14 acquires the current position information of the movable body 1 at regular intervals. The position information acquisition means 14 then stores the acquired position information into the storage unit 16. The position information acquisition means 14 is a device that is mounted on a vehicle and can acquire GPS (Global Positioning System) information, for example.
The setting means 15 selects the first mode 164 or the second mode 165 according to specified conditions, and sets the selected mode to the image acquisition means 11. Further, when the acquired position information 163 is within the range of the specific area 162 on the map information 161, the setting means 15 selects the second mode 165 and sets it to the image acquisition means 11.
The specified conditions are conditions using some or all of position information, time information, weather, road information, driving information and the like, for example. An example of the conditions using position information is whether the distance between the current position and a specific position is within a specified value or not, or whether it is within the range of a specific area or not. An example of the conditions using time information is conditions for determining a season based on the present day of month, or conditions for determining a time period such as the daytime, the nighttime or the evening based on hours and minutes. To use the time information, it is necessary to add an element for acquiring information about the current time or the like.
An example of the conditions using the weather is the state of clouds, whether the weather is sunny, cloudy, rainy or the like. To use the weather, it is necessary to add an element for acquiring information about the weather or the like. An example of the conditions using road information is conditions for determining whether a road type is an expressway or a local street, the width of road, a road sign, a road shape, a road surface condition or the like. To use the road information, it is necessary to add an element for acquiring information about the road on which a vehicle is traveling or the like. An example of the conditions using driving information is conditions for determining whether vehicle speed, brake control information or the like exceeds a specified value or not. To use the driving information, it is necessary to add an element for acquiring speed or brake control information from a vehicle or the like.
As described above, in the second embodiment, it is possible to change the type of a filter to be used for taking an image according to situations. Thus, when it is easy to determine the outside environment, it is possible to use the first mode to reduce the power consumption or the like of the movable body 1. On the other hand, when it is difficult to determine the outside environment, it is possible to immediately change the mode to the second mode using a plurality of filter characteristics.
A third embodiment is a modified example of the first embodiment described above. While the image acquisition means 11 includes two imaging devices in the first embodiment described above, an image acquisition means 11a according to the third embodiment includes one imaging device.
As described above, the same advantages as in the first embodiment described above can be obtained also in the third embodiment.
The warning device 10b includes an image acquisition means 11b, a detection means 12b, a determination means 17, and a warning means 13b. The image acquisition means 11b takes images of the area around the movable body 1b successively in parallel by a plurality of different filter characteristics while the movable body 1b is traveling. The image acquisition means 11b then acquires the plurality of taken images as a plurality of time-series image groups for each of the filter characteristics. Stated differently, the image acquisition means 11b acquires an image group including a plurality of images taken in succession. Then, the image acquisition means 11b acquires a plurality of image groups based on the plurality of filter characteristics, respectively. The image acquisition means 11b includes imaging devices 111 and 112. The imaging devices 111 and 112 respectively correspond to the plurality of filter characteristics. Each of the imaging devices 111 and 112 takes images by the corresponding filter characteristics. Note that the number of imaging devices included in the image acquisition means 11b is not limited to two. For example, the image acquisition means 11b may include three or more imaging devices respectively corresponding to three or more filter characteristics. Note that the imaging device is a camera, for example.
The detection means 12b performs detection of a specified object on each of images in the plurality of image groups. The detection means 12b then generates a detection result for each image. Note that a method of detecting an object in one image by the detection means 12b is the same as the method by the detection means 12 described above.
The determination means 17 compares object detection results in a plurality of image groups in chronological order, and determines the degree of movement of an object. “Comparing detection results in chronological order” means comparing detection results in images taken at the times adjacent to each other. At this time, detection results may be compared between images in each image group, or between images belonging to different image groups. Alternatively, detection results may be compared between images belonging to different image groups and taken at the same time. Further, an average of detection results between images belonging to different image groups and taken at a certain time may be calculated, and an average between images taken at an adjacent time may be compared.
Further, “the degree of movement” includes the degree indicating whether an object is moving closer to or farther from the movable body 1b, which is, the degree of approach or the speed of approach of an object to the movable body 1b, the speed of movement or the direction of movement of an object and the like, for example. Further, “determining the degree of movement of an object” includes determining whether the speed of approach or the like is higher than a reference value, for example. Alternatively, when an object moves crossing a road, the probability of a collision between the object and the movable body 1b may be calculated from the moving direction and the moving speed, the traveling direction and the traveling speed of the movable body 1b, and whether the probability of a collision exceeds a reference value or not may be determined, for example.
The warning means 13b issues a warning in accordance with the determined degree of movement. For example, the warning means 13b issues a warning when an object is approaching the movable body 1b from the degree of movement. Particularly, the warning means 13b issues a higher level of warning when the degree of approach exceeds a reference value. Further, the warning means 13b issues a higher level of warning also when the probability of a collision exceeds a reference value as described above.
The storage unit 140b is a storage device such as a memory. The storage unit 140b stores a warning program 141b, a first image group 1421b, a second image group 1422b, a detection result 143 and an approach speed 144. The warning program 141b is a computer program in which processing, which is a warning method, of the warning device 10b according to the fourth embodiment is implemented.
The first image group 1421b is a time-series set of a plurality of taken images 14211, 14212, . . . that are obtained by taking images of the area around the movable body 1 in succession by the first filter characteristics while the movable body 1b is traveling. It is assumed that the images 14211, 14212, . . . are taken successively in this order, for example. Further, the second image group 1422b is a time-series set of a plurality of taken images 14221, 14222, . . . that are obtained by taking images of the area around the movable body 1 in succession by the second filter characteristics, in parallel with taking images in the first image group 1421b. It is assumed that the images 14221, 14222, . . . are taken successively in this order, for example. In sum, the images 14211 and the 14221 are taken at the corresponding time, and the images 14212 and the 14222 are taken at the corresponding time.
The detection result 143 is a result of trying to detect an object in each of the taken images. The detection result 143 exists for each taken image. The approach speed 144 is one example of the degree of approach of an object to the movable body 1b, and it is a relative moving speed of an object to the movable body 1b. The approach speed 144 is calculated from detection results in a plurality of images or the like in consideration of the time the images are taken.
The processor 110 reads the warning program 141b from the storage unit 140b and executes it. The warning device 10b thereby operates as the image acquisition means 11b, the detection means 12b, the determination means 17, the warning means 13b and the like according to the fourth embodiment by using the IF unit 120, the imaging devices 131 and 132 as appropriate.
Further, in parallel with Step S41, the imaging device 112 takes images of the area around the movable body 1b in succession by the second filter characteristics while the movable body 1b is traveling (S44). The image acquisition means 11b then acquires the taken images as the second image group (S45). To be specific, each time an image is taken, the image acquisition means 11b stores the taken image in association with the second image group 1422b in the storage unit 140b. Then, the detection means 12b performs detection of a specified object on each image in the second image group (S46). To be specific, the detection means 12b reads the taken image 14221 or the like that is associated with the second image group 1422b stored in the storage unit 140b, performs detection for each image, and stores a detection result into the storage unit 140b. Note that the detection means 12b may perform the detection processing in Steps S43 and S46 each time images are taken by the imaging devices 111 and 112.
After that, the determination means 17 compares detection results in the first image group and the second image group in chronological order, and determines the degree of movement of the object (S47). For example, the determination means 17 calculates the speed of approach of the object to the movable body 1b based on the detection results, and determines the degree of movement of the object by using the speed of approach. Then, the warning means 13b issues a warning in accordance with the determined degree of movement (S48).
As described above, in the fourth embodiment, while the movable body 1b is traveling, images of the area around the movable body 1b are taken in succession in parallel by a plurality of filter characteristics. At this time, the taken images are sorted, in chronological order, into different image groups corresponding to different filter characteristics. In the images taken by a plurality of filter characteristics, it is likely that the quality of an image taken by at least one of the filter characteristics in each time period attains the quality equivalent to an image taken by appropriate filter characteristics even under the circumstances where it is difficult to determine the outside environment of the movable body 1b. Therefore, by comparing object detection results on each of the images in chronological order, it is possible to determine the degree of movement of an object. It is thereby possible to issue an appropriate warning to a driver in accordance with the degree of movement of an object. This enhances the safety.
Particularly, by determining the degree of movement with use of the speed of approach, it is possible to make a determination about the danger of a collision more directly, which improves the accuracy of warning as well.
Examples 1 to 3 that show the detailed processing of Steps S47 and S48 in
In the example 1, the speed of approach is calculated for each image group, and a warning is issued based on the image group with a higher speed of approach. Processing of the example 1 is described hereinafter with reference to
In
Likewise, the determination means 17 makes a comparison between images in the second image group in chronological order and calculates the speed of approach (S52). In the example of
Next, the determination means 17 selects an image group with a higher speed of approach among the plurality of image groups (S53). Specifically, the determination means 17 compares the speeds of approach calculated in Steps S51 and S52 and determines in which of the first image group 31 and the second image group 32 the speed of approach is higher and selects one. Then, the determination means 17 determines the degree of movement of the object by using the speed of approach in the selected image group (S54). For example, the determination means 17 determines whether the selected speed of approach is equal to or higher than a specified reference value or not, and when it is equal to or higher than the specified reference value, determines that the object is approaching the movable body 1b at a higher speed than normal.
After that, the warning means 13b issues a warning in accordance with the speed of approach by using the selected image group (S55). For example, when the speed of approach is equal to or higher than a reference value, the warning means 13b issues a higher level of warning than when the speed of approach is lower than the reference value. The issuance of a higher level of warning is described hereinafter. For example, the warning means 13b may modify an image belonging to the selected image group so as to highlight the object included in the image and then output the image to the display device 21. Note that highlighting may include drawing a rectangle line around an object, placing a mark on or above an object, changing the color of an object or the like, for example. In other words, the warning means 13b may perform processing corresponding to the second warning processing described above.
On the other hand, the case where the level of warning is not high compared with the above (for example, when the speed of approach is lower than a reference value) is also described as an example. In this case, the image that belongs to the selected image group is output to the display device 21 and displayed on the screen. At this time, the warning means 13b performs processing such as displaying the image belonging to the selected image group on the display device 21 without any modification, or, when modifying the image and displaying it on the display device 21, performs processing such as drawing a thinner line around an object, placing a different mark above an object, or changing the color of an object to be modified compared with the case where the level of warning is high. Further, in the case where the warning means 13b outputs a warning tone through the speaker 22, it performs processing such as changing the volume or changing the warning tone compared with the case where the level of warning is high. In other words, the warning means 13b may perform processing corresponding to the first warning processing described above.
As described above, in the example 1 of the fourth embodiment, the degree of movement of an object is determined by using a group of images taken by a more appropriate filter in the outside environment during a certain period of time, and it is thereby possible to issue an appropriate warning with use of an image with higher detection accuracy.
Because the object detection accuracy is high in each image that belongs to a group of images taken by an appropriate filter, a difference in the position of an object between images that are adjacent in chronological order is clear. Thus, the movement of the object can be clearly grasped. On the other hand, because the object detection accuracy is low in each image that belongs to a group of images taken by an inappropriate filter, a difference in the position of an object between images that are adjacent in chronological order is not clear. Thus, it is difficult to determine whether the object is moving or not. Therefore, the speed of approach is calculated to be relatively higher in the former than in the latter. Accordingly, the detection accuracy by a filter is relatively high in the image group where the speed of approach is calculated to be higher. By outputting an image in the image group with high detection accuracy on the screen or the like, a driver can adequately recognize the danger.
Further, for example, when the speed of approach exceeds a reference value, it is highly urgent and therefore the level of warning is further increased, thereby giving an appropriate warning to a driver. On the contrary, when an object is detected in an image taken at a certain time and no object is detected in an image taken at a later time, it is likely that the object is detected by mistake, or the object has moved away from the movable body 1b. In such a case, it is determined not to issue a warning, thereby avoiding an excessive warning.
In the example 2, an image with high detection accuracy is selected for each time period in order to deal with a sudden change in the outside environment. Processing of the example 2 is described hereinafter with reference to
First, the determination means 17 compares images corresponding to the time t1 and selects one image with higher detection accuracy (S61). For example, a comparison of detection accuracy between images corresponding to the time may be made by detecting the degree of matching by pattern matching techniques or the like with use of a parameter related to the detected appearance and a comparative parameter, which are used for detection results described above, and comparing the results. Next, the determination means 17 compares images corresponding to the time t2 and selects one image with higher detection accuracy (S62). Likewise, the determination means 17 compares images corresponding to the time t3 and selects one image with higher detection accuracy (S63). Note that the order of performing Steps S61 to S63 is not limited thereto.
In the example of
After that, the determination means 17 compares the selected images at the time t1 to t3 in chronological order and calculates the speed of approach (S64). In the example of
Then, the determination means 17 determines the degree of movement of the object by using the calculated speed of approach (S65). The warning means 13b then issues a warning in accordance with the speed of approach by using each of the selected images (S66). Note that, in Steps S65 and S66, the level of warning may be increased as shown in
As described above, in the example 2 of the fourth embodiment, it is possible to appropriately determine the degree of movement of an object and issue an appropriate warning in the case where there is a sudden change in the outside environment within a certain period of time. The case where there is a sudden change in the outside environment within a certain period of time is, for example, the case where the brightness of the surrounding area abruptly changes, such as before and after entering a tunnel, before and after coming out of a tunnel, when the weather changes quickly due to rapid movement of clouds or the like. In such a case, while the detection accuracy of the first image group taken by the first filter characteristics is higher until a certain point of time, the detection accuracy of the first image group decreases halfway through, and the detection accuracy of the second image group taken by the second filter characteristics can become higher. Further, when the brightness is not stable due to the sunset and clouds, the image group with higher detection accuracy can be in alternation as shown in
Because the example 2 makes a comparison of the detection accuracy for each time an image is taken, it is possible to select images taken by appropriate filter characteristics for the brightness at each time and form an image group with high detection accuracy by a set of the selected images. It is thereby possible to calculate the speed of approach with high reliability and issue an appropriate warning.
In the example 3, when it is difficult to determine appropriate filter characteristics for the outside environment among a plurality of filter characteristics, images taken by the respective filter characteristics are synthesized to enhance the detection accuracy. Processing of the example 3 is described hereinafter with reference to
First, the determination means 17 compares images corresponding to the time t1 and specifies the position of an object in the image (S71). Next, the determination means 17 compares images corresponding to the time t2 and specifies the position of an object in the image (S72). Likewise, the determination means 17 compares images corresponding to the time t3 and specifies the position of an object in the image (S73). Note that, the processing of specifying the position may calculate the average of the positions of an object in the images to be compared. Further, the images to be compared may be synthesized. Note that the order of performing Steps S71 to S73 is not limited thereto.
In the example of
After that, the determination means 17 compares the specified positions in chronological order and calculates the speed of approach (S74). In the example of
Then, the determination means 17 determines the degree of movement of the object by using the calculated speed of approach (S75). The warning means 13b then issues a warning in accordance with the speed of approach by using the specified position of the object in each time period (S76). Note that, in Steps S75 and S76, the level of warning may be increased as shown in
As described above, in the example 3 of the fourth embodiment, the average or the like is calculated from detection results of the respective image groups, and therefore the effect of false determination can be suppressed. Further, when it is in the early evening, for example, the brightness in the area around the movable body 1b is between the daytime and the nighttime, and it is sometimes difficult to determine which of the first filter characteristics and the second filter characteristics is appropriate for use. Even in such a case, by calculating an intermediate position between the two detection results, it is possible to specify the position of an object with adequate accuracy. It is thereby possible to maintain the object detection accuracy by using a plurality of image groups in a comprehensive manner.
A fifth embodiment is a modified example of the fourth embodiment described above. While the image acquisition means 11b includes two imaging devices in the fourth embodiment described above, an image acquisition means according to the fifth embodiment includes one imaging device. Note that the configuration of the image acquisition means according to the fifth embodiment is the same as shown in
As described above, the same advantages as in the fourth embodiment described above can be obtained also in the fifth embodiment. Further, any of the examples 1 to 3 described above can be applied to the fifth embodiment.
Note that the detection means 12b may compare the detection results between images in each of image groups in chronological order, calculate the speed of approach to the object for each of the image groups, select the image group with the highest speed of approach among the plurality of image groups, and determine the degree of movement of the object in the selected image group.
The above-described examples can be combined as desirable. Specifically, the second embodiment and the third embodiment may be combined.
Although the exemplary embodiment of the present invention is described in the foregoing, the present invention is not restricted to the above-described configuration, and various changes, modifications and combinations as would be obvious to one skilled in the art may be made without departing from the scope of the invention.
Arbitrary processing of the on-vehicle device described above may be implemented by causing a CPU (Central Processing Unit) to execute a computer program to perform given processing. In this case, the computer program can be stored and provided to the computer using any type of non-transitory computer readable medium. The non-transitory computer readable medium includes any type of tangible storage medium. Examples of the non-transitory computer readable medium include magnetic storage media (such as floppy disks, magnetic tapes, hard disk drives, etc.), optical magnetic storage media (e.g. magneto-optical disks), CD-ROM (Read Only Memory), CD-R, CD-R/W, and semiconductor memories (such as mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory), etc.). The program may be provided to a computer using any type of transitory computer readable medium. Examples of the transitory computer readable medium include electric signals, optical signals, and electromagnetic waves. The transitory computer readable medium can provide the program to a computer via a wired communication line such as an electric wire or optical fiber or a wireless communication line.
Further, in addition to the case where the functions of the above-described exemplary embodiment are implemented by causing a computer to execute a program for implementing the functions of the above-described exemplary embodiment, the case where the functions of the above-described exemplary embodiment are implemented by this program in cooperation with the an OS (Operating System) or application software running on the computer is also included in the exemplary embodiment of the present invention. Further, the case where all or part of the processes of this program are executed by a function enhancement board inserted into the computer or a function enhancement unit connected to the computer to implement the functions of the above-described exemplary embodiment is also included in the exemplary embodiment of the present invention.
According to the embodiment, it is possible to provide a warning device, a warning method and a warning program for issuing an appropriate warning based on detection of an object even under the circumstances where it is difficult to determine the outside environment of a movable body.
Further, according to the embodiment, it is possible to provide a warning device, a warning method and a warning program for maintaining the accuracy of detecting the movement of an object and issuing an appropriate warning even under the circumstances where it is difficult to determine the outside environment of a movable body.
The exemplary embodiment is applicable to a warning device that is mounted on a movable body including a vehicle, and it has an industrial applicability.
| Number | Date | Country | Kind |
|---|---|---|---|
| 2015-046584 | Mar 2015 | JP | national |
| 2015-046585 | Mar 2015 | JP | national |
This application is a Continuation of U.S. patent application Ser. No. 16/591,537, filed on Oct. 2, 2019, which claims priority to U.S. patent application Ser. No. 15/698,099, filed on Sep. 7, 2017, which is a Bypass Continuation of International Application No. PCT/JP2016/001136, filed on Mar. 2, 2016, which is based upon and claims the benefit of priority from Japanese patent application No. 2015-046584 filed on Mar. 10, 2015 and Japanese patent application No. 2015-046585 filed on Mar. 10, 2015, the disclosures of which are incorporated herein in its entirety by reference.
| Number | Name | Date | Kind |
|---|---|---|---|
| 11423664 | Imbe | Aug 2022 | B2 |
| 20080197284 | Ebenstein et al. | Aug 2008 | A1 |
| 20120320212 | Aimura | Dec 2012 | A1 |
| 20130018547 | Ogata | Jan 2013 | A1 |
| Number | Date | Country |
|---|---|---|
| 101246219 | Aug 2008 | CN |
| 102782741 | Nov 2012 | CN |
| 102881186 | Jan 2013 | CN |
| 2015276 | Jan 2009 | EP |
| H07-160957 | Jun 1995 | JP |
| 2009-55553 | Mar 2009 | JP |
| 2011-146049 | Jul 2011 | JP |
| 2012-203884 | Oct 2012 | JP |
| Entry |
|---|
| International Search Report and Written Opinion for PCT/JP2016/001136, dated May 24, 2016, in 7 pages. |
| Number | Date | Country | |
|---|---|---|---|
| 20220351528 A1 | Nov 2022 | US |
| Number | Date | Country | |
|---|---|---|---|
| Parent | 16591537 | Oct 2019 | US |
| Child | 17866371 | US | |
| Parent | 15698099 | Sep 2017 | US |
| Child | 16591537 | US | |
| Parent | PCT/JP2016/001136 | Mar 2016 | US |
| Child | 15698099 | US |
