The disclosure relates in general to a cross-domain image comparison method and a cross-domain image comparison system.
In recent years, self-driving cars have accidents on the road. Therefore, various driving simulation tests before self-driving cars are very important. In particular, some accident videos are used to predict the self-driving response capability. If a real vehicle crash is used to produce the real accident video, it must inevitably require a relatively high cost. Computer graphics are useful to assist the production of similar accident videos and has become a good alternative.
However, how high the confidence of the synthesized accident video generated by computer is an important factor for the success of the driving simulation test. The synthesized accident video and the real accident video must have a certain level of similarity. The synthesized accident video and the real accident video are generated by different devices. That is to say, the synthesized accident video and the real accident video are cross-domain. The synthesized accident video and the pixels in the real accident video are quite different in pixel level, and it is difficult to be compared. Traditional comparison method cannot obtain the similarity between the synthesized accident video and the real accident video. Therefore, researchers are working on developing a cross-domain image comparison method to assist the self-driving simulation test, or other applications.
The disclosure is directed to a cross-domain image comparison method and a cross-domain image comparison system.
According to one embodiment, a cross-domain image comparison method includes the following steps. Two videos in cross-domain are obtained. The videos are generated by different types of devices. A plurality of semantic segmentation areas are obtained from one frame of each of the videos. A region of interest pair (ROI pair) is obtained according to moving paths of the semantic segmentation areas in the videos. Two bounding boxes and two central points of the ROI pair are obtained. A similarity between the frames is obtained according to the bounding boxes and the central points.
According to another embodiment, a cross-domain image comparison system is provided. The cross-domain image comparison system includes an inputting unit, a semantic segmentation unit, a ROI unit, a bounding box unit and a similarity unit. The inputting unit is used for obtaining two videos in cross-domain. The videos are generated by different types of devices. The semantic segmentation unit is used for obtaining a plurality of semantic segmentation areas from one frame of each of the videos. The ROI unit is used for obtaining a region of interest pair (ROI pair) according to moving paths of the semantic segmentation areas in the videos. The bounding box unit is used for obtaining two bounding boxes and two central points of the ROI pair. The similarity unit is used for obtaining a similarity between the frames according to the bounding boxes and the central points.
In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. It will be apparent, however, that one or more embodiments may be practiced without these specific details. In other instances, well-known structures and devices are schematically shown in order to simplify the drawing.
Please refer to
Please refer to
Please refer to
Next, in the step S120, the semantic segmentation unit 120 obtains a plurality of semantic segmentation areas S11, S12, S21, S22 from the frames F1, F2 of the videos VD1, VD2. As shown in
Then, in the step S130, the ROI unit 130 obtains regions of interest pair (ROI pair) R01, R02 according to moving paths of the semantic segmentation areas S11, S12, S21, S22 in the videos VD1, VD2. The semantic segmentation areas S11, S12 corresponding the ROI pair R01 are obtained from the different videos VD1, VD2. The semantic segmentation areas S21, S22 corresponding the ROI pair R02 are obtained from the different videos VD1, VD2. As shown in
According to the moving path of the semantic segmentation area S12 in the video VD1 and the moving path of the semantic segmentation area S22 in the video VD2, the ROI unit 130 finds that the moving path of the semantic segmentation area S12 and the moving of the semantic segmentation area S22 are similar. The semantic segmentation area S12 and the semantic segmentation area S22 are deemed as another identical object, so the semantic segmentation area S12 and the semantic segmentation area S22 are linked to be the ROI pair R02. After obtaining the ROI pair R01, the ROI pair R01 can be analyzed to obtain the similarity between the frame F1 and the frame F2. Similarly, after obtaining the ROI pair R02, the ROI pair R02 can be analyzed to obtain the similarity between the frame F1 and the frame F2. If the similarity between the semantic segmentation area S11 and the semantic segmentation area S21 in the ROI pair R01 is high, then it can be inferred that the frame F1 and the frame F2 have high similarity; if the similarity between the semantic segmentation area S12 and the semantic segmentation area S22 in the ROI pair R02 is high, then it can be inferred that the frame F1 and the frame F2 have high similarity.
Afterwards, in the step S140, the bounding box unit 140 obtains bounding boxes B11, B12, B21, B22 and central points C11, C12, C21, C22 of the ROI pairs R01, R02. As shown in
For example, the central points C11, C12, C21, C22 may be the intersections of diagonal lines of the bounding boxes B11, B12, B21, B22 respectively. So far, the cross-domain image comparison system 100 already obtains the counters, the bounding boxes B11, B12, B21, B22 and the central points C11, C12, C21, C22 of the semantic segmentation areas S11, S12, S21, S22 in the ROI pairs R01, R02. According to the above mentioned information, the similarity between the frame F1 and the frame F2 can be obtained in the following steps.
Then, in the step S150, the similarity unit 150 obtains the similarity between the frame F1 and the frame F2 at least according to the bounding boxes B11, B12, B21, B22 and the central points C11, C12, C21, C22. As shown in
Please refer to
Please refer to
As shown in
The position similarity analyzer 151 obtains the position similarity degree simpos of the ROI pair R01 and the ROI pair R02 according to the above information. The position similarity degree simpos may be calculated according to the following equation (1).
|ROI pair| is the number of the ROI pairs R01, R02. In this case, the number of the ROI pairs R01, R02 is 2. That is to say, if the central point C11 is close to the central point C21, and the central point C12 is close to the central point C22, the position similarity degree simpos will approach to 1. On the contrary, if the central point C11 is far from the central point C21, and the central point C12 is far from the central point C22, the position similarity degree simpos will approach to 0.
Please refer to
As shown in
The angle similarity analyzer 152 obtains the angle similarity degree simorie of the ROI pair R01 and the ROI pair R02 according to the relative angles r1, r2. The angle similarity degree simorie may be calculated according to the following equation (2).
That is to say, if the degree of inclination of the bounding box B11 is close to that of the bounding box B21 and the degree of inclination of bounding box B12 is close to that of the bounding box B22, the angle similarity degree simorie will approach to 1. On the contrary, if the degree of inclination of the bounding box B11 is far from that of the bounding box B21 and the degree of inclination of the bounding box B12 is far from that of the bounding box B22, the angle similarity degree simorie will approach to 0.
Refer to
The size similarity analyzer 153 obtains the size similarity degree simsize of the ROI pair R01 and the ROI pair R02 according to the diagonal lengths d1,1, d1,2, d2,1, d2,2. The size similarity degree simsize may be calculated according to the following equation (3).
That is to say, if the size of the bounding box B11 is close to the size of the bounding box B21 and the size of the bounding box B12 is close to the size of the bounding box B22, the size similarity degree simsize will approach to 1. On the contrary, if the size of the bounding box B11 is far from the size of the bounding box B21 and the size of the bounding box B12 is far from the size of the bounding box B22, the size similarity degree simsize will approach to 0.
Please refer to
As shown in
Afterwards, the process proceeds to the step S155. As shown in
According to the embodiments described above, the cross-domain image comparison system 100 and the cross-domain image comparison method apply the semantic segmentation technology to reduce the complexity of the videos in cross-domain, and analyzes the ROI pairs to obtain the similarity. As such, the similarity between the videos and the video in cross-domain can be obtained to assist the application of the self-driving simulation, the dance training or the gymnastic training. The operation of those elements is illustrated via a flowchart.
It will be apparent to those skilled in the art that various modifications and variations can be made to the disclosed embodiments. It is intended that the specification and examples be considered as exemplary only, with a true scope of the disclosure being indicated by the following claims and their equivalents.
Number | Name | Date | Kind |
---|---|---|---|
7702185 | Keating | Apr 2010 | B2 |
8532396 | Rezazadeh et al. | Sep 2013 | B2 |
8712156 | Bronstein et al. | Apr 2014 | B2 |
9008424 | Cordara et al. | Apr 2015 | B2 |
20050002568 | Chupeau et al. | Jan 2005 | A1 |
20160379055 | Loui | Dec 2016 | A1 |
20180025622 | Lindoff et al. | Jan 2018 | A1 |
20200174490 | Ogale | Jun 2020 | A1 |
20200202615 | Kwon | Jun 2020 | A1 |
Number | Date | Country |
---|---|---|
103218810 | Jul 2013 | CN |
105857547 | Jun 2016 | CN |
106203242 | Dec 2016 | CN |
107092870 | Aug 2017 | CN |
109670555 | Apr 2019 | CN |
201118763 | Jun 2011 | TW |
201933182 | Aug 2019 | TW |
WO 2015196937 | Dec 2015 | WO |
Entry |
---|
Li et al., “A Novel Visual-Vocabulary-Translator-Based Cross-Domain Image Matching”, IEEE Access, vol. 5, 2017, pp. 23190-23203. |
Lin et al., “Cross-Domain Visual Matching via Generalized Similarity Measure and Feature Learning”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, No. 6, Jun. 2017, pp. 1089-1102. |
Melo et al., “Cross-domain image matching improved by visual attention”, Journal of VSCG, ISSN 1213-6972, vol. 22, 2014, pp. 65-72. |
Shrivastava et al., “Data-driven Visual Similarity for Cross-domain Image Matching”, SA '11: Proceedings of the 2011 SIGGRAPH Asia Conference, Article No. 154, Dec. 2011, pp. 1-9. |
Xu et al., “Cross-Paced Representation Learning With Partial Curricula for Sketch-Based Image Retrieval”, IEEE Transactions on Image Processing, vol. 27, No. 9, Sep. 2018, pp. 4410-4421. |
Zhang et al., “Sketch-Based Cross-Domain Image Retrieval via Heterogeneous Network”, VCIP 2017, Dec. 10-13, 2017, pp. 1-4. |
Taiwanese Office Action and Search Report, dated Dec. 31, 2020, for Taiwanese Application No. 109103872. |
Number | Date | Country | |
---|---|---|---|
20210201093 A1 | Jul 2021 | US |