1. Field of Invention
The present invention relates to an efficient three dimensional scanning of an object with reduced computation time, memory requirements, disc storage and network bandwidth usage requirements.
2. Summary of the Invention
The new algorithm aims to select best keyframes from a set of frames (e.g. a video stream) for subsequent processing with the main application in 3D scanning. The proposed scheme works both in online mode (when frames are captured one-by-one and keyframe selection is accomplished on-the-fly) and in offline mode (when all frames are already captured). The proposed algorithm simultaneously fulfills several criteria: the online process should be intuitive for the user and convey his/her intent, the scanned entity (object, person, room, etc.) should be covered from all view angles, the selected images should have the highest level of details to allow texture of the best quality.
All available raw frames for a scan contain redundant information so it is possible to select only several keyframes to achieve reduction in computation time, memory requirements, disk storage and network bandwidth usage. The problem is that in using a naïve approach (e.g. just selecting every 10th frame) the result is a degradation of the quality of a 3D model because such an approach can drop occasionally a high-quality frame and keep a blurred frame. So the goal of the present invention is to develop an algorithm of keyframe selection than can bring the advantages but without harming the final result and user experience.
Two algorithms were developed for selection of keyframes: the first is for online mode, and the second one is for offline mode.
1.1. For each new frame in the timeframe:
1.2. use the best frame in the timeframe as the keyframe (e.g. remember it in memory, write to a disk, send to cloud, etc.)
Online keyframe selection is commonly achieved by adding a new keyframe when a user moves too far from the position of a previous keyframe (measured e.g. by geometric distances or by decreased stability of tracking) and without taking into account quality of this keyframe. However, the situation when a user departs too far from a previous position is usually caused by fast camera movements and so the selected keyframe can be blurry. The presently disclosed online keyframe selection algorithm finds the accidental pauses in the continuous motion of a camera and takes a keyframe at exactly this point and so get significantly less blurry frames. Also it better conveys the intent of a user and gives him/her intuitive behavior. For example, if a user wants to scan a part with high level of details and scans this part thoroughly, the proposed method will select more keyframes for this part than for other regions that a user did not spend much time on. Offline selection takes into account all available data and produces keyframe suitable both for meshing and texturing. Usual strategies for offline keyframe selection in 3D reconstruction aim to select keyframes only for reliable determination of camera poses, their internal parameters and locations of features but ignore requirements of subsequent essential tasks: meshing and texturing. The present strategy resolves this problem and allows us to obtain triangulated and textured 3D models of high quality. Also a novel efficient way is introduced to compute similarity between two point clouds, which reflects coverage of a scanned entity by these two clouds.
The invention is not limited by the embodiments described above which are presented as examples only but can be modified in various ways within the scope of protection defined by the appended patent claims.
Thus, while there have been shown and described and pointed out fundamental novel features of the invention as applied to a preferred embodiment thereof, it will be understood that various omissions and substitutions and changes in the form and details of the devices illustrated, and in their operation, may be made by those skilled in the art without departing from the spirit of the invention. For example, it is expressly intended that all combinations of those elements and/or method steps which perform substantially the same function in substantially the same way to achieve the same results are within the scope of the invention. Moreover, it should be recognized that structures and/or elements and/or method steps shown and/or described in connection with any disclosed form or embodiment of the invention may be incorporated in any other disclosed or described or suggested form or embodiment as a general matter of design choice. It is the intention, therefore, to be limited only as indicated by the scope of the claims appended hereto.
Jaccard, Paul (1912), “The distribution of the flora in the alpine zone”, New Phytologist 11: 37-50.
Lance, G. N., & Williams, W. T. (1967). A general theory of classificatory sorting strategies 1. Hierarchical systems. The computer journal, 9(4), 373-380.
D. Marr (1982). Vision. San Francisco: Freeman
Ahmed, M. T., Dailey, M. N., Landabaso, J. L., & Herrero, N. (2010, May). Robust Key Frame Extraction for 3D Reconstruction from Video Streams. In VISAPP (1) (pp. 231-236).
Rashidi, A., Dai, F., Brilakis, I., & Vela, P. (2013). Optimized selection of key frames for monocular videogrammetric surveying of civil infrastructure. Advanced Engineering Informatics, 27(2), 270-282.
Park, M. G., & Yoon, K. J. (2011). Optimal key-frame selection for video-based structure-from-motion. Electronics letters, 47(25), 1367-1369.
Knoblauch, D., Hess-Flores, M., Duchaineau, M. A., Joy, K. I., & Kuester, F. (2011). Non-parametric sequential frame decimation for scene reconstruction in low-memory streaming environments. In Advances in Visual Computing (pp. 359-370). Springer Berlin Heidelberg.
Dong, Z., Zhang, G., Jia, J., & Bao, H. (2009, September). Keyframe-based real-time camera tracking. In Computer Vision, 2009 IEEE 12th International Conference on (pp. 1538-1545). IEEE.
This application claims the benefit of U.S. Provisional Application No. 62/151,520, filed Apr. 23, 2015, the entire content of which is incorporated by reference.
Number | Date | Country | |
---|---|---|---|
62151520 | Apr 2015 | US |