Aspects and embodiments of the invention are most generally directed to an optical imaging system, methods pertaining thereto, and applications thereof; more particularly to a panoramic optical imaging system, methods pertaining thereto, and applications thereof; and, most particularly to a panoramic optical imaging system that has zero or substantially no parallax, methods pertaining thereto, and applications thereof.
Current 360 degree systems without parallax employ an arrangement of mirrors to scan the image and are limited by an imaging speed of 10 frames per second (fps). Google uses a 360 degree camera with refractive lenses developed by Immersive Media to capture photos for its Streetview software. The photos must be post-processed and corrected for parallax, costing time, which reduces Google's ability to scale its Streetview initiatives. Fisheye lenses provide wide angle imaging but at the cost of high distortion. Distortion is the physical result of mapping a large spherical object onto a small flat image plane.
Some companies have developed optical systems to simplify the process of taking a panoramic image. Rather than rotating the camera to get multiple shots, all of the photos are captured simultaneously with many cameras imaging different parts of the scene. Immersive Media and Greypoint Imaging have developed single shot 360 degree cameras that are available for varying price tags between $10,000 and $100,000. Both companies develop software to automatically correct for the artifacts (parallax) created in the image and offer a better solution than panoramas captured by one camera, e.g., the iPhone camera. The software, however, is not perfect and many artifacts still exist in the images. Anecdotally, Google, had one person carry a Dodeca 360 camera (offered by Immersive Media) around the Grand Canyon, and had to employ programmers to correct the images frame by frame for the artifacts induced by parallax.
Parallax and the Chief Rays of an Optical System
Parallax is defined as “the effect whereby the position or direction of an object appears to differ when viewed from different positions, e.g., through the viewfinder and the lens of a camera.” Parallax is created as a result of stitching together images from multiple cameras, each with its own unique perspective of the world.
Referring to
The chief ray plays a critical role in the parallax created by stitching together multiple images.
The search for an algorithm that can correct for parallax has been going on for many years. Many solutions have been developed but even with the most sophisticated algorithms to date, artifacts are still left in panoramic images. For some, this may not be a problem as software engineers can be hired to fix the images frame by frame; however, for the general consumer this option of correcting each image is not feasible. A better solution is needed that effectively corrects for parallax before such a system can be made available to the consumer market. It is preferable to solve the problem of reducing parallax in an image optically, rather than computationally.
Current designs created for single shot panoramic imaging suffer from parallax because they are created from imaging systems with overlapping fields of view. FIG. 5 is taken from U.S. Pat. No. 2,696,758. This figure illustrates how parallax is created in the 360 degree imaging systems 50 available today. The field of views 28 overlap and a triangle that appears at the edge of the FOV 28 for the bottom lens system will appear at around 0.707 times the FOV 28 in the imaging system on top. Thus, the triangle is mapped to different image points for each camera 25. On the bottom it is mapped to the full FOV 28 (the edge of the image).
The inventor has thus recognized the advantages and benefits of a panoramic imaging system and associated methods in which there is no parallax, and where the parallax is eliminated optically rather than by post-processing software. Such a system would have applications including providing a scalable way to map the streets of the planet; allowing for the creation of virtual tours, both of cities and of private institutions; high frame-rate video surveillance; military applications including drone and tank technology; an alternative for fisheye lenses which provide wide angle imaging at the cost of high distortion.
An aspect of the invention is a multicamera panoramic imaging system having no parallax. According to a non-limiting embodiment, the multicamera panoramic imaging system includes a plurality of discrete, imaging systems disposed in a side-by-side array, wherein a field of view of each discrete, imaging system is conjoined with a field of view of each adjacent discrete imaging system, further wherein a stencil of chief rays at the edge of the field of view of any one of the discrete imaging systems will be substantially parallel to a stencil of chief rays at the edge of the field of view of any adjacent ones of the discrete imaging systems such that all of the substantially parallel stencils of chief rays appear to converge to a common point when viewed from object space. In various non-limiting embodiments, the multicamera panoramic imaging system may include or be further characterized by the following features, limitations, characteristics either alone or in various combinations thereof:
An aspect of the invention is a method for forming an image of an object having no parallax. According to a non-limiting embodiment, the method includes providing a panoramic imaging system, wherein the panoramic imaging system comprises a plurality of discrete imaging systems each characterized by a field of view; and constraining a stencil of chief rays at the edge of the field of view of every one of the discrete imaging systems to be substantially parallel to a stencil of chief rays at the edge of the field of view of an immediately adjacent one of the discrete imaging systems such that all of the parallel stencils of chief rays appear to converge to a common point when viewed from object space, wherein the imaging system is parallax-free. In various non-limiting embodiments, the panoramic imaging method may include or be further characterized by the following features, limitations, characteristics, steps either alone or in various combinations thereof:
An aspect of the invention is a method for designing a (substantially) parallax-free, panoramic imaging system. According to a non-limiting embodiment, the method includes determining an overall panoramic imaging system geometry, wherein the overall panoramic imaging system comprises a plurality of discrete, imaging systems having respective fields of view, disposed in a side-by-side array such that the fields of view of adjacent imaging systems conjoin; designing the discrete imaging systems such that a stencil of chief rays at the edge of the field of view of one of the discrete imaging systems will be substantially parallel to a stencil of chief rays at the edge of the field of view of an adjacent one of the discrete imaging systems such that the substantially parallel stencil of chief rays would appear to converge to a common point when viewed from object space. In various non-limiting embodiments, the panoramic imaging method may include or be further characterized by the following features, limitations, characteristics, steps either alone or in various combinations thereof:
For a panoramic camera to achieve minimal parallax, the field of views (FOV) of the imaging systems must not overlap. Thus, the chief ray at the edge of the FOV must approach the optical system parallel to the chief rays at the edge of the adjacent optical system.
NP Point (No Parallax Point)
To aid in the understanding of the previous concept, we define a term referred to as the No Parallax Point (NP Point). The NP Point 160 is an abstraction used for understanding how the chief rays 140 at the edge of the FOV 130 can physically be made to lie parallel to one another and what rules they should follow. The NP Point 160 is the point where the chief rays 140 at the edge of adjacent optical systems 110 intersect the optical axis when viewing the system from object space 135 for a panoramic imaging system 100 without parallax.
According to the embodied invention, the NP Points for each imaging system must lie in the same location. That is to say, that the rays of adjacent optical systems must be parallel.
An embodiment of the present invention relates to a multicamera panoramic imaging system 100, where the fields of adjacent imaging units 110 merge to form the composite field of view of the entire imaging system, as illustrated in the schematic of
In the instant exemplary embodiment, the rays striking the edge of one imaging unit are constrained to lie parallel to the incoming rays of an adjacent imaging unit so that both imaging systems share the same set of edge rays. As seen in the 3-dimensional model of
The embodied panoramic imaging system 200 utilizes the aforementioned technique of designing an imaging system with a NP point 260 behind the image sensor (225), and combines multiple lens systems (210) in a dodecahedron geometry, to create a 360 degree FOV camera (200) with minimal or no parallax.
The first lens element 270 will be shaped into the surface of a regular pentagon 267. The complete system will be composed of 12 discrete imaging units 210, each with a common NP point 260 for rays along the edge of the pentagon 267 and constrained to have incident angles meeting the geometry specified by that of a dodecahedron.
A dodecahedron is a polyhedron with 12 surfaces. A polyhedron is a three dimensional solid consisting of a collection of polygons joined at the edges. Each side of the dodecahedron is a regular pentagon (a pentagon with equal length sides). Dodecahedrons have some important geometrical properties that must be understood in order to design a lens system 210 utilizing the geometry. The properties will be discussed in turn next after briefly discussing why the first lens must be shaped into the surface of a pentagon.
By using a circularly edged lens as the first element 270 in the dodecahedron geometry, it is not possible to capture all information in the 360 degree field of view using the current technique of aligning edge rays. The missing area from where the first lens 270 is inscribed in the pentagon 267 (shaded region in
The following description is meant to illustrate the geometry of a dodecahedron and is necessary when creating a lens system 210 utilizing the aforementioned NP technique and a dodecahedron geometry, but is not essential for the purposes of creating the no parallax, panoramic imaging system 200 embodied herein.
Property 1: Diameter of Circle Circumscribing Regular Pentagon
For each of the 12 individual lens systems 210, the first lens 270 will be designed such that it circumscribes each of the regular pentagons 267 of the dodecahedron as shown in
D=a/sin(36°)=1.7013a
In the equation above, “a” is the side length of the regular pentagon. The first lens element 270 of each system (210) will fully circumscribe each pentagon 267 and so the diameter of the first lens element 270 for each system (210) is given as 1.7013a as illustrated in
Property 2: Inscribed Sphere Touching Center of Each Pentagon
The radius of an inscribed sphere (tangent to each of the dodecahedron's faces) is:
This radius is the distance from the center 280 of the dodecahedron, which will be the NP Point 260 for each lens (210) in this design, and the center of the pentagon's face, which coincides with the center (optical axis) of the first lens element 270 in a system occupying that pentagon. This point is at the center of each pentagon face. The length between the NP point 260 and the center of the dodecahedron is constrained to be 1.1135a where a is the length of one of the pentagon sides, as illustrated in
Property 3: Mid-Radius of Dodecahedron
The mid-radius is the point connecting the center of the dodecahedron and the middle of each edge. This length is given as follows:
This equation constrains the distance between the top of the pentagon face and the NP Point, as illustrated in
Constraints
The geometric properties of a dodecahedron constrain the design of the 12 lenses (210) that will embody it. In particular, we have the following four parameters based upon the description given above:
1. Diameter of 1st lens element 270: 1.7013a;
2. Distance from 1st lens element 270 to center of dodecahedron: 1.1135a;
3. Distance from top of 1st lens element 270 to center of dodecahedron: 1.31a;
4. FOV=37.3777 degrees Given any two of the first three constraints, we have that the angle between the optical axis 220 of the lens (210) and the top of the first lens element 270 is 37.3777 degrees (see
tan−1((1.7013/2)/1.1135)−37.377°
We want this angle of 37.37 degrees to be the field of view 230 of the lens (210). This will ensure that the NP Point 260, that is the point where the chief ray 240 of the blending (the blending angle being the full FOV) intersects the optical axis 220 in object space 235, lies at the center 280 of the dodecahedron. All of the other constraints will ensure that the lens elements 278 lie before the NP Point 260 and that the elements 278 fall within the 31.717 degree half angle cone of light.
Diameter of Other Lens Elements and Sensor
With the four constraints given above, we know what the size of each lens element 278 after the first (270) must be in order to fit into the dodecahedron geometry. In order for the preceding lens elements (278) to fit, any lens or sensor element must fit inside of the 31.717 degree cone of light beginning at the center of the dodecahedron and tangential to the diameter of the first lens element 270. As the distance from the first lens element 270 increases, the diameter of the preceding lens elements 278 will decrease proportionally (see
The maximum diameter of any lens element 278 or sensor (225) preceding the first can be found geometrically to be less than or equal to (1.1135a−D)*tan(31.716 degrees) where D is the distance of that element from the first lens element 270.
Thus, we now have the five constraints that will allow this lens system 210 to match the geometry of a dodecahedron and permit 360 degree imaging:
A geometry for the lenses (210) was chosen. Platonic solids have the property that they are composed of many solids of equal geometry and volume. For a system imaging 360 degrees, this allows the composite imaging system to be made from the same replicated lens design. A dodecahedron geometry was chosen because it is approximately spherical in its geometry.
In order for the edge rays of one imaging unit 210 to lie parallel to those of an adjacent unit, they must enter at the same angle. The angle shared by both imaging units is that of the dodecahedrons edge surface. At the center of the edge surface, the angle with respect to the center of the dodecahedron center is 31.717 degrees, as illustrated in
In order to make the rays along adjacent imaging units 210 match, the first lens 270 of the imaging unit 210 is cut into a pentagon 267, matching the surface of the dodecahedron. At the center of the edge, the ray 240 striking the surface enters with an angle of incidence of 31.717 degrees. At the corner of the edge, the angle of incidence of an entering ray is 37.377 degrees. At all points along the edge of the lens (270), the angle of incidence of an entering ray is made to match the geometry of the dodecahedron surface.
The angle of incidence for 37 rays along the edge of the pentagon lens was calculated using trigonometry, knowing the distance from the center of the dodecahedron to the center of the pentagon face, and knowing the distance from the center 280 of the dodecahedron to the edge point in question as shown in the
A diagram illustrating the ray constraints is shown in
The instant application is a continuation application of U.S. application Ser. No. 17/742,964, which is a continuation application of U.S. application Ser. No. 16/845,721, now U.S. Pat. No. 11,363,194, which is a continuation application of U.S. application Ser. No. 16/420,752, now U.S. Pat. No. 10,659,688, which is a continuation application of U.S. application Ser. No. 15/309,180, now U.S. Pat. No. 10,341,559 B2, which is a 371 of international application No. PCT/US2015/029146, which claims the benefit of U.S. provisional application No. 61/989,136 filed May 6, 2014, the subject matter of which is incorporated herein by reference in its entirety.
Number | Name | Date | Kind |
---|---|---|---|
2696758 | Angenieux | Dec 1954 | A |
5023725 | McCutchen | Jun 1991 | A |
5130794 | Ritchey | Jul 1992 | A |
5703604 | McCutchen | Dec 1997 | A |
6141034 | McCutchen | Oct 2000 | A |
6549650 | Ishikawa et al. | Apr 2003 | B1 |
6618093 | Levy | Sep 2003 | B1 |
6870680 | Yoshikawa et al. | Mar 2005 | B2 |
7006123 | Yoshikawa et al. | Feb 2006 | B2 |
7015954 | Foote et al. | Mar 2006 | B1 |
7119961 | Yoshikawa et al. | Oct 2006 | B2 |
7126756 | Yoshikawa et al. | Oct 2006 | B2 |
7268805 | Yoshikawa et al. | Sep 2007 | B2 |
7277118 | Foote | Oct 2007 | B2 |
7440637 | Schechner et al. | Oct 2008 | B2 |
7515177 | Yoshikawa et al. | Apr 2009 | B2 |
7548373 | Yoshikawa et al. | Jun 2009 | B2 |
7620909 | Park et al. | Nov 2009 | B2 |
7664340 | Yoshikawa | Feb 2010 | B2 |
7710463 | Foote | May 2010 | B2 |
7782357 | Cutler | Aug 2010 | B2 |
7817354 | Wilson | Oct 2010 | B2 |
7834910 | Delorme et al. | Nov 2010 | B2 |
9503638 | Nguyen et al. | Nov 2016 | B1 |
20040051805 | Yoshikawa | Mar 2004 | A1 |
20060114332 | Yoshikawa | Jun 2006 | A1 |
20070091195 | Yoshikawa | Apr 2007 | A1 |
20070182812 | Ritchey | Aug 2007 | A1 |
20110069189 | Venkataraman et al. | Mar 2011 | A1 |
20110080487 | Venkataraman et al. | Apr 2011 | A1 |
20110304612 | Ohyama | Dec 2011 | A1 |
20130076900 | Mrozek | Mar 2013 | A1 |
20140146132 | Bagnato et al. | May 2014 | A1 |
Number | Date | Country |
---|---|---|
2009210672 | Aug 2009 | AU |
1745337 | Mar 2006 | CN |
1593999 | Nov 2005 | EP |
2003162018 | Jun 2003 | JP |
2004184862 | Jul 2004 | JP |
2007110228 | Apr 2007 | JP |
2399073 | Oct 2010 | RU |
2007046498 | Apr 2007 | WO |
2012066642 | May 2012 | WO |
2013154433 | Oct 2013 | WO |
Entry |
---|
International Search Report in PCT/US2015/029146 dated Jul. 14, 2015. |
Written Opinion of the International Searching Authority in PCT/US2015/029146 dated Nov. 12, 2015. |
Examination report dated Nov. 17, 2021 from Canadian Intellectual Property Office, Appl. No. 2,953,337 (PCT No. US2015029146). |
Examination report dated Sep. 16, 2019 from China National Intellectual Property Administration, Appl. No. 201580034186.5. |
Notice of Grant dated Jan. 13, 2021 from China National Intellectual Property Administration, Appl. No. 201580034186.5. |
Examination report dated Jul. 12, 2021 from China National Intellectual Property Administration, Appl. No. 202010190853.1 (divisional of Appl. No. 201580034186.5). |
Examination report dated Jul. 9, 2018 from European Patent Office, Appl. No. 15 723 390.9. |
Examination report dated Nov. 27, 2019 from European Patent Office, Appl. No. 15 723 390.9. |
Examination report dated Jul. 15, 2020 from European Patent Office, Appl. No. 15 723 390.9. |
Intention to Grant dated Jun. 3, 2021 from European Patent Office, Appl. No. 15 723 390.9. |
Search Opinion dated Jun. 21, 2021 from European Patent Office, Appl. No. 21162087.7 (continuation of Appl. No. 15 723 390.9). |
Examination report dated Jan. 7, 2020 from Intellectual Property India, Appl. No. 201617041154. |
Decision of Refusal dated Feb. 25, 2020 from Japanese Patent Office, Appl. No. 2017-511155. |
Notice of Reasons for Refusal dated Jan. 28, 2021 from Japanese Patent Office, Appl. No. 2017-511155. |
Trial and Appeal Decision dated Jul. 12, 2021 from Japanese Patent Office, Appl. No. 2017-511155. |
Notice of Reasons for Refusal dated Jun. 24, 2021 from Japanese Patent Office, Appl. No. 2020-109314 (continuation of Appl. No. 2017-511155). |
Decision of Refusal dated Jan. 11, 2022 from Japanese Patent Office, Appl. No. 2020-109314 (continuation of Appl. No. 2017-511155). |
Notice of Non-Final Rejection dated Dec. 18, 2018 from Korean Intellectual Property Office, Appl. No. 10-2016-7032965. |
Notice of Allowance of Patent dated Nov. 4, 2019 from Korean Intellectual Property Office, Appl. No. 10-2016-7032965. |
Notice of Non-Final Rejection dated Jul. 28, 2020 from Korean Intellectual Property Office, Appl. No. 10-2020-7002743 (divisional of Appl. No. 10-2016-7032965). |
Notice of Allowance dated Jan. 25, 2021 from Korean Intellectual Property Office, Appl. No. 10-2020-7002743 (divisional of Appl. No. 10-2016-7032965). |
Written Opinion from Intellectual Property Office of Singapore in corresponding Application No. 11201609780P, dated Sep. 7, 2017. |
Moore, Dustin, “Where Is the Minimum Parallax Point?” Apr. 21, 2012. |
Notice of Eligibility of Grant from Intellectual Property Office of Singapore in corresponding Application No. 11201609780P, dated Jun. 21, 2018. |
Patent Search Report dated Dec. 14, 2018 from Federal Service for Intellectual Property (ROSPATENT) Federal Institute of Industrial Property, in corresponding Russia Patent Application No. 2016145439/28(073015). |
Office Action (Enquiry) dated Dec. 21, 2018 from Patent Office of Russian Federation, Federal Institute of Industrial Property, in corresponding Russia Patent Application No. 2016145439/28(073015). |
Zhang, Cha et al., “A Survey on Image-Based Rendering,” Electrical and Computer Engineering, Carnegie Mellon University, Jun. 2003, Pittsburgh, PA. |
Qi, Zhi et al., “Overcoming Parallax and Sampling Density Issues in Image Mosaicing of Non-Planar Scenes,” Centre for Intelligent Machines, McGill University. |
McCutchen, David, “A Dodecahedral Approach to Immersive Imagine and Display,” Computer Graphics, May 1997, pp. 35-37. |
Peleg, Shmuel et al., “Panoramic Mosaics by Manifold Projection,” Proc. of Int'l Conf. on Computer Vision and Pattern Recognation, 1997: 338-343, San Juan, Puerto Rico. |
Qi Zhi, Towards Dynamic Mosaic Generation with Robustness to Parallax Effects, Department of Electrical Engineering, McGill University, Montreal, Quebec, 2008. |
Shum, Heung-Yeung et al., “Construction and Refinement of Panoramic Mosaics with Global and Local Alignment,” Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271), Jan. 7, 1998, Bombay, India. |
Wilburn, Bennett et al., “High Performance Imaging Using Large Camera Arrays,” Association of Computing Machinery, Inc., U.S., 2005. |
Moore, Dustin, “Where is the Minimum Parallax Point?”, 2012, United States. |
Icosahedron, http://en.wikipedia.org/wiki/Icosahedron, May 4, 2012. |
Notification of Reasons for Refusal in corresponding Japanese Patent Application No. 2017-511155, dated Mar. 19, 2019. |
Number | Date | Country | |
---|---|---|---|
20230388642 A1 | Nov 2023 | US |
Number | Date | Country | |
---|---|---|---|
61989136 | May 2014 | US |
Number | Date | Country | |
---|---|---|---|
Parent | 17742964 | May 2022 | US |
Child | 18231656 | US | |
Parent | 16845721 | Apr 2020 | US |
Child | 17742964 | US | |
Parent | 16420752 | May 2019 | US |
Child | 16845721 | US | |
Parent | 15309180 | US | |
Child | 16420752 | US |