Patent Application
20170180702
This application claims priority from European Application No. 15307067.7, entitled “Method and System for Estimating the Position Of A Projection Of A Chief Ray On A Sensor Of A Light-Field Acquisition Device,” filed on Dec. 18, 2015, the contents of which are hereby incorporated by reference in its entirety.
The present disclosure relates to image or video camera calibration. More precisely, the present disclosure generally relates to a method and a system for estimating the position of a projection of a chief ray on a sensor of an image acquisition device, notably a plenoptic or multi-lens camera.
The present section is intended to introduce the reader to various aspects of art, which may be related to various aspects of the present disclosure that are described and/or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.
Image acquisition devices project a three-dimensional scene onto a two-dimensional sensor. During operation, a conventional capture device captures a two-dimensional (2-D) image of the scene representing an amount of light that reaches a photosensor (or photodetector or photosite) within the device. However, this 2-D image contains no information about the directional distribution of the light rays that reach the photosensor (which may be referred to as the light-field).
Moreover, it is more and more frequent to post-process the image data captured by the sensor, and to run computational photography algorithms on the acquired signals.
However, in order for such image data processing to be performed correctly, it is necessary to have accurate calibration data relating to the image acquisition device used to capture such image or video data.
Notably, when considering a sensor device observing an object space from the image space of an optical system, it is necessary to estimate, for each pixel of the sensor, to which direction(s), or beam(s), it corresponds in the object space (i.e. which portion of the object space is sensed by this pixel). In the present disclosure, the terms “object space” and “image space” respectively stand for the input and output optical spaces usually defined in the optical design discipline. Hence, the “object space” is the observed scene in front of the main lens of an image acquisition device, while the “image space” is the optical space after the optical system of the image acquisition device (main lens, microlenses, . . . ) where the imaging photosensor captures an image.
Among the required calibration data, what is first needed is to identify the chief ray direction in the object space of the beam corresponding to a sensor pixel. The chief ray corresponds to the optical axis of an optical system.
According to known prior art techniques, optical systems calibration mainly use checkerboards or grids of points in the object space to estimate the position of corners or intersection points on the acquired images in the image space. For a given optical configuration (a given zoom/focus of the optical acquisition device), grid points or corners positions are estimated with sub-pixel image processing techniques, and these estimates are provided to a model generalizing the estimated positions to the entire field of view.
Such a perspective projection model is usually taken as a starting point for optical acquisition devices calibration. It is then supplemented with distortion terms, in order to get very precise calibration of all pixels in the camera.
In “A Generic Camera Model and Calibration Method for Conventional, Wide-Angle, and Fish-Eye Lenses”, Pattern Analysis and Machine Intelligence, IEEE Transactions on 28, no. 8 (2006): 1335-1340, Kannala et al. consider that the perspective projection model is not suitable for fish-eye lenses, and suggest to use a more flexible radially symmetric projection model. This calibration method for fish-eye lenses requires that the camera observe a planar calibration pattern.
In “Multi-media Projector-Single Camera Photogrammetric System For Fast 3d Reconstruction”, International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, Commission V Symposium, pp. 343-347. 2010, V. A. Knyaz proposes the use of a multimedia projector to simultaneously calibrate several cameras in a 3D reconstruction context.
Existing calibration methods hence rely on a global model transforming the geometry in the object space to the geometry in the image space. However, such prior art techniques are not suited for light field acquisition devices, which show a complex design and embed optical elements like lenslet arrays, which do not always follow specifications with all the required precision.
It is actually recalled that light-field capture devices (also referred to as “light-field data acquisition devices”) have been designed to measure a four-dimensional (4D) light-field of the scene by capturing the light directions from different viewpoints of that scene. Thus, by measuring the amount of light traveling along each beam of light that intersects the photosensor, these devices can capture additional optical information (information about the directional distribution of the bundle of light rays) for providing new imaging applications by post-processing. The information acquired/obtained by a light-field capture device is referred to as the light-field data.
Light-field capture devices are defined herein as any devices that are capable of capturing light-field data. There are several types of light-field capture devices, among which:
For light field acquisition devices, a precise model of the optics (including defects such as microlens array deformations or misalignment) is more complex than with classical single pupil optical systems. Moreover, with light field acquisition devices, blur or vignetting can affect image forming, distorting the relationship between a source point and its image on the sensor. Last, the notion of stationary Point Spread Function, which is used when calibrating conventional image acquisition devices, does not hold for light field acquisition devices.
It is hence necessary to provide calibration techniques, which are suited for calibrating light field acquisition devices.
Actually, the goal of plenoptic camera calibration is to determine the positions of the centers of the micro-images formed on the sensor by the array of micro-lenses. Such centers are also called micro-centers. The positions of micro-centers are indeed required for computing focal stacks or matrices of views, i.e. for turning raw plenoptic data into workable imaging material. Yet, micro-centers localization is not straightforward, notably because peripheral pixels are especially difficult to exploit since they are dark.
Specific calibration methods and models have hence been proposed for plenoptic or camera arrays acquisition, as in “Using Plane+Parallax for Calibrating Dense Camera Arrays”, Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on, vol. 1, pp. 1-2. IEEE, 2004 by Vaish et al. This document describes a procedure to calibrate camera arrays used to capture light fields using a planar+parallax framework.
It would be desirable to provide a new technique for estimating the chief ray projection on the sensor of a light field acquisition device, which would allow calibrating light field acquisition devices with higher precision. More precisely, it would be desirable to provide a new technique for estimating the position of the chief ray on the sensor of a light field acquisition device, which could form a prior step for a set of robust methods allowing to spot the centers of micro-images on the whole image formed on the sensor.
According to an embodiment, a method for estimating the position, on a sensor of a light-field acquisition device, of a projection of a chief ray, corresponding to an optical axis of said light-field acquisition device is provided, which comprises:
The present disclosure thus relies on a different approach for estimating the position of the chief ray projection on the sensor of a plenoptic camera.
Actually, it consists in estimating the position of the chief ray within the raw picture formed on the sensor through uniform white lighting, thanks to an analysis of the shapes formed on this raw image by the optical system of the light-field acquisition device. When considering a plenoptic camera, it comprises a main lens (or a set of lenses which may be assimilated to a unique main lens), a photo-sensor, and a micro-lens array located between the main lens and the sensor. The main lens focuses the subject onto, or near, the micro-lens array. The micro-lens array splits the incoming light rays according to their directions. As a consequence, micro-images form on the sensor.
The present disclosure advantageously makes use of the optical (aka artificial) vignetting, which affects peripheral micro-images shape, making them look like a cat's eye, or an almond, while central micro-images show the same shape as the micro-lenses, i.e. are usually circular or hexagonal. It further relies on cross-correlation computation, which is an interesting mathematical tool for finding repeating patterns, and which allows to achieve shape matching on the raw image formed on the sensor.
Hence, by analyzing the shapes of the micro-images formed on the sensor, it is possible to determine a center of symmetry for this repeating pattern of shapes, which gives a coarse estimate of the position of the chief ray projection on the sensor.
It is important to note that this center of symmetry may be sought in the whole raw image, for brute force search, or in a candidate area, corresponding to a restricted central area of the raw image, for lighter computational load.
It is recalled that, in the present disclosure, the chief ray corresponds to the optical axis of the optical system of the light-field acquisition device. Furthermore, it is assumed that the chief ray also corresponds to the axis of a cylinder defined by the rims surrounding front and rear elements of the main lens of the light-field acquisition device.
The coarse estimate of the chief ray projection on the sensor may be used in a further plenoptic camera calibration process for recovering the whole pattern of micro-centers (i.e. centers of the micro-images) in the raw white image.
According to an embodiment, such a method also comprises refining said coarse estimate of the position of the chief ray projection by analyzing energy fall-off on part of the candidate area.
Hence, the present disclosure also relies on the effect of natural vignetting, which makes peripheral pixels darker than center pixels, in order to refine the coarse estimate of the chief ray projection. Shape matching's coarse result thus initializes the range of research for illumination falloff characterization, which is computationally more expensive, but delivers more accurate estimation of the chief ray projection.
According to an embodiment, such a method further comprises:
The coarse estimation obtained through optical vignetting analysis with shape matching is interesting in that it is computationally inexpensive. However, its accuracy is limited to the radius r of a micro-image, or to √{square root over (2)}·r. It is thus important to refine this coarse estimate, starting with a set of refinement candidates, which may for example be chosen at a distance smaller than √{square root over (2)}·r, or smaller than 2r, to the coarse estimate.
For example, the refinement candidate set comprises pixels which distance to the coarse estimate is smaller than or equal to twice the radius of a micro-image.
Such a refinement step aims at determining a radial symmetry center for the illumination pattern of the raw image.
According to an embodiment, concentric circles forming the concentric rings are distant from each other from a distance r targeted to correspond to a radius of a micro-image.
The present disclosure also concerns a method for calibrating a light-field acquisition device, comprising a main lens, a sensor, and a micro-lens array placed between said main lens and said sensor, a micro-lens of said micro-lens array forming a micro-image on said sensor, which comprises:
estimating the position, on said sensor, of a projection of a chief ray, corresponding to an optical axis of said light-field acquisition device, as described previously;
determining positions of centers of said micro-images using said estimated position of the projection of said chief ray.
Such a calibration method may either deal directly with a coarse estimate of the chief ray projection, or with a refined estimate, resulting from illumination falloff characterization.
The present disclosure also concerns a computer program product downloadable from a communication network and/or recorded on a medium readable by a computer and/or executable by a processor, comprising program code instructions for implementing a method as described previously.
The present disclosure also concerns a non-transitory computer-readable medium comprising a computer program product recorded thereon and capable of being run by a processor, including program code instructions for implementing a method as described previously.
Such computer programs may be stored on a computer readable storage medium. A computer readable storage medium as used herein is considered a non-transitory storage medium given the inherent capability to store the information therein as well as the inherent capability to provide retrieval of the information therefrom. A computer readable storage medium can be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. It is to be appreciated that the following, while providing more specific examples of computer readable storage mediums to which the present principles can be applied, is merely an illustrative and not exhaustive listing as is readily appreciated by one of ordinary skill in the art: a portable computer diskette; a hard disk; a read-only memory (ROM); an erasable programmable read-only memory (EPROM or Flash memory); a portable compact disc read-only memory (CD-ROM); an optical storage device; a magnetic storage device; or any suitable combination of the foregoing.
The present disclosure also concerns a system for estimating the position, on a sensor of a light-field acquisition device, of a projection of a chief ray, corresponding to an optical axis of said light-field acquisition device, comprising:
According to an embodiment, the processor is further configured to refine the coarse estimate of the position of the chief ray projection by analyzing energy fall-off on the candidate area.
According to a further embodiment, the processor is further configured to:
More generally, all the assets and features described previously in relation to the method for estimating the position of the chief ray projection on the sensor of a light-field acquisition device also apply to the present system for estimating the position of the position of a chief ray.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not restrictive of the disclosure, as claimed.
It must also be understood that references in the specification to “one embodiment” or “an embodiment”, indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.
The present disclosure can be better understood with reference to the following description and drawings, given by way of example and not limiting the scope of protection, and in which:
The components in the figures are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the disclosure.
The general principle of the present disclosure relies on the analysis of the optical vignetting effect affecting a raw image formed on the sensor of a light-field acquisition device, in order to determine a coarse estimate of the position of the chief ray projection on this sensor. Such a coarse estimate may be advantageously used in further calibration methods of the light-field acquisition device. Alternately, analyzing the effect of natural vignetting on this raw image allows refining such a coarse estimate.
The present disclosure will be described more fully hereinafter with reference to the accompanying figures, in which embodiments of the disclosure are shown. This disclosure may, however, be embodied in many alternate forms and should not be construed as limited to the embodiments set forth herein. Accordingly, while the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that there is no intent to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the disclosure as defined by the claims. Like numbers refer to like elements throughout the description of the figures.
It will be understood that, although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element without departing from the teachings of the disclosure.
While not explicitly described, the present embodiments and variants may be employed in any combination or sub-combination.
It must be noted that, in the foregoing, the exemplary embodiments are described in relation to a plenoptic camera.
As schematically illustrated in
The raw image of the scene obtained as a result is the sum of all the micro-images acquired from respective portions of the photosensor. This raw image contains the angular information of the light-field. In fact, the angular information is given by the relative position of pixels in the micro-images with respect to the centre of these micro-images. Based on this raw image, the extraction of an image of the captured scene from a certain point of view, also called “de-multiplexing”, can be performed by concatenating the raw pixels covered by each micro-image. This process can also be seen as a data conversion from a 2D raw image into a 4D light-field.
It is hence very important to determine the positions of the centres of the micro-images, also called micro-centres. The positions of micro-centres are indeed required for computing focal stacks or matrices of views, i.e. for turning raw plenoptic data into workable imaging material.
However, shapes and intensities of the micro-images are affected by vignetting, which may be simply described as the shading of the image towards its margin. Vignetting has several causes, among which:
According to an embodiment of the disclosure, the shapes and intensities of the micro-images on sensor 30 are analyzed, in order to estimate the position of the chief ray within the raw picture of
The main lens of a camera is classically made up of several lenses, centered on a same axis, and surrounded on their outer periphery by rims. Given their overall circular shapes, the lenses placed one behind another and surrounded by rims form a cylinder 50, delimited by the front and rear elements of the main lens. Reference numeral 51 denotes the axis of cylinder 50. Reference numeral 52 designates the intersection of the cylinder axis 51 with sensor 30.
In the following, it is assumed that the center of symmetry of cat's eye shapes and the center of radial symmetry of illumination falloff both correspond to the same point 52 on sensor 30, i.e. the projection of the chief ray on the sensor.
At step 62, a candidate area, comprising potential positions of the chief ray projection is selected within raw image I 61. Such a candidate area may be the whole picture I 61, when it is desired to carry out the search for the position of the chief ray projection in the whole picture. Otherwise, for a lighter computational load, the candidate area may be a restricted central area within the raw picture I 61, for example covering 25% of the total area of the raw image.
The candidate area is denoted as Ω=1, W
×
1, H
. It is possible to choose W=H
where W and H respectively denote the width and the height of the candidate area, in pixels.
The process of
The candidate area Ω corresponds to a set of potential symmetry centers 71 in raw image I 61. Let (μ, ν)εΩ be pixel coordinates in the candidate area. At step 63, for each candidate pixel (μ0, ν0) corresponding to a possible symmetry center in the candidate area Ω, the following score is computed:
In other words, for each candidate pixel (μ0, ν0) in the candidate area Ω, we compute a cross-correlation of the raw image I 61 restricted to the candidate area Ω with a symmetrical image obtained by applying central symmetry with respect to the candidate pixel (μ0, ν0) to the raw image I 61 restricted to the candidate area Ω. Each cross-correlation computation 63i provides a cross-correlation score R (μ0, ν0) for the candidate pixel (μ0, ν0).
At step 64, the cross-correlation scores obtained for all pixels in the candidate area Ω are compared and the highest cross-correlation score is selected:
(x,y)=arg max{R(μ0,ν0)}
The pixel (x, y) associated to the highest cross-correlation score R(μ0, ν0) gives a coarse estimate 65 of the position of the chief ray projection on the sensor 30. Actually, the highest score indicates the most likely position of the chief ray projection with respect to optical vignetting, as illustrated by
Such a coarse estimate (x, y) 65 may be used in further calibration techniques, for example for determining the positions of the micro-centers of the micro-images 40 for plenoptic camera 100.
However, as illustrated by
According to an embodiment of the present disclosure described in relation to
The input image is the raw white plenoptic image I 61. Knowing the coarse estimate (x, y) 65 of the chief ray projection on sensor 30, we first determine at step 92 a set of pixels close to the coarse estimate (x, y) 65, called a refinement candidate set, which are candidates for this refinement step. For each candidate (μ0, ν0) in the refinement candidate set, natural vignetting is analyzed by energy fall-off measurement on concentric rings centered on the candidate (μ0, ν0) at step 93.
Let (ri)1≦i≦N denote a sequence of increasing radii, with e.g. rn=n·r, r>0 and rN≦min(W, H). Considering a potential candidate for radial symmetry center (μ0, ν0), we consider the rings bounded by two consecutive circles:
D
i={(μ,ν)εΩ:ri<(μ−μ0)2+(ν−ν0)2≦ri+1}
Raw plenoptic samples are summed within the ring in order to get:
where |·| denotes the cardinal number of a set, r denotes a radius in pixels, n, just like i, denotes an index in the sequel of rn, Dn, Sn, which starts from 1 and ends at N (note that it is pointless to consider rN>min(W, H)), Di denotes a ring bounded by two consecutive discs and Si is the mean light intensity averaged on every pixel of Di.
Because of natural illumination falloff due to natural vignetting, the sequence of (Si)1≦i≦N is monotonically decreasing if centered on the chief ray. Thus, degree of natural vignetting can be analyzed with a value of Si. Therefore, we consider the corresponding series, normalized by its first occurrence:
With respect to natural vignetting, the most likely position of the chief ray projection on sensor 30 corresponds to the lowest final score, which is selected at step 94:
(xR,yR)=arg min{TN(μ0,ν0)}
The position (xR, yR) associated to the lowest energy fall-off score gives a refined estimate 95 of the position of the chief ray projection on the sensor 30. Such a refined estimate of the position of the chief ray projection on sensor 30 may be used in further calibration methods, which are out of the scope of the present disclosure, and are not described here in further details.
It is important to carefully tune the sequence (ri)1≦i≦N of increasing radii. Choosing r of the magnitude of the radius of a micro-image 40 is relevant. However, other values may also be chosen.
The refinement process of
An apparatus 200 illustrated in
The processor 101 controls operations of the apparatus 200. The storage unit 102 stores at least one program to be executed by the processor 101, and various data, including data relating to the estimate position of the chief ray projection or to the selected candidate area, parameters used by computations performed by the processor 101, intermediate data of computations performed by the processor 101, and so on. The processor 101 may be formed by any known and suitable hardware, or software, or a combination of hardware and software. For example, the processor 101 may be formed by dedicated hardware such as a processing circuit, or by a programmable processing unit such as a CPU (Central Processing Unit) that executes a program stored in a memory thereof. As the cross-correlation computations of step 63 (
The storage unit 102 may be formed by any suitable storage or means capable of storing the program, data, or the like in a computer-readable manner Examples of the storage unit 102 include non-transitory computer-readable storage media such as semiconductor memory devices, and magnetic, optical, or magneto-optical recording media loaded into a read and write unit. The program causes the processor 101 to perform a process for estimating the position of the chief ray projection on the sensor of a light field acquisition device according to an embodiment of the present disclosure as described previously.
The input device 103 may be formed by a keyboard, a pointing device such as a mouse, or the like for use by the user to input commands. The output device 104 may be formed by a display device to display, for example, the calibration data of the light field acquisition device, including the coarse and refined estimate of the chief ray projection. The input device 103 and the output device 104 may be formed integrally by a touchscreen panel, for example. The input device 103 may be used by an operator for selecting, on the raw image, the candidate area, comprising potential positions of the chief ray projection. Such a candidate area may then be stored into storage unit 102.
The interface unit 105 provides an interface between the apparatus 200 and an external apparatus. The interface unit 105 may be communicable with the external apparatus via cable or wireless communication. In this embodiment, the external apparatus may be the plenoptic camera 100 and a uniform whit light source for lighting plenoptic camera 100. In this case, the raw white plenoptic image formed on the sensor 30 by the light source can be input from the plenoptic camera 100 to the apparatus 200 through the interface unit 105, then stored in the storage unit 102.
Although only one processor 101 is shown on
In a specific embodiment, such a processor 101 also comprises a refinement module for refining such a coarse estimate by illumination falloff analysis on the raw image. Such a refinement module comprises several parallel units for computing energy fall-off scores for a set of candidate pixels by:
Such a refinement module also comprises a module for determining, as a refined estimate of the position of the chief ray projection, the pixel in the refinement candidate set associated to the lowest energy fall-off score.
These modules and units may also be embodied in several processors 101 communicating and co-operating with each other.
The present disclosure thus provides a system and method allowing precise identification of the position of chief rays in the observed object space corresponding to individual pixels of a light field sensor. It thus provides a precise technique for calibrating light field data acquisition devices, and notably plenoptic cameras.
| Number | Date | Country | Kind |
|---|---|---|---|
| 15307067.7 | Dec 2015 | EP | regional |
