The present application relates to a method for calibrating a 3D camera, an apparatus for capturing 3D images, and a computer-readable medium.
A 3D camera is able to capture a three dimensional (3D) scene such that a 3D representation of the scene can be displayed at an appropriate display apparatus. A 3D camera may comprise, for example, a stereo camera or a texture plus depth camera.
When using a 3D camera one needs initially to calibrate it. This means that one needs to estimate the camera parameters (i.e. its extrinsic and intrinsic parameters) in order to carry out the registration of the two images obtained.
These could be either two textures or the combination of one texture and one depth map. Image registration is required for the correct alignment of the two images, this is necessary to allow a correct 3D experience to be delivered at a 3D display apparatus.
The camera parameter estimation is typically performed with the help of a calibration panel 100, which is illustrated in
A particular 3D camera will have a particular calibration panel associated with it. This is necessary so that the physical dimensions of the panel are known to the camera. Using a similar patterned panel with a different scale would give an inaccurate calibration of the camera. When the camera images its own calibration panel, it can derive actual physical distances from the image. Based on these real distances, the camera parameters are estimated.
Alternatively, a 3D camera can be calibrated automatically from the information in the images without a calibration panel. This is done by matching points that represent the same scene location from the two images and then estimating the camera parameters. Unfortunately, this method, although convenient, is limited as no absolute values of the camera are determined. Indeed, the scene distances in reality are unknown so there is not sufficient information for full calibration. In other words, a fundamental problem with automatic parameter estimation is the lack of scale information from the scene, preventing the estimation of absolute camera parameter values. Further, camera calibration without a calibration panel is far more computationally demanding than calibration with a calibration panel.
A problem with the calibration panel solution is that the calibration panel is not always available. In general, this panel is big (typically A4 or A3 size) and one does not carry it around. Also, the calibration software is adapted to a unique size of the panel, which means that other calibration panels having similar patterns but with different measurements can not be used instead.
There is provided herein a solution to the problem of camera calibration when there is a lack of prior information available, that is, when an appropriate calibration panel is not available.
The methods described herein are aimed at devices that incorporate touch surfaces. Such devices may use touchable screens, as contemporary tablets and smartphones do. A device can obtain a good estimate of the size and the form of a body part such as a user's hand by detecting it when applied to the touch surface. Once this is done, the hand can be used as a calibration object by detecting salient features (e.g. the fingers tips), which are then used to estimate the camera parameters. The calibration can be improved by using a pose estimation algorithm to determine an arrangement of the user's hand from the image(s).
Accordingly, there is provided a method for calibrating a 3D camera. The method comprises detecting measurements of a user's body part on a touch surface, and capturing at least one image of the user's body part with the 3D camera. The method further comprises applying the detected measurements to the at least one image to estimate parameters of the 3D camera.
Using the above method, no specific calibration panel is required and yet absolute camera parameter values can be determined. As a consequence, camera calibration becomes an easy task for an average user, and can be performed without specialist equipment.
The application of the detected measurements to the at least one image may comprise applying the measurements to a model template to generate a 3D model of the user's body part, and fitting the 3D model to the at least one image. The fitting may be done using a pose estimation algorithm, or a segmentation algorithm.
The parameters of the 3D camera may be intrinsic and/or extrinsic parameters. The extrinsic parameters (sometimes called external parameters) may comprise at least one of: orientation, rotation, location, or translation of the camera. The intrinsic parameters (sometimes called internal parameters) may comprise at least one of: pixel size, pixel skew, or focal length.
The estimation of parameters of the 3D camera may comprise: estimating a camera projection matrix; and recovering intrinsic and extrinsic parameters from the projection matrix. The method may further comprise refining the recovered intrinsic and extrinsic parameters using nonlinear optimization.
The body part may be a hand. The body part may be bigger than the touch surface, and the detecting measurements of a user's body part may comprise multiple detections each of different sections of the body part. The method may further comprise obtaining a measurement of the complete body part from a composite of the multiple detections.
There is further provided an apparatus for capturing 3D images, the apparatus comprising a touch surface, a 3D camera and an image processor. The touch surface is for detecting measurements of a user's body part. The 3D camera is arranged to capture at least one image of the user's body part. The image processor is arranged to apply the detected measurements to the at least one image and to estimate parameters of the 3D camera, the parameters for calibrating the 3D camera.
Using the above apparatus, no specific calibration panel is required and yet absolute camera parameter values can be determined. As a consequence, camera calibration becomes an easy task for an average user, and can be performed without specialist equipment. The 3D camera may be arranged to capture a plurality of images of the user's body part.
The image processor may be further arranged to apply the measurements of a user's body part to a template body part model to generate a 3D model of the user's body part; and to fit the 3D model to the at least one image of the user's body part. The fitting may be done using a pose estimation algorithm.
The parameters of the 3D camera may be intrinsic and/or extrinsic parameters. The extrinsic parameters (sometimes called external parameters) may comprise at least one of: orientation, rotation, location, or translation of the camera. The intrinsic parameters (sometimes called internal parameters) of the camera comprise at least one of: pixel size, pixel skew, or focal length.
The estimation of parameters of the 3D camera may comprise: estimating the camera projection matrix, and recovering intrinsic and extrinsic parameters from the projection matrix.
The touch surface may be a touch screen. The apparatus may be a tablet computer or smartphone. The touch surface may be a handle or part of a casing of the apparatus. The body part may be a hand.
There is further provided a computer-readable medium, carrying instructions, which, when executed by computer logic, causes said computer logic to carry out any of the methods defined herein.
There is further provided a computer-readable storage medium, storing instructions, which, when executed by computer logic, causes said computer logic to carry out any of the methods defined herein. The computer program product may be in the form of a non-volatile memory or volatile memory, e.g. an EEPROM (Electrically Erasable Programmable Read-only Memory), a flash memory, a disk drive or a RAM (Random-access memory).
A method and apparatus for 3D camera calibration will now be described, by way of example only, with reference to the accompanying drawings, in which:
In the examples herein we consider the user's hand as the body part used for 3D camera calibration, but it should be noted that any other object or parts of the body could be used provided that it can be sensed by the touch sensitive surface.
The user is initially requested to place her/his hand over the touch sensitive surface. The touch sensitive surface may be a touch screen or a touch sensitive surface incorporated into a carrying handle of the device. The device determines the area of the surface that is covered by the hand. Since the physical size of the surface is fixed, and known to the device, the size of the hand can be determined. The measurements taken from the touch surface may comprise the length of the fingers, the width of the palm, etc. These measurements are applied to a generic human hand model to generate a 3D model of the user's hand. In embodiments where another part of the user's body is used for camera calibration, a generic model of that part of a human body is used. The user may be given a choice as to which body part to use for camera calibration. At least one generic model of a body part is stored in the device.
Then, the user is requested to show the hand in front of the 3D camera with different distances to the camera and different hand poses. The arrangement and pose of the hand can be determined from a camera view using a segmentation algorithm, or pose estimation algorithm.
Of course, the hand may show a different pose to the one detected by the touch sensitive surface. Even if the user intends to show the same pose, the pose may not be the same shape as when pressed against the touch sensitive surface, for example the fingers may not have the same separation. As a result, the size of the hand in the picture is calculated and its pose is also estimated using the 3D model of the user's hand. With this information, the distances in the picture are deduced.
The salient features of the hand (e.g. finger tips) are detected in the camera image and the camera parameters are estimated with the deduced information.
The model comprises a plurality of joints 510 to 520, connected by segments. This is a 3D model, meaning that the segments may be moved around the joints in a 3D space. To create such a model of the user's hand, the length of the segments is determined from the measurements taken by the touch sensitive surface.
This model of the user's hand is then fit to an image from the 3D camera. In computer vision, there are methods and techniques to fit deformable models with a 2D image of a 3D scene. One example is “Human body pose estimation with particle swarm optimisation” by Ivekovic S, Trucco E, Petillot Y R, and published in Evolutionary Computation, 2008 Winter; 16(4):509-28, the human body pose estimation method described in this paper is incorporated herein by reference.
Once the model is fitted to the image, one knows the location of each joint of the user's hand in each calibration image (in as many images are needed to perform the calibration). Where the 3D camera comprises two 2D cameras, simultaneous images from each camera are taken of the hand in a pose, and the same pose is fit to each view, but rotated to allow for the slightly different perspective of each camera. With this information, the required camera parameters can be determined.
With reference to
The touch sensitive surface 710 is arranged to detect measurements of a user's body part, such as their hand. The 3D camera 720 is arranged to capture an image of the user's body part. The image processor 730 is arranged to apply the detected measurements to the image of the user's body part and to estimate parameters of the 3D camera, the parameters for calibrating the 3D camera.
The image processor 730 is arranged to receive instructions which, when executed, cause the processor 730 to carry out the above described method. The instructions may be stored on the memory 735.
An advantage of this solution is that no specific calibration panel is required and still absolute camera parameter values can be estimated. Likewise, different calibration panels may be used. As a consequence, camera calibration becomes an easy task for an average user, and can be performed without specialist equipment, and moreover with an item that the majority of users always have two of available to them.
The solution described herein is particularly applicable to tablet computers having 3D cameras built in. But this solution could be used with a touch surface smaller than the user's hand if multiple overlapping impressions of a user's hand are taken to allow the device to piece together a composite impression of the user's hand.
It may also be noted that such devices are typically calibrated upon manufacture. However, camera sensors may move (e.g. during transportation), and sooner or later a new calibration is required. Only a minor disturbance to the original setup is required to result in a 3D effect which is uncomfortable to a viewer of the captured 3D scene. As such, having a easily performed calibration process could be very useful.
It will be apparent to the skilled person that the exact order and content of the actions carried out in the method described herein may be altered according to the requirements of a particular set of execution parameters. Accordingly, the order in which actions are described and/or claimed is not to be construed as a strict limitation on order in which actions are to be performed.
It should be noted that the above-mentioned embodiments illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. The word “comprising” does not exclude the presence of elements or steps other than those listed in a claim, “a” or “an” does not exclude a plurality, and a single processor or other unit may fulfill the functions of several units recited in the claims. Any reference signs in the claims shall not be construed so as to limit their scope.
Filing Document | Filing Date | Country | Kind |
---|---|---|---|
PCT/EP2012/069353 | 10/1/2012 | WO | 00 |