This application claims the benefit and priority of European Patent Application Serial No. 12306080.8, filed on Sep. 7, 2012, which is incorporated by reference herein.
The invention relates to the field of image stabilization in pico-projectors and more specifically in handheld devices equipped with projectors and additional sensors.
Handheld devices like smartphones are more and more used for the visualization of images or movies. Nevertheless the small size of their screens does not help to watch videos in good conditions. Projection features are being integrated into handheld devices to solve their issue of small screen size. That is why handheld devices like smartphones are more and more equipped with projectors. That way, any wall or surface can be used as a projection surface. This enables a large visual presentation from a very small projector.
Projectors embedded in handheld devices like smartphones are called pico-projectors. The issue of image stabilization does not arise from office projectors as those devices can rest on a stable platform or they can be fixed externally and do not move during a projection session. In the case of pico-projectors, they are sustained by instable object like a hand, so they will suffer vibrations and slight orientation and position changes during the projection, called “hand shake”.
The lack of stability in the projection of a sequence of images can produce several defects. Keystone is a bad orientation of the projector relative to the projection surface; inner rotation is a bad orientation of the projector relative to its own central beam; gap is a bad displacement of the projector relative to the projection area on the projection surface, zoom is a changing distance of the projector with the projection surface. Instability corresponds to a projection session where one or several projected images of the sequence will suffer from one or many of those defects. While the issue of image stabilization due to vibrations in image projection can be solved in office projectors by fixing physically the body of the projector, for Pico-projectors, image stabilization due to hand shake requires image processing techniques.
In the prior art, many correction techniques relate to keystone, but not stabilization. Some of the cited defects are not mentioned. Correcting only the image deformation for example is not sufficient to correct stabilization. Image stabilization requires more data than managing keystone or bad spatial orientation alone. For keystone, knowing the elevation and inclinations angles with the projection surface is necessary.
In some prior art where the image stabilization issue is addressed, there is needed a complete set of motion sensors to compensate for handshake. In a first prior art, for example the patent application US20120113514, a combination of gyroscopes and accelerometers on all three axes is used to compute a corrective signal input to feed a video controller to reduce the apparent motion of the image to be projected. In a second prior art, for example the patent U.S. Pat. No. 6,753,907, an un-calibrated camera is used to observe the projected image; and the image to be displayed is pre-warped so that the distortions induced by the misaligned projection system will be compensated. Generally speaking, techniques with heavy image processing, as they can't be applied in real time during a movie projection, are not image stabilization techniques.
How to manage image stabilization during projection in a real time process and with a low cost approach is still to question in handheld devices. Image stabilization systems are classified into two main categories. The optical or mechanical image stabilizer employs a prism assembly that moves opposite to the shaking of camera for stabilization. The optical image stabilizers are hardware dependent and require built-in devices such as servo motors making it voluminous and costly. The digital or electronic image stabilizer compensates for the image sequence by employing motion sensors to detect the device movement for compensation and performs image compensation through image processing algorithms.
An object of embodiments of the present invention is to alleviate at least partly the above mentioned drawbacks. More particularly, embodiments of the invention aim at improving image stabilization in pico-projectors during a projection session. It is taken advantage of the several devices and sensors handheld devices can be equipped with for image stabilization. An object of the invention is, in a projection session, to detect a movement and to compensate for it easily and at low cost.
The object of the present invention is achieved by a method for the projection of a sequence of images onto a display surface by a handheld device having an embedded projector, the method comprising, for an image of the sequence of images to be projected: a step of evaluation wherein a spatial position variation is evaluated with respect to a reference spatial position of the handheld device; and the spatial position variation is defined by a spatial orientation variation and a spatial vector variation; the spatial vector variation being determined by 3 independent coordinates; and at least one of the 3 independent coordinates is provided by a stereoscopic sensor embedded in the handheld device a step of compensation wherein a compensated image is generated from the image to be projected depending on the spatial position variation; and a step of projecting onto the display surface the compensated image.
Embodiments can comprise one or more of the following features:
The object of the present invention is also achieved with a data storage medium having recorded thereon a computer program comprising instructions for performing the method. The object of the present invention is also achieved by a handheld device having an embedded projector capable of projecting a sequence of images onto a display surface; said handheld device comprising an evaluation unit adapted to evaluate a spatial position variation with respect to a reference spatial position of said handheld device; said spatial position variation being defined by a spatial orientation variation and a spatial vector variation; said spatial vector variation being determined by 3 independent coordinates; and said handheld device comprising a compensation unit adapted to generate a compensated image from an image of the sequence of images to be projected, depending on said spatial position variation; and said handheld device comprising a stereoscopic sensor configured to deliver at least one of the 3 independent coordinates determining said spatial vector variation; and said embedded projector being adapted to project onto said display surface said compensated image.
Embodiments can comprise one or more of the following features:
Further features and advantages of embodiments of the invention will appear from the following description of some embodiments of the invention, given as non-limiting examples, with reference to the accompanying drawings listed hereunder.
a to
a is an image in a frame correctly projected;
b is an image in a frame badly projected because of elevation;
c is an image in a frame badly projected because of inclination;
d is an image in a frame badly projected because of inner rotation;
e is an image in a frame badly centered because of gap;
f is an image of a frame badly displayed because of a zoom issue;
When there is no image stabilization, the quality of the video can decrease.
a is an image of a frame correctly projected. In
b is an image of a frame badly projected because of an arisen elevation of the handheld projector. In
c is an image of a frame badly projected because of an arisen inclination of the handheld projector. In
d is an image of a frame badly projected because of an arisen inner rotation. In
e is an image of a frame badly centered because of an arisen translation gap. In
f is an image of a frame badly displayed because of a zoom issue. In
With an image stabilization system, the resulting image at the projection surface is targeting the projection area and is not misty owing to vibration. The goal is to place the projector display in a fixed target area on the projection surface, despite jitter and vibrations during the projection. At each time step the position of the projector is collected and the image to be projected is compensated so as to fulfill the requirement of good projection quality. Several kinds of defects in the projection of images onto the projection surface will need to be compensated. The first defect is the orientation of the pico-projector in respect of the projection surface and the second and third defects are the translation and the remoteness with the target point in the projection surface. The orientation includes elevation, inclination and inner rotation. The translation includes horizontal or vertical gap on the one hand, and depth with the target area on the other hand. Keystone correction needs an estimate of the orientation of the projector relative to the surface. Image stabilization consists in fixing the projection on the surface even under hand jitter. As image stabilization needs both orientation and position of the projector relative to the surface, keystone correction is the easiest case.
Digital image stabilizer can be chosen advantageously. It does not need any mechanical or optical devices and is suitable for handheld devices. Digital stabilization systems use completely electronic processing to control the image stability. It uses more software algorithms and less hardware components to compensate for the disturbances. This makes digital stabilization more portable and cost effective compared with other methods. In digital stabilization, spatial positions variations are obtained by taking consecutive two frames of the sequence and performing a series of operations over the frames. Because of exhaustive image processing operations, spatial position variation evaluation is the most time consuming and difficult part in digital stabilization.
Sensors data and vectors calculations can be used as following.
With the stereoscopic system, knowing what happened along the Z axis direction is possible. Thus a 2-axis accelerometer is enough for the remaining translation components. In practice, information from 3-axes gyroscopes, 2-axes accelerometers and stereoscopic sensors can be mixed together to enable the pico projector to know the spatial vector variation. A new position of the handheld device 4 corresponds to a translation and a rotation. The translation can be split into firstly the gap to the target of the central beam CB during the projection, and secondly the distance to the projection surface 3. Pico projectors are not supposed to be tied to target on the projection surface 3; it is one of their advantages and an easy to use feature. The distance to the projection surface 3 is compensable with an automatic zooming feature, capable of magnification when the distance is increasing and capable of reduction when the distance is lowering.
The projector can be enriched with an evaluation unit and a compensation unit as following. After spatial position evaluation, motion compensation part is responsible for correcting unintentional motions. It is about the alignment of the frames with respect to the estimated jitter through an inverse transformation process. In this part, same amount of movements are given to the frames in the inverse direction with the jitter in order to obtain stabilized video sequence.
An exemplary embodiment of the method according to the present invention for stabilizing image recordings adapted for projected data will consist firstly by calculating what action is required to keep the image stable from information by motion and proximity sensors and secondly by detecting a movement and compensating for it to get stabilization. The correction can be applied in real time, during the projection of a movie. The whole stabilization is split with a distinction between stabilization in translation and stabilization in rotation along 2 or 3 axes. Hand shaking can be slight flicker or slight vibrations; the amplitude of this movement is often limited to an offset.
The evaluation unit can work as following. Using
To obtain linear movements of the camera, acceleration data are converted into displacement data. Some smartphones are equipped with 3-way axis device which is used to determine the handheld device physical position. The accelerometer can tell when the handheld device is tilted, rotated, or moved. In practice the spatial vector variation is determined by 3 independent coordinates in a coordinate system whereas the spatial orientation variation is determined by 3 independent angles. For spatial bad orientation we need to get an additional rotation angle of the pico-projector around its own central beam axis. Image stabilization will require another triplet corresponding to the 3 independent coordinates of a spatial point in a Cartesian coordinate system representative of the reference mark.
The movement to compensate for can be described in a 6-axis information vector (x, y, z, Rx, Ry, Rz). For each movement, a specific correction will be attributed. For a translation movement towards the X axis, the correction is a Shift on X axis. For a translation movement towards the Y axis, the correction is a shift on Y axis. For a translation movement towards the Z axis, the correction can be a Zoom In or a Zoom Out. For a rotation movement around X axis or Pitch, the keystone correction is to be made by a rotation along vertical axis. For a rotation movement around Y axis or Roll, the keystone correction is to be made by a rotation along horizontal axis. For an inner rotation movement, there is a rotation correction needed along Z axis or Yaw. For a combination of movements, one correction is a combination of associated corrections. For an image of the sequence of images to be projected, a spatial position variation is evaluated with respect to a reference spatial position.
The spatial position variation consists in a spatial orientation variation and a spatial vector variation. Accordingly, the 3 independent angles are preferably provided by a 3 axes gyroscope embedded in the handheld device. The Z-coordinate is provided by a stereoscopic sensor. The X and Y coordinates are provided by a 2-axes accelerometer. Conventional stereo vision is usually achieved with two cameras that are mounted in a known relationship to each other and are synchronized to take images at the same instant. Contrary to a one camera system, stereoscopic system brings the depth information. To measure depth a stereoscopic camera consists of two cameras that capture two different, horizontally shifted perspective viewpoints. This results in a disparity of objects in the recorded scene between the two cameras views depending on their depth. The depth and disparity are related in relationships using the focal length of the cameras and the inter-axial separation between the two lenses of the stereoscopic camera.
There are commonly two possible ways of taking stereoscopic pictures: by using special two-lens stereo cameras which is an optical system with two lens but only one camera or by using systems with two single-lens cameras (which are two separate cameras joined together). Stereoscopic pictures allow us to calculate the distance from the cameras to the chosen object within the picture. The distance is calculated from differences between the pictures and additional technical data like focal length and distance between the cameras.
Practically, the evaluation can be performed through an evaluation unit 10 adapted to evaluate a spatial position variation with respect to a reference spatial position of the handheld device; the spatial position variation being defined by a spatial orientation variation and a spatial vector variation.
The compensation can be performed in several possible implementations. For example one possible implementation is the one that always uses two consecutive frames from the input image sequence to estimate the spatial position variation, which is referred to as frame-to-frame algorithm. In that case, the reference spatial position will be variable between two images of the sequence of images to be projected. Another possible implementation is the one that keeps a reference image and uses it to estimate the variation between the reference and the current input image, which is referred to as the frame-to-reference algorithm. In that case, the reference spatial position will be constant for all images of the sequence of images to be projected. Any combination of those two implementations can of course be used, for example the refreshing of the reference spatial position every n images, with n a constant number. Anyway, in our case, a compensated image is generated from each image to be projected depending on the spatial position variation. At the end, what is projected onto the projection surface is the compensated image instead of the original image.
Any of inverse transformations methods well known can be used to generate a compensated image resulting from the input image to be projected and from sensors data about the spatial position variation. For example graphics processing unit implementations or digital image processing techniques. Generally speaking, for GPU implementations, the spatial transformation can develop a triangle mesh with one triangle defined by 3 vertices or points. The transformation is executed by texture mapping from the rectilinear mesh of the input image to the transformed shape of the destination image. In digital image processing implementation, the spatial transformation consists of spatially defined 2-dimensional image re-sampling or scaling filter. The scaling operation is performed with different scaling ratios in different parts of the image, according to the defined transformation. Practically, the compensation can be performed through a compensation unit 11 adapted to generate a compensated image from an image of the sequence of images to be projected, according to the spatial position variation.
Using
An exemplary embodiment of the handheld device according to the present invention is as following.
The sensors according to the present invention can be connected to the handheld device on chip or they can be externally connected to it, serving as possible extensions of the handheld device. In either case the extension will be considered to be embedded to the handheld device. Presently, pico-projectors or stereoscopic sensors considered here are said to be embedded in the handheld device.
The intelligence of the system can be implemented in a computer readable medium. The handheld device may comprise a computer readable medium such that computers programs are loadable into data-processing units and capable of executing embodiments of the present invention.
The invention has been described with reference to preferred embodiments. However, many variations are possible within the scope of the invention.
Number | Date | Country | Kind |
---|---|---|---|
12306080.8 | Sep 2012 | EP | regional |