Patent Application
20140210794
The present invention generally relates to correction of touch screen camera geometry. The invention relates particularly, though not exclusively, to automatic geometry calibration of an infrared-based multitouch display.
In a camera-based multitouch display, the cameras are not always placed exactly as designed. This is caused e.g. by finite mechanical tolerances that cause some variation in the physical setup of multitouch display systems. Hence, for optimal performance, it is necessary to calibrate each display separately, for each of its cameras. The parameters to calibrate involve:
camera sensor region of interest (ROI), i.e. the area of the camera sensor that is used for touch tracking;
camera lens distortion parameters (such as barrel distortion);
focus point for performing anti-distortion calculations; and
offset of the image from desired target location.
a shows various errors in images of 16 cameras. In
The camera ROI placement can be adjusted per-camera in order to account for the physical differences between different cameras. Also camera distortion parameters can be optimized per camera, as illustrated by in
Recognition of pointing objects from an infra-red based multitouch display depends on the quality of the operation of the cameras. Camera errors increase difficulty of accurately recognizing boundaries objects that point the touch screen and of determining when an object actually touches the display.
According to a first example aspect of the invention there is provided a method comprising:
the method further comprising:
The method may further comprise detecting a start command and responsively starting the producing of the image, the determining of the geometric distortions and the calculating of the calibration parameters.
The group of infrared cameras may be configured to take images of the calibration patterns through the touch screen.
The forming of the one or more calibration patterns onto the touch screen may comprise producing the one or more calibration patterns with a calibration device operable to produce the one or more calibration patterns when placed on the touch screen.
The forming of the one or more calibration patterns onto the touch screen may comprise producing corresponding visible light patters by the touch screen. The visible light patters may be configured to form an infrared light component of a corresponding shape and/or the infrared cameras may be configured to operate also on visible light wavelengths.
Surprisingly, typical commercially available infrared cameras and flat display panels have sufficiently overlapping ranges of operating wavelengths so that the display can be used to produce calibration patterns for the one or more infrared image capture units. In addition to flat panel displays, video projectors can also be used for image generation, with similar overlapping visible/infrared wavelengths. It is appreciated that in case of projector based displays, the calibration geometric distortions may also account for possible geometric errors in the projector image.
The forming of the one or more calibration patterns onto the touch screen by producing corresponding visible light patters by the touch screen may enable inherently matching the calibration patters with the touch screen and thereby avoiding possible alignment errors that may occur when using a separate calibration device.
The forming of the one or more calibration patterns onto the touch screen by producing corresponding visible light patters by the touch screen may be emphasized by using a reflective cover that is arbitrarily placed on the touch screen to reflect light emitted by the touch screen to the group of infrared cameras.
The cover may have a fluorescent surface configured to absorb visible light and responsively reflect infrared light.
When forming the one or more calibration patterns onto the touch screen by producing corresponding visible light patters by the touch screen, the resulting image of the touch screen may be too noisy for determining of the geometric distortions. Hence, the method may further comprise taking with each of the group of infrared cameras a set of plural images to improve signal to noise ratio.
The method may further comprise optimizing the region of interest of each camera. The optimizing of the region of interest may involve the steps of:
The taking of the plural images may involve displaying visible images on the screen and:
The method may further comprise normalizing the images of the calibration pattern according to the formula of NPe=(PPe−PBe)/(PWe−PBe), wherein NPe represents the normalized image pixel value.
The method may further comprise reducing noise from the normalized images of the calibration pattern by noise reduction filtering.
The method may further comprise detecting edges of the calibration patterns. The curvature may be calculated for the detected edges of the calibration patterns. The calculating of the curvature may be performed line segment by line segment.
The method may further comprise calculating of curvatures of the detected edges. Average of the curvatures, AC, may be calculated. The method may further comprise calculating mutual angles of the edges. The method may further comprise calculating an average angle of the edges. An average angle deviation AA may be calculated from the calculated angles of the edges or from the average angle of the edges.
A combined error measure may be calculated by summing the AC and AA.
The method may further comprise calculating for each edge the deviation from the nearest multiple of 90 degree angle.
The method may further comprise calculating the average deviation AD from the nearest 90 degree angle.
A combined error measure may calculated by summing the AC, AA and AD.
The method may further comprise calculating error functions such that when the geometric errors in the image of the touch screen is compensated using the error functions, the combined error measure is smaller than with at least one other set of error functions.
The error functions may measure one or more of the following items:
According to a second example aspect of the invention there is provided an apparatus comprising:
The means for forming of the one or more calibration patterns may comprise a calibration screen. The calibration screen may comprise a passive element configured to cast patterns produced by the touch screen to the group of infrared cameras. The passive element may comprise a sheet or plate configured, when placed on the touch screen, to reflect patterns produced by the touch screen to the group of infrared cameras.
Advantageously, a passive element may produce calibration patterns independent of the alignment of the passive element on the surface of the touch screen and thereby remove the need to accurately align the calibration screen with relation to the touch screen.
The calibration screen may comprise an active element configured produce a calibration pattern for the group of infrared cameras. The active element may comprise one or more infrared illumination units. The infrared illumination units may comprise infrared light emitting diodes. Alternatively or additionally, the infrared illumination units may comprise windows in a sheet or plate, which windows are at least partly transparent for infrared light so as to pass ambient infrared light through the windows to the group of infrared cameras.
The calibration screen may comprise one or more fluorescent patterns configured to absorb visible light and responsively reflect infrared light so as to produce infrared calibration patterns when illuminated by the touch screen.
The apparatus may further comprise means for implementing a method according to any embodiment of the first aspect.
According to a third example aspect of the invention there is provided a computer program comprising computer executable program code which when executed by a processor causes an apparatus to perform the method of the first aspect.
The processor may be formed of one or more discrete circuitries.
According to a fourth example aspect of the invention there is provided a computer readable memory medium comprising the computer program code of the third aspect. The memory medium may be non-transient. The memory medium may be formed of one or more discrete circuitries.
According to a fifth aspect of the invention there is provided an apparatus comprising:
The calibration screen may be reflective in wavelengths on which the one or more infrared image capture units operate.
The apparatus may further comprise a display configured to display images on the display surface under control of the processor.
The display and the one or more infrared image capture units may have overlapping ranges of operating wavelengths for enabling the display to produce calibration patterns for the one or more infrared image capture units.
The processor may further be configured to show different visible patterns on the display for producing different calibration patterns and detecting how the one or more infrared image capture units respond to different patterns.
The processor may further be configured to perform a calibration process according to any method of the first aspect.
The calibration screen may have a fixed pattern suited for providing a reference for the calibrating. The fixed pattern may be printed.
The calibration screen may contain a pattern that appears in infrared light.
The cover may be configured to produce a pattern with light emitting diodes. The light emitting diodes may be infrared light emitting diodes.
The calibration screen may have windows configured to produce a pattern when the calibration screen is illuminated from behind with infrared light.
The calibration screen may be placed directly in contact with the display, or placed further away.
The calibration screen may be a cover sheet.
Any foregoing memory medium may comprise a digital data storage such as a data disc or diskette, optical storage, magnetic storage, holographic storage, opto-magnetic storage, phase-change memory, resistive random access memory, magnetic random access memory, solid-electrolyte memory, ferroelectric random access memory, organic memory or polymer memory. The memory medium may be formed into a device without other substantial functions than storing memory or it may be formed as part of a device with other functions, including but not limited to a memory of a computer, a chip set, and a sub assembly of an electronic device.
Different non-binding exemplary aspects and embodiments of the present invention have been illustrated in the foregoing. The above embodiments are used merely to explain selected aspects or steps that may be utilized in implementations of the present invention. Some embodiments may be presented only with reference to certain aspects of the invention. It should be appreciated that corresponding embodiments may apply to other aspects as well.
Some example embodiments of the invention will be described with reference to the accompanying drawings, in which:
a to 10d show captured and differently processed patterns.
In the following description, like numbers denote like elements.
In the following examples of various embodiments of the invention an LCD screen is used as an example of an image layer. The LCD screen may comprise a film that forms a visible image and optionally also other elements such as background illumination, infrared (IR) illumination, incoming IR intensity detection across the screen, and/or one or more speakers for outputting sound. It is appreciated that the invention may likewise be applied with any other types of image layers as well, or screens for simpler expression. Generally, an image layer is any means for producing a visible image for a user, such as an electric display, a screen displaying a projected image or a substance carrying printed, dyed image, or woven image. However, the image layer should be sufficiently transparent or translucent for IR light that is used for detecting a pointing object through the image layer as will be explained in more detail in the following.
The system 400 comprises as an outmost element or as facing to the user 413 a touching surface 401 such as a toughened glass plate, then an LCD film 402 as a display screen and a diffusion layer 403 behind the screen.
The diffusion layer 403 is configured to spread the light coming from a background light source (described in the following), so that an image displayed on the screen appears even from many directions.
In order to support the aforementioned optical elements, there is provided a casing 404. The casing 404 comprises a back wall 405 attached to side walls 406. A background light source 408 may be located in the casing 404 for background illumination of the screen 402.
It should be appreciated that the structure presented in
The side walls 406 may be coated from the inner side with some reflective material in order to deliver maximum amount of light to the screen and finally to users of the system.
Inside the casing, there is an image capture unit 407 that is in this particular case has an array of IR cameras configured to see through the diffusion layer 403, screen 402 and the touching surface 401 any objects placed near or at the touching surface 401. Further in
The distance between the background light source 408 and the diffusion layer 403 may generally depend on the space that the camera setup requires. The distance may be shortened for more compact sized system e.g. by moving the camera forward and/or turning the camera around (to point away from the screen) and using a mirror or prism in front of the camera to capture the activities on the screen.
The camera 407 and the IR LEDs 410 may not be seen from the outside of the system if their shadow is negligent e.g. due to the highly diffused illumination coming from the background lights and as the reflective inner side of the system provides evens further illumination of the display surface. Further, the diffusion layer 403 may prevent the users from seeing any details from inside the system.
The system 400 may comprise invisible light sources 410 such as IR light sources for facilitating input detection on IR wavelengths. Additionally or alternatively, the system 400 may comprise an audio speaker 414 for providing audible signals to the user 413.
Furthermore, the system 400 may comprise one or more buttons 409 for receiving user input. For instance, there may be a dedicated calibration initiation button 409 so located that administrative personnel only can reach the button.
In one embodiment, there are 8 columns×4 rows of cameras 407 and invisible light sources each covering a different sub-region of the touching surface 401. The different sub-regions may have a some overlap (e.g. 2, 5 or 10%) or the different sub-regions may be non-overlapped.
In
The control unit 411 further comprises a processor 507 configured to control the operation of the control unit 411 according to the software 504 by executing computer executable program code contained by the software in the work memory 502. Alternatively, the control unit may be configured to execute the software in place in the non-volatile memory in which case the work memory may not be necessary. The control unit further comprises an input/output unit (I/O) 508 for exchanging signals with other elements of the system 400 and optionally also with external equipment. The I/O 508 may comprise e.g. any one or more of a universal serial bus port, a local area network port, an ISA bus, a PCI express bus, an IR port, a Bluetooth element, and a parallel port. Alternatively to being configured capable of communicating with external equipment, the system 400 may be provided with a transferable memory reception unit 509 such as a CD-ROM or DVD-ROM drive, memory card reader or memory stick reader which enables replacing part of the non-volatile memory e.g. for updating information to be displayed on the LCD screen 402.
In order to control the operation of various components of the system and to obtain the captured image, there are connections between the control unit or particularly its input/output unit 508 and other components of the system 400, while not shown in sake of clarity of the drawing. The control unit has generally the task of receiving a signal from the camera 407, detecting if and where the touching surface 401 is pointed at and typically also outputting the determination in a standard way e.g. emulating a computer drawing tablet, mouse or other known pointing device.
Generally, the control unit operation may comprise following acts:
It is appreciated that while the control unit may consist of one separate unit, the control unit 411 may alternatively be integrated with any other element. The control unit 411 may comprise two or more discrete elements each for one or more of the aforementioned acts.
Unfortunately, the patterns produced by the screen 402 may be barely visible in the images captured by the cameras. However, the inventor has noticed that these barely visible images, that are not nearly good enough for normal calibration of cameras, can be digitally enhanced 860 before the calibration 870 to make the calibration images sufficiently visible. The digital enhancing 870 may involve for instance the following sub-steps:
1. Record n camera pictures, while the screen is completely black;
2. Record n camera pictures while the screen is completely white;
3. Record n camera pictures while a given calibration pattern is on the screen
4. Repeat step 3 for each calibration pattern.
n may be selected from a range 64 to 256. In alternative embodiments, n may be given lower or higher values based on an optimization wherein the calibration images become sufficiently clear while the time required for the calibration is not excessive. For instance, n may be empirically selected such that the calibration images are recognized in the images at least for 95, 99, or 99.5% of the systems 400. It is also appreciated that the calibration method may be performed differently in a manufacturing site and at customer premises.
Once we have this record of images we can easily create normalized versions of the patterns, assuming that,
To obtain normalized version of PP, we perform the following arithmetic operation on the images (per pixel) illustrated by equation (1):
NP=(PP−PB)/(PW−PB) (1)
The resulting image may still be too noisy. Therefore, as phase 5, the noise can be reduced with well-known noise reduction algorithms, for example median filtering or low-pass filtering. After noise reduction, the images produced by the cameras may be enhanced that far that the calibration images shown in
The process of
In one alternative embodiment, no calibration sheet is used at all. Instead of forming the calibration pattern by radiating or reflecting infrared light with the calibration sheet, the calibration pattern is formed by forming with the display and used to filter ambient light so that the calibration pattern is seen by the infrared cameras. For instance, LCD screens absorb infrared light selectively depending on each display pixel's current state and thus LCD screens can also produce the calibration pattern even without any sheet placed on the display. The formed calibration pattern may have less contrast than the calibration patterns that are produced using e.g. a reflective sheet on the display and internal infrared light sources that cast light through the LCD screen and also back (to the extent that the infrared light reflects from the reflective sheet). However, the disclosed methods for enhancing the detection of the calibration patterns are effective also in conjunction with this embodiment.
Let us next consider various calibration processes that aid automatic calibration of the system 400, with reference to captured and differently processed patterns shown in
b shows a somewhat corrected camera image, based on source image (
c shows a camera image with edge lines as detected with an edge-walk algorithm, based on the image of
d shows a fully geometry corrected output, after the automatic calibration process.
The automatic calibration process involves creating error functions that represent different problems with the image. For example image 10b shows two kinds of error:
In this case we can construct an error function by performing the following process:
1. Detect the edges of the stripes in
2. Calculate the curvature of each line segment (there are 7 lines in this case rather than 8, as the lines too close to the edge of the image may be ignored, since they may disappear from the image area, or contain anomalies related to the edge of the image.
3. Calculate the average of the curvatures (AC)
4. Calculate the angles of the lines
5. Calculate the average angle of the lines
6. Calculate the average deviation from the average angle (AA)
7. For each line, calculate the deviation from the nearest multiple of 90 degree angle.
8. Calculate the average deviation from the nearest 90 degree angle (AD)
Before averaging the curvatures or angle differences the line segments may be remapped, to guide the optimization process. For example the error weight of a single line curvature may be calculated using a non-linear formula. By summing AC, AA, and AD we get an error function representing a combined effect of these three types of error. It is appreciated that additional error functions may be developed to counter specific types of error. We can then use any kind of numeric optimization method to minimize the error function. For instance, heuristics, simulated annealing, gradient descend, genetic algorithms etc. are methods suited for this purpose. We can further use a combination of different methods to improve the robustness and the speed of the optimization process. The optimization can be carried out in multiple passes, with each pass optimizing a subset of all parameters, with a different or similar algorithm, and different or similar error function.
Since the external device is a simple sheet of white material, the external device is very easy to produce, distribute and replace. Further, since the calibration patterns are directly on the screen, it is certain that the calibration pattern is exactly aligned with the screen and there is no offset due to misplacement of the calibration sheet.
Various technical effects of different embodiments of invention have been presented directly or implied by the preceding description. Some of these are summarized here:
The ROI of each camera may be optimized before proceeding with the calibration process disclosed with reference to
Various embodiments have been presented. It should be appreciated that in this document, words comprise, include and contain are each used as open-ended expressions with no intended exclusivity.
The foregoing description has provided by way of non-limiting examples of particular implementations and embodiments of the invention a full and informative description of the best mode presently contemplated by the inventors for carrying out the invention. It is however clear to a person skilled in the art that the invention is not restricted to details of the embodiments presented above, but that it can be implemented in other embodiments using equivalent means or in different combinations of embodiments without deviating from the characteristics of the invention.
Furthermore, some of the features of the above-disclosed embodiments of this invention may be used to advantage without the corresponding use of other features. As such, the foregoing description shall be considered as merely illustrative of the principles of the present invention, and not in limitation thereof. Hence, the scope of the invention is only restricted by the appended patent claims.
| Filing Document | Filing Date | Country | Kind | 371c Date |
|---|---|---|---|---|
| PCT/FI2011/050660 | 7/18/2011 | WO | 00 | 3/17/2014 |
