The functionality of many computing systems and other devices relies on effective display of information using a display. More recently, the display has also been used in an interactive manner as a direct input device. For instance, a display might be equipped with touch sense resistive and/or capacitive arrays to detect a portion of the display being contacted.
Some prior interactive displays use “vision capture” technology in which a camera is positioned behind the display, the display consisting of one or more layers of transparent or semitransparent material. An infrared illuminator is also positioned behind the display to illuminate an object in front of or in contact with the display. Illumination light (i.e., light originating from the illuminator) reflected from the object is received back into the camera, which takes a picture of the reflected light. The picture is taken as electronic input to the system. Since the placement of the object in front of the display influences its image taken by the camera, the object may be used to input information into the system.
Some of the illumination light is reflected not off the object, but off of the relatively flat surfaces of the transparent or semi-transparent layer(s) forming the display. The result is that the camera will receive a specular reflection of relatively strong intensity at a specific area of the display. The specular reflection may be so strong that it would be difficult to distinguish any image actually reflected from an input object within the area of the specular reflection. The specular reflection may even saturate the camera in that specific area. The effect is somewhat analogous to the situation in which a person looks downward at a shallow pond on a sunny day. The person might be able to view the bottom of the pond except for the area that is at or close to the sun's blinding reflection.
Accordingly, specular reflections can adversely impact the ability to use an interactive display as input, especially if the input object is positioned at an area in which the camera is experiencing a specular reflection.
Although not required, embodiments of the present invention relate to an interactive display in which alternating illuminators are used to reduce or cancel specular reflections. The interactive display includes multiple illuminators and at least one camera placed behind one or more display layers. The interactive display uses a phase management mechanism that controls illumination phases of the interactive display. During the first phase, the first illuminator is controlled to dominate over the second illuminator. For example, the first illuminator may be turned on and the second illuminator may be turned off. In that state, the camera is caused to take an image. During the second phase, the second illuminator is controlled to dominate over the first illuminator.
A consolidated image is then formulated using a combination of the first and second images. The consolidated image may be formed such that specular reflections from the first image are deemphasized or even cancelled using portions of the second image, or vice versa. If the light sources are positioned such that their specular reflections are not overlapping, then the specular reflections effect may be mitigated or even eliminated.
This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
The appended drawings are used in order to more particularly describe embodiments of the present invention. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered to be limiting of its scope, the embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
Embodiments of the present invention extend to an interactive display that reduces specular reflections by alternating illumination phases.
The illuminator 802 emits light in many directions. Some of the light is emitted from the display, but does not interfere with the display operation since it is of a different frequency spectrum than the displayed image. Other light reflects from objects placed on the front surface (i.e., on or over the upper surface in
For example, to illustrate the interference of spectral reflection, three rays of light 821, 822 and 823 are illustrated as being emitted from the illuminator 802. Rays 821 and 822 are reflected from objects in front of the display 810, and thus represent valid input information. Ray 823, on the other hand, reflects off of a flat surface of the display layers 810, and thus represents spectral reflection. Spectral reflection can thus be observed by the camera 801 at approximately the region of object A. Accordingly, object A may not be observable or may be difficult to observe by the camera 801. Object B, on the other hand, is not close to the area of spectral reflection, and thus may be observed without concern for spectral reflection. The interactive display 800 may thus not work properly if the input object is placed on certain regions of the display corresponding to spectral reflection.
Similarly, when the second illuminator 103 is on and the first illuminator 102 is off, a specular reflection will be experienced, but in a different region of the image perceived from the camera 101. After all, the illuminators 102 and 103 are positioned at different positions behind the display layer 10. While illuminator 103 emits many rays of light, only three rays of light 121, 122 and 123 are shown. Ray 121 reflects off of object A and is received into the camera 101. Ray 122 reflects off of object B and is received into the camera 101. Ray 123 reflects off of the display layer 10, thereby representing specular reflection, and is received into camera 101. With illuminator 103 being on and illuminator 102 being off, objects A and B can both be perceived by the camera 101, since they are both further away from the area of specular reflection that is caused by the illuminator 103. Nevertheless, if there were an object closer to the area of specular reflection, that object might be much more difficult for the camera 101 to perceive while the illuminator 103 is on.
Returning to
This description will now set forth several definitions before returning to the main description of the state diagram 200 of
In this description and in the claims, an illuminator being “turned off” does not necessarily mean that light is not emitting from the turned off illuminator or that power is not being supplied to the turned off illuminator. Some illumination devices have a much faster turn on time if they are kept in an “off” state in which they have some power provided to the illuminator. Accordingly, in fast sampling interactive displays, the illuminator may be considered turned off, if it is still emitting light or being provided power, albeit to a perhaps much lesser extent than when in an “on” state. That said, illuminators may presently be powered-down and powered-up very quickly. Accordingly, in a very feasible embodiment, the dominating illuminator may be turned on, while the non-dominating illuminator may be turned off without any power at all.
Returning now to the state transition diagram 200 of
At the appropriate time, the phase management mechanism 130 transitions to the second phase (state transition arrow 212). In that state, the second illuminator is controlled to dominate over the first illuminator (state 222). The camera is then caused to take the second image (act 232 within state 222). For example, the image 300B of
Although the state transition diagram 200 illustrates a third and fourth state as well, the interactive display 100 may be limited to two phases, the first phase and the second phase, which may be repeated.
Returning to
The image construction mechanism 140 may construct the image in a variety of possible ways. In one example, the image construction mechanism 140 formulates the right half of the image using the right half of image 300A, which lacks the specular reflection. In that example, the left half of the image is formulated using the left half of the image 300B, which lacks a specular reflection. Accordingly, an entire image may be formulated without specular reflection once every two phases. Although the left half and right half of the final image may represent the image at slightly different times, this time might be minimized by increasing the sampling rate using a higher camera sampling rate and illuminator transition speed. Alternatively, the motion in front of the screen could be kept slower so as to prevent the central boundary in the final image from being too discontinuous. As another alternative, the user interface may be designed such that input is kept away from the central boundary at which the left and right halves of the image connect.
In another embodiment, rather than using the left half and the right half of the various images 300A and 300B, all of one image might be used except for the area of the spectral reflection. For example, image 300A might be used in its entirety except for the area of the spectral reflection 321. The area of the spectral reflection might then be populated using the corresponding area of the other image 300B. It is notable that the area within image 300B that corresponds to the spectral reflection 321 of image 300A lacks spectral reflection, and is entirely outside of the spectral reflection 322 of image 300B. Accordingly, the final image lacks any spectral reflection.
The boundary for what is included within the final image may be chosen differently so long as portions of each image are used. Significant benefit may be derived using the principles of the present invention if the areas that are included within any given position of the final position are chosen from the corresponding image that lacks spectral reflection at that position. However, after having read this description, there may be those who attempt to use the principles of the present invention to reduce, rather than eliminate, spectral reflections.
Such motivations may be due to technical limitations such as, for example, perhaps the spectral reflection from one of the images overlaps to some extent the spectral reflection of the other image. Thus, unless there are additional illumination phases and illuminators, elimination of the spectral image will not be feasible in the overlapping area of spectral reflection.
Motivations may also be for pragmatic reasons, such as attempts to design around the principles of the present invention in the mistaken belief that the principles of the present invention apply only if the spectral reflection is completely eliminated in the final image. Those of ordinary skill in the art will recognize, after having read this description, that significant benefit may be derived using the principles of the present invention, even if some areas of spectral reflection are chosen to be in the final image, so long as the spectral reflection is reduced in the final image as compared to at least one of the input images.
There may be more than two illumination phases. For example, if spectral reflections from the first two illuminators are overlapping somewhat, it may be helpful to have a third phase is which a third illuminator is turned on, while the first and second illuminators are turned off In the first and/or second phases, this third illuminator may be turned off Accordingly, it is more likely that portions of all three images may be selected so as to create an image that has significantly reduced or eliminated spectral reflections. Using these same principles, fourth or even more illumination phases may prove helpful to reduce spectral reflection given the principles of the present invention.
Illumination phases are helpful not just in cancelling specular reflection, but also in cancelling ambient light using the principles of the present invention. For instance, suppose that the illuminators are operating using infrared light. In many environments, there may be considerable amounts of ambient infrared light. For instance, the sun emits considerable amounts of infrared light. Such ambient light might also adversely impact the ability of the camera to detect the input object as distinguished from ambient light.
Referring to the state diagram 200 of
If three phases are used, the image construction mechanism might calculate the image using the following Equation (1):
FinalImage=(phase #1 right half image+phase #2 left half image)−phase #3 image (1)
Although four phases are shown in the state diagram 200 of
In order to simplify the operation of reducing or cancelling ambient light, a fourth phase may be used. Referring back to the state diagram 200 of
As represented by the timing diagram 400C of
Regardless, if four phases are used, the image construction mechanism 140 may use all four images in the final image. For instance, the right half of the first image (e.g., image 300A of
FinalImage right half=(phase #1 right half)−(phase #3 right half) (2)
On the other hand, the left half of the final image might be calculated using the following Equation (3)
FinalImage left half=(phase #2 left half)−(phase #4 left half) (3)
During the first phase, all of the corresponding first array of illuminators 502 (including interconnected illuminators 502A though 5020) illuminate, whereas all of the corresponding second array of illuminators 503 (including interconnected illuminators 503A through 503J) are turned off During the second phase, all of the corresponding second array of illuminators 503 are turned on, whereas all of the corresponding first array of illuminators 502 are turned off The array of cameras 501 (including cameras 501A through 501P) all take images at approximately the same time since they are interconnected. Each camera is responsible for imaging its corresponding grid portion of the display.
The structure 500 of
In the first phase, the illuminators 502 are on, and the illuminators 503 are off, the right half of each of the images for all of the even cameras 501I through 501P are read, whereas the left half of each of the images for all of the odd cameras 501A through 501H are read.
In the next phase, called “phase #3” in the
In the next phase, called “phase #2” in the
In the next phase, called “phase #4”), all illuminators are off, the right half of each of the images for all of the even cameras 501I through 501P are read, whereas the left half of each of the images for all of the odd cameras 501A through 501H are read.
For each grid area corresponding to a single camera in the array of cameras, the grid image may be obtained using Equations (2) and (3) presented above. The grid images may then be combined to form the entire image.
For completeness,
The phase management mechanism 600 also includes an illumination control mechanism 602, which may control the illuminators to be in the appropriate state given the phase. For instance, illumination control mechanism 602 may cause the first illuminator to turn on and the second illuminator to turn off when in the first phase (state 221 of
The phase management mechanism 600 also includes a camera control mechanism 603 that is configured to cause the cameras to take the appropriate images during the appropriate illumination phases (corresponding to acts 231 through 234 of
The phase management mechanism 600 may be external to the cameras and illuminators or may be partially or fully encompassed within one or more of the cameras and/or illuminators. The phase management mechanism 600 may be a centralized control module, or may be distributed across multiple modules.
Accordingly, the embodiments have been described in which an interactive display alternates illuminators to reduce or cancel specular reflections. The present invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.
| Number | Name | Date | Kind |
|---|---|---|---|
| 4586817 | Ehemann, Jr. | May 1986 | A |
| 4896029 | Chandler | Jan 1990 | A |
| 4972093 | Cochran | Nov 1990 | A |
| 4992650 | Somerville | Feb 1991 | A |
| 5109537 | Toki | Apr 1992 | A |
| 5153418 | Batterman | Oct 1992 | A |
| 5291564 | Shah | Mar 1994 | A |
| 5319214 | Gregory | Jun 1994 | A |
| 5436639 | Arai | Jul 1995 | A |
| 5483261 | Yasutake | Jan 1996 | A |
| 5515452 | Penkethman | May 1996 | A |
| 5526177 | Fantone | Jun 1996 | A |
| 5528263 | Platzker | Jun 1996 | A |
| 5541419 | Arackellian | Jul 1996 | A |
| 5684530 | White | Nov 1997 | A |
| 5821930 | Hansen | Oct 1998 | A |
| 5831601 | Vogeley | Nov 1998 | A |
| 5835692 | Cragun | Nov 1998 | A |
| 5900863 | Numazaki | May 1999 | A |
| 5920688 | Cooper | Jul 1999 | A |
| 5940076 | Sommers | Aug 1999 | A |
| 5973315 | Saldana | Oct 1999 | A |
| 5973689 | Gallery | Oct 1999 | A |
| 6061091 | Van de Poel | May 2000 | A |
| 6067369 | Kamei | May 2000 | A |
| 6088019 | Rosenberg | Jul 2000 | A |
| 6088612 | Blair | Jul 2000 | A |
| 6094509 | Zheng | Jul 2000 | A |
| 6111565 | Chery | Aug 2000 | A |
| 6128003 | Smith | Oct 2000 | A |
| 6154214 | Uyehara | Nov 2000 | A |
| 6181343 | Lyons | Jan 2001 | B1 |
| 6225620 | Campbell et al. | May 2001 | B1 |
| 6243492 | Kamei | Jun 2001 | B1 |
| 6266061 | Doi | Jul 2001 | B1 |
| 6269172 | Rehg | Jul 2001 | B1 |
| 6323503 | Hecht | Nov 2001 | B1 |
| 6339748 | Hiramatsu | Jan 2002 | B1 |
| 6340119 | He | Jan 2002 | B2 |
| 6400836 | Senior | Jun 2002 | B2 |
| 6414672 | Rekimoto et al. | Jul 2002 | B2 |
| 6448987 | Easty | Sep 2002 | B1 |
| 6469722 | Kinoe | Oct 2002 | B1 |
| 6476378 | Nougaret | Nov 2002 | B2 |
| 6478432 | Dyner | Nov 2002 | B1 |
| 6520648 | Stark | Feb 2003 | B2 |
| 6522395 | Barnji | Feb 2003 | B1 |
| 6529183 | MacLean | Mar 2003 | B1 |
| 6545663 | Arbter | Apr 2003 | B1 |
| 6600475 | Gutta | Jul 2003 | B2 |
| 6604682 | Wakamiya | Aug 2003 | B2 |
| 6614422 | Rafii | Sep 2003 | B1 |
| 6633338 | Pelsue et al. | Oct 2003 | B1 |
| 6636621 | Thebaud | Oct 2003 | B2 |
| 6639594 | Zhang | Oct 2003 | B2 |
| 6654007 | Ito | Nov 2003 | B2 |
| 6663244 | Wichner | Dec 2003 | B1 |
| 6690363 | Newton | Feb 2004 | B2 |
| 6710770 | Tomasi | Mar 2004 | B2 |
| 6714221 | Christie | Mar 2004 | B1 |
| 6720949 | Pryor | Apr 2004 | B1 |
| 6750877 | Rosenberg | Jun 2004 | B2 |
| 6781069 | Silverstein | Aug 2004 | B2 |
| 6788411 | Lebens | Sep 2004 | B1 |
| 6788813 | Cooper | Sep 2004 | B2 |
| 6791530 | Vernier | Sep 2004 | B2 |
| 6804396 | Higaki | Oct 2004 | B2 |
| 6812907 | Gennetten | Nov 2004 | B1 |
| 6840627 | Olbridch | Jan 2005 | B2 |
| 6888960 | Penev | May 2005 | B2 |
| 6895104 | Wendt | May 2005 | B2 |
| 6956963 | Ulrich | Oct 2005 | B2 |
| 6959102 | Peck | Oct 2005 | B2 |
| 6975360 | Slatter | Dec 2005 | B2 |
| 7006128 | Xie | Feb 2006 | B2 |
| 7007236 | Dempski | Feb 2006 | B2 |
| 7058217 | Thomber | Jun 2006 | B2 |
| 7075687 | Lippert | Jul 2006 | B2 |
| 7079238 | Vaez-Iravani et al. | Jul 2006 | B2 |
| 7084859 | Pryor | Aug 2006 | B1 |
| 7095401 | Liu | Aug 2006 | B2 |
| 7119897 | Vaez-Iravani et al. | Oct 2006 | B2 |
| 7120280 | Biswas | Oct 2006 | B2 |
| 7136537 | Pilu et al. | Nov 2006 | B2 |
| 7144117 | Kojima | Dec 2006 | B2 |
| 7161578 | Schneider | Jan 2007 | B1 |
| 7168813 | Wong | Jan 2007 | B2 |
| 7204428 | Wilson | Apr 2007 | B2 |
| 7268774 | Pittel | Sep 2007 | B2 |
| 7281661 | Zhu et al. | Oct 2007 | B2 |
| 7310085 | Holloway | Dec 2007 | B2 |
| 7327376 | Shen | Feb 2008 | B2 |
| 7342574 | Fujioka | Mar 2008 | B1 |
| 7372977 | Fujimura | May 2008 | B2 |
| 7375810 | Nikoonahad et al. | May 2008 | B2 |
| 7379562 | Wilson | May 2008 | B2 |
| 7391507 | Chism, II | Jun 2008 | B2 |
| 7394459 | Bathiche | Jul 2008 | B2 |
| 7397464 | Robbins | Jul 2008 | B1 |
| 7398927 | Olmstead et al. | Jul 2008 | B2 |
| 7404146 | Bennetts | Jul 2008 | B2 |
| 7418671 | Hama | Aug 2008 | B2 |
| 7467380 | Kurlander | Dec 2008 | B2 |
| 7492357 | Morrison | Feb 2009 | B2 |
| 7499027 | Brigham | Mar 2009 | B2 |
| 7515143 | Keam | Apr 2009 | B2 |
| 7519223 | Dehlin | Apr 2009 | B2 |
| 7525538 | Bathiche | Apr 2009 | B2 |
| 7576725 | Bathiche | Aug 2009 | B2 |
| 7613358 | Wilson | Nov 2009 | B2 |
| 7630002 | Jenkins | Dec 2009 | B2 |
| 7665041 | Wilson et al. | Feb 2010 | B2 |
| 7701437 | Thursfield | Apr 2010 | B2 |
| 7724242 | Hillis et al. | May 2010 | B2 |
| 7728821 | Hillis et al. | Jun 2010 | B2 |
| 7787706 | Yee | Aug 2010 | B2 |
| 7907128 | Bathiche | Mar 2011 | B2 |
| 7911444 | Yee | Mar 2011 | B2 |
| 8060840 | Billow | Nov 2011 | B2 |
| 20010012001 | Rekimoto | Aug 2001 | A1 |
| 20010036299 | Senior | Nov 2001 | A1 |
| 20010055006 | Sano | Dec 2001 | A1 |
| 20020006786 | Mine | Jan 2002 | A1 |
| 20020041700 | Thebaud | Apr 2002 | A1 |
| 20020113882 | Pollard | Aug 2002 | A1 |
| 20020125411 | Christy | Sep 2002 | A1 |
| 20020181774 | Ishikura | Dec 2002 | A1 |
| 20020186221 | Bell | Dec 2002 | A1 |
| 20030156756 | Gokturk | Aug 2003 | A1 |
| 20030161524 | King | Aug 2003 | A1 |
| 20040001113 | Zipperer | Jan 2004 | A1 |
| 20040005920 | Soltys | Jan 2004 | A1 |
| 20040090524 | Belliveau | May 2004 | A1 |
| 20040155902 | Dempski | Aug 2004 | A1 |
| 20040196371 | Kono | Oct 2004 | A1 |
| 20050050476 | SanGiovanni | Mar 2005 | A1 |
| 20050064936 | Pryor | Mar 2005 | A1 |
| 20050110964 | Bell | May 2005 | A1 |
| 20050122306 | Wilcox | Jun 2005 | A1 |
| 20050122308 | Bell | Jun 2005 | A1 |
| 20050134853 | Ingleson et al. | Jun 2005 | A1 |
| 20050151850 | Ahn | Jul 2005 | A1 |
| 20050162381 | Bell | Jul 2005 | A1 |
| 20050212753 | Marvit | Sep 2005 | A1 |
| 20050226467 | Hatano | Oct 2005 | A1 |
| 20050227217 | Wilson | Oct 2005 | A1 |
| 20050238201 | Shamaie | Oct 2005 | A1 |
| 20050245302 | Bathiche | Nov 2005 | A1 |
| 20050255434 | Lok | Nov 2005 | A1 |
| 20060033835 | Pollard et al. | Feb 2006 | A1 |
| 20060034492 | Siegel | Feb 2006 | A1 |
| 20060036944 | Wilson | Feb 2006 | A1 |
| 20060043194 | Barkan et al. | Mar 2006 | A1 |
| 20060056662 | Thieme | Mar 2006 | A1 |
| 20060092267 | Dempski | May 2006 | A1 |
| 20060152741 | Quist | Jul 2006 | A1 |
| 20060163355 | Olmstead et al. | Jul 2006 | A1 |
| 20060178212 | Penzias | Aug 2006 | A1 |
| 20060202974 | Thielman | Sep 2006 | A1 |
| 20070063981 | Galyean | Mar 2007 | A1 |
| 20070075163 | Smith et al. | Apr 2007 | A1 |
| 20070126717 | Cohen | Jun 2007 | A1 |
| 20070279494 | Aman | Dec 2007 | A1 |
| 20080036732 | Wilson et al. | Feb 2008 | A1 |
| 20080122786 | Pryor | May 2008 | A1 |
| 20090121894 | Wilson et al. | May 2009 | A1 |
| 20090262070 | Wilson | Oct 2009 | A1 |
| 20110181551 | Yee | Jul 2011 | A1 |
| 20120023423 | Bilow | Jan 2012 | A1 |
| Number | Date | Country |
|---|---|---|
| 0690407 | Jan 1996 | EP |
| 1233606 | Feb 2002 | EP |
| 2756077 | May 1998 | FR |
| 2004-259247 | Sep 2004 | JP |
| 2004-319364 | Nov 2004 | JP |
| 2006-031941 | Feb 2006 | JP |
| 2007-514242 | May 2007 | JP |
| 102001005203 | Jan 2001 | KR |
| 1020040072652 | Aug 2004 | KR |
| 1020050051554 | Jun 2005 | KR |
| WO 9118383 | Nov 1991 | WO |
| WO 97-14075 | Apr 1997 | WO |
| WO 9819292 | May 1998 | WO |
| WO 0184251 | Nov 2001 | WO |
| WO 2005-057399 | Jun 2005 | WO |
| Number | Date | Country | |
|---|---|---|---|
| 20080180530 A1 | Jul 2008 | US |